Defending Against Adversarial Examples by Regularized Deep Embedding
Authors: Yao Li
Martin Renqiang Min
Wenchao Yu
Cho-Jui Hsieh
Thomas Lee
Erik Kruus
Venue:  (2020)

Abstract: Recent studies have demonstrated the vulnerability of deep convolutional neural networks against adversarial examples. Inspired by the observation that the intrinsic dimension of image data is much smaller than its pixel space dimension and the vulnerability of neural networks grows with the input dimension, we propose to embed high-dimensional input images into a low-dimensional space to perform classification. However, arbitrarily projecting the input images to a low-dimensional space without regularization will not improve the robustness of deep neural networks. We propose a new framework, Embedding Regularized Classifier (ER-Classifier), which improves the adversarial robustness of the classifier through embedding regularization. Experimental results on several benchmark datasets show that, our proposed framework achieves state-of-the-art performance against strong adversarial attack methods.

Under review as a conference paper at ICLR 2020
DEFENDING AGAINST ADVERSARIAL EXAMPLES BY REGULARIZED DEEP EMBEDDING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent studies have demonstrated the vulnerability of deep convolutional neural networks against adversarial examples. Inspired by the observation that the in- trinsic dimension of image data is much smaller than its pixel space dimension and the vulnerability of neural networks grows with the input dimension, we pro- pose embed high-dimensional input images into low-dimensional space perform classification. However, arbitrarily projecting the input images to low- dimensional space without regularization will not improve the robustness of deep neural networks. We propose a new framework, Embedding Regularized Clas- sifier Ela-Classif,er). which improves the adversarial robustness of the classifier through embedding regularization. Experimental results several benchmark datasets show that, our proposed framework achieves state-of-the-art performance against strong adversarial attack methods.
INTRODUCTION
Deep neural networks (DNNs) have been widely used for tackling numerous machine learning prob- lems that were once believed to be challenging. With their remarkable ability of fitting training data, DNNs have achieved revolutionary successes in many fields such as computer vision, natu- ral language progressing, and robotics. However, they were shown to be vulnerable to adversarial examples that are generated by adding carefully crafted perturbations original images. The ad- versarial perturbations can arbitrarily change the network's 's prediction but often too small to affect human recognition (Szegedy et al.||20 [2133 Kurakin al. 2016). This phenomenon brings out security concerns for practical applications of deep learning.
Two main types of attack settings have been considered in recent research (Goodfellow et al. Carlini & Wagner] [017aa] Chen et al.] 2017 Papernot et al.] (017): black-box and white-box settings. In the black-box setting, the attacker can provide any inputs and receive the corresponding predictions. However, the attacker cannot get access to the gradients whereas in the white-box setting, the attacker is allowed to analytically compute the model's 's gradi- ents, and have full access to the model architecture and weights. In this paper, focus defending against the white-box attack which is the harder task.
model parameters under this setting;
Recent work (Sbbie- et al., 2018) presented both theoretical arguments and empirical one-to-one relationship between input dimension and adversarial vulnerability, showing that the vulnerability of neural networks grows with the input dimension. Therefore, reducing the data di- mension may help improve the robustness of neural networks. Furthermore, a consensus in the high- dimensional data analysis community is that, a method working well on the high-dimensional data is because the data is not really of high- digh-dimension (Levina & Bickel| 2005). These high-dimension low dimensional space. Hence, carefully reducing
data, such images, are actually embedded the input dimension may improve the robustness of the model without sacrificing performance.
Inspired by the observation that the intrinsic dimension of image data is actually much smaller than its pixel space dimension (Levina & Bickel 2005) and the vulnerability of model grows with its input dimension Siionn-Gabrie et al., 2018), we propose a defense framework that embeds input images into low-dimensional space using deep encoder and performs classification based on the latent embedding with a classifier network. However, an arbitrary projection does not guaran- tee improving the robustness of the model. because there are lot of mapping functions including non-robust ones from the raw input space to the low-dimensional space capable of minimizing the classification loss. To constrain the mapping function, employ distribution regularization in the

---

Under review as a conference paper at ICLR 2020
embedding space leveraging optimal transport theory. We call Embedding Regularized Classifier classsssif..rr. To be more specific, we introduce a discrimina- tor in the latent space which tries to separate the generated code vectors from the encoder network and the ideal code vectors sampled from a prior distribution, i.e., a standard Gaussian distribution. Employing similar powerful competitive mechanism demonstrated by Generative Adversarial Networks (Goodfellow et al. 2014), the discriminator enforces the embedding space of the model follow the prior distribution.
classification framework
In our ER-Classifier framework, the encoder and discriminator structures together project the input data to a low-dimensional space with nice shape, then the classifier per- forms prediction based on the low- dimensional embedding. Based on the optimal transport theory, the pro- posed ER-Classifier minimizes the discrepancy between the distribution of the true label and the distribu- tion of the framework output, thus only retaining important features for classification in the embedding space. With small embedding dimension, the effect of the adversarial perturbation is largely diminished through the projection process.
Re(K)
Classiia GF-)
x-P
Generated Code
Ideal Code
Disciininaoo Ry()
2-P2
Figure 1: Overview of ER-Classifier framework
We compare ER- ER-Classifier with other state-of of-the-art defense methods on MNIST, CIFARIO, STL10 and Tiny Imagenet. Experimental results demonstrate that our proposed ER- RR-Classifier outperforms other methods by a large margin. To sum up, this paper makes the following three main contribu- tions:
A novel unified end-to-end robust deep neural network framework against adversarial tacks is proposed, where the input image is first projected to low-dimensional space and then classified.
An objective is induced to minimize the optimal transport cost between the true class dis- tribution and the framework output distribution, guiding the encoder and discriminator to project the input image low-dimensional space without losing important features for classification.
Extensive experiments demonstrate the robustness of proposed ER-Classife frame- work under the white-box attacks, and show that ER-Classifier outperforms other state-of- the-art approaches on several benchmark image datasets.
As far as we know, our approach is the first that applies optimal transport theory, i.e., a Wasser- stein distance regularization, bottleneck embedding layer of deep neural network in purely supervised learning setting without considering any reconstruction loss, although optimal transport theory or discriminator loss has been applied to generative models in an unsupervised learning setting (Makhzani et al., 2015) Tolstikhin et al.) 20177;; (2) Our method is also the first that es- tablishes the connection between a Wasserstein distance regularization and the robustness of deep neural networks for defending against adversarial examples.
RELATED WORK
In this section, we summarize related work into three categories: attack methods, defense mecha- nisms and optimal transport theory. We first discuss different white-box attack methods, followed by description of different defense mechanisms against, and finally optimal transport theory.
2.1 ATTACK METHIOSS
Under the white-box setting, attackers have all information about the targeted neural network, in- cluding network structure and gradients. Most white-box attacks generate adversarial examples based the gradient of loss function with respect the input. An algorithm called fast gradi- ent sign method (FGSM) was proposed in (Goodfellow et al.) which generates adversarial ex- amples based the sign of gradient. Many other white-box attack methods have been proposed
2

---

Under review as a conference paper at ICLR 2020
recently (Moosavi-Dezfooli al., 2016: Chen et al. 2018 Madry et al. 2017 Carlini & Wagner 2017b), and among them C&W and PGD attacks have been widely used to test the robustness of machine learning models.
C&W attack: The adversarial attack method proposed by Carlini and Wagner (Carlini & Wagner, 2017b) is one of the strongest white-box attack methods. They formulate the adversarial example generating process as an optimization problem. The proposed objective function aims at increasing the probability of the target class and minimizing the distance between the adversarial example and the original input image. Therefore, C&W attack can be viewed as a gradient-descent based adversarial attack.
The projected gradient descent attack is proposed by (Madry et al. 2017), which finds adversarial examples in an e-ball of the image. The PGD attack updates in the direction that decreases the probability of the original class most, then projects the result back to the c-ball of the input. An advantage of PGD attack over C&W attack that it allows direct control of distortion level by changing E, while for C& W attack, one can only do so indirectly via
PGD attack:
tuning.
Both C& W attack and PGD attack have been frequently used to benchmark the defense algorithms due to their effectiveness (Athalye et al. 2018). In this paper, we mainly use -PGD untargeted attack to evaluate the effectiveness of the defense method under white-box setting.
Instead of crafting different adversarial perturbation for different input image, an algorithm was proposed by (Moosavi-Dezfooli et al., 2017) to construct a universal perturbation that causes natural images to be misclassified. However, since this universal perturbation is image-agnostic, it is usually larger than the image-specific perturbation generated by PGD and C&W.
2.2 DEFENSE MECHIANISMS
Many works have been done improve the robustness of deep neural networks. To defend against adversarial examples, defenses that aim to increase model robustness fall into three main categories: augmenting the training data with adversarial examples to enhance the existing classifiers (Madry) et al. 2017: Na et al., 2017, Goodfellow et al.)); ii) leveraging model-specific strategies to enforce model properties such as smoothness (Papernot et al. 2016); and, iii) trying to remove adversarial perturbations from the inputs (Xie et al. 2017] (Samangouei et al. 20188) Meng & Chen 2017). We
select three representative methods that are effective under white-box setting.
Augmenting the training data with adversarial examples can increase the robustness of the deep neural network. Madry et al. (Madry et al. (017) recently introduced a min- max formulation against adversarial attacks. The proposed model not only trained on the original dataset but also adversarial example in the e-ball of each input image.
Adversarial training:
Random Self-Ensemble: Another effective defense method under white-box setting is RSE (Lu
et al. 2017). The authors proposed a "noise layer". which fuses output of each layer with Gaussian noise. They empirically show that the noise layer can help improve the robustness of deep neural networks. The noise layer is applied in both training and testing phases, so the prediction accuracy will not be largely affected.
Defense-GAN:
Defense-GAN (Samangouei et al. 2018) leverages the expressive capability of GANs to defend deep neural networks against adversarial examples. It is trained project input images onto the range of the GAN's generator to remove the effect of the adversarial perturba- tion. Another defense method that uses the generative model to filter out noise is MagNet proposed by (Meng & Chen 2017). However, the differences between ER- ER-CCassifier and the two methods are obvious. ER-Classifier focuses on reducing the dimension, and performing classification based on the low-dimensional embedding, while Defense-GAt and MagNet mainly apply the generative model to filter out the adversarial noise, and both Defense-GAt and MagNet perform classifica- tion on the original dimension space. (Samangouei et al.) 2018) showed that Pefense-GAN is more robust than MagNet, so we only compare with Defense- GAN in the experiments.
Other Related Methods: Zhang et al. 2018 regularizes the latent space with Gaussian Mixture Model and applies KL-divergence to do the optimization. However, our method employs a simple but nice-shaped Gaussian prior for Wasserstein distance minimization to constrain the global shape of the latent embeddings, while permitting high freedom for the shapes of individual class distribu- tions of latent embeddings. We want the classifier decide the optimal class-specific distributions of
3

---

Under review as a conference paper at ICLR 2020
latent embeddings. Miyato et al. (2018) shares similar idea to adversarial learning, but it employs virtual labels generated by a current classifier to identify search directions that can smooth the output label distribution of the classifier and is best suitable for semi-supervised learning. Please note that both methods in (Zhang et al. 2018) Miyato et al. 2018) are designed for improving generalization performance but not for defending against adversarial examples. A recent paper (myas et al.) 2019) shows that adversarial examples are purely human phenomenon and models tend to learn features that are not robust yet generalize well. We show that our Wasserstein distance regularization helps to identify robust features, which will be discussed later.
Notations In this paper, we use (x and 22 distortion metries to measure similarity. We report lxc distance in the normalized [0, 1] space, so that a distortion of 0.031 corresponds to 8/256, and l2 distance as the total root-mean root-mean-square square distortion normalized by the total number of pixels.
We use calligraphic letters for sets (i.e., X'), capital letters for random variables (i.e., X)), and lower case letters for their values (i.e., X). The probability distributions are denoted with capital letters (i.e., Px) and corresponding densities with lower case letters (i.e., px).
3 PROPOSED FRAMEWORK: EMBEDDING REGULARIZED CLASSIFIER
3.1 FRAMEWORK DETAILS
We propose novel defense framework, ER-Classifier, which aims at projecting the image data to a www-dimesiioall space to remove noise and stabilize the classification model by minimizing the optimal transport cost between the true label distribution Py and the distribution of the ER- C.Casssiiier output (Pc). An overview of the framework shown in Figure [ The encoder and discriminator structures together help diminish the effect of the adversarial perturbation by projecting input data to a space lower dimension, then the classifier part performs classification based on the low- dimensional embedding.
Mathematically, input images XE E = .Rd are projected to a low-dimensional embedding vector Z E Z = H through the encoder Qu. The discriminator D., discriminates between the generated code 2 Q (Z and the ideal code Z Pz. The classifier C, performs classification based on the generated code Ž, producing output U E U E" where is the number of classes. The label of is denoted as Y E U.
The Kantorovich's 's distance induced by the optimal transport problem is given by inf
Wo(Pr, Pe) rEP(Y-PP, UU-))) B(y,))-r {c(Y,U)):
where P(Y ~ Pr. ~ Pc) is the set of all joint distributions of (Y,U) with marginals Py and Pc. and c(y,") 4 x x R4 is any measurable cost function. W.(Py, Pc) measures the divergence between probability distributions Py and Pc. When the probability measures are on a metric space, the p-th root of W. is called the p- p-Wasserstein distance.
To minimize the Wasserstein distance between the distribution of the true label (Py) and the dis- tribution of the ER-Classifier output (Pc), we can prove that sufficient to find a conditional distribution Q x such that its marginal distribution Qz is identical to a prior distribution Pz. The theorem and the proof are deferred to the Appendix. In this paper, we apply standard Gaussian as our prior distribution Pz, but other priors may be used for different cases. The final objective of ER-Classifier is: inf X
Epx Eo(zx) {((f(x), C(Z)) + XD(Qz. P2),
(1)
where Q can be a deterministic encoder as focused by this paper due to its simplicity or stochastic encoder as the one in a standard Variational Autoencoder, > 0 is hyper- ppram-prrreeter and D is an arbitrary divergence between Qz and Pz.
To estimate the divergences between Qz and Pz, we apply a GAN-based framework, fitting a discriminator to minimize the I-Wasserstein distance between Qz and Pz:
inf W W(Qz, 1ep(2=Qg.2-Pa) 2~PP)
E 1(222)r112 Z||.
4

---

Under review as a conference paper at ICLR 2020
We have also tried the Saisenn-hannon divergence, but expected, Wasserstein distance provides more stable training and better results. When training the framework, the weight clipping method proposed in Wasserstein GAN (Arjovsky et al., 2017) is applied to help stabilize the training of discriminator D The training algorithm is summarized in Algorithm[]
tow-dimensional space, resulting in gen- erated code (2). Another ideal code (z) sampled from the prior distribution, and the discriminator D, discriminates between the ideal code (positive data) and the generated code (negative data). The classifier (C,) predicts the image label based on the generated code (?).
At training stage, the encoder Q₂ first maps the input
Algorithm Training ER-Classifier
1: Input: Regularization coefficient À > 0, encoder Qø, discriminator D and classifier C,,
2: Note: l stands for the cross-entropy loss.
3: while (1,77,7 not converged do
4: Sample (01131), Vi),..., (Z yn)) from the training set
5: Sample {"1, ... Zn from the prior Pz
6: Directly obtain or sample ž; from (Z)z)) for i 1,.., n
7: Update D,, by ascending the following objective by 1-step Adam:
(2) (
1=1
8: Update Qs and C, by descending the following objective by 1-step Adam:
((C)(Q((Z))).)
9: Update Qs by ascending the following objective by 1-step Adam:
(Q,(()))
10: end while
At inference time, only the encoder Qg and the classifier C, are used. The input image is first mapped to low-dimensional space by the encoder (ä ( then the latent code ă is fed into the classifier to obtain the predicted label.
The main goal of ER-Classifier is leveraging input space dimension reduction to remove adversarial perturbations. Therefore, other defense methods can also benefit from this property. Our framework trained with min-max robust optimization (Madry al. 2017).
3.2 JUSTIFICATIONS OF OUR FRAMEWORK
There are two Wasserstein distances (W-distances) in our framework. One is the W-distance hetween the aggregated latent embedding distribution Q(z) and the prior distribution Pzz and the other one is the W-distance between the true label distribution Py and the distribution of the ER-Classifier output (Pc). In Algorithm T we are minimizing the first one. The theorem in the Appendix shows that minimizing the first W-distance in combination with minimizing standard cross-entropy loss as done Algorithm i is equivalent minimizing the second W-distance, which guarantees that the training process is not distracted from the main goal of the framework, classification. That is to say, Algorithm|] [ will result classifier with the following property: the global output distribution of the classifier will match the global ground-truth label distribution in the data matter whether the encoder Q & is deterministic or stochastic (the second W-distance is automatically minimized),
It's hard to analyze the importance of the theorem in the Appendix if we just look at a deterministic encoder. Let's convert this deterministic encoder to a stochastic encoder that outputs a Gaussian with fixed variance and the mean being the same its corresponding deterministic version. The theory tells us that, by minimizing the first W-distance over all sampled z from this stochastic encoder and the standard cross-entropy loss we will automatically minimize the second W-distance
5

---

Under review as a conference paper at ICLR 2020
and preserve the global label frequency in the dataset, even though these 2's are only c-close the deterministic encoding features of training data.
Moreover, find that minimizing the W-distance helps the encoder identify some robust features instead of non-robust features (yyas et al, 2017). because our proposed regularization constrains the c-ball around each Qs (Z)X)) to contribute to preserving the global label distribution in the data, even with X integrated out. From this perspective, we can view our proposed framework as "a supervised variant" of Generative Adversarial Network Wasserstein Autoencoder in which the Generator or Decoder is replaced by Classifier that generates labels from low-dimensional latent embeddings preserving global label frequency in the training dataset. Replacing W-distance with KL divergence loses all these nice properties.
In our framework, we use a simple but nice-shaped Gaussian prior Pz for W-distance minimization to constrain the global shape of the latent embeddings, while permitting high freedom for the shapes of individual class distributions latent embeddings. We want the classifier to decide the optimal class-specift distributions of latent embeddings. In addition, it is interesting to explore how to set e-ball to make sure the stochastic encoder to best align the latent embedding z to human-perceived robust features, which will be left as future work.
4 EXPERIMENTS
In this section, compare the performance of our proposed algorithm (ER-Classifier) with other state-of-the-art defense methods on several benchmark datasets:
MNIST (LeCun 1998): handwritten digit dataset, which consists of 60, 000 training im- ages and 10, 000 testing images. Theses are 28 x 28 black and white images in ten different classes.
CIFAR1O (Krizhevsky & Hinton, 2009): natural image dataset, which contains 50, 000 training images and 10, 000 testing images in ten different classes. These are low resolution 32 x 32 color images.
STL10 (Coates et al., 201 :: color image dataset similar to CIFARIO, but contains only
5. 000 training images and 8, 000 testing images in ten different classes. The images are of higher resolution 96 x 96.
Tiny Imagenet (Deng et al. 2009); ; a subset of Imagenet dataset. Tiny Imagenet has 200 classes, and each class has 500 training images, 50 testing images, making a challenging benchmark for defense task. The resolution of the images is 64 x 64.
Various defense methods have been proposed to improve the robustness of deep neural networks. Here we compare our algorithm with state-of-the-art methods that are robust in white-box setting. Madry's adversarial training (Madry's Adv) has been recognized as one of the most successful defense method in white-box setting, shown in (Athalye et al. 2018).
Random Self-Ensemble (RSE) method introduced by (Liu et al. in the neural network, achieving similar performance to Madry's s adversarial training algorithm.
2017) adds stochastic components
Another method we would like to compare with is Defense-GAN (Samangouei et al., 2018). first trains generative adversarial network to model the distribution of the training data. At inference time, it finds close output to the input image and feed that output into the classifier. This process "projects" input images onto the range of GAN's generator, which helps remove the effect of ad- versarial perturbations. In (Samangouei et al. 2018), the author demonstrated the performance of Defense- GAN on MNIST and Mashion-MNIST. so will compare our method with Defense-GAN on MNIST.
Since the main goal of ER-Classifier is using dimension reduction to improve adversarial robustness, other defense methods can also benefit from this property. The proposed ER-Classifier is trained with min-max robust optimization (Madry et al., 2017). To demonstrate the dimension reduction ability of ER-Classifier, we include a variant ER-Classifier which trains ER-Classifies without min-max robust optimization.
4.1 EVALUATE MODELS UNDER WHITE-BOX 1 -PD ATTACK
In this section, we evaluate the defense methods against lx- PGD untargeted attack, which is one of the strongest white-box attack methods. Models are evaluated under different distortion level (e),

---

Under review as a conference paper at ICLR 2020
MNIIT
CIFAROO
STL10
Malyy Ale
23 Lizergeh atuck)
681 081 66r 2:4 (arerigh atlcck)
530 021 221
5005 530g 2005 0003
spallon b4reng sblakk
Figure 2:
Testing accuracy under PPG attack on four different datasets: MNIST, CIFARIO, STLIO and Tiny Imagenet.
and the larger the distortion the stronger the attack. Depending on the image scale and type, different datasets are sensitive to different strength of attack.
Models on MNIST are evaluated under distortion level from 0 to 0.4 by 0.025. Models on CIFARIO and STL10 are evaluated under [0, 0.06, 0.005]. Models on Tiny Imagenet are evaluated under c E [0, 0.02, 0.002]. As mentioned in the notation part, all the distortion levels are reported in the normalized [0, 1] space. The experimental results shown in Figure 2 To demonstrate the results more clearly, we show part of the results in Table
Based on Figure 2 and Table ! can see that ERR-Clasifier is the most robust one variety of datasets. ER-Classifier without min-max ro- bust optimization can also improve the robustness of deep neural net-
Data
Defense
0.1
0.2
0.3
0.4
Madry's Adv
98.7
97.5
93.8
85.5
20.8
MNIST
ER ER-Classifier
99.1
98.7
97.2
94.9
71.1
Data
Defense
0.015
0.03
0.045
0.06
work. Compare the performance of
ER-Classifier" with the performance of model without defense method (No Defense), we can see that ER- Classifier is much more robust than the model with no defense method on all benchmark datasets. Besides, when the distortion level (c) is large, ER- RR-CClasifier tends to perform bet- ter than other state-of-the-art defense methods on MNIST, CIFARI0 and Tiny Imagenet. This phenomenon is obvious on CIFAR10 and it even per- forms better than ER-Classifier when the attack strength is strong. The reason easier to regularize the embedding space.
CIFARIO
Madry's 's Adv 82.6
68.0
42.3
21.6
12.0
ER-Clussifier
84.0
67.5
51.3
35.8
23.3
Madry's 's Adv
63.6
53.5
36.8
25.0
18.7
STL10
ER- ER-Classifier
60.7
52.1
40.3
30.6
24.5
Data
Defense
0.004
0.01
0.016
0.02
Madry's Adv
$7.3
48.6
26.5
15.1
12.0
Tiny Imagenet
FR- FR-Classifier
54.6
36.7
25.6
21.1
Table I: Testing accuracy (%) under different strength of PGD attacks. The table shows the results of ER-Classifier and Madry's 's adversarial training (Madry's Adv). The better accuracy marked in bold.
might be that without min-max robust optimization, it is
We also compare Defense-GAN with our method ER- Classifier on MNIST. Although Defense-GAN was shown to be partly broken by (Athalye al.) 2018 Ilas et al. 2017). both ER-Classifier and Defense- GAN share the similar idea of projecting the input to a learned manifold, and comparing to Defense-GAN important to demonstrate the advantage of our novel Wassserstein distance regularization. Please note that Defense-GAN is not our major comparison baseline in this paper. Both ER-Classifier and Defense-Gar GAN are evaluated against the L C& untargeted at- tack, one of the strongest white-box attack proposed in (Carlini & Wagner 2017b). Defense- GAN evaluated using the method proposed in (Athalye et al. 2018). and the code is available on githut ER-CClasiffer is evaluated against 12-C& W untargeted attack with the same hyper-parameter val- ues as those used in the evaluation of Defense-GAN. The results under le 0.005 threshold are shown in Table Based on Table
Method
Testing Accuracy
Defense-Gan GAN
55.0
UR-CCassifier
99.1
Table 2: Testing accuracy (%) of two de- fense methods under C&W attack with L2 S 0.005.
Classifier is much more robust than Defense-GAN un-
Publicly available at
bttps://ggithi htw.:/tttuu.o.ooaa.oommoo shatha http://aithtaaaall//ooooeeeeeeee
tree/mmsterrdeeeene
7

---

Under review as a conference paper at ICLR 2020
MMST
CIFAR10
STL10
Tiny Imagenet
2.-
Mateeens VIE-AA ALA
Denase VEGGA EDA
Mo Neloeess MEDA
04-
titere ChCoss CLA
tostire accurso
RCA
ain
8015
opsten of attark)
epeton (strengtt of athack) epstoon (ttrnggth afthe sthack)
(eteegth
Figure 3: Testing accuracy of E-CLA, VAE-CLA and ER-CClassifier" under -PG attack on four different datasets: MNIST, CIFARIO, STL10 and Tiny Imagenet.
der the l2 < 0.005 threshold. Since (Samangouei et al. 2018) did not evaluate Defense-GAN on CIFARIO, STL10 and Tiny Imagenet. without details of GAN structure, we can not compare with Defense-GAN on these datasets.
4.2 EVALUATE MODELS UNDER BLACK-BOL BOX ATTACK
We evaluate Madry's 's adversarial training, classsssi,,,r. and ER Classifier against recently pro- posed black-box attack method called Nattack (Li et al. 2019 on CIFARIO. Nattack is only performed the first 100 images of CIFAR10 since the attack process takes long time. We port the accuracy = number of correctly classified number of attacked images (exactly 100). The accuracy of Madry's adv, ER-Classifirr, and ER-Classifier" is, respectively, 38%, 43%, and 32%. ER-Classifier still outperforms Madry's adv.
4.3 EVALUATE THE EFFECT OF DISCRIMINATOR
ER- Classifier framework consists of three parts, and the classification task is done by the encoder Q₂ and classifier C,. Without the discriminator part, the encoder can also project the input images to low-dimensional space. However, arbitrarily projecting the images to low-dimensional space with only the encoder part cannot improve the robustness of the model. In contrast, sometimes decreases the robustness of the model.
To show that arbitrarily projecting the input images to a low-dimensional space can not improve the robustness, we fit framework with only the encoder and classifier part (E-CLA), where the encoder and classifier have the same structures as in ER-Classifier, and compare E-CLA with the ER-Classifier framework. For fair comparison, both structures are trained without min-max robust optimization. The results are shown in Figure 3
robust than just the encoder and classifier structure on MNIST, CIFARIO and Tiny Imagenet. It is also more robust on STLI0 but not that much. The reason might be that there are only 5, 000 training images in STL10 and the resolution is 96 x 96. Therefore, it is harder to learn good embedding with limited amount of images. However, even when the number of training images is limited, ER- Classifier still much more robust than the E-CLAA structure. This observation demonstrates that regularization on the embedding space helps improve the adversarial robustness. Notice that the performance of E-CLA structure is similar to the performance of model without defense method on CIFAR 10, STL10 and Tiny Imagenet, and worse on MNIST, which means the robustness of RR-assssifier does not come from the structure design.
Based on Figure 3 can observe that ER-Classifier is much
Variational auto-encoder can project the images to low-dimensional space and use Kullback-Leibler divergence loss to regularize the embedding distribution, which does not need discriminator struc- ture. Therefore, we also tried VAE-CLA, which applies Variational auto-encoder structure to do the projection and regularization. The experimental results in Figure show that VAE-CLA does not perform as well as ER-Classifier. Based on the observation of the Kullback-Leibler loss and classification loss during the training process, it seems difficult for VAE-CLA balance between the two tasks. The reason might be that Kullback-Leibler distances are not sensible cost functions when learning distributions supported by low dimensional manifolds (Arjovsky et al.
2017)
8

---

Under review as a conference paper at ICLR 2020
4.4 PRIOR SELECTION
ER- ER-Classifier does not have restrictions on the choice of prior. However, the selec- tion of prior important as imposes different restrictions on the embedding space. Three different prior distributions are tested on MNIST and CIFARIO datasets. They are standard Gaussian, Uniform( -3, 3. 3) and Cauchy(O, 1), where Cauchy(O, 1) has the same support standard Gaussian but is heavy tailed and 99.7% of the standard Gaussian points lies within 3.. All the models are trained without min-max robust optimization. and the experimental results are shown in Figure 4 Based on the results, all three priors work well, but standard Gaussian performs best on both datasets.
MNIST
CIFARI0
1.5
Gaundan
Gaustion
tasting acouracy
0.7
0.3
0.-
0.5
0.00
0.0 0.05
opsiion (strongth attack)
opsiton (strength attack)
Figure 4: Testing accuracy of models with different prior distributions under -PG attack.
Ding et al. (Ding et al. 2019) prove that adversarial robustness is sensitive to the input data dis- tribution, and if the data uniformly distributed in the input space, no algorithm can achieve good robustness. They also empirically show that corneree/conceetraaeed data distributions tend to achieve better robustness. This helps explain why regularizing the embedding space can help improve ro- bustness. Though the projection process reduces the input dimension, the embedding space is still large. Prior distribution helps push the embedding space to be more concentrated, reducing the valid perturbation space.
Details of hyper-parameter parameter selection, model structure and code are included in the supplementary part. Embedding space visualization also be found in the supplementary material.
CONCLUSION
In this paper, we propose new defense framework, ER-Classifier, which projects the input images to low-dimensional space to remove adversarial perturbation and stabilize the model through mini- mizing the discrepancy between the true label distribution and the framework output distribution. We empirically show that ER- Classifier is much more robust than other wate-of--he-ant aart defense methods on several benchmark datasets. Future work will include further exploration of the low-dimensional space to improve the robustness of deep neural network.
REFERENCES
Martin Arjovsky. Soumith Chintala, and Léon Bottou. arXiv:1701.07877 077775 2017.
Wasserstein gan.
arXiv preprint
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. arXiv preprint 2018.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings the 10th ACM Workshop Artificial Intelligence and Security, AlSec '17, pp. 3-14, New York, NY, USA, 2017a. ACM. ISBN 978-1-4503-5202- 4. doi: 10.1145/3128572.144442 URL http: acm. 0tt//10.114/3112800
3140444
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Security and Privacy (SP), 2017 IEEE Symposium on, pp. 39-57. IEEE, 2017b.
Pin-Ye Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order opti- mization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 15-26. ACM, 2017.

---

Under review as a conference paper at ICLR 2020
Pin-Y Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsich. Ead: elastic-net net attacks to deep neural networks via adversarial examples. In AAAI, 2018.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-2215,,2 2011.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248-255 leee, 2009.
Gavin Weiguang Ding, Kry Yik Chau Lui, Xiaomeng Jin, Luyu Wang. and Ruitong Huang. On the sensitivity of adversarial robustness input data distributions. arXiv preprint arXiv: 190.00360, 2019.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples (2014). arXiv preprini 1426521
Andrew Ilyas, Ajil Jalal, Eirini Asteri, Constantinos Daskalakis, and Alexandros G Dimakis. The robust manifold defense: Adversarial training using generative models. arXiv preprint 1rX1v:1712.09196. 2017.
Andrew Hyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. arXiv preprint arXiv: 1905.02775 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv: 160700233, 2016.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nmistf/ 1998.
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. In Advances in neural information processing systems, pp. 777-784, 2005.
Yandong Li, Lijun Li, Liqiang Wang, Tong Zhang, and Boqing Gong. Nattack: Learning the dis-
tributions of adversarial examples for improved black-box attack deep neural networks. In ICML, pp. 3866-38766 2019.
Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsich. Towards robust neural networks via random self-ensemble. arXiv preprint arXiv:1712.00673, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov)2557-225 2605, 2008.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.
Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0608. 2017.
Alireza Makhzani. Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint 15 2015.
Dongyu Meng and Hao Chen. Magnet: two-pronged defense against adversarial examples. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 137- ACM, 2017.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and supervsseee learning. IEEE transactions on pattern analysis and machine intelligence, 1979,199911 2018.
10

---

Under review as a conference paper at ICLR 2020
Seyed-Mohsen Moosavi-Dezfoole Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp.
2016.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. arXiv preprint, 2017.
Taesik Na, Jong Hwan Ko, and Saibal Mukhopadhyay. Cascade adversarial machine learning regu- larized with unified embedding. arXiv preprint arXiv: 02582, 2017.
Nicolas Papernot, Patrick McDanicl, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense adversarial perturbations against deep neural networks. In Security and Privacy (SP), 2016 IEEE Symposium on, pp. 582-597. IEEE, 2016.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506-519, ACM, 2017.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint arXiv: 1805.06605 2018.
Carl-Johann Gabrie,, Yann Ollivier, Bernhard Schölkopf. Léon Bottou, and David Lopez- Paz. Adversarial vulnerability of neural networks increases with input dimension. arXiv preprint 180..0120,..1441,, 2018.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. encoders. arXiv preprint arXiv:1711.01558 2017.
Wasserstein auto-
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization, arXiv preprint arXiv:1711.01991, 2017.
Shufei Zhang, Kaizhu Huang, Jianke Zhu, and Yang Liu. Manifold adversarial learning, 2018.

---

Under review as a conference paper at ICLR 2020
APPENDIX
THEORETICAL ANALYSIS OF OUR FRAMEWORK
Mathematically, input images X E I2d are projected to a low-dimensional embedding vector Z E z = E through the encoder Qs. The discriminator D,, discriminates between the generated code 7 ~ 2=Q2(22 (2) and the ideal code Z Pzz The classifier C, performs classification based on the generated code Z, producing output U E U R", where is the number of classes. The label of X is denoted as Ut
The ER-Classifies framework embeds important classification features by minimizing the discrep- ancy between the distribution of the true label (Py) and the distribution of the framework output (Pc). In the framework, the classifier (Po(U1)) Z)) maps a latent code Z sampled from fixed dis- tribution in a latent space Z, to the output U E l = Rm. The density of ER-Classifier output is defined as follow:
po(u)= = p Yu e U.
In this paper apply standard Gaussian our prior distribution Pz, but other priors may be used for different cases. Assume there is an oracle f : - L assigning the image data (X E X its true label (Y U). We want to minimize the optimal transport cost between the distribution of the true label (Py) and the distribution of the ER-Classifier output (Pc).
OPTIMAL TRANSPORT THEORY
There are various ways to define the distance or divergence between the target distribution and the model distribution. In this paper, we turn the optimal transport theory (Villani 2008), which provides a much weaker topology than many others. In real applications, data is usually embedded in space of much lower dimension, such as non-linear manifold. Kullback-Leible divergence, Jensen-Shamoon divergence and Total Variation distance are not sensible cost functions when learn- ing distributions supported by lower dimensional manifolds (Arjovsky et al.) (0077). In contrast, the optimal transport cost is more sensible in this setting. Kantorovich's 's distance induced by the optimal transport problem is given by
inf W (PYPY Pc) rEP(Y~Py, PoPP)
B(r,))- (c(Y
where P(Y ~ U Pc) ) the set of all joint distributions of (Y, U) with marginals Py and Pc. and c(y.u) u): 4 1 + R4 is any measurable cost function. W (P Pc)) measures the divergence between probability distributions Py and Pc. When the probability measures are on a metric space, the p-th root of W, is called the p- p-ssssesstein distance.
To minimize the optimal transport cost between the distribution of the true label (Pr) and the dis- tribution of the ER-Classifier output (Pc), it is sufficient to find conditional distribution Q( such that its marginal distribution Q z is identical to the prior distribution Pz.
Theorem For Pe as defined above with a deterministic Po(UIZ) and any function C: 2 U
inf DEP(YNTy,UPPa)
EX (E(Y,U)}
inf Ep EqIzI) ((((()) Q:Qx=P2
where I' E P(Y ~ Py,U ~ Pc) is the set of all joint distributions of (Y,U) with marginals Py and Pc, and f(y,u) u) : 14 + R4, is any measurable cost function. Qz the marginal distribution of Z when X ~ Px and Z ~ QZZ)). (The proof is presented later.)
Therefore, optimizing over the objective the r.h.s is equivalent to minimizing the discrepancy between the true label distribution (Py) and the output distribution Pc. thus the important classi- fication features embedded in the low-dimensional space. This is the core idea of the paper,
12

---

Under review as a conference paper at ICLR 2020
summarizing the high- dinenssinnii data in space of much lower dimension without losing impor- tant features for classification. To implement the r.h.s objective, the constraint on Q can be relaxed by adding penalty term. The final objective of ER- ER-Classifier is:
inf Epx (O(II) {4(((), C(Z)) + XD(Qz. Pz), (
(3)
where any nonparametric set of probabilistic encoders, À 0 a hyper-parameter and an arbitrary divergence between Qz and Pz.
To estimate the divergences between Qz and Pz. we apply a GAN-based framework, fitting discriminator to minimize the 1-Wasserstein distance between Qz and Pz:
inf W (QQ2z P2)= rEP(2~Q,,2-Pp) 2-PP)
E (Z Z||-
We have also tried the Sensen-Shammen divergence, but expected, Wasserstein distance provides more stable training and better results. When training the framework, the weight clipping method proposed in Wasserstein GAN (Arjovsky et al. (017) is applied help stabilize the training of discriminator D...
PROOF OF THEOREM
The proof of Theorem 1 is adapted from the proof of Theorem in Tolstikhin et al. 2017). Consider certain sets of joint probability distributions of three random variables (X, 2) E X u x Z. X can be taken as the input images, U as the output of the framework, and as the latent codes. Zu, Z) represents joint distribution of variable pair (U, Z). where Z is first sampled from Pz and then from Pu(Ul Pa defined in is the marginal distribution of U when (U, 2) Po,z-
The joint distributions r(X, U) or couplings between values of X and U can be written as U) ((U)X))x(XX due the marginal constraint. [(U] can be decomposed into an encoding distribution ( and the generating distribution (c Z), and Theorem mainly shows how factor it through Z.
In the first part, we will show that if Pc(UIZ) are Dirac measures, we have
inf EXxUU (((((X);U)
reP(X-Px.U-ne)
inf Exx,Ur (()(())U)) TEPx.
where P(X- PX,U Pc) denotes the set of all joint distributions of (X, U) with marginals Px, Po. and likewise for P(X ~ Px, 7 ~ Pz). The set of all joint distributions of (X, U, Z) such that X Px, (U, the sets of marginals on (X, U) and (X, Z) induced by Px,U.z-
Pa,z, and (U II X
denoted by Px, Px, and Px,z denote
From the definition, it is clear that Px, P(Px, Therefore, we have
inf TEP(X-Px.U-Pe) E(X,U)- ((((())
< inf Exx,U)er UL), rePx.
(5)
The identity is satisfied if (C) are Dirac measures, such as C(Z). This is proved by the following Lemma in (Tolstikhin et al. 2017). -5pt
Lemma Pxu C P(Px, Pc) with identity if Pc = 2) are Dirac for all z E Z. (see details in (Tolstikhin al. 2017).)
In the following part, we show that
inf (x.0))(((((()))) rEPx.0
inf Ep, Bozzx) (((((), QQ2==2
13

---

Under review as a conference paper at ICLR 2020
Based on the definition, P(Px: Pc), Px. and Px,v depend the choice of conditional distri- butions Zc( but Px,z does not. It is also easy to check that Px.2 = P(( ~ Px. ~ Pz). The tower rule of expectation, and the conditional independence property of Px,u, implies
inf B(X,U)~ (((((x), TEPx.0
inf EXX (UU())
TEPx.U.2 EP Pr
inf
TEPx,U,
inf Ep E Ex-P(I)) ((2)
TEPx,u,
inf Exz (((((X),
TEPx.2
inf Epx Bo(z)x) ((((X),C(Z)) Q:Qz=Pz
Finally, since Y = f(X), is easy to get
inf TEP(Y~Py.U-Re) BXrU)~ (((YU))
inf TEP(X~PX
E E (E((()),
(8)
Now 4 and are proved and the three together prove Theorem
Our proposed framework readily applies to non-deterministic case. If the classifier part is non- deterministie, Lemma [ provides only the inclusion of sets Px,u S P(Px,, Pv), and we can get an upper bound on the Wasserstein distance between the ground-truth and predicted label distributions:
inf TEP(X~Px.UwPo) B(x,))rr {L(((X))U) <
inf E(x,y)r (e(f(X).U) rEPx.
inf rEPx~Px 2-P2 Exx,zy~ (Ilf(X) C
where we assume the conditional distributions Po(U|Z = 2) have mean values C(2) E Rd and marginal variances of.... o3 2 0 for all Z. where C Z X. and l(y. u) lll u/| The above upper bound is derived by: =
inf E(x,))- TEPx.0
inf rePx.u,2
Ep Bo (Il(() U
(10)
and
(1X - -
= =II((X) C(2)]2 BU- -(2) + EU- (I - U11
II - C(Z)) Σ of
(11)
In equation 11 the second term of the second last row becomes since the optimization will drive f(X) C()) to zero.
HYPER-PARAMETER SELECTION
DIMENSION OF EMBEDDING SPACE
One important ppprmppareettr for the ER-Classifier the dimension of the embedding space. the dimension is too small, important features "collapsed" onto the same dimension, and if the dimension is too large, the projection will not extract useful information, which results in too much noise and instability. The maximum likelihood estimation of intrinsic dimension proposed in
14

---

Under review as a conference paper at ICLR 2020
CIFARTO
6.9
6m-d 6m--66
6e 0m-20
Tostle accuraay
04
terting
attaclo
epston (fteeggh sttack)
opstion attack)
Figure 5: Testing accuracy of models with different embedding dimensions under 1 -PG attack.
MNIST
CIFAR10
STL10
Maddy:es
2
testing accuracy
0.8
testing accuracy
0.8-
Mudrysapsiomm
testing accuracy
0.5
0.6
Madfyiepdionn..o 0.04
0.5
0.4
0.4
Kodrycpster--.5
Kadryep-fo--G55
6.4
0.3
Madnysputer--.88
Kadhyapsee--.777
0.2
0.2
0.0
0.1
0.2
0.3
0.4
0.00 0.01 0.02 0.03 0.04 0.05
0.06
0.00
0.01 0.02 0.03 0.04 0.65 0.06
epsilon (strength the attack)
epsilon (strength the attack)
epsilon (strength of the attack)
Figure 6: Testing accuracy of models with different MNIST, CIFARIO and STL10.
Levina & Bickel 2005 is used to calculate the intrinsic dimension of each image dataset, serving guide for selecting the embedding dimension. The sample size used in calculating the intrinsic dimension is 1, 000, and changing the sample size does not influence the results much. Based on the intrinsic dimension calculated by (Levina & Bickel 2005). we test several different values around the suggested intrinsic dimension and evaluate the models against (x -PD attack. All models are trained without min-max robust optimization, and the experimental results are shown in Figure
The final embedding dimension is chosen based on robustness, number of parameters, and testing accuracy when there is no attack. The final embedding dimensions and suggested intrinsic dimen- sions are shown in Table
Data MNIST CIFARIO
Data dim.
Intrinsic dim.
Embedding dim.
X x
13
3 32
17
16
STLIO
3
20
16
Tiny Imagenet
x
Table 3: Pixel space dimension, intrinsic dimension calculated by (Levina & Bickel 2005). and final embedding dimension used.
the embedding dimension close the calculated intrinsic dimension usually offers better results except on MNIST, One explanation may be that MNIST is simple handwritten digit dataset, so performing classification on MNIST may require that many dimensions.
Based on Figure 5
EPSILON SELECTION
Epsilon (c) is an important hyper-pann parameter for adversarial training. When doing Madry's adversar- ial training, we test the model robustness with different and choose the best one. The experiment results are shown in Figure
Based on Figure we use e = 0.3, 0.03, 0.03 in Madry's adversarial training on MNIST, CIFARIO and STL10 respectively. For Tiny Imagenet, we use 0.01. To make fair comparison, we use the same when training ER-Classifier.
'Code publicly available http://///gghhhh.ooo///////oooo
15

---

Under review as a conference paper at ICLR 2020
EMBEDDING VISUALIZATION
In this section, we compare the embedding learned by Classie+ structure (E-CLA) and the embedding learned by ER-Classifier on several datasets without min-max robust optimization. We first generate embedding of testing data using the encoder ( (z)), then project the em- bedding points (ž) to 2-D space by ISNt(Maaten & Hinton 2008). Then we generate adversarial images (Fadt) against E-CLA and ER- El-Csassifier using PGD attack. The adversarial embedding is generated by feeding the adversarial images into the encoder (Zade Q (Tade)). Finally, we project the adversarial embedding points (Zadn) to 2-D space. The results are shown in Figure 7 The plots in the first and second rows are embedding visualization plots for E-CLA, and the plots in the third and last rows are the embedding visualization plots for ER-Classifier. In adversarial embedding visualization plots, the misclassified point is marked "down triangle", which means the PGD attack successfully changed the prediction, and the correctly classified point is marked as "point", which means the attack fails.
Based on Figure 7 we can see that E-CLA can learn a good embedding on legitimate images of MNIST. Embedding points for different classes separated on the 2D space, but under adversarial attack, some embedding points of different classes are mixed together. However, ER-Classifier generate good separated embeddings on both legitimate and adversarial images. On CIFARIO. 10, the E-CLA can not generate good separated embeddings on either legitimate images or adversarial images, while ER-Classifier can generate good separated embeddings for both.
PSEUDO-CODE
Code for reproduction will be made available online at github later. The pseudocode for training ER-Classifier shown in Listing I.
MODEL STRUCTURE
MNIST, STL10 and Tinylmagenet classifier structures used for baseline methods shown in Fig- ure g We use VGG19 for the baseline methods on CIFAR10. Details of Classifier structures on the four benchmark datasets shown in Figure 10
16

---

Under review as a conference paper at ICLR 2020
Original Embodding E-CLA MNIST
Adversarial Embedding E-CLA MNIST
Classffed
Classi ed Wrong
Original Embedding E-CLA CIFAR10
Adversarial Embedding E-CLA CIFAR10
Classifed Right Wrong
Original Embedding of ER-classtfeer on MNIST
Adversarial Embedding of ER-classifier on MNIST
Classifed Right
Classilied Wrong
Original Embodding of ER-classifie on CIFAR10
Adversarial Embedding of ER-classifior on CIFAR10
Classiflad
Clossifiad Whonn
Figure 7: 2D embeddings for E-CLA and ER-Classifier on MNIST and CIFAR10. Larger visualiza- tion figures are also shown in the appendix.
17

---

Under review as a conference paper at ICLR 2020
Original Embedding of E-CLA on MNIST
18

---

Under review as a conference paper at ICLR 2020
Adversarial Embedding of E-CLA on MNIST
Classified Right
Classified Wrong
19

---

Under review as a conference paper at ICLR 2020
Original Embedding of E-CLA on CIFAR10
20

---

Under review as a conference paper at ICLR 2020
Adversarial Embedding of E-CLA on CIFAR10
Classified Right
Classified Wrong
21

---

Under review as a conference paper at ICLR 2020
Original Embedding of ER-classifier on MNIST
22

---

Under review as a conference paper at ICLR 2020
Adversarial Embedding of ER-classifier on MNIST
Classified Right
Classified Wrong
23

---

Under review as a conference paper at ICLR 2020
Original Embedding of ER-classifier on CIFAR10
24

---

Under review as a conference paper at ICLR 2020
Adversarial Embedding of ER-classifier on CIFAR10
Classified Right
Classified Wrong
25

---

Under review as a conference paper at ICLR 2020
def train. weiineer-cii ce..eerrcllaaeeee ar.cll((rain.ooade test.loader, encoder, discriminator,
classifier, other-hyper bererrhypp..ppameeen.. parameters):
criterion CrossEntropyLoss
encoder. train discriminator. train()
classifier train()
# Optimizers
enc optim optim.Adam enccnder. parameters (). Ir 1r)
dis.optim optim Adam discriminator parameters ()
0.5 Ir)
10
cla.optim = optim optim.Adaml Adam classifier classiiier.parammtteess/
parameters () Ir = 0.05 * Ir)
11
enc.scheduler StepLR (cnc_optim.
step-size=30) gamma=0.5)
dis-scheduler StepLR dis_oppim..
step-size=30. gamma=0.51
13
cla.scleu StepLR (cla.optim. one torch. Tensor ([1]) mone = one -1
step-size =30, gamma=0.5)
14
15
for epoch in range (num :
step = =
18
for images. labels in tqdm train loader):
19
encoder. uncoder.zero-grra. zero-grad
20
discriminator. zero-grad classifier.. )
22
Min-Max Robust Optimization
23
images = adv get (images, classificr encoder)
24
# ======== Train Discriminator ==ss====
frozen-parane (encoder)
frozen-params celassicclsssfferr.
27
free-params discriminator)
28
z.fake sample.z(prior, n.. bateh_size, sigma)
29
d_fake discriminatorrto.vaau
z_real encoder
real discriminator var real))
32
disc fake LAMBDA * fake mean
fake.mean
33
disc real LAMBDA * d real mean
34
disc-fake backward (one)
disc.real backward mone)
36
diss_loss disc_fake disc_real
37
dis.optim.stepy ()
38
clip-params prn-params(discriiiie (diseriminator)
#= Train Classifier and Encoder
free-params (encoder)
41
free free perepparasscclassiini (classifier)
frozen (discriminator) paraammddisriiiiatttor
12
43
pred_labels = classifier (encoder tvr (images))
class_loss LAMBDA0 criterion labed_labels llbels))
class_loss.backwee backward
46
cla_optim. enc step
47
48
# ========= Train Encoder ==s-_=== #
free-params (encoder)
50
Trozen.params (classifiee)
51
frozen-params discriminator)
52
z-real encoder (images)
d_real discriminator (encoder (encoder Variable (images data)))
d_loss LAMBDAI (d.real. (.eann ())
55
d.loss backward
56
enc ontio. att..pmim.mpp((( step += savefile cocefileffile_nnn (file_name. encoder. discriminator, classifier dataset= dataset)
57
59
return classifier. encoder
Listing I: Pseudocode for training ER-Classifier
26

---

Under review as a conference paper at ICLR 2020
Thp mopeled fend at Stackee
Peatres Seqarta:
Bc Restraad
D2 Condass and 2 3 mide a
the thd
BC Reed
D20 Camaalaa hord stael Bastdeel Rovilreel
13: Mec lecrdd stdd andled aod and dhttoo oo mosoriidd
MNNIT Baseline Strccure
122 Rel Jisteo
and tuctra
(comv2) Com261 kemell doc-15 5) 2016--1 IIl
120 Cambatde hard acaet Brattaell Sosttell
Icomz) Comoeeg, 50, hemel sieett, St. strder11, 31
[t:]) Uneaninn foatures-800 out teatureu-500 NaseTruel
D2 Dipooauee orme wers is aesert Sveesarts
[+]) Unwanin feetorennton out festureerin biserTnue)
D3 DacMeraash tame tber drder peddingd chstari of madertion
Slup bostre Biseees
01 derdi bi, leved wir showet 16 p II
D7) Coociie berd dark i Speekaee of
01 Mereeeee
serertann Aic rock partr Meoethed
hil lirsted
(1: canvecia de, samel ohe a siart io 10
DV Conaa 53 birn serll in undeell & Saadkaggh of
Bitamiiedaaa conlete momertundd strarTun Ted norr Intefa
01
made Sanee
[L) Crehit)) XD bere crall it cadeecl Rudddael St
Neraarolien uusd andeed pat sapa, diaooed oo madenato
01 Cendo 128 tamel teach mise)) 1) metsiggll
(1 la larhhaerddsttk es ledt nessun 6.1 Pate Yirh rentin Thae
117 Crebett)) 532 lire sheel it seddef Konkkeel 3I
Mus net Uiber an de-3 predingd diaberl
Baitkentellt epolett
11h Saweatti 220 anded steets Tt dridedt 31 pandrrpt HI
Emooso canta movermme
Crodlit) 512, lored sertt 11 (oideel) Shond II
(8)) (n)-nnteoraaa epuitets manerumms stteectie, Yodk parvig strotrue
Mashoob jas net 2leer aed prd and didterl,, moderfadee
(10 wahall3 25% anded strel 1t ardeetl
(172
Sopertal
(196 WAWA 25% storts 11 ardeett,11 16
(29) kanova Mig modde di, movertune Liffeee trakk mnauin
D2 dasadiedd11
dest poedingss diatorns,
Haurerr Securdial
13 Bescddiedd
15 Urcele Istorode ofl Intresto buceise
Figure 8: Baseline Structure for MNIST
1202 Euciriated
Inan Souarial
0t Uneals feor wembororrrrrggiooo deserios out fnserred bastree)
Parury Seqarcal
Roowe ou meuse 113 Beeria
(0 kelifwwan
Fellinghel
10 lnnale Denereatt out fontroskty bostiee
01 Cenci Xe. s bere gidee pnnke
00 kettinphoo
(7 SeUunng aea
8: Un Pestrmeet ort Iutusedd, Sarthal
NI Inglomed Iil.
M1 Cenvittk Lomd 8 oiteen 1o IIl
Inarl Soueriel
tawdspt4
mndels (1 10
(1x Seluinneenn
Ms Satriel 128, sordd joott sottee & pobingeli 11, akeefial
R4: fandipis berd antel pastogg oo
[SS:
2ach quote:3 novertanrs, aftertrus trek puriin Trai
NC keujozaa
03 P55 512 lemd stral it antopp contretl I1 tonifaae
Halbodiffannd siteed diabred roterfil
ln Wireoo)
86 Conesiba 2034 lervel alseet, 4t undeon 3 puddngel i1 dleefaldd
100 lilegao))
Ni Larthem28@t00 epe Ie-i, monertannl 1 affire The tock nining evechhee
D2: (ami St1 lord soad godeo It pastkaee 12
(awd lerel andeel
fotturss Iow
Be Unppter
[2) Mahhefifoon soud untee god
MUST 1R Caotier
Inal) Souerial
anttch cantrnlll
futwooso,, Kanuivel
[1): Lepllnd
sp--e-22
09 unew) Nasrretto, ost futueesess Magetts
Som28511 311 landl Auddee 31sadiigeii 30 131 badaaeed
Ht test spode 25 morerrnn-D dire trek corrin sto-Trss
25) Leewnn bettam frase los Free
Figure 9: Details of Embedding Regularized Classifier Structures
27

---

Under review as a conference paper at ICLR 2020
fasurl Sup seperrda
pastngel:
01 Feithigton
81 conraajo same cow aiotnggh
HL Ne becbee thed
0-3
21 SaUAArpkec)
121 facllrpacd
NacheoSaaree yattings riearr Fali
111 Besi Boureh meeets Tran monny, Tooe
[R: Cenia 18th hand Jaant 4t anaen it addings Stan indl
12 Iisttentrii
[11) Sellaa
18: bikatursts meslec
DR1 BouEspace
1M susssnsseee epodtess covertaned
cecaitts Iice kane wou mater watreg Stturrdd
D81 Eechomaa01s spene 2. vomer strresttn 2nd ruming jolnine
(1) Con0ato0n cosa tame ane undieg At pidlegi lae Slll
122 tetoHommaa00t ce monemmmei
111 ler fonrhii12 tedt moeeds alee Tran and monnng was Toae
D41 Cemenaaos lacel anec indeest auloctin
D8 Baliferraloot inmectteee
Metelug
(2) Eupla))
DE1 Intchammalia sprlle 703a8t streett 2nd noma Ihtel mite
D8: Eathhomb511s sprree vomerts mill attrenttn 2id ruman Iheehee
Irr learth wetteent ert mendon bouther
[1) Badd honbhlir opott useent atreeee and roming dat
1x Leacch Infacood0 bntennot beefag
Leaiig Indoomchh bet tenstie
Ingeol Adaonaapheelldd (add)
Figure 10: Details of Embedding Regularized Classifier Structures
28


Reviews:
[{"review_id": "S1gqzHfaYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new regularization technique called Embedding Regularization to improve the adversarial robustness.  The idea is to use generative adversarial networks (GAN) to perform inference on the latent space by matching the aggregated posterior of the hidden space vector with a prior distribution. The proposed strategy could be combined with adversarial training to achieve state-of-the-art adversarial accuracy on several benchmark datasets. \n\nOverall, I find the idea interesting and the experimental results promising. The following are my detailed comments.\n\na. About the algorithm \nThe idea of incorporating a GAN based model to regularize the representation learning is not completely novel. In [1], a similar approach has been considered for unsupervised learning (auto-encoders).  Besides the omission of reference, a few points about the algorithm need to be clarified :\n\na.1 What is the motivation of using the different of discriminator as loss in line 7 of Algorithm 1?\nIn the standard GAN literature, the discriminator loss is usually in the form log(D(z_{true})) + log (1 - D(z_{generated})). Is there any intuition to prefer the current formulation comparing to the this standard formulation?\n\na.2 Why do we separate the training of classification loss and regularized loss (line 8 and line 9)? \nFrom the optimization perspective, there is no difficulty to optimize 8+9 together, which corresponds better to the loss described in equation (2). (Just fix the discriminator and train the encoder and classifier jointly) Is there any reason to prefer the current alternative training rather than joint training? \n\na.3 How large are the discriminator loss versus generator loss?\nIn the standard GAN training, it is crucial to balance the discriminator loss and the generator loss. A plot comparing the different loss in line 7, line 8  and line 9 will be helpful to understand the role of discriminator in the framework. In particular,  it would be interesting to see whether there is a difference when combining with/without adversarial training.\n\nb. About the experiments \nb.1 Why Gaussian distribution is a good prior distribution?\nIntuitively, we would expect the latent distribution to be well clustered in several class which is clearly not the property of a Gaussian distribution. It is very curious to me why imposing a Gaussian distribution could improve the robustness. Have you tried to use clustered distribution as a prior? (examples could be find in [1]) If not, it would be interesting to try and compare the results.\n\nb.2 On CIFAR10, why the non-robust training outperform the robust training?\nIt is curious to see that on CIFAR10, ER-classifier- outperforms ER-Classifier with large epsilon. This is surprising since the min-max robust training explicitly minimize the robust loss. \n\nb.3 Have you tried to vary the regularization parameter lambda?\nIt would be interesting to see how the performance changes when varying the regularization parameter lambda. By the way, how is lambda determined in the current experiments?\n\nMinor comments: \nb.4 There is a stronger version of Defense-GAN introduced in [2]\nb.5 The study on the dimension of embedding space is interesting, is the non robust training giving similar results? \n\n[1] Adversarial Autoencoders, Makhzani et al. 2015\n[2] The Robust Manifold Defense: Adversarial Training using Generative Models, Ilyas et al, 2017", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJedaZ7EYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary\n========\nThis paper proposes to make neural networks robust to adversarial examples via dimensionality reduction.\nThe paper builds on and compares to prior approaches (e.g., MagNet and Defense-GAN) that are known to be broken, which casts serious doubts on the validity of the performed experimental evaluation.\nThe theoretical part of the paper is also very hard to follow.\nFor these reasons, I recommend rejection.\n\n\nComments\n=========\n\nThe way I understand the proposed framework, it is trying to compress the data into a form that is as close as possible to a standard Gaussian distribution, while maintaining classification accuracy.\nThe intuition, formulated next to Figure 1, seems to be that compressing the data will force the network to only retain the most important features for classification, which will thus lead to higher robustness.\nI have two main concerns with this approach:\n- First, recent work by Ilyas et al (\"Adversarial examples are not bugs, they are features\") seems to suggest that models tend to learn features that are not robust yet generalize well. It is unclear why compressing the data would remove such features.\n- Second, it is unclear why constraining the compressed data to be close to a Gaussian would have a positive effect on robustness. The authors claim that this is to remove some \"pathological\" mappings from the input space to the embedding space. Can you explain what you mean here? What are these pathological mappings and why should they be the source of non-robustness?\n\nThe theoretical analysis in Sections 2.3and 3. is very hard to follow. The authors first mention that \"optimal transport theory [...] provides a much weaker topology than many other [distances]\". What do you mean by this? What is the advantage of optimal transport theory that you are trying to exploit?\nWhen Kantorovich’s distance is first introduced, the reader has no idea what any of the symbols represent. What are Y, U, \\mathcal{U}, P_Y, P_C, etc?\nThe authors then mention a relation to the Wasserstein distance and to Wasserstein GANs. How does this relate to this paper?\n\nI could not understand how Algorithm 1 related to the discussion in Section 3.\nAlgorithm 1 simply uses a standard cross-entropy loss to train the classifier C. Is this what you mean by minimizing the optimal transport cost between P_Y and P_C? This seems like a very convoluted way of justifying the use of the most standard loss function in ML.\nAlgorithm 1, step 6 mentions sampling from Q(Z|x_i). But isn't the encoder Q deterministic? If not, where does the randomness come from?\n\nIn the experimental section, the authors compare to Defense-GAN, which was shown to be broken in \"The Robust Manifold Defense: Adversarial Training using Generative Models\" by Jalal et al.\nThe authors mention that they evaluate robustness using an untargeted white-box PGD attack, but they do not specify which objective their attack is optimizing. As this papers proposed a different classification pipeline, adaptive the attacks to this pipeline is crucial.\n\nThe experimental results (Figure 2 & 3) raise a number of questions:\n- Why does the ER-classifier variant without adversarial training perform so much better than the one with? The author's explanation about the difficulty of optimizing over the embedding space should be supported. \n- Do the attacks reach 0% accuracy for large enough epsilon? The very high accuracy even for large epsilon suggests that the evaluated attacks are not evaluating the right objective to fool the classifier. E.g., on MNIST, eps=0.4 can nearly destroy the whole image so the accuracy should be expected to be lower.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BketebWQ9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a adversarial learning framework that tries to align the hidden features of data with simple prior distributions.  A training strategy similar to GAN  was exploited. The proposed framework was argued that it can well deal with the adversarial perturbations. Some experiments were conducted, verifying that the proposed algorithm seems useful and robust.\n\nThe main comments are listed as follows:\n\n(1) To the reviewer, the adversarial framework to use a simple prior to align the hidden features seems new. However, in the literatures, there are some adversarial training algorithms which did similar things. For example, C1 tried to force the hidden features of deep learning follow a Gaussian distribution with the smooth regularizer. Though different,  I believe they share some common motivation, the authors may want to discuss and even compare this reference.\nSimilarly, in the literature, the so-called Center Loss will also basically force the means of different data are as further as possible, this is also relevant to the paper.\n\n(2) The authors have reported a series of experiments, which is great. However, they only evaluated their model's robustness in case of PGD attack. This is very different from many adversarial literature which normally would discuss various attack such as l_2 attack, FSGM, and even black-box attack.  The evaluations may be more convincing  if more attacks can be tested on the proposed method.\n\n(3) Further to (2), it is noted that the paper simply compared their approach's robustness  with Madry's adv, it would be more convincing to compare the other recently adversarial training algorithms like VAT (C2).\n\n(4) It is good that the paper proposed a different way in dealing with adversarial examples. In comparison, the current work studying adversarial examples is based on the robust framework of minimax trying to impose the worst-case perturbation. It would be interesting to see if the proposed work can be also further applied in the minimax  framework and examine if a further robustness can be achieved.\n\n(5) Some visualization may be interesting. For example, a visualization how Q would change as the training continues. This may be used to check the convergence property of the proposed algorithm.   \n\nC1:Manifold Adversarial Learning, S. Zhang et al.,  https://arxiv.org/pdf/1807.05832\nC2:Virtual Adversarial Training: a Regularization Method for Supervised and Semi-supervised Learning, T. Miyato et al. arXiv preprint arXiv:1704.03976, 2017.\n\n\n==================\n\nThanks for the response made by the authors. Thought these response resolved some of my concerns, it is still not very convincing.  On one hand, it seems that the authors did not  comment (4) and (5).  On the other hand, the paper may be still in lack of comparisons and/or discussions with the related work. After PGD, there are a lot of new robust approaches. Overall, I enjoy reading this paper but I would still think the paper could have further room to be  enhanced. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper suggests a new way to defend against adversarial attacks on neural networks. Two of the reviewers were negative, one of them (the most experienced in the subarea) strongly negative. One reviewer is weakly positive. The main two concerns of the reviewers are insufficient comparisons with SOTA and lack of clarity. The authors' response, though detailed, has not convinced the reviewers and has not alleviated their concerns. 

Decision: Reject