Year: 2019

Abstract:
Recent studies have demonstrated the vulnerability of deep convolutional neural networks against adversarial examples. Inspired by the observation that the intrinsic dimension of image data is much smaller than its pixel space dimension and the vulnerability of neural networks grows with the input dimension, we propose to embed high-dimensional input images into a low-dimensional space to perform classification. However, arbitrarily projecting the input images to a lowdimensional space without regularization will not improve the robustness of deep neural networks. We propose a new framework, Embedding Regularized Classifier (ER-Classifier), which improves the adversarial robustness of the classifier through embedding regularization. Experimental results on several benchmark datasets show that, our proposed framework achieves state-of-the-art performance against strong adversarial attack methods.

heading: 1 INTRODUCTION
text: Deep neural networks (DNNs) have been widely used for tackling numerous machine learning problems that were once believed to be challenging. With their remarkable ability of fitting training data, DNNs have achieved revolutionary successes in many fields such as computer vision, natural language progressing, and robotics. However, they were shown to be vulnerable to adversarial examples that are generated by adding carefully crafted perturbations to original images. The adversarial perturbations can arbitrarily change the network’s prediction but often too small to affect human recognition (Szegedy et al., 2013; Kurakin et al., 2016). This phenomenon brings out security concerns for practical applications of deep learning.
Two main types of attack settings have been considered in recent research (Goodfellow et al.; Carlini & Wagner, 2017a; Chen et al., 2017; Papernot et al., 2017): black-box and white-box settings. In the black-box setting, the attacker can provide any inputs and receive the corresponding predictions. However, the attacker cannot get access to the gradients or model parameters under this setting; whereas in the white-box setting, the attacker is allowed to analytically compute the model’s gradients, and have full access to the model architecture and weights. In this paper, we focus on defending against the white-box attack which is the harder task.
Recent work (Simon-Gabriel et al., 2018) presented both theoretical arguments and an empirical one-to-one relationship between input dimension and adversarial vulnerability, showing that the vulnerability of neural networks grows with the input dimension. Therefore, reducing the data dimension may help improve the robustness of neural networks. Furthermore, a consensus in the highdimensional data analysis community is that, a method working well on the high-dimensional data is because the data is not really of high-dimension (Levina & Bickel, 2005). These high-dimensional data, such as images, are actually embedded in a low dimensional space. Hence, carefully reducing the input dimension may improve the robustness of the model without sacrificing performance.
Inspired by the observation that the intrinsic dimension of image data is actually much smaller than its pixel space dimension (Levina & Bickel, 2005) and the vulnerability of a model grows with its input dimension (Simon-Gabriel et al., 2018), we propose a defense framework that embeds input images into a low-dimensional space using a deep encoder and performs classification based on the latent embedding with a classifier network. However, an arbitrary projection does not guarantee improving the robustness of the model, because there are a lot of mapping functions including non-robust ones from the raw input space to the low-dimensional space capable of minimizing the classification loss. To constrain the mapping function, we employ distribution regularization in the
embedding space leveraging optimal transport theory. We call our new classification framework Embedding Regularized Classifier (ER-Classifier). To be more specific, we introduce a discriminator in the latent space which tries to separate the generated code vectors from the encoder network and the ideal code vectors sampled from a prior distribution, i.e., a standard Gaussian distribution. Employing a similar powerful competitive mechanism as demonstrated by Generative Adversarial Networks (Goodfellow et al., 2014), the discriminator enforces the embedding space of the model to follow the prior distribution.
In our ER-Classifier framework, the encoder and discriminator structures together project the input data to a low-dimensional space with a nice shape, then the classifier performs prediction based on the lowdimensional embedding. Based on the optimal transport theory, the proposed ER-Classifier minimizes the discrepancy between the distribution of the true label and the distribution of the framework output, thus only retaining important features for classification in the embedding space. With a small embedding dimension,
the effect of the adversarial perturbation is largely diminished through the projection process.
We compare ER-Classifier with other state-of-the-art defense methods on MNIST, CIFAR10, STL10 and Tiny Imagenet. Experimental results demonstrate that our proposed ER-Classifier outperforms other methods by a large margin. To sum up, this paper makes the following three main contributions:
• A novel unified end-to-end robust deep neural network framework against adversarial attacks is proposed, where the input image is first projected to a low-dimensional space and then classified. • An objective is induced to minimize the optimal transport cost between the true class distribution and the framework output distribution, guiding the encoder and discriminator to project the input image to a low-dimensional space without losing important features for classification. • Extensive experiments demonstrate the robustness of our proposed ER-Classifier framework under the white-box attacks, and show that ER-Classifier outperforms other state-ofthe-art approaches on several benchmark image datasets.
As far as we know, our approach is the first that applies optimal transport theory, i.e., a Wasserstein distance regularization, to a bottleneck embedding layer of a deep neural network in a purely supervised learning setting without considering any reconstruction loss, although optimal transport theory or a discriminator loss has been applied to generative models in an unsupervised learning setting (Makhzani et al., 2015; Tolstikhin et al., 2017); (2) Our method is also the first that establishes the connection between a Wasserstein distance regularization and the robustness of deep neural networks for defending against adversarial examples.

heading: 2 RELATED WORK
text: In this section, we summarize related work into three categories: attack methods, defense mechanisms and optimal transport theory. We first discuss different white-box attack methods, followed by a description of different defense mechanisms against, and finally optimal transport theory.

heading: 2.1 ATTACK METHODS
text: Under the white-box setting, attackers have all information about the targeted neural network, including network structure and gradients. Most white-box attacks generate adversarial examples based on the gradient of loss function with respect to the input. An algorithm called fast gradient sign method (FGSM) was proposed in (Goodfellow et al.) which generates adversarial examples based on the sign of gradient. Many other white-box attack methods have been proposed
recently (Moosavi-Dezfooli et al., 2016; Chen et al., 2018; Madry et al., 2017; Carlini & Wagner, 2017b), and among them C&W and PGD attacks have been widely used to test the robustness of machine learning models.
C&W attack: The adversarial attack method proposed by Carlini and Wagner (Carlini & Wagner, 2017b) is one of the strongest white-box attack methods. They formulate the adversarial example generating process as an optimization problem. The proposed objective function aims at increasing the probability of the target class and minimizing the distance between the adversarial example and the original input image. Therefore, C&W attack can be viewed as a gradient-descent based adversarial attack.
PGD attack: The projected gradient descent attack is proposed by (Madry et al., 2017), which finds adversarial examples in an -ball of the image. The PGD attack updates in the direction that decreases the probability of the original class most, then projects the result back to the -ball of the input. An advantage of PGD attack over C&W attack is that it allows direct control of distortion level by changing , while for C&W attack, one can only do so indirectly via hyper-parameter tuning.
Both C&W attack and PGD attack have been frequently used to benchmark the defense algorithms due to their effectiveness (Athalye et al., 2018). In this paper, we mainly use l∞-PGD untargeted attack to evaluate the effectiveness of the defense method under white-box setting.
Instead of crafting different adversarial perturbation for different input image, an algorithm was proposed by (Moosavi-Dezfooli et al., 2017) to construct a universal perturbation that causes natural images to be misclassified. However, since this universal perturbation is image-agnostic, it is usually larger than the image-specific perturbation generated by PGD and C&W.

heading: 2.2 DEFENSE MECHANISMS
text: Many works have been done to improve the robustness of deep neural networks. To defend against adversarial examples, defenses that aim to increase model robustness fall into three main categories: i) augmenting the training data with adversarial examples to enhance the existing classifiers (Madry et al., 2017; Na et al., 2017; Goodfellow et al.); ii) leveraging model-specific strategies to enforce model properties such as smoothness (Papernot et al., 2016); and, iii) trying to remove adversarial perturbations from the inputs (Xie et al., 2017; Samangouei et al., 2018; Meng & Chen, 2017). We select three representative methods that are effective under white-box setting.
Adversarial training: Augmenting the training data with adversarial examples can increase the robustness of the deep neural network. Madry et al. (Madry et al., 2017) recently introduced a minmax formulation against adversarial attacks. The proposed model is not only trained on the original dataset but also adversarial example in the -ball of each input image.
Random Self-Ensemble: Another effective defense method under white-box setting is RSE (Liu et al., 2017). The authors proposed a “noise layer”, which fuses output of each layer with Gaussian noise. They empirically show that the noise layer can help improve the robustness of deep neural networks. The noise layer is applied in both training and testing phases, so the prediction accuracy will not be largely affected.
Defense-GAN: Defense-GAN (Samangouei et al., 2018) leverages the expressive capability of GANs to defend deep neural networks against adversarial examples. It is trained to project input images onto the range of the GAN’s generator to remove the effect of the adversarial perturbation. Another defense method that uses the generative model to filter out noise is MagNet proposed by (Meng & Chen, 2017). However, the differences between ER-Classifier and the two methods are obvious. ER-Classifier focuses on reducing the dimension, and performing classification based on the low-dimensional embedding, while Defense-GAN and MagNet mainly apply the generative model to filter out the adversarial noise, and both Defense-GAN and MagNet perform classification on the original dimension space. (Samangouei et al., 2018) showed that Defense-GAN is more robust than MagNet, so we only compare with Defense-GAN in the experiments.
Other Related Methods: Zhang et al. (2018) regularizes the latent space with Gaussian Mixture Model and applies KL-divergence to do the optimization. However, our method employs a simple but nice-shaped Gaussian prior for Wasserstein distance minimization to constrain the global shape of the latent embeddings, while permitting high freedom for the shapes of individual class distributions of latent embeddings. We want the classifier to decide the optimal class-specific distributions of
latent embeddings. Miyato et al. (2018) shares a similar idea to adversarial learning, but it employs virtual labels generated by a current classifier to identify search directions that can smooth the output label distribution of the classifier and is best suitable for semi-supervised learning. Please note that both methods in (Zhang et al., 2018; Miyato et al., 2018) are designed for improving generalization performance but not for defending against adversarial examples. A recent paper (Ilyas et al., 2019) shows that adversarial examples are purely human phenomenon and models tend to learn features that are not robust yet generalize well. We show that our Wasserstein distance regularization helps to identify robust features, which will be discussed later.
Notations In this paper, we use l∞ and l2 distortion metrics to measure similarity. We report l∞ distance in the normalized [0, 1] space, so that a distortion of 0.031 corresponds to 8/256, and l2 distance as the total root-mean-square distortion normalized by the total number of pixels.
We use calligraphic letters for sets (i.e., X ), capital letters for random variables (i.e., X), and lower case letters for their values (i.e., x). The probability distributions are denoted with capital letters (i.e., PX ) and corresponding densities with lower case letters (i.e., pX ).

heading: 3 PROPOSED FRAMEWORK: EMBEDDING REGULARIZED CLASSIFIER

heading: 3.1 FRAMEWORK DETAILS
text: We propose a novel defense framework, ER-Classifier, which aims at projecting the image data to a low-dimensional space to remove noise and stabilize the classification model by minimizing the optimal transport cost between the true label distribution PY and the distribution of the ER-Classifier output (PC). An overview of the framework is shown in Figure 1. The encoder and discriminator structures together help diminish the effect of the adversarial perturbation by projecting input data to a space of lower dimension, then the classifier part performs classification based on the lowdimensional embedding.
Mathematically, input images X ∈ X = Rd are projected to a low-dimensional embedding vector Z ∈ Z = Rk through the encoder Qφ. The discriminator Dγ discriminates between the generated code Z̃ ∼ Qφ(Z|X) and the ideal code Z ∼ PZ . The classifier Cτ performs classification based on the generated code Z̃, producing output U ∈ U = Rm, where m is the number of classes. The label of X is denoted as Y ∈ U . The Kantorovich’s distance induced by the optimal transport problem is given by
Wc(PY , PC) := inf Γ∈P(Y∼PY ,U∼PC) E(Y,U)∼Γ {c(Y,U)} ,
whereP(Y ∼ PY , U ∼ PC) is the set of all joint distributions of (Y,U) with marginals PY and PC , and c(y, u) : U × U 7→ R+ is any measurable cost function. Wc(PY , PC) measures the divergence between probability distributions PY and PC . When the probability measures are on a metric space, the p-th root of Wc is called the p-Wasserstein distance.
To minimize the Wasserstein distance between the distribution of the true label (PY ) and the distribution of the ER-Classifier output (PC), we can prove that it is sufficient to find a conditional distribution Q(Z|X) such that its marginal distribution QZ is identical to a prior distribution PZ . The theorem and the proof are deferred to the Appendix. In this paper, we apply standard Gaussian as our prior distribution PZ , but other priors may be used for different cases. The final objective of ER-Classifier is:
inf Q(Z|X)∈Q EPXEQ(Z|X) {`(f(X),C(Z))}+ λD(QZ , PZ), (1)
where Q can be a deterministic encoder as focused by this paper due to its simplicity or stochastic encoder as the one in a standard Variational Autoencoder, λ > 0 is a hyper-parameter and D is an arbitrary divergence between QZ and PZ .
To estimate the divergences between QZ and PZ , we apply a GAN-based framework, fitting a discriminator to minimize the 1-Wasserstein distance between QZ and PZ :
W (QZ , PZ) = inf Γ∈P(Z̃∼QZ ,Z∼PZ) E(Z̃,Z)∼Γ‖Z̃ − Z‖.
We have also tried the Jensen-Shannon divergence, but as expected, Wasserstein distance provides more stable training and better results. When training the framework, the weight clipping method proposed in Wasserstein GAN (Arjovsky et al., 2017) is applied to help stabilize the training of discriminator Dγ . The training algorithm is summarized in Algorithm 1.
At training stage, the encoder Qφ first maps the input x to a low-dimensional space, resulting in generated code (z̃). Another ideal code (z) is sampled from the prior distribution, and the discriminator Dγ discriminates between the ideal code (positive data) and the generated code (negative data). The classifier (Cτ ) predicts the image label based on the generated code (z̃).
Algorithm 1 Training ER-Classifier 1: Input: Regularization coefficient λ > 0, encoder Qφ, discriminator Dγ , and classifier Cτ . 2: Note: ` stands for the cross-entropy loss. 3: while (φ, γ, τ) not converged do 4: Sample {(x1, y1), ..., (xn, yn)} from the training set 5: Sample {z1, ..., zn} from the prior PZ 6: Directly obtain or sample z̃i from Qφ(Z|xi) for i = 1, ..., n 7: Update Dγ by ascending the following objective by 1-step Adam:
λ
n n∑ i=1 Dγ(zi)−Dγ(z̃i)
8: Update Qφ and Cτ by descending the following objective by 1-step Adam:
1
n n∑ i=1 `(Cτ (Qφ(xi)), yi)
9: Update Qφ by ascending the following objective by 1-step Adam:
λ
n n∑ i=1 Dγ(Qφ(xi))
10: end while
At inference time, only the encoder Qφ and the classifier Cτ are used. The input image x is first mapped to a low-dimensional space by the encoder (z̃ = Qφ(x)), then the latent code z̃ is fed into the classifier to obtain the predicted label.
The main goal of ER-Classifier is leveraging input space dimension reduction to remove adversarial perturbations. Therefore, other defense methods can also benefit from this property. Our framework is trained with min-max robust optimization (Madry et al., 2017).

heading: 3.2 JUSTIFICATIONS OF OUR FRAMEWORK
text: There are two Wasserstein distances (W-distances) in our framework. One is the W-distance between the aggregated latent embedding distribution Q(z) and the prior distribution PZ , and the other one is the W-distance between the true label distribution PY and the distribution of the ER-Classifier output (PC). In Algorithm 1, we are minimizing the first one. The theorem in the Appendix shows that minimizing the first W-distance in combination with minimizing a standard cross-entropy loss as done in Algorithm 1 is equivalent to minimizing the second W-distance, which guarantees that the training process is not distracted from the main goal of the framework, classification. That is to say, Algorithm 1 will result in a classifier with the following property: the global output distribution of the classifier will match the global ground-truth label distribution in the data no matter whether the encoder Qφ is deterministic or stochastic (the second W-distance is automatically minimized).
It’s hard to analyze the importance of the theorem in the Appendix if we just look at a deterministic encoder. Let’s convert this deterministic encoder to a stochastic encoder that outputs a Gaussian z with a fixed variance and the mean being the same as its corresponding deterministic version. The theory tells us that, by minimizing the first W-distance over all sampled z’s from this stochastic encoder and the standard cross-entropy loss, we will automatically minimize the second W-distance
and preserve the global label frequency in the dataset, even though these z’s are only -close to the deterministic encoding features of training data.
Moreover, we find that minimizing the W-distance helps the encoder identify some robust features instead of non-robust features (Ilyas et al., 2017), because our proposed regularization constrains the -ball around each Qφ(Z|X) to contribute to preserving the global label distribution in the data, even with X integrated out. From this perspective, we can view our proposed framework as “a supervised variant” of Generative Adversarial Network or Wasserstein Autoencoder in which the Generator or Decoder is replaced by a Classifier that generates labels from low-dimensional latent embeddings preserving global label frequency in the training dataset. Replacing W-distance with KL divergence loses all these nice properties.
In our framework, we use a simple but nice-shaped Gaussian prior PZ for W-distance minimization to constrain the global shape of the latent embeddings, while permitting high freedom for the shapes of individual class distributions of latent embeddings. We want the classifier to decide the optimal class-specific distributions of latent embeddings. In addition, it is interesting to explore how to set -ball to make sure the stochastic encoder to best align the latent embedding z to human-perceived robust features, which will be left as future work.

heading: 4 EXPERIMENTS
text: In this section, we compare the performance of our proposed algorithm (ER-Classifier) with other state-of-the-art defense methods on several benchmark datasets:
• MNIST (LeCun, 1998): handwritten digit dataset, which consists of 60, 000 training images and 10, 000 testing images. Theses are 28×28 black and white images in ten different classes. • CIFAR10 (Krizhevsky & Hinton, 2009): natural image dataset, which contains 50, 000 training images and 10, 000 testing images in ten different classes. These are low resolution 32× 32 color images. • STL10 (Coates et al., 2011): color image dataset similar to CIFAR10, but contains only 5, 000 training images and 8, 000 testing images in ten different classes. The images are of higher resolution 96× 96. • Tiny Imagenet (Deng et al., 2009): a subset of Imagenet dataset. Tiny Imagenet has 200 classes, and each class has 500 training images, 50 testing images, making it a challenging benchmark for defense task. The resolution of the images is 64× 64.
Various defense methods have been proposed to improve the robustness of deep neural networks. Here we compare our algorithm with state-of-the-art methods that are robust in white-box setting. Madry’s adversarial training (Madry’s Adv) has been recognized as one of the most successful defense method in white-box setting, as shown in (Athalye et al., 2018).
Random Self-Ensemble (RSE) method introduced by (Liu et al., 2017) adds stochastic components in the neural network, achieving similar performance to Madry’s adversarial training algorithm.
Another method we would like to compare with is Defense-GAN (Samangouei et al., 2018). It first trains a generative adversarial network to model the distribution of the training data. At inference time, it finds a close output to the input image and feed that output into the classifier. This process “projects” input images onto the range of GAN’s generator, which helps remove the effect of adversarial perturbations. In (Samangouei et al., 2018), the author demonstrated the performance of Defense-GAN on MNIST and Fashion-MNIST, so we will compare our method with Defense-GAN on MNIST.
Since the main goal of ER-Classifier is using dimension reduction to improve adversarial robustness, other defense methods can also benefit from this property. The proposed ER-Classifier is trained with min-max robust optimization (Madry et al., 2017). To demonstrate the dimension reduction ability of ER-Classifier, we include a variant ER-Classifier− which trains ER-Classifier without min-max robust optimization. 4.1 EVALUATE MODELS UNDER WHITE-BOX l∞-PGD ATTACK
In this section, we evaluate the defense methods against l∞-PGD untargeted attack, which is one of the strongest white-box attack methods. Models are evaluated under different distortion level ( ),
and the larger the distortion the stronger the attack. Depending on the image scale and type, different datasets are sensitive to different strength of attack.
Models on MNIST are evaluated under distortion level from 0 to 0.4 by 0.025. Models on CIFAR10 and STL10 are evaluated under ∈ [0, 0.06, 0.005]. Models on Tiny Imagenet are evaluated under ∈ [0, 0.02, 0.002]. As mentioned in the notation part, all the distortion levels are reported in the normalized [0, 1] space. The experimental results are shown in Figure 2. To demonstrate the results more clearly, we show part of the results in Table 1.
Based on Figure 2 and Table 1, we can see that ER-Classifier is the most robust one on a variety of datasets. ER-Classifier without min-max robust optimization can also improve the robustness of deep neural network. Compare the performance of ER-Classifier− with the performance of model without defense method (No Defense), we can see that ERClassifier− is much more robust than the model with no defense method on all benchmark datasets. Besides, when the distortion level ( ) is large, ER-Classifier− tends to perform better than other state-of-the-art defense methods on MNIST, CIFAR10 and Tiny Imagenet. This phenomenon is obvious on CIFAR10 and it even performs better than ER-Classifier when
the attack strength is strong. The reason might be that without min-max robust optimization, it is easier to regularize the embedding space.
We also compare Defense-GAN with our method ERClassifier on MNIST. Although Defense-GAN was shown to be partly broken by (Athalye et al., 2018; Ilyas et al., 2017), both ER-Classifier and DefenseGAN share the similar idea of projecting the input to a learned manifold, and comparing to Defense-GAN is important to demonstrate the advantage of our novel Wassserstein distance regularization. Please note that Defense-GAN is not our major comparison baseline in
this paper. Both ER-Classifier and Defense-GAN are evaluated against the l2-C&W untargeted attack, one of the strongest white-box attack proposed in (Carlini & Wagner, 2017b). Defense-GAN is evaluated using the method proposed in (Athalye et al., 2018), and the code is available on github1. ER-Classifier is evaluated against l2-C&W untargeted attack with the same hyper-parameter values as those used in the evaluation of Defense-GAN. The results under l2 ≤ 0.005 threshold are shown in Table 2. Based on Table 2, ER-Classifier is much more robust than Defense-GAN un-
1Publicly available at https://github.com/anishathalye/obfuscated-gradients/ tree/master/defensegan
der the l2 ≤ 0.005 threshold. Since (Samangouei et al., 2018) did not evaluate Defense-GAN on CIFAR10, STL10 and Tiny Imagenet, without details of GAN structure, we can not compare with Defense-GAN on these datasets.

heading: 4.2 EVALUATE MODELS UNDER BLACK-BOX ATTACK
text: We evaluate Madry’s adversarial training, ER-Classifier, and ER-Classifier−, against a recently proposed black-box attack method called Nattack (Li et al., 2019)2 on CIFAR10. Nattack is only performed on the first 100 images of CIFAR10 since the attack process takes a long time. We report the accuracy = number of correctly classified / number of attacked images (exactly 100). The accuracy of Madry’s adv, ER-Classifier, and ER-Classifier− is, respectively, 38%, 43%, and 32%. ER-Classifier still outperforms Madry’s adv.

heading: 4.3 EVALUATE THE EFFECT OF DISCRIMINATOR
text: ER-Classifier framework consists of three parts, and the classification task is done by the encoder Qφ and classifier Cτ . Without the discriminator part, the encoder can also project the input images to a low-dimensional space. However, arbitrarily projecting the images to a low-dimensional space with only the encoder part cannot improve the robustness of the model. In contrast, sometimes it even decreases the robustness of the model.
To show that arbitrarily projecting the input images to a low-dimensional space can not improve the robustness, we fit a framework with only the encoder and classifier part (E-CLA), where the encoder and classifier have the same structures as in ER-Classifier, and compare E-CLA with the ER-Classifier framework. For a fair comparison, both structures are trained without min-max robust optimization. The results are shown in Figure 3.
Based on Figure 3, we can observe that ER-Classifier is much more robust than just the encoder and classifier structure on MNIST, CIFAR10 and Tiny Imagenet. It is also more robust on STL10 but not that much. The reason might be that there are only 5, 000 training images in STL10 and the resolution is 96× 96. Therefore, it is harder to learn a good embedding with limited amount of images. However, even when the number of training images is limited, ER-Classifier is still much more robust than the E-CLA structure. This observation demonstrates that regularization on the embedding space helps improve the adversarial robustness. Notice that the performance of E-CLA structure is similar to the performance of model without defense method on CIFAR10, STL10 and Tiny Imagenet, and worse on MNIST, which means the robustness of ER-Classifier does not come from the structure design.
Variational auto-encoder can project the images to low-dimensional space and use Kullback–Leibler divergence loss to regularize the embedding distribution, which does not need discriminator structure. Therefore, we also tried VAE-CLA, which applies Variational auto-encoder structure to do the projection and regularization. The experimental results in Figure 3 show that VAE-CLA does not perform as well as ER-Classifier. Based on the observation of the Kullback–Leibler loss and classification loss during the training process, it seems difficult for VAE-CLA to balance between the two tasks. The reason might be that Kullback–Leibler distances are not sensible cost functions when learning distributions supported by low dimensional manifolds (Arjovsky et al., 2017).
2https://github.com/Cold-Winter/Nattack

heading: 4.4 PRIOR SELECTION
text: ER-Classifier does not have restrictions on the choice of prior. However, the selection of prior is important as it imposes different restrictions on the embedding space.
Three different prior distributions are tested on MNIST and CIFAR10 datasets. They are standard Gaussian, Uniform(−3, 3) and Cauchy(0, 1), where Cauchy(0, 1) has the same support as standard Gaussian but is heavy tailed and 99.7% of the standard Gaussian points lies within [−3, 3]. All the models are trained without min-max robust optimization, and the experimental results are shown in Figure 4. Based on the results, all three priors work well, but standard Gaussian performs best on both datasets.
Ding et al. (Ding et al., 2019) prove that adversarial robustness is sensitive to the input data distribution, and if the data is uniformly distributed in the input space, no algorithm can achieve good robustness. They also empirically show that cornered/concentrated data distributions tend to achieve better robustness. This helps explain why regularizing the embedding space can help improve robustness. Though the projection process reduces the input dimension, the embedding space is still large. Prior distribution helps push the embedding space to be more concentrated, reducing the valid perturbation space.
Details of hyper-parameter selection, model structure and code are included in the supplementary part. Embedding space visualization can also be found in the supplementary material.

heading: 5 CONCLUSION
text: In this paper, we propose a new defense framework, ER-Classifier, which projects the input images to a low-dimensional space to remove adversarial perturbation and stabilize the model through minimizing the discrepancy between the true label distribution and the framework output distribution. We empirically show that ER-Classifier is much more robust than other state-of-the-art defense methods on several benchmark datasets. Future work will include further exploration of the low-dimensional space to improve the robustness of deep neural network.

heading: APPENDIX

heading: THEORETICAL ANALYSIS OF OUR FRAMEWORK
text: Mathematically, input images X ∈ X = Rd are projected to a low-dimensional embedding vector Z ∈ Z = Rk through the encoder Qφ. The discriminator Dγ discriminates between the generated code Z̃ ∼ Qφ(Z|X) and the ideal code Z ∼ PZ . The classifier Cτ performs classification based on the generated code Z̃, producing output U ∈ U = Rm, where m is the number of classes. The label of X is denoted as Y ∈ U . The ER-Classifier framework embeds important classification features by minimizing the discrepancy between the distribution of the true label (PY ) and the distribution of the framework output (PC). In the framework, the classifier (PC(U |Z)) maps a latent code Z sampled from a fixed distribution in a latent space Z , to the output U ∈ U = Rm. The density of ER-Classifier output is defined as follow:
pC(u) := ∫ Z pC(u|z)pZ(z)dz, ∀u ∈ U . (2)
In this paper we apply standard Gaussian as our prior distribution PZ , but other priors may be used for different cases. Assume there is an oracle f : X 7→ U assigning the image data (X ∈ X ) its true label (Y ∈ U). We want to minimize the optimal transport cost between the distribution of the true label (PY ) and the distribution of the ER-Classifier output (PC).

heading: OPTIMAL TRANSPORT THEORY
text: There are various ways to define the distance or divergence between the target distribution and the model distribution. In this paper, we turn to the optimal transport theory (Villani, 2008), which provides a much weaker topology than many others. In real applications, data is usually embedded in a space of a much lower dimension, such as a non-linear manifold. Kullback-Leibler divergence, Jensen-Shannon divergence and Total Variation distance are not sensible cost functions when learning distributions supported by lower dimensional manifolds (Arjovsky et al., 2017). In contrast, the optimal transport cost is more sensible in this setting. Kantorovich’s distance induced by the optimal transport problem is given by
Wc(PY , PC) := inf Γ∈P(Y∼PY ,U∼PC) E(Y,U)∼Γ {c(Y,U)} ,
whereP(Y ∼ PY , U ∼ PC) is the set of all joint distributions of (Y, U) with marginals PY and PC , and c(y, u) : U × U 7→ R+ is any measurable cost function. Wc(PY , PC) measures the divergence between probability distributions PY and PC . When the probability measures are on a metric space, the p-th root of Wc is called the p-Wasserstein distance.
To minimize the optimal transport cost between the distribution of the true label (PY ) and the distribution of the ER-Classifier output (PC), it is sufficient to find a conditional distribution Q(Z|X) such that its marginal distribution QZ is identical to the prior distribution PZ .
Theorem 1 For PC as defined above with a deterministic PC(U |Z) and any function C : Z 7→ U
inf Γ∈P(Y∼PY ,U∼PC)
E(Y,U)∼Γ {`(Y,U)}
= inf Q:QZ=PZ EPXEQ(Z|X) {`(f(X),C(Z))} ,
where Γ ∈ P(Y ∼ PY , U ∼ PC) is the set of all joint distributions of (Y, U) with marginals PY and PC , and `(y, u) : U × U 7→ R+ is any measurable cost function. QZ is the marginal distribution of Z when X ∼ PX and Z ∼ Q(Z|X). (The proof is presented later.)
Therefore, optimizing over the objective on the r.h.s is equivalent to minimizing the discrepancy between the true label distribution (PY ) and the output distribution PC , thus the important classification features are embedded in the low-dimensional space. This is the core idea of the paper,
summarizing the high-dimensional data in a space of much lower dimension without losing important features for classification. To implement the r.h.s objective, the constraint on QZ can be relaxed by adding a penalty term. The final objective of ER-Classifier is:
inf Q(Z|X)∈Q EPXEQ(Z|X) {`(f(X),C(Z))}+ λD(QZ , PZ), (3)
whereQ is any nonparametric set of probabilistic encoders, λ > 0 is a hyper-parameter and D is an arbitrary divergence between QZ and PZ .
To estimate the divergences between QZ and PZ , we apply a GAN-based framework, fitting a discriminator to minimize the 1-Wasserstein distance between QZ and PZ :
W (QZ , PZ) = inf Γ∈P(Z̃∼QZ ,Z∼PZ) E(Z̃,Z)∼Γ‖Z̃ − Z‖.
We have also tried the Jensen-Shannon divergence, but as expected, Wasserstein distance provides more stable training and better results. When training the framework, the weight clipping method proposed in Wasserstein GAN (Arjovsky et al., 2017) is applied to help stabilize the training of discriminator Dγ .

heading: PROOF OF THEOREM 1
text: The proof of Theorem 1 is adapted from the proof of Theorem 1 in (Tolstikhin et al., 2017). Consider certain sets of joint probability distributions of three random variables (X,U,Z) ∈ X × U × Z . X can be taken as the input images, U as the output of the framework, and Z as the latent codes. PC,Z(U,Z) represents a joint distribution of a variable pair (U,Z), where Z is first sampled from PZ and then U from PC(U |Z). PC defined in (2) is the marginal distribution of U when (U,Z) ∼ PC,Z .
The joint distributions Γ(X,U) or couplings between values of X and U can be written as Γ(X,U) = Γ(U |X)PX(X) due to the marginal constraint. Γ(U |X) can be decomposed into an encoding distribution Q(Z|X) and the generating distribution PC(U |Z), and Theorem 1 mainly shows how to factor it through Z.
In the first part, we will show that if PC(U |Z) are Dirac measures, we have
inf Γ∈P(X∼PX ,U∼PC)
E(X,U)∼Γ {`(f(X), U)}
= inf Γ∈PX,U
E(X,U)∼Γ {`(f(X), U)} , (4)
where P(X ∼ PX , U ∼ PC) denotes the set of all joint distributions of (X,U) with marginals PX , PC , and likewise for P(X ∼ PX , Z ∼ PZ). The set of all joint distributions of (X,U,Z) such that X ∼ PX , (U,Z) ∼ PC,Z , and (U ⊥ X)|Z are denoted by PX,U,Z . PX,U and PX,Z denote the sets of marginals on (X,U) and (X,Z) induced by PX,U,Z . From the definition, it is clear that PX,U ⊆ P(PX , PC). Therefore, we have
inf Γ∈P(X∼PX ,U∼PC)
E(X,U)∼Γ {`(f(X), U)}
≤ inf Γ∈PX,U E(X,U)∼Γ {`(f(X), U)} , (5)
The identity is satisfied if PC(U |Z) are Dirac measures, such as U = C(Z). This is proved by the following Lemma in (Tolstikhin et al., 2017). -5pt
Lemma 1 PX,U ⊆ P(PX , PC) with identity if PC(U |Z = z) are Dirac for all z ∈ Z . (see details in (Tolstikhin et al., 2017).)
In the following part, we show that
inf Γ∈PX,U
E(X,U)∼Γ {`(f(X), U)}
= inf Q:QZ=PZ EPXEQ(Z|X) {`(f(X),C(Z))} . (6)
Based on the definition, P(PX , PC), PX,U,Z and PX,U depend on the choice of conditional distributions PC(U |Z), but PX,Z does not. It is also easy to check that PX,Z = P(X ∼ PX , Z ∼ PZ). The tower rule of expectation, and the conditional independence property of PX,U,Z implies
inf Γ∈PX,U
E(X,U)∼Γ {`(f(X), U)}
= inf Γ∈PX,U,Z
E(X,U,Z)∼Γ {`(f(X), U)}
= inf Γ∈PX,U,Z EPZEX∼P (X|Z)EU∼P (U |Z) {`(f(X), U)}
= inf Γ∈PX,U,Z EPZEX∼P (X|Z) {`(f(X),C(Z))}
= inf Γ∈PX,Z
E(X,Z)∼Γ {`(f(X),C(Z))}
= inf Q:QZ=PZ EPXEQ(Z|X) {`(f(X),C(Z))} (7)
Finally, since Y = f(X), it is easy to get
inf Γ∈P(Y∼PY ,U∼PC)
E(Y,U)∼Γ {`(Y,U)}
= inf Γ∈P(X∼PX ,U∼PC)
E(X,U)∼Γ {`(f(X), U)} (8)
Now (4), (6) and (8) are proved and the three together prove Theorem 1.
Our proposed framework readily applies to non-deterministic case. If the classifier part is nondeterministic, Lemma 1 provides only the inclusion of sets PX,U ⊆ P(PX , PU ), and we can get an upper bound on the Wasserstein distance between the ground-truth and predicted label distributions:
inf Γ∈P(X∼PX ,U∼PC) E(X,U)∼Γ {`(f(X), U)} ≤ inf Γ∈PX,U E(X,U)∼Γ {`(f(X), U)}
≤ d∑ i=1 σ2i + inf Γ∈PX∼PX,Z∼PZ E(X,Z)∼Γ { ‖f(X)−C(Z)‖2 } , (9)
where we assume the conditional distributions PC(U |Z = z) have mean values C(z) ∈ Rd and marginal variances σ21 , ..., σ 2 d ≥ 0 for all z ∈ Z , where C : Z → X , and `(y, u) = ‖y − u‖2. The above upper bound is derived by:
inf Γ∈PX,U
E(X,U)∼Γ { ‖f(X)− U‖2 } = inf
Γ∈PX,U,Z EPZEX∼P (X|Z)EU∼P (U |Z){‖f(X)− U‖2}
(10)
and
EU∼P (U |Z){‖f(X)− U‖2} = EU∼P (U |Z){‖f(X)−C(Z) + C(Z)− U‖2} = ‖f(X)−C(Z)‖2 + EU∼P (U |Z){< f(X)−C(Z),C(Z)− U >}+ EU∼P (U |Z){‖C(Z)| − U‖2}
= ‖f(X)−C(Z)‖2 + d∑ i=1 σ2i . (11)
In equation 11, the second term of the second last row becomes 0 since the optimization will drive f(X)−C(Z) to zero.

heading: HYPER-PARAMETER SELECTION

heading: DIMENSION OF EMBEDDING SPACE
text: One important hyper-parameter for the ER-Classifier is the dimension of the embedding space. If the dimension is too small, important features are “collapsed” onto the same dimension, and if the dimension is too large, the projection will not extract useful information, which results in too much noise and instability. The maximum likelihood estimation of intrinsic dimension proposed in
(Levina & Bickel, 2005)3 is used to calculate the intrinsic dimension of each image dataset, serving as a guide for selecting the embedding dimension. The sample size used in calculating the intrinsic dimension is 1, 000, and changing the sample size does not influence the results much. Based on the intrinsic dimension calculated by (Levina & Bickel, 2005), we test several different values around the suggested intrinsic dimension and evaluate the models against l∞-PGD attack. All models are trained without min-max robust optimization, and the experimental results are shown in Figure 5.
The final embedding dimension is chosen based on robustness, number of parameters, and testing accuracy when there is no attack. The final embedding dimensions and suggested intrinsic dimensions are shown in Table 3.
Based on Figure 5, the embedding dimension close to the calculated intrinsic dimension usually offers better results except on MNIST. One explanation may be that MNIST is a simple handwritten digit dataset, so performing classification on MNIST may not require that many dimensions.

heading: EPSILON SELECTION
text: Epsilon ( ) is an important hyper-parameter for adversarial training. When doing Madry’s adversarial training, we test the model robustness with different and choose the best one. The experiment results are shown in Figure 6.
Based on Figure 6, we use = 0.3, 0.03, 0.03 in Madry’s adversarial training on MNIST, CIFAR10 and STL10 respectively. For Tiny Imagenet, we use = 0.01. To make a fair comparison, we use the same when training ER-Classifier.
3Code publicly available at https://github.com/OFAI/hub-toolbox-python3
EMBEDDING VISUALIZATION
In this section, we compare the embedding learned by Encoder+Classifier structure (E-CLA) and the embedding learned by ER-Classifier on several datasets without min-max robust optimization. We first generate embedding of testing data using the encoder (z̃ = Qφ(x)), then project the embedding points (z̃) to 2-D space by tSNE(Maaten & Hinton, 2008). Then we generate adversarial images (xadv) against E-CLA and ER-Classifier using l∞-PGD attack. The adversarial embedding is generated by feeding the adversarial images into the encoder (z̃adv = Qφ(xadv)). Finally, we project the adversarial embedding points (z̃adv) to 2-D space. The results are shown in Figure 7. The plots in the first and second rows are embedding visualization plots for E-CLA, and the plots in the third and last rows are the embedding visualization plots for ER-Classifier. In adversarial embedding visualization plots, the misclassified point is marked as “down triangle”, which means the PGD attack successfully changed the prediction, and the correctly classified point is marked as “point”, which means the attack fails.
Based on Figure 7, we can see that E-CLA can learn a good embedding on legitimate images of MNIST. Embedding points for different classes are separated on the 2D space, but under adversarial attack, some embedding points of different classes are mixed together. However, ER-Classifier can generate good separated embeddings on both legitimate and adversarial images. On CIFAR10, the E-CLA can not generate good separated embeddings on either legitimate images or adversarial images, while ER-Classifier can generate good separated embeddings for both.

heading: PSEUDO-CODE
text: Code for reproduction will be made available online at github later. The pseudocode for training ER-Classifier is shown in Listing 1.

heading: MODEL STRUCTURE
text: MNIST, STL10 and TinyImagenet classifier structures used for baseline methods are shown in Figure 8. We use VGG19 for the baseline methods on CIFAR10. Details of ER-Classifier structures on the four benchmark datasets are shown in Figure 9-10.
Original Embedding of E-CLA on MNIST
Adversarial Embedding of E-CLA on MNIST Classified Right Classified Wrong
Original Embedding of E-CLA on CIFAR10
Adversarial Embedding of E-CLA on CIFAR10 Classified Right Classified Wrong
Original Embedding of ER-classifier on MNIST
Adversarial Embedding of ER-classifier on MNIST Classified Right Classified Wrong
Original Embedding of ER-classifier on CIFAR10
Adversarial Embedding of ER-classifier on CIFAR10 Classified Right Classified Wrong
1 d e f t r a i n e r c l a ( t r a i n l o a d e r , t e s t l o a d e r , encoder , d i s c r i m i n a t o r , 2 c l a s s i f i e r , o t h e r h y p e r p a r a m e t e r s ) : 3 c r i t e r i o n = nn . C r o s s E n t r o p y L o s s ( ) 4 e n c o d e r . t r a i n ( ) 5 d i s c r i m i n a t o r . t r a i n ( ) 6 c l a s s i f i e r . t r a i n ( ) 7 # O p t i m i z e r s 8 e n c o p t i m = opt im . Adam( e n c o d e r . p a r a m e t e r s ( ) , l r = l r ) 9 d i s o p t i m = opt im . Adam( d i s c r i m i n a t o r . p a r a m e t e r s ( ) , l r = 0 . 5 ∗ l r )
10 c l a o p t i m = opt im . Adam( c l a s s i f i e r . p a r a m e t e r s ( ) , l r = 0 . 0 5 ∗ l r ) 11 e n c s c h e d u l e r = StepLR ( enc op t im , s t e p s i z e =30 , gamma = 0 . 5 ) 12 d i s s c h e d u l e r = StepLR ( d i s o p t i m , s t e p s i z e =30 , gamma = 0 . 5 ) 13 c l a s c h e d u l e r = StepLR ( c l a o p t i m , s t e p s i z e =30 , gamma = 0 . 5 ) 14 one = t o r c h . Tensor ( [ 1 ] ) 15 mone = one ∗ −1 16 f o r epoch i n r a n g e ( num epoch ) : 17 s t e p = 0 18 f o r images , l a b e l s i n tqdm ( t r a i n l o a d e r ) : 19 e n c o d e r . z e r o g r a d ( ) 20 d i s c r i m i n a t o r . z e r o g r a d ( ) 21 c l a s s i f i e r . z e r o g r a d ( ) 22 # ======== Min−Max Robus t O p t i m i z a t i o n ======== # 23 images = a d v g e t ( images , c l a s s i f i e r , e n c o d e r ) 24 # ======== T r a i n D i s c r i m i n a t o r ======== # 25 f r o z e n p a r a m s ( e n c o d e r ) 26 f r o z e n p a r a m s ( c l a s s i f i e r ) 27 f r e e p a r a m s ( d i s c r i m i n a t o r ) 28 z f a k e = s a m p l e z ( p r i o r , n z , b a t c h s i z e , s igma ) 29 d f a k e = d i s c r i m i n a t o r ( t o v a r ( z f a k e ) ) 30 z r e a l = e n c o d e r ( images ) 31 d r e a l = d i s c r i m i n a t o r ( t o v a r ( z r e a l ) ) 32 d i s c f a k e = LAMBDA ∗ d f a k e . mean ( ) 33 d i s c r e a l = LAMBDA ∗ d r e a l . mean ( ) 34 d i s c f a k e . backward ( one ) 35 d i s c r e a l . backward ( mone ) 36 d i s s l o s s = d i s c f a k e − d i s c r e a l 37 d i s o p t i m . s t e p ( ) 38 c l i p p a r a m s ( d i s c r i m i n a t o r ) 39 # ======== T r a i n C l a s s i f i e r and Encoder ======== # 40 f r e e p a r a m s ( e n c o d e r ) 41 f r e e p a r a m s ( c l a s s i f i e r ) 42 f r o z e n p a r a m s ( d i s c r i m i n a t o r ) 43 p r e d l a b e l s = c l a s s i f i e r ( e n c o d e r ( t o v a r ( images ) ) ) 44 c l a s s l o s s = LAMBDA0 ∗ c r i t e r i o n ( p r e d l a b e l s , l a b e l s ) 45 c l a s s l o s s . backward ( ) 46 c l a o p t i m . s t e p ( ) 47 e n c o p t i m . s t e p ( ) 48 # ======== T r a i n Encoder ======== # 49 f r e e p a r a m s ( e n c o d e r ) 50 f r o z e n p a r a m s ( c l a s s i f i e r ) 51 f r o z e n p a r a m s ( d i s c r i m i n a t o r ) 52 z r e a l = e n c o d e r ( images ) 53 d r e a l = d i s c r i m i n a t o r ( e n c o d e r ( V a r i a b l e ( images . d a t a ) ) ) 54 d l o s s = LAMBDA1 ∗ ( d r e a l . mean ( ) ) 55 d l o s s . backward ( one ) 56 e n c o p t i m . s t e p ( ) 57 s t e p += 1 58 s a v e f i l e ( f i l e n a m e , encoder , d i s c r i m i n a t o r , c l a s s i f i e r , d a t a s e t = d a t a s e t ) 59 r e t u r n c l a s s i f i e r , e n c o d e r
Listing 1: Pseudocode for training ER-Classifier

Review summaries:
The paper proposed a adversarial learning framework that tries to align the hidden features of data with simple prior distributions.  A training strategy similar to GAN  was exploited. The proposed framework was argued that it can well deal with the adversarial perturbations. Some experiments were conducted, verifying that the proposed algorithm seems useful and robust.

This paper proposes a new regularization technique called Embedding Regularization to improve the adversarial robustness.  The idea is to use generative adversarial networks (GAN) to perform inference on the latent space by matching the aggregated posterior of the hidden space vector with a prior distribution. The proposed strategy could be combined with adversarial training to achieve state-of-the-art adversarial accuracy on several benchmark datasets.