Year: 2020

Abstract:
We propose a novel adversarial training method which leverages both the local and global information to defend adversarial attacks. Existing adversarial training methods usually generate adversarial perturbations locally in a supervised manner and fail to consider the data manifold information in a global way. Consequently, the resulting adversarial examples may corrupt the underlying data structure and are typically biased towards the decision boundary. In this work, we exploit both the local and global information of data manifold to generate adversarial examples in an unsupervised manner. Specifically, we design our novel framework via an adversarial game between a discriminator and a classifier: the discriminator is learned to differentiate the latent distributions of the natural data and the perturbed counterpart, while the classifier is trained to recognize accurately the perturbed examples as well as enforcing the invariance between the two latent distributions. We conduct a series of analysis on the model robustness and also verify the effectiveness of our proposed method empirically. Experimental results show that our method substantially outperforms the recent state-of-the-art (i.e. Feature Scattering) in defending adversarial attacks by a large accuracy margin (e.g. 17.0% and 18.1% on SVHN dataset, 9.3% and 17.4% on CIFAR-10 dataset, 6.0% and 16.2% on CIFAR-100 dataset for defending PGD20 and CW20 attacks respectively).

heading: 1 INTRODUCTION
text: Deep Neural Networks (DNNs) have achieved impressive performance on a broad range of datasets, yet can be easily fooled by adversarial examples or perturbations (LeCun et al., 2015; He et al., 2016; Gers et al., 1999). Adversarial examples have been shown to be ubiquitous beyond different tasks such as image classification (Goodfellow et al., 2014), segmentation (Fischer et al., 2017), and speech recognition (Carlini & Wagner, 2018). Overall, adversarial examples raise great concerns about the robustness of learning models, and have drawn enormous attention over recent years.
To defend adversarial examples, great efforts have been made to improve the model robustness (Kannan et al., 2018; You et al., 2019; Wang & Zhang, 2019; Zhang & Wang, 2019). Most of them are based on the adversarial training, i.e. training the model with adversarially-perturbed samples rather than clean data (Goodfellow et al., 2014; Madry et al., 2017; Lyu et al., 2015). In principle, adversarial training is a min-max game between the adversarial perturbations and classifier. Namely, the indistinguishable adversarial perturbations are designed to mislead the output of the classifier, while the classifier is trained to produce the accurate predictions for these perturbed input data. Currently, the adversarial perturbations are mainly computed by enforcing the output invariance in a supervised manner (Madry et al., 2017). Despite its effectiveness in some scenarios, it is observed recently that these approaches may still be limited in defending adversarial examples.
In particular, we argue that these current adversarial training approaches are typically conducted in a local and supervised way and fail to consider globally the overall data manifold information; such information however proves crucially important for attaining better generalization. As a result, the generated adversarial examples may corrupt the underlying data structure and would be typically biased towards the decision boundary. Therefore, the well-generalizing features inherent to the data distribution might be lost, which limits the performance of the DNNs to defend adversarial examples even if adversarial training is applied (Ilyas et al., 2019a; Schmidt et al., 2018). For illustration, we have shown a toy example in Figure 1. As clearly observed, adversarially-perturbed examples gen-
erated by PGD, one of the most successful adversarial training method, corrupt the data manifold, which would inevitably lead to poor performance if the training is conducted based on these perturbed examples. On the other hand, the current state-of-the-art method Feature Scattering (Zhang & Wang, 2019) can partially alleviate this problem but still leads to corruptions on the data manifold.
To address this limitation, we propose a novel method called Adversarial Training with Latent Distribution (ATLD) which additionally considers the data distribution globally in an unsupervised fashion. In this way, the data manifold could be well preserved, which is beneficial to attain better model generalization. Moreover, since the label information is not required when computing the adversarial perturbations, the resulting adversarial examples would not be biased towards the decision boundary. This can be clearly observed in Figure 1(d).
Our method can be divided into two steps: first, we train the deep model with the adversarial examples which maximize the variance between latent distributions of clean data and adversarial counterpart rather than maximizing the loss function. We reformulate it as a minimax game between a discriminator and a classifier. The adversarial examples are crafted by the discriminator to make different implicitly the latent distributions of clean and perturbed data, while the classifier is trained to decrease the discrepancy between these two latent distributions as well as promoting accurate classification on the adversarial examples as Figure 2 shows. Then, during the inference procedure, we generate the specific perturbations through the discriminator network to diminish the impact of the adversarial attack as shown in Figure 6 in Appendix.
On the empirical front, with the toy examples, we show that our proposed method can preserve more information of the original distribution and learn a better decision boundary than the existing adversarial training method. We also test our method on three different datasets: CIFAR-10, CIFAR100 and SVHN with the famous PGD, CW and FGSM attacks. Our ATLD method outperforms the state-of-the-art methods by a large margin. e.g. ATLD improves over Feature Scattering (Zhang & Wang, 2019) by 17.0% and 18.1% on SVHN for PGD20 and CW20 attacks. Our method also shows a large superiority to the conventional adversarial training method (Madry et al., 2017), boosting the performance by 32.0% and 30.7% on SVHN for PGD20 and CW20 attacks.

heading: 2 RELATED WORK
text: Adversarial Training. Adversarial training is a family of techniques to improve the model robustness (Madry et al., 2017; Lyu et al., 2015). It trains the DNNs with adversarially-perturbed samples instead of clean data. Some approaches extend the conventional adversarial training by injecting the adversarial noise to hidden layers to boost the robustness of latent space (Ilyas et al., 2019b; You et al., 2019; Santurkar et al., 2019; Liu et al., 2019). All of these approaches generate the adversarial examples by maximizing the loss function with the label information. However, the structure of the data distribution is destroyed since the perturbed samples could be highly biased towards the non-optimal decision boundary (Zhang & Wang, 2019). Our proposed method has a similar training scheme with adversarial training by replacing clean data with the perturbed one. Nevertheless, our method generates the adversarial perturbations without the label information which weakens the impact of non-optimal decision boundary and can retain more information of the underlying data distribution.
Manifold-based Adversarial Training. Song et al. (2017) propose to generate the adversarial examples by projecting on a proper manifold. Zhang & Wang (2019) leverage the manifold information in the forms of inter-sample relationship within the batch to generate adversarial adversarial perturbations. Virtual Adversarial Training and Manifold Adversarial Training are proposed improve model generalization and robustness against adversarial examples by ensuring the local smoothness of the data distribution (Zhang et al., 2018; Miyato et al., 2017). Some methods are designed to enforce the local smoothness around the natural examples by penalizing the difference between the outputs of adversarial examples and clean counterparts (Kannan et al., 2018; Chan et al., 2020; Jakubovitz & Giryes, 2018). All of these methods just leverage the local information of the distribution or manifold. Differently, our method generates the perturbations additionally considering the structure of distribution globally.
Unsupervised Domain Adversarial Training. Domain Adversarial Training shares a training scheme similar to our method where the classifier and discriminator compete with each other (Odena et al., 2017; Long et al., 2018; Ganin et al., 2016). However, its objective is to reduce the gap between the source and target distributions in the latent space. The discriminator is used to measure the divergence between these two distributions in the latent space. The training scheme of our method is also based on competition between the classifier and discriminator. Different from the previous framework, the discriminator of our method is used to capture the information of distributions of adversarial examples and clean counterparts in the latent space which helps generate the adversarial perturbations.
GAN-based Adversarial Training Methods. Several GAN-based methods leverage GANs to learn the clean data distribution and purify the adversarial examples by projecting them on clean data manifold before classification (Meng & Chen, 2017; Metzen et al., 2017). The framework of GAN can also be used to generate the adversarial examples (Baluja & Fischer, 2018). The generator produces the adversarial examples to deceive both the discriminator and classifier; the discriminator and classifier attempt to differentiate the adversaries from clean data and produce the correct labels respectively. Some adversary detector networks are proposed to detect the adversarial examples which can be well aligned with our method (Gong et al., 2017; Grosse et al., 2017). In these works, a pretrained network is augmented with a binary detector network. The training of the pretrained network and detector involves generating adversarial examples to maximize their losses. Differently, our method generates the adversarial examples just to minimize the loss of the discriminator and feed them as the training set to the classifier. Such adversarial examples are deemed to induce most different latent representations from the clean counterpart.

heading: 3 BACKGROUND

heading: 3.1 ADVERSARIAL TRAINING
text: Let us first introduce the widely-adopted adversarial training method for defending against adversarial attacks. Specifically, it solves the following minimax optimization problem through training.
min θ {E(x,y)∼D[ max x′∈Sx L(x′, y; θ)]}, (1)
where x ∈ Rn and y ∈ R are respectively the clean data samples and the corresponding labels drawn from the dataset D, and L(·) is the loss function of the DNN with the model parameter θ ∈ Rm. Furthermore, we denote the clean data distribution as Q0, i.e. x ∼ Q0. , and denote x′ ∈ Rn as perturbed samples in a feasible region Sx , {z : z ∈ B(x, ) ∩ [−1.0, 1.0]n} with B(z, ) , {z : ‖x− z‖∞ ≤ } being the `∞-ball at center x with radius . By defining fθ(·) as the mapping function from the input layer to the last latent layer, we can also rewrite the loss function of the DNN as l(fθ(x), y) where l(·) denotes the loss function calculated from the last hidden layer of the DNN, e.g. the cross entropy loss as typically used in DNN.
Whilst the outer minimization can be conducted by training to find the optimal model parameters θ, the inner maximization essentially generates the strongest adversarial attacks on a given set of model parameters θ. In general, the solution to the minimax problem can be found by training a network minimizing the loss for worst-case adversarial examples, so as to attain adversarial robustness. Given a set of model parameters θ, the commonly adopted solution to the inner maximization problem can lead to either one-step (e.g., FGSM) or multi-step (e.g., PGD) approach (Madry et al., 2017). In
particular, for a given single point x, the strongest adversarial example x′ at the t-th iteration can be iteratively obtained by the following updating rule:
xt+1 = ΠSx(x t + α · sgn(∇xL(xt, y; θ))), (2)
where ΠSx(·) is a projection operator to project the inputs onto the region Sx, sgn(·) is the sign function, and α is the updating step size. For the initialization, x0 can be generated by randomly sampling in B(x, ).
It appears in (1) that each perturbed sample x′ is obtained individually by leveraging its loss function L(x′, y; θ) with its label y. However, without considering the inter-relationship between samples, we may lose the global knowledge of the data manifold structure which proves highly useful for attaining better generalization. This issue has been studied in a recent work (Zhang & Wang, 2019) where a new method named feature scattering made a first step to consider the inter-sample relationship within the batch; unfortunately this approach did not take the full advantages of the global knowledge of the entire data distribution. In addition, relying on the maximization of the loss function, the adversarially-perturbed data samples may be highly biased towards the decision boundary, which potentially corrupts the structure of the original data distribution, especially when the decision boundary is non-optimal (see Figure 1 again for the illustration).

heading: 3.2 DIVERGENCE ESTIMATION
text: To measure the discrepancy of two distributions, statistical divergence measures (e.g., KullbackLeibler and Jensen-Shannon divergence) have been proposed. In general, given two distributions P and Q with a continuous density function p(x) and q(x) respectively, f -divergence is defined as Df (P||Q) , ∫ X q(x)f ( p(x) q(x) ) dx. The exact computation of f -divergence is challenging, and the estimation from samples has attracted much interest. For instance, leveraging the variational methods, Nguyen et al. (2010) propose a method for estimating f -divergence from only samples; Nowozin et al. (2016) extend this method by estimating the divergence with learning the parameters of discriminator. Specifically, the f -divergence between two distributions P and Q can be lowerbounded using Fenchel conjugate and Jensen’s inequality (Nowozin et al., 2016).
Df (P||Q) = ∫ X q(x) sup t∈domf∗ {tp(x) q(x) − f∗(t)}dx
≥ sup T∈τ ( ∫ X p(x)T (x)dx− ∫ X q(x)f∗(T (x))dx)
= sup W
(Ex∼P[gf (VW (x))] + Ex∼Q[−f∗(gf (VW (x)))]),
(3)
where VW : X → R is a discriminator network with parameter W and gf : R → domf∗ is an output activation function which is determined by the type of discriminator. τ is an arbitrary class of functions T : X → R. f is a convex lower-semicontinous function and f∗ is its conjugate defined by f∗(t) = supu∈domf [ut− f(u)]. The objective of discriminator for GANs is a special case of (3) with the activation function gf (t) = − log(1+e−t) and f∗(g) = − log(2−eg). It approximates the Jense-Shannon divergence between real and fake distributions. Arjovsky et al. (2017) also develop a method to estimate the Wasserstein-distance by neural network. In this paper, these methods will be used to estimate the Jensen-Shannon divergence between latent distributions induced by adversarial and clean examples.

heading: 4 ADVERSARIAL TRAINING WITH LATENT DISTRIBUTION
text: As discussed in Section 3.1, the conventional adversarial training methods rely on the knowledge of data labels. As a result, the local information to generate adversarial examples may be biased toward the decision boundary; such individual adversarial example generation does not capture the global knowledge of the data manifold.
To alleviate these limitations, we propose a novel method to compute the perturbed samples by leveraging the global knowledge of the whole data distribution and then disentangling them from the data labels and the loss function. Generally speaking, the perturbations are generated to enlarge the variance between latent distributions induced by clean and adversarial data.
Formally, we try to identify the set of adversarial examples Xadv that yield in the latent space a distribution P ∗θ through fθ(·) that is the most different from the latent distribution Qθ induced by the clean samples Xorg = {x : x ∼ Q0}, without resorting to the corresponding labels Y . In other words, the resulting adversarial examples can be deemed as manifold adversarial examples, which ‘deceive’ the manifold rather than fool the classifier as defined in the traditional adversarial examples. It is noted that the latent space to be perturbed could be any hidden layer though it is defined in the last hidden layer before the softmax of a DNN in this paper. The optimization problem of the proposed adversarial training can then be reformulated as follows:
min θ {Efθ(xadv)∼P∗θ [l(fθ(x adv), y)] +Df (P ∗ θ ||Qθ)} (4)
s.t. P ∗θ = arg max Pθ∈P [Df (Pθ||Qθ)] (5)
where l(·) and y are similarly defined as before, and Df (·) is the f -divergence measure of two distributions. P = {P : fθ(x′) ∼ P subject to ∀x ∼ Q0, x′ ∈ B(x, )} is the feasible region for the latent distribution Pθ which is induced by the set of perturbed examplesXp through fθ(·). fθ(x′) and fθ(xadv) represents the latent features of the perturbed example x′ and adversarial example xadv respectively. Intuitively, we try to obtain the worst latent distribution P ∗θ which is induced by Xadv through fθ(·) within the region P , while the model parameter θ is learned to minimize the classification loss on the latent feature fθ(xadv) ∼ P ∗θ (or equivalently adversarial example xadv ∈ Xadv) and the f -divergence between the latent distributions P ∗θ and Qθ induced by adversarial examples Xadv and clean data Xorg.
It is still challenging to solve the above optimization problem, since both the objective function and the constraint are entangled with the adversarial examples Xadv and the model parameters θ. To make the problem more tractable, we propose a novel Adversarial Training with Latent Distribution (ATLD) method. In the next subsection, by taking into account the entire data distribution globally, we first focus on the constraint and identify the adversarial samples Xadv through the maximization problem. We then solve the minimization of the objective function with the adversarial training procedure. To further enhance the performance, we add specific perturbations named Inference with Manifold Transformation (IMT) in Section 4.2 to input samples for enforcing them towards the more separable natural data manifold. Finally, we classify the transformed data points with the adversarially-trained model.

heading: 4.1 GENERATING ADVERSARIAL EXAMPLES FOR TRAINING
text: First, we optimize the constraint (5) to generate the adversarial examples or its induced distribution P ∗θ for training. Intuitively, the adversarial examples Xadv are crafted to maximize the divergence between the latent distributions induced by natural examplesXorg and adversarial counterpartXadv in an unsupervised fashion since no knowledge of labels Y is required. Together with the objective function in (4), our proposed adversarial training method is to minimize such divergence as well as the classification error for adversarial examples Xadv .
However, it is a challenging task to evaluate the divergence between two latent distributions. To make it more tractable, we leverage a discriminator network for estimating the Jensen-Shannon divergence between two distributions P ∗θ /Pθ and Qθ according to Section 3.2. It is noted again that the class label information is not used for generating adversarial examples. Hence the adversarial examples are still generated in an unsupervised way. Then, by using (3), the optimization problem in (4) and (5) can be approximated as follows in a tractable way.
min θ { N∑ i=1 L(xadvi , yi; θ)︸ ︷︷ ︸ Lf + sup W N∑ i=1 [logDW (fθ(x adv i )) + (1− logDW (fθ(xi)))︸ ︷︷ ︸ Ld ] }
s.t. xadvi = arg max x′i∈B(xi, ) [logDW (fθ(x ′ i)) + (1− logDW (fθ(xi)))︸ ︷︷ ︸
Ld
]
(6)
where N denotes the number of training samples and DW denotes the discriminator network with the sigmoid activation function and parameter W . fθ(xi) is the latent feature of the clean sample xi. DW is used to determine whether the latent feature is from adversary manifold (output the
manifold label of the latent feature). For ease of description, we represent the components in Eq. (6) as two parts: Lf and Ld. Ld is the manifold loss and Lf represents the loss function of the classifier network.
We now interpret the above optimization problem. By comparing Eq. (6) and Eq. (4), it is observed that the Jensen-Shannon divergence between P ∗θ andQθ is approximated by supW ∑N i=1 Ld, and the
minimization of the classification loss on adversarial examples is given by minθ ∑N i=1 Lf . The problem (6) is optimized by updating parameters θ and W and crafting adversarial examples {xadvi }Ni=0 iteratively. The whole training procedure can be viewed as the game among three players: the classifier, discriminator, and adversarial examples. The discriminator DW is learned to differentiate the latent distributions of the perturbed examples and clean data via maximizing the loss Ld while the classifier fθ is trained to (1) enforce the invariance between these two distributions to confuse the discriminatorDW by minimizing the loss Ld, and (2) classify the adversarial examples as accurately as possible by minimizing Lf . For each training iteration, the adversarial examples are crafted to make different the adversarial latent distribution and natural one by maximizing Ld. Although DW cannot measure the divergence between the two latent distributions exactly at the first several training steps, it can help evaluate the divergence between distributions induced by perturbed examples and clean ones when the parameters W converges.
However, when the latent distributions are multi-modal, which is a real scenario due to the nature of multi-class classification, it is challenging for the discriminator to measure the divergence between such distributions. Several work reveals that there is a high risk of failure in using the discriminator networks to measure only a fraction of components underlying different distributions (Arjovsky & Bottou, 2017; Che et al., 2016). Ma (2018) also shows that two different distributions are not guaranteed to be identical even if the discriminator is fully confused. To alleviate such the problem, we additionally train the discriminator DW to predict the class labels for latent features as (Odena et al., 2017; Long et al., 2018). As a result, the problem (6) can then be reformulated as:
min θ { N∑ i=1 L(xadvi , yi; θ)︸ ︷︷ ︸ Lf + sup W N∑ i=1 [logD0W (fθ(x adv i )) + (1− logD0W (fθ(xi)))︸ ︷︷ ︸
L0d
]
+ min W
[l(D1:CW (fθ(xi)), yi) + l(D 1:C W (fθ(x adv i )), yi)]︸ ︷︷ ︸
L1:Cd } s.t. xadvi = arg max
x′i∈B(xi, ) [logD0W (fθ(x ′ i)) + (1− logD0W (fθ(xi))︸ ︷︷ ︸
L0d
]
(7)
HereD0W is the first dimension of the output of the discriminator, which indicates the manifold label of the latent features; D1:CW are the remaining C dimensions of the output of DW , used to output the class label of the latent feature; C denotes the number of classes, and L0d and L 1:C d are the manifold loss and the classification loss for the discriminator network respectively. (The detailed derivation for Eq. (6) and Eq. (7) can be seen in Appendix.) The detailed training procedure of our framework is depicted in Figure 2.
Remarks. It is worth noting that the labeled information is not required for generating adversarial examples. Therefore, our method prevents the perturbed examples from highly biasing towards the decision boundary and more information of the original distribution structure is preserved. In addition, since the discriminator is trained with the whole dataset (both clean and adversarial examples), it captures the global information of data manifold. Consequently, by training with adversarial examples generated according to the manifold loss of the discriminator, our method can improve the model robustness against adversarial examples with the global structure of data distribution.

heading: 4.2 INFERENCE WITH MANIFOLD TRANSFORMATION
text: To enhance the generalization of ATLD, we further develop a new inference method with manifold transformation. Although adversarially-trained models can well recognize adversarial examples, there are still potential examples which are easily misclassified especially for unseen data. In other words, the generalization to adversarial examples is hard to achieve due to the more complex distribution of adversarial examples (Schmidt et al., 2018; Zhai et al., 2019). To alleviate this problem,
our proposed inference method first pushes adversarial examples towards the manifold of natural examples which is simpler and further away from the decision boundary than the adversarial distribution. Then the more separable adjusted examples are classified by the adversarially-trained model. Specifically, the input sample is fed into our adversarially-trained model and the discriminator outputs the probability of such a sample lying on the adversarial manifold. If this probability is higher than a certain threshold, we compute the transformed example xt by adding the specific perturbation r∗ to the input sample x to reduce such a probability. This perturbation can be computed as:
r∗ = arg min ‖r‖∞≤ logD0W (fθ(x+ r)). (8)
Intuitively, the reduction of probability of this data point lying on adversarial manifold indicates that this point moves towards the benign example manifold after adding perturbation r∗. In other words, it becomes more separable since the benign example manifold is further away from the decision boundary. When the probability of the image lying on adversary manifold is lower than threshold, we still add such a perturbation to input image to make it more separable but with smaller magnitude. In the experiment part, we show this perturbation can move the adversarial examples away from the decision boundary. The whole inference procedure can be seen in Figure 5 in Appendix.

heading: 5 EXPERIMENTS
text: We conduct experiments on the widely-used datasets, e.g., CIFAR-10, SVHN, and CIFAR-100. Following the Feature Scattering method (Zhang & Wang, 2019), we leverage the wideresnet (Zagoruyko & Komodakis, 2016) as our basic classifier and discriminator model structure. During the training phase, the initial learning rate is empirically set to 0.1 for all three datasets. We train our model 400 epochs with the transition epoch 60, 90 and the decay rate 0.1. The input perturbation budget is set to = 8 with the label smoothing rate as 0.5. We use L∞ perturbation in this paper including all the training and evaluation.
We evaluate the various models on white-box attacks and black-box attacks. Under the white-box attacks, we compare the accuracy of the proposed method with several competitive methods, including: (1) the original wideresnet (Standard) trained with natural examples; (2) Traditional Adversarial Training with PGD (AT) (Madry et al., 2017); (3) Triplet Loss Adversarial training (TLA) (Mao et al., 2019); (4) Layer-wise Adversarial Training (LAT): injecting adversarial perturbation into the latent space (Sinha et al., 2019); (5) Bilateral: adversarial perturb on examples and labels both (Wang & Zhang, 2019); (6) Feature-scattering: generating adversarial examples with considering interrelationship of samples (Zhang & Wang, 2019). These comparison algorithms present the most competitive performance on defending adversarial attack. Under the black-box attacks, we compare four different algorithms used to generate the test time attacks: Vanilla training with natural examples, adversarial training with PGD, Feature Scattering, and our proposed model.

heading: 5.1 DEFENDING WHITE-BOX ATTACKS
text: We show the classification accuracy under several white-box attacks on CIFAR-10, CIFAR-100, SVHN in this section. We first report the accuracy on CIFAR-10 in Table 1 with the attack iterations T = 20, 40, 100 for PGD (Madry et al., 2017) and CW (Carlini & Wagner, 2017). We also conduct more experiments to further evaluate the robustness of our proposed method against more recent attacks, e.g. AutoAttack (Croce & Hein, 2020) and RayS (Chen & Gu, 2020)) as shown in Appendix B.2. As observed, overall, our proposed method achieves a clear superiority over all the defence approaches on both the clean data and adversarial examples (except that it is slightly inferior to Feature Scattering in FGSM). We also observe one exception that the standard model performs the best on clean data. Our approach performs much better than the other baseline models on PGD and CW attack. Particularly, we improve the performance of the recent state-of-the-art method Feature Scattering almost 3.1% and 5.2% under PGD20 and CW20 attack respectively. With the implementation of Inference with Manifold Transformation (IMT), our approach (ATLD-IMT) is 8.9% and 17.4% higher than the Feature Scattering under PGD20 and CW20 attack respectively. However, the performance on clean data is declined from 93.3% to 86.4% since IMT appears to have a negative effect for classifying clean data. In order to reduce the impact of IMT on the natural data, a threshold is used to limit the perturbation of IMT based on the output of discriminator. The perturbation is halved if the output of discriminator is within the range of [0.3, 0.7] (ATLD-IMT+). Under such setting, our approach could achieve high performance on adversarial attacks without sacrificing its accuracy on clean data.
Similarly, the accuracy on CIFAR-100 and SVHN are shown in Table 2 with the attack iterations T = 20, 100 for both PGD and CW for conciseness. Although our method is slightly weaker than Feature Scattering under FGSM attack on CIFAR-100, overall, our proposed method ATLD achieves state-of-the-art performance over all the other approaches under various adversarial attacks. Furthermore, our ATLD-IMT version exceeds Feature Scattering by almost 19.2% and 23.8% against the attack of CW100 on CIFAR-100 and SVHN respectively. More details about the defense of whitebox attacks under different attack budgets can be seen in Appendix.

heading: 5.2 DEFENDING BLACK-BOX ATTACKS
text: To further verify the robustness of ATLD, we conduct transfer-based black-box attack experiments on CIFAR-10. More black-box attack results on CIFAR-100 and SVHN are listed in Appendix.
Four different models are used for generating test time attacks including the Vanilla Training model, the Adversarial Training with PGD model, the Feature Scattering Training model and our model. As demonstrated by the results in Table 3, our proposed approach can achieve competitive performance almost in all the cases. Specifically, ATLD outperforms Feature Scattering significantly in 8 cases while it demonstrates comparable or slightly worse accuracy in the other 3 cases.
It deserves our attention that ATLD-IMT appears to have a negative impact on the black-box attacks though it stills performs much better than PGD. This may be explained in several aspects. On one hand, the distributions of adversarial examples produced by different models may differ significantly in the latent space; on the other hand, our discriminator lacks the ability to deal with the unseen distributions since the discriminator only distinguishes one type of adversarial examples from the natural data during training. We will leave the investigation of this topic as future work.

heading: 6 CONCLUSION
text: We have developed a novel adversarial training method which leverages both the local and global information to defend adversarial attacks in this paper. In contrast, existing adversarial training methods mainly generate adversarial perturbations in a local and supervised fashion, which could however limit the model’s generalization. We have established our novel framework via an adversarial game between a discriminator and a classifier: the discriminator is learned to differentiate globally the latent distributions of the natural data and the perturbed counterpart, while the classifier is trained to recognize accurately the perturbed examples as well as enforcing the invariance between the two latent distributions. Extensive empirical evaluations have shown the effectiveness of our proposed model when compared with the recent state-of-the-art in defending adversarial attacks in both the white-box and black-box settings.

heading: APPENDIX

heading: A LIST OF MAJOR NOTATION
text: For clarity, we list the major notations that are used in our model.
• Xorg = {x : x ∼ Q0}: the set of clean data samples, where Q0 is its underlying distribution;
• Xp = {x′ : x′ ∈ B(x, ), ∀x ∼ Q0}: the set of perturbed samples, the element x′ ∈ Xp is in the -neighborhood of the clean example x ∼ Q0;
• fθ: the mapping function from input to the latent features of the last hidden layer (i.e., the layer before the softmax layer);
• Qθ: the underlying distribution of the latent feature fθ(x) for all x ∈ Xorg; • Pθ: the underlying distribution of the latent feature fθ(x′) for all x′ ∈ Xp;
• P: the feasible region of the latent distribution Pθ, which is defined as P , {P : fθ(x′) ∼ P subject to ∀x ∼ Q0, x′ ∈ B(x, )}.
• Xadv: the set of the worst perturbed samples or manifold adversarial examples, the element xadv ∈ Xadv are in the -neighborhood of clean example x ∼ Q0;
• P ∗θ : the worst latent distribution within the feasible region P which leads to the largest divergence or the underlying distribution of the latent feature fθ(xadv) for all xadv ∈ Xadv;

heading: B ADDITIONAL EXPERIMENT DETAILS

heading: B.1 MODEL ROBUSTNESS AGAINST PGD AND CW ATTACKER UNDER DIFFERENT ATTACK BUDGETS
text: We further evaluate the model robustness against PGD and CW attacks under different attack budgets with a fixed attack step of 20. These results are shown in Figure 3. It is observed that the performance of Adversarial Training with the PGD method (AT) drops quickly as the attack budget increases. The Feature Scattering method (FS) can improve the model robustness across a wide range of attack budgets. The proposed approach ADLT-IMT further boosts the performance over Feature Scattering by a large margin under different attack budgets especially under CW attack, except that our ADLTIMT is slightly inferior to Feature Scattering under PGD attack with budget = 20 on CIFAR-10.

heading: B.2 MODEL ROBUSTNESS AGAINST AUTOATTACK AND RAYS
text: As shown in (Croce & Hein, 2020; Chen & Gu, 2020), several models (such as Feature Scattering) could achieve high enough robustness against PGD and CW attack, but they may fail to defend more stronger attacks. To further evaluate the model robustness against stronger attacks, we evaluate the robustness of our proposed method IMT+ against AutoAttack (Croce & Hein, 2020) and RayS (Chen & Gu, 2020) attacks with L∞ budget = 8 on CIFAR-10 and CIFAR-100.
We first compare the accuracy of the proposed ATLD-IMT+ with several competitive methods on CIFAR-10 in Table 4 to defend the AutoAttack (AA) and Rays attacks, including: (1) Traditional Adversarial Training with PGD (AT) (Madry et al., 2017); (2) TRADES: trading adversarial robustness off against accuracy (Zhang et al., 2019); (3) Feature Scattering: generating adversarial examples with considering inter-relationship of samples (Zhang & Wang, 2019); (4) Robustoverfitting: improving models adversarial robustness by simply using early stop (Rice et al., 2020); (5) Pretraining: improving models adversarial robustness with pre-training (Hendrycks et al., 2019); (6)WAR: mitigating the perturbation stability deterioration on wider models (Wu et al., 2020); (7) RTS: achieving high robust accuracy with semisupervised learning procedure (self-training) (Carmon et al., 2019); (8) Gowal et al. (2020): achieving state-of-the-art results by combining larger models, Swish/SiLU activations and model weight averaging.
These comparison algorithms attain the most competitive performance on defending AA attack. As observed, overall, our proposed method achieves a clear superiority over all the defence approaches on both the clean data and adversarial examples (except on clean data, ours is slightly inferior to Gowal et al. (2020) which is however trained with additional data). Note that Pretraining, WAR and Gowal et al. (2020) with footnote require additional data for training (e.g. unlabeled data, pretraining).
We also report the accuracy of ATLD-IMT+ with the state-of-the-arts methods on CIFAR-100 in Table 5 against the AutoAttack (AA). Our proposed method again achieves on both the clean data and AA attacked examples significant better performance than all the other defense approaches (without data augmentation). Furthermore, it is noted that, while our ATLD-IMT+ method is just slightly inferior to Gowal et al. (2020) (which is trained with additional data), it is substantially ahead of the normal version of Gowal et al. (2020).

heading: B.3 BLACK-BOX RESULTS ON SVHN AND CIFAR-100
text: We conduct more evaluations on the transfer-based black-box attacks on SVHN and CIFAR-100. We report the results in Table 6. It can be observed that our proposed method overall outperforms Feature Scattering in most of the cases on SVHN. Surprisingly, the Adversarial Training method, i.e. the PGD, performs better than our method and Feature Scattering method in three cases. This also partially reveals the more challenging nature of defending black-box attacks than white-box attacks.
On CIFAR-100, it can be observed that our method and Feature Scattering are comparable. The performance of these two methods differs little though our method outperforms Feature Scattering significantly under PGD20 and CW20 against adversarial attacks generated from the Feature Scattering model.
Overall, though the proposed ATLD method may not lead to remarkably higher performance than the current state-of-the-art algorithms in defending black-box attacks (as we observed in the case of white-box attacks), it still generates overall better or comparable performance. We will again leave the further exploration of defending black-box attacks as our future work.
B.4 ILLUSTRATION OF THE OVERLAID BOUNDARY CHANGE OF DIFFERENT METHODS
We conduct a toy example in Figure 4 to illustrate the effect on how the various methods would affect the decision boundary after the adversarial training is applied. In Figure 4, (a) shows the decision boundary trained with clean data; (b) shows the decision boundary adversarially trained with the perturbed samples by PGD; (c) presents the decision boundary given by the adversarial training of Feature Scattering; and (d) illustrates the decision boundary trained from our proposed ATLD. Clearly, both the PGD (Figure 4(b)) and the FS (Figure 4(c)) vary the original decision boundary significantly. Moreover, it can be observed that the adversarial training with PGD corrupts the data manifold completely. On the other hand, FS appears able to retain partially the data manifold information since it considers the inter-sample relationship locally. Nonetheless, its decision boundary appears non-smooth, which may hence degrade the performance. In contrast, as shown in Figure 4(d), our proposed method considers to retain the data manifold globally, which varies the
decision boundary slightly. This may explain why our proposed ATLD method could outperform the other approaches.

heading: B.5 FURTHER DETAILS OF ATLD-IMT
text: We elaborate the training procedure of our IMT in this section. The overall architecture of ATLDIMT is plotted in Figure 5. A test sample x is fed into the classifier, and the discriminator outputs the prediction. A special perturbation in IMT is then computed from the loss DW and added back to x; in this way, the sample would be pushed towards the manifold of natural samples, which is supposed to be further away from the decision boundary. The prediction of the transformed xt by the adversarially-trained classifier will then be output as the label of x.
To illustrate clearly the effect of our ATLD-IMT, we conduct additional toy experiments as shown in Figure 6 where we respectively plot the clean or natural data, perturbed data attacked by PGD, and adjusted data by ATLD-IMT in (a), (b), and (c). Moreover, the decision boundary is given by ATLD
in all the three sub-figures. In (a), it deserves our attention that the boundary learned by ATLD could classify natural data well compared to the PGD and Feature Scattering as shown in Section A.3. As observed in (b), the perturbations generated by PGD will push the natural samples toward or even cross the decision boundary. Our proposed IMT can push the samples towards the manifold of natural examples as observed in (c). Since the manifold of natural examples would be more separable, this may further increase the classification performance as observed in the experiments.
Under review as a conference paper at ICLR 2021
B.6 ILLUSTRATION OF VECTOR FIELD OF DIFFERENT PERTURBATION SCHEMES
test
test
test

heading: C DETAILED DERIVATION
text: In this section, we provide the details about the derivation for the main objective function (6) and elaborate how to compute the adversarial examples and the transformed examples.

heading: C.1 DERIVATION FOR MAIN OBJECTIVE FUNCTION (6)
text: We start with minimizing the largest f -divergence between latent distributions Pθ and Qθ induced by perturbed example x′ and natural example x. And we denote their corresponding probability density functions as p(z) and q(z). According to Eq. (3), we have
min θ max Qθ Df (Pθ||Qθ) = min θ max q(z) ∫ Z q(z) sup t∈domf∗ {tp(z) q(z) − f∗(t)}dx
≥ min θ max q(z) sup T∈τ ( ∫ Z p(z)T (z)dz − ∫ Z q(z)f∗(T (z))dz)
= min θ max Qθ sup W
{ Ez∼Pθ [gf (VW (z))] + Ez∼Qθ [−f∗(gf (VW (z)))] } = min
θ sup W
{ Ex∼D { max
x′∈B(x, ) [gf (VW (fθ(x
′)))] + [−f∗(gf (VW (fθ(x))))] }}
(9)
To compute the Jensen-Shannon divergence between Pθ andQθ, we set gf (t) = − log(1+e−t) and f∗(g) = − log(2− eg). Then, we have
min θ max Qθ DJS(Pθ||Qθ) ≥ min θ sup W
{ Ex∼D { max
x′∈B(x, ) [logDW (fθ(x
′)))] + [1− logDW (fθ(x))))] }} (10)
where DW (x) = 1/(1 + e−VW (x)). (10) is equivalent to optimize the lower bound of JensenShannon divergence between Pθ and Qθ. With disentangling the computation of adversarial examples from Eq. (10) and further considering the classification loss for the classifier Lf and the
discriminator L1:Cd , we can obtain the final objective:
min θ { sup W N∑ i=1 [logD0W (fθ(x adv i )) + (1− logD0W (fθ(xi))︸ ︷︷ ︸
L0d
]
+L(xadvi , yi; θ)︸ ︷︷ ︸ Lf + min W [l(D1:CW (fθ(xi)), yi) + l(D 1:C W (fθ(x adv i )), yi)]︸ ︷︷ ︸
L1:Cd
} ,
s.t. xadvi = arg max x′i∈B(xi, ) [logD0W (fθ(x ′ i)) + (1− logD0W (fθ(xi))︸ ︷︷ ︸
L0d
]
(11)

heading: C.2 COMPUTATION FOR ADVERSARIAL EXAMPLE AND TRANSFORMED EXAMPLE
text: To compute the adversarial example, we need to solve the following problem:
xadvi = arg max x′i∈B(xi, ) [logD0W (fθ(x ′ i)) + (1− logD0W (fθ(xi))︸ ︷︷ ︸
L0d
] (12)
It can be reformulated as computing the adversarial perturbation as follows:
radvi = arg max‖r‖∞≤ [L0d(xi + ri, θ)] (13)
We first consider the more general case ‖r‖p ≤ and expand (13) with the first order Taylor expansion as follows:
radvi = arg max‖r‖p≤ [L0d(xi, θ)] +∇xFT ri (14)
where F = L(xi, θ). The problem (14) can be reduced to:
max ‖ri‖p=
∇xFT ri (15)
We solve it with the Lagrangian multiplier method and we have
∇xFri = λ(‖ri‖p − ) (16)
Then we make the first derivative with respect to ri:
∇xF = λ rp−1i p( ∑ j(r j i ) p)1− 1 p
(17)
∇xF = λ p ( ri )p−1
(∇xF) p p−1 = (
λ p ) p p−1 ( ri )p (18)
If we sum over two sides, we have∑ (∇xF) p p−1 = ∑ ( λ
p )
p p−1 (
ri
)p (19)
‖∇xF‖p ∗ p∗ = ( λ p )p ∗ ∗ 1 (20)
where p∗ is the dual of p, i.e. 1p + 1 p∗ = 1. We have
( λ
p ) = ‖∇xF‖p∗ (21)
By combining (18) and (21), we have
r∗i = sgn(∇xF)( |∇xF| ‖∇xF‖p∗ ) 1 p−1
= sgn(∇xL0d)( |∇xL0d| ‖∇xL0d‖p∗ ) 1 p−1
(22)
In this paper, we set p to∞. Then we have
r∗i = lim p→∞ sgn(∇xL0d)( |∇xL0d| ‖∇xL0d‖p∗ ) 1 p−1
= sgn(∇xL0d)( |∇xL0d| ‖∇xL0d‖1 )0 = sgn(∇xL0d)
(23)
Then we can obtain the adversarial example:
x∗i = xi + sgn(∇xL0d) (24)
To compute the transformed example, we need to solve the following problem:
r∗ = arg min ‖r‖∞≤ logD0W (fθ(x+ r)). (25)
With the similar method, we can easily get the transformed example xt
xt = x− sgn(∇x logD0W ). (26)

Review summaries:
This paper presents a framework for adversarial robustness via incorporating local and global structure of the data manifold. Specifically, the key motivation is that standard adversarial methods typically use only sample specific perturbations for generating the adversarial examples, and thus using them for robustness of the learning model is limited. Instead the paper proposes to capture the global data manifold as well in the robustifying framework. To this end, an objective is presented (4,5) that uses latent data distributions, with the goal that the adversarial perturbations should maximize the f-divergence against the latent distribution of the clean samples. Experiments are provided on several datasets and demonstrate significant performance improvements.

The paper analyzes the property of local and global data manifold for adversarial training. In particular, they used a discriminator-classifier model, where the discriminator tries to differentiate between the natural and adversarial space, and the classifier aims to classify between them while maintaining the constraints between local and global distributions. The authors implemented the proposed method on several datasets and achieved good performance. They also compared with several whitebox and blackbox methods and proved superiority.