Improving Model Robustness with Latent Distribution Locally and Globally
Authors: Zhuang QIAN
Shufei Zhang
Kaizhu Huang
Qiufeng Wang
Rui Zhang
Xinping Yi
Venue:  (2021)
Keywords: adversarial example
robustness
data manifold
adversarial training

Abstract: We propose a novel adversarial training method which leverages both the local and global information to defend  adversarial attacks. Existing adversarial training methods usually generate adversarial perturbations  locally in a supervised manner and fail to consider the data manifold information in a global way. Consequently, the resulting  adversarial examples may corrupt the underlying data structure and are typically biased towards the decision boundary. In this work, we exploit both the local and global information of data manifold to generate adversarial examples in an unsupervised manner. Specifically, we design our novel framework  via an adversarial game between  a discriminator and a classifier: the discriminator is  learned to differentiate the latent distributions  of the natural data and the perturbed counterpart, while the classifier is trained to recognize accurately the perturbed examples as well as enforcing the invariance between the two latent distributions. We conduct a series of analysis on the model robustness and also verify the effectiveness of our proposed method empirically. Experimental results show that our method  substantially outperforms the recent state-of-the-art (i.e. Feature Scattering) in defending adversarial attacks  by a large accuracy margin  (e.g. $17.0\%$ and $18.1\%$ on SVHN dataset, $9.3\%$ and $17.4\%$ on CIFAR-10 dataset, $6.0\%$ and $16.2\%$ on CIFAR-100 dataset for defending PGD20 and CW20 attacks respectively).

Under review as a conference paper at ICLR 2021
IMPROVING MODEL ROBUSTNESS WITH LATENT DIS- TRIBUTION LOCALLY AND GLOBALLY
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a novel adversarial training method which leverages both the local and global information defend adversarial attacks. Existing adversarial training methods usually generate ad adversarial perturbations locally in a supervised manner and fail to consider the data manifold information in a global way. Consequently, the resulting adversarial examples may corrupt the underlying data structure and are typically biased towards the decision boundary. In this work, we exploit both the local and global information of data manifold to generate adversarial exam- ples in an unsupervised manner. Specifically, we design our novel framework via adversarial game between a discriminator and classifier: the discriminator is learned to differentiate the latent distributions of the natural data and the perturbed counterpart, while the classifier is trained to recognize accurately the perturbed examples as well as enforcing the invariance between the two latent distributions. We conduct a series of analysis on the model robustness and also verify the effec- tiveness of our proposed method empirically. Experimental results show that our method substantially outperforms the recent ofate-of-the-ar (i.e. Feature Scatter- ing) in defending adversarial attacks by a large accuracy margin (e.g. 17.0% and 18.1% 1% on SVHN dataset, 9.3% and 17.4% on CIFAR- 10 dataset, 6.0% and 16.2% on CIFAR-100 dataset for defending PGD20 and CW20 attacks respectively).
INTRODUCTION
Deep Neural Networks (DNNs) have achieved impressive performance broad range of datasets,
yet can be easily fooled by adversarial examples or perturbations (LeCCnn et al. 2015 He et al.) 2016 Gers et al. 1999). Adversarial examples have been shown to be ubiquitous beyond different tasks such as image classification (Goodfellow et al. 2014), segmentation (Fischer et al. 2017). , and speech recognition (Carlini & Wagner, 2018). Overall, adversarial examples raise great concerns about the robustness of learning models, and have drawn enormous attention over recent years.
To defend adversarial examples, great efforts have been made improve the model robustness (Kan- nan et al. 2018) You et al. 2019: Wang & Zhang 2019 /Zhang & Wang 2019). Most of them are based on the adversarial training. than clean data (Goodfellow et al.) 2014) Madry et al., 2017] Lyu et al., 2015). In principle, adver- sarial training is a min-max game between the adversarial perturbations and classifier. Namely, the indistinguishable adversarial perturbations are designed to mislead the output of the classifier, while the classifier trained produce the accurate predictions for these perturbed input data. Currently, the adversarial perturbations are mainly computed by enforcing the output invariance in supervised manner (Madry al., (017). Despite its effectiveness some scenarios, is observed recently that these approaches may still be limited in defending adversarial examples.
training the model with adversarially aallly---eee perturbed samples rather
In particular, we argue that these current adversarial training approaches are typically conducted in a local and supervised way and fail to consider globally the overall data manifold information; such information however proves crucially important for attaining better generalization. As a result, the generated adversarial examples may corrupt the underlying data structure and would be typically biased towards the decision boundary. Therefore, the well-gene rell--eneralizing features inherent to the data distribution might be lost, which limits the performance of the DNNs to defend adversarial examples even if adversarial training is applied (Iyys et al., 2019a Schhmidt et al. 2018). For illustration, we have shown toy example in Figure]] [ As clearly observed, adversarially perturbed examples gen-

---

Under review as a conference paper at ICLR 2021
(a) Original Data
(b) PGD
(c) Feature Scattering
(d) Ours
Illustrative example of different perturbation schemes. (a) Original data, Perturbed data using (b) PGD: a supervised adversarial generation method (c) Feature Scattering, and (d) the pro- posed ATLD method. The overlaid boundary is from the model trained on clean data.
Figure 1:
erated by PGD, one of the most successful adversarial training method, corrupt the data manifold, which would inevitably lead to poor performance if the training is conducted based on these per- turbed examples. On the other hand, the current state-of-the-ee method Feature Scattering (Zhang & Wang, 2019) can partially alleviate this problem but still leads to corruptions on the data manifold.
To address this limitation, we propose a novel method called Adversarial Training with Latent Dis- tribution (ATLD) which additionally considers the data distribution globally in unsupervised fashion. In this way, the data manifold could be well preserved, which is beneficial to attain better model generalization. Moreover, since the label information is not required when computing the ad- versarial perturbations, the resulting adversarial examples would not be biased towards the decision boundary. This can be clearly observed in Figure dd).
Our method can be divided into two steps: first, we train the deep model with the adversarial exam- ples which maximize the variance between latent distributions of clean data and adversarial coun- terpart rather than maximizing the loss function. We reformulate it as minimax game between discriminator and classifier. The adversarial examples are crafted by the discriminator to make different implicitly the latent distributions of clean and perturbed data, while the classifier is trained to decrease the discrepancy between these two latent distributions as well as promoting accurate classification the adversarial examples as Figure 2 shows. Then, during the inference procedure, we generate the specific perturbations through the discriminator network to diminish the impact of the adversarial attack as shown in Figure 6 Appendix.
On the empirical front, with the toy examples, we show that our proposed method can preserve more information of the original distribution and learn better decision boundary than the existing adversarial training method. We also test our method on three different datasets: CIFAR-10, 10, CIFAR- 100 and SVHN with the famous PGD, CW and FGSM attacks. Our ATLD method outperforms the state-of-the-aat methods by a large margin. e.g. ATLD improves over Feature Scattering ÇZhang & Wang 2019) by 17.0% and 18.1% on SVHN for PGD20 and CW20 attacks. Our method also shows a large superiority to the conventional adversarial training method (Madry et al. (2017), boosting the performance by 32.0% and 30.7% on SVHN for PGD20 and CW20 attacks.
RELATED WORK
Adversarial Training. Adversarial training is family of techniques to improve the model robust- ness (Madry et al.) 2017 Lyu et al. 2015). It trains the DNNs with adversarially-perturi
samples instead of clean data. Some approaches extend the conventional adversarial training by injecting the adversarial noise to hidden layers to boost the robustness of latent space (Ilyas et al. 2019b] You al. 2019 /anturkar et al., 2019 Liu et al. 2019p. All of these approaches generate the adver- sarial examples by maximizing the loss function with the label information. However, the structure of the data distribution destroyed since the perturbed samples could be highly biased towards the non-optimal decision boundary (Zhang & Wang 2019). Our proposed method has similar train- ing scheme with adversarial training by replacing clean data with the perturbed one. Nevertheless, method generates the adversarial perturbations without the label information which weakens the impact non-optimal decision boundary and can retain more information of the underlying data distribution.
2

---

Under review as a conference paper at ICLR 2021
Manifold-based Adversarial Training. Song et al. (2017) propose generate the adversarial ex- amples by projecting on a proper manifold. Zhang & Wang (2019) leverage the manifold information in the forms of inter-sample relationship within the batch generate adversarial adversarial per- turbations. Virtual Adversarial Training and Manifold Adversarial Training are proposed improve model generalization and robustness against adversarial examples by ensuring the local smooth- ness of the data distribution (Zhang et al.) 20188] MMyato et al., 2017). Some methods are designed to enforce the local smoothness around the natural examples by penalizing the difference between the outputs of adversarial examples and clean counterparts (Kannan et al. 2018) Chan et al.) 2020) Jakubovitz Giryes 2018). All of these methods just leverage the local information of the distri- bution or manifold. Differently, our method generates the perturbations additionally considering the structure of distribution globally.
Unsupervised Domain Adversarial Training. Domain Adversarial Training shares training scheme similar to our method where the classifier and discriminator compete with each other (Odena et al. 2017 Long et al. 2018 Ganin et al. 2016). However, its objective is to reduce the gap be- tween the source and target distributions in the latent space. The discriminator used to measure the divergence between these two distributions in the latent space. The training scheme of our method is also based on competition between the classifier and discriminator. Different from the previous framework, the discriminator of method is used to capture the information of distributions of adversarial examples and clean counterparts in the latent space which helps generate the adversarial perturbations.
GAN-based Adversarial Training Methods. Several GAN-based methods leverage GANs to learn the clean data distribution and purify the adversarial examples by projecting them on clean data manifold before classification (Meng & Chen 2017 Metzen et (207)) The framework of GAN can also be used to generate the adversarial examples (Baluja Fischer 2018 The generator produces the adversarial examples to deceive both the discriminator and classifier; the discriminator and classifier attempt to differentiate the adversaries from clean data and produce the correct labels respectively. Some adversary detector networks are proposed to detect the adversarial examples which can be well aligned with our method (Gong et al. 2017 Grosse et al. 2017). In these works, pretrained network is augmented with binary detector network. The training of the pretrained network and detector involves generating adversarial examples to maximize their losses. Differently, method generates the adversaria examples just to minimize the loss of the discriminator and feed them as the training set to the classifier. Such adversarial examples are deemed to induce most different latent representations from the clean counterpart.
BACKGROUND
3.1
ADVERSARIAL TRAINING
Let us first introduce the widely-adopted adversarial training method for defending against adver- sarial attacks. Specificallyy solves the following minimax optimization problem through training.
min (x(.))) max L(z', yy 0)]}, x'ES.
(1)
where I E Ra and y E are respectively the clean data samples and the corresponding labels drawn from the dataset D, and L(-) is the loss function of the DNN with the model parameter 0 E TRM. Furthermore, we denote the clean data distribution as Qn. i.e. x ~ Qo- and denote o' R" as perturbed samples in a feasible region S₂ & (z : Z E B(x,e) [-100 1.0]") with B(2,c) é {2 : || - 211x0 b0 S <} being the lo at center x with radius e. By defining fe(-) as the mapping function from the input layer to the last latent layer, we can also rewrite the loss function of the DNN as ((fo(z)), y) where 1(.) denotes the loss function calculated from the last hidden layer of the DNN, e.g. the cross entropy loss as typically used in DNN.
Whilst the outer minimization can be conducted by training to find the optimal model parameters 0, the inner maximization essentially generates the strongest adversarial attacks given set of model parameters 8. In general, the solution to the minimax problem can be found by training a network minimizing the loss for worst-case adversarial examples, as to attain adversarial robustness. Given a set of model parameters 6, the commonly adopted solution to the inner maximization problem can lead to either one-step (e.g., FGSM) multi-step (e.g., PGD) approach (Madry al. 2017). In
3

---

Under review as a conference paper at ICLR 2021
particular, for given single point z, the strongest adversarial example x' the t-th iteration can be iteratively obtained by the following updating rule:
(2) is projection operator to project the inputs onto the region SN, sgn(-) is the sign function, and a is the updating step size. For the initialization, x can be generated by randomly sampling in e(r,e).
rt = Ils, (x' + a -a--ggh sgn(V y; 0)))
where [.)
appears in I that each perturbed sample x' is obtained individually by leveraging its loss function L(x', y; 8) with its label However, without considering the inter-relationship between samples, we may lose the global knowledge of the data manifold structure which proves highly useful for attaining better generalization. This issue has been studied in recent work (Zhang & Wang 2019) where a new method named feature scattering made a first step to consider the inter-sample rela- tionship within the batch; unfortunately this approach did not take the full advantages of the global knowledge of the entire data distribution. In addition, relying on the maximization of the loss func- tion, the adversarially perturbed eerrrrrrr data samples may be highly biased towards the decision boundary, which potentially corrupts the structure the original data distribution, especially when the decision boundary is non-optimal (see Figure T again for the illustration).
3.2 DIVERGENCE ESTIMATION
To measure the discrepancy of two distributions, statistical divergence measures (e.g.. Kullback- Leibler and Censen-Shannon divergence) have been proposed. general, given two distributions P and with continuous density function p(z) and ((x) respectively, f-divergence is defined as (TIIQ) - Jxx {(x)
plar) dr. The exact computation of f-divergence challenging, and q(x))
the estimation from samples has attracted much interest. For instance, leveraging the variational
methods, Nguyen et al. (2110) propose method for estimating f-divergence from only samples; Nowozin et al. (2016) extend this method by estimating the divergence with learning the parameters of discriminator. Specifically, the f-divergence between two distributions P and Q can be lower- bounded using Fenchel conjugate and Jensen's 's inequality (Nowozin et al. 2016).
/ ((x) sup
{t (()())
(Q())
ft(()
EEdomff
((x)
sup( p(x)T(x)dx= / g(z)) TEr x (o
(3)
W
where Vw : X + R is a discriminator network with parameter W and gʃ R + domf" is an output activation function which is determined by the type of discriminator.
an arbitrary class of
functions 7 X + IR. f is convex lower-semicontinous function and f is its conjugate defined by (1) supugdom. jut- - f(w)]. The objective of discriminator for GANs is a special case of (3)
with the activation function ş (t) = log(1 + e and f"(g)= = - log(2 2
e²) It approximates the
divergence between real and fake distributions. Arjovsky et al. 2017 also develop
method to estimate the maaseersteinddistane by neural network. In this paper, these methods will be
used estimate the Jensen-Shannon divergence between latent distributions induced by adversarial and clean examples.
4 ADVERSARIAL TRAINING WITH LATENT DISTRIBUTION
As discussed in Section 3.1 the conventional adversarial training methods rely on the knowledge of data labels. As result, the local information generate adversarial examples may be biased toward the decision boundary; such individual adversarial example generation does not capture the global knowledge of the data manifold.
To alleviate these limitations, we propose a novel method to compute the perturbed samples by leveraging the global knowledge of the whole data distribution and then disentangling them from the data labels and the loss function. Generally speaking, the perturbations are generated to enlarge the variance between latent distributions induced by clean and adversarial data.
4

---

Under review as a conference paper at ICLR 2021
Formally, we try to identify the set of adversarial examples Xadv that yield in the latent space distribution Pÿ through fo(-) that is the most different from the latent distribution Qo induced by the clean samples Xorg {x other words, the resulting adversarial examples can be deemed as manifold adversarial examples, which 'deceive' the manifold rather than fool the classifier defined in the traditional adversarial examples. It is noted that the latent space to be perturbed could be any hidden layer though it defined in the last hidden layer before the softmax of DNN in this paper. The optimization problem of the proposed adversarial training can then be reformulated as follows:
Qob. without resorting to the corresponding labels Y. In
min 6
(B yo; (1 v)) + (y(Pal1Qu)
(4)
s.t. Pj -aag max ( P&E
(5)
where 1(-) and y are similarly defined as before, and (-(-) is the f-divergence measure of two distributions. P P= foo the latent distribution Po which is induced by the set of perturbed examples Xp through fo(-). fo (xx) and So (x)dv) represents the latent features of the perturbed example x' and adversarial example aade respectively. Intuitively, we try to obtain the worst latent distribution Pÿ which is induced by Xodu through fe(-) within the region P, while the model parameter 0 is learned to minimize the classification loss on the latent feature fe(wd)y) ~ Pe (or equivalently adversarial example x zdd e Xadv) and the f-divergence between the latent distributions Pÿ and Qo induced by adversarial examples Xado and clean data Xorg-
P subject to Vz Qo. x' E B(x,c) c)} is the feasible region for
It is still challenging to solve the above optimization problem, since both the objective function and the constraint are entangled with the adversarial examples Xado and the model parameters 0. To make the problem more tractable, propose novel Adversarial Training with Latent Distribution (ATLD) method. In the next subsection, by taking into account the entire data distribution globally, first focus on the constraint and identify the adversarial samples Xade through the maximization problem. We then solve the minimization of the objective function with the adversarial training procedure. To further enhance the performance, we add specific perturbations named Inference with Manifold Transformation (IMT) in Section [42 to input samples for enforcing them towards the more separable natural data manifold. Finally, we classify the transformed data points with the adversarialll-trainee model.
4.1 GENERATING ADVERSARIAL EXAMPLES FOR TRAINING
First, we optimize the constraint to generate the adversarial examples or its induced distribution Pj for training. Intuitively, the adversarial examples Xadv are crafted to maximize the divergence between the latent distributions induced by natural examples Xorg and adversarial counterpart Xade in an unsupervised fashion since no knowledge of labels Y required. Together with the objective function in (4) our proposed adversarial training method is to minimize such divergence as well as the classification error for adversarial examples X Xadv-
However, it is a challenging task to evaluate the divergence between two latent distributions. To make more tractable, we leverage a discriminator network for estimating the Sensen-Shannoe divergence between two distributions P7 Pa and Qa according to Section 3.2 It is noted again that the class label information is not used for generating adversarial examples. Hence the adversarial
examples are still generated in an unsupervised way. Then, by using (3). the optimization problem in and (5) can be approximated as follows in a tractable way.
min Σ
L( yb:
+ sup W
Dog Dw (fo(a)d))) + (1 log Dw (fo
La
(6)
s.t.
2 adv
arg
max (
log Dw (w(((())) + (1 log Dw (w((a()))))
La
where N denotes the number of training samples and Dw denotes the discriminator network with the sigmoid activation function and parameter W. fo (2.) is the latent feature of the clean sample Tz. Dw is used to determine whether the latent feature is from adversary manifold (output the
5

---

Under review as a conference paper at ICLR 2021
manifold label of the latent feature). For ease of description, we represent the components in Eq. @ as two parts: Ly and La. La is the manifold loss and L, represents the loss function of the classifier network.
We now interpret the above optimization problem. By comparing Eq. 6 and Eq.
it is observed
that the Jensen-Shannon divergence between P/ and Qa is approximated by supw == La and the minimization of the classification loss on adversarial examples is given by ming >i Lg. The prob- lem (6) is optimized by updating parameters 0 and W and crafting adversarial examples {xzde) ) } iteratively. The whole training procedure can be viewed as the game among three players: the clas- sifier, discriminator, and adversarial examples. The discriminator Dw learned to differentiate the latent distributions of the perturbed examples and clean data via maximizing the loss La while the classifier fo is trained to (1) enforce the invariance between these two distributions to confuse the discriminator Dw by minimizing the loss La. and (2) classify the adversarial examples as accurately as possible by minimizing Lg. For each training iteration, the adversarial examples are crafted to make different the adversarial latent distribution and natural one by maximizing La. Although Dw cannot measure the divergence between the two latent distributions exactly at the first several train- ing steps, it can help evaluate the divergence between distributions induced by perturbed examples and clean ones when the parameters W converges.
However, when the latent distributions multi-class classification, it is challenging for the discriminator to measure the divergence between such distributions. Several work reveals that there a high risk of failure in using the discriminator networks to measure only a fraction of components underlying different distributions (Arjovsky) & Bottou 2017 Che al.] 2016). Ma (2018) also shows that two different distributions are not guaranteed to be identical even if the discriminator is fully confused. To alleviate such the problem, we additionally train the discriminator Dw to predict the class labels for latent features as (Odena
multi-modal, which is a real scenario due to the nature of
et al. 2017 Long et al.) 2018). As result, the problem (6) can then be reformulated as:
min Σe
L Langdv Ms: 0):0)
+ sup W
llo Diy (fo (f(()dd)) + (1 log Di (fo(z:)))
47
+ min
(min|((Dip
((Tt (Jo(z:)), .b) + I(D (Jo(add))) D1
W
s.t. 2 arg
max ((B()
log DW (fa(z))) + (1 - log DV (fo(z:))
L7
Here Df. is the first dimension of the output of the discriminator, which indicates the manifold label of the latent features; DIC are the remaining C dimensions of the output of Dw, used output the class label of the latent feature; C denotes the number of classes, and L. and L the manifold loss and the classification loss for the discriminator network respectively. (The detailed derivation for Eq. T and Eq. 77) can be seen in Appendix.) The detailed training procedure of our framework is depicted in Figure
Remarks. It is worth noting that the labeled information is not required for generating adversarial examples. Therefore, our method prevents the perturbed examples from highly biasing towards the decision boundary and more information of the original distribution structure is preserved. In addi- tion, since the discriminator is trained with the whole dataset (both clean and adversarial examples), it captures the global information of data manifold. Consequently, by training with adversarial ex- amples generated according to the manifold loss of the discriminator, our method can improve the model robustness against adversarial examples with the global structure of data distribution.
4.2 INFERENCE WITH MANIFOLD TRANSFORMATTION
To enhance the generalization of ATLD, we further develop new inference method with manifold transformation. Although adversarially-train trained models can well recognize adversarial examples, there are still potential examples which are easily misclassified especially for unseen data. In other words, the generalization to adversarial examples is hard to achieve due to the more complex distri- bution of adversarial examples (Schmidt et al. 2018 Zhai al. 2019). To alleviate this problem,

---

Under review as a conference paper at ICLR 2021
classificr fo
aL 30
loss Ly
natural example xt
class label
forwardprop
backprop
latent feature
0L aL aw + aw
loss Ly +L
adversarial
aL4 8
class label
example undo
manifuld label
UL 0
aade esgml
loss La
Ox
discriminstor Dw
Figure 2: Overall architecture of ATLD and its training procedure. 1) The natural example is fed into the network, and the discriminator outputs its prediction. The manifold loss La computed with the prediction and true label and generates the adversarial example zade (blue arrow). 2) Both the clean and adversarial sample are fed into the network to train the classifier (green arrow) and the discriminator (yellow arrow) iteratively.
our proposed inference method first pushes adversarial examples towards the manifold of natural examples which is simpler and further away from the decision boundary than the adversarial distri- bution. Then the more separable adjusted examples are classified by the pursarially-trainne model. Specifically, the input sample fed into our adversarially-traiine puts the probability of such sample lying on the adversarial manifold. If this probability is higher than certain threshold, we compute the transformed example x' by adding the specific perturbation r" to the input sample to reduce such probability. This perturbation can be computed as:
model and the discriminator out-
arg min log Dl (fo(2 + r)).
(8)
|erllse
Intuitively, the reduction of probability of this data point lying on adversarial manifold indicates that this point moves towards the benign example manifold after adding perturbation r". In other words, becomes more separable since the benign example manifold further away from the decision boundary. When the probability of the image lying on adversary manifold is lower than threshold, we still add such a perturbation to input image to make it more separable but with smaller magnitude. In the experiment part, we show this perturbation can move the adversarial examples away from the decision boundary. The whole inference procedure can seen Figure Appendix.
EXPERIMENTS
We conduct experiments the widely-used datasets, c.g., CIFAR-I0, 10, SVHN, and CIFAR- 100. Following the Feature Scattering method (Zhang Wang 2019), we leverage the wideres- basic classifier and discriminator model structure. Dur-
net (Zagoruyko & Komodakis 2016) ing the training phase, the initial learning rate is empirically set to 0.1 for all three datasets. We train our model 400 epochs with the transition epoch 60, 90 and the decay rate 0.1. The input perturbation budget is set to e = 8 with the label smoothing rate as 0.5. We use Los perturbation in this paper including all the training and evaluation.
We evaluate the various models on white-box attacks and black-box attacks. Under the white-box attacks, we compare the accuracy of the proposed method with several competitive methods, includ- ing: (1) the original wideresnet (Standard) trained with natural examples; (2) Traditional Adversar- ial Training with PGD (AT) (Madry al. 2017); (3) Triplet Loss Adversarial training (TLA) (Mao) et al. )019): : (4) Layer-wise Adversarial Training (LAT): injecting adversarial perturbation into the latent space (Sinha et al. 2019); (5) Bilateral: adversarial perturb on examples and labels both (Wang & Zhang 2019); (6) Feature-scattering: generating adversarial examples with considering inter- relationship of samples (Zhang & Wang 2019). These comparison algorithms present the most competitive performance defending adversarial attack. Under the black-box attacks, com- pare four different algorithms used to generate the test time attacks: Vanilla training with natural examples, adversarial training with PGD, Feature Scattering, and our proposed model.

---

Under review as a conference paper at ICLR 2021
5.1 DEFENDING WHITE-BOS ATTACKS
We show the classification accuracy under several white-box attacks on CIFAR-10, 10, CIFAR- 100, SVHN in this section. We first report the accuracy on CIFAR-10 Table with the attack itera- tions 7 20, 40, 100 for PGD (Madry et al.] 2017) and CW (Carlini & Wagner 2017). We also conduct more experiments to further evaluate the robustness of our proposed method against more recent attacks, e.g. AutoAttack (Croce & Hein 2020) and RayS (Chen & Gu 2020 shown in Appendix B.2 As observed, overall, our proposed method achieves a clear superiority over all the defence approaches
both the clean data and adversarial examples (except that it is slightly inferior to Feature Scattering in FGSM). We also observe one exception that the standard model per- forms the best on clean data. Our approach performs much better than the other baseline models PGD and CW attack. Particularly, we improve the performance of the recent state-of-the-art method Feature Scattering almost 3. % and 5.2% under PGD20 and CW20 attack respectively. With the implementation of Inference with Manifold Transformation (IMT), our approach (ATLD-IMT) is 8.9% and 17.4% higher than the Feature Scattering under PGD20 and CW20 attack respectively. However, the performance on clean data is declined from 93.3% to 86.4% since IMT appears to have a negative effect for classifying clean data. In order to reduce the impact of IMT on the natural data, threshold is used to limit the perturbation of IMT based on the output of discriminator. The perturbation is halved the output discriminator is within the range of [0.3, 0.7] (ATLD-IMT+)) Under such setting, our approach could achieve high performance adversarial attacks without sacrificing its accuracy on clean data.
Similarly, the accuracy on CIFAR-100 and SVHN are shown in Table 2 with the attack iterations = 20, 100 for both PGD and CW for conciseness. Although our method is slightly weaker than Feature Scattering under FGSM attack CIFAR-100, overall, our proposed method ATLD achieves state-of-the-ae performance over all the other approaches under various adversarial attacks. Further- more, our ATLD-IMA IMT version exceeds Feature Scattering by almost 19. 19.2% and 23.8% against the attack of CW 100 on CIFAR-100 and SVHN respectively. More details about the defense of white- box attacks under different attack budgets can be seen in Appendix.
Table Accuracy under different White-box Attack attack on CIFAR-10
ACCURACY UNDER WHITE-BOX AFTACK (E
MODELS
CLEAN
FGSM
PGD20
PGD40
PGD100
CW20
CW40
CW100
STANDARD
95.60
36.90
0.00
0.00
0.00
0.00
0.00
0.00
85.70
54,90
44,90
44.80
44,80
45.70
45.60
45,40
TLA
86.21
58.88
$1.59
LAT
87.80
53.04
BII ATERAL
91.20
70.70
57.50
55.20
56.20
$3.80
ES
90.00
78.40
70.50
70.30
68.60
62.40
62.10
60.60
ATLD
93.34
87.91
73.38
72.82
66.26
ATLD-IMT
86.42
84.62
79.48
77.06
74.20
79.81
77.14
75.20
ATLD IMT+
77.17
Table 2: Accuracy under different White-box Attack attack on CIFAR-100 and SVHN
MODELS
CIFAR- 100te- $)
SVHING S)
CLEAN
FGSM
PGD20
PGD100
CW20
CW100
CLEAN
FGSM
PGD20
PGD100
CW20
CW100
STANDARD
79.00
10.00
0.00
0.00
47.20
$3.00)
0.30
0.10
AT
59.90
25.50
22.60
22.30
23,20
23.00
93.90
68.10
47.90
16.00
48,70
47,30
LAT
60.94
27.05
26.41
60.94
27.03
26.41
48,
73,90
61.00
17.20
16.20
34.60
30.60
96.20
83,50
62,90
52.00
61,30
50.80
ATED
74.50
57.38
45.45
48.20
40.32
40.36
96.90
93,30
08.42
35.46
66.32
$2.62
ATLD-IMT
63.17
56.64
$3.26
$3.35
50.80
49.77
85.85
89.07
79.93
74.70
79.44
74.59
ATLD-IMT
46.92
46.15
90.34
78.33
77.52
71.04
5.2 DEFENDING BLACK-BOX ATTACKS
To further verify the robustness ATLD, we conduct transfer-based black-box attack experiments CIFAR-10. More black-box attack results CIFAR-100 and SVHN are listed in Appendix.
8

---

Under review as a conference paper at ICLR 2021
Four different models are used for generating test time attacks including the Vanilla Training model, the Adversarial Training with PGD model, the Feature Scattering Training model and our model. As demonstrated by the results in Table our proposed approach can achieve competitive performance almost in all the cases. Specifically, ATLD outperforms Feature Scattering significantly in 8 cases while it demonstrates comparable or slightly worse accuracy in the other cases.
deserves our attention that ATLD-IMT appears to have a negative impact on the black-box attacks though it stills performs much better than PGD. This may be explained in several aspects. On one hand, the distributions of adversarial examples produced by different models may differ significantly in the latent space; on the other hand, our discriminator lacks the ability to deal with the unseen distributions since the discriminator only distinguishes one type of adversarial examples from the natural data during training. We will leave the investigation of this topic as future work.
Table Accuracy under black-box attack on CIFAR-10
DEFENSE MODELS
AFTACKED MODELS
VANILLA TRAINING
ADVERSARIAL TRAINING
PRATURE SCATTERING
OURS
CW20
CW20
AT
84.62
84.89
84.83
72.20
63.77
63.27
82.26
80.36
79.31
82.56
81.34
79.97
FS
88.64
89.25
89.31
77.18
66.59
66.40
82.80
81.01
78:34
82.99
81.56
79.92
91.32
54.47
ATLD-IMT
83.10
87.17
86
73.66
61.59
61.62
84.37
81.18
78.18
83.68
81.05
77.95
ATLD-IMT+
86.53
89.93
89.91
79.16
65.61
65.07
85.76
83.75
81.40
83.90
81.98
79.77
CONCLUSION
We have developed novel adversarial training method which leverages both the local and global information to defend adversarial attacks in this paper. In contrast, existing adversarial training methods mainly generate adversarial perturbations
local and supervised fashion, which could however limit the model's generalization. We have established our novel framework via an adver- sarial game between a discriminator and a classifier: the discriminator is learned to differentiate globally the latent distributions of the natural data and the perturbed counterpart, while the classifier is trained to recognize accurately the perturbed examples as well as enforcing the invariance be- tween the two latent distributions. Extensive empirical evaluations have shown the effectiveness of our proposed model when compared with the recent state-of-the-art in defending adversarial attacks in both the white-box and black-box settings.
REFERENCES
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In Stat, volume 1050, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. In arXiv preprint arXin:1701.07875, 2017.
Shumeet Baluja and lan Fischer. Learning to attack: Adversarial transformation networks. Thirty- second aaai conference artificial intelligence, 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 20/7
IEEE Symposium Security and Privacy (SP), pp. 39-57. IEFF, 2017.
Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks on speech-to- text. In 2018 IEEE Security and Privacy Workshops (SPW), pp. 1-7. IEEE, 2018.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems, pp- 11192-11203, 2019.
Alvin Chan, Yi Tay, Yew-Soon Ong, and Jie Fu. Jacobian adversarially regularized networks for robustness. In International Conference Learning Representation, 2020.

---

Under review as a conference paper at ICLR 2021
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks. 2016.
Jinghui Chen and Quanquan Gu. Rays: A ray searching method for hard-label adversarial attack. In Proceedings of the 26rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2020.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free free attacks. In ICML, 2020.
Volker Fischer, Mummadi Chaithanya Kumar, Jan Hendrik Metzen, and Thomas Brox. Adversarial examples for semantic image segmentation. In arXiv preprint arXiv:/703 001,00101, 2017.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. In The Journal of Machine Learning Research, volume 17, pp. 2096-2030. JMLR. org. 2016.
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with Istm. IET, 1999.
Zhitao Gong, Wenlu Wang, and Wei-Shinn Ku. Adversarial and clean data are not twins. In arXiv preprint 1rXiv:1704. 04966, 2017.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In 1rXiv:1412.6552,, 2014.
Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03593, 2020.
Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick McDaniel. On the (statistical) detection adversarial examples. In arXiv preprint arXiv: 1702006600 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. Proceedings of the International Conference on Machine Learning, 2019.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pp. 125 136, 2019a.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are bugs, they are features. In Advances in Neural Information Processing Systems, pp. 125-136, 2019b.
Daniel Jakubovitz and Raja Giryes. Improving dnn robustness to adversarial attacks using jacobian regularization. In European Conference on Computer Vision, pp. 514-529, 2018.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. arXiv:1803.06373. 2018
Adversarial logit pairing. In
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. In nature, volume 521, pp. 436. Nature Publishing Group, 2015.
Aishan Liu, Xianglong Liu, Chongzhi Zhang, Hang Yu, Qiang Liu, and Junfeng He. Training robust deep neural networks via adversarial noise propagation. In arXiv:1909.0900 090099999999..... 2019.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650, 2018.
10

---

Under review as a conference paper at ICLR 2021
Chunchuan Lyu, Kaizhu Huang, and Hai-Ning Liang. A unified gradient regularization family for adversarial examples. 2015 IEEE International Conference on Data Mining, pp. 301-309. IEEE, 2015.
Tengyu Ma. Generalization and equilibrium in generative adversarial nets (gans) (invited talk). In the 50th Annual ACM SIGACT Symposium, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In
2017.
Chengzhi Mao, Ziyuan Zhong, Junfeng Yang, Carl Vondrick, and Baishakhi Ray. Metric learning for adversarial robustness. In Advances in Neural Information Processing Systems, pp. 478-489,
2019.
Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. In Proceedings of the 2017 ACM SIGSAC conference computer and communications security, pp. 135-147, 147, 2017.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. In arXiv preprint arXiv:1702.04267, 2017.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. In arXiv:1704.00 00777.03777, 2017.
XuanLong Nguyen, Martin Wainwright, and Michael I Jordan. Estimating divergence function- als and the likelihood ratio by convex risk minimization. In IEEE Transactions Information Theory, volume 56, pp. 5847-5861. IEEE, 2010.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: using variational divergence minimization. In Advances neural information processing systems, pp. 271-279, 2016.
Training generative neural samplers
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In International conference on machine learming, pp. 2642-2651, 2017.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. arXiv preprint arXiv:2002. /569, 2020.
Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Computer vision with single (robust) classifier. In arXiv: 1906.0943, 2019.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data. Advances in Neural Information Processing Systems, pp. 5014-5026, 2018.
Abhishek Sinha, Mayank Singh, Nupur Kumari, Balaji Krishnamurthy, Harshitha Machiraju, and VN Balasubramanian. Harnessing the vulnerability of latent layers in adversarially trained mod- els. In arXiv: 1et::00000005550 2019.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In arXiv preprint arXiv: 1710.1006606666 2017.
Jannyu Wang and Haichao Zhang. Bilateral adversarial training: Towards fast training of more robust models against adversarial attacks. Proceedings of the IEEE International Conference on Computer Vision, 2019.
Boxi Wu, Jinghui Chen, Deng Cai, Xiaofei He, and Quanquan Gu. Does network width really help adversarial robustness? arXiv preprint arXiv:2010.01279 2020.
Zhonghui You, Jinmian Ye, Kunming Li, Zenglin Xu, and Ping Wang. Adversarial noise layer: Regularize neural network by adding noise. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 909-913. IEFR, 2019.

---

Under review as a conference paper at ICLR 2021
Sergey Zagoruyko and Nikos Komodakis. arXin:1605.07146. 2016.
Wide residual networks.
In arXiv preprint
Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John Hopcroft, and Liwei Wang. Adversarially robust generalization just requires more unlabeled data. arXiv preprint FrXiv:1906.00 2019.
Haichao Zhang and Jianyu Wang. Defense against adversarial attacks using feature scattering-based adversarial training. In Advances in Neural Information Processing Systems, pp. 1829-1839, 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint MrXiv:/901.08577 2019.
Shufei Zhang, Kaizhu Huang, Jianke Zhu, and Yang Liu. Manifold adversarial learning. In arXiv:1887.005832, 2018.
APPENDIX
A LIST OF MAJOR NOTATION
For clarity, we list the major notations that are used in our model.
Xorg tion;
{x : Qo): the set of clean data samples, where Qo is its underlying distribu-
Xp {a' : E B(r,e), Vz Qo}: the set of perturbed samples, the element of Xp is in the e-neighborhood of the clean example ~ Qo:
Jo: the mapping function from input the latent features of the last hidden layer (i.e., the layer before the softmax layer);
Q*: the underlying distribution of the latent feature fo(x:) for all Xorgi
Po: the underlying distribution of the latent feature fo(x') for all E Xp:
P: the feasible region of the latent distribution Ps. which is defined as P fe(x') subject Yz Qo, x' (zx,e)) e)}
Xadv: the set of the worst perturbed samples or manifold adversarial examples, the element addi Xado are in the c-neighborhood of clean example Qu:
Pt: the worst latent distribution within the feasible region P which leads to the largest divergence or the underlying distribution of the latent feature fa(aadv) for all zade E >Xavo
B ADDITIONAL EXPERIMENT DETAILS
B.1 MODEL ROBUSTNESS AGAINST PGD AND CW ATTACKER UNDER DIFFERENT ATTACK BUDGETS
We further evaluate the model robustness against PGD and CW attacks under different attack budgets with fixed attack step of 20. These results are shown in Figure is observed that the performance of Adversarial Training with the PGD method (AT) drops quickly as the attack budget increases. The Feature Scattering method (FS) can improve the model robustness across a wide range of attack Feature Scattering by a large margin under different attack budgets especially under CW attack, except that our ADLT- IMT is slightly inferior Feature Scattering under PGD attack with budget 20 CIFAR-10.
budgets. The proposed approach ADLT-IME further boosts the performance
12

---

Under review as a conference paper at ICLR 2021
Accusoc (6)
Accurncy (%6)
60-
Aceupcy
40
60-
40
20
AGLT-MI
F5
<0
ADUT-IIT
20
10
20-
16
Rudget
Rudget
Atnick Budget
(a) PGD on CIFAR-10
(b) PGD on CIFAR-100
(c) PGD on SVHN
80-
50
Accurazy (N))
60-
Accuraey
40
A1
Accuracy
80-
AT
ADUT-MMT
12
16
16
20
16
20
Atlack Budget
Altack Dudget
Atack Budget
(d) CW CIFAR-
(c) CW on CIFAR 100
(f) CW svhn
Figure 3: Model performance under PGD and CW attacks with different attack budgets.
B.2 MODEL ROBUSTNESS AGAINST AUTOATTACK AND RAYS
As shown in (Croce & Hein 2020 Chen & Gu 2020). several models (such as Feature Scattering) could achieve high enough robustness against PGD and CW attack, but they may fail defend more stronger attacks. To further evaluate the model robustness against stronger attacks, we evaluate the robustness of our proposed method IMT+ against AutoAttack (Croce & Hein 2020) and RayS (Chen
& Gu 2020) attacks with Lox budget 8 on CIFAR- 10 and CIFAR-100.
We first compare the accuracy of the proposed ATL.DIMMT+ with several competitive methods on CIFAR-10 in Table 4 to defend the AutoAttack (AA) and Rays attacks, including: (1) Traditional
Adversarial Training with PGD (AT) (Madry et al. 2017); (2) TRADES: trading adversarial ro-
bustness off against accuracy (Zhagg al. 2019); (3) Feature Scattering: generating adversar-
ial examples with considering inter-relationshpp of samples (Zhang & Wang. 2019;; (4) Robust-
overfitting:
improving models adversarial robustness by simply using early stop (Rice et al. 2020
(5) Pretraining: improving models adversarial robustness with pre-training (Hendrycks et al. 2019
(A): mitigating the perturbation stability deterioration wider models (Wu et al. (020);
RTS: achieving high robust accuracy with semisupervised learning procedure (self- selff-taainng! (Cr-
mon al., 2019); (8) Gowal et al. 2020 achieving ofate-of-the-arr the-art results by combining larger models, Swish/SiLU activations and model weight averaging.
These comparison algorithms attain the most competitive performance on defending AA attack. As observed, overall, our proposed method achieves clear superiority over all the defence approaches on both the clean data and adversarial examples (except on clean data, ours is slightly inferior to Gowal et al.] (2020) which is however trained with additional data). Note that Pretraining, WAR Gowal al. (2020) with footnote require additional data for training (e.g. unlabeled data, pre- training).
We also report the accuracy of ATLD-IMT+ with the state-of-the-arts methods on CIFAR-100 in Table 5 against the AutoAttack (AA). Our proposed method again achieves on both the clean data and AA attacked examples significant better performance than all the other defense approaches (without data augmentation). Furthermore, it is noted that, while our XTLD-IMIT method is just slightly inferior to Gowal et al.) (2020) (which is trained with additional data), it is substantially ahead of the normal version of Gowal et al. 2020
13

---

Under review as a conference paper at ICLR 2021
Table 4: Accuracy under AutoAttack (AA) and RayS on CIFAR-10
CIFAR- 1ate--00t== = 8)
MODELS
CLEAN
CW 100
AA
RAYS
AT
87.14
50.60
44.04
50.70
TRADES
84.92
56.43
53.08
57.30
FS
89.98
60.6
36.64
44.50
ROREST-ONEIPITING
85.34
58.00
53.42
58.60
PRETRAINING"
87.11
57.40
54.92
60.10
85.60
59.78
RTS
89.69
62.50
59.53
64.6
GOWAL AT. 2020
85.29
57.14
$7.20
GOWAL
2020
91.10
65.87
65.88
ALLD-IMI
86.42
75.20
70.49
70.60
ATLD-IMT+
90.78
74.46
70.60
81.68
INDICATES MODELS WHICH REQUIRES ADDITIONAL DATA FOR TRAIN-
Table 5: Accuracy under AutoAttack (AA) on CIFAR-I00
CIFAR-IOD(C 8)
MODELS
CLEAN
CW100
AA
RAYS
OVVERTTTIIG
53.83
28.10
18.95
PRETRAINING"
59.23
33.50
28.42
GOWAL 2020
60.86
30.67
30.03
GOWAL LT AL 2020
69.15
37.70
36.88
41.98
ATLD-IME
68.17
46.15
32.36
43,91
INDICATES MODELS wilChn REQUIRES ADDITIONAL, DATA TRAIN-
ING.
B.3 BLACK BOX RESULTS ON SVHN AND CIFAR-100
We conduct more evaluations on the transfer-based black-box attacks on SVHN and CIFAR- 100. We report the results in Table It can be observed that our proposed method overall outperforms Feature Scattering in most of the cases on SVHN. Surprisingly, the Adversarial Training method, i.e. the PGD, performs better than our method and Feature Scattering method in three cases. This also partially reveals the more challenging nature of defending black-box attacks than white-box attacks.
CIFAR-100, can observed that our method and Feature Scattering are comparable, The performance of these two methods differs little though our method outperforms Feature Scatter- ing significantly under PGD20 and CW20 against adversarial attacks generated from the Feature Scattering model.
Overall, though the proposed ATLD method may not lead to remarkably higher performance than the current state-of-the-at aht algorithms in defending black- box attacks (as observed in the case of white- box attacks), it still generates overall better or comparable performance. We will again leave the further exploration of defending black-box attacks as future work.
B.4 ILLUSTRATION OF THE OVERLAID BOUNDARY CHANGE OF DIFFERENT METHODS
We conduct a toy example in Figure to illustrate the effect on how the various methods would affect the decision boundary after the adversarial training is applied. In Figure (a) shows the decision boundary trained with clean data; (b) shows the decision boundary ad adversarially trained with the perturbed samples by PGD; (c) presents the decision boundary given by the adversarial training of Feature Scattering: and (d) illustrates the decision boundary trained from our proposed ATLD. Clearly, both the PGD (Figure 4(b) and the FS (Figure [()) vary the original decision boundary significantly. Moreover, it can be observed that the adversarial training with PGD corrupts the data manifold completely. On the other hand, FS appears able to retain partially the data man- ifold information since it considers the inter-sample relationship locally. Nonetheless, its decision boundary appears non-smooth, which may hence degrade the performance. In contrast, as shown in Figure 4(d) our proposed method considers to retain the data manifold globally, which varies the
14

---

Under review as a conference paper at ICLR 2021
Table 6: Accuracy under black-box attack SVHN and CIFAR-100
(SVHN)
DEFENSE MODELS
VANILLA TRAINING
AnVERSARIAL TRAINING
PRATURE SCATTERING
Ouas
FGSM
PGD20
CW20
FGSM
PGD20
CW20
FGSM
PGD20
CW20
FGSM
PGD20
CW20
89.17
FS
$0.89
36.65
87. 10
82.27
63.59
65.18
96.59
81.91
82.52
93.11
81.65
82 26.46
AILD
$2.81
89.39
89,49
83.37
66,00
65,28
96.42
91.88
92.36
91.75
36,46
57.37
ATLD-IMT+
80.27
82.18
82.53
79.61
64.63
64.84
90.33
86.84
82.06
87.97
83.27
83.71
ATTACKED MODELS (CIFAR 100)
DEFENSE MODELS
VANILLA TRAINING
ADVERSARIAL TRAINING
PEATURE SCATTERING
OUBS
FGSM
PG1920
CW20
PGSM
PG1220
CW20
FGSM
PGD20
CW20
FGSM
PGD220
CWW20
AT
60.46
60.88
60.92
47.16
40.37
39.53
61.61
61.20
60.78
60.71
60.99
60.96
FS
66.51
71.47
71.88
$6.96
41.99
46.37
74.00
62.07
62.07
69.35
63.36
63.50
67.30
67.20
ATLD-IMT
$7.01
59.32
ATLD-IMT+
57.27
61.40
60.66
48.15
39.07
41.30
63.09
61.48
60.78
61.32
60.95
59.93
decision boundary slightly. This may explain why our proposed ATLD method could outperform the other approaches.
(a) Original
Adversarial Training
(c) Feature Scattering
(d) Ours
Figure 4: The overlaid decision boundary after the various adversarial training is applied
B.5 FURTHER DETAILS OF ATLD-IMT
We elaborate the training procedure of our IMT in this section. The overall architecture of ATLD- IMT plotted Figure 5 A test sample fed into classifier, and the discriminator outputs the prediction. A special perturbation in IMT is then computed from the loss Dw and added back to x: in this way, the sample would be pushed towards the manifold of natural samples, which is supposed be further away from the decision boundary. The prediction of the transformed xf by the adversarially-trrne -trained classifier will then be output as the label x.
To illustrate clearly the effect of our ATLD-IMT, Figure where we respectively plot the clean or natural data, perturbed data attacked by PGD, and adjusted data by ATLD-IMT in (a), (b), and (c). Moreover, the decision boundary is given by ATLD
conduct additional toy experiments shown in
15

---

Under review as a conference paper at ICLR 2021
in all the three sub-figures. In (a), it deserves attention that the boundary learned by ATLD could classify natural data well compared to the PGD and Feature Scattering as shown Section A.3. As observed in (b), the perturbations generated by PGD will push the natural samples toward or even cross the decision boundary. Our proposed IMT can push the samples towards the manifold of natural examples observed in (c). Since the manifold of natural examples would be more separable, this may further increase the classification performance as observed in the experiments.
classifier fo
adversarial example example
class label
2
Intent feature
forwardprop
backprop
transformed example xf
class label
manifold label
esgmi
Dig Ox
Dig Ax
log Di
discriminator Dw
Figure 5: Detailed Procedure of IMT. 1) The natural example or adversarial example x is fed into the network, and the discriminator outputs its prediction. The loss log Dw is computed and the transformed example xf (red arrow) is then generated. 2) The transformed sample is fed into the network and classified by the adversarially-trainee network.
(a) Clean data
(b) Perturbed data by PGD
(c) Adjusted data ATLD-IME
Figure 6: Illustration of ATLD-IMT. The decision boundary is given by ATLD in all the three sub- figures, while (a) shows clean data, (b) draws perturbed data attacked by PGD, and (c) plots adjusted data by ATLD-IMTT. Our proposed IMT can push the samples towards the manifold of natural exam- ples as observed in (c). Since the manifold of natural examples would be more separable, this may further increase the classification performance.
16

---

Under review as a conference paper at ICLR 2021
B.6 ILLUSTRATION OF VECTOR FIELD OF DIFFERENT PERTURBATION SCHEMES
(a) Adversarial Training by PGD
(b) Feature Scattering
(c) Ours
Figure 7: Vector field illustration of different perturbation schemes. (a) PGD, (b) Feature Scattering. (c) the proposed ATLD method. The overlaid boundary is from the model trained on clean data. The figure plots the direction of adversarial perturbations at different points. worth noting that most directions of the adversarial perturbations for the conventional adversarial training methods point to decision boundary. indicates that the resulting adversarial examples are easily biased towards the decision boundary which potentially corrupts the structure of the underlying distribution. The perturbation directions for Feature Scattering and our method are influenced by decision boundary less.
C DETAILED DERIVATION
In this section, we provide the details about the derivation for the main objective function (6) and elaborate how to compute the adversarial examples and the transformed examples.
C.I DERIVATION FOR MAIN OBJECTIVE FUNCTION (6)
We start with minimizing the largest f-divergence between latent distributions Po and Qp induced by perturbed example x² and natural example æ. And we denote their corresponding probability density functions as p(z) and g(=). According to Eq. 3 we have
sup [1. (1 (1)))d
min max (p(Pol)Qo) = min max ((=)
((2)
8
Qo
tEdomf-
a(2))
p(z)T(z)dz / g(2))
2 min max sup( a(=) Ter
(9)
min maxsup Qo
Ex (Vw + (
W
min E max
(Vx
(fo( (o
W
z'E((a.e)
To compute the Jensen- Shannon divergence between Py and Qo. we set (t = - -o and (g)
log(2 e²). Then, have
min max DJs(Po 2 min sup W
BANDY max [log Dw Dw (fo(x)))) + log Dw (To (10) r'B((a,))
Qo
where (w(x) e-w(+e-Vv(() 10) equivalent to optimize the lower bound of Jensen- Shannon divergence between Po and Qo. With disentangling the computation of adversarial ex- amples from Eq. (10) and further considering the classification loss for the classifier Ly and the
17

---

Under review as a conference paper at ICLR 2021
discriminator L can obtain the final objective:
N
min sup
log Dir (Fo((o))))))) (1 log (io(f((x)))
+ Lxz y::0) 0) min
( (DW (fo(T.)). v.) + ((DW (fa(z;
(II)
W
41 log DW (fo(z() + (1 - log Div (fo(zz))
s.t. 2i ado
= arg max xfEB(e,,e)
47
C.2 COMPUTATION FOR ADVERSARIAL EXAMPLE AND TRANSFORMED EXAMPLE
To compute the adversarial example, need to solve the following problem:
Fi ade
arg max ((;,))))))
log DW (f)(z)) + (1 log DW (z)())))
(12)
E2
can be reformulated as computing the adversarial perturbation as follows:
F adv
arg max [L](2 + + 0)] |rll
(13)
We first consider the more general case ||r||p I e and expand 13 with the first order Taylor expan-
sion as follows:
adv
arg max [Z 0)] + V.F" ||llpSe
(14)
where F L(x;, 0). The problem
14] can be reduced to:
max V
ra
(15)
We solve it with the Lagrangian multiplier method and we have
V VaFri
A(IIrilpp e)
(16)
Then we make the first derivative with respect to
V=F= 1
(17)
p (-/)
X ( - p-1
V
p
)
( pー p
(18)
If we sum over two sides, we have
E(V.A)
(19)
A0' x1
|V
F p'
(20)
p
where p" is the dual p, i.e. 1
"
p We have
|
V..FFl1 112-
(21)
18

---

Under review as a conference paper at ICLR 2021
By combining (88 and 21 we have
V
r =esgn Fv
(22)
esgnn V
In this paper, we p Then we have
lim 8-200
sgn(V
L
)
|
14
(23)
esgn (egn(V.... |
)
esgn( (V
Then we can obtain the adversarial example:
x x4 + esgn (2L2)
(24)
To compute the transformed example, we need to solve the following problem:
arg min log DW (fe(æ + +)).
(25)
Sr
With the similar method, we can easily get the transformed example x²
x² = esgn(V,, log D(y).
(26)
19


Reviews:
[{"review_id": "vKNSD7gK_U", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a framework for adversarial robustness via incorporating local and global structure of the data manifold. Specifically, the key motivation is that standard adversarial methods typically use only sample specific perturbations for generating the adversarial examples, and thus using them for robustness of the learning model is limited. Instead the paper proposes to capture the global data manifold as well in the robustifying framework. To this end, an objective is presented (4,5) that uses latent data distributions, with the goal that the adversarial perturbations should maximize the f-divergence against the latent distribution of the clean samples. Experiments are provided on several datasets and demonstrate significant performance improvements.\n\nPros:\n1. The key idea of using the global data manifold into the robustifying framework is quite interesting.\n2. Experiments demonstrate good empirical benefits of the approach.\n\nCons: \n1. While, the paper seemed well organized in the beginning, I got totally lost with Eq. (4-5). As I see, this objective is inaccurate and needs significant refinement. Specifically, it is unclear how is x^{adv} is related to x, and how is x^{adv} related to P*_theta? The paper tries to explain this objective in the paragraph below, but the explanation is very confusing as well.  A few other things that could help here:\na) It is said that \"Q_theta and P_\\theta* are the latent distributions induced by the natural example x\". How can a single data point induce a distribution? Do you assume the feature map from a hidden layer of a network represents a distribution? If so, in what sense? \nb) \"The adversarial example is crafted to induce the worst case distribution P*\". How is it crafted and what is the relation between P* and x? This is the key connection that is missing from (4-5).\n\n2. Moving along, Section 4.1 is organized very poorly as well. I believe too many concepts are tied together into one formulation in (6), making it hard to decipher. For example, why to include the classifier D^{1:C} within this formulation? Why not talk about it elsewhere and focus on the meat of the objective, systematically? \n\n3. Further, as I understand, x^{adv} is the first step that happens in (6), however, there is no \"adversary\" in this case, instead is finding a perturbed sample x' that maximizes the f-divergence. In what sense is x^{adv} then an adversarial sample? Perhaps the paper should re-define what is the definition of an adversarial example that it is using, to clearly state what the idea is. Technically, there is no requirement that the point x^{adv} found by this step will promote any data misclassification; however can be any point that is within a B(x,\\eps) ball from x, and that happens to maximize this divergence loss. Note that none of the other components D_W, f_theta, etc. are well trained in doing this optimization. So they could also be sub-optimal (in the sense of what the paper argues in the beginning of Page 4).\n\n4. Why is the middle formula in (6) minimizing over W to have both x and x^adv matched with the same label? Again, where is the adversary here? Or for that matter, how will the proposed approach achieve adversarial robustness ? \n\nMinor comments:\na. What is \\tau and T in (3)? \nb. How is f_\\theta defined in (6)? \nc. The paper writes that back and forth that there is no use of label information in the setup, however has labels used in discriminator in (6). This is very confusing. \nd. There is also reference to data manifold and manifold label in Figure 2, but these are not clearly explained. What precisely is the data manifold? Is it the latent distribution for a specific label? \ne. Page 4, top para: \"without considering the inter-relationship between data samples\". Won't this relation be captured implicitly through the neural network parameters theta when perturbations on all the samples are used in the training process?\n\nOverall, I think this paper needs a thorough revision to explain well its technical contributions. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "pEPBJlVSWN5", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper considers the local and global information in adversarial attacks for adversarial training, where the authors design an adversarial framework containing a discriminator and a classifier. The idea is interesting and the paper is easy to follow. \n\nHowever, I have still some concerns below: \n- The novelty of this work combines the idea of PGD (local information) and Feature-Scatter (global information) .\n- More importantly, the evaluation is no enough, even though Feature-Scatter considers the global information, but many attack methods have shown the robustness of Feature-Scatter was overestimated, such as [1][2][3] and so on. So I think evaluating on PGD and CW  is not enough.\n- There are few analysis experiments for the proposed method, more analysis experiments are needed besides the comparision.\n\n[1] Feature Attack: https://openreview.net/forum?id=Syejj0NYvr&noteId=rkeBhuBMjS\n\n[2] RayS: A Ray Searching Method for Hard-label Adversarial Attack. KDD 2020.\n\n[3] Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. ICML 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "cIp99ASc2ES", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new method of improving model robustness by generating adversarial samples that are regularized by their latent distribution through f-divergence, whereas existing literature only uses local manifold property such as smoothness. \n\nThe method is well-motivated and the clarity of the paper is good. The experimental results are compared with several competitive baselines and the improvement looks significant (Although I am not familiar with the state-of-the-art experimental results). \n\nProofread is needed for the sentence \"The adversarial examples are crafted by ... \" on page 2 and several other small typos. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "EIlDln-C5Ne", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper analyzes the property of local and global data manifold for adversarial training. In particular, they used a discriminator-classifier model, where the discriminator tries to differentiate between the natural and adversarial space, and the classifier aims to classify between them while maintaining the constraints between local and global distributions. The authors implemented the proposed method on several datasets and achieved good performance. They also compared with several whitebox and blackbox methods and proved superiority. \n\nThis paper was, in general, well written. The authors provided a good visualization of their analysis. Using local and global information for adversarial training is intuitive. The authors provided a good theoretical background to establish their method. The empirical evaluations show promising results. \n\nSome major concerns are listed as follows:\n1. It is not clear how equations 4 and 5 are realized using discriminator and classifier. \n2. What kind of perturbations are chosen? It looks like all the experiments are with L-infinity. Does this observation hold for other ones?\n3.  If the attackers leverage the global and local data manifold, can they bypass this attack? ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a framework for adversarial robustness by incorporating local and global structures of the data manifold. In particular, the authors use a discriminator-classifier model, where the discriminator tries to differentiate between the original and adversarial spaces and the classifier aims to classify between them. The authors implement the proposed approach on several datasets and the experimental results demonstrate performance improvements. The idea of using the global data manifold into addressing robustness of the learning model is interesting. However, the technical contribution and novelty have not been explained very well.
Decision: Reject