GAT: Generative Adversarial Training for Adversarial Example Detection and Robust Classification
Authors: Xuwang Yin
Soheil Kolouri
Gustavo K Rohde
Venue:  (2020)
Keywords: adversarial example detection
adversarial examples classification
robust optimization
ML security
generative modeling
generative classification

Abstract: The vulnerabilities of deep neural networks against adversarial examples have become a significant concern for deploying these models in sensitive domains. Devising a definitive defense against such attacks is proven to be challenging, and the methods relying on detecting adversarial samples are only valid when the attacker is oblivious to the detection mechanism. In this paper we propose a principled adversarial example detection method that can withstand norm-constrained white-box attacks. Inspired by one-versus-the-rest classification, in a K class classification problem, we train K binary classifiers where the i-th binary classifier is used to distinguish between clean data of class i and adversarially perturbed samples of other classes. At test time, we first use a trained classifier to get the predicted label (say k) of the input, and then use the k-th binary classifier to determine whether the input is a clean sample (of class k) or an adversarially perturbed example (of other classes). We further devise a generative approach to detecting/classifying adversarial examples by interpreting each binary classifier as an unnormalized density model of the class-conditional data. We provide comprehensive evaluation of the above adversarial example detection/classification methods, and demonstrate their competitive performances and compelling properties. Code is available at https://github.com/xuwangyin/GAT-Generative-Adversarial-Training

Published as a conference paper at ICLR 2020
GAT: GENERATIVE ADVERSARIAL TRAINING FOR ADVERSARIAL EXAMPLE DETECTION AND ROBUST CLASSIFICATION
Xuwang Yin
Soheil Kolouri
Department of Electrical and Computer Engineering University of Virginia
Information and Systems Sciences Laboratory HRL Laboratories, LLC. skolouri @hr1 com
xy4cm@vi rgi rigiiiaa. edu edu
Gustavo K. Rohde
Department of Electrical and Computer Engineering University of Virginia
gustavo@virginia.cem
ABSTRACT
The vulnerabilities of deep neural networks against adversarial examples have become a significant concern for deploying these models in sensitive domains. Devising definitive defense against such attacks proven to be challenging. and the methods relying on detecting adversarial samples are only valid when the attacker is oblivious to the detection mechanism. this paper we pro- pose principled adversarial example detection method that withstand norm- constrained white-box attacks. Inspired by one-versus-the-res classification, in K class classification problem, train K binary classifiers where the i-th bi- nary classifier is used to distinguish between clean data of class i and adver- sarially perturbed samples of other classes. At test time, first use trained classifier to get the predicted label (say k) of the input, and then use the k-th binary classifier determine whether the input is clean sample (of class or an adversarially perturbed example (of other classes). We further devise a generative approach to detecting/Classifying adversarial examples by interpreting each binary classifier as an unnormalized density model of the class-conditional data. We provide comprehensive evaluation of the above adversarial example de- tection/classfiicttion methods, and demonstrate their competitive performances and compelling properties. Code available at https: / liggttuuub com/ !
INTRODUCTION
Deep neural networks have become the staple of modern machine learning pipelines, achieving state- of-the-art performance extremely difficult tasks in various applications such as computer vision (He et al., 2016), speech recognition (Amodei et al., 2016), machine translation (Vaswani et al., 2017), robotics (Levine al., 2016), and biomedical image analysis (Shen et al., 2017). Despite their outstanding performance, these networks are shown to be vulnerable against various types of adversarial attacks, including evasion attacks (aka, inference perturbation attacks) (Szegedy al., 2013; Goodfellow et al., 2014 Carlini & Wagner, 2017b; Su et al., 2019) and poisoning attacks (Liu et al., 2017: Shafahi al., 2018). These vulnerabilities in deep neural networks hinder their deployment in sensitive domains including, but not limited to, health care, finances, autonomous driving, and defense-rede related applications and have become major security concern.
Due to the mentioned vulnerabilities, there has been a recent surge toward designing defense mech- anisms against adversarial attacks (Gu Rigazio, 2014; Jin al., 2015; Papernot et al., 2016b;
A thorough evaluation of the proposed method available Tramer al. (2020).

---

Published as a conference paper at ICLR 2020
Bastani et al., 2016; Madry et al., 2017; Sinha al., 2018), which has in turn motivated the de- sign of stronger attacks that defeat the proposed defenses (Goodfellow et al., 2014; Kurakin et al., 2016b;a; Carlini & Wagner, 2017b; Xiao et al., 2018; Athalye et al., 2018; Chen et al., 2018; He et al., 2018). Besides, the proposed defenses have been shown to be limited and often not effective and easy to overcome (Athalye et al., 2018). Alternatively, large body of work has focused on detection of adversarial examples (Bhagoji et al., 2017: Feinman et al., 2017: Gong et al., 2017; Grosse et al., 2017; Metzen al., 2017: Hendrycks & Gimpel, 2017: Li & Li, 2017: Xu al., 2017: Pang et al., 2018; Roth et al., 2019; Bahat et al., 2019; Ma et al., 2018; Zheng & Hong, 2018; Tian et al., 2018). While training robust classifiers focuses on maintaining performance in presence of adversarial examples, adversarial detection only cares for detecting such examples.
The majority of the current detection mechanisms focus on non-adaptive threats, for which the attacks are not specifically tuned/tailored to bypass the detection mechanism, and the attacker is oblivious to the detection mechanism. In fact, Carlini & Wagner (2017a) and Athalye et al. (2018) showed that the detection methods presented in (Bhagoji et al., 2017: Feinman et al., 2017; Gong et al., 2017: Grosse et al., 2017; Metzen et al., 2017: Hendrycks & Gimpel, 2017: Li & Li, 2017; Ma et al., 2018), are significantly less effective than their claims under adaptive attacks. Overall, current solutions are mostly heuristic approaches that cannot provide performance guarantees.
In this paper we propose detection mechanism that can with- stand adaptive attacks. The idea is to partition the input space into subspaces based on the original classifier's decision bound- ary, and then perform clcan/adversarial example classification the subspaces. The binary classifier in each subspace is trained to dis- tinguish in-class samples from adversarially perturbed samples of other classes, At inference time, we first use the original classi- fier to get an input sample's s predicted label k, and then use the k-th binary classifier to identify whether the input is a clean sample (of class k)) or an adversarially perturhed sample (of other classes). Fig. 1 provides schematic illustration of the proposed approach.
Twciton
000 Natunn
XXX Petuctatiiss
Figure I: A conceptual vi- sualization of the proposed adversarial example detection mechanism.
Our specific contributions are: (1) We develop a principled adversarial example detection method that can withstand adaptive attacks. Empirically, our best models improve previous state-of-the-ar mean Le distortion from 3.68 to 5.65 on MNIST dataset, and from 1.1 to 1.5 on CIFARIO 10 dataset. (2) We study powerful and versatile generative classification models derived from our detection framework and demonstrate their competitive performances over discriminative robust classifiers. While discriminative robust classifiers are vulnerable to rubbish examples, inputs that have confident predictions under our models have interpretable features.
RELATED WORKS
Adversarial attacks Since the pioneering work of Szegedy et al. (2013), a large body of work has focused on designing algorithms that achieve successful attacks on neural networks (Goodfellow al., 2014; Moosavi-Dezfooli et al.. 2016; Kurakin et al., 2016b; Chen al., 2018; Papernot al., 2016a; Carlini & Wagner, 2017b). More recently, iterative projected gradient descent (PGD), has been empirically identified as the most effective approach for performing norm constrained attacks, and the attack reasonably approximates the optimal attack (Madry et al., 2017).
A Adversarial example detection The majority of the methods developed for detecting adversarial attacks are based on the following core idea: given trained K-class classifier, f IRd - (1...), ). and its corresponding clean training samples, D = {=; E IR ] generate a set of adversarially attacked samples DD' = (a' E R and devise a mechanism to discriminate 'D from D'. For instance, Gong et al. (2017) use this exact idea and learn binary classifier to distinguish the clean and adversarially perturbed sets. Similarly, Grosse et al. (2017) append a new "attacked" class to the classifier, f. and re-train a secured network that classifies clean images, E D, into the K classes and all attacked images, x' E D', to the (K + 1)-th class. In contrast to Gong et al. (2017); Grosse et al. (2017), which aim detecting adversarial examples directly from the image content, Metzen et al. (2017) trained a binary classifier that receives as input the intermediate layer features extracted from the classifier network and distinguished D from D' based on such input features. More
2

---

Published as a conference paper at ICLR 2020
importantly, Metzen et al. (2017) considered the so-called case of adaptiveldynamic adversary and proposed to harden the detector against such attacks using a similar adversarial training approach as in Goodfellow et al. (2014). Unfortunately, the mentioned detection methods are significantly less effective under an adaptive adversary equipped with a strong attack (Carlini & Wagner, 2017a; Athalye al., 2018).
PROPOSED APPROACH TO DETECTING ADVERSARIAL EXAMPLES
classifier f Dr
clussiferr f D
classifier f D
cliss
class
cluss
Dz
Dy
D2
Ds
Da
class
clase
class
cluss
class
class
f(e) 1, Di(r) T: adv example
f(a) - 2, Dg(a) T: adv exomple
f(x) = 3, Dg(x) > T: clean sample
Figure 2: A schematic illustration of the proposed method for determining whether an input sample (represented by the gray star) is an adversarial example. The first figure shows the case where is predicted by f as class and then x is identified as an adversarial example by Dj. The following two figures shows the other two cases where is respectively predicted as class 2 and class 3 and then D2 and D3 is respectively used to predict whether x is an adversarial example.
The proposed approach to detecting adversarial examples is based on the following simple idea. Assume there an input sample x, and it is predicted i by the classifier S, then ær is either true sample of class & (assuming no misclassification) or an adversarially perturbed sample of other classes. To determine which is the case we can use binary classifier that is specifically trained to distinguish between clean samples of class & and adversarially perturbed samples of other classes. Because & can be any one of the K class, we need to train total of K binary classifier in order to have a complete solution. Fig. 2 provides a schematic illustration of the above detection idea. We next provide a mathematical justification and more details about how to train the detection model.
In K(K 2 2) class classification problem, given a dataset of clean samples D = { 24 E Rdd along with labels [ 3/ E {1,.... K}. let f Rdd + {1,.... K} be classifier on D, and D' be set of p-norm bounded adversarial examples computed from D: D {x + 5 f(z + 6) * y: fx) y, x E D, E S). S {6 Rd || S e). We use D/ {a fx) k, E D)
and D' = {x= f(x) = k x E D'} to respectively denote the clean samples and adversarial examples which are predicted by f as class k. Let H = ) where de I + [0. 1] is binary classifier trained to distinguish between samples from D/ (assigned class 1) and samples
from D' (assigned as class O). We use dx to model p(x E DÍ (x)) and predict æ E D when (x(x)) > 2 and D' when (x) < 1 Consider the following procedure to determine whether a sample æ is an adversarial example (i.e., whether it comes from D or D'):
First obtain the estimated class label k f(x). then use the k-th binary clas- sifier to predict: if dr (z) >= 2 then categorize x as a clean sample, otherwise categorize it as an adversarial example.
This algorithm be viewed as binary classifier, and its accuracy is given by
Σ
1x du (x)
2 x
E
D +
Ha dx s 1.2 E D'LH
(1)
D + D
Crucially, because the errors of individual binary classifiers are independent, maximizing Eq. (1) is equivalent to optimizing the performances of individual binary classifiers. de solves the binary clas- sification problem of distinguishing between samples from DÍ and samples from / k and therefore
3

---

Published as a conference paper at ICLR 2020
can be trained with binary classification objective:
0 arg arg min Dn [[ddx(x; (x)d)((;:))))) 0)] + AND (x; 0x))
(2)
8L
where L is a loss function that measures the discrepancy between dg 's's output and the supplied label (e.g., the negative log likelihood loss). In order to harden du against adaptive attacks, we follow Madry et al. (2017) and incorporate the adversary into the training objective:
(d min p(0x); where p(0x)= =EZDDf Lses max 8k
+E [ (())(x;
((=))()
(3)
where D {x: f(x) * k. # k E D). and we assume that is large enough such that VIr E D/ 38 E 8, s.t. f(x + 8) = k.
The equality constraint f(x 6) = k in Eq. (3) complicates the inner maximization. We observe that by dropping this constrain we have the following upper bound of the first loss term:
max ( L(dx(r + 0;0x), 0) < max L(dx (x + 0).
BES
Because we are minimizing [(dx(x ( 0;))), 0), we can instead minimizing this upper bound, which gives us the unconstrained objective
p(Ox) = E
DDD
max L(d (xr 5;0s).0) + EEWDY (<de(<;;0)) BES
(4)
We can further simply this objective by using the fact that when D is used as the training set, f can overfit on D such that Di {x+ y/ + k} and Dr. are respectively good approximations of D and o
max L(d (=d+t HEWDD [ ((:)(=:(
min p(0x), where p(0x) E. 6
(5)
BES
In words, each binary classifier is trained using clean samples of particular class and adversarial examples (with respect to da) created from samples of other class. The inner maximization is solved using the PGD attack (Madry et al., 2017). We use the negative log likelihood loss as L and minimize it using gradient-bart based optimization methods.
From classification point of view, that has rejection option: given input æ and its prediction label & = f(x), if d; T, then rejected, otherwise it's classified as We will refer to this classification model as an integrated classifier.
reformulate the above detection algorithm classifier
3.1 A GENERATIVE APPROACH TO ADVERSARIAL. EXAMPLE DETECTION/CLASSSIICAAIII
The proposed approach makes use of a trained classifier f to get the predicted label, but f is not strictly necessary: we can use H {d+ ) in place of f to do classification.
We can interpret H as an set-versus-the-ree (OVR) classifier. In a K class classification problem, OVR classifier consists of K binary classifiers, with each one trained to solve a two-class problem of separating samples in a particular class from samples not in that class. H differs from traditional OVR classifier in that de trained to distinguish between samples class l and adversarially perturbed samples of other classes, but because the loss adversarial inputs is an upper bound of the loss on clean samples, the binary classifier should also be able to separate samples of class OVR classifier, the classification rule from clean samples of other classes. When H is viewed
(()) arg max di (I)).
We can also interpret H. generative classifier. Our experiments show that de has strong generative property: performing adversarial attacks on de causes visual features of class & to appear in the attacked data (in some cases, the attacked data become valid instance of class k). Although
4

---

Published as a conference paper at ICLR 2020
similar phenomenon is observed in standard adversarial training (Tsipras et al., 2018; Engstrom et al., 2019; Santurkar et al., 2019), the generative property of our model seems to be much stronger than that of softmax adversarially robust classier (Fig. 4, Fig. 6, and Fig. 7). These results motivate us to reinterpret de as an unnormalized density model (i.e., an energy-hased model (LeCun et al,, obtain the class-conditional probability
2006)) of the class-k data. This interpretation allows of an input by:
p(z]k) exp( exp(-E,, (x))
Zμ
where E, (x) = -Zdd (2), with zd, being the logit output of da, and
Zx = exp((E, (x)))xx
(8)
is an intractable normalizing constant known as the partition function. We can then apply the Bayes classification rule to obtain a generative classifier:
H arg max [k]=))= = arg max p(x:/k)p(R
arg max 2dz (z),
(9)
p(z)
k
k
where we have assumed all partition functions Zz, k 1.... K and class priors p(k), k to be equal. Because we explicitly model p(ar, k). we can use this quantity to reject low probability inputs which can be any samples that do not belong to class In this work we focus on the scenario where low probability inputs are adversarially perturbed samples of other classes and the rejected samples are considered as adversarial examples. Because du (z) is computed by applying the logistic sigmoid function to zd, (zr), and the logistic sigmoid function is a monotonically increasing function of its argument, the generative classifier (Eq. (9)) is equivalent to the OVR classifier (Eq. (6)).
K
In the following sections, will use integrated detection refer to the original detection approach where we make use an extra classifier f. and generative detection to refer to this alternative approach where we first use the generative classifier Eq. (9) to get the predicted label & of an input x, and then use dg to determine whether x adversarial input.
EVALUATION METHODOLOGY
4.1 ROBUSTNESS TEST
We first validate the robustness of individual binary classifiers by following the standard methodol- ogy for robustness testing: we train the binary classifier with PGD attack configured with a particular combination of step-size and number of steps, and then test the binary classifier's performance under PGD attacks configured with different combinations of step-sizes and number of steps. We use AUC (area under the ROC Curve) as the detection performance metric. AUC is an aggregated measure- ment of detection performance across a range of thresholds, and can be interpreted as the probability that the binary classifier assigns a higher score to random positive sample than to a random negative example. For a given dy, the AUC is computed on the set ((x, 0) D' ((z, 1) D/),
where D A {or + arg max; (2 + 5; 0z), 0) x E D())
4.2 ADVERSARIAL EXAMPLE DETECTION PERFORMANCE
Having validated the robustness of individual binary classifier, we evaluate the overall performance of the proposed approach to detecting adversarial examples. According to the detection algorithm, first obtain the predicted label i f()), ). and then use the k--th binary classifier's logit output to predict: if zd, (x) 2 T, then & is a clean sample, otherwise it is an adversarially perturbed sample.
We use D (T;, 94) 1 to denote the test set that contains clean samples, and D' = {(x; + 8., yi ) to denote the corresponding perturbed test set. For a given T, we compute the true positive rate (TPR) D and false positive rate (FPR) on D' (here, clean samples in the positive class). These two metrics are respectively defined as
TPR=
1x D
zdg (x)
2 T,k T,k= f(x). (2. DHI
(10)
5

---

Published as a conference paper at ICLR 2020
and
FPR
[D'] (x: ) + y, (z. v) E D')
(11)
We observe that for the norm ball constraint we considered the experiments, not all perturbed samples can cause misclassification J. so we use f(x) & in the FPR definition to constrain that only adversarial inpuls that actually cause misclassification can be counted as false positives.
Given a clean sample r and its groundtruth label y. we consider three approaches to creating the corresponding adversarial example x'. Here we will focus on untargeted attacks.
Classifier attack This attack corresponds to the scenario where the adversary is oblivious to the de- tection mechanism. Inspired by the CW attack (Carlini Wagner, 2017b), the adversarial example x' is computed by minimizing,
L(z)= = = max (z())
(12)
*#
where 2y(x') is the classifier's logit outputs.
Detector attack In this scenario adversarial examples are produced by attacking only the detec- tor. We first construct detection function H by aggregating the logit outputs of individual binary classifiers:
zH (2), 2d, (z).
(13)
The adversarial example x' is then computed by minimizing
L(x') max z (x') (z');- iyy
(14)
According to our detection rule, a low value of a binary classifier's logit output indicates the de- tection of adversarial example, and therefore by minimizing the negative of the logit output make the adversarial input harder to detect. H can also be used with the CW loss Eq. (12) or the cross-entropy loss, but we find the attack based Eq. (14) be most effective.
Combined attack The combined attack is an adaptive method that considers both the classifier and the detector. We consider adaptive attack of Carlini & Wagner (201 7a) which has been shown to be effective against existing detection methods. We first construct a new detection function H with Eq. (13) and then use H's largest logit output maxkæy zH (z)) (low value of this quantity indicates detection of an adversarial example) and the classifier logit outputs 2 to construct new classifier g:
loss functions for the combined attack. The first is based on the
(2)(Tr);
if < K,
(x); = {(
(15)
maxj zH (z)) + 1) max, 2)(2),
if = K+1
The adversarial example x' is then computed by minimizing the loss function
I(x') max =g(x'): -max 24 2y (x'),.
(16)
if
In practice we observe that the optimization of Eq. (16) tends to stuck at the point where maxi3fy zy (x'); keeps changing signs while max z (x), staying as a large negative number (which indicates detection). In light the above issues we derive a more effective attack by com- bining Eq. (12) and Eq. (14):
T(x') [2)(=') maxi/y (x');;
if zf((") 2 maxi/y 2y(x');
(17)
maxx/y 2n (xx);
else.
In words, if x is not yet adversarial example on f (case 1), optimize it for that goal, otherwise optimize for evading the detection (case 2).
We note that the above three attacks are for the original detection approach (i.e., integrated detec- tion). The generative detection approach (Section 3.1) does not make use of f and we use Eq. (14) to create adversarial examples for generative detection.

---

Published as a conference paper at ICLR 2020
4.3 ROBUST CLASSIFICATION PERFORMANCE
In robust classification,, the accuracy of a softmax robust classifier is evaluated on the original test test (the standard accuracy) and the adversarially perturbed test (the robus! accuracy). Because the generative classifier comes with the reject option, we use slightly different metrics. On the clean test dataset D {o yt )t we define the standard accuracy as the fraction of samples that are correctly classified (f(x) = y) and at the same time not rejected (2dg (x) 2 T):
Accuracy = N {x : 2d, (x) 2 T, = (z),
(z, E DH--
(18)
In the adversarially perturhed test dataset D' = [1x yi === we will consider data sample as properly handled when it is rejected (zd; (x) < 7), regardless of whether it causes miselassification. In this way, only misclassified (f)) # y) and unrejected (zd, (x) 2 T) samples counted as errors:
Error N Ht za, (x) 2 T, f(x), # u. (2, E D'H
(19)
To compare different classifiers under the same metrics, compute the error of softmax robust classifier on DD as
Error N Hr g(x) # y, (z. y) E D'H.
(20)
We respectively use Eq. (17) and Eq. (14) to compute the D' for the integrated classifier and the generative classifier.
EXPERIMENTS
5.1 MNIST
We train four detection models (each consists of ten binary classifiers) by using different combi- nations of p-norm and perturbation limit (Table 5). The adversarial examples used for training and validation are computed using PGD) attacks of different steps and step sizes (Table 5). At each step of PGD attack we use the Adam optimizer to perform gradient descent, both for L2-based and Loo -baed attacks. Appendix A.I provides more training details.
Robustness results Table and Table show that do and d1 are able to withstand PGD attacks configured with different steps and step-size, for both Ly-based and -based attacks. The binary classifiers also exhibit robustness when the attack uses p-norm or perturbation limit that are different from those used for training the model (Table 8). The models also robust when the attacks use multiple random restarts (Table 9).
Table 1: AUC scores of the first two binary classifiers (d1, d2) tested with different configurations of PGD attacks. In each step of the PGD attack we the Adam optimizer to perform gradient descent.
PGD attack steps, step size
L c = 0.3 model d
Lge dj
= 0.5 model PGD attack steps, step size
Lg = 2.5 model dj
L
5.0 model dg
d2
d2
d2
d
200, 0.01 2000, 0.005
0.99959 0.999558
0,99971 0.99971
0,99830
0.99869
200,
0,99962 0.99927
O. 0.99968
0,99578 0.99529
0.99987 0.99918
0.99796
0.99861
2000, 0.05
0.99900
Table Mean L2 distortion (higher is better) of perturbed samples when the detection method has 1.0 FPR on the perturbed MNIST test set and 0.95 TPR on the clean MNIST test set.
Detection method
Mean L2 distortion
State-of of-the-art (Carlini & Wagner, 2017a)
3.68
Ours (generative detection with T.x. E 0.3 hinary classifiers)
4.40
Ours (generative detection with T.o E 0.5 binary classifiers)
5.65
Detection results Fig. 3a shows the performances of integrated detection and generative detection under different attacks. Combined attack with Eq. (17) is the most effective attack against integrated
7

---

Published as a conference paper at ICLR 2020
detection, and is much more effective than the combined attack with Eq. (16). Overall, genera- tive detection outperforms integrated detection when they are evaluated under their respective most effective attack. It is also interesting to note that when the adversarial examples created by at- tacking only the classifier (Eq. (12)), integrated detection is able to perfectly detect these adversarial examples (see the red curve that overlaps the y-axis).
Given that generative detection is the most effective approach among the proposed approaches, we compare with state-of-the-art detection methods (Carlini & Wagner, 2017a). Table 2 shows that generative detection outperforms the state-of-the-an method by large margins. Appendix B provides details about how the mean L2 distortions are computed.
Classification results Figure 3b shows the standard and robust classification performances of the proposed classifiers and state-of-the-art softmax robust classifier (Madry al., 2017). Our models provide the reject option that allows the user to find balance between standard accuracy and robust error by adjusting the rejection threshold. We observe that stronger attack
0.4) breaks the softmax robust classifier (as indicated by the right red cross), while the generative classifier still exhibits robustness, even though both models are trained with the Logo c 0.3 constraint.
0.965
robust classeferr
True True Postbva Rata
0.96
0.975
classifier (eps-0.4)
Integrated detection Icambined attack)
integrated detection Icombined attack ew lestt
0.965
integrated detection idetectors attack!
0.960
Integrated classifier
Integrated detection telassifier attack)
Generative detection
0.955
Generative classifier (eps-
Generative classifier
0.950
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.2
0.4
0.6
0.8
Fabse Positive
Errer perturbed MMST
(a)
(b)
Figure 3: (a) Performances of integrated detection and generative detection under T 0.3 constrained attacks. (b) Performances of the integrated classifier and generative classifier under Loo e 0.3 constrained and Loe 0.4 constrained attacks. The performances of the softmax robust classifier (Madry et al., 2017) (accuracy 0.984, етrоr 0.08 at 0.3, and accuracy 0.984, error 0.941 at 0.4) are marked with red crosses. PGD attack steps 100, step size 0.01.
Figure 4 shows perturbed samples produced by performing targeted attacks against the generative classifier and softmax robust classifier. The generative classifier's perturbation samples have distin- guishable visible features of the target class, indicating that individual binary classifiers have learned the class conditional distributions, and the perturbations have to change the semantics for a success- ful attack. In contrast, perturbations introduced by attacking the softmax robust classifier are not interpretable, even though they can cause high logit output of the target classes (see Figure for the logit outputs distribution).
5.2 CIFARIO
On CIFARIO we train a single detection model that consists of ten binary classifiers using L c constrain PGD attack of steps 40 and step size 0.5 (note that the scale of c and step size here is 0-255, as opposed to 0-1 as
case MNIST). The softmax robust classifier (Madry et al., 2017) that we compare with is also trained with Lox. 8 constraint but with different step size (Appendix C.2.2 provides a discussion on the effects of step size). Appendix A.2 provides the training details.
Robustness results Table 3 shows that dj and d2 withstand PGD attacks configured with different steps and step-size. In Appendix C.2.1 we report random restart test results, cross-norm and pross-perturbation test results, and robustness test result for L2 based models.
Detection results Consistent with the MNIST result, Fig. 5 shows that combined attack with Eq. (17) is the most effective attack against integrated detection, and generative detection simi- larly outperforms integrated detection. Table shows that generative detection outperforms the state-of-thed the- adversarial detection method.
8

---

Published as a conference paper at ICLR 2020
samples
samp les (ge nerative
samples darsmerr
6
D
f
U
0
A
/
A
2
1
2
7
&
7
2
4
7
2
2
2
2
2-
A 2 2 + S 4 S
2
3
3
5
S
5
5
3
3
3
3
3
4
9
4
4
9
9
6
5
5
5 6 / 7
6
/
6
0
6
/
4
6
6
C &
5
2
7
7
>
7
7 O
7
7077 T 7
8
2
Y
C
8
C
L & C
4
9
2
9
S
Figure 4: Clean samples and corresponding perturbed samples produced by performing targeted attack against the generative classifier and robust classifier (Madry et al., 2017). Targets from top row to bottom row are digit class from 0 to 9. We perform the targeted attack by maximizing the logit output of the targeted class, using Lo 0.4 constrained PGD attack of steps 100 and step size 0.01. Both classifiers are trained with Lo E 0.3 constraint.
Table AUC scores the first two CIFARIO Lis e = 8 binary classifiers (d1, d2) under Lox é constrained PGD attacks of different steps and step-size.
PGD attack steps, step-size
d1
dg
20, 2.0
0,9224
0.9533
40, 0.5
0,9234
0.9553
200, 0.1
0,9231
0.9550
200, 0.5
0,9205
0,9504
500,
0.9203
0,9500
Table 4: CIFARIO mean Le distortion (higher is better) of perturbed samples when the detection method has 1.0 FPR perturbed set and 0.95 TPR the clean set. Appendix B provides details about how the mean L2 distances are computed.
Detection method
Mean La distortion (0-1 scale)
State-of saate-of-the-are art (Carlini & Wagner, 2017a) Ours (generative detection)
1.1
1.5
robunt clessiferr
Accunety on CPARIO test
robaa adassifer (epam12)
0.6
0.5
Toue
0.3
0.4
integrated didection (combired atlack)
integrated detection (combined attack loss)
(dotector attack)
integrated dotection (cbssttier
Generative dassiter
Generative detection
Denurative class (upom
(a)
(b)
Figure 5: (a) Performances generative detection and integrated detection under Loo 8 attack. (b) Performances of integrated classifier (discussed in Section 4.3) and generative classifier under Leo = 8 constrained and Loc 12 constrained attacks. The performances of the robust classi- fier (Madry et al., 2017) (accuracy 0.8735, error 0.5311 I at = 8, and accuracy 0.8735, error 0.7087 e 12) are annotated. PGD attack step size 2.0, steps 20 for e = 8, and 30 for 12.
Classification results Contrary to MNIST's result, we did not observe a dramatic decrease in the softmax robust classifier's 's performance when increase the perturbation limit to 12 (Fig. 5b). Integrated classification can reach the standard accuracy of a regular classifier, but at the cost of significantly increased error the perturbed set. Fig. 6 shows some perturbed samples produced

---

Published as a conference paper at ICLR 2020
by attacking the generative classifier and robust classifier. While these two classifiers have similar errors on the perturbed set, samples produced by attacking the generative classifier have more visible features of the attacked classes, suggesting that the adversary needs to change more semantic to cause the same error.
Fig. 7 and Fig. 11 demonstrate that unrecognizable images able to cause high logit outputs of the softmax robust classifier. This phenomenon highlights a major defect of the softmax robust classifier: they can be easily fooled by unrecognizable inputs (Nguyen et al., 2015; Goodfellow et al., 2014; Schott et al., 2018). In contrast, samples that cause high logit outputs of the generative classifier all have clear semantic meaning. In Figure 14 we present image synthesis results using Loo c = 16 constrained detectors. In Appendix D we provide Gaussian noise attack results and discussion about the interpretability of the generative classification approach.
Natural samples
Perturbed samples (generative classifirr)
Perturbed samples (robust classifier!
Figure 6: Clean samples and corresponding perturbed samples by performing targeted attack against the generative classifier and robust classifier (Madry et al., 2017). The targeted attack is performed by maximizing the logit output of the targeted class. We use L c = 12 constrained PGD attack of steps 30 and step size 2.0 to produce these samples.
Generater ted by atta cking generative daassifier
Generated attacking robust classifier
VD7
Figure 7: Images generated from class conditional Gaussian noise by performing targeted attack against the generative classifier and robust classifier. We use PGD attack of steps 60 and step size 0.5 x 255 to perform To2 30 255 constrained attack (same Santurkar et al. (2019). The Gaussian noise inputs from which these two plots are generated are the same. Samples not selected.
CONCLUSION
We studied the problem of adversarial example detection under the robust optimization frame- work and proposed a novel detection method that can withstand adaptive attacks. Our formula- tion leads to generative modeling technique which called generative adversarial training (GAT). GAT's capability to learn class conditional distributions further gives rise to generative de- rection/classificatinn approaches that show competitive performance and improved interpretability.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, al. Deep speech End-to- end speech recognition in english and mandarin. In International conference on machine learning, pp. 173-182, 2016.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Cireumventing defenses adversarial examples. arXiv preprint arXiv:1802.00424, 2018. URL https:
00420. 00420.
Yuval Bahat, Michal Irani, and Gregory Shakhnarovich. Natural and adversarial error detection using invariance to image transformations. arXiv preprint arXiv:1902.0023:: 2019.
Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya Nori, and An- tonio Criminisi. Measuring neural net robustness with constraints. In Advances in neural infor- mation processing systems, pp.
2016.
Arjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal. Dimensionality reduction as a defense against evasion attacks machine learning classifiers. arXiv preprint 1xXiv:1704.026533 2017.
Nicholas Carlini and David Wagner. Adversarial examples are easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Security, pp. 3-14. ACM, 2017a. URL https: / arwiivvvvvvv//aav wwwrivv.orree org/ abs 11005 07263.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017b.
Pin- Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsich. Ead: elastic-net attacks to deep neural networks via adversarial examples. In Thirty-second AAAI conference on artificial intelligence, 2018.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Alek- sander Madry. Adversarial robustness as a prior for learned representations. arXiv preprint arXiv:1906. 1.X0019906.00445, 2019.
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint 17X3v:1703.004100 2017. https: http:::://axxvvv/a////oe abs/1700 00410.
Zhitao Gong, Wenlu Wang, and Wei-Shinn Ku. Adversarial and clean data are not twins. arXiv preprint arXiv::7044 04960, 2017. URL https: httpp:////xxvv.rr//// wtwpaiss/ro..////ooe//ooo/ssssss abs///////// /1704 04960.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 2014. URL org/ abs 412. 6572.
Kathrin Grosse, Praveen Manoharan, Nicolas Michael the (statistical) detection adversarial examples. CORR, 2017.
Michael Backes, and Patrick McDaniel. On
Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint arXiv:1412.5068 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Warren He, Bo Li, and Dawn Song. Decision boundary analysis of adversarial examlees. In Interna- tional Conference on Learning Representations, 2018. URL https: http::///peeeeeeeeeeeeeee Forum?id-BkpiPMa
Dan Hendrycks and Kevin Gimpel. Early methods for detecting adversarial images. ICLR, 2017. URL wwt.tttpp///s.. /ttp///avvvvv// org/ abs///b///// 1008. 00530.

---

Published as a conference paper at ICLR 2020
Jonghoon Jin, Aysegul Dundar, and Eugenio Culurciello. Robust convolutional neural networks under adversarial noise. arXiv preprint 11X::1511..66366, 2015.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint 1rXiv 02233,, 2016a. URL http:://///r wwtpxiii.rrg///aee /a/////////aa 167. 02533.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint urXiv:1611.01266 2016b.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
S Levine, C Finn, Darrell, and P Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17:1334-1321 1373, 2016.
Xin Li and Fuxin Li. Adversarial examples detection in deep networks with convolutional filter statistics. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5775-5783, 2017. URL https://aaaatit. www./oog///a...ooooooo..oo...e// /ttpa/ab//11..177
Yuntao Liu, Yang Xie, and Ankur Srivastava. Neural trojans. In 2017 IEEE International Conference Computer Design (ICCD), pp. 45-48. IEEE, 2017.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality, arXiv preprint 1rXiv:1801.02613, 2018. URL org/abs/ 1801.11111 02613.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0608 2017. URL https: arxiv. wwwpiiioorggaaaee wogg/bs//////0 /1706. 06083.
Madryl.a, CIFAR10 Adversarial Examples Challenge.
cifarl0_challenge, a.
MadryLab. MNIST Adversarial Examples Challenge. httpps///gittub..cmm/maaaa mmist_challenge b.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversar- ial perturbations. CoRR, abs/1702.04267, 2017. URL https///wwwwwww///aaaa/e 04267.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2574-2582, 2016.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436- 2015.
Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection of adversarial ex- amples. In Advances in Neural Information Processing Systems, pp. ht https: : http://axvvvv.o///// org/ alss/ 100.. 00633.
2018. URL
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European Sympo- sium on Security and Privacy (FuroS&P), pp. 372-387. IEFF, 2016a.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pp. 582-597. IEEE, 2016b.
Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: A statistical test for detecting adversarial examples. arXiv preprint arXiv: arXiv:1902.00 1902.0492999 18, 2019.
12

---

Published as a conference paper at ICLR 2020
Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Image synthesis with a single (robust) classifier, 2019.
Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on mnist. arXiv preprint arXiv:/805.09/90 2018
Ali Shafahi, Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Advances in Neural Information Processing Systems, pp. 6103-6113 13, 2018.
Dinggang Shen, Guorong Wu, and Heung-ll Suk. Deep learning in medical image analysis. Anmual review of biomedical engineering, 19:221 2428 2017.
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifiable distributional robustness with principled adversarial training. In International Conference on Learning Representations, 2018.
URL https: /wppeepeeeeieeeeeeeeeeeeeeeeeeee net//frummmfm
forum? i id=Hk 6xP
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation, 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:13122 13/2.6199, 2013. URL https: org/ abs abs/1312. 6199.
Shixin Tian, Guolei Yang, and Ying Cai. Detecting adversarial examples through image transfor- mation. In Thirly-Second AAAI Conference on Arlificial Intelligence, 2018.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. arXiv preprint arXiv:2002.088: 083447, 2020.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry, Robustness may be odds with accuracy, stat, 1050:11, 2018. URL https:///axxv.or////// abs/ 12152.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp. 5998-6008 2017.
Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating ad- versarial examples with adversarial networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 3905-3911. AAAI Press, 2018.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint arXiv: 1704.0 155, 2017.
Zhihao Zheng and Pengyu Hong. Robust detection of adversarial attacks by modeling the intrinsic properties of deep neural networks. In Advances in Neural Information Processing Systems, pp- 7913-7922, 2018
A TRAINING DETAILS
A.1 MNIST TRAINING
We use 50K samples from the original training set for training and the remaining 10K samples for validation, and report the test performance based the checkpoint which has the best validation performance. All binary classifiers are trained for 100 epochs, where each iteration we sample 32 in-class samples the positive samples, and 32 out-class samples to create adversarial examples which will be used as negative samples.
All binary classifier models 32 and 64 filters, and a fully connected layer size 1024; more details of the network architecture can be found in Madry al. (2017).
neural network consisting of two convolutional layers each with
13

---

Published as a conference paper at ICLR 2020
Table 5: Training setups for MNIST detection models
La models
Age models
6 2.5
< 5.0
0.3
0.5
PGD attack steps, step-size (training)
100,
200. 0.1
100, 0.01
100, 0.01
PGD attack steps, step-size (validation)
200,
200,00
200, 0.01
A.2 CIFARIO TRAINING
We train CIFARIO binary classifiers using ResNet model (same as the one used by Madry et al. (2017); MadryLab (a)). To speedup training, we take advantage of a clean trained classifier: subnetwork of f that defines the output logit 2f(-)x is essentially a "binary classifier" that would output high values samples of class k; and low values for others. The binary classifier is then trained by finetuning the subnetwork using objective 5. The pretrained classifier has test accuracy of 95.01% (fetched from MadryLab (a)).
At each iteration of training sample a batch of 300 samples, from which in-class samples used as positive samples, while an equal number of out-of-class samples are used for crafting ad- versarial examples. Adversarial examples for training L2 and Loo models are both optimized using normalized steepest descent based PGD attacks (MadryLab, b). We report results based on the best performances the CIFARIO 10 test (thus don't claim generalization performance of the proposed method).
B COMPUTING MEAN L2 DISTANCE
We first find the detection threshold T with which the detection system has 0.95 TPR. We construct new loss function by adding weighted loss term that measures perturbation size to objective 14
L(x') max zH (x'); + ||2| x112- ifa
(21)
We then use unconstrained PGD attack to optimize L(x'). We use binary search to find the optimal c,
where in each bsearch attempt if is false positive (max, zи (x'); + and maxişy zи (x'); >> T) we consider the current effective and continue with a larger The configurations for performing binary search and PGD attack are detailed in Table The e upper bound is established such that with this upper bound, samples except those that inherently misclassified by the generative classifier, could be perturbed as a false positive. With these settings, our MNIST Lox = 0.3 and Too 0.5 generative detection models respective reached 1.0 FPR and 0.9455 FPR, and CIFAR10 generative model reached 0..9955 FPR.
Table 6: Binary search and PGD attack configurations on MNIST and CIFARIO dataset
Datasct
Iniuial e
Jowerr bound
c upper bound
bscarch depth
PGD steps
PGD step size
Threshold
PGD optimizer
MNIST
0.0
0.0
8.0
20
1000
1.0 (0-1 scalee
3,6
Adam
CIFARIO
0.0
0.0
1.0
100
2.56 (0-255 scale)
-5.0
comalized steepest descent
C MORE EXPERIMENTAL RESULTS
C.1 MORE MNIST RESULTS
14

---

Published as a conference paper at ICLR 2020
Table 7: AUC scores of the first two binary classifiers (d1, d2. MNIST) tested with different configurations of PGD attacks. In each step of the PGD attack use normalized gradient as in Madry et al. (2017) (the update rules for Lz-based and based attacks are respectively VV f(x,)))
(( and 2n
En+1 Zn /
PGD attack steps, step
Loe c 0.3 model
Loo
0.5 model
PGD attack steps, step
1/2 2.5 model
L2 5.0 model
d1
d2
dj
dg
d2
dj
dg
200,
0.99962
0,99973
0.99820
0,99901
200,
0.99906 0.99855
0,99916
0.99960 0.99237
0,99997
2000, 0.005
0.99959
0.99971
0.99795
0.99872
2000, 0.05
0.99883
0.99994
Table 8: AUC scores of the first two binary classifiers (d1, d2, MNIST) under cross-norm and cross- perturbation attacks. Lo -based attacks use steps 200 and step-size 0.01, and L2 based attacks uses steps 200 and siep-size 0.1.
dg
Attack
0.5
Ly
5.0
Ios 0.3
0,99959
0.99966
0.99927
0.99928
0,99971
0,99967
0,90949
0,99984
Toe 0.994366
0.9983
0.99339
0.997677
0.99778
0.99869
0.99397
0.99961
12 2.5
0.99974
0.99999
0.99962
0.99944
0.99965
0.99955
0.99968
0.99987
Lue 0,96421
0.98816
0,97747
0.99577
0,98268
0,98687
0,98117
0,99986
Table 9: AUC scores of d, (MNIST) under fixed start and multiple random restarts attacks. The Loo 0.5 model is attacked using Loo = 0.5 constrained PGD attack of steps 200 and step size 0.01, and the L2 = 5.0 model is attacked using L2 5.0 constrained PGD attack of steps 200 and step size 0.1.
Lo 0.5 model
Lg c model
fixed start 50 random restarts
0. 0.99830
0.99578 0.99501
0.99776
Table 10: AUC scores of Loo c 0.3 binary classifiers. constrained PGD attacks of steps 200 and step size 0.01.
Results obtained with Loo c 0.3
41
d2
de
dy
dg
dg
d10
AUC
0,99959
0.99971
0,99876
0.99861
0. 0,99859 0,99861
0.99795
0,99863
0.99687
0.99418
Table 11: AUC scores Loo e 0.5 binary classifiers. Results obtained with Loo e 0.5 constrained PGD attacks of steps 200 and step size 0.01.
dj
d2
dg
d4
dg
de
dy
ds
dg
d1o
AUC
0.99830
0.99869
0.99327
0.99355
0.99314
0,99228
0,99424
0,99439
0,97875
0.9769
True positive rate on MNIST test set
1.00
Accuracy MNSIT test set
1.00
0.98
0.98
0.96
0.96
0.94
0.94
Generative detection
Generative classifier
0.92
Generative detection (xent loss)
0.92
Generative classifier (xent loss)
Generative detection (ew loss)
Generative classifier (cw loss)
0.90
0.2
0.4
0.8
1.0
False positive perturbed MNIST test
Error perturbed MNIST test
Figure 8: Perforrance of generative detection (a) and generative classification (b) on MNIST dataset under attacks with different loss functions. Please refer to Madryl.ab (b) for the implementations of cross-entropy loss and CW loss based attacks.
15

---

Published as a conference paper at ICLR 2020
generative classifier
robust classifier
outputs natural samples class
natural samples class logit outputs of generated samples
lagit outputs of generated samples
80
60
40
20
10
-10
Figure 9: Distributions of class I's logit outputs of clean samples from class and perturbed samples from the first row of Figure 4 (MNIST dataset).
16

---

Published as a conference paper at ICLR 2020
C.2 MORE CIFARIO RESULTS
C.2.1 MORE ROBUSTNESS TEST RESULTS
Table 12: AUC scores of CIFARIO dj under fixed start and multiple random restarts attacks. The
Los E 2.0 model is attacked using PGD attack of steps 10 and step size 0.5, and the Loo e 8.0 model is attacked using PGD attack of steps 40 and step size 0.5.
Loo 2.0 model
Loo 8.0 model
fixed start 10 random starts
0.9866
0.9234
0.9866
0.9233
Table 13: AUC scores of CIFARIO d1 (trained with Loo norms and perturbation limits.
8) under PGD attacks with different
PGD attack
AUC
Le S0-coostrained (steps 20. step-size 10)
0.9814
Loo 2-constrained (sleps 10, step-size 0.5)
0,9841
Table 14: AUC scores of CIFARIO di, d2 (trained with L2 e SO-constrained PGD attack of steps 20 and step size 10) under L2-based PGD attacks of different steps and step-size.
L 80-constrained PGD attack steps, step size
d
d2
20, 10
0.9839
0.9924
50, 5.0
0.9837
0.9922
Table 15: AUC scores of all CIFARIO L 2.0 binary classifiers under Loo 2.0-constrained PGD attack steps 10 and step size 0.5.
dj
du
d3
dg
dg
d7
dg
d1u
AUC 0.9866 0.9926
0.9721
0.9501
0.9773
0.9636
0.9859
0.9908
0.9930
0.9916
Table 16: AUC scores of all CIFARIO Lo 8.0 binary classifiers under L.o 8.0-constrained PGD attack steps 40 and step size 0.5.
dj
d2
dg
ds
dg
dg
dg
d10
AUC
0.9234
0.9553
0.8393
0.7893
0.8494
0.8557
0.9071
0.9276
0.9548
0.9370
17

---

Published as a conference paper at ICLR 2020
Accurory on CIFARIO tost set
0.8
0.8
Truc Pocitive
0.6
0.4
0.2
0.2
Generative classifler
Generative delection (xent loss)
Generative classifier (xent loss)
Generative detection (ew less)
Generative classifier (cw loss)
0.0
0.3
0.4
0.5
0.6
0.0
0.1
False Positive Rate
Error on perturbed CIFAR10 test set
(b)
Figure 10: Performance of generative detection (a) and generative classification (b) on CIFARIO dataset under attacks with different loss functions. Cross-entropy and CW loss is only able to out- performs loss 14 when detection threshold is low (over 0.9 TPR). Please refer to MadryLab (a) for the implementations of cross-entropy loss and CW loss based attacks.
generative classifier
robust classifier
160
logit outputs of natural samples of class
logit outputs of natural samples of class
140
logit outputs of generated samples
logit outputs generated samples
40
20
330
200
10
10
20
30
-30
-20
10
10
20
Figure 11: Distributions of class 1's logit outputs of clean samples of class and generated samples from the first row of Figure 7 (CIFARI0 dataset).
18

---

Published as a conference paper at ICLR 2020
C.2.2 TRAINING STEP SIZE AND ROBUSTNESS
We found training with adversarial examples optimized with a sufficiently small step size to be critical for model robustness. In table 17 we tested two Li = 2.0 binary classifiers respectively trained with 0.5 and 1.0 step size. The step size 1.0 model not robust when tested with much smaller step size. We observe that when training the step size 1.0 model, training set adv AUC reached 1.0 in less than one hundred iterations, but test set clean AUC plummeted around 0.95 and couldn't recover thereafter. (Please refer to Figure 12 for the definitions of adv AUC and nat AUC.)
Table 17: AUC scores of two Lo c 2.0 d1 models trained with different steps and step-sizes.
Training steps, step-size
Attack steps, step-size
10, 0.5
10, 1.0
10, 0.5
0.9866
0,9965
40, 0.1
0.9892
0.8848
C.2.3 EFFECTS OF PERTURBATION LIMIT
To study the effects of perturbation limit, analyze the training dynamics of one Loc 2.0 constrained and one Lis e = 8.0 constrained binary classifiers. In Figure 12 we show the training and testing history of these
models. The 2.0 model history shows that by adversarial finetuning the model reaches robustness in just a few thousands of iterations, and the performance on clean samples is preserved (test clean AUC begins 0.9971, and ends at 0.9981). Adversarial finetuning on the = 8.0 model didn't It converge after an extended 20K iterations of training. The gap between train adv AUC and test adv AUC of the 8.0 model more pronounced, and we observed decrease of test clean AUC from 0.9971 to 0.9909.
These results suggest that training with larger perturbation limit is more time and resource consum- ing, and could lead to performance decrease clean samples. The benefit is that the detector is pushed to a better approximation of the target data distribution. As an illustration, perturbations generated by attacking the naturally trained classifier (corresponds to 0 perturbation limit) don't 't have clear semantics, while perturbed samples the Loo e 8.0 model are completely recognizable.
Figure 13,
k 0, E 2.0 model
km O, Lu E 8.0 model
0.95
0.95
AUC
0.90
0.90
train nat AUC
train net. AUC
train adv AUC
train adv AUC
AUC
test adv AUC
test adv AUC
0.80
0.80
5000
10000
15000
20000
10000
20000
30000
40000
iteration
Figure 12: Training and testing AUC histories of two binary classifiers. Adv AUC the AUC score computed on {(x, 0) x E D'()) U {(x,1) x E DE}. and nat AUC the score computed on ((x, 0) x E DD U ((x, 1) x e D().
19

---

Published as a conference paper at ICLR 2020
for 2.0 Aec
delectrr
fo mi
Natural
Trated
Clocall
Figure 13: Perturbed samples produced by attacking the k (airplane) detectors and the natural trained classifier's 1st logit output, All samples reached the same La perturbation of I 1200 (produced using PGD attacks of step size 10.0).
Figure 14: Images generated from class conditional Gaussian noise by attacking L c = 16 con- strained CIFAR10 detectors. we use I.2 100 x 255 constrained PGD attack of steps 200 and step size 0.5 x 255. Samples not selected.
20

---

Published as a conference paper at ICLR 2020
C.3 IMAGENET RESULTS
On ImageNet we show GAT' induces detection robustness and supports the learning of class condi- tional distributions. Our experiment based on Restricted ImageNet (Tsipras al., 2018), subset of ImageNet that has its samples reorganized into customized categories. The dog category consists of images of different breeds collected from ImageNet class from 151 268. We trained dog class detector by finetuning pre-trained ResNet50 (He et al., 2016) model. The dog category covers range of ImageNet classes, with each one having its logit output. We use the subnetwork defined by the logit output of class 151 as the detector (in principle logit output of other classes in the range should also work). Due to computational resource constraints, we only validated the robustness of Loo 0.02 trained detector (trained with PGD attack of steps 40 and step size 0.001), and we present the result Table 18. (On Restricted ImageNet in the case Loo scenario Tsipras et al. (2018) only demonstrates the robustness of a 0.005 constrained model). Please refer to Appendix C.3 for more results on adversarial example generation and image synthesis.
Table 18: AUC scores the dog detector under different strengths of Too E = 0.02 constrained PGD attacks
Attack steps, step size
40, 0.001
100, 0.001
200,
0.001
0.002
200, 0.002
200, 0.0005
AUC
0.9720
0.9698
0,9692
0.9703
0.9690
0.9698
21

---

Published as a conference paper at ICLR 2020
(a) L2 c = 3.5 trained softmax robust classifier with La 40 constrained PGD attack of steps 60 and step
size 1.0 (Santurkar et al., 2019).
(b) Top 0.02 trained detector with L2 40 constrained PGD attack of steps 60 and step size 1.0.
(c) Lo 0.05 trained detector with L2 40 constrained PGD attack of steps 60 and step size 1.0.
(d) Leo 0.1 trained detector with La 40 constrained PGD attack of steps 60 and step size 1.0.
(e) Loe 0.1 trained detector with L2 100 constrained PGD attack steps step size 10.0.
(f) Too 0.3 trained detector with 100 constrained PGD attack of steps 100 and step size 10.0.
Figure 15: ImageNet 224 x 224 3 random samples generated from class conditional Gaussian noise by attacking softmax robust classifier and detector models trained with different constrains. Note than large perturbation models didn't 't reach robustness. Please refer to Santurkar et al. (2019) for the detail about how the class conditional Gaussian is estimated.
22

---

Published as a conference paper at ICLR 2020
(a)
(b)
Figure 16: Perturbed samples produced by attacking the Loo = 0.3 trained dog detector using
L2 = 30 constrained PGD attack of steps 100 and step size 5. Top rows are original images, and second rows are attacked images.
23

---

Published as a conference paper at ICLR 2020
Figure 17
Figure 18: More 224 224 random samples generated by attacking the Tooe 0.3 trained
detector with T22 = 100 constrained PGD attack of steps 100 and step size 10.0.
24

---

Published as a conference paper at ICLR 2020
D GAUSSIAN NOISE ATTACK AND MODEL INTERPRETABILITY
In this section we use Gaussian noise attack experiment to motivate a comparative analysis of the interpretabilities of our generative classification approach and discriminative robust classification approach (Madry et al., 2017).
We first discuss how these two approaches determine the posterior class probabilities.
For the discriminative classifier, the posterior probabilities are computed from the logit outputs of exp(ε)(#))) (a)a)
the classifier using the softmax function p(k] 2z) Σラ= exp()y(*),)) For the generative classifier, the
posterior probabilities computed in two steps: in the first, we train the base detectors, which is the process of solving the inference problem of determining the joint probability p(z, k), and in the
second, we use Bayes rule to compute the posterior probability p(k|z) p(z) Σ exp(ea, (e))
p(a.k)
exp(ze, (x))
Coincidentally, the formulas for computing the posterior probabilities take the same form. But in our approach, the exponential of the logit output of a detector (i.e., exp(*a, (a))) has clear probabilistic interpretation: it's the unnormalized joint probability of the input and the corresponding class category. We use Gaussian noise attack to demonstrate that this probabilistic interpretation is consistent with visual perception.
We start from Gaussian noise image, and gradually perturb it to cause higher and higher logit out- puts. This is implemented by targeted PGD attack against logit outputs of these two classification models. The resulting images in Figure 19 show that, in model, the logit output increase di- rection, i.e. the join probability increase direction, indicates the class semantic changing direction; while for the discriminative robust model, the perturbed image computed by increasing logit outputs are not as clearly interpretable. In particular, the perturbed images that cause high logit outputs of the softmax robust classifiers are not recognizable.
In summary, as a generative classification approach that explicitly models class conditional distri- butions, our system offers a probabilistic view of the decision making process of the classification problem; adversarial attacks that rely on imperceptible or uninterpretable noises are not effective against such a system.
E COMPUTATIONAL COST ISSUE
In this section we provide an analysis of the computational cost of our generative classification approach. In terms of memory requirements, if assume the softmax classifier (i.e., the discrim- inative softmax robust classifier) and the detectors use the same architecture (i.c., only defer in the final layer) then the detector based generative classifier is approximately K times more expensive than the K K-class softmax classifier. This also means that the computational graph of the generative classifier is K times larger than the softmax classifier. Indeed, in the CIFAR10 10 task, on our Quadro M6000 24GB GPU (TensorFlow 1.13.1), the inference speed of the generative classifier is roughly ten times slower than the sofumax classifier.
We next benchmark the training speed of these two types of classifiers.
The generative classifier has K logit outputs, with each one defined by the logit output of a detector. Same with the softmax classifier, except that the K outputs share the parameters in the convolutional base. Now consider ordinary adversarial training on the softmax classifier and generative adversarial training on the generative classifier. To train the softmax classifier, we use batches of N samples. For the generative classifier, we train each detector with batches of M samples (M positive samples and M negative samples). At each iteration, we need to respectively compute N and M x K adversarial examples for these two classifiers. Now we test the speed of the following two scenarios: 1) compute the gradient w.r.t. to N samples gradient w.r. to M x K samples on K computational graphs, with each graph working on M samples. We assume that in scenario 2 all the computational graphs are loaded to GPUs, and thus their computations are in parallel.
single computational graph, and 2) compute the
In CIFAR10 experiment, used batches consisting of 30 positive samples and 30 negative samples to train each ResNet50 binary classifiers. In Madry et al. (2017), the softmax classifier was trained with batches of 128 samples. In this case, K = 10, M = 30, and N 128. On our GPU,
25

---

Published as a conference paper at ICLR 2020
Gaussian noise attack on generative classifier
Gaussian noise attack robust classifier
loga ore G
Iechrom out 14
loaft out 19.6
Ieth outt 21.0
lodft out 22.6
logit out -70.5
logt out
iogt-out-99
luge 31.4
out 31.5
logit oUE 5.0
ough 15.0
loght 23.5
Night out. 24.0
loght 23.0
ligit ou
huttion 15.
100f 19.0
Frift 0. 22.0
looR oUT 26.0
Kos OUE
1001-01
lugh 25
lnsfrout 2.9
lngh out 6.0
HoR 14.0
1000 out 23.5
12 Out
logh OUT 25.5
KOS OUE
1001-01
IO9TOOE
lest 6.0
Iostrount 6
IIR 15.0
100h
1007 24.0
indt.ast
1001 mut
leght out 6.5
out:
1osh 22.0
Image generated by attacking the generative classifier (based on Lox 16 trained de- tectors) and discriminative softmax robust classifier (Madry et al., 2017) using (the same) Gaussian noise image. We used unconstrained L2 PGD attack of step size 0.5*255. The five columns corre- sponding to the perturbed images at step 0, 50, 100, 150, and 200.
Figure 19:
scenario 1 took 683 ms 6.76 ms per loop, while scenario 2 took 1.85 s + 42.7 ms per loop. In this case, we could expect generative adversarial training to be about 2.7 times slower than ordinary adversarial training, if not considering parameter gradient computation.
In practice, large batch size is almost always preferred. And our method won't 't compare as favorably we choose to use one.
F DENSITY ESTIMATION ON SYNTHETIC DATASETS
While ordinary discriminative training only learns good decision boundary, GAT is able to learn the underlying density functions that generate the training data. Results ID (Figure 20) and 2D benchmark datasets (Figure 21) show that through properly configured generative adversarial training, detectors' output recover target density functions.
26

---

Published as a conference paper at ICLR 2020
Urunnrry Carnoo Eagenuo output
Dancd isgyna wurport
Domne dcidey
The funcion
10
2.6
33
30
0.4
23
22
10
Figure 20: Ordinary discriminative training and generative adversarial training on real 1D data. The positive class data (blue points) are sampled from mixture of Gaussians (mean 0.4 with std 0.01, and mean 0.6 with std 0.005, each with 250 samples). Both the blue and red data has 500 samples. The estimated density function computed using Gibbs distribution and network logit outputs. PGD attack steps 20, step size 0.05, and perturbation limit 0.3.
#
68
66
0.8
06
86
04
60
1.0
1.0
90
28
68
2.4
66
0.5
100
6.
03
04
8.5
04
0.0
0.2
044
0.5
Figure 21: 2D datasets (top row, blue points are class data, and red points are class 0 data, both have 1000 data points) and sigmoid outputs of GAT trained models (bottom row). The architecture PGD attack steps 10, step
of the ML.P model for solving these tasks 55500500-5000500-55500 size 0.05, and perturbation limit I 0.5.
27


Reviews:
[{"review_id": "rklkefNcFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper studies the adversarial detection problem within the robust optimization framework. They propose an adversarial detection and a generative modeling technique called asymmetrical adversarial training (AAT). With one detector for each class discriminating natural data from adversarially perturbed data, AAT can learn class-conditional distributions, which further result in generative detection/classification methods with competitive performance. Experimental results are provided on MNIST, CIFAR10 and Restricted ImageNet, compared with CW method as baseline.\n\nThe paper is well written with detailed experimental results. I'd suggest accepting the paper.\n\nTo my understanding, the objective function of AAT is similar to GAN's, while there is a detector for each class discriminating natural data from adversarially perturbed data instead of generated data. They incorporate the attack into the training objective with three attacking scenarios: classifier attack, detectors attack, and combined attack. They also introduce integrated classification of the classifier and detectors with the reject option. Further, they demonstrate ATT promotes the learning of class-conditional distributions and leads to generative classifiers. They claim in addition to more robust classification, ATT also gives rise to improved interpretability, which I'm not convinced of with given experimental results.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SklzDtn-9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for adaptive adversarial example detection. The authors propose to construct the adversarial subspace detector based on Asymmetrical Adversarial Training (AAT). The proposed model is composed of both classifier and adversarial detector, where the classifier makes the classification prediction and the adversarial detector evaluate if the input sample is natural of adversarial. The goal of the objective function is to minimize the adversarial detector error given large enough perturbation budget.\n\nThe authors provide extensive experimental results showing the promising performance of the model in detecting various types of adversarial attack. I have several concerns regarding the model and experiments:\n\n1) Since D^{'^{f}}_k \\subset D^f_k, would the model minimize w.r.t. both the loss of L(h(x, \\theta), 0) and L(h(x, \\theta), 1)? Would this cause unstable training?\n\n2) Maybe I missed it, but it seems that the objective function in Eq. (5) is based on the adversarial detector. How could the classification performance of classifier f be guaranteed in training?\n\n3) What does the cross mark mean in Fig. 2(b) and 4(b)?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xdR_S6YH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review: The paper addresses the adversarial example detection problem. The framework proposed in the paper divides the input space into subspaces based on a classifier’s output and trains detectors on these subspaces to classify a natural sample (classified as that class) from an adversarial one fooling the network. The goal is to use a robust optimization approach to enable detection methods to withstand adaptive/dynamic attacks. Hence, an asymmetrical adversarial training (AAT) regime is employed which presents solving a min-max problem. AAT supports the detectors to learn class conditional distributions, motivating generative detection/classification approaches. There are three different attacking scenarios and evaluation shows that the combined attack turns out to be most effective (as it fools both the classifier and detectors) against integrated detection. The paper also demonstrates empirical improvements over state of the art detection techniques with higher L2 distortion of perturbed samples.\n\nPros:\n- With the vulnerabilities associated with neural networks, the motivation behind building defense mechanisms against adversarial attacks has been well-justified.\n- Most of the evaluation metrics look appropriate and well-defined.\n- It was interesting to observe the perturbed samples produced by attacking generative classifier. While they exhibited visible features of the target class, these perturbations had to be different on a sematic level to be distinguished from the natural samples. \n\n\nCons:\n- While the idea to partition into subspaces and learn a different detection for each of them is novel, it involves training multiple detectors one by one that can be computationally expensive. The loss function for different attacking scenarios uses the outputs from all the detectors, which can also be expensive, especially when there are a lot of classes.\n- To deal with extremely unbalanced data sets when training the detector, the solutions used in the paper resamples to balanced the positive and negative class data. This would mean throwing off most of the data, I would see how it affects the training. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work addresses the problem of detecting an adversarial attack. This is a challenging problem as the detection mechanism itself is also vulnerable to attack. The paper proposes asymmetrical adversarial training as a robust solution. This approach partitions the feature space according to the output of the robust classifier and trains an adversarial example detector per partition. The paper demonstrates improvements over state-of-the-art detection techniques. 

All three reviewers recommend acceptance of this work. Some positive points include the paper being well-written with strong experimental evidence. One potential difficulty with the proposed approach is the additional computational cost associated with a per class adversarial attack detector. The authors have responded to this concern by claiming that the straightforward version of their approach is K times slower (10 in the case of 10 classes), but their integrated version is 2x slower as they only run the detector associated with the example-specific class prediction. We encourage the authors to include a discussion on computational cost in the final version. In addition, there was a community comment about black-box testing which will be of relevance to many in the community. The authors have already provided additional experiments to address this question as well as code to reproduce the new experiment. 

Overall, the paper addresses an important problem with a two-step solution of training a robust model and detecting potentially perturbed samples per class. This is a novel solution with comprehensive experiments and therefore recommend acceptance. 

Decision: Accept (Poster)