Confidence-Calibrated Adversarial Training: Towards Robust Models Generalizing Beyond the Attack Used During Training
Authors: David Stutz
Matthias Hein
Bernt Schiele
Venue:  (2020)
Keywords: Adversarial Training
Adversarial Examples
Adversarial Robustness
Confidence Calibration

Abstract: Adversarial training is the standard to train models robust against adversarial examples. However, especially for complex datasets, adversarial training incurs a significant loss in accuracy and is known to generalize poorly to stronger attacks, e.g., larger perturbations or other threat models. In this paper, we introduce  confidence-calibrated adversarial training (CCAT) where the key idea is to enforce that the confidence on adversarial examples decays with their distance to the attacked examples. We show that CCAT preserves better the accuracy of normal training while robustness against adversarial examples is achieved via confidence thresholding. Most importantly, in strong contrast to adversarial training, the robustness of CCAT generalizes to larger perturbations and other threat models, not encountered during training. We also discuss our extensive work to design strong adaptive attacks against CCAT and standard adversarial training which is of independent interest. We present experimental results on MNIST, SVHN and Cifar10.

Under review as a conference paper at ICLR 2020
CONFIDENCE-CALIBRABH ADVERSARIAL TRAINING and Detection: MORE ROBUST MODELS GENERALIZ- ING BEYOND THE ATTACK USED DURING TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial training is the standard to train models robust against adversarial ex- amples. However, especially for complex datasets, adversarial training incurs a significant loss accuracy and known to generalize poorly to stronger at- tacks, c.g., larger perturbations or other threat models. In this paper, we introduce confidence-calibratee
adversarial training (CCAT) where the key idea is to en- force that the confidence on adversarial examples decays with their distance to the attacked examples. We show that CCAT preserves better the accuracy of normal training while robustness against adversarial examples is achieved via confidence thresholding, i.e., detecting adversarial examples based on their confidence. Most importantly, in strong contrast to adversarial training, the robustness of CCAT generalizes to larger perturbations and other threat models, not encountered dur- ing training. For evaluation, we extend the commonly used robust test error our detection setting, present an adaptive attack with backtracking and allow the attacker to select, per test example, the worst-case adversarial example from mul- tiple black- and white-box attacks. We present experimental results using Looo Lag. Li and Lo attacks on MNIST, SVHN and Cifarl0.
INTRODUCTION
Deep neural networks have shown tremendous improvements in various learning tasks including ap- plications in computer vision, natural language processing or text processing. However, the discov- ery of adversarial examples, i.e., nearly imperceptibly perturbed inputs that cause mis-classification, has revealed severe security threats, as demonstrated by attacking popular computer vision services such Google Cloud Vision (Iyas et al. (018a) or Clarifai (Liu et al. 2016 Bhagoji et al. 2017b). As the number of safety- and arivicy-critical applications is increasing, e.g., autonomous driving or more important.
medical imaging, this problem becomes
In practice, adversarial training and its variants Szegedy et al. 2013 Goodfellow et al. 2014 Madry al. 2078), i.e., training on adversarial examples, be regarded as the sfate-of-the-art to obtain models robust against adversarial examples. In contrast to many other defenses, and to the best of our knowledge, adversarial training has not been broken so far. However, adversarial training is known to increase test error significantly. Only on simple datasets such as MNIST (LeCun et al.) 1998,, adversarial training is able preserve accuracy. This observation is typically described as a trfde-off between robustness and accuracy (Tipras et al. 2018 Stutz et al. 2019 Raghunathan lt al. 2019- Zhang et al. 2 Furthermore, while adversarial training leads to robust models for the employed threat model used during training, the obtained robustness does not translate to other
threat models, e.g., other T.g-balls (Sharma Chen 2017
Song et al. 2008 Mudry el al.) 2018 larger perturbations than the ones used
Tramer& & Boneh 2019 Li al. Kang et al. 2019.
during training or distal adversarial examples (Hei et al. 2019).
Contributions:
We address both problems of adversarial training: the drop in accuracy datasets such as Cifar10 (Krizhevsky) 2009) and the poor "generalization" of robustness to larger pertur- bations unseen threat models. To this end, introduce confidence--llibraaee adversarial training (CCAT) based on the idea that the predicted confidence for an adversarial example should decrease with its distance the attacked example. Specifically, we bias the network to predict

---

Under review as a conference paper at ICLR 2020
convex combination of uniform and (correct) one-hot distribution for adversarial examples, which becomes uniform as the distance to the attacked test example increases. Thus, CCAT implicitly biases the network predict the uniform distribution over classes beyond the adversarial examples seen during training. In contrast, standard adversarial training forces the network to classify ad- versarial examples correctly with maximal confidence but provides guidance how to extrapolate beyond adversarial examples seen during training. Overall, CCAT allows to detect previously unseen adversarial examples, e.g., large perturbations, different Lo attacks or distal adversarial examples, by confidence thresholding, i.e, detection.
We also discuss worst-case evaluation methodology using an adaptive attack to compare our detection- approach with adversarial training. First, we generalize the commonty used robust test error (Madry et al. 2018 Scholl et al.) 2019) to our detection setting. Second, for each test example, the worst-case adversarial example, i.e,, with highest confidence, is selected from multiple white- and black-box attacks. Using an adapted attack based on (Madry et al. [018] Dong et al. 2018 augmented with novel backtracking scheme, we show that CCAT produces models with im- proved accuracy on Cifarl0 and robustness to various unseen adversarial examples: larger pertur- bations, Lzg L1 and Ly attacks, distal adversarial examples and corrupted examples MNIST(-C), LeCun et al. 1998 Mu & Gilmer] 2019) SVHN (Netzer et al.) 2017)), Cifar10(- (CC (Hendrycks & Dietterich 2019) We will make our code for CCAT and evaluation publicly available.
Outline: After reviewing related work in Sec.
1.1 we discuss adversarial training (aadry et al. we present our confidence-hhressoolded robust fest
2018 and introduce CCAT in Sec. 2 In Sec. error and present experimental results in Sec. Finally, we conclude in Sec. 5
1.1 RELATED WORK
Adversarial examples can roughly be divided into white-box attacks, i.e.. with access to the models, its weights and gradients, c.g. (Goodfellow et al. 2014 Madry et al.)
Adversarial Examples:
2017 CCarlini & Wagner, (0017)). and black-box attacks, i.c., only with access to the output of the model, e.g. (Chen et al. 2017 Brendel & Bethge, 2017: Su et al. 2017 Iyas et al. 201 & Kasiviswanathan, (017). Adversarial examples were also found to be transferable between similar models (Liu et al. 2016 Xie et al. 2018). While these attacks
Sarkar et al.
are intended be (nearly) imperceptible, visible adversarial transformations, e.g., rotations or rotations (Engstrom et al. [2077] 1177:2199-00 Dumontt et al. 2018; Alaisarii el al. 2018 Xiao el al.) 2018),
or patches (Liu et al. 2018] Lee KKotte| 2019 Brown et al. 2017 Karmon et al. [2018] Zajac
let al.) [2019) have also been proposed and transferred to the physical world (Athalye et al. 2018b
Li et al.] 2019b Kurakin et al.) 2016). We refer to recent surveys (Barreno et al. 2006] Yuan
ee al. 2017 Akhtar & Mian] 2018 Biggio & Roli 2018a] for more details.. Recently, white-box attacks utilizing projected gradient ascent to maximize cross-entrops loss or surrogate objectives, e.g., (Madry et al.) 2017, Dong et al.) 2018 Carlini & Wagner] (017b), have become standard.
Instead, we directly maximize the confidence in any but the true class, similar to (Hein et al. 2019 Goodfellow et al. 2019). to attack our proposed training procedure, CCAT.
Adversarial Training: Many defenses against adversarial attacks have been proposed, see, c.g., Yuaan et al.] 2017| Akhtar & Mian 2018 Biggio & Roli 2018b), of which some have been shown to be ineffective, e.g., in (Athalye et al., 2018a; |0188]|Athalyy & Carlini 2018). Currently, adversarial training. i.e., training on adversarial examples, is the de-facto standard to obtain robust models.
While adversarial training was proposed in different variants (Szegedy et al.] 2013 Zantedeschi al. 2017. Miyato et al. 2016; Huang et al. 2015) (Shaham 20188 Sinha et al. 2018 Madry et al. 2017 Wang et al. 2019). the formulan lation by Madry et (2017) received considerable atten- tion and has been extended in various ways (Lee al. 2017 7 Ye & Zhu 2018 Liu et al. 2019). For example, in (Shafahi et al. 2018 Pérolat et al. 2018). ad versarial traming is applied to universal adversarial example, in (Cai et al. 2018), curriculum learning is used, and in (Trmer et al., 2017] Grefenstette et al. 2018) ensemble adversarial training is proposed. Adversarial training is also used in provable/certified defenses (Kolter & Wongl 2017 Zhang & Evans 2018). The increased sample complexity of adversarial training (Schmidt et al. (008)) has been addressed in (Lamb et al. 2019 by training on interpolated examples or in (Carmon et al. 2019 Uesato et al. 2019) using unlabeled examples. The dependence the threat model used during training was addressed in (Tramer & Boneh 2019) by using multiple threat models during training. Finally, the frequently observed trade-off between accuracy and robusiness has been studied both theoretically and empir-
2

---

Under review as a conference paper at ICLR 2020
ically TTipras et al. 2018 Stutz al. 2019 Zhang al. 2019 Raghunathan et al. 2019. CCAT differs from standard adversarial training in the used attack objective and the imposed distribution over the labels, which tends towards uniform distribution for large perturbations.
Instead of correctly classifying adversarial examples, as intended in adversarial train- ing, several works [Gong et al. 2017 Rouhani al. 2018 Grosse al.
Detection:
2017 Feinman et al. 2017 Liao et al. 2 Ma et al. 2018 Amsaleg et al. 2017 Metzen et al. 2017 Bhagoji et al.
2017 Hendrycks Gimpel] 2017: Li & Li 2017] Pang et al. 2018) try instead to detect adver- sarial examples. However, several detectors have been shown to be ineffective against adaptive attacks, i.e., adversaries aware of the used detection mechanism (Carlini Wagner 2017a). Re- cently, the detection of adversarial examples based on their confidence, similar to our approach with CCAT, has also been discussed (Pang et al.) 2018); however, the authors build on (Feinman et al. 2017 which has been shown to be ineffective (Carlini & Wagner 2017a). as well. Goodfellow et al. 22019 instead, focus on evaluating confidence-based detection methods using adaptive, targeted attacks maximizing confidence. Our attack, introduced in Sec. 2.2 is similar in spirit, however, untargeted by nature and, thus, suitable for usage with CCAT.
2 CONFIDENCE CALIBRATION OF ADVERSARIAL EXAMPLES
Adversarial training, as described by Madry et al. 2018), has become standard to train robust mod- els. However, accuracy on challenging datasets such as Cifarl0 is reduced and robustness does not generalize to larger perturbations, e.g., with respect to the used L constraint, or different threat models in other Lp norms. With CCAT, we intend to address both shortcomings: Instead of training models to correctly classify adversarial examples, we want to detect adversarial examples based on their confidence. In the following, we first review adversarial training, before discussing the modifications needed for our CCAT, thereby turning robustness against adversarial examples into a detection problem.
Notation: We consider a classifier ʃ Rd - R where K is the number of classes and fa denotes the confidence for class k. While we assume that the cross-entropy loss L is used for training. our approach can also be used with other losses. Given æ E R classified correctly as y = argmax (x())), an adversarial example x + is defined as a "small" perturbation such that argmax & fa( + 6) * y. i.e., the classifier changes its decision. The strength of the change is measured by some -n-norm with p E {0, 1, 1, oo); p oo is a popular choice in the literature as this leads to the smallest perturbation per feature/pixel.
2.1 ADVERSARIAL TRAINING
Standard adversarial training, as formulated by Madry et al. (2018), is given as the following min- max problem:
min E max C(f(z + 8: w))
(1)
[1
with w being the classificr's parameters and L being the cross-entropy loss. During mini-batch training the inner maximization problem.
max L(f(x + 88 10
(2)
approximately solved. In addition to the Loc consttrann box constraint, i.e., 2; (x++ + 8)₂ e [0, 1J. is enforced for images. Note that maximizing the cross-entropy loss is equivalent to finding the adversarial example with minimal confidence in the true class. For neural networks, this is generally a non-convex optimization problem. In (Madry al. 2018) the problem is tackled using projected gradient descent (PGD), which is typically initialized using a random with ||0] < e. During adversarial training, PGD is applied to the batch of every iteration in order update the classifier's 's parameters on the obtained adversarial examples. At test time one uses the best out of several random restarts to assess robustness.
In contrast to adversarial training as proposed in Madry et al. 2018). which computes adversarial examples for the full batch in each iteration, others compute adversarial examples only for half the
3

---

Under review as a conference paper at ICLR 2020
AT
CCAT
Confidence Calibration. For ad- versarial training (AT) and our confidence- calibrated adversarial training (CCAT) with Ppow = 10 using the power transition in Eq. (6) we plot the probabilities for all ten classes along adversarial directions. Adversarial examples were computed using our Lsg-PGD-Corg at- tack, cf. Sec. [4.7| using projected gradient de- scent (Madry et al. 2018) the confidence of the adversarial examples is maximized for T 1000 and Lise -oosstrnttt bustness of AT does not generalize beyond the e = 0.03-ball as high confidence adversarial ex- amples can be found for larger perturbations, whereas CCAT predicts close to uniform con- fidence after some transition phase allowing to easily detect adversarial examples.
Figure 1:
Coslidrans per Clooo
Confióence per Cloos
0.5
0.8
0.5
00.6
0.4
0.4
0.2
0.02
0.04
0.00
0.02
0.01
0.06
Perturbation Distance
Pertarbattin Distamnce
Coellibaess ger Cloos
Couffdance per Class
0.5
0.03. The ro-
0.4
0.2
0.2
0.
0.02
0.00
0.01
0.05 0.04
Perturbation Distance
Pesturbation Distance
Classes 1.... 10
2013), i.e., instead of training only on adversarial examples, each batch is divided 50% into clean and 50% into adversarial examples. Compared to Eq. (I. 50%/50% adversarial training effectively minimizes
examples of each batch (Szegedy et al.,
min E
max So [(f(x + 6; v) B[C(f((;,) w). w)
50% adversarial training
50% "elean" training
This improves test accuracy on clean examples compared to 100% adversarial training, i.e., Eq. ①.. but typically leads to worse robustness. Intuitively, by balancing both terms in Eq. (3) the trade-off between accuracy and robustness can already be optimized to some extent (Stutz et al. 2019).
There are two problems with adversarial training: First, the e-ball around training examples might include examples from other classes. Then, the attack, i.e., the inner maximization Eq. ① will focus on these regions such that adversarial training for these examples gets "stuck". This case is illustrated in our theoretical toy dataset in Sec. 2.3 and has also been considered in related work Jacobsen et al. 20/9b al. Here, both 100% and 50%/50% adversarial training. ef. Eq. ① and 3). are not able to find the Bayes optimal classifier in a fully deterministic problem, i.e., zero Bayes error. This might contribute the observed drop in accuracy for adversarial training on, e.g., Cifar10 Krizhevsky| 2009). Second, and more importantly, adversarial training as in Eq. ( does not guide the classifier how to extrapolate beyond the used e-ball during training. Even worse, enforces high confidence predictions within the e-ball, which clearly cannot be extrapolated to arbitrary regions. is not surprising that adversarial examples can often be found right beyond the e-ball, i.e., using larger perturbations than at training time or other threat models, e.g., other ITp norm balls.
2.2
CONFIDENCE-CALIBRATEE ADVERSARIAL TRAINING
Addressing these problems requires only few but effective modifications as outlined in the description of our proposed confidence-calirrtte adversarial training (CCAT) in Alg. [ during training, we bias the network to predict a uniform distribution over the classes on adversarial examples that are sufficiently far away from the original training examples. Subsequently, during testing, adversarial examples can be detected by confidence thresholding - adversarial examples receive near-uniform confidence while test examples receive high-confidence. Thus, given an example x, the ideal attacker during training maximizes the confidence in any arbitrary other label k + y instead of minimizing the confidence in the true label y. as in Eq.
max max fa + +.: 1e kyn
(4)
where fü denotes the probability f assigns to class ls (i.e., in practice, f is the output of the soft- max layer). A similar objective but for targeted attacks has been used in (Goodfellow et al.) 2019). whereas our goal is untargeted attack and thus our objective is the maximal confidence over all
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Pseudo-code of confidence-calibran adversarial training (CCAT). The main changes compared to regular adversarial training as, e.g.. described in (Madry et al. 2018) or SSzegdyy et al. 2013), are in the attack (line 4) and the probability distribution the classes (line 6.7). which becomes more uniform as distance ||0|| increases.
1: while true do
2: choose random batch (x1, U1), .., (IB, yB)
3: for b 1,.... f//2 do
4: (maximize confidence other classes than true one for adversarial example Ty Eq.
5:
6, argmax || [ maXk/ys fx( 6)
6: ã == 2g - 5b
7:
(probability over classes of ig becomes more uniform as |,6 increases, Eq.
8:
À == e -- or À == (1 - min(1, (61l=///))))))
9:
{úr is convex combination of one hot and uniform distribution over the classes, Eq.
5
10:
übb == hone_hot(ys) + K
11: end for
12: {corresponds to 50%/50% adversarial training, Eq. 3
13: update parameters using Σ C(f (6) + + B/= ((f(xb). vo)
14:
end while
other classes. Then, during training, CCAT biases the classifier towards predicting uniform distribu- tions on adversarial examples by using the following distribution as target in the cross-entropy loss:
p()) = (k)
N)((k) 1.... K.
Here, (y. (k) is the original "one- hot" distribution, i.e., (k) = iff = = y and Py (k) = 0 otherwise with y being the true label, and u(A) = 1/K is the uniform distribution. Thus, enforce convex combination of the original label distribution and the uniform distribution which is controlled by the parameter X. We choose À decrease with the distance ||0]|c of the adversarial example to the attacked example I with the intention to enforce uniform predictions when || = 8. Then, the network is encouraged to extrapolate this uniform distribution beyond the used e-ball. Even if extrapolation does not work perfectly, the uniform distribution is much more meaningful for ex- trapolation in the region between classes compared to high-confident predictions as encouraged in standard adversarial Iraining. We consider two variants of transitions, i.e., controlling the trade-off between one-hot and uniform distribution:
= - 6
("exponential transition" (exp))
(6)
(1 - min l+))))
("power transition" (pow)
This ensures that for O we impose the original (one-hot) label. For growing 6, however, the influence of the original label decays proportional to || The speed of decay is controlled by the parameter p. For the exponential transition, we always have bias towards the true label as even for large p. À will be non-zero. In case of the power transition, X = 0 for ||o'|oe > e, meaning pure uniform distribution enforced. is important note that in CCAT in Alg. [ we train on
50% clean and 50% adversarial examples in each batch, as in Eq. 3 Training only on adversarial examples will not work as we the network has no incentive predict correct labels.
2.3
ADVERSARIAL TRAINING YIELDS ACCURATE MODELS
Proposition I analyzes 100% adversarial training and its 50%/50% variant well confidence- calibrated variant, CCAT, to show that there exist problems where both 100% and 50%/50% adver- sarial training are unable to reconcile robustness and accuracy, as recently discussed (Tsipras al.) 2018 Stulz et al.) 2019 Raghunathan el al.) 2019 Zhang et al. 2019). However, our CCAT's is able obtain both robustness and accuracy given that in Eq. (6) chosen appropriately.
We consider classification problem with points deterministic labels, that is p(y = 2/z = 0) 1 and p(y = 1|x = e) = 1 and the problem is fully determined by the probability po p(x = 0) p(æ =
Proposition 1.
0 and
in R. with
Po- The Bayes error of this
5

---

Err(T) corresponds to taking Under review as a conference paper at ICLR 2020
classification problem is zero. The predicted probability distribution over the classes is p(ylx) egu(r)
esn (91(0)+002(0)) where R + R and we assume that the function X IR4 - [0, 1] is monotonically
decreasing and X(0) = 1. The Bayes optimal classifier for the cross-entropy loss of
adversarial training on 100% adversarial examples, of. Eq. yields an error of min 1 po}.
adversarial training with 50% adversarial and 50% clean examples per batch, cf. Eq. 3 yields an error of min -po}.
CCAT on 50% clean and 50% adversarial examples, cf. Alg. 7 yields zero error if X(e) <
min {
Po 1-po } 11-po S
3 CONFIDENCE-TIRESHOLDDE THRESHOLDED ROBUST TEST ERROR (RERR)
CCAT runs in two-stage process: First, examples are rejected by confidence thresholding. Ideally, all adversarial examples are rejected as CCAT encourages low-confidenee adversarial examples. Second, we evaluate robustness and accuracy on the non-rejected (i.e., correctly or incorrectly clas- sified) examples. In the first stage (i.e., confidence thresholding). we consider successful adversarial examples negatives and correctly classified test examples positives. Then, we report the area under the ROC curve, i.e., ROC AUC, which shows how well we can discriminate adversarial exam- ples from correctly classified clean examples. In the second stage (i.e., after detection by confidence thresholding), we fix the threshold T al true positive rate (TPR) of 99% for correctly classified test samples, i.e., the network is allowed to reject most 1% of correctly classified test examples. We note that this also improves accuracy as errors on test examples typically have lower confidence. Then, we intend use test error (Err) and robust test error (RErr), as also used in (Madry et al.) 2018, to evaluate adversarial training. However, extending RErr to take into account confidence- thresholding as in our detection setting is non non-trivial: for example, correctly classified examples can can have lower confidence than their corresponding adversarial examples. Thus, given confidence threshold T, we define the confidence-thresholdee robust test error REr as follows:
N
REM(()) ((n))y, (((E))2 +
Σ 1 ((..)) ((a.) lo(x.)2T n=1
Σ lc(xw)22 + Σ Lo(aw)r (c(w))2 I ((z))=n f(an)yu n=1
Here, T is the confidence-threshold fixed on a held-out validation set, ((In; yn) ) are test exam- ples, c(x.)= maxk I's (In) denotes the classifier's confidence on Fa. [xx, == argmax & f((()r) denotes the classifier's decision, and ā, are adversarial examples. The numerator counts the number of incorrectly classified test examples n,, with c(x,)) 2 7 (first term) and the number of successful adversarial examples T, on correctly classified test examples with c(ū,) ) 2 T (second term). The denominator counts test examples In with c(x,)) 2 r (first term) and the number of successful ad- versarial examples i, with c(²,,) 2 T but where the corresponding test example Ir has c( < J (second term). The latter takes care of the special case where adversarial examples have higher confidence than their corresponding test examples, mentioned above and is encouraged by the objective of our attack in Eq. (4). In total this yields a correct fraction within [o, 1] and for 7 = O, Eq. ( reduces to the "standard" RErr. The only the first terms in both numerator and denominator.
4 EXPERIMENTS
We compare normal training, adversarial training (AT) and our CCAT on MNIST, (LeCun et al. 1998) SVHN (Netzer et al. (201)) and C0 (Krizhevsky) 2009). We use 20 He et al. 2016 implemented in PyTorch (Paszke et al.) 2017]. initialized following He et al. 2015) and trained using stochastic gradient descent with batch size of 100 for 100 or 200 epochs (MNIST and SVHN/Cifar10, respectively). As detailed in Sec. D we report ROC AUC (higher is better) as well as confidence-thresholdee test error (Err; lower is better) and robust test error (RErr; lower is better); Err is reported on the full test sets, while ROC AUC and RErr are reported on the first 1000 atlacked test examples. More details and experimental results can be found in Appendix B

---

Under review as a conference paper at ICLR 2020
0.6
0.6
Figure 2: Momentum and back- tracking, Our L PGD-Conf attack, of Sec. [. with 40 iterations with momentum and backtracking (left) and without both (right). We plot the ob- jective of Eq. 4 over iterations for 10 samples (different colors).
Error
0.4
Erro
0.4
0.2
30
40
0
10
20
30
Itrration
Iteration
Attacks on Different Test Examples
4.1 ATTACKS
Whx and adaptive attacks: We follow (Madry et al. 2018) and use projected gradient descent (PGD) to minimize the negatives of Eq. (2) and (4); we denote them as PGD-CE and PGD-Conf. The perturbation 8 is initialized uniformly over direction and distance; for PGD-Conf, we additionally use 8 = as initialization. Different from (Madry et al.] 2018), we run exactly T iterations (no early stopping) and take the perturbation corresponding to the best objective of the I iterations. In addition to momentum, as (Dong et al.] 2018), we propose to use an adaptive learning rate in combination with backtracking scheme to improve the attacks: after each iteration, the computed update is only applied if it improves the objective; otherwise the learning rate is reduced. We also implemented PGD for L2. L1 and Lo attacks; note that for each Lip, P E {00, 2, 1, 1,0). PGD-Conf has been explicitly designed to attack our CCAT. We train on Los attacks using T = 40 iterations with 0.3 (MNIST) or 0.03 (SVHN/Cifar For evaluation, 10 random retries for PGD-Conf: T 200 with 50 random retries for PGD-CE. For L2, L, and Lo atlacks, we
use T 2000 iterations and
3, 10, 15 (MNIST) or 1, 7,85, 10 (SVHN/CCifarOO, respectively, partly following Tramer & Boneh 2019). Learning rates have been tuned for each by and objective (i.e., CE or Conf) independently.
Black-box attacks: For the Lix case, we additionally use random sampling, the attack by Ilyas et al. (2018)), adapted with momentum and backtracking optimizing Eq. (4) for T 2000 iterations with 10 attempts, a variant of (Narodytska & Kasiviswanathan (017) with T = 2000 iterations and the "cube" attack (Andriushchenko) 2019) with T 5000 iterations. We also consider the cube attack in the L2 setting and the atlack of (Cruce & Hein 2019) in the Lo selling. The black-box attacks ensure that our defense avoids, e.g., gradient masking as described in (Athalye et al. 2018a).
4.2 WORST-CASE EVALUATION METHODOLOGY
Instead of reporting robustness against individual attacks, as commonly done in the literature, we use a worst-case evaluation scheme: For each individual test example, all adversarial examples from multiple attempts (i.e., multiple random restarts) and across all (white- and black-box) attacks are accumulated. Subsequently, per text example, only the adversarial example with highest confi- dence is kept; these are the worst-case adversarial examples for each test example. practice this scheme allows to accumulate results from different attacks and thus corresponds to much stronger robustness evaluation than usual.
SVHN: Attack Ablation with RErr for 7 7P9%TTP (Los attack with e = 0.03)
Optimization
momentum+backtradd
mom
Initialization
zero
rand
Iterations T
200
2000
4000
4000
300
300
AT
38.4
46.2
49.9
50.1
51.8
38.1
30.8
AT Conf
27.4
40.5
46.9
47.3
48.1
28.5
23.8
CCAT
4.0
5.0
22.8
23.3
5.2
2.6
2.6
Table 1: Attack ablation study on SVHN. Comparison of our Log PGD-Conf attack with 0.03 on the test set for different number of iterations T and configurations of momentum, backtracking and initialization. As backtracking needs additional forward pass per iteration, we compare = 200 with backtracking to T = 300 without. Attacks on adversarial training (AT) succeed within few iterations, but more difficult against CCAT and require initialization zero
7

---

Under review as a conference paper at ICLR 2020
Tree Podtive Taste (TPR)
Robast Teer Beree (RFRr)
0.8
81
AT
0.8
0.0
DCCT, Plww
10
0.85
0.98
0.4
0.6
00.03
0.2
0.2
0.4
Confideace
0.2
Test Confidenace
0.1
0.5
0.8
0.2
CCAT. 10
0.5
0.8
0.2
11:
0.3
Falan Powition (FPR)
Confidense Threshold
Narmal
AT
CCAT, Paon 10
0.9
011
Figure 4: ROC and RErr curves on SVHN. Left: ROC curves, i.c., FPR against TPR when distinguishing correctly classified test examples from successful adversarial examples by confi- dence. Right: RErr against confidence threshold т. For evaluation, we choose T in order to obtain 99%TPR. As described in the text, Rßr sub- sumes both Err and FPR. Curves based on worst- case examples across all tested Lo attacks.
01
0.2 0.1 0.6 0.55
0
0.2
0.4
0.5 0.5
Adversacial Confidence
Adversarial Confidence
Figure 3: Confidence histograms on SVHN.
For AT (left) and CCAT (right) with Ppew = 10 (right), we show confidence histograms corre- sponding to correctly classified test examples (top) and successful adversarial examples (bot- tom). We consider the worst-case adversarial ex- amples across all tested Lo attacks for e 0.03.
4.3 ABLATION STUDY
Momentum and Backtracking: Fig. 2 illustrates the advantage of momentum and the proposed backtracking scheme for PGD-Conf with T' = 40 iterations on 10 test examples of SVHN, As shown in Fig. 2 and Tab. D better objective values can be achieved within fewer iterations and avoiding oscillation which is important at training time. However, also at test time, Tab. T shows that attacking our CCAT model effectively requires up to T = 2000 iterations and zero initialization so that REr for 7 @Q99TPP stagnates. In contrast, PGD-Conf performs better against AT even for smaller T and without momentum or backtracking. Thus, finding high-confidence adversarial examples against CCAT is more difficuult than for AT. Overall, this illustrates immense effort put into attacking our proposed defense with an adapted attack, novel optimization techniques, and large number of iterations.
Evaluation Metrics: Fig. shows ROC and RErr curves on SVHN, considering AT and CCAT with Ppow 10, i.e., using the power transition from Eq. (6). The ROC curves, and the corresponding AUC value, quantify how well (successful) adversarial examples can be detected, i.e., distinguished from (correctly classified) test examples. Note our conservative choice to use the confidence thresh- old 7 at 99%TPR, wrongly rejecting at most 1% correctly classified examples. We note that T depends only on correctly classified test examples, not adversarial examples, and REл implicitly includes FPR as well as Err. We focus on the confidence thresholding will naturally also improve results for AT, resulting in a fair comparison, i.e, AT is also allowed to benefit from our two-stage detection setting: for Err this can be seen in Tab. 2 On SVHN and Cifarl0, we found the power transition with Prow 10 from Eq. (6) to work best. Up to Ppow 10, performance regarding RErr for @ 99%% TPR continuously improves and after (Poow = 10 performance stagnates, as shown in detail the appendix. On MNIST, interest- ingly, exponential transition with Pexp performs best against Loe attacks; assume that the
variants of RErr and Err as
MNIST: Test Error
SVHN: Test Error
SVHN: Test Error
Standard
Detection @99%TPP
Standard == 0
Detection T999%TPR
Standard
Detection
0
0
T T99%%TP
Em
Err
Er
Err
Em
Err
in %
in %
in %
in %
in %
in %
Normal
0.4
0.1
3.6
2.6
8.3
7.4
AT
0.5
0
3.4
2.5
16.6
15.5
CCAT
0.3 (0.3)
0.1 (0.1)
2.9
2.1
10.1
6.7
Table Test errors on MNIST, SVHN and Cifar10. Err forforr = O, iee. standard evaluation, and @ @99%IPR i.e., detection evaluation, comparing normal training, AT and CCAT. Especially Cifar10, AT increases Err significantly, in both settings. CCAT, in contrast, is able to preserve the Err of the normally trained model better.
8

---

Under review as a conference paper at ICLR 2020
MNIST:
Main Results for Detection T099%TPR
(Topo 6 0.30 during training, standard evaluation, i.c., T O, Appendix
Loc, c 0.30
Loc, C
0.40
L2,
Li. 10.00
Lou 15
(seen)
(unseen)
(unseen)
(unseen)
(unseen)
ROC AUC
REM
ROC
RET
ROC
REm
ROC
REm
ROC
REm
in %
AUC
in %
AUC
in %
AUC
in %
AUC
in %
AT
0.97
1.0
0.20
100.0
0.73
81.3
0.93
5.2
0.99
2.5
CCAT, Pesp =7
0.99
7.7
0.94
40.0
1.00
1.4
0.96
14.7
0.99
7.8
CCAT, Ppos =10 10
0.96
21.9
0.94
29.0
1.00
0.1
1.00
1.6
0.99
7.8
SVHN: Main Results for Detection @ @99%TPR
(Loo, c 0.03 during training, standard evaluation, i.c., r
0, in Appendix
L c 0.03
Loos = 0.06
La, = 1
L1. 7.85
Lo. 10
(scen)
(unseen)
(unseen)
(unscen)
(unscen)
ROC
ROC
RErr
ROC
REr
ROC
RErr
ROC
REr
AUC
in %
AUC
AUC
in %
AUC
in
AUC
in %
AT
0.55
55.6
0.32
88.3
0.26
92.0
0.34
91.9
0.90
73.4
CCAT
0.70
38.5
0.70
46.0
0.91
18.5
0.89
20.8
1.00
2.7
CIFARIO: Main Results for Detection @99%TTP
(Lor é 0.03 during training, standard evaluation, i.c., r O. Appendix
Loo, c 0.03
Looo 0.06
L2.
Li. c 7.85
Lo.
(scen)
(unseen)
(unseen)
(unscen)
(unscen)
ROC
ROC
REm
ROC
REm
ROC
RErr
ROC
REr
AUC
in %
AUC
AUC
in %
AUC
AUC
in %
AT
0.64
62.3
0.35
93.6
0.59
73.9
0.61
68.0
0.80
74.1
CCAT
0.60
67.9
0.43
91.5
0.77
46.2
0.78
45.2
0.98
20.9
Table 3: Main results MNIST, SVHN and Cifar10. Comparison of AT and CCAT on MNIST (top); SVHN (middle) and Cifar10 (bottom). Per threat model, i.e., for Lox, L22, In and Lo at- tacks, report worst-case results across all tested attacks; the used values are reported in the corresponding columns. During training, Los attacks with SVHš/Cifar10 were used; adversarial examples from the remaining threat models were not en- countered during training. We report ROC AUC as well as confidence-threshen @99%TIPR. CCAT is competitive with AT on the Loe attacks seen during training, but generalizes significantly better to previously unseen attacks.
0.3 on MNIST and 0.03 on
Err and RErrfor
slight bias towards the true label preserved in the exponential transition helps. However, even MNIST, the power transition improves robustness against previously unseen attacks, e.g., L2. L1 or Lo attacks, such that we include results for both.
4.4 RESULTS
In Tab.3 we report the main results of our paper, namely robustness across all evaluated Lee, Lg, L and Tag attacks; for T. include results for the used during training and increased While on MNIST, CCAT incurs a drop of roughly 6% in RErr, and on Cifar a drop of roughly 5%, both against Loo with the same e as during training, it significantly outperforms AT on SVHN, by more than 16%. On SVHN and Cifar10, Err is additionally improved on Cifarl0, the improvement is particularly significant with roughly 6%. For L attacks with larger e, robustness of AT degrades significantly, while CCAT is able to preserve robustness to some extent, especially on SVHN. Only on 10, iΕm degrades similarly models, i.e., Iz2, In and Lo attacks, CCAT outperforms AT significantly. We note that on all datasets, the considered e-balls for Toz, In and Lo are not contained the Thsl-ball used during training. Overall. robustness of AT degrades significantly under threat models not seen during training, while CCAT generalizes to new threat models much better.
AT. In terms of generalization to previously unseen threat
In Tab. 4 we also report results for detecting uniform noise and adversarial uniform noise (i.e., distal adversarial examples) based on their confidence. For the latter, we sample uniform noise and subse- quently use PGD-Conf to maximize the confidence (without considering any true label in Eq. (4) in the L around the noise point. Although ROC AUC values are high on uniform noise, an FPR of 40% or higher shows that normal training and AT assigns high confidence uniform noise. When

---

Under review as a conference paper at ICLR 2020
MNIST (C 0.3)
SVHN (c 0.03)
CIFARIO (c 0.03)
Detection T@99%TPR
Detection
Detection
=@99%TPR
+@99%TPM
ROC
FPR
ROC
FPR
ROC
FPR
AUC
in %
AUC
in %
AUC
in %
Rand
Normal
0.34
100.0
0.95
72.8
0.83
87.1
AT'
0.99
44.5
1.00
0.9
0.60
100.0
CCAT
1.00
0.0
1.00
0.0
1.00
0.0
Normal
0.34
100.0
0,43
100,0
0.49
100.0
Dist
AT
0.63
100.0
0.89
98.8
0.31
100.0
CCAT
1.00
0.0
1.00
0.0
1.00
0.0
Table 4: Results for noise and distal adversarial examples on MNIST, SVHN and C0far10.
Robustness against uniform noise ("Rand") and distal adversarial examples ("Dist"), i.c., high- confidence adversarial computed on uniform noise using our Loc PGD-Conf attack by considering Eq. 4 without any true label;
0.3 on MNIST, c = 0.03 SVHN/Cifar10. We report ROC AUC and FPR for a confidence threshold of 7 @ 99%9TP.. In contrast to normal training and AT, CCAT is able to reliably detect all (adversarial) noise examples.
maximizing confidence on uniform noise, FPR approaches 100% on all datasets. In contrast CCAT allows to separate these attacks perfectly from test examples. Furthermore, Tab. presents results on MNIST-C and Cifarl0-C (Mu & Gilmer 2019 Hendrycks & Dietterich 2019, containing the MNIST and Cifarl0 test examples with various corruptions (e.g., spatial transformations, bright- ness/contrast changes, noise, etc.). Again, we consider our two-stage detection approach, reporting ROC AUC for detecting corrupted examples based on confidence and confidence confidence-hrrssholde Err on the corrupted examples. The results are averaged over all available corruptions (15 on MNIST, 19 on Cifar10), As can be seen, allowing confidence thresholding is beneficial in this scenario for AT and CCAT. However, CCAT, outperforms normal training and AT significantly.
5 CONCLUSION
We proposed confidence-callibraeedd adversarial training (CCAT) which addresses two problems of standard adversarial training: an apparent accuracysrobustnes problem, i.e., adversarial training tends to worsen accuracy on difficult datasets, and, more importantly, the lack of "generalizable" robustness, i.e., obtaining robust models against threat models not encountered during training. CCAT achieves comparable or better robustness against the threat model at training time, i.e., Loo- constrained adversarial examples, with better accuracy Cifar10. However, in strong contrast to adversarial training, CCAT generalizes to stronger Los attacks as well as L2, L1, Lo attacks and distal adversarial examples by allowing detection based on iments, we further introduced a generalized robust test error adapted to our detection-setting and used a worst-case evaluation methodology, both of independent interest the community.
For our exper-
Corrupted MNIST:
Corrupted CIFAR10:
Detection @99%TPR
Detection C TPR
(training: Loc. 6 0.30)
(training: Lox. c 0.03)
Average Results over 15 Corruptions
Average Results over 15 Corruptions
ROC
Er
ROC
Err
AUC
in
AUC
Normal
0.75
32.8
0.57
12.3
AT
0.80
12.6
0.53
16.2
CCAT
0.95
4.0
0.66
8.5
Table 5: Results on Corrupted MNIST and Cifar10. Performance corrupted examples (spatial
transformations, brighuness/cottrust changes, noise etc.) We report ROC AUC and confidence- thresholded Err on corrupted examples for @999%IPR. Compared to normal training and AT, CCAT detects better corrupted examples (higher ROC AUC) and its test error Err on non-rejected examples is lower.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. arXiv.org, abs/1801.00553, 2018.
Rima Alaifari, Giovanni S. Alberti, and Tandri Gauksson. Adef: an iterative algorithm construct adversarial deformations. arXiv.org, abs/ 804.07729, 2018.
Laurent Amsaleg, James Bailey, Dominique Barbe, Sarah M. Erfani, Michael E. Houle, Vinh Nguyen, and Milos Radovanovic. The vulnerability of learning adversarial perturbation in- creases with intrinsic dimensionality. In WIFS, 2017.
Maksym Andriushchenko. Provable adversarial defenses for boosting. Master's thesis, Saarland University, August 2019.
Anish Athalye and Nicholas Carlini. On the robustness of the CVPR 2018 white-box adversarial example defenses. arXivorr,, abs/1804.03286, 2018.
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv.org, 802.00420, 2018a.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In ICML, pp. 284-293, 2018b.
Marco Barreno, Blaine Nelson, Russell Sears, Anthony D. Joseph, and J. D. Tygar. Can machine learning be secure? In AsiaCCS, 2006.
Arjun Nitin Bhagoji. Daniel Cullina, and Prateek Mittal. Dimensionality reduction defense against evasion attacks on machine learning classifiers. arXiv.org, abs/1704.02654, 2017a.
Arjun Nitin Bhagoji, Warren He, Bo Li, and Dawn Song. Exploring the space black-box attacks on deep neural networks. arXiv.org, abs/1712.09433 2017b.
Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. arXiv.org abs/1 0bss/711..00 112.0314,, 2018a.
Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 84:317-331 2018b.
Wieland Brendel and Matthias Bethge. Comment "biologically inspired protection of deep net- works from adversarial attacks". arXiv.org, abs/11004.1157, 2017.
Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv.org, abbs/1712.0966 2017.
Qi-Zhi Cai, Chang Liu, and Dawn Song. Curriculum adversarial training. In IJCAI, pp. 3740-3743 2018.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. arXiv.org, abs/1705.07263, 2017a.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In SP, 2017b.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi. Unlabeled data improves adversarial robustness. arXiv.org, 13511905.13736. 2019.
Pin-V Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. ZOO: Zeroth order opti- mization based black-box attacks to deep neural networks without training substitute models. In AlSec, 2017.
Francesco Croce and Matthias Hein. Sparse and imperceivable adversarial attacks. abs/1909.05040,, 2019.
arXiv.org.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In CVPR, 2018.

---

Under review as a conference paper at ICLR 2020
John C. Duchi, Shai Shalev-Shwarte Yoram Singer, and Tushar Chandra. Efficient projections onto the / --all for learning in high dimensions. ICML, 2008
Beranger Dumont, Simona Maggio, and Pablo Montalvo. Robustness of rotation-equivarian net-
works adversarial perturbations. arXiv.org, abs/ abs/1802.06627, 2018.
Logan Engstrom, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. rotation and translation suffice: Fooling CNNs with simple transformations, arXiv.org, abs/1712.00279) 2017.
Logan Engstrom. Brandon Tran, Dimitris Tsipras. Ludwig Schmidt, and Aleksander Madry. Ex- ploring the landscape of spatial robustness. ICMI., 2019.
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts, arXiv.org, abs/1703.00411 2017.
Zhitao Gong, Wenlu Wang, and Wei-Shinn Ku. Adversarial and clean data are not twins. arXiv.org, abs/1704.04960, 2017.
lan Goodfellow, Yao Qin, and David Berthelot. Evaluation methodology for attacks against confidence thresholding models, 2019. URL https:://openrevi.ew fer.meomfffiidd
H1ç0piA9tQ
Ian J Goodffllow, Jonathon Shlens. and Christian Szegedy. Explaining and harnessing adversarial examples. arXivorg, (os..5422667 2014.
Edward Grefenstette, Robert Stanforth, Brendan O'Donoghue, Jonathan Uesato, Grzegorz Swirszez, and Pushmeet Kohli. Strength in numbers: Trading-off robustness and computation via adversarially-trained ensembles. arXiv.org, abs/1811.09368 2018
Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick McDaniel. On the (statistical) detection of adversarial examples. arXiv.org, abs/ abs/1702.00280, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing huvan-level performance on imagenet classification. In ICCV, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. CVPR, 2019.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv.org, abs/1903. 12261, 2019.
Dan Hendrycks and Kevin Gimpel. Early methods for detecting adversarial images. In ICLR, 2017.
Ruitong Huang. Bing Xu, Dale Schuurmans, and Csaba Szepesvári. Learning with strong adver- sary. arXiv.org, abs/151 103034, 2015.
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In ICML, 2018a.
Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Black-box adversarial attacks with bandits and priors. arXiv.org, hbbs/1807.07978 2018b.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In /CML, 2015.
Jörn-Henrik Jacobsen, Jens Behrmann, Nicholas Carlini, Florian Tramèr, and Nicolas Papernot. Exploiting excessive invariance caused by norm-bounded adversarial robustness. arXiv.org, abs/ 0bs//193..14844 2019a.
Jörn-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invari- causes adversarial vulnerability. In ICLR, 2019b.
12

---

Under review as a conference paper at ICLR 2020
Daniel Kang, Yi Sun, Dan Hendrycks, Tom Brown, and Jacob Steinhardt. Testing robustness against unforeseen adversaries. arXiv.org, hbss/190..0006,, 2019.
Danny Karmon, Daniel Zoran, and Yoav Goldberg. Lavan: Localized and visible adversarial noise. In ICML, 2018.
J. Zico Kolter and Eric Wong. Provable defenses against adversarial examples via the convex outer adversarial polytope. arXiv.org,
2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv.org. abs/1607 002533.0.033,.... 2016.
Alex Lamb, Vikas Verma, Juho Kannala, and Yoshua Bengio. Interpolated adversarial train- ing: Achieving robust neural networks without sacrificing too much accuracy. arXiv.org, abs/1906.067888 06784, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. of the IEEE, 86(11):278-2324, 1998.
Hyeungill Lee, Sungyeob Han, and Jungwoo Lee. Generative adversarial trainer: Defense to adver- sarial perturbations with GAN, arXiv.org, abs/1705.03387, 2017.
Mark Lee and Zico Kolter. On physical adversarial patches for object detection. arXiv.org, abs/1906.1 11897, 2019.
Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin. On norm-agnostic robustness of ad- versarial training. arXiv.org, Abs/1905.066 06455, 2019a.
Juncheng Li, Frank R. Schmidt, and J. Zico Kolter. Adversarial camera stickers: A physical camera- based attack deep learning systems. In ICML, 2019b.
Xin Li and Fuxin Li. Adversarial examples detection in deep networks with convolutional filter statistics. In ICCV, pp. 5775-5783, 2017.
Fangzhou Liao, Ming Liang. Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In CVPR, 2018.
Xin Liu, Huanrui Yang, Linghao Song, Hai Li, and Yiran Chen. Dpatch: Attacking object detectors with adversarial patches. arXiv.org, ftss/806.00299, 2018.
Xuanqing Liu, Yao Li, Chongruo Wu, and Cho-Jui Hsieh. Adv-bnn: Improved adversarial defense
through robust bayesian neural network. In ICLR, 2019.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial exam- ples and black-box attacks. arXiv.org, abs/161 161 1. .02770, 2016.
Xingjun Ma, Bo Li. Yisen Wang adn Sarah M. Erfani, Sudanthi Wijewickrema, Michael E. Houle, Grant Schoenebeck. Dawn Song, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. arXiv.org, 00/118001.211111 2018
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. arXiv.org, 00s/1706.06083, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ICLR, 2018.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXivoorg, abs/1702.00266) 2017.
Takeru Miyato, Shin-ichi Maeda, Masamori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing with virtual adversarial training. ICLR, 2016.
13

---

Under review as a conference paper at ICLR 2020
Norman Mu and Justing Gilmer. Mnist-c: A robustness benchmark for computer vision. ICML Workshops, 2019.
Nina Narodytska and Shiva Prasad Kasiviswanathan. Simple black-box adversarial attacks on deep neural networks. CVPR Workshops, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. NeurIPS, 2011.
Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection of adversarial exam- ples. In NeurlPS, pp. 4584 4594, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NeurlPS Workshops, 2017.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duchesnay. Scikit-learn: Machine learning in Python. JMLR, 12:2825-2830, 2011.
Julien Pérolat, Mateusz Malinowski, Bilal Piot, and Olivier Pietquin. Playing the game of universal adversarial perturbations. CoRR, abs/1809.07800 2018.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang. John C. Duchi, and Percy Liang. Adversarial training hurt generalization. arXiv.org, abs//196.00003,, 2019.
Bita Darvish Rouhani, Mohammad Samragh, Tara Javidi, and Farinaz Koushanfar. Towards safe deep learning: Unsupervised defense against generic adversarial attacks, 2018. URL https:
Sayantan Sarkar, Ankan Bansal, Upal Mahbub, and Rama Chellappa. UPSET and ANGRI Break- ing high performance image classifiers. arXiv.org, abs/1707 159, 2017.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Ad- versarially robust generalization requires more data. CoRR, arXiv.org, 2018
Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on MNIST. In ICLR, 2019.
Ali Shafahi, Mahyar Najibi, Zheng Xu, John P. Dickerson, Larry S. Davis, and Tom Goldstein. Universal adversarial training. arXiv.org, abs/181 1304, 2018
Uri Shaham, Yutaro Yamada, and Sahand Negahban. Understanding adversarial training: Increasing local stability of supervised models through robust optimization. Nearocomputing. 307:195-204, 2018.
Yash Sharma and Pin- Yu Chen. Attacking the madry defense model with 1l-based adversarial ex- amples. arXiv.org, abs/1710. 10733, 2017.
Aman Sinha, Hongseok Namkoong, and John C. Duchi. Certifiable distributional robustness with principled adversarial training. ICLR, 2018
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Generative adversarial examples. arXiv.org, htt//88550789./ 2018
David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and general- ization. CVPR, 2019.
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. arXiv.org, abs/1710.088644, 2017.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv.org,
2013.
14

---

Under review as a conference paper at ICLR 2020
Florian Tramer and Dan Boneh. Adversarial training and robustness for multiple perturbations. arXiv.org, abs/1904.13000, 2019.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Dan Boneh, and Patrick D. McDaniel. Ensemble adversarial training: Attacks and defenses. arXiv.org, 00s/1705507204, 2017.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv.org. ab2/1805.12152, 2018.
Jonathan Uesato. Jean-Baptisse Alayrac, Po-Sen Huang, Robert Stanforth, Alhussein Fawzi, and Pushmeet Kohli. Are labels required for improving adversarial robustness? arXiv.org,
abs/1905.13725, 2019.
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In ICML, 2019.
Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song, Spatially trans- formed adversarial examples. arXiv.org, abs/1801 .02612, 2018.
Cihang Xie, Zhishuai Zhang, Jianyu Wang, Yuyin Zhou, Zhou Ren, and Alan L. Yuille. Improving transferability of adversarial examples with input diversity. arXiv.org, abs/1803.06975 2018.
Nanyang Ye and Zhanxing Zhu. Bayesian adversarial learning. In NeurlPS, 2018.
Xiaoyong Yuan, Pan He, Qile Zhu, Rajendra Rana Bhat, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. arXiv.org, abs/1711.07 2017.
Michal Zajac, Konrad Zolna, Negar Rostamzadeh, and Pedro O. Pinheiro. Adversarial framing for image and video classification. In AAAI Workshops, 2019.
Valentina Zantedeschi, Maria-Irina Nicolae, and Ambrish Rawat. Efficient defenses against adver- sarial attacks. In AlSec, 2017.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
Xiao Zhang and David Evans. Cost-sensitive robustness against adversarial examples. arXiv.org, abs/1 0bbs/1810.00 00l//1800.0925, 2018
15

---

Under review as a conference paper at ICLR 2020
A PROOF OF PROPOSITION 1
Proposition 2. We consider classification problem with two points = 0 and in I with deterministic labels, that is p(y = 2/ = 0) and p(y = 1|x = and the problem is fully determined by the probability Po p(x = 0) p( - =
Pu- The Bayes error of this classification problem is zero. The predicted probability distribution over the classes is p(ylz) =
e e91 4e92lny) where g R R2 and assume that the function À .R++ [0, 1] is monotonically
decreasing and X(0) = The Bayes optimal classifier for the cross-entropy loss of
udversarial training on 100% adversarial examples, cf. Eq. yields an error of min[po, po}-
adversarial training with 50% adversarial and 50% clean examples per batch, cf. Eq. 3 yields an error of min{po: -P))
CCAT on 50% clean and 50% adversarial examples, cf. Alg. yields zero error if X(e)
<
min
1-
T-pa
P0
dealing with three different probability distributions the labels: the true one p(y(æ)) the imposed one during training p(y(z) and the predicted one p(y)z)) We also note that p depends on X through Eq. 5 and X itself a function the norm ||81| through Eq. 16 here, this dependence is made explicit by writing (()) This makes the expressions for the expected loss of CCAT slightly more complicated. We first derive the Bayes optimal classifier and its loss for CCAT. We introduce
Proof. First, we stress that
g1 (0) 92()), b= - 92(e).
(8)
and express the logarithm of the predicted probabilities (confidences) of class 1 and 2 in terms of these quantities.
(a)
llg (1 ea) ( log (1 + e (1)-
ifr = 0 if = é
- -log = 2/xr- = =x)= -log
(0) (w)
[log(1 + eb)
log 1|x x) log
e
(x)
log (1 (0
flog(1 4 log (1+e
if 0 if =
(0)
e92(1))
For our confidence-calibratee adversarial training one first has to solve
i2() argmax max k + 6).
5 ktj
We get with the expressions above (note that we have binary classification problem):
0 ifa >
(1() = argmax ply = 2
else.
11611
{
ifa > b else.
(2) = argmax P(y = 1)
= Se
ifa >>
(1) argmax ply = 2/c 00)
0
else.
1 Se
if
8 (2) argmax p0y= = 1
Se
-c
else.
Note that in CCAT for 50% of the batch we use the standard cross-entropy loss and for the other 50% we use
2
L (b, (110((1)))
- =
j bog (p(y
+00))
16

---

Under review as a conference paper at ICLR 2020
The corresponding expected loss is then given by
E (1t_0///(oolllaaao00000aaaaaaa11
=p(x= = O)ı (1,,,,(((0((6666000000)000000000 0) + + = T)R (.p & (()
where = = Pa and =) == 1 With the true conditional probabilities
ply = m we ger
-
2 = Σ py==s x)
For our problem it holds with ply = 2| plu 11 1-==0 = 1 (by assumption). Thus,
=+) = Σ = -0-0000000000000000-00>>>----0 (100
2
[[4,,,/A(00((,,,]+]))====-- = - Σ
log (plv = = -
3=1
As [y)]l is either 0 or ||e|| and x(0) we in the following for simplicity the notation
X([|e]). Moreover, note that
(+ (1-)) (1-1)
ify J.
Pu(A)o= -
else
where K is the number of classes. Thus, K = 2 in our example and we note that X (->) 14A
With this we can write the total loss (remember that we have half normal cross-entropy loss and half the loss for the adversarial part with the modified "labels") as
(1 + A)
(1- A) log(1 + + +
I(a, b) Poo log(1 + e 1l)2 + lacb
log(1 + +
2
2
(1 + 1)
log(1 e +ea) (1- A) log(1 1 + e0)]
+ (1 - pp) log(1 + + "0-) + lasb
2
2
+ log(1 + e log(1 e +==))(1- Po).
where we have omitted a global factor 2 for better readability (the last row is the cross-entropy loss). We distinguish two sets in the optimization. First consider the case a 2 b. Then it is easy to that in order to minimize the loss we have a = b.
e
O. = 2
1t e Pa
I-e-a po)
This yields en = L-Pe or a = log I-po and the minimum for a 2 is attained on the boundary
of the domain of a < b. The other case is a S b. We get
0,L + X)
(1 (1-1 A)
6
e
po) (1op
(1 (1-
+ Po
2
1
2
1 ea
+ e
(1 A)
c
(1 (1-1) X)
h
0,4
+
Po + (1 po)
2
c
1 1 b
+ b
This yields the solution
1+1 (1 po)
1-1 + (1 - po) Po
a- = llog
ba = log
Pn + 1 (1 - poo)
1+
2
Po
17

---

Under review as a conference paper at ICLR 2020
It is straightforward to check that a" b" for all Po < 1, indeed we have
1 (1- 2 Po + (1
1+1
2 Pu) - Po
po)
1- 2 Pu + (1 - po)
po
Po 2
2
2ppo pd
if <. Po < and note that À < 1 by assumption. We have a < and thus g2(0) > 91 (0) (Bayes optimal decision for x 0) if
1 Po
X,
Po
and b* > 0 and thus gı(c) > g2(c) (Bayes optimal decision for x if
Po X.
Po
Thus we recover the Bayes classifier if
Po
Po
<
min
Po
Po
Now we consider the approach by Madry et loss can be written as
2018 with 100% adversarial training. The expected
max =0 Lyy, f(x 8) E 6 f(x
=p(x 0)p(y = 2/2 x) max log (p(y 2 0)) - log (ply 2/z = e))}
+ (1 - p(z = 0) p(y = 1|2 = z) max {- log (p(y = 1|2 = 0 (ply = 1|2 = }))
=p(x = 0) max { log (p(y = 2/ = 0)), - log (p(y = 2/x= = ={))
+ +1 p(z = 0)) max - log (ōly = 1| = 0)), - -log (ply 1|1 $)}
This yields in terms of the parameters a, the expected loss:
L(a, b) = max log(1 + )²), log(1 + e²) po + max log(1 + tc= + 1o +--)](1- po)
The expected loss is minimized if a b as then both maxima are minimal. This results in the expected loss
L(a) = loog11111 + et Po + log(1 + e e)()(1 - Po).
The critical point is attained at a" = 6* = log
This is never Bayes optimal for all 0 < Po <
Po
Note that b" =a > 0 if pa < po
- po and thus the error is given by po, similar a* b² < 0 if
1- Po < po and thus the error is given by 1 po- We get an error of min 1 po} of the Bayes optimal solution of 100% adversarial training.
Next we consider 50% adversarial plus 50% clean training. The expected loss
E max L(, f(x S(x + 6)] + EE L(v, [E(LO, f(x ( ((x-))) 6)).
can be written
Llabb) max log(1 e²),, log(1 + }
+ max log(1 + ee
) }) - Pu)
+ +log(1 + en)) Po + log(1 + +e-+e++ )1 - -po)
We make a case distinction. a 2 h, then the loss reduces to
L(a, b) log(1 + e e + log(1 + te )1 pp)
+ log(1 + e" )) + log(1 + E -1 - Po)
> a(a,a)
2ogg(1 en )o + 2log + e - - po)
18

---

Under review as a conference paper at ICLR 2020
MNIST: Attack Ablation with RFm for TP9%
(Tocc attack with e 0.3)
Optimization Iniiiaiization
momentum banbnntum-bbckracck
momentum
zero
rand
zero
zero
Iterations T
40
200
2000
4000
4000
60
300
60
300
AT
0.4
0.4
0.4
0.3
0.6
0.4
0.6
0.4
0.4
Conf
0.8
0.8
1.0
1.0
1.0
CCAT
0.6
3.3
4.9
6.8
5.5
0.9
3.8
0.0
0.1
SVHN: Attack Ablation with RErr for @ @99%IPR
(Leb attack with 0.03)
Optimization
momentum+backtrace
momentum
Initialization
zero
rand
zero
zero
Iterations T
40
200
2000
4000
4000
60
300
60
300
AT
38.4
46.2
49.9
50.1
51.8
37.7
38.1
29.9
30.8
AT Conf
27.4
40.5
46.9
47.3
48.1
27.1
28.5
21.1
23.8
CCAT
4.0
5.0
22.8
23.3
5.2
2.6
2.6
2.6
2.6
Cifar10: Attack Ablation with REir for 7 99% TPR
(T attack with = 0.03)
Optimization
momentum + backtrack
momentum
Initialization
rand
Iterations 7'
200
2000
4000
4000
60
300
60
300
60.9
60.8
60.8
60.8
60.9
60.9
$7.4
57.6
AT Conf
60.4
60.6
60.5
60.5
60.9
60.4
60.6
56.2
56.6
CCAT
14.8
16.2
40.2
41.3
34.9
7.2
7.2
7.2
7.2
Table 6: Detailed attack ablation studies MNIST, SVHN and Cifar10. Complementary to
Tab. D we compare our Loe PGD-Conf, as introduced in Sec. 4.7 attack with T iterations and dif- ferent combinations of momentum, backtracking and initialization all three datasets. We consider AT, AT trained with PGD-Conf, and CCAT; we report REi for confidence threshold 7 @ IPR..PR. As backtracking requires additional forward pass per iteration, attacks without backtracking to be comparable to attacks with 40 and T = 200 with backtrack- ing.
T 60 and T 300 for
Solving for the critical point yields a" =log [--po) = b" Next we consider the set S b. This yields the loss
L(a, b) = log(1 + e² e0)po + log(1 + te (1 po)
+ log(1 + )') + log(1 + ") po)
P-Po b" which fulfills S b. Actually, it Solving for the critical point yields a' log coincides with the solution found already for 100% adversarial training. One does not recover the Bayes classifier for any 0 < Po < 1.
no
Moreover, note that
> 0
if Pa
b
7
if Pa
Thus, we classify = 0 correctly, if Pu by min(po. Po).
1 and x correctly if Po < 2. Thus the error is given
B EXPERIMENTS
We give additional details on experimental setup, specifically regarding attacks, training and the used evaluation metrics. Afterwards, we include additional experimental results, including ablation studies, results for 98% true positive rate (TPR), results per attack, results per corruption on MNIST. MMu Gilmer 2019) and CifarlO-C (Hendrycks Dietterich 2019) and comparison with
Pang et al. 2018
19

---

Under review as a conference paper at ICLR 2020
Algorithm 2 Pseudo-code for the used projected gradient descent (PGD) procedure to maximize
Eq. or Eq. (10) subject to constraints 2; I; + à; [0, 1] and |6]|,o 6; in practice,
the procedure is applied batches of imputs. The algorithm is also easily adapted work with L no-oor; only the projections on line and needs to adapted.
input: example I with label y/
input: number of iterations T
input:
learning rate y, momentum B, learning rate factor a input: initial 8!"), e.g., Eq. (II) or 8(0)
1: := {best objective achieved]
2: I + g(0) {best adversarial example]
3: g( := 0 (accumulated gradients}
4: for = 0....., do
5: (projection onto Lo e-ball and on [0, 1]:}
6: clip 0!t)) e-
7: clip F; (/ [0, 1]
8:
(forward and backward pass to get objective and gradient:)
9:
(10) = F(r+ 8(4),
10:
g == sign ((x + ,()),))
11:
(keep track of adversarial example resulting in best objective:}
12:
if (1) > v then
13:
== (*)
14:
I := x 8(1) end if
15:
16:
(iteration only meant to check whether last update improved objective:}
17:
if = T then
18:
break
end if
20:
fintegrate momentum term:
21:
(()) : (t-1) + -1- (01-8)gi (t)
("try" the update step and objective increases:
23:
g(t) := 5(t) (g
clip ((() to [-
24:
clip v, + 8(00 to [0.1]
25:
26:
((0) = F(r+ 7())) )(),))
27:
fonly keep the update if the objective increased: otherwise decrease learning rate:}
if ((t) (1)) then
8(++1) = 8(t)
29:
30:
else
31:
77=7% %/-
32:
end if
33:
end for
34: return 2,
B. ATTACKS
Complementary to the description of the projected gradient descent (PGD) attack by Madry et al. (208)) and our adapted attack, we provide a detailed algorithm in Alg. We note that the objective maximized in (Madry al. 2018) is
F(r+d,y) = [(f(x) 6
(9)
where denotes the cross-entropy loss, f(;:w) denotes the model and (a, y) an input-label pair from the test set. Our adapted attack, in contrast, maximizes
F(x + 8, y) maxk4y fa(x + w)
(10)
Note that the maximum over labels, i.e., maX is explicitly computed during optimization; this means that in contrast (Goodfellow et al. 2019), do not K targeted attack and subse-
20

---

Under review as a conference paper at ICLR 2020
MNIST (Loo attack with 0.3)
Cifarl0 (Loo attack with 0.03)
CCAT, Avp
0.
CCTT Apm
0.4
0.4
0.0
01
0.2
0.4
0.6
0.8
0
0.2
041
0.5
0.5
0.2
0.4
0.6
0.5
0
6.2
0.4
0.6
0.8
Test Confidence
Test ConSidence
Tos Confidence
Test Coufidence
0.4
AT
CU.T PPay
0.15
CCAT. Ppm - 10
0.1
0.2
0/06
0.05
0.2
0.8
0
0.2 0.4 0.6 088
0.2
0.4
0.5
08
0.8
Adbersarial Confideence
Advernatial Confideence
Advensarial Confudence
Advorsariul Confidence
Figure 5: Confidence histograms on MNIST and Cifarl0. As in Fig. we show histograms of confidences on correctly classified test examples (top) and on successful adversarial examples (bottom) for both AT and CCAT. Note that on AT, the number of successful adversarial examples is usually lower than on CCAT, i.e., reflects the RErr for T = 0 in Tab. for CCAT in contrast, nearly all adversarial examples are successful, while only a part has high confidence. Histograms obtained for the worst-case adversarial examples across all tested Tox attacks.
quently take the muximum-confideee one, where K is the number of classes. We denote these two variants as PGD-CE and PGD-Conf, respectively. Deviating from (Madry et al., 2018), we initial- ize 8 uniformly over directions and norm (instead of uniform initialization over the volume of the e-ball):
8
8 = ME
8' ~ A(,, - 10,
(1I)
where 5' is sampled from a standard Gaussian and u E [0. 1] from a uniform distribution. We also consider zero initialization, i.c., 8 O. For random initialization we always consider multiple attempts, 10 for PGD-Conf and 50 for PGD-CE, with zero initialization, we use only 1 attempt.
Both PGD-CE and PGD-Cont can also be applied using the L2, Lı and Lo norms following the description above. Then, gradient normalization, i.e., the signum in Line 70 in Alg. for the Loe norm, the projection, and the the initialization need to be adapted. For the L2 norm, the gradient is normalized by dividing by the Lz norm; for the Ta norm only the 1% largest values (in absolute terms) of the gradient are kept and normalized by their I, norm; and for the Lo norm, the gradient is normalized by the L, norm. Projection is easily implemented for the L22 norm, while we follow the algorithm of Duchi et al. (2008) for the L1 project; for the Lo projection, only the e-largest values are kept. Similarly, initialization for L2 and Li are simple by randomly choosing a direction (as in Eq. TT)) and then normalizing by their norm. For Lu, we randomly choose pixels with probability 2/3c/ (He((HDD) D) and set them uniformly random values E [0, 1). where H x W x D is the image size. In experiments, we found that tuning the learning rate for PGD with LA and Lo constraints (independent of the objective, i.e., Eq. (9) or Eq. (10) is much more difficult. Additionally, PGD using the Lo norm seems to get easily stuck in sub-optimal local optima.
Alg. also gives more details on the employed momentum and backtracking scheme. These two "tricks" add two additional hyper-parameters to the number of iterations T and the learning rate y, namely the momentum parameter 99 and the learning rate factor a. After each iteration, the computed update, already including the momentum term, is only applied if this improves the objective. This is checked through an additional forward pass. If not, the learning rate is divided by a, and the update is rejected. Alg. inccldes this scheme as an algorithm for an individual test example with label for brevity; however, extending it to work on batches, which is used in our paper, is straight- forward. In practice, for PGD-CE, with T 200 iterations, we use 3 0.9 and = 1.25; for PGD-Conf, with T' 2000 iterations, we use 3 = 0.9 and a = 1.1.
We also give more details the used black-box attacks. For random sampling, we apply Eq. [1] T = 5000 times in order to maximize Eq. (0)). We also implemented the black-box attack of II lya
et al. 2018a) using a population of 50 and variance of 0.1 for estimating the gradient in Line of
Alg. 2 a detailed algorithm is provided in (Ilyss et al.) 20188)). We use a learning rate of 0.001 (note
that the gradient is signed, as in (Madry et al. 2018)) and also integrated momentum with / = 0.9
21

---

Under review as a conference paper at ICLR 2020
MNIST (Loo attack with 0.3)
Cifarl0 (Loo attack with 0.03)
Trme Positive Rate (TPR)
True Poilivv Rate (IPR))
Relaux (REi))
0.8
0.5
0.5
0.8
0.0
0.0
0.6
0.6
0.4
0.4
0.4
0.4
0.2
Rebuue
0.2
0.2
(FPR)
Confidence Threshold
False Poeitive (FPR)
Coufiikanee Thresbold
Normal
AT
CCAT, Pexp
Normal
AT
CCAT, Ppow 10
Figure 6: ROC and RErr curves on MNIST and Cifar10. ROC curves, i.c. FPR plotted against
TPR for possible confidence thresholds т, and RErr curves, i.e., RErr over confidence threshold r for AT and CCAT, including different parameters. Worst-case adversarial examples across all Loo attacks were tested.
and backtracking as described in Alg. with a 1.1 and T 2000 iterations. We use zero and random initialization; the latter case we allow 10 random retries. For the simple black-box attack we follow the algorithmic description in (Narodylska & Kasiviswanathan (017) considering only axis-aligned perturbations size e per pixel. We run the attack for = 2000 iterations and allow 10 random retries. Finally, we use variant of the cube attack proposed in (Andriushchenko 2019). We run the attack for T 5000 iterations with a probability that, except for (lyys et al. 201 8a), these attacks are not gradient-buse based and do not approximate the gradient.
change 0.05. We emphasize
B.2 TRAINING
As described in Sec. 4 we follow the ResNet-20 architecture by He et al. (2016) implemented in PyTorch (Paszke et al. 2017). For training we use a batch size of 100 and train for 100 and 200 epochs on MNIST and SVHN/Cifar10, 10. respectively: this holds for normal training, adversarial training (AT) and confidence-calibrated adversarial training (CCAT). For the latter two, we PGD-CE and PGD Conf, respectively, for T = 40 iterations, momentum and backtracking (3 = 0.9, learning
of 0.05, 0.01 and 0.005 MNIST, SVHN and Cifar10. For PGD-Conf we use a learning rate of 0.05. For training, we use standard stochastic gradient descent, starting with learning rate of 0.1 on MNIST/SVHN and 0.075 on Cifar10. The learning rate is multiplied by 0.95 after each epoch. We do not use weight decay; but the network includes batch normalization (loffe Szegedy 2015). On SVHN and Cifar10, we use random cropping, random flipping (only Cifarl0) and contrast augmentation during training. We always train on 50% clean and 50% adversarial examples per batch, i.e., each batch contains both clean and adversarial examples which is important when using batch normalization.
1.5). For PGD-CE
B.3 EVALUATION METRICS
For reproducibility and complementing the discussion in the main paper, we describe the used eval- uation metrics and evaluation procedure in more detail. Adversarial examples are computed on the first 1000 examples of the test set; the used confidence threshold is computed on the last 1000 exam- ples of the test set; test errors are computed on all test examples minus the last 1000. As we consider multiple attacks, and some attacks allow multiple random attempts, we always consider the worst case adversarial example per test example and across all attacks/attempts; the worst-case is selected based confidence.
ROC AUC: compute ROC curves, and the area under the curve, i.c., ROC AUC, we define negatives successful adversarial examples (on correctly classified test examples) and positives as the corresponding correctly classified test examples. The ROC AUC as well as the curve itself can easily be calculated using (Pedregosa al. 2011). Practically, the generated curve could be used to directly estimate a threshold corresponding to pre-determined true positive rate (TPR). However, this requires interpolation; after trying several constant interpolation schemes, concluded that the
22

---

Under review as a conference paper at ICLR 2020
SVHN: AT with Lno PGD-Conf, 0.03
Cludlece
E/M
6:36
0.34
Piecatlation
Poetriatice
Portuzbation
Portechation
Pictuctation
Cantabon
2
1-4
Eitl
LUZ2
1030
0.02
Pertadbation Daterxe
Portarhation Datexee
Perterattin Dutcoo
Petrccation Didoow
Petrccation Distaner
SVHN: CCAT, Apow
10 with Loe PGD-Conf, 0.03
Corlibon
0.00
04
0.4
62
022
2M
3.08
5452
1.00
0.22
6.90
Nestallation Thelarce
Pestrbation Thstarce
Pertotation
Peturlation Biatoe
D'etturtation
Codibbaee
0.4
0.3
0.32
0.03
0.08
04:
6.22
2.01
6.06
Perssthatic
Pettcbation
Portzzbatin Dituzee
Cifarl0: AT with Loe PGD-Come 0.03
Pamarbatics
Persstatiin
Partebation
Pettecation
Pistuctttion
Cifar10: CCAT, Ppow
10 with Loo PGD-Conf. c 0.03
0.0
Codidenns
0.4
043
0136
0135
0124
Pocurhation
Porturhation
Pictuchation
Plctor Natioon
Classes
Figure 7: Effect of confidence calibration on SVHN and Cifar10. Confidences for classes along adversarial directions for AT and CCAT, Poow = 10. Adversarial examples were computed using PGD-Conf with T 2000 iterations and 0.001 and zero initialization. For both AT and CCAT, we show the first ten examples of the test set on SVHN, and the first five examples of the test set on Cifar10.
results are distorted significantly, especially for TPRs close to 100%. Thus, we followed a simpler scheme mentioned above: held out validation set of size 1000 (the last 1000 samples of the test set), we sorted the corresponding confidences, and picked the confidence threshold in order to obtain the desired TPR, e.g., 99%, on this exactly.
23

---

Under review as a conference paper at ICLR 2020
MNIST (PGD-Conf, Loo with 0.3)
y=3,5=a y= y=4.-3 y=d.aa y=a,g=2 2 y=T,f=1 y=2,5=2 y=3,9=a y=1.s11 y=4.-4 y=7-=7 y=
Je(3) 0.30 1.00 -,(3) (01)) (02)=55 0..5 f+03) 1.00 Jel)) e.st A(t) 40)-1.0 - ((3 -100-.00 418)- 1.0 (11))
9
AT
= y=4.j- y=4.ia y= y=2,f=1 y=s,g=a y=1.f=3 y=4.-a y=7.-4 y=9,5=4
Jx( 1200==1000 (2(9) 1.00 -00=11000 (al)) = f402= 0.54 fx00) 1.00 (x(2) =.53) (u12)
CCAT
4
4
3
4
SVHN (PGD-Conf, Lo with c = 0.03)
CCAT AT
5
5
2-49e6
2-8ge2
SHAS-2
p=land
x=1d=2
S
5
Cifar10 (PGD-Conf, Loo with 0.03)
CCAT AT
swApws 401-00
2=76-0
1,00-1000
1,0
Figure 8: Adversarial examples against AT and CCAT on MNIST, and Cifar10. LoD- Conf adversarial examples for MNIST, e = 0.3, and SVIIN/Cifar10, e = 0.03, for AT and CCAT. In all cases, twelve test examples have been attacked. On the first six examples, AT has difficulties, the last six examples, CCAT has difficulties. We also report the true label y. the target label ğ of the adversarial examples and the corresponding confidence fz(()..
Robust Test Error: For clarity, we repeat our confidence-integrated definition of the robust test error:
N 1 ((m))) vn f(an) 1r(āx)2r
Σ ((x))i L((a))2r + Σ
REm(()
==1
(12)
Σ 10()))2 + Σ
Tewwer ((())22 ((x)=vn f(a))uu
As described in the main paper, Eq. quantifies the performance of classifier with reject-option at a specific confidence threshold t on both clean and adversarial examples. The regular robust test also be written
error, corresponding Eq. 112 at
Σ 1 (a,))vn
Σ
1 1((=.) v. 1(5.))
REr
(13)
Σ
1
m=1
24

---

Under review as a conference paper at ICLR 2020
MNIST: AT with Loe PGD-Conf, 0.3
2
4 you
24 4
7
Golginge
264
y=
9= 1500.4
Interpodatinn FActoe I
fg 1-009995
Inmpptatt Fac
f==1
MNIST: CCAT, Pexp
= 7 with Los (GD-Conf. c = 0.3
2 2
24 4
-0.5
7
204
0.2
Interyolation PactOe
0.4
0.0
0.5
fy=1
5=8
fu=1
u=
Innppiatao Factic
4.5
3.2
fy=1
9=2 8-10.1
f=
MNIST: CCAT, Prpo
10 with Loo PGD-Conf, 0.3
Cuggang ger Slaw
2 2
2.2 4
7
20.4
Interpdation Factoe
1==1
D=2
9-4 f--0.11
Iroppiatoo
3.5
f=
y=2
fg=0.12
y=4 f=1
Classes
2,
3,
5.
6
8. 9.
10
Figure 9: Confidence calibration between test examples MNIST. We plot the confidence for all classes when interpolating linearly between test examples: (1 - K) + 632 for two test examples different x2. Additionally, we
In and V2i xn is fixed and we show two examples corresponding show the corresponding images for k = 0, i.e,, I1, K = 0.5, i.e., the mean image, and k = 1, i.e., I2. with the corresponding labels and confidences. As can be seen, CCAT with power transi- tion, cf. Eq. (6). is able to perfectly predict uniform distribution between test examples, while the exponential transition enforces a stronger bias towards the true labels.
The robust test error is easy handle it quantifies overall performance, including generalization on clean examples (first term in Eq. (13) and robustness on adversarial examples corresponding to correctly classified clean examples (second term in Eq. [33). Additionally, the robust test error lies in [o, 1]. Generalizing Eq. (13) to > 0 is non-trivial due to the following considerations: First, when integrating confidence threshold, the reference set (i.e., the denominator) needs to be adapted. Otherwise. the metric will not reflect the actual performance after thresholding, i.e., reject- ing examples- specifically. the values are not comparable across different confidence thresholds т. Then, Eq. (3) might be adapted as follows:
N
Σ
(on) 1 12(a))2r
Σ 1l (am)) 1 sun 1 oo(k))2r
REM(())
n=1
(14)
Σ 1o(z)12r
n=1
Note that in the nominator, we only consider clean and adversarial examples with confidence above the threshold i.e., c(x,,) > 7 and c(ār,) 2 Similarly, the denominator has been adapted ac- cordingly, as after rejection, the reference set changes, too. However, this formulation has a problem when adversarial examples obtain higher confidence than the original clean examples. Thus, second, we need to account for the case where c(ū.,) > + 2 e(x,)) i.e., an adversarial example obtains a higher confidence than the corresponding clean example. Then, the nominator may exceed the de- nominator, resulting in a value larger than one. To mitigate this problem, we need to include exactly this case in the denominator. Formalizing this case, we see that corresponds exactly the second term in the denominator or Eq. 1
N
Σ
( 4 f(aw)v
(15)
n=1
Overall, with Eq. 12 we obtain a metric that lies within [0, 1]. is comparable across thresholds r and, for reduces the regular robust test error used in related work (Madry al. 2018
25

---

Under review as a conference paper at ICLR 2020
MNIST: Main Results for Detection T99%%TPR
(T.ge, 6 0.30 during training
Loc, c
0.30
Loc, c 0.40
L2,
L1. 10.00
Lo,
15
(seen)
(unseen)
(unseen)
(unseen)
(unseen)
ROC AUC
REM
ROC
ROC
REM
ROC
REM
ROC
REm
in %
AUC
in %
AUC
in %
AUC
in %
AUC
in %
AT
0.97
0.5
0.20
100.0
0.73
67.6
0,93
3.7
0.99
0.8
CCAT, Pesp
0.99
5.6
0.94
29.2
1.00
0.6
0.96
10.9
0.99
S.1
CCAT, Ppos 10
0.96
18.1
0.94
21.2
1.00
0.0
1.00
1.1
0.99
3.5
SVHN: Main Results for Defection @9%%TPP
(Leo, c 0.03 during training)
Loo, c
0.03
Looo = 0.06
La, c
L1, = 7.85
Lo. 10
(scen)
(unscen)
(unscen)
(unscen)
(unscen)
ROC
REr
ROC
REr
ROC
RErr
ROC
REr
AUC
in %
AUC
AUC
in %
AUC
in
AUC
in %
AT
0.55
52.9
0.32
86.9
0.26
91.3
0.34
90.9
0.90
56.8
CCAT
0.70
36.5
0.70
34.8
0.91
15.6
0.89
17.6
1.00
2.1
CIFARIO: Main Results for Detection @98%TPP
(Loc, c 0.03 during training
Loo,
0.03
Lioo 0.06
L2.
Li. c 7.85
Lo.
(seen)
(unseen)
(unseen)
(unscen)
(unscen)
ROC AUC
ROC
REm
ROC
REm
ROC
RErr
ROC
REr
%
AUC
AUC
in %
AUC
AUC
in %
AT
0.64
61.9
0.35
93.6
0.59
73.6
0.61
67.6
0.80
71.8
CCAT
0.60
67.2
0.43
90.9
0.77
45.7
0.78
44.0
0.98
17.8
Table 7: Main results for 98%TPR MNIST, SVHN and Cifar10. While reporting results for
99%TPR in the main paper, cf. Tab. reducing the TPR requirement for
to
98%TPR generally improves results for both AT' and CCAT, but only slightly. Again, report Err and RErr for = 0 and 7 +998TPP as well as ROC AUC.
B.4 ABLATION STUDY
Complementing Sec. 4 we include ablation studies for MNIST, SVHN and Cifarl0; for our attack in Tab. and training in Tab. 9
Regarding the proposed attack, i.c., PGD-Conf, using momentum and backtracking, Tab. shows that the main observations for SVHN can be transferred MNIST and Cifar10. Only the improve- ment of backtracking and momentum over just using momentum cannot be confirmed on MNIST. Note that for fair comparison, iteeaaionn with backtracking are equivalent to 3/2T iterations without backtracking; which is why we include results for T 60 and T = 300. However, the importance of using enough iterations, i.e., T = 2000, and zero initialization to attack CCAT still clearly visible. Interestingly, against AT SVHN, more iterations are also beneficial, while this is not required on MNIST and Cifar10.
Tab. 9 also reports results for CCAT with different transitions, cf. Eq. (6). and values for p. As
mentioned before, on SVHN and Cifar10, power transition with Ppow = 10 works best; for larger p performance stagnates. It is also important note that the power transition does not preserve bias towards to true label, i.e., for the maximum possible perturbation (11ó = e), Eq. (6) forces the network predict a purely uniform distribution. This is in contrast to the exponential transition. where the true one-hot distribution always receives non-zero weight. On MNIST, we found this to work considerably better.
B.5 ANALYSIS
For further analysis, Fig. shows confidence histograms for AT and CCAT on MNIST and Cifar10. The confidence histograms for CCAT reflect the expected behavior: adversarial examples are mostly successful in changing the label, which is supported by high REı values for confidence threshold т = 0, but their confidence is pushed towards uniform distributions. For AT, contrast, successful adversarial examples fewer in total generally obtain high confidence; this results in confidence
26

---

Under review as a conference paper at ICLR 2020
MNIST: Supplementary Results for Detection and Standard Settings (Leu attack with 0.30 during training)
Detection Setting
Standard Setting
T T@99%TV TPR
T=0
ROC
Err
REm
Err
REm
Attack
Training
AUC
in %
in %
AT
0.97
0.0
1.0
1.0
0.5
7.2
Pang et al. 2018
1.00
0.2
2.5
2.2
0.5
100.0
Loo, é = 0.30
CCAT
0.99
0.1
7.7
1.0
0.5
100.0
0.73
0.0
81.3
1.0
0.5
Pang et al. 2018
0.99
0.2
10.9
2.2
0.5
100.0
I2, c
CCAT
1.00
0.1
1.4
1.0
0.5
82.6
SVHN: Supplementary Results for Detection and Standard Settings
(Loo attack with = 0.03 during training)
Detection Setting
Standard Setting
@ #99%TTP
T=0
ROC AUC
Err
REmr
Err
RErr
Attack
Training
in %
in %
T
in %
in %
AT
0.55
2.5
55.6
0.6
3,4
57.3
1oo0 0.03
Pang al. 2018
0.87
3.0
92.2
2.2
4.0
99.1
CCAT
0.70
38.5
0.6
2.9
AT
0.26
2.5
92.0
0.6
3.4
92.4
I2,
Pang al. 2018
0.87
3.0
81.7
2.2
4.0
93.9
CCAT
0.91
2.1
18.5
0.6
2.9
81.8
CIFARIO: Supplementary Results for Detection and Standard Settings
(Loc attack with 0.03 during training)
Detection Sctting
Standard Sctting
r T099%TPP
T=0
ROC AUC
Err
RFm
Err
REr
Attack
Training
in
in %
T
in %
0.64
15.1
62.3
0.3
16.6
62.7
Loo, é 0.03
Pang et al. (2018)
0.70
19.4
83.2
2.2
20.0
99.8
CCAT
0.60
8.7
67.9
0.4
10.1
96.7
AT
0.59
15.1
73.9
0.3
16.6
74.4
-=
Pang 2018
0.66
19.4
83.1
2.2
20.0
97.7
CCAT
0.77
8.7
46.2
0.4
10.1
81.9
Table 8: Comparison with (Pang et al. 2018) on MNIST, SVHN and Cifar10. We report "stan- dard" Err and RErr, i.e., T ("Standard Setting)), well their confidecce thresholded variants for @99%TPR ("Detection Setting"), cf. Sec. 3 On MNIST their method is competitive regarding RErr after unffdenne-thresholdiee alization to Le attacks. However, on SVHN and Cifar10, the approach cannot be considered robust anymore regarding our rigorous evaluation; this might also be due the fact that no adversarial examples are used during training.
even outperforming AT when it comes to the gener-
thresholding being not effective for AT. This behavior, however, is less pronounced on MNIST. Here, the exponential transition results in adversarial examples with confidence slightly higher than uniform confidence, i.e., 0.1. This might be the result of preserving a bias towards the true label through Eq. (6). In fact, for lower p. we found that this behavior is pronounced, until, for very small the behavior of AT is obtained.
In Fig. 7 we plot the probabilities for all ten classes along an adversarial direction. We note that these directions do not necessarily correspond to successful adversarial examples. Instead, we chose the first 10 test examples on SVHN. The adversarial examples were obtained using our Loc. PGD- Conf attack with T 2000 iterations and zero initialization for 0.03. For AT, we usually observe a change in predictions along these directions; some occur within || S c, corresponding to successful adversarial examples, some occur for ||o]|oo > corresponding unsuccessful ad- versarial examples (within e). However, AT always assigns high confidence. Thus, when allowing larger adversarial perturbations at time, robustness of AT reduces significantly. For CCAT, in
27

---

Under review as a conference paper at ICLR 2020
contrast, there are only few such cases; more often, the model achieves near uniform prediction for small ||o||o and extrapolates this behavior beyond the e-ball used for training. On SVHN, this behavior successfully allows to generalize the robustness to larger adversarial perturbations. Fur- thermore, these plots illustrate why using more iterations at test time, and using techniques such as momentum and backtracking, are necessary to find adversarial examples as the objective becomes more complex compared to AT.
Fig. g shows concrete adversarial examples against AT and CCAT on MNIST and SVHN. Among the first 200 test examples, we picked the first six examples where AT is mistaken and (non-overlapping) six examples where CCAT has difficulties, i.e., assigns high confidence. On SVHN, for example, CCAT is able to detect adversarial examples with low confidence where AT assigns incorrect classes. On MNIST, in contrast, CCAT, cannot recover mistakes made by AT. Similarly, there exist examples where AT assigns the correct class while CCAT assigns the wrong class with very high confidence. On MNIST, we additionally found adversarial examples against CCAT to represent more "localized" changes in the Los norm compared to AT. Overall, however, we did not find any systematic difference between strong adversarial against AT and CCAT.
In Fig. 9 on MNIST, we udditionally illustrate the advantage of CCAT with respect to the loy example in Proposition ! Here, we consider the case where the e-balls of two training or test examples (in different classes) overlap. As we show in Proposition [ adversarial training is not able to handle such cases, resulting in the trade-off between accuracy in robustness reported in the literature Usipras al.) 2018: [Stut al.) 2019- RRghhmathan al. 2019 Zhang al. 2019. This is because adversarial training enforces high-confidence predictions on both e-balls (corresponding to different classes), resulting in an obvious conflict. CCAT, in contrast, enforces uniform predictions throughout the largest parts of both e-balls, resolving the conflict. Finally, Fig. D also shows that CCAT is indeed able to extrapolate the uniform predictions beyond the e-ball used during training.
B.6 RESULTS
In the following, we present and discuss complementary results corresponding to the main results of our paper as presented in Tab. 3 and 4 To this end, we consider requiring only 98% TPR instead of 99% TPR, and most importantly, break down our analysis by the different white-box and black-bon. attacks used.
Main results for 98% TPR: Tab. 7 reports results (corresponding to Tab. B) requiring only 98%TPR.
This implies, that compared to 99%TPR, up to 1% more correctly classified test examples can be rejected. For relatively simple tasks such as MNIST and SVHN, where Err values are low, this is a significant "sacrifice". However, as can be seen, robustness in terms of RErr only improves slightly. We found that the same holds for 95%TPR, however, rejecting more than 2% correctly classified examples seems prohibitive large for the considered datasets.
Per-Attack Results: Finally Tab. 10 11 and 12 we break down the results of Tab. regarding the used attacks. For simplicity we focus on PGD-CE and PGD-Conf while reporting the used black- box attacks together, i.., taking the worst-case adversarial examples across all black-box attacks. For comparison, we also include non-thresholded Err and RĒrr for the results from the main paper. On MNIST, where AT performs very well in practice, it is striking that for 4/3c = 0.4 even black- box attacks are able reduce robustness completely, resulting in high REr.. This observation also transfers to SVHN and Cifar 10. For CCAT, black-box attacks are only effective on Cifarl0, where they result in roughly 87% REm with T @999%TRR. For the Lz L1 and Lo attacks we can make similar observations, except on MNIST, where AT seems similarly robust against L1 and Ly attacks as CCAT. Across all Lp norms, it can also be seen that PGD-CE performs significantly worse against our CCAT compared to AT, which shows that it is essential to optimize the right objective to evaluate the robustness of defenses and adversarially trained models, i.e., maximize confidence against CCAT
Results on Corrupted MNIST/Cifar10: We also conducted experiments on MNIST-C Muu & Gilmer 2019) and CifarlO-C [Tendrycks & Dietterich 2019). These datasets are variants of MNIST and Cifar10 that contain common perturbations of the original images obtained from var- ious types of noise, blur or transformations: examples include zoom or motion blue, Gaussian and shot noise, rotations, translations and shear. Tab. [1 [ppresents the per-corruption results on MNIST- C and CifarIO-C. Here, al all I includes all corruptions and mean reports the average results across
28

---

Under review as a conference paper at ICLR 2020
all corruptions; note that, due the thresholding. leaving different numbers of corrupted examples after detection, the distinction between all and mean is meaningful. Striking is the performance of CCAT noise corruptions such as gaussian noise or shot noise. Here, CCAT is able to detect 100% of the corrupted examples (cf. the true negative rate (TNR) in Tab. 14). resulting in thresholded Err of 0%. This is in stark contrast AT, exhibiting Err of roughly 15% after detection on Cifarl0-C. On the remaining corruptions, CCAT is able to perform slightly better than AT, which is often due to higher detection rate, i.e., higher ROC AUC. On, CifarI0, the generally lower Err of CCAT also contributes to the results. Overall, this illustrates that CCAT is able to pre- serve the inductive bias of predicting near-uniform distribution on noise similar to Loe adversarial examples seen during training.
Comparison with (Pang et al.) 2018: We also compare AT and CCAT with the approach by Pang et al. 2018) We implemented the reverse cross-entropy error in PyTorch following the official code The reverse cross-entropy loss encourages the non-maximum predictions (i.e., the case of correct classification, the probabilities of the "other" classes) to be uniform. Then, adversarial examples are detected based on statistics of the predicted distributions. We used the non-maximum element entropy (nonME) detector instead of the K-density detector (Feinman al. 2017 () which
has been shown to be ineffective against adaptive attacks in (Carlini & Wagner 2017a). We note that |Pang al. 2018) do not train on adversarial examples as AT CCAT. Our evaluation
metrics need not be changed; however, thresholding is done on the nonME detector instead of on the confidence, as for CCAT. In Tab. [ it can be seen, that the approach works considerably well on MNIST, regarding both Lise and L2 attacks. Again, we report the worst results across PGD- CE, PGD-Con GGD-Conf and the considered black-box attacks. However, on SVHN and Cifar10, we are able to break to robustness of the approach, reaching 92% (thresholded) RErr on SVHN and 99% RTE on Cifar10. On SVHN, the models performs slightly better against Le attacks, with 81,, but can not be considered robust anymore.
httpss///githhbb.cmm//2333 htter/////eeeeeeerrroo---at---e-
29

---

0 ("Stan- Under review as a conference paper at ICLR 2020
MNIST: Training Ablation Detection and Standard Settings (Loo attack with 0.30 during training)
Detection Setting
Standard Setting
7 *@99%TPP
ROC AUC
Err
REm
Err
RFr
%
in %
Normal
0.34
0.1
100.0
1.0
0.4
100.0
AT
0.97
0.0
0.4
1.0
0.5
5.6
AT Conf
0.98
0.1
1.1
1.0
0.5
6.0
CCAT, Pesp
0.99
0.1
11.9
1.0
0,4
88.1
CCAT, Pesp
0.98
0.1
10.4
1.0
0.5
95.2
CCAT, Peep =
0.99
0.1
10.8
1.0
0.4
88.7
CCAT, Pexp
0.99
0.1
7.8
1.0
0.4
83.5
CCAT, Pesp
0.99
0.1
5.7
1.0
0.5
74.7
CCAT, Peep
0.98
0.1
13.3
1.0
0.3
94.3
CCAT,
0.99
0.1
11.0
1.0
0.3
86.8
CCAT, Peep 10
0.99
0.1
14.3
1.0
0.4
97.1
CCAT, Ppon 10
0.95
0.1
17.3
1.0
0.3
64.8
SVHN: Training Ablation for Detection and Standard Settings (Lose attack with 0.03 during training)
Detection Setting
Standard Setting
7 @ 99%9TTP
7
ROC AUC
Err
REr
Err
RErr
in %
in %
T
in %
Normal
0.17
2.6
99.9
0.8
3.6
99.9
0.55
2.5
54.9
0.6
3.4
AT Conf
0.61
2.8
52.5
0.6
3.7
58.7
CCAT, Ppon
0.74
2.2
43.0
0.3
2.7
82.4
CCAT, Ppnn 2
0.68
2.1
44.2
0.5
2.9
79.6
CCAT, Ppon
0.68
1.8
35.8
0.6
2.7
CCAT, Ppow 6
0.64
1.8
32.8
0.7
2.9
72.1
CCAT, Ppow 8
0.63
2.2
42.3
0.6
2.9
84.6
CCAT, Ppow 10
0.67
2.1
38.5
0.6
2.9
91.0
CCAT, Ppon 12
0.67
1.9
36.3
0.6
2.8
CCAT, CIFARIO: Training Ablation for Detection Standard Settings
0.66
2.0
54.7
0.5
2.9
73.1
(Leo attack with e = 0.03 during training)
Detection Setting
Standard Setting
T99%TPR
0
ROC AUC
Err
REm
Err
REr
in %
in
in %
Normal
0.20
7.4
100.0
0.6
8.3
100.0
AT
0.65
15.1
60.9
0.3
16.6
61.3
AT Conf
0.63
15.1
61.5
0.4
16.1
61.7
CCAT, Ppnn
0.63
8.7
72.4
9.7
CCAT, Ppon 2
0.60
8.4
70.6
0.4
9.7
95.1
CCAT, Ppow 4
0.61
8.6
66.3
0.4
9.8
93.5
CCAT. Ppon 6
0.54
8.0
69.8
0.4
9.2
94.1
CCAT, Ppon
0.58
8.5
65.3
0.4
9.4
CCAT, Ppon 10
0.60
8.7
63.0
0.4
10.1
95.0
CCAT, Ppow 12
0.62
9.4
63.0
0.3
10.1
96.6
CCAT, Peap
0.66
11.7
68.6
0.2
13.2
77.4
Table 9: Training ablation studies MNIST, SVHN and Cifar10. We report results for different
and transitions, cf. Eq. (6). We report RЕл and Err with confidence threshold
dard Setting") and 7 TPR. as well ROC AUC ("Detection Setting"). The models are tested against our L PGD-Conf attack with 7 2000 iterations and zero as well as random initialization, as discussed Sec. 4.7 On MNIST, the exponential transition. especially Pexp = 7 performs best; on Cifar10, the power transition with Ppow 10 works best performance stagnates for Ppow > 10. SVHN, we also use Prow 10, although Ppow = 6 shows better results. However, against larger c-balls, found that Ppow 10 works significantly better.
30

---

Under review as a conference paper at ICLR 2020
Supplementury
Settings
(Looo 0.30 during training)
Detection Setting
Standard Setling
90%TTP
Attack
Training
ROC AUC
Ei
REm
Eirr
REm
AT Poy
0.97
0.0
1.0
1.0
D.5
7.2
(Troc 0.30)
CCAT, #po==10
0.96
0.1
21.9
1.0
0.3
90.5
AT
0.97
0.0
0.4
1.0
0.5
5.6
PGD Comí (Loo. 0.30)
Aog =o
74.7
CCAT, 10
0.95
0.1
17.3
1.0
0.3
64.8
AI CCAT, Plo =7
0.97
0.0
0.8
1.0
0.5
6.7
PGD (I. 0.30)
1.00
0.1
4.3
1.0
0.5
100.0
CCAT, Peaw 10
1.00
0.1
0.1
1.0
0.3
100.0
0.98
0.0
1.0
0.5
7.2
Black-Box (Loo. 0.30)
CCAT, Aoy=7
1.00
0.1
1.0
0.5
100.0
CCAT, Ppw==1
1.00
0.1
7.4
1.0
0.3
88.6
AT
0.20
100.0
Worst-Csse (Loo. 0.40)
CCAT, Amp=7
0.94
0.1
40.0
1.0
0.5
100.0
Poow
AT
0.36
0.0
97.8
1.0
0.5
99.8
PGD Conf (Loo. 0.40)
CCAT, Asy=7
0.96
0.1
15.4
1.0
0.5
97.1
Ppow
A1
0,20
0.0
100.0
1.0
0.5
100.0
PGD (Loc. 0.40)
CCAT, Aop=7
0.97
0.1
29.6
1.0
0.5
100.0
CCAT, flow
1.00
0.1
2.5
1.0
0.3
100.0
AT Asa
0.23
0.0
100.0
1.0
D.5
100.0
Black-Box (T 0.40)
0.1
1.0
0.5
CCAT,
0.99
0.1
10.7
1.0
0.3
91.8
AT CCAT, =7
0.73
0.0
81.3
1.0
0.5
98.8
WonttCaasc(Lg,
1.00
0.1
1.0
0.5
82.6
CCAT, Apo==1 110
1.00
0.1
0.1
1.0
0.3
22.1
AI
0.98
0.0
0.2
1.0
0.5
3.4
PGD Conf Conf
CCAT, Pow 77
1.00
0.1
1.0
0.5
4.4
CCAT, Peaw 10
1.00
0.1
0.1
1.0
0.3
9.5
AT
PGD CE (Izg.
CCAT, 7
1.00
0.1
0.9
1.0
0.5
82.4
CCAT, Abow
1.00
0.1
0.1
1.0
0.3
100.0
AI CCAT, Am
Black Box (Lgg =
1.00
0.1
0.6
1.0
0.5
38.4
Pawe
AT
0,93
0.0
5,2
1.0
0.5
18.0
Worst Case (I.1.c = 10.00)
CCAT, Pay =7
0.96
0.1
14.7
1.0
0.5
35.8
Paow
1.00
A1
0.93
0.0
4.2
1.0
0.5
14.8
PGD Conf (Li, 10.00)
CCAT, Apg =7
0.96
0.1
14.5
1.0
0.5
35.7
CCAT, Pro
1.00
0.1
1.6
1.0
0.3
17.7
A1 Ang
U.66
0.0
3.8
1.0
0.5
15.5
CB(L,, 10.00)
CCAT, =10==1
1.00
0.1
0.1
1.0
0.3
99.7
AT
0.99
0.0
2.5
1.0
0.5
93.9
Worst-Case (Lp. 15)
CCAT, Aog =7
0.99
0.1
1.0
0.5
65.9
CCAT, 100
0.99
0.1
7.8
1.0
0.3
88.8
0.8
0.0
1.0
0.5
5.2
PGD Conf (Lg. 15)
CCAT, =7
0.99
0.1
4.1
1.0
0.5
18.3
CCAT, Apow 10
0.99
0.1
3.7
1.0
0.3
13.4
AT
PGD CE (Lo, 15)
CCAT,
0.98
0.1
6.2
1.0
0.5
18.6
CCAT, Apow
0.98
0.1
5,4
1.0
0.3
16.5
AT
1.00
0.0
1.0
0.5
93.9
Black Box (Lg. 15)
CCAT,
=7
1.00
0.1
0.1
1.0
0.5
65.7
Prow
1.00
Table 10: Per-attack results MNIST. Per-attack results considering PGD-CE, in Madry al.
2018 our PGD-Conf and the remaining black-box attacks for all threat models, ie., Tooes Toz, In and Lop text. The used values reported in the left-most column. For the black-box attacks, we take the per-example worst-case across all black-box attacks.
31

---

Under review as a conference paper at ICLR 2020
SVHN: Supplementary Results for Detection and Standard Settings
(Loc. 0.03 during training)
Detection Setting
Standard Setting
7 T@99%TTV TPR
T=0
ROC AUC
Err
REm
Err
REm
Attack
Training
in %
in %
in %
AT
0.55
2.5
55.6
0.6
3.4
57.3
Worst-Case (Lox, = 0.03)
CCAT
0.70
38.5
0.6
2.9
AT
0.55
2.5
54.9
0.6
3,4
56.9
Conf (Toc, 0.03)
CCAT
0.67
2.1
38.5
0.6
2.9
91.0
2.5
43.2
0.6
3.4
50.7
PGD CE (Too, 0.03)
CCAT
1.00
2.1
2.6
0.6
2.9
94.9
AT
0.95
2.5
30.8
0.6
3.4
46.2
Black-Box (Lio,, c 0.03)
CCAT
1.00
2.1
6.3
0.6
2.9
79.5
0.32
2.5
88.3
0.6
3.4
89.0
Worst-Case (Lno, 0.06)
CCAT
0.70
2.1
46.0
0.6
2.9
99.8
AT
0.32
2.5
84.7
0.6
3,4
86.1
PGD Conf (Lex, c 0.06)
CCAT
0.70
2.1
0.6
2.9
AT
0.66
2.5
88.0
0.6
3.4
88.9
PGD CE (Loc, 0.06)
CCAT
0.99
2.1
17.2
0.6
2.9
100.0
AT
0.78
2.5
82.3
0.6
3.4
84.0
Black-Box (Loc, c 0.06)
CCAT
2.1
0.6
2.9
AT
0.26
2.5
92.0
0.6
3,4
92.4
Worst-Case(L2, 6 1)
CCAT
0.91
2.1
18.5
0.6
2.9
81.8
AT
0.50
2.5
77.1
0.6
3.4
PGD Conf (L2, 1)
CCAT
0.90
2.1
18.4
0.6
2.9
74.4
AT
0.27
2.5
92.0
0.6
3.4
92.4
PGD CE (L2, 6 1)
CCAT
0.99
2.1
3.7
0.6
2.9
100.0
AT
0.98
2.5
13.7
0.6
3.4
29.8
Black-Box (L2, 1)
CCAT
1.00
2.1
2.6
0.6
2.9
100.0
AT
0.34
2.5
91.9
0.6
3,4
92.5
Worst-Case (J,1, 6 7.85)
CCAT
0.89
2.1
20.8
0.6
2.9
63.5
AT
0.37
2.5
87.9
0.6
3.4
88.8
PGD Conf (I1, 7.85)
CCAT
0.92
2.1
13.4
0.6
2.9
50.3
AT
0.51
2.5
88.9
0.6
3.4
89.6
PGD CE (L1, = 7,85)
CCAT
0.99
0.6
2.9
100.0
AT
0.90
2.5
73.4
0.6
3.4
89.2
Worst-Case (Lo, 10)
CCAT
1.00
2.1
2.7
0.6
2.9
77.1
AT
2.5
51.3
0.6
3.4
PGD Conf (To, 10)
CCAT
1.00
2.1
2.7
0.6
2.9
63.9
AT
0.89
2.5
60.8
0.6
3.4
68.9
PGD CE (Lg, = 10)
CCAT
1.00
2.1
2.6
0.6
2.9
96.9
0.98
2.5
35.0
0.6
3.4
86.7
Black-Box (Lo. 10)
CCAT
1.00
2.1
2.6
0.6
2.9
100.0
Table 11: Per-attack results on SVHN. Per-attack results considering PGD-CE, as in Madry et al.
2018). our PGD-Conf and the remaining black-box attacks for all threat models, i.e.. Loc, Lzz and Tap see text. The used values are reported in the left-most column. For the black-box attacks, take the per-example worst-case across all black-box attacks.
32

---

Under review as a conference paper at ICLR 2020
CIFAR10: Supplementary Results for Detection and Standard Settings
(Lec, 0.03 during training)
Detection Setting
Standard Setting
7 7@99% TPR
T=0
ROC AUC
Err
REm
Err
REr
Attack
Training
in %
in %
in %
AT
0.64
15.1
62.3
0.3
16.6
62.7
Worst-Case (Lox, = 0.03)
CCAT
8.7
67.9
0.4
10.1
AT
0.65
15.1
60.9
0.3
16.6
61.3
Conf (Toc, 0.03)
CCAT
0.60
8.7
63.0
0.4
10.1
95.0
AT
15.1
61.4
16.6
PGD CE (Too, 0.03)
CCAT
0.99
8.7
9.8
0.4
10.1
100.0
AT
0.70
15.1
56.9
0.3
16.6
57.3
Black-Box (Lio,, c 0.03)
CCAT
0.90
8.7
49.3
0.4
10.1
96.4
0.35
15.1
93.6
16.6
93.7
Worst-Case (Lno, 0.06)
CCAT
0.43
8.7
91.5
0.4
10.1
99.2
AT
0.37
15.1
92.1
0.3
16.6
92.2
PGD Conf (Le,, c 0.06)
CCAT
0.49
8.7
66.8
0.4
10.1
AT
0.40
15.1
93.6
0.3
16.6
93.7
PGD CE (Loc, 0.06)
CCAT
0.98
8.7
10.4
0.4
10.1
100.0
AT
0.50
15.1
87.1
0.3
16.6
87.2
Black-Box (Loc, c 0.06)
CCAT
0.78
8.7
0.4
10.1
99.5
AT
0.59
15.1
73.9
0.3
16.6
74.4
Worst-Case(L2, 6 1)
CCAT
0.77
8.7
46.2
0.4
10.1
81.9
AT
0.63
15.1
64.9
16.6
PGD Conf (L2, 1)
CCAT
0.78
8.7
45.2
0.4
10.1
80.9
AT
0.61
15.1
73.9
0.3
16.6
74.6
PGD CE (L2, é 1)
CCAT
0.95
8.7
18.9
0.4
10.1
100.0
AT
0.81
15.1
35.8
0.3
16.6
36.9
Black-Box (L2, 1)
CCAT
1.00
8.7
8.8
0.4
10.1
100.0
AT
0.61
15.1
68.0
0.3
16.6
68.3
Worst-Case (J,1, 6 7.85)
CCAT
0.78
8.7
45.2
0.4
10.1
AT
0.61
15.1
66.8
0.3
16.6
67.1
PGD Conf (I1, 7.85)
CCAT
0.84
8.7
35.7
0.4
10.1
73.5
AT
0.71
15.1
58.5
0.3
16.6
60.5
PGD CE (L1, = 7,85)
CCAT
8.7
0.4
10.1
AT
0.80
15.1
74.1
0.3
16.6
75.9
Worst-Case (Lo, 10)
CCAT
0.98
8.7
20.9
0.4
10.1
55.7
AT
0.74
15.1
44.3
16.6
45.1
PGD Conf (To, 10)
CCAT
0.98
8.7
12.5
0.4
10.1
34.8
AT
0.76
15.1
49.4
0.3
16.6
51.4
PGD CE (Lg, = 10)
CCAT
1.00
8.7
7.6
0.4
10.1
79.3
0.91
15.1
69.6
16.6
75.9
Black-Box (Lo. 10)
CCAT
0.99
8.7
17.8
0.4
10.1
99.1
Table 12: Per-attack Lo and L2 results on Cifar10. Per-attack results considering PGD-CE, as in Madry et al. (2018), our PGD-Conf and the remaining black-box attacks for all threat models, i.e. T. T.z, Ta and To, see text. The used values are reported in the left-most column. For the black-box attacks, we take the per-example worst-case across all black-bo attacks.
33

---

Under review as a conference paper at ICLR 2020
MNIST: Suppiementary Results for Detoction and Standard Settings
(Tag attack with e 0.30 during training)
Standard
Detection Setting
Setting
T8 1899%TPP IPR
ROC AUC
FPR
TNR
Emr
Eir
Comuptinn
Training
in
Normal
0.75
82.8
17.2
31.9
1.0
36.4
Al
0.80
59.0
41.0
1.0
26.9
CCAT
0.95
40.3
1.0
25.1
Normal
0.75
82.8
17.2
32.8
1.0
36.3
AT
0.80
59.0
12.6
1.0
26.9
mear
CCAT
0.95
40.4
59.7
4.0
1.0
25.1
Normal
0.36
100.0
0.0
90.2
1.0
90.2
brightness CCAT
Al
99.1
84.3
1.00
0.0
100,0
0.0
1.0
88.3
Normal
0.91
62.4
37.6
34.8
1.0
45.6
AT
0.96
28.8
71.2
33.4
1.0
53.2
canny_edges
CCAT
0.97
45.1
54.9
47.0
1.0
51.5
Normal
dotted.line
AT
0.77
72.8
27.2
0.9
1.0
7.9
CCAT
0.92
74,2
25.8
1.8
1.0
7.6
Normal
AT
0.90
29.7
70.3
10.4
1.0
$9.0
Normal
0.87
71.6
28.4
57.2
1.0
56.2
Al
0.82
67.4
32.6
1.2
1.0
11.0
gluss.blur
Normal
0.87
72.4
27.6
79.2
1.0
81.3
AI
0.98
13.9
86.
18.8
1.0
61.4
impulse_noise
CCAT
1.00
0.0
100.0
1.0
47.1
Normal
0.86
73.9
26.1
29.5
1.0
37.2
AT
0.62
90.6
1.0
mot blui
CCAT
1.00
0.1
99.9
0.0
1.0
1.7
Normal
0.70
92,4
7.6
1.7
1.0
4.6
AI
0.64
rotate
CCAT
0.94
84.3
15.7
0.5
1.0
3.8
Normal
0.84
89.5
1.0
AT
0.86
78.5
3.0
scale
CCAT
0.85
91.0
9.0
0.3
1.0
2.3
Normal
AT
0.56
95.1
4.9
0.1
1.0
0.9
CCAT
0.93
89.1
10.9
1.0
0.9
Normal
0.74
AT
0.62
91.3
8.7
0.2
1.0
1.9
0.87
Normal
0.85
72.8
27.2
18.9
1.0
28.1
AT
0.65
88.3
11.7
0.6
1.0
3.6
spacter
CCAT
1.00
Normal
0.92
61.1
38.9
69.6
1.0
70,2
AT
0.98
12.2
87.8
83.8
1.0
81.3
stripe
CCAT
1.00
100.0
1.0
75.8
Normal
0.72
95.2
4.8
0.3
1.0
1.6
translate
CCAT
0.86
91.7
8.3
0.1
1.0
1.3
Normal
0.83
74.3
25.5
16.3
1.0
24.8
zigzag
CCAT
0.97
42.6
$7.4
9.3
1.0
21.1
Table 13: Per-corruptions results on MNIST-C. Results on MNIST-C, broken down by individual corruptions (first column); all includes all corruptions and mean the averaged results over all corruptions. We report ROC AUC, FPR and additionally the true negative rate (TNR) in addition to the thresholded and unthresholded Err the corrupted examples.
34

---

Under review as a conference paper at ICLR 2020
CIFARIO: Supplementary Resalts Detection Standard Settings
(oa attack with 0.03 during training)
Standard
Detection Setting 91%R
Setting T=0
ROC AUC
TNR
Eir
Comuption
Training
in %
in %
in %
15
Normal
0.57
12:2
0.6
all
AT
0.53
96.2
3.8
16.2
0.3
18.1
72.1
Normal
0.57
97.1
12.3
0.6
13.7
mean
AT
0.53
96.2
3.8
16.2
0.3
18.1
CCAT
0.66
72.1
Normal
0.50
98.1
7.5
0.6
8,4
brightness
AT'
0.50
97.0
14.9
0.3
16.5
CCAT
0.54
94.8
0.4
10.4
Normal
0.52
98.1
1.9
8.4
0.6
9.4
contrast
0.60
94.1
17.1
0.3
20.0
CCAT
0.55
96.6
10.3
0.4
11.9
Normal
0.50
98.1
7.4
0.6
8.4
defocus
96.8
17.2
CCAT
0.49
97.5
2.5
9.2
0.4
10,5
Normal
0.60
97.3
12.1
0.6
13.5
elastic testiottaanoomm
0.57
95.8
19.0
0.3
21.1
CCAT
0.54
96.5
3.5
13.2
0.4
14.9
Normal
Cog
AT
0.57
94.7
5.3
15.7
0.3
18.3
CCAT
0.55
96.0
9.0
0.4
11.1
0.57
frost
AT
0.53
95.9
4.1
16.0
0.3
18.0
0.65
88.1
12.5
Normal
0.50
98.0
7.4
0.6
8.4
gaussian.blur
AT
0.51
96.8
3.2
15.5
0.3
17.1
0.49
Normal
0.62
96.2
15.7
0.6
17.7
gaussian.roise
0.51
96.8
15.5
0.3
17.0
CCAT
1.00
0.0
100.0
0.4
84.9
Normal
0.76
92.5
7.5
40.8
0.6
43.1
glass_blur
0.58
95.4
18.0
0.3
20.2
CCAT
0.93
31.1
68.9
14.7
0.4
31.8
Normal
0.59
97.0
3.0
13.1
0.6
14.3
impuise_roise
96.4
17.9
CCAT
1.00
0.1
99.9
0.0
0.4
61.0
Normal
0.59
96.9
12.3
0.6
13.7
joeg compreomrresion
0.51
96.8
15.5
0.3
17.0
CCAT
0.59
93.3
6.7
9.3
0.4
12.1
Normal
motion.blur
AT
0.55
95.9
4.1
16.8
0,3
18.9
CCAT
0.52
96.5
11.9
0.4
13.6
0.54
10.9
pixelate
AT
0.51
96.7
3.3
15.3
0,3
17.0
0.4
Normal
0.55
97.5
10.0
0.6
11.3
aaturate
AT
0.55
95.3
4.7
18.2
0.3
20.5
CCAT
0.48
13.0
Normal
0.58
97.0
3.0
12.1
0.6
13.7
shot.roise
AT
0.51
97.1
15.4
0.3
16.7
CCAT
1.00
100.0
0.4
84.5
Normal
0.57
97.3
2.7
12.1
0.6
13.55
CCAT
0.58
95.6
4.4
11.2
0.4
13.3
Normal
0.34
97.3
2.5
9.4
0.6
10.6
spatter
CCAT
0.58
95.2
4.8
9.3
0.4
11.6
Normal
0.57
97.0
12.4
0.6
13.8
speckle_roise
AT
0.51
97.1
15.2
0.3
16.7
CCAT
1.00
0.0
100.0
0.0
0.4
83.0
Normal
96.7
scom-blur
AT
0.56
95.7
4.3
17.2
0.3
19.2
CCAT
0.52
97.1
13.9
0.4
15.6
Table 14: Per-corruptions results Cifar Results CifarIO-C focusing individual ruptions (first column); all includes all corruptions and mean are the averaged results over all corruptions. We report ROC AUC, FPR and additionally the true negative rate (TNR) in addition to the thresholded and unthresholded Err on the corrupted examples.
35


Reviews:
[{"review_id": "Skx1F-7VtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Edit after rebuttal*\nSee the comment below for a response to the author's rebuttal. I have updated my score to a weak accept.\n\nSummary\n========\nThis paper proposes a more granular adversarial training scheme, where the model is trained to have decaying confidence as the size of the adversarial perturbations increase. \nThe idea is quite natural and well motivated and presented. The paper suffers in its theoretical and evaluation sections though, which were both fairly confusing to me. I lean towards rejection for this paper but would consider changing my score if the presentation was improved.\n\n\nComments\n=========\nThe observation that adversarial training enforces high confidence in the entirety of the epsilon ball is interesting, and the motivation for the proposed scheme is sound.\n\nYet, while the intuition of CCAT for addressing the accuracy-robustness trade-off makes sense to me, the purported beneficial effect for perturbations outside the given epsilon ball are not as clear.\nThe authors note in Section 3.1 that adversarial training gives no indications on how the model should behave outside the given epsilon ball, or on other threat models. While this is true, it is not clear how CCAT supposedly remedies this. By making the confidence decay within the epsilon ball, the training also doesn't explicitly enforce any behavior outside this ball or on other threat models. So if CCAT indeed provides benefits in those settings, it isn't clear why those gains should be expected.\n\nThe exposition of the technique in Section 3.2. is very clear, and Figure 1 nicely illustrates the effect of CCAT on the class confidences during an attack.\n\nThe theoretical toy example in Section 3.3. is a bit confusing, and I'm not convinced that the proof of Proposition 1 is correct. First, a classification problem parametrized by the class imbalance is a little weird in this setting. I have a few questions about the proof:\n- You seem to be assuming that the model's confidence is calculated using a softmax. This assumption should be stated.\n- The expressions for \\hat{p}(y=2|x=x) and \\hat{p}(y=1|x=x) are confusing to me. Why are you taking a log over the softmax? These probabilities are now negative... Also, in the second step, an equality of the form e^x / (e^y + e^x) = 1+e^(y-x) is used. This is incorrect, it should be 1/(1+e^(y-x)). In the end, the values log(1+e^a) and log(1+e^(-b)) are not in [0, 1] so they cannot be probabilities.\n- From there on, it isn't clear what the rest of the proof is really saying\n\nThe experimental section is quite thorough and the results seem convincing. The considered adaptive attack, that optimizes (4) seems natural given that the defense thresholds on the model's confidence. I wonder how this is optimized though: Do you try a targeted attack for every possible target other than the true class?\n\nI found the conflation of evaluation metrics quite confusing. It seems the authors know this and have done an effort to explain their choices, but I feel like this could still be improved further. For example, some of the metrics (e.g., AUC) should be maximized, while others (Err or RErr) should be minimized. This makes the reading of the result tables quite difficult.I would also suggest dropping one digit of precision in the Tables for better readability.\n\nOverall, I have some mixed feelings about this paper. The idea is simple and well explained. The motivation, theoretical analysis, and reporting of results are somewhat confusing though. Some extra work on the presentation could definitely help this paper.\n\nSome additional references\n========================\n- In the introduction, when mentioning the trade-off between robustness and accuracy, you should cite \"Robustness May Be at Odds with Accuracy\" by Tsipras et al. rather than Schmidt et al. (that paper shows that robustness may require more training data).\n- Also in the introduction, you could consider referencing the following papers on robustness to different threat models:\n    * Engstrom et al., \"Exploring the Landscape of Spatial Robustness\"\n    * Tramer and Boneh, \"Adversarial Training and Robustness for Multiple Perturbations\"\n- In the related work, you reference a number of works for adversarial training but not the original idea in the paper by Szegedy et al.\n- In Section 3.1., the observation that the epsilon-ball around training examples can cross class boundaries is further analyzed in \"Excessive Invariance Causes Adversarial Vulnerability\" and \"Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness\" by Jacobsen et al.\n\nTypos\n=====\np.6 enumerator -> numerator", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxBFIb8KS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes the \"confidence-calibrated adversarial training (CCAT)\" to train robust DNNs against adversarial examples. The idea is to perform adversarial training with gradually smoothed network predictions (in an exponential or power rate to perturbation size). This new approach seems can help detection and defense at the same time to some extent. While the idea is quite interesting, the paper is poorly written and many times hard to read. \n\nHere are some detailed comments:\n1. Introduction. It is hard to get how the \"robustness-accuracy trade-off\" and the \"poor generalization to other or stronger attacks\" are addressed by CCAT. I can hardly agree that \"standard adversarial training strongly depends on the training attack\". PGD-AT generalizes pretty well to other types of attacks such as CW. It needs more concrete evidence to make such a claim.\n2. Related work can be improved. There are a few places inaccurate descriptions, like \"White-box attacks utilizing projected gradient ascent to ...\". And since the focus of this paper is adversarial training, related works deserves more detailed discussions.\n3. Section 3, the 100%/50% adversarial training is not properly explained. For example,  in (3), I cannot tell how the 50% comes from, as input x in the two components are the same. \n4. The confidence calibration seems related to the reverse cross entropy (RCE) proposed in [*] for detection?\n5. Figure 1, missing legends. I cannot find where the PGD-Conf is defined when came across Figures 1/2.\n6. Experiments. The settings are not standard, which makes results hard to interpret. The Err, RErr metrics are confusing. Not sure what 99% TPR is for, detection?\n7. Hard to tell the real improvement. For example , in Table 2, CIFAR-10, L_{\\infty}=0.0.3, CCAT has even lower RErr (\\tau =0) and ROC (\\tau@99%TPR) than standard AT.\n\n[*] Pang, Tianyu, et al. \"Towards robust detection of adversarial examples.\"\n\nAdvances in Neural Information Processing Systems. 2018.\nMissing citations  of a few recent papers in adversarial training:\n[1] Zhang, Hongyang, et al. \"Theoretically principled trade-off between robustness and accuracy.\" ICML, 2019.\n[2] Wang, Yisen, et al. \"On the Convergence and Robustness of Adversarial Training.\" ICML, 2019.\n[3] Carmon, Yair, et al. \"Unlabeled data improves adversarial robustness.\" NeurIPS, 2019.\n[4] Uesato, et al. \"Are Labels Required for Improving Adversarial Robustness?\" NeurIPS, 2019.\n\nA few additional high-level suggestions: \n1) Show some concrete examples where standard adversarial training fails while CCAT is not. \n2) Show the formulation of CCAT a little earlier in Section 3. Then link back to standard adversarial training or other defense methods. Not the other way around like it did now (taking too long to get to the idea of CCAT). \n3. Use a standard experiment setting, and compare 1-2 more existing methods fairly. \n4. Show understanding and analysis on CIFAR-10 dataset instead of SVHN.\n\n=============\nI appreciate the substantial revisions made during the rebuttal. The paper reads much better now. Unfortunately, my rating remains the same. My major concern goes to the unusually setting adopted in this paper. Detection and defense are two different types of adversarial research. I appreciate the efforts this paper has put to try combining them together into a unified robust defense (eg. detection first, then defense). Specifically, the use of confidence thresholding (equivalent to detection) before defense. This is also the main reason that causes the other reviewers hard to understand the real impact made in this paper. It is not fair to directly compare AT with CCAT, as AT does not have the detection component. I suggest the authors to include an existing detection method (LID (Ma et al. 2018), or Mahalanobis score for AT detection (\"A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks\"), before applying AT for prediction. Overall, the authors have done a good rebuttal, but it still requires much improvement to be accepted.\n\nTypo:\n1. Table 3, the 3rd column should be Cifar10 not SVHN.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkeh7P29tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n====== AFTER READING THE AUTHOR RESPONSE ======\n\nMany thanks for the extensive response and the respective revision from the author(s).\nMainly, I found the main results are adjusted in the revision to rather demonstrate its good detection performance at 99% TPR, and I feel the message of the manuscript becomes more strengthen. However, at the same time I feel the proposed method would be slightly less motivated without the improved results of the \"pure\" robust accuracy, as the method itself is anyway a variant of adversarial training. There are several works that specially focus on detecting adversarial examples [1, 2], and comparing the results with them on diverse threat models would more strengthen the paper. \n\nIn overall, I keep my score unchanged to the current version of manuscript.\n\n[1] A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks, NeurIPS 2018\n[2] The Odds are Odd: A Statistical Test for Detecting Adversarial Examples, ICML 2019\n\n===============================================\n\nThe paper proposes a new adversarial training scheme to improve generalization of robustness over unseen threat models, e.g. larger perturbations or different noise distributions. The key idea is to impose uniform confidence on the adversarial examples depending on the distance from the original example, based on a pre-determined distance metric, e.g. L-infinity distance. Experimental results shows its effectiveness on detecting adversarial examples and unseen robustness for MNIST, SVHN, and CIFAR-10 datasets, under L-infinity and L-2 adversaries.\n\nIn overall, I agree that improving generalization over unforeseen adversaries is a very important problem in adversarial training, and the paper addresses this problem with a novel approach. The manuscript is generally well-presented with clear motivation. In particular, I appreciated the simplicity of the proposed idea, and the thoroughness of experiments as a defense paper, e.g. presenting per-example worst-case results across diverse attacks. However, I am currently on a slightly negative side, due to some unclear points in the experimental results. I would like to increase the score if the issue could be addressed, regarding the importance of the problem and their approach, which seems valuable to be shared in the community.\n\nMainly, it is still hard for me to interpret the presented experimental results at the positive side, unless the authors could further address on the important points of the results. In general, the results are not that clear as claimed, especially when tau=0, to show that CCAT improves robustness: At the original threat model (L-inf with the smallest epsilon), CCAT shows much inferior results across all the datasets. It does improves the L-2 results, except for CIFAR-10 which should be a bare baseline to show the scalability of the method. Although the paper also point out that CCAT sometimes achieves much lower clean test error, but sometimes this also signals the less robustness. I hope the paper could justify such points in the Table 2 for better presentation.\n\n- Perhaps ResNet-20 is too small for CIFAR-10 tasks, as the training loss would hardly minimized into 0? I think WRN-10 is a more fair standard for CIFAR-10 in AT, and wonder if the result could show more effectiveness (or get more aligned tendency across datasets) of the proposed method if the capacity of network is increased.\n- Apart from the thoroughness of the attack methods assumed, considering only L-inf and L-2 adversaries may be not enough to claim the \"generalization ability\" of a defense. Could the method also improves robustness against other attacks, e.g. general corruption (such as MNIST-C, CIFAR-10-C), or unrestricted adversarial examples?\n- Eq 4: Does it mean that the logits other than the 2nd predictions are not considered when generating adversarial examples? Personally, I don't much get the motivation of using this objective, even while it is described in below Eq. 4.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a confidence-calibrated adversarial training (CCAT). The key idea is to enforce that the confidence on adversarial examples decays with their distance to the attacked examples. The authors show that CCAT can achieve better natural accuracy and robustness. After the author response and reviewer discussion, all the reviewers still think more work (e.g., improving the motivation to better position this work, conducting a fair comparison with adversarial training which does not have adversarial example detection component) needs to be done to make it a strong case. Therefore, I recommend reject.
Decision: Reject