Title: CONFIDENCE-CALIBRATED ADVERSARIAL TRAINING and Detection: MORE ROBUST MODELS GENERALIZ-ING BEYOND THE ATTACK USED DURING TRAINING

Abstract: Adversarial training is the standard to train models robust against adversarial examples. However, especially for complex datasets, adversarial training incurs a significant loss in accuracy and is known to generalize poorly to stronger attacks, e.g., larger perturbations or other threat models. In this paper, we introduce confidence-calibrated adversarial training (CCAT) where the key idea is to enforce that the confidence on adversarial examples decays with their distance to the attacked examples. We show that CCAT preserves better the accuracy of normal training while robustness against adversarial examples is achieved via confidence thresholding, i.e., detecting adversarial examples based on their confidence. Most importantly, in strong contrast to adversarial training, the robustness of CCAT generalizes to larger perturbations and other threat models, not encountered during training. For evaluation, we extend the commonly used robust test error to our detection setting, present an adaptive attack with backtracking and allow the attacker to select, per test example, the worst-case adversarial example from multiple black-and white-box attacks. We present experimental results using L ∞ , L 2 , L 1 and L 0 attacks on MNIST, SVHN and Cifar10.

1. INTRODUCTION:

Deep neural networks have shown tremendous improvements in various learning tasks including applications in computer vision, natural language processing or text processing. However, the discovery of adversarial examples, i.e., nearly imperceptibly perturbed inputs that cause mis-classification, has revealed severe security threats, as demonstrated by attacking popular computer vision services such as Google Cloud Vision (Ilyas et al., 2018a) or Clarifai (Liu et al., 2016; Bhagoji et al., 2017b) . As the number of safety-and privacy-critical applications is increasing, e.g., autonomous driving or medical imaging, this problem becomes even more important.

In practice, adversarial training and its variants (Szegedy et al., 2013; Goodfellow et al., 2014; Madry et al., 2018) , i.e., training on adversarial examples, can be regarded as the state-of-the-art to obtain models robust against adversarial examples. In contrast to many other defenses, and to the best of our knowledge, adversarial training has not been broken so far. However, adversarial training is known to increase test error significantly. Only on simple datasets such as MNIST (LeCun et al., 1998) , adversarial training is able to preserve accuracy. This observation is typically described as a trade-off between robustness and accuracy (Tsipras et al., 2018; Stutz et al., 2019; Raghunathan et al., 2019; Zhang et al., 2019) . Furthermore, while adversarial training leads to robust models for the employed threat model used during training, the obtained robustness does not translate to other threat models, e.g., other L p -balls (Sharma & Chen, 2017; Song et al., 2018; Madry et al., 2018; Tramèr & Boneh, 2019; Li et al., 2019a; Kang et al., 2019) , larger perturbations than the ones used during training or distal adversarial examples (Hein et al., 2019) .

Contributions::

We address both problems of adversarial training: the drop in accuracy on datasets such as Cifar10 (Krizhevsky, 2009) and the poor "generalization" of robustness to larger perturbations or unseen threat models. To this end, we introduce confidence-calibrated adversarial training (CCAT) based on the idea that the predicted confidence for an adversarial example should decrease with its distance to the attacked example. Specifically, we bias the network to predict a convex combination of uniform and (correct) one-hot distribution for adversarial examples, which becomes uniform as the distance to the attacked test example increases. Thus, CCAT implicitly biases the network to predict the uniform distribution over classes beyond the adversarial examples seen during training. In contrast, standard adversarial training forces the network to classify adversarial examples correctly with maximal confidence but provides no guidance how to extrapolate beyond adversarial examples seen during training. Overall, CCAT allows to detect previously unseen adversarial examples, e.g., large perturbations, different L p attacks or distal adversarial examples, by confidence thresholding, i.e., detection.

We also discuss a worst-case evaluation methodology using an adaptive attack to compare our detection-based approach with adversarial training. First, we generalize the commonly used robust test error (Madry et al., 2018; Schott et al., 2019) to our detection setting. Second, for each test example, the worst-case adversarial example, i.e., with highest confidence, is selected from multiple white-and black-box attacks. Using an adapted attack based on (Madry et al., 2018; Dong et al., 2018) , augmented with a novel backtracking scheme, we show that CCAT produces models with improved accuracy on Cifar10 and robustness to various unseen adversarial examples: larger perturbations, L 2 , L 1 and L 0 attacks, distal adversarial examples and corrupted examples on MNIST(-C), (LeCun et al., 1998; Mu & Gilmer, 2019 ) SVHN (Netzer et al., 2011) , Cifar10(-C) (Hendrycks & Dietterich, 2019) . We will make our code for CCAT and evaluation publicly available.

Outline: After reviewing related work in Sec. 1.1, we discuss adversarial training (Madry et al., 2018) and introduce CCAT in Sec. 2. In Sec. 3 we present our confidence-thresholded robust test error and present experimental results in Sec. 4. Finally, we conclude in Sec. 5.

1.1. RELATED WORK:

Adversarial Examples: Adversarial examples can roughly be divided into white-box attacks, i.e., with access to the models, its weights and gradients, e.g. (Goodfellow et al., 2014; Madry et al., 2017; Carlini & Wagner, 2017b) , and black-box attacks, i.e., only with access to the output of the model, e.g. (Chen et al., 2017; Brendel & Bethge, 2017; Su et al., 2017; Ilyas et al., 2018b; Sarkar et al., 2017; Narodytska & Kasiviswanathan, 2017) . Adversarial examples were also found to be transferable between similar models (Liu et al., 2016; Xie et al., 2018) . While these attacks are intended to be (nearly) imperceptible, visible adversarial transformations, e.g., rotations or rotations (Engstrom et al., 2017; 2019; Dumont et al., 2018; Alaifari et al., 2018; Xiao et al., 2018) , or patches (Liu et al., 2018; Lee & Kolter, 2019; Brown et al., 2017; Karmon et al., 2018; Zajac et al., 2019) have also been proposed and transferred to the physical world (Athalye et al., 2018b; Li et al., 2019b; Kurakin et al., 2016) . We refer to recent surveys (Barreno et al., 2006; Yuan et al., 2017; Akhtar & Mian, 2018; Biggio & Roli, 2018a) for more details.. Recently, white-box attacks utilizing projected gradient ascent to maximize cross-entropy loss or surrogate objectives, e.g., (Madry et al., 2017; Dong et al., 2018; Carlini & Wagner, 2017b) , have become standard. Instead, we directly maximize the confidence in any but the true class, similar to (Hein et al., 2019; Goodfellow et al., 2019) , to attack our proposed training procedure, CCAT.

Adversarial Training: Many defenses against adversarial attacks have been proposed, see, e.g., (Yuan et al., 2017; Akhtar & Mian, 2018; Biggio & Roli, 2018b) , of which some have been shown to be ineffective, e.g., in (Athalye et al., 2018a; Athalye & Carlini, 2018) . Currently, adversarial training, i.e., training on adversarial examples, is the de-facto standard to obtain robust models. While adversarial training was proposed in different variants (Szegedy et al., 2013; Zantedeschi et al., 2017; Miyato et al., 2016; Huang et al., 2015; Shaham et al., 2018; Sinha et al., 2018; Madry et al., 2017; Wang et al., 2019) , the formulation by Madry et al. (2017) received considerable attention and has been extended in various ways (Lee et al., 2017; Ye & Zhu, 2018; Liu et al., 2019) . For example, in (Shafahi et al., 2018; Pérolat et al., 2018) , adversarial training is applied to universal adversarial example, in (Cai et al., 2018) , curriculum learning is used, and in (Tramèr et al., 2017; Grefenstette et al., 2018) ensemble adversarial training is proposed. Adversarial training is also used in provable/certified defenses (Kolter & Wong, 2017; Zhang & Evans, 2018) . The increased sample complexity of adversarial training (Schmidt et al., 2018) has been addressed in (Lamb et al., 2019) by training on interpolated examples or in (Carmon et al., 2019; Uesato et al., 2019) using unlabeled examples. The dependence on the threat model used during training was addressed in (Tramèr & Boneh, 2019 ) by using multiple threat models during training. Finally, the frequently observed trade-off between accuracy and robustness has been studied both theoretically and empir-ically (Tsipras et al., 2018; Stutz et al., 2019; Zhang et al., 2019; Raghunathan et al., 2019) . CCAT differs from standard adversarial training in the used attack objective and the imposed distribution over the labels, which tends towards a uniform distribution for large perturbations.

Detection: Instead of correctly classifying adversarial examples, as intended in adversarial training, several works (Gong et al., 2017; Rouhani et al., 2018; Grosse et al., 2017; Feinman et al., 2017; Liao et al., 2018; Ma et al., 2018; Amsaleg et al., 2017; Metzen et al., 2017; Bhagoji et al., 2017a; Hendrycks & Gimpel, 2017; Li & Li, 2017; Pang et al., 2018) try instead to detect adversarial examples. However, several detectors have been shown to be ineffective against adaptive attacks, i.e., adversaries aware of the used detection mechanism (Carlini & Wagner, 2017a) . Recently, the detection of adversarial examples based on their confidence, similar to our approach with CCAT, has also been discussed (Pang et al., 2018) ; however, the authors build on (Feinman et al., 2017) which has been shown to be ineffective (Carlini & Wagner, 2017a) , as well. Goodfellow et al. (2019) , instead, focus on evaluating confidence-based detection methods using adaptive, targeted attacks maximizing confidence. Our attack, introduced in Sec. 2.2, is similar in spirit, however, untargeted by nature and, thus, suitable for usage with CCAT.

2. CONFIDENCE CALIBRATION OF ADVERSARIAL EXAMPLES:

Adversarial training, as described by Madry et al. (2018) , has become standard to train robust models. However, accuracy on challenging datasets such as Cifar10 is reduced and robustness does not generalize to larger perturbations, e.g., with respect to the used L ∞ constraint, or different threat models in other L p norms. With CCAT, we intend to address both shortcomings: Instead of training models to correctly classify adversarial examples, we want to detect adversarial examples based on their confidence. In the following, we first review adversarial training, before discussing the modifications needed for our CCAT, thereby turning robustness against adversarial examples into a detection problem.

Notation: We consider a classifier f : R d → R K where K is the number of classes and f k denotes the confidence for class k. While we assume that the cross-entropy loss L is used for training, our approach can also be used with other losses. Given x ∈ R d , classified correctly as y = argmax k f k (x), an adversarial example x + δ is defined as a "small" perturbation δ such that argmax k f k (x + δ) = y, i.e., the classifier changes its decision. The strength of the change δ is measured by some L p -norm with p ∈ {0, 1, 2, ∞}; p = ∞ is a popular choice in the literature as this leads to the smallest perturbation per feature/pixel.

2.1. ADVERSARIAL TRAINING:

Standard adversarial training, as formulated by Madry et al. (2018) , is given as the following minmax problem:

EQUATION

with w being the classifier's parameters and L being the cross-entropy loss. During mini-batch training the inner maximization problem,

EQUATION

is approximately solved. In addition to the L ∞ -constraint, a box constraint, i.e., xi = (x + δ) i ∈ [0, 1], is enforced for images. Note that maximizing the cross-entropy loss is equivalent to finding the adversarial example with minimal confidence in the true class. For neural networks, this is generally a non-convex optimization problem. In (Madry et al., 2018) the problem is tackled using projected gradient descent (PGD), which is typically initialized using a random δ with δ ∞ ≤ .

During adversarial training, PGD is applied to the batch of every iteration in order to update the classifier's parameters on the obtained adversarial examples. At test time one uses the best out of several random restarts to assess robustness.

In contrast to adversarial training as proposed in (Madry et al., 2018) ρ pow = 10 using the power transition in Eq. ( 6), we plot the probabilities for all ten classes along adversarial directions. Adversarial examples were computed using our L ∞ -PGD-Conf attack, cf. Sec. 4.1: using projected gradient descent (Madry et al., 2018) the confidence of the adversarial examples is maximized for T = 1000 and L ∞ -constraint = 0.03. The robustness of AT does not generalize beyond the = 0.03-ball as high confidence adversarial examples can be found for larger perturbations, whereas CCAT predicts close to uniform confidence after some transition phase allowing to easily detect adversarial examples. examples of each batch (Szegedy et al., 2013) 

min w E max δ ∞≤ L(f (x + δ; w), y) 50% adversarial training + E L(f (x; w), y) 50% "clean" training (3)

This improves test accuracy on clean examples compared to 100% adversarial training, i.e., Eq. ( 1), but typically leads to worse robustness. Intuitively, by balancing both terms in Eq. (3), the trade-off between accuracy and robustness can already be optimized to some extent (Stutz et al., 2019) .

There are two problems with adversarial training: First, the -ball around training examples might include examples from other classes. Then, the attack, i.e., the inner maximization in Eq. (1), will focus on these regions such that adversarial training for these examples gets "stuck". This case is illustrated in our theoretical toy dataset in Sec. 2.3 and has also been considered in related work (Jacobsen et al., 2019b; a) . Here, both 100% and 50%/50% adversarial training, cf. Eq. ( 1) and (3), are not able to find the Bayes optimal classifier in a fully deterministic problem, i.e., zero Bayes error. This might contribute to the observed drop in accuracy for adversarial training on, e.g., Cifar10 (Krizhevsky, 2009) . Second, and more importantly, adversarial training as in Eq. (1) does not guide the classifier how to extrapolate beyond the used -ball during training. Even worse, it enforces high confidence predictions within the -ball, which clearly cannot be extrapolated to arbitrary regions. It is not surprising that adversarial examples can often be found right beyond the -ball, i.e., using larger perturbations than at training time or other threat models, e.g., other L p norm balls. 

2.2. CONFIDENCE-CALIBRATED ADVERSARIAL TRAINING:

max δ ∞≤ max k =y f k (x + δ; w) (4)

where f k denotes the probability f assigns to class k (i.e., in practice, f is the output of the softmax layer). A similar objective but for targeted attacks has been used in (Goodfellow et al., 2019) , whereas our goal is an untargeted attack and thus our objective is the maximal confidence over all Algorithm 1 Pseudo-code of confidence-calibrated adversarial training (CCAT). The main changes compared to regular adversarial training as, e.g., described in (Madry et al., 2018) or (Szegedy et al., 2013) , are in the attack (line 4) and the probability distribution over the classes (line 6,7), which becomes more uniform as distance δ ∞ increases.

1: while true do 2:

choose random batch (x 1 , y 1 ), . . . , (x B , y B ). 

EQUATION

Here, p y (k) is the original "one-hot" distribution, i.e., p y (k) = 1 iff k = y and p y (k) = 0 otherwise with y being the true label, and u(k) = 1 /K is the uniform distribution. Thus, we enforce a convex combination of the original label distribution and the uniform distribution which is controlled by the parameter λ. We choose λ to decrease with the distance δ ∞ of the adversarial example to the attacked example x with the intention to enforce uniform predictions when δ ∞ = δ. Then, the network is encouraged to extrapolate this uniform distribution beyond the used -ball. Even if extrapolation does not work perfectly, the uniform distribution is much more meaningful for extrapolation in the region between classes compared to high-confidence predictions as encouraged in standard adversarial training. We consider two variants of transitions, i.e., controlling the trade-off λ between one-hot and uniform distribution:

λ =e -ρ δ ∞ ("exponential transition" (exp)) λ =(1 -min(1, δ ∞ / )) ρ

("power transition" (pow))

This ensures that for δ = 0 we impose the original (one-hot) label. For growing δ, however, the influence of the original label decays proportional to δ ∞ . The speed of decay is controlled by the parameter ρ. For the exponential transition, we always have a bias towards the true label as even for large ρ, λ will be non-zero. In case of the power transition, λ = 0 for δ ∞ ≥ , meaning a pure uniform distribution is enforced. It is important to note that in CCAT in Alg. 1 we train on 50% clean and 50% adversarial examples in each batch, as in Eq. (3). Training only on adversarial examples will not work as we the network has no incentive to predict correct labels.

2.3. CONFIDENCE-CALIBRATED ADVERSARIAL TRAINING YIELDS ACCURATE MODELS:

Proposition 1 analyzes 100% adversarial training and its 50%/50% variant as well as our confidencecalibrated variant, CCAT, to show that there exist problems where both 100% and 50%/50% adversarial training are unable to reconcile robustness and accuracy, as recently discussed (Tsipras et al., 2018; Stutz et al., 2019; Raghunathan et al., 2019; Zhang et al., 2019) . However, our CCAT is able to obtain both robustness and accuracy given that λ in Eq. ( 6) is chosen appropriately. Proposition 1. We consider a classification problem with two points x = 0 and x = in R with deterministic labels, that is p(y = 2|x = 0) = 1 and p(y = 1|x = ) = 1 and the problem is fully determined by the probability p 0 = p(x = 0) as p(x = ) = 1 -p 0 . The Bayes error of this classification problem is zero. The predicted probability distribution over the classes is p(y|x) = e gy (x) e g 1 (x) +e g 2 (x) , where g : R d → R 2 and we assume that the function λ : R + → [0, 1] is monotonically decreasing and λ(0) = 1. The Bayes optimal classifier for the cross-entropy loss of Then, we intend to use test error (Err) and robust test error (RErr), as also used in (Madry et al., 2018) , to evaluate adversarial training. However, extending RErr to take into account confidencethresholding as in our detection setting is non-trivial: for example, correctly classified examples can can have lower confidence than their corresponding adversarial examples. Thus, given confidence threshold τ , we define the confidence-thresholded robust test error RErr as follows:

EQUATION

Here, τ is the confidence-threshold fixed on a held-out validation set, 4). In total this yields a correct fraction within [0, 1] and for τ = 0, Eq. ( 7) reduces to the "standard" RErr. The confidence-thresholded Err(τ ) corresponds to taking only the first terms in both numerator and denominator.

{(x n , y n )} N n=1 are test exam- ples, c(x n ) := max k f k (x n ) denotes the classifier's confidence on x n , f (x n ) := argmax k f k (x n ) denotes the

4. EXPERIMENTS:

We compare normal training, adversarial training (AT) and our CCAT on MNIST, (LeCun et al., 1998) SVHN (Netzer et al., 2011) and Cifar10 (Krizhevsky, 2009) . We use ResNet-20 (He et al., 2016) , implemented in PyTorch (Paszke et al., 2017) , initialized following (He et al., 2015) and trained using stochastic gradient descent with batch size of 100 for 100 or 200 epochs (MNIST and SVHN/Cifar10, respectively). As detailed in Sec. 3, we report ROC AUC (higher is better) as well as our confidence-thresholded test error (Err; lower is better) and robust test error (RErr; lower is better); Err is reported on the full test sets, while ROC AUC and RErr are reported on the first 1000 attacked test examples. More details and experimental results can be found in Appendix B. We plot the objective of Eq. ( 4) over iterations for 10 samples (different colors).

4.1. ATTACKS:

White-box and adaptive attacks: We follow (Madry et al., 2018) and use projected gradient descent (PGD) to minimize the negatives of Eq. ( 2) and ( 4); we denote them as PGD-CE and PGD-Conf. The perturbation δ is initialized uniformly over direction and distance; for PGD-Conf, we additionally use δ = 0 as initialization. Different from (Madry et al., 2018) , we run exactly T iterations (no early stopping) and take the perturbation corresponding to the best objective of the T iterations. In addition to momentum, as in (Dong et al., 2018) , we propose to use an adaptive learning rate in combination with a backtracking scheme to improve the attacks: after each iteration, the computed update is only applied if it improves the objective; otherwise the learning rate is reduced. We also implemented PGD for L 2 , L 1 and L 0 attacks; note that for each L p , p ∈ {∞, 2, 1, 0}, PGD-Conf has been explicitly designed to attack our CCAT. We train on L ∞ attacks using T = 40 iterations with = 0.3 (MNIST) or = 0.03 (SVHN/Cifar10). For evaluation, we use T = 2000 iterations and 10 random retries for PGD-Conf; T = 200 with 50 random retries for PGD-CE. For L 2 , L 1 and L 0 attacks, we set to 3, 10, 15 (MNIST) or 1, 7.85, 10 (SVHN/Cifar10), respectively, partly following (Tramèr & Boneh, 2019) . Learning rates have been tuned for each L p and objective (i.e., CE or Conf) independently.

Black-box attacks: For the L ∞ case, we additionally use random sampling, the attack by Ilyas et al. (2018a) , adapted with momentum and backtracking optimizing Eq. ( 4) for T = 2000 iterations with 10 attempts, a variant of (Narodytska & Kasiviswanathan, 2017) with T = 2000 iterations and the "cube" attack (Andriushchenko, 2019) with T = 5000 iterations. We also consider the cube attack in the L 2 setting and the attack of (Croce & Hein, 2019) in the L 0 setting. The black-box attacks ensure that our defense avoids, e.g., gradient masking as described in (Athalye et al., 2018a) .

4.2. WORST-CASE EVALUATION METHODOLOGY:

Instead of reporting robustness against individual attacks, as commonly done in the literature, we use a worst-case evaluation scheme: For each individual test example, all adversarial examples from multiple attempts (i.e., multiple random restarts) and across all (white-and black-box) attacks are accumulated. Subsequently, per text example, only the adversarial example with highest confidence is kept; these are the worst-case adversarial examples for each test example. In practice, this scheme allows to accumulate results from different attacks and thus corresponds to a much stronger robustness evaluation than usual. However, also at test time, Tab. 1 shows that attacking our CCAT model effectively requires up to T = 2000 iterations and zero initialization so that RErr for τ @99%TPR stagnates. In contrast, PGD-Conf performs better against AT even for smaller T and without momentum or backtracking. Thus, finding high-confidence adversarial examples against CCAT is more difficult than for AT. Overall, this illustrates our immense effort put into attacking our proposed defense with an adapted attack, novel optimization techniques, and large number of iterations.

Evaluation Metrics: Fig. 4 shows ROC and RErr curves on SVHN, considering AT and CCAT with ρ pow = 10, i.e., using the power transition from Eq. ( 6). The ROC curves, and the corresponding AUC value, quantify how well (successful) adversarial examples can be detected, i.e., distinguished from (correctly classified) test examples. Note our conservative choice to use the confidence threshold τ at 99%TPR, wrongly rejecting at most 1% correctly classified examples. We note that τ depends only on correctly classified test examples, not adversarial examples, and RErr implicitly includes FPR as well as Err. We focus on the confidence-thresholded variants of RErr and Err as confidence thresholding will naturally also improve results for AT, resulting in a fair comparison, i.e., AT is also allowed to benefit from our two-stage detection setting; for Err this can be seen in Tab. 2. On SVHN and Cifar10, we found the power transition with ρ pow = 10 from Eq. ( 6) to work best. Up to ρ pow = 10, performance regarding RErr for τ @99% TPR continuously improves and after ρ pow = 10 performance stagnates, as shown in detail in the appendix. On MNIST, interestingly, exponential transition with ρ exp = 7 performs best against L ∞ attacks; we assume that the slight bias towards the true label preserved in the exponential transition helps. However, even on MNIST, the power transition improves robustness against previously unseen attacks, e.g., L 2 , L 1 or L 0 attacks, such that we include results for both.

MNIST: Test Error Standard τ = 0 Detection τ @99%TPR Err in % Err in % Normal 0.4 0.1 AT 0.5 0 CCAT 0.3 (0.3) 0.1 (0.

4.4. RESULTS:

In Tab. 3, we report the main results of our paper, namely robustness across all evaluated L ∞ , L 2 , L 1 and L 0 attacks; for L ∞ we include results for the used during training and an increased . While on MNIST, CCAT incurs a drop of roughly 6% in RErr, and on Cifar10 a drop of roughly 5%, both against L ∞ with the same as during training, it significantly outperforms AT on SVHN, by more than 16%. On SVHN and Cifar10, Err is additionally improved -on Cifar10, the improvement is particularly significant with roughly 6%. For L ∞ attacks with larger , robustness of AT degrades significantly, while CCAT is able to preserve robustness to some extent, especially on SVHN. Only on Cifar10, RErr degrades similarly to AT. In terms of generalization to previously unseen threat models, i.e., L 2 , L 1 and L 0 attacks, CCAT outperforms AT significantly. We note that on all datasets, the considered -balls for L 2 , L 1 and L 0 are not contained in the L ∞ -ball used during training. Overall, robustness of AT degrades significantly under threat models not seen during training, while CCAT generalizes to new threat models much better.

In Tab. 4 we also report results for detecting uniform noise and adversarial uniform noise (i.e., distal adversarial examples) based on their confidence. For the latter, we sample uniform noise and subsequently use PGD-Conf to maximize the confidence (without considering any true label in Eq. ( 4)) in the L ∞ -ball around the noise point. Although ROC AUC values are high on uniform noise, an FPR of 40% or higher shows that normal training and AT assigns high confidence to uniform noise. When 

A PROOF OF PROPOSITION 1:

Proposition 2. We consider a classification problem with two points x = 0 and x = in R with deterministic labels, that is p(y = 2|x = 0) = 1 and p(y = 1|x = ) = 1 and the problem is fully determined by the probability p 0 = p(x = 0) as p(x = ) = 1 -p 0 . The Bayes error of this classification problem is zero. The predicted probability distribution over the classes is p(y|x) = e gy (x)

e g 1 (x) +e g 2 (x) , where g : R d → R 2 and we assume that the function λ : R + → [0, 1] is monotonically decreasing and λ(0) = 1. The Bayes optimal classifier for the cross-entropy loss of

• adversarial training on 100% adversarial examples, cf. Eq. (1), yields an error of min{p 0 , 1 -p 0 }.

• adversarial training with 50% adversarial and 50% clean examples per batch, cf. Eq. (3), yields an error of min{p 0 , 1 -p 0 }.

• CCAT on 50% clean and 50% adversarial examples, cf. Alg. 1, yields zero error if λ( ) < min p0 1-p0 , 1-p0 p0 .

Proof. First, we stress that we are dealing with three different probability distributions over the labels: the true one p(y|x), the imposed one during training p(y|x) and the predicted one p(y|x).

We also note that p depends on λ through Eq. (5), and λ itself is a function of the norm δ through Eq. ( 6); here, this dependence is made explicit by writing p(λ)(y|x). This makes the expressions for the expected loss of CCAT slightly more complicated. We first derive the Bayes optimal classifier and its loss for CCAT. We introduce

EQUATION

and express the logarithm of the predicted probabilities (confidences) of class 1 and 2 in terms of these quantities.

log p(y = 2|x = x) = -log e g2(x) e g1(x) + e g2(x) = log 1 + e g1(x)-g2(x) = log 1 + e a if x = 0 log(1 + e b ) if x = .

log p(y = 1|x = x) = -log e g1(x) e g1(x) + e g2(x) = log 1 + e g2(x)-g1(x) = log(1 + e -a ) if x = 0 log 1 + e -b if x = .

For our confidence-calibrated adversarial training one first has to solve

δ * x (j) = argmax δ ∞ ≤ max k =j p(y = k | x + δ).

We get with the expressions above (note that we have a binary classification problem):

δ * 0 (1) = argmax δ ∞ ≤ p(y = 2|0 + δ) = 0 if a > b else. . δ * 0 (2) = argmax δ ∞ ≤ p(y = 1|0 + δ) = if a > b 0 else. . δ * (1) = argmax δ ∞ ≤ p(y = 2| + δ) = - if a > b 0 else. . δ * (2) = argmax δ ∞ ≤ p(y = 1| + δ) = 0 if a > b - else. .

Note that in CCAT for 50% of the batch we use the standard cross-entropy loss and for the other 50% we use

L py λ( δ * x (y) ) (x), p(x) = - 2 j=1 py λ( δ * x (y) ) (y = j | x = x) log p y = j | x = x + δ * x (j) .

The corresponding expected loss is then given by

E L py λ( δ * x (y) ) (x), p(x) = E E L py λ( δ * x (y) ) (x), p(x) |x =p(x = 0)E L(p y (λ( δ * 0 (y) ))(0), p(0))|x = 0 + p(x = )E L pY λ( δ * (y) ) ( ), p( ) |x = ,

where p(x = 0) = p 0 and p(x = ) = 1-p(x = 0) = 1-p 0 . With the true conditional probabilities p(y = s | x = x) we get

E[L(p y (λ(δ))(x), p(x))|x = x] = 2 s=1 p(y = s | x = x)L(p s (λ( δ * x (s) ))(x), p(x)) = - 2 s=1 p(y = s | x = x) 2 j=1 ps ( λ(δ * x (s) ))(y = j | x = x) log (p(y = j | x = x + δ * x (s)))

For our problem it holds with p(y = 2 | x = 0) = p(y = 1 | x = ) = 1 (by assumption). Thus,

E[L(p y (λ(δ))(x), p(x))|x = 0] = - 2 j=1 p2 (λ( δ * 0 (2) ))(y = j | x = 0) log (p(y = j | x = x + δ * 0 (2))) E[L(p y (λ(δ))(x), p(x))|x = ] = - 2 j=1 p1 (λ( δ * (1) ))(y = j | x = ) log (p(y = j | x = x + δ * (1)))

As δ * x (y) is either 0 or and λ(0) = 1 we use in the following for simplicity the notation λ = λ( ). Moreover, note that

py (λ)(y = j|x = x) = λ + (1-λ) K if y = j, (1-λ) K else ,

where K is the number of classes. Thus, K = 2 in our example and we note that λ

+ (1-λ) 2 = 1+λ

2 . With this we can write the total loss (remember that we have half normal cross-entropy loss and half the loss for the adversarial part with the modified "labels") as

L(a, b) =p 0 log(1 + e a )1 a≥b + 1 a<b (1 + λ) 2 log(1 + e b ) + (1 -λ) 2 log(1 + e -b ) + (1 -p 0 ) log(1 + e -b )1 a≥b + 1 a<b (1 + λ) 2 log(1 + e -a ) + (1 -λ) 2 log(1 + e a ) + log(1 + e a )p 0 + log(1 + e -b )(1 -p 0 ),

where we have omitted a global factor 1 2 for better readability (the last row is the cross-entropy loss). We distinguish two sets in the optimization. First we consider the case a ≥ b. Then it is easy to see that in order to minimize the loss we have a = b.

∂ a L = 2 e a 1 + e a p 0 - e -a 1 + e -a (1 -p 0 )

This yields e a = 1-p0 p0 or a = log 1-p0 p0 and the minimum for a ≥ b is attained on the boundary of the domain of a ≤ b. The other case is a ≤ b. We get

∂ a L = (1 + λ) 2 -e -a 1 + e -a + (1 -λ) 2 e a 1 + e a (1 -p 0 ) + p 0 e a 1 + e a ∂ b L = (1 + λ) 2 e b 1 + e b + (1 -λ) 2 -e b 1 + e -b p 0 + (1 -p 0 ) -e -b 1 + e -b

This yields the solution

a * = log 1+λ 2 (1 -p 0 ) p 0 + 1-λ 2 (1 -p 0 ) , b * = log 1-λ 2 p 0 + (1 -p 0 ) 1+λ 2 p 0 It is straightforward to check that a * < b * for all 0 < p 0 < 1, indeed we have 1+λ 2 (1 -p 0 ) p 0 + 1-λ 2 (1 -p 0 ) = 1+λ 2 (1 -p 0 ) p 0 1+λ 2 + 1-λ 2 = 1 -p 0 -(1-λ) 2 (1 -p 0 ) p 0 1+λ 2 + 1-λ 2 < 1-λ 2 p 0 + (1 -p 0 )

1+λ 2 p 0 if 0 < p 0 < 1 and note that λ < 1 by assumption. We have a * < 0 and thus g 2 (0) > g 1 (0) (Bayes optimal decision for x = 0) if

1 > 1 -p 0 p 0 λ,

and b * > 0 and thus g 1 ( ) > g 2 ( ) (Bayes optimal decision for x = ) if

1 > p 0 1 -p 0 λ.

Thus we recover the Bayes classifier if

λ < min 1 -p 0 p 0 , p 0 1 -p 0 .

Now we consider the approach by Madry et al. (2018) with 100% adversarial training. The expected loss can be written as . This is never Bayes optimal for all 0 < p 0 < 1. Note that b * = a * > 0 if p 0 < 1 -p 0 and thus the error is given by p 0 , similar a * = b * < 0 if 1 -p 0 < p 0 and thus the error is given by 1 -p 0 . We get an error of min{p 0 , 1 -p 0 } of the Bayes optimal solution of 100% adversarial training.

E max δ ≤∞ L(y, f (x + δ) = E E max δ ≤∞ L(y, f (x + δ)|x =p(x =

Next we consider 50% adversarial plus 50% clean training. The expected loss Moreover, we note that

E max δ ≤∞ L(y, f (x + δ) + E L(y, f (x + δ) ,

a * = b * = > 0 if p 0 < 1 2 , < 0 if p 0 > 1

2 . Thus, we classify x = 0 correctly, if p 0 > 1 2 and x = correctly if p 0 < 1 2 . Thus the error is given by min{p 0 , 1 -p 0 }.

B EXPERIMENTS:

We give additional details on our experimental setup, specifically regarding attacks, training and the used evaluation metrics. Afterwards, we include additional experimental results, including ablation studies, results for 98% true positive rate (TPR), results per attack, results per corruption on MNIST-C (Mu & Gilmer, 2019) and Cifar10-C (Hendrycks & Dietterich, 2019 ) and a comparison with (Pang et al., 2018) .

Algorithm 2 Pseudo-code for the used projected gradient descent (PGD) procedure to maximize Eq. ( 9) or Eq. ( 10) subject to the constraints xi = x i + δ i ∈ [0, 1] and δ ∞ ≤ ; in practice, the procedure is applied on batches of inputs. The algorithm is also easily adapted to work with a L 2 -norm; only the projections on line 6 and 24 needs to be adapted.

input: example x with label y input: number of iterations T input: learning rate γ, momentum β, learning rate factor α input: initial δ (0) , e.g., Eq. ( 11) or δ (0) = 0 1: v := 0 {best objective achieved} 2: x := x + δ (0) {best adversarial example} 3: g (-1) := 0 {accumulated gradients} 4: for t = 0, . . . , T do 5:

{projection onto L ∞ -ball and on [0, 1]:}

6: clip δ (t) i to [-, ] 7: clip x i + δ (t) i to [0, 1] 8:

{forward and backward pass to get objective and gradient:} 9: v (t) := F(x + δ (t) , y) 10: g (t) := βg (t-1) + (1 -β)g (t) 22:

g (t) := sign ∇ δ (t) F(x + δ (t) ,

{"try" the update step and see if objective increases:} γ := γ /α 32:

end if 33: end for 34: return x, ṽ

B.1 ATTACKS:

Complementary to the description of the projected gradient descent (PGD) attack by Madry et al. (2018) and our adapted attack, we provide a detailed algorithm in Alg. 2. We note that the objective maximized in (Madry et al., 2018) is F(x + δ, y) = L(f (x + δ; w), y) (9) where L denotes the cross-entropy loss, f (•; w) denotes the model and (x, y) is an input-label pair from the test set. Our adapted attack, in contrast, maximizes F(x + δ, y) = max k =y f k (x + δ; w).

(10) Note that the maximum over labels, i.e., max k =y is explicitly computed during optimization; this means that in contrast to (Goodfellow et al., 2019) quently take the maximum-confidence one, where K is the number of classes. We denote these two variants as PGD-CE and PGD-Conf, respectively. Deviating from (Madry et al., 2018) , we initialize δ uniformly over directions and norm (instead of uniform initialization over the volume of the -ball):

EQUATION

where δ is sampled from a standard Gaussian and u ∈ [0, 1] from a uniform distribution. We also consider zero initialization, i.e., δ = 0. For random initialization we always consider multiple attempts, 10 for PGD-Conf and 50 for PGD-CE, with zero initialization, we use only 1 attempt.

Both PGD-CE and PGD-Conf can also be applied using the L 2 , L 1 and L 0 norms following the description above. Then, gradient normalization, i.e., the signum in Line 10 in Alg. 2 for the L ∞ norm, the projection, and the the initialization need to be adapted. For the L 2 norm, the gradient is normalized by dividing by the L 2 norm; for the L 1 norm only the 1% largest values (in absolute terms) of the gradient are kept and normalized by their L 1 norm; and for the L 0 norm, the gradient is normalized by the L 1 norm. Projection is easily implemented for the L 2 norm, while we follow the algorithm of Duchi et al. (2008) for the L 1 project; for the L 0 projection, only the -largest values are kept. Similarly, initialization for L 2 and L 1 are simple by randomly choosing a direction (as in Eq. ( 11)) and then normalizing by their norm. For L 0 , we randomly choose pixels with probability 2 /3 /(HW D) and set them to a uniformly random values u ∈ [0, 1], where H × W × D is the image size. In experiments, we found that tuning the learning rate for PGD with L 1 and L 0 constraints (independent of the objective, i.e., Eq. (9) or Eq. ( 10)) is much more difficult. Additionally, PGD using the L 0 norm seems to get easily stuck in sub-optimal local optima.

Alg. 2 also gives more details on the employed momentum and backtracking scheme. These two "tricks" add two additional hyper-parameters to the number of iterations T and the learning rate γ, namely the momentum parameter β and the learning rate factor α. After each iteration, the computed update, already including the momentum term, is only applied if this improves the objective. This is checked through an additional forward pass. If not, the learning rate is divided by α, and the update is rejected. Alg. 2 includes this scheme as an algorithm for an individual test example x with label y for brevity; however, extending it to work on batches, which is used in our paper, is straight-forward.

In practice, for PGD-CE, with T = 200 iterations, we use β = 0.9 and α = 1.25; for PGD-Conf, with T = 2000 iterations, we use β = 0.9 and α = 1.1.

We also give more details on the used black-box attacks. For random sampling, we apply Eq. ( 11) T = 5000 times in order to maximize Eq. ( 10). We also implemented the black-box attack of Ilyas et al. (2018a) using a population of 50 and variance of 0.1 for estimating the gradient in Line 10 of Alg. 2; a detailed algorithm is provided in (Ilyas et al., 2018a) . We use a learning rate of 0.001 (note that the gradient is signed, as in (Madry et al., 2018) ) and also integrated a momentum with β = 0.9 and backtracking as described in Alg. 2 with α = 1.1 and T = 2000 iterations. We use zero and random initialization; in the latter case we allow 10 random retries. For the simple black-box attack we follow the algorithmic description in (Narodytska & Kasiviswanathan, 2017) considering only axis-aligned perturbations of size per pixel. We run the attack for T = 2000 iterations and allow 10 random retries. Finally, we use a variant of the cube attack proposed in (Andriushchenko, 2019) . We run the attack for T = 5000 iterations with a probability of change of 0.05. We emphasize that, except for (Ilyas et al., 2018a) , these attacks are not gradient-based and do not approximate the gradient. The ROC AUC as well as the curve itself can easily be calculated using (Pedregosa et al., 2011) . Practically, the generated curve could be used to directly estimate a threshold corresponding to a pre-determined true positive rate (TPR). However, this requires interpolation; after trying several constant interpolation schemes, we concluded that the results are distorted significantly, especially for TPRs close to 100%. Thus, we followed a simpler scheme as mentioned above: on a held out validation set of size 1000 (the last 1000 samples of the test set), we sorted the corresponding confidences, and picked the confidence threshold in order to obtain the desired TPR, e.g., 99%, on this set exactly. Robust Test Error: For clarity, we repeat our confidence-integrated definition of the robust test error:

EQUATION

As described in the main paper, Eq. ( 12) quantifies the performance of a classifier with reject-option at a specific confidence threshold τ on both clean and adversarial examples. The regular robust test error, corresponding to Eq. ( 12) at τ = 0, can also be written as x 1 and x 2 ; x 1 is fixed and we show two examples corresponding to different x 2 . Additionally, we show the corresponding images for κ = 0, i.e., x 1 , κ = 0.5, i.e., the mean image, and κ = 1, i.e., x 2 , with the corresponding labels and confidences. As can be seen, CCAT with power transition, cf. Eq. ( 6), is able to perfectly predict a uniform distribution between test examples, while the exponential transition enforces a stronger bias towards the true labels.

EQUATION

The robust test error is easy to handle as it quantifies overall performance, including generalization on clean examples (first term in Eq. ( 13)) and robustness on adversarial examples corresponding to correctly classified clean examples (second term in Eq. ( 13)). Additionally, the robust test error lies in [0, 1]. Generalizing Eq. ( 13) to τ > 0 is non-trivial due to the following considerations: First, when integrating a confidence threshold, the reference set (i.e., the denominator) needs to be adapted. Otherwise, the metric will not reflect the actual performance after thresholding, i.e., rejecting examples -specifically, the values are not comparable across different confidence thresholds τ . Then, Eq. ( 13) might be adapted as follows:

EQUATION

)

Note that in the nominator, we only consider clean and adversarial examples with confidence above the threshold τ , i.e., c(x n ) ≥ τ and c(x n ) ≥ τ . Similarly, the denominator has been adapted accordingly, as after rejection, the reference set changes, too. However, this formulation has a problem when adversarial examples obtain higher confidence than the original clean examples. Thus, second, we need to account for the case where c(x n ) > τ ≥ c(x n ), i.e., an adversarial example obtains a higher confidence than the corresponding clean example. Then, the nominator may exceed the denominator, resulting in a value larger than one. To mitigate this problem, we need to include exactly this case in the denominator. Formalizing this case, we see that it corresponds exactly to the second term in the denominator or Eq. ( 12):

EQUATION

Overall, with Eq. ( 12) we obtain a metric that lies within [0, 1], is comparable across thresholds τ and, for τ = 0, reduces to the regular robust test error as used in related work (Madry et al., 2018) . Table 7 : Main results for 98%TPR on MNIST, SVHN and Cifar10. While reporting results for 99%TPR in the main paper, cf. Tab. 3, reducing the TPR requirement for confidence-thresholding to 98%TPR generally improves results for both AT and CCAT, but only slightly. Again, we report Err and RErr for τ = 0 and τ @98%TPR as well as ROC AUC.

B.4 ABLATION STUDY:

Complementing Sec. 4, we include ablation studies for MNIST, SVHN and Cifar10; for our attack in Tab. 6 and training in Tab. 9.

Regarding the proposed attack, i.e., PGD-Conf, using momentum and backtracking, Tab. 6 shows that the main observations for SVHN can be transferred to MNIST and Cifar10. Only the improvement of backtracking and momentum over just using momentum cannot be confirmed on MNIST. Note that for fair comparison, T iterations with backtracking are equivalent to 3 /2T iterations without backtracking; which is why we include results for T = 60 and T = 300. However, the importance of using enough iterations, i.e., T = 2000, and zero initialization to attack CCAT is still clearly visible. Interestingly, against AT on SVHN, more iterations are also beneficial, while this is not required on MNIST and Cifar10.

Tab. 9 also reports results for CCAT with different transitions, cf. Eq. ( 6), and values for ρ. As mentioned before, on SVHN and Cifar10, power transition with ρ pow = 10 works best; for larger ρ performance stagnates. It is also important to note that the power transition does not preserve a bias towards to true label, i.e., for the maximum possible perturbation ( δ ∞ = ), Eq. ( 6) forces the network to predict a purely uniform distribution. This is in contrast to the exponential transition, where the true one-hot distribution always receives a non-zero weight. On MNIST, we found this to work considerably better.

B.5 ANALYSIS:

For further analysis, Fig. 5 shows confidence histograms for AT and CCAT on MNIST and Cifar10. The confidence histograms for CCAT reflect the expected behavior: adversarial examples are mostly successful in changing the label, which is supported by high RErr values for confidence threshold τ = 0, but their confidence is pushed towards a uniform distributions. For AT, in contrast, successful adversarial examples -fewer in total -generally obtain high confidence; this results in confidence (Pang et al., 2018) on MNIST, SVHN and Cifar10. We report "standard" Err and RErr, i.e., τ = 0 ("Standard Setting"), as well as their confidence thresholded variants for τ @99%TPR ("Detection Setting"), cf. Sec. 3. On MNIST their method is competitive regarding RErr after confidence-thresholding, even outperforming AT when it comes to the generalization to L 2 attacks. However, on SVHN and Cifar10, the approach cannot be considered robust anymore regarding our rigorous evaluation; this might also be due to the fact that no adversarial examples are used during training.

thresholding being not effective for AT. This behavior, however, is less pronounced on MNIST.

Here, the exponential transition results in adversarial examples with confidence slightly higher than uniform confidence, i.e., 0.1. This might be the result of preserving a bias towards the true label through Eq. ( 6). In fact, for lower ρ, we found that this behavior is pronounced, until, for very small ρ, the behavior of AT is obtained.

In Fig. 7 , we plot the probabilities for all ten classes along an adversarial direction. We note that these directions do not necessarily correspond to successful adversarial examples. Instead, we chose the first 10 test examples on SVHN. The adversarial examples were obtained using our L ∞ PGD-Conf attack with T = 2000 iterations and zero initialization for = 0.03. For AT, we usually observe a change in predictions along these directions; some occur within δ ∞ ≤ , corresponding to successful adversarial examples, some occur for δ ∞ > , corresponding to unsuccessful adversarial examples (within ). However, AT always assigns high confidence. Thus, when allowing larger adversarial perturbations at test time, robustness of AT reduces significantly. For CCAT, in contrast, there are only few such cases; more often, the model achieves a near uniform prediction for small δ ∞ and extrapolates this behavior beyond the -ball used for training. On SVHN, this behavior successfully allows to generalize the robustness to larger adversarial perturbations. Furthermore, these plots illustrate why using more iterations at test time, and using techniques such as momentum and backtracking, are necessary to find adversarial examples as the objective becomes more complex compared to AT. On MNIST, we additionally found adversarial examples against CCAT to represent more "localized" changes in the L ∞ norm compared to AT. Overall, however, we did not find any systematic difference between strong adversarial against AT and CCAT.

In Fig. 9 , on MNIST, we additionally illustrate the advantage of CCAT with respect to the toy example in Proposition 1. Here, we consider the case where the -balls of two training or test examples (in different classes) overlap. As we show in Proposition 1, adversarial training is not able to handle such cases, resulting in the trade-off between accuracy in robustness reported in the literature (Tsipras et al., 2018; Stutz et al., 2019; Raghunathan et al., 2019; Zhang et al., 2019) . This is because adversarial training enforces high-confidence predictions on both -balls (corresponding to different classes), resulting in an obvious conflict. CCAT, in contrast, enforces uniform predictions throughout the largest parts of both -balls, resolving the conflict. Finally, Fig. 9 also shows that CCAT is indeed able to extrapolate the uniform predictions beyond the -ball used during training.

B.6 RESULTS:

In the following, we present and discuss complementary results corresponding to the main results of our paper as presented in Tab. 3 and 4. To this end, we consider requiring only 98% TPR instead of 99% TPR, and most importantly, break down our analysis by the different white-box and black-box attacks used.

Main results for 98% TPR: Tab. 7 reports results (corresponding to Tab. 3) requiring only 98%TPR. This implies, that compared to 99%TPR, up to 1% more correctly classified test examples can be rejected. For relatively simple tasks such as MNIST and SVHN, where Err values are low, this is a significant "sacrifice". However, as can be seen, robustness in terms of RErr only improves slightly. We found that the same holds for 95%TPR, however, rejecting more than 2% correctly classified examples seems prohibitive large for the considered datasets.

Per-Attack Results: Finally Tab. 10, 11 and 12 we break down the results of Tab. 3 regarding the used attacks. For simplicity we focus on PGD-CE and PGD-Conf while reporting the used blackbox attacks together, i.e., taking the worst-case adversarial examples across all black-box attacks.

For comparison, we also include non-thresholded Err and RErr for the results from the main paper.

On MNIST, where AT performs very well in practice, it is striking that for 4 /3 = 0.4 even blackbox attacks are able to reduce robustness completely, resulting in high RErr. This observation also transfers to SVHN and Cifar10. For CCAT, black-box attacks are only effective on Cifar10, where they result in roughly 87% RErr with τ @99%TPR. For the L 2 , L 1 and L 0 attacks we can make similar observations, except on MNIST, where AT seems similarly robust against L 1 and L 0 attacks as CCAT. Across all L p norms, it can also be seen that PGD-CE performs significantly worse against our CCAT compared to AT, which shows that it is essential to optimize the right objective to evaluate the robustness of defenses and adversarially trained models, i. Comparison with (Pang et al., 2018) : We also compare AT and CCAT with the approach by Pang et al. (2018) . We implemented the reverse cross-entropy error in PyTorch following the official code1 . The reverse cross-entropy loss encourages the non-maximum predictions (i.e., in the case of correct classification, the probabilities of the "other" classes) to be uniform. Then, adversarial examples are detected based on statistics of the predicted distributions. We used the non-maximum element entropy (nonME) detector instead of the K-density detector (Feinman et al., 2017) which has been shown to be ineffective against adaptive attacks in (Carlini & Wagner, 2017a) . We note that (Pang et al., 2018) do not train on adversarial examples as AT or CCAT. Our evaluation metrics need not be changed; however, thresholding is done on the nonME detector instead of on the confidence, as for CCAT. In Tab. 8, it can be seen, that the approach works considerably well on MNIST, regarding both L ∞ and L 2 attacks. Again, we report the worst-case results across PGD-CE, PGD-Conf and the considered black-box attacks. However, on SVHN and Cifar10, we are able to break to robustness of the approach, reaching 92% (thresholded) RErr on SVHN and 99% RTE on Cifar10. On SVHN, the models performs slightly better against L 2 attacks, with 81%, but can not be considered robust anymore. 6). We report RErr and Err with confidence threshold τ = 0 ("Standard Setting") and τ @99%TPR as well as ROC AUC ("Detection Setting"). The models are tested against our L ∞ PGD-Conf attack with T = 2000 iterations and zero as well as random initialization, as discussed in Sec. 4.1. On MNIST, the exponential transition, especially ρ exp = 7 performs best; on Cifar10, the power transition with ρ pow = 10 works best -performance stagnates for ρ pow > 10.

On SVHN, we also use ρ pow = 10, although ρ pow = 6 shows better results. However, against larger -balls, we found that ρ pow = 10 works significantly better. 

:

https://github.com/P2333/Reverse-Cross-Entropy

Figures and Tables:

Addressing these problems requires only few but effective modifications as outlined in the description of our proposed confidence-calibrated adversarial training (CCAT) in Alg. 1: during training, we bias the network to predict a uniform distribution over the classes on adversarial examples that are sufficiently far away from the original training examples. Subsequently, during testing, adversarial examples can be detected by confidence thresholding -adversarial examples receive near-uniform confidence while test examples receive high-confidence. Thus, given an example x, the ideal attacker during training maximizes the confidence in any arbitrary other label k = y instead of minimizing the confidence in the true label y, as in Eq. (1):

other classes than true one for adversarial example xb , Eq. (4):} 5: δ b := argmax δ ∞≤ max k =y b f k (x b + δ) 6: xb := x b + δ b 7: {probability over classes of xb becomes more uniform as δ b ∞ increases, Eq. (6):} 8: λ := e -ρ δ b ∞ or λ := (1 -min(1, δ ∞ / )) ρ 9: {ỹ b is convex combination of one hot and uniform distribution over the classes, Eq. f (x b ), ỹb ) + B b= B /2 L(f (x b ), y b ) 14: end while other classes. Then, during training, CCAT biases the classifier towards predicting uniform distributions on adversarial examples by using the following distribution as target in the cross-entropy loss:

classifier's decision, and xn are adversarial examples. The numerator counts the number of incorrectly classified test examples x n with c(x n ) ≥ τ (first term) and the number of successful adversarial examples xn on correctly classified test examples with c(x n ) ≥ τ (second term). The denominator counts test examples x n with c(x n ) ≥ τ (first term) and the number of successful adversarial examples xn with c(x n ) ≥ τ but where the corresponding test example x n has c(x n ) < τ (second term). The latter takes care of the special case where adversarial examples have higher confidence than their corresponding test examples, as mentioned above and is encouraged by the objective of our attack in Eq. (

Figure 2: Momentum and backtracking. Our L ∞ PGD-Conf attack, cf. Sec. 4.1, with 40 iterations with momentum and backtracking (left) and without both (right). We plot the objective of Eq. (4) over iterations for 10 samples (different colors).

Figure 3: Confidence histograms on SVHN. For AT (left) and CCAT (right) with ρ pow = 10 (right), we show confidence histograms corresponding to correctly classified test examples (top) and successful adversarial examples (bottom). We consider the worst-case adversarial examples across all tested L ∞ attacks for = 0.03.

0)p(y = 2|x = x) maxlog p(y = 2|x = 0) , -log p(y = 2|x = )+ (1 -p(x = 0))p(y = 1|x = x) maxlog p(y = 1|x = 0) , -log p(y = 1|x = ) =p(x = 0) maxlog p(y = 2|x = 0) , -log p(y = 2|x = ) + (1 -p(x = 0)) maxlog p(y = 1|x = 0) , -log p(y = 1|x = )This yields in terms of the parameters a, b the expected loss: L(a, b) = max log(1 + e a ), log(1 + e b ) p 0 + max log(1 + e -a ), log(1 + e -b ) (1 -p 0 ) The expected loss is minimized if a = b as then both maxima are minimal. This results in the expected loss L(a) = log(1 + e a )p 0 + log(1 + e -a )(1 -p 0 ). The critical point is attained at a * = b * = log 1-p0 p0

can be written as L(a, b) = max log(1 + e a ), log(1 + e b ) p 0 + max log(1 + e -a ), log(1 + e -b ) (1 -p 0 ) + log(1 + e a )p 0 + log(1 + e -b )(1 -p 0 ) We make a case distinction. If a ≥ b, then the loss reduces to L(a, b) = log(1 + e a )p 0 + log(1 + e -b )(1 -p 0 ) + log(1 + e a )p 0 + log(1 + e -b )(1 -p 0 ) ≥ L(a, a) = 2 log(1 + e a )p 0 + 2 log(1 + e -a )(1 -p 0 )

. 4.1, attack with T iterations and different combinations of momentum, backtracking and initialization on all three datasets. We consider AT, AT trained with PGD-Conf, and CCAT; we report RErr for confidence threshold τ @99%TPR. As backtracking requires an additional forward pass per iteration, we use T = 60 and T = 300 for attacks without backtracking to be comparable to attacks with T = 40 and T = 200 with backtracking. Solving for the critical point yields a * = log 1-p0 p0 = b * . Next we consider the set a ≤ b. This yields the loss L(a, b) = log(1 + e b )p 0 + log(1 + e -a )(1 -p 0 ) + log(1 + e a )p 0 + log(1 + e -b )(1 -p 0 ) Solving for the critical point yields a * = log 1-p0 p0 = b * which fulfills a ≤ b. Actually, it coincides with the solution found already for 100% adversarial training. One does not recover the Bayes classifier for any 0 < p 0 < 1.

Figure 5: Confidence histograms on MNIST and Cifar10. As in Fig. 3, we show histograms of confidences on correctly classified test examples (top) and on successful adversarial examples (bottom) for both AT and CCAT. Note that on AT, the number of successful adversarial examples is usually lower than on CCAT, i.e., reflects the RErr for τ = 0 in Tab. 3; for CCAT in contrast, nearly all adversarial examples are successful, while only a part has high confidence. Histograms obtained for the worst-case adversarial examples across all tested L ∞ attacks.

Figure 6: ROC and RErr curves on MNIST and Cifar10. ROC curves, i.e. FPR plotted against TPR for all possible confidence thresholds τ , and RErr curves, i.e., RErr over confidence threshold τ for AT and CCAT, including different ρ parameters. Worst-case adversarial examples across all L ∞ attacks were tested.

TRAINING As described in Sec. 4, we follow the ResNet-20 architecture by He et al. (2016) implemented in PyTorch (Paszke et 2017). For training we use a batch size of 100 and train for 100 and 200 epochs on MNIST and SVHN/Cifar10, respectively: this holds for normal training, adversarial training (AT) and confidence-calibrated adversarial training (CCAT). For the latter two, we use PGD-CE and PGD Conf, respectively, for T = 40 iterations, momentum and backtracking (β = 0.9, α = 1.5

Figure 7: Effect of confidence calibration on SVHN and Cifar10. Confidences for classes along adversarial directions for AT and CCAT, ρ pow = 10. Adversarial examples were computed using PGD-Conf with T = 2000 iterations and γ = 0.001 and zero initialization. For both AT and CCAT, we show the first ten examples of the test set on SVHN, and the first five examples of the test set on Cifar10.

Figure 8: Adversarial examples against AT and CCAT on MNIST, SVHN, and Cifar10. L ∞ -PGD-Conf adversarial examples for MNIST, = 0.3, and SVHN/Cifar10, = 0.03, for AT and CCAT. In all cases, twelve test examples have been attacked. On the first six examples, AT has difficulties, on the last six examples, CCAT has difficulties. We also report the true label y, the target label ỹ of the adversarial examples and the corresponding confidence f ỹ (x).

, ρexp = 7 with L∞ PGD-Conf, = 0

Figure 9: Confidence calibration between test examples on MNIST. We plot the confidence for all classes when interpolating linearly between test examples: (1 -κ)x 1 + κx 2 for two test examplesx 1 and x 2 ; x 1 is fixed and we show two examples corresponding to different x 2 . Additionally, we show the corresponding images for κ = 0, i.e., x 1 , κ = 0.5, i.e., the mean image, and κ = 1, i.e., x 2 , with the corresponding labels and confidences. As can be seen, CCAT with power transition, cf. Eq. (6), is able to perfectly predict a uniform distribution between test examples, while the exponential transition enforces a stronger bias towards the true labels.

Fig. 8 shows concrete adversarial examples against AT and CCAT on MNIST and SVHN. Among the first 200 test examples, we picked the first six examples where AT is mistaken and (non-overlapping) six examples where CCAT has difficulties, i.e., assigns high confidence. On SVHN, for example, CCAT is able to detect adversarial examples with low confidence where AT assigns incorrect classes. On MNIST, in contrast, CCAT, cannot recover mistakes made by AT. Similarly, there exist examples where AT assigns the correct class while CCAT assigns the wrong class with very high confidence.On MNIST, we additionally found adversarial examples against CCAT to represent more "localized" changes in the L ∞ norm compared to AT. Overall, however, we did not find any systematic difference between strong adversarial against AT and CCAT.

, i.e., instead of training only on adversarial examples, each batch is divided 50% into clean and 50% into adversarial examples. Compared to Eq. (1), 50%/50% adversarial training effectively minimizes

Table content: <table/>

Attack

Table content: <table><tr><td>Optimization</td><td/><td colspan="3">momentum+backtrack</td><td/><td>mom</td><td>-</td></tr><tr><td>Initialization</td><td/><td/><td>zero</td><td/><td>rand</td><td>zero</td><td>zero</td></tr><tr><td>Iterations T</td><td>40</td><td colspan="3">200 2000 4000</td><td>4000</td><td>300</td><td>300</td></tr><tr><td>AT</td><td colspan="2">38.4 46.2</td><td>49.9</td><td>50.1</td><td>51.8</td><td>38.1</td><td>30.8</td></tr><tr><td>AT Conf</td><td colspan="2">27.4 40.5</td><td>46.9</td><td>47.3</td><td>48.1</td><td>28.5</td><td>23.8</td></tr><tr><td>CCAT</td><td>4.0</td><td>5.0</td><td>22.8</td><td>23.3</td><td>5.2</td><td>2.6</td><td>2.6</td></tr></table>

Test Main Results for Detection τ @99%TPR (L∞, = 0.30 during training, standard evaluation, i.e., τ = 0, in Appendix B)

Table content: <table><tr><td/><td colspan="2">SVHN: Test Error</td><td colspan="2">SVHN: Test Error</td></tr><tr><td/><td>Standard</td><td>Detection</td><td>Standard</td><td>Detection</td></tr><tr><td/><td>τ = 0</td><td>τ @99%TPR</td><td>τ = 0</td><td>τ @99%TPR</td></tr><tr><td/><td>Err</td><td>Err</td><td>Err</td><td>Err</td></tr><tr><td/><td>in %</td><td>in %</td><td>in %</td><td>in %</td></tr><tr><td/><td>3.6</td><td>2.6</td><td>8.3</td><td>7.4</td></tr><tr><td/><td>3.4</td><td>2.5</td><td>16.6</td><td>15.5</td></tr><tr><td>1)</td><td>2.9</td><td>2.1</td><td>10.1</td><td>6.7</td></tr></table>

Main

Table content: <table/>

Results for noise and distal adversarial examples on MNIST, SVHN and Cifar10. Robustness against uniform noise ("Rand") and distal adversarial examples ("Dist"), i.e., highconfidence adversarial computed on uniform noise using our L ∞ PGD-Conf attack by considering Eq. (4) without any true label; we use = 0.3 on MNIST, = 0.03 on SVHN/Cifar10. We report ROC AUC and FPR for a confidence threshold of τ @99%TPR. In contrast to normal training and AT, CCAT is able to reliably detect all (adversarial) noise examples.

Table content: <table><tr><td/><td/><td colspan="2">MNIST ( = 0.3)</td><td colspan="2">SVHN ( = 0.03)</td><td colspan="2">CIFAR10 ( = 0.03)</td></tr><tr><td/><td/><td colspan="2">Detection</td><td colspan="2">Detection</td><td colspan="2">Detection</td></tr><tr><td/><td/><td colspan="2">τ @99%TPR</td><td colspan="2">τ @99%TPR</td><td colspan="2">τ @99%TPR</td></tr><tr><td/><td/><td>ROC</td><td>FPR</td><td>ROC</td><td>FPR</td><td>ROC</td><td>FPR</td></tr><tr><td/><td/><td>AUC</td><td>in %</td><td>AUC</td><td>in %</td><td>AUC</td><td>in %</td></tr><tr><td>Rand</td><td>Normal AT CCAT</td><td>0.34 0.99 1.00</td><td>100.0 44.5 0.0</td><td>0.95 1.00 1.00</td><td>72.8 0.9 0.0</td><td>0.83 0.60 1.00</td><td>87.1 100.0 0.0</td></tr><tr><td>Dist</td><td>Normal AT CCAT</td><td>0.34 0.63 1.00</td><td>100.0 100.0 0.0</td><td>0.43 0.89 1.00</td><td>100.0 98.8 0.0</td><td>0.49 0.31 1.00</td><td>100.0 100.0 0.0</td></tr></table>

Results

Table content: <table/>

Detailed

Table content: <table><tr><td/><td colspan="6">MNIST: Attack Ablation with RErr for τ @99%TPR</td><td/><td/><td/></tr><tr><td/><td colspan="4">(L∞ attack with = 0.3)</td><td/><td/><td/><td/><td/></tr><tr><td>Optimization</td><td/><td colspan="4">momentum+backtrack</td><td colspan="2">momentum</td><td>-</td><td/></tr><tr><td>Initialization</td><td/><td/><td>zero</td><td/><td>rand</td><td/><td>zero</td><td colspan="2">zero</td></tr><tr><td>Iterations T</td><td colspan="4">40 200 2000 4000</td><td>4000</td><td>60</td><td>300</td><td colspan="2">60 300</td></tr><tr><td>AT</td><td>0.4</td><td>0.4</td><td>0.4</td><td>0.3</td><td>0.6</td><td>0.4</td><td>0.6</td><td>0.4</td><td>0.4</td></tr><tr><td>AT Conf</td><td>0.8</td><td>0.8</td><td>0.8</td><td>0.8</td><td>1.1</td><td>1.0</td><td>1.1</td><td>1.0</td><td>1.0</td></tr><tr><td>CCAT</td><td>0.6</td><td>3.3</td><td>4.9</td><td>6.8</td><td>5.5</td><td>0.9</td><td>3.8</td><td>0.0</td><td>0.1</td></tr><tr><td/><td colspan="6">SVHN: Attack Ablation with RErr for τ @99%TPR</td><td/><td/><td/></tr><tr><td/><td colspan="4">(L∞ attack with = 0.03)</td><td/><td/><td/><td/><td/></tr><tr><td>Optimization</td><td/><td colspan="3">momentum+backtrack</td><td/><td colspan="2">momentum</td><td>-</td><td/></tr><tr><td>Initialization</td><td/><td colspan="2">zero</td><td/><td>rand</td><td colspan="2">zero</td><td colspan="2">zero</td></tr><tr><td>Iterations T</td><td>40</td><td colspan="3">200 2000 4000</td><td>4000</td><td>60</td><td>300</td><td>60</td><td>300</td></tr><tr><td>AT</td><td colspan="2">38.4 46.2</td><td>49.9</td><td>50.1</td><td>51.8</td><td colspan="2">37.7 38.1</td><td colspan="2">29.9 30.8</td></tr><tr><td>AT Conf</td><td colspan="2">27.4 40.5</td><td>46.9</td><td>47.3</td><td>48.1</td><td colspan="2">27.1 28.5</td><td colspan="2">21.1 23.8</td></tr><tr><td>CCAT</td><td>4.0</td><td>5.0</td><td>22.8</td><td>23.3</td><td>5.2</td><td>2.6</td><td>2.6</td><td>2.6</td><td>2.6</td></tr><tr><td/><td colspan="6">Cifar10: Attack Ablation with RErr for τ @99%TPR</td><td/><td/><td/></tr><tr><td/><td colspan="4">(L∞ attack with = 0.03)</td><td/><td/><td/><td/><td/></tr><tr><td>Optimization</td><td/><td colspan="3">momentum+backtrack</td><td/><td colspan="2">momentum</td><td>-</td><td/></tr><tr><td>Initialization</td><td/><td colspan="2">zero</td><td/><td>rand</td><td colspan="2">zero</td><td colspan="2">zero</td></tr><tr><td>Iterations T</td><td>40</td><td colspan="3">200 2000 4000</td><td>4000</td><td>60</td><td>300</td><td>60</td><td>300</td></tr><tr><td>AT</td><td colspan="2">60.9 60.8</td><td>60.8</td><td>60.8</td><td>60.9</td><td colspan="2">60.9 60.9</td><td colspan="2">57.4 57.6</td></tr><tr><td>AT Conf</td><td colspan="2">60.4 60.6</td><td>60.5</td><td>60.5</td><td>60.9</td><td colspan="2">60.4 60.6</td><td colspan="2">56.2 56.6</td></tr><tr><td>CCAT</td><td colspan="2">14.8 16.2</td><td>40.2</td><td>41.3</td><td>34.9</td><td>7.2</td><td>7.2</td><td>7.2</td><td>7.2</td></tr></table>

y)

Table content: <table><tr><td>11:</td><td>{keep track of adversarial example resulting in best objective:}</td></tr><tr><td>12: 13: 14:</td><td>if v (t) &gt; v then v := v (t) x := x + δ (t)</td></tr><tr><td>15:</td><td>end if</td></tr><tr><td>16:</td><td>{iteration T is only meant to check whether last update improved objective:}</td></tr><tr><td>17:</td><td>if t = T then</td></tr><tr><td>18:</td><td>break</td></tr><tr><td>19:</td><td>end if</td></tr><tr><td>20:</td><td>{integrate momentum term:}</td></tr><tr><td>21:</td><td/></tr></table>

, we do not run K targeted attack and subse-

Table content: <table><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="10">Cifar10 (L ∞ attack with = 0.03)</td></tr><tr><td>1</td><td/><td>AT</td><td/><td/><td/><td/><td>1</td><td/><td/><td colspan="3">CCAT, ρexp = 7</td><td/><td/><td/><td/><td/><td>AT</td><td/><td/><td/><td/><td>0.6</td><td/><td colspan="3">CCAT, ρpow = 10</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0.4</td><td/><td/><td/><td/><td/><td/><td/><td>0.4</td><td/><td/><td/><td/></tr><tr><td>0.5</td><td/><td/><td/><td/><td/><td/><td>0.5</td><td/><td/><td/><td/><td/><td/><td/><td>0.2</td><td/><td/><td/><td/><td/><td/><td/><td>0.2</td><td/><td/><td/><td/></tr><tr><td>0</td><td>0</td><td>0.2</td><td>0.4</td><td>0.6</td><td>0.8</td><td>1</td><td>0</td><td colspan="2">0</td><td>0.2</td><td>0.4</td><td>0.6</td><td>0.8</td><td>1</td><td>0</td><td colspan="2">0</td><td>0.2</td><td>0.4</td><td>0.6</td><td>0.8</td><td>1</td><td>0</td><td>0</td><td>0.2</td><td>0.4</td><td>0.6</td><td>0.8</td><td>1</td></tr><tr><td/><td/><td colspan="4">Test Confidence</td><td/><td/><td/><td/><td colspan="4">Test Confidence</td><td/><td/><td/><td/><td colspan="4">Test Confidence</td><td/><td/><td/><td colspan="4">Test Confidence</td></tr><tr><td>0.4</td><td/><td>AT</td><td/><td/><td/><td/><td colspan="2">0.1</td><td/><td colspan="3">CCAT, ρexp = 7</td><td/><td/><td colspan="2">0.15</td><td/><td>AT</td><td/><td/><td/><td/><td/><td/><td colspan="3">CCAT, ρpow = 10</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">0.1</td><td/><td/><td/><td/><td/><td/><td>0.4</td><td/><td/><td/><td/></tr><tr><td>0.2</td><td/><td/><td/><td/><td/><td/><td colspan="2">0.05</td><td/><td/><td/><td/><td/><td/><td colspan="2">0.05</td><td/><td/><td/><td/><td/><td/><td>0.2</td><td/><td/><td/><td/></tr><tr><td>0</td><td>0</td><td colspan="4">0.2 Adversarial Confidence 0.4 0.6 0.8</td><td>1</td><td colspan="2">0</td><td>0</td><td colspan="4">0.2 Adversarial Confidence 0.4 0.6 0.8</td><td>1</td><td colspan="2">0</td><td>0</td><td colspan="4">0.2 Adversarial Confidence 0.4 0.6 0.8</td><td>1</td><td>0</td><td>0</td><td colspan="4">0.2 Adversarial Confidence 0.4 0.6 0.8</td><td>1</td></tr></table>

). For PGD-CE we use a learning rate of 0.05, 0.01 and 0.005 on MNIST, SVHN and Cifar10. For PGD-Conf we use a learning rate of 0.05. For training, we use standard stochastic gradient descent, starting with a learning rate of 0.1 on MNIST/SVHN and 0.075 on Cifar10. The learning rate is multiplied by 0.95 after each epoch. We do not use weight decay; but the network includes batch normalization(Ioffe & Szegedy, 2015). On SVHN and Cifar10, we use random cropping, random flipping (only Cifar10) and contrast augmentation during training. We always train on 50% clean and 50% adversarial examples per batch, i.e., each batch contains both clean and adversarial examples which is important when using batch normalization.B.3 EVALUATION METRICSFor reproducibility and complementing the discussion in the main paper, we describe the used evaluation metrics and evaluation procedure in more detail. Adversarial examples are computed on the first 1000 examples of the test set; the used confidence threshold is computed on the last 1000 examples of the test set; test errors are computed on all test examples minus the last 1000. As we consider multiple attacks, and some attacks allow multiple random attempts, we always consider the worst case adversarial example per test example and across all attacks/attempts; the worst-case is selected based on confidence.

Table content: <table/>

Comparison with

Table content: <table><tr><td/><td colspan="6">MNIST: Supplementary Results for Detection and Standard Settings</td><td/></tr><tr><td/><td colspan="4">(L∞ attack with = 0.30 during training)</td><td/><td/><td/></tr><tr><td/><td/><td/><td colspan="2">Detection Setting</td><td/><td colspan="2">Standard Setting</td></tr><tr><td/><td/><td/><td colspan="2">τ @99%TPR</td><td/><td>τ =0</td><td/></tr><tr><td colspan="2">Attack Training</td><td>ROC AUC</td><td>Err in %</td><td>RErr in %</td><td>τ</td><td>Err in %</td><td>RErr in %</td></tr><tr><td/><td>AT</td><td>0.97</td><td>0.0</td><td>1.0</td><td>1.0</td><td>0.5</td><td>7.2</td></tr><tr><td>L∞, = 0.30</td><td>Pang et al. (2018) CCAT</td><td>1.00 0.99</td><td>0.2 0.1</td><td>2.5 7.7</td><td>2.2 1.0</td><td>0.5 0.5</td><td>100.0 100.0</td></tr><tr><td/><td>AT</td><td>0.73</td><td>0.0</td><td>81.3</td><td>1.0</td><td>0.5</td><td>98.8</td></tr><tr><td>L2, = 3</td><td>Pang et al. (2018) CCAT</td><td>0.99 1.00</td><td>0.2 0.1</td><td>10.9 1.4</td><td>2.2 1.0</td><td>0.5 0.5</td><td>100.0 82.6</td></tr><tr><td/><td colspan="6">SVHN: Supplementary Results for Detection and Standard Settings</td><td/></tr><tr><td/><td colspan="4">(L∞ attack with = 0.03 during training)</td><td/><td/><td/></tr><tr><td/><td/><td/><td colspan="2">Detection Setting</td><td/><td colspan="2">Standard Setting</td></tr><tr><td/><td/><td/><td colspan="2">τ @99%TPR</td><td/><td>τ =0</td><td/></tr><tr><td colspan="2">Attack Training</td><td>ROC AUC</td><td>Err in %</td><td>RErr in %</td><td>τ</td><td>Err in %</td><td>RErr in %</td></tr><tr><td/><td>AT</td><td>0.55</td><td>2.5</td><td>55.6</td><td>0.6</td><td>3.4</td><td>57.3</td></tr><tr><td>L∞, = 0.03</td><td>Pang et al. (2018)</td><td>0.87</td><td>3.0</td><td>92.2</td><td>2.2</td><td>4.0</td><td>99.1</td></tr><tr><td/><td>CCAT</td><td>0.70</td><td>2.1</td><td>38.5</td><td>0.6</td><td>2.9</td><td>97.8</td></tr><tr><td/><td>AT</td><td>0.26</td><td>2.5</td><td>92.0</td><td>0.6</td><td>3.4</td><td>92.4</td></tr><tr><td>L2, = 1</td><td>Pang et al. (2018)</td><td>0.87</td><td>3.0</td><td>81.7</td><td>2.2</td><td>4.0</td><td>93.9</td></tr><tr><td/><td>CCAT</td><td>0.91</td><td>2.1</td><td>18.5</td><td>0.6</td><td>2.9</td><td>81.8</td></tr><tr><td/><td colspan="6">CIFAR10: Supplementary Results for Detection and Standard Settings</td><td/></tr><tr><td/><td colspan="4">(L∞ attack with = 0.03 during training)</td><td/><td/><td/></tr><tr><td/><td/><td/><td colspan="2">Detection Setting</td><td/><td colspan="2">Standard Setting</td></tr><tr><td/><td/><td/><td colspan="2">τ @99%TPR</td><td/><td>τ =0</td><td/></tr><tr><td colspan="2">Attack Training</td><td>ROC AUC</td><td>Err in %</td><td>RErr in %</td><td>τ</td><td>Err in %</td><td>RErr in %</td></tr><tr><td/><td>AT</td><td>0.64</td><td>15.1</td><td>62.3</td><td>0.3</td><td>16.6</td><td>62.7</td></tr><tr><td>L∞, = 0.03</td><td>Pang et al. (2018)</td><td>0.70</td><td>19.4</td><td>83.2</td><td>2.2</td><td>20.0</td><td>99.8</td></tr><tr><td/><td>CCAT</td><td>0.60</td><td>8.7</td><td>67.9</td><td>0.4</td><td>10.1</td><td>96.7</td></tr><tr><td/><td>AT</td><td>0.59</td><td>15.1</td><td>73.9</td><td>0.3</td><td>16.6</td><td>74.4</td></tr><tr><td>L2, = 1</td><td>Pang et al. (2018)</td><td>0.66</td><td>19.4</td><td>83.1</td><td>2.2</td><td>20.0</td><td>97.7</td></tr><tr><td/><td>CCAT</td><td>0.77</td><td>8.7</td><td>46.2</td><td>0.4</td><td>10.1</td><td>81.9</td></tr></table>

that, due to the thresholding, leaving different numbers of corrupted examples after detection, the distinction between all and mean is meaningful. Striking is the performance of CCAT on noise corruptions such as gaussian noise or shot noise. Here, CCAT is able to detect 100% of the corrupted examples (cf. the true negative rate (TNR) in Tab. 14), resulting in a thresholded Err of 0%. This is in stark contrast to AT, exhibiting a Err of roughly 15% after detection on Cifar10-C. On the remaining corruptions, CCAT is able to perform slightly better than AT, which is often due to higher detection rate, i.e., higher ROC AUC. On, Cifar10, the generally lower Err of CCAT also contributes to the results. Overall, this illustrates that CCAT is able to preserve the inductive bias of predicting near-uniform distribution on noise similar to L ∞ adversarial examples as seen during training.

Table content: <table/>

Training ablation studies on MNIST, SVHN and Cifar10. We report results for different ρ and transitions, cf. Eq. (

Table content: <table><tr><td/><td colspan="4">MNIST: Training Ablation for Detection and Standard Settings</td><td/><td/></tr><tr><td/><td colspan="3">(L∞ attack with = 0.30 during training)</td><td/><td/><td/></tr><tr><td/><td/><td colspan="2">Detection Setting</td><td/><td colspan="2">Standard Setting</td></tr><tr><td/><td/><td colspan="2">τ @99%TPR</td><td/><td/><td>τ = 0</td></tr><tr><td/><td>ROC AUC</td><td>Err in %</td><td>RErr in %</td><td>τ</td><td>Err in %</td><td>RErr in %</td></tr><tr><td>Normal</td><td>0.34</td><td>0.1</td><td>100.0</td><td>1.0</td><td>0.4</td><td>100.0</td></tr><tr><td>AT</td><td>0.97</td><td>0.0</td><td>0.4</td><td>1.0</td><td>0.5</td><td>5.6</td></tr><tr><td>AT Conf</td><td>0.98</td><td>0.1</td><td>1.1</td><td>1.0</td><td>0.5</td><td>6.0</td></tr><tr><td>CCAT, ρexp = 3</td><td>0.99</td><td>0.1</td><td>11.9</td><td>1.0</td><td>0.4</td><td>88.1</td></tr><tr><td>CCAT, ρexp = 4</td><td>0.98</td><td>0.1</td><td>10.4</td><td>1.0</td><td>0.5</td><td>95.2</td></tr><tr><td>CCAT, ρexp = 5</td><td>0.99</td><td>0.1</td><td>10.8</td><td>1.0</td><td>0.4</td><td>88.7</td></tr><tr><td>CCAT, ρexp = 6</td><td>0.99</td><td>0.1</td><td>7.8</td><td>1.0</td><td>0.4</td><td>83.5</td></tr><tr><td>CCAT, ρexp = 7</td><td>0.99</td><td>0.1</td><td>5.7</td><td>1.0</td><td>0.5</td><td>74.7</td></tr><tr><td>CCAT, ρexp = 8</td><td>0.98</td><td>0.1</td><td>13.3</td><td>1.0</td><td>0.3</td><td>94.3</td></tr><tr><td>CCAT, ρexp = 9</td><td>0.99</td><td>0.1</td><td>11.0</td><td>1.0</td><td>0.3</td><td>86.8</td></tr><tr><td>CCAT, ρexp = 10</td><td>0.99</td><td>0.1</td><td>14.3</td><td>1.0</td><td>0.4</td><td>97.1</td></tr><tr><td>CCAT, ρpow = 10</td><td>0.95</td><td>0.1</td><td>17.3</td><td>1.0</td><td>0.3</td><td>64.8</td></tr><tr><td/><td colspan="4">SVHN: Training Ablation for Detection and Standard Settings</td><td/><td/></tr><tr><td/><td colspan="3">(L∞ attack with = 0.03 during training)</td><td/><td/><td/></tr><tr><td/><td/><td colspan="2">Detection Setting</td><td/><td colspan="2">Standard Setting</td></tr><tr><td/><td/><td colspan="2">τ @99%TPR</td><td/><td/><td>τ = 0</td></tr><tr><td/><td>ROC AUC</td><td>Err in %</td><td>RErr in %</td><td>τ</td><td>Err in %</td><td>RErr in %</td></tr><tr><td>Normal</td><td>0.17</td><td>2.6</td><td>99.9</td><td>0.8</td><td>3.6</td><td>99.9</td></tr><tr><td>AT</td><td>0.55</td><td>2.5</td><td>54.9</td><td>0.6</td><td>3.4</td><td>56.9</td></tr><tr><td>AT Conf</td><td>0.61</td><td>2.8</td><td>52.5</td><td>0.6</td><td>3.7</td><td>58.7</td></tr><tr><td>CCAT, ρpow = 1</td><td>0.74</td><td>2.2</td><td>43.0</td><td>0.3</td><td>2.7</td><td>82.4</td></tr><tr><td>CCAT, ρpow = 2</td><td>0.68</td><td>2.1</td><td>44.2</td><td>0.5</td><td>2.9</td><td>79.6</td></tr><tr><td>CCAT, ρpow = 4</td><td>0.68</td><td>1.8</td><td>35.8</td><td>0.6</td><td>2.7</td><td>80.4</td></tr><tr><td>CCAT, ρpow = 6</td><td>0.64</td><td>1.8</td><td>32.8</td><td>0.7</td><td>2.9</td><td>72.1</td></tr><tr><td>CCAT, ρpow = 8</td><td>0.63</td><td>2.2</td><td>42.3</td><td>0.6</td><td>2.9</td><td>84.6</td></tr><tr><td>CCAT, ρpow = 10</td><td>0.67</td><td>2.1</td><td>38.5</td><td>0.6</td><td>2.9</td><td>91.0</td></tr><tr><td>CCAT, ρpow = 12</td><td>0.67</td><td>1.9</td><td>36.3</td><td>0.6</td><td>2.8</td><td>81.8</td></tr><tr><td>CCAT, ρexp = 7</td><td>0.66</td><td>2.0</td><td>54.7</td><td>0.5</td><td>2.9</td><td>73.1</td></tr><tr><td/><td colspan="4">CIFAR10: Training Ablation for Detection and Standard Settings</td><td/><td/></tr><tr><td/><td colspan="3">(L∞ attack with = 0.03 during training)</td><td/><td/><td/></tr><tr><td/><td/><td colspan="2">Detection Setting</td><td/><td colspan="2">Standard Setting</td></tr><tr><td/><td/><td colspan="2">τ @99%TPR</td><td/><td/><td>τ = 0</td></tr><tr><td/><td>ROC AUC</td><td>Err in %</td><td>RErr in %</td><td>τ</td><td>Err in %</td><td>RErr in %</td></tr><tr><td>Normal</td><td>0.20</td><td>7.4</td><td>100.0</td><td>0.6</td><td>8.3</td><td>100.0</td></tr><tr><td>AT</td><td>0.65</td><td>15.1</td><td>60.9</td><td>0.3</td><td>16.6</td><td>61.3</td></tr><tr><td>AT Conf</td><td>0.63</td><td>15.1</td><td>61.5</td><td>0.4</td><td>16.1</td><td>61.7</td></tr><tr><td>CCAT, ρpow = 1</td><td>0.63</td><td>8.7</td><td>72.4</td><td>0.3</td><td>9.7</td><td>95.3</td></tr><tr><td>CCAT, ρpow = 2</td><td>0.60</td><td>8.4</td><td>70.6</td><td>0.4</td><td>9.7</td><td>95.1</td></tr><tr><td>CCAT, ρpow = 4</td><td>0.61</td><td>8.6</td><td>66.3</td><td>0.4</td><td>9.8</td><td>93.5</td></tr><tr><td>CCAT, ρpow = 6</td><td>0.54</td><td>8.0</td><td>69.8</td><td>0.4</td><td>9.2</td><td>94.1</td></tr><tr><td>CCAT, ρpow = 8</td><td>0.58</td><td>8.5</td><td>65.3</td><td>0.4</td><td>9.4</td><td>93.2</td></tr><tr><td>CCAT, ρpow = 10</td><td>0.60</td><td>8.7</td><td>63.0</td><td>0.4</td><td>10.1</td><td>95.0</td></tr><tr><td>CCAT, ρpow = 12</td><td>0.62</td><td>9.4</td><td>63.0</td><td>0.3</td><td>10.1</td><td>96.6</td></tr><tr><td>CCAT, ρexp = 7</td><td>0.66</td><td>11.7</td><td>68.6</td><td>0.2</td><td>13.2</td><td>77.4</td></tr></table>

Per-attack results on MNIST. Per-attack results considering PGD-CE, as inMadry et al. (2018), our PGD-Conf and the remaining black-box attacks for all threat models, i.e., L ∞ , L 2 , L 1 and L 0 , see text. The used values are reported in the left-most column. For the black-box attacks, we take the per-example worst-case across all black-box attacks.

Table content: <table><tr><td/><td colspan="4">MNIST: Supplementary Results for Detection and Standard Settings</td><td/><td/><td/></tr><tr><td/><td/><td colspan="2">(L∞, = 0.30 during training)</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td colspan="2">Detection Setting</td><td/><td colspan="2">Standard Setting</td></tr><tr><td/><td/><td/><td>τ @99%TPR</td><td/><td/><td>τ =0</td><td/></tr><tr><td>Attack</td><td>Training</td><td>ROC AUC</td><td>Err in %</td><td>RErr in %</td><td>τ</td><td>Err in %</td><td>RErr in %</td></tr><tr><td/><td>AT</td><td>0.97</td><td>0.0</td><td>1.0</td><td>1.0</td><td>0.5</td><td>7.2</td></tr><tr><td>Worst-Case (L∞, = 0.30)</td><td>CCAT, ρexp=7</td><td>0.99</td><td>0.1</td><td>7.7</td><td>1.0</td><td>0.5</td><td>100.0</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>0.96</td><td>0.1</td><td>21.9</td><td>1.0</td><td>0.3</td><td>90.5</td></tr><tr><td/><td>AT</td><td>0.97</td><td>0.0</td><td>0.4</td><td>1.0</td><td>0.5</td><td>5.6</td></tr><tr><td>PGD Conf (L∞, = 0.30)</td><td>CCAT, ρexp=7</td><td>0.99</td><td>0.1</td><td>5.7</td><td>1.0</td><td>0.5</td><td>74.7</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>0.95</td><td>0.1</td><td>17.3</td><td>1.0</td><td>0.3</td><td>64.8</td></tr><tr><td/><td>AT</td><td>0.97</td><td>0.0</td><td>0.8</td><td>1.0</td><td>0.5</td><td>6.7</td></tr><tr><td>PGD CE (L∞, = 0.30)</td><td>CCAT, ρexp=7</td><td>1.00</td><td>0.1</td><td>4.3</td><td>1.0</td><td>0.5</td><td>100.0</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>1.00</td><td>0.1</td><td>0.1</td><td>1.0</td><td>0.3</td><td>100.0</td></tr><tr><td/><td>AT</td><td>0.98</td><td>0.0</td><td>1.0</td><td>1.0</td><td>0.5</td><td>7.2</td></tr><tr><td>Black-Box (L∞, = 0.30)</td><td>CCAT, ρexp=7</td><td>1.00</td><td>0.1</td><td>0.4</td><td>1.0</td><td>0.5</td><td>100.0</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>1.00</td><td>0.1</td><td>7.4</td><td>1.0</td><td>0.3</td><td>88.6</td></tr><tr><td/><td>AT</td><td>0.20</td><td>0.0</td><td>100.0</td><td>1.0</td><td>0.5</td><td>100.0</td></tr><tr><td>Worst-Case (L∞, = 0.40)</td><td>CCAT, ρexp=7</td><td>0.94</td><td>0.1</td><td>40.0</td><td>1.0</td><td>0.5</td><td>100.0</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>0.94</td><td>0.1</td><td>29.0</td><td>1.0</td><td>0.3</td><td>96.9</td></tr><tr><td/><td>AT</td><td>0.36</td><td>0.0</td><td>97.8</td><td>1.0</td><td>0.5</td><td>99.8</td></tr><tr><td>PGD Conf (L∞, = 0.40)</td><td>CCAT, ρexp=7</td><td>0.96</td><td>0.1</td><td>15.4</td><td>1.0</td><td>0.5</td><td>97.1</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>0.92</td><td>0.1</td><td>20.6</td><td>1.0</td><td>0.3</td><td>72.5</td></tr><tr><td/><td>AT</td><td>0.20</td><td>0.0</td><td>100.0</td><td>1.0</td><td>0.5</td><td>100.0</td></tr><tr><td>PGD CE (L∞, = 0.40)</td><td>CCAT, ρexp=7</td><td>0.97</td><td>0.1</td><td>29.6</td><td>1.0</td><td>0.5</td><td>100.0</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>1.00</td><td>0.1</td><td>2.5</td><td>1.0</td><td>0.3</td><td>100.0</td></tr><tr><td/><td>AT</td><td>0.23</td><td>0.0</td><td>100.0</td><td>1.0</td><td>0.5</td><td>100.0</td></tr><tr><td>Black-Box (L∞, = 0.40)</td><td>CCAT, ρexp=7</td><td>0.99</td><td>0.1</td><td>3.9</td><td>1.0</td><td>0.5</td><td>100.0</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>0.99</td><td>0.1</td><td>10.7</td><td>1.0</td><td>0.3</td><td>91.8</td></tr><tr><td/><td>AT</td><td>0.73</td><td>0.0</td><td>81.3</td><td>1.0</td><td>0.5</td><td>98.8</td></tr><tr><td>Worst-Case(L2, = 3)</td><td>CCAT, ρexp=7</td><td>1.00</td><td>0.1</td><td>1.4</td><td>1.0</td><td>0.5</td><td>82.6</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>1.00</td><td>0.1</td><td>0.1</td><td>1.0</td><td>0.3</td><td>22.1</td></tr><tr><td/><td>AT</td><td>0.98</td><td>0.0</td><td>0.2</td><td>1.0</td><td>0.5</td><td>3.4</td></tr><tr><td>PGD Conf (L2, = 3)</td><td>CCAT, ρexp=7</td><td>1.00</td><td>0.1</td><td>0.0</td><td>1.0</td><td>0.5</td><td>4.4</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>1.00</td><td>0.1</td><td>0.1</td><td>1.0</td><td>0.3</td><td>9.5</td></tr><tr><td/><td>AT</td><td>0.93</td><td>0.0</td><td>11.5</td><td>1.0</td><td>0.5</td><td>29.2</td></tr><tr><td>PGD CE (L2, = 3)</td><td>CCAT, ρexp=7</td><td>1.00</td><td>0.1</td><td>0.9</td><td>1.0</td><td>0.5</td><td>82.4</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>1.00</td><td>0.1</td><td>0.1</td><td>1.0</td><td>0.3</td><td>100.0</td></tr><tr><td/><td>AT</td><td>0.73</td><td>0.0</td><td>80.1</td><td>1.0</td><td>0.5</td><td>98.7</td></tr><tr><td>Black-Box (L2, = 3)</td><td>CCAT, ρexp=7</td><td>1.00</td><td>0.1</td><td>0.6</td><td>1.0</td><td>0.5</td><td>38.4</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>1.00</td><td>0.1</td><td>0.1</td><td>1.0</td><td>0.3</td><td>100.0</td></tr><tr><td/><td>AT</td><td>0.93</td><td>0.0</td><td>5.2</td><td>1.0</td><td>0.5</td><td>18.0</td></tr><tr><td>Worst-Case (L1, = 10.00)</td><td>CCAT, ρexp=7</td><td>0.96</td><td>0.1</td><td>14.7</td><td>1.0</td><td>0.5</td><td>35.8</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>1.00</td><td>0.1</td><td>1.6</td><td>1.0</td><td>0.3</td><td>17.7</td></tr><tr><td/><td>AT</td><td>0.93</td><td>0.0</td><td>4.2</td><td>1.0</td><td>0.5</td><td>14.8</td></tr><tr><td>PGD Conf (L1, = 10.00)</td><td>CCAT, ρexp=7</td><td>0.96</td><td>0.1</td><td>14.5</td><td>1.0</td><td>0.5</td><td>35.7</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>1.00</td><td>0.1</td><td>1.6</td><td>1.0</td><td>0.3</td><td>17.7</td></tr><tr><td/><td>AT</td><td>0.96</td><td>0.0</td><td>3.8</td><td>1.0</td><td>0.5</td><td>15.5</td></tr><tr><td>PGD CE (L1, = 10.00)</td><td>CCAT, ρexp=7</td><td>0.98</td><td>0.1</td><td>4.2</td><td>1.0</td><td>0.5</td><td>12.1</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>1.00</td><td>0.1</td><td>0.1</td><td>1.0</td><td>0.3</td><td>99.7</td></tr><tr><td/><td>AT</td><td>0.99</td><td>0.0</td><td>2.5</td><td>1.0</td><td>0.5</td><td>93.9</td></tr><tr><td>Worst-Case (L0, = 15)</td><td>CCAT, ρexp=7</td><td>0.99</td><td>0.1</td><td>7.8</td><td>1.0</td><td>0.5</td><td>65.9</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>0.99</td><td>0.1</td><td>7.8</td><td>1.0</td><td>0.3</td><td>88.8</td></tr><tr><td/><td>AT</td><td>0.98</td><td>0.0</td><td>0.5</td><td>1.0</td><td>0.5</td><td>5.2</td></tr><tr><td>PGD Conf (L0, = 15)</td><td>CCAT, ρexp=7</td><td>0.99</td><td>0.1</td><td>4.1</td><td>1.0</td><td>0.5</td><td>18.3</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>0.99</td><td>0.1</td><td>3.7</td><td>1.0</td><td>0.3</td><td>13.4</td></tr><tr><td/><td>AT</td><td>0.98</td><td>0.0</td><td>2.4</td><td>1.0</td><td>0.5</td><td>17.7</td></tr><tr><td>PGD CE (L0, = 15)</td><td>CCAT, ρexp=7</td><td>0.98</td><td>0.1</td><td>6.2</td><td>1.0</td><td>0.5</td><td>18.6</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>0.98</td><td>0.1</td><td>5.4</td><td>1.0</td><td>0.3</td><td>16.5</td></tr><tr><td/><td>AT</td><td>1.00</td><td>0.0</td><td>0.0</td><td>1.0</td><td>0.5</td><td>93.9</td></tr><tr><td>Black-Box (L0, = 15)</td><td>CCAT, ρexp=7</td><td>1.00</td><td>0.1</td><td>0.1</td><td>1.0</td><td>0.5</td><td>65.7</td></tr><tr><td/><td>CCAT, ρpow=10</td><td>1.00</td><td>0.1</td><td>1.4</td><td>1.0</td><td>0.3</td><td>88.9</td></tr></table>

Per-attack results on SVHN. Per-attack results considering PGD-CE, as inMadry et al. (2018), our PGD-Conf and the remaining black-box attacks for all threat models, i.e., L ∞ , L 2 , L 1 and L 0 , see text. The used values are reported in the left-most column. For the black-box attacks, we take the per-example worst-case across all black-box attacks.

Table content: <table><tr><td colspan="7">CIFAR10: Supplementary Results for Detection and Standard Settings</td><td/></tr><tr><td/><td colspan="4">(L∞, = 0.03 during training)</td><td/><td/><td/></tr><tr><td/><td/><td/><td colspan="2">Detection Setting</td><td/><td colspan="2">Standard Setting</td></tr><tr><td/><td/><td/><td colspan="2">τ @99%TPR</td><td/><td>τ =0</td><td/></tr><tr><td colspan="2">Attack Training</td><td>ROC AUC</td><td>Err in %</td><td>RErr in %</td><td>τ</td><td>Err in %</td><td>RErr in %</td></tr><tr><td>Worst-Case (L∞, = 0.03)</td><td>AT CCAT</td><td>0.64 0.60</td><td>15.1 8.7</td><td>62.3 67.9</td><td>0.3 0.4</td><td>16.6 10.1</td><td>62.7 96.7</td></tr><tr><td>PGD Conf (L∞, = 0.03)</td><td>AT CCAT</td><td>0.65 0.60</td><td>15.1 8.7</td><td>60.9 63.0</td><td>0.3 0.4</td><td>16.6 10.1</td><td>61.3 95.0</td></tr><tr><td>PGD CE (L∞, = 0.03)</td><td>AT CCAT</td><td>0.67 0.99</td><td>15.1 8.7</td><td>61.4 9.8</td><td>0.3 0.4</td><td>16.6 10.1</td><td>62.3 100.0</td></tr><tr><td>Black-Box (L∞, = 0.03)</td><td>AT CCAT</td><td>0.70 0.90</td><td>15.1 8.7</td><td>56.9 49.3</td><td>0.3 0.4</td><td>16.6 10.1</td><td>57.3 96.4</td></tr><tr><td>Worst-Case (L∞, = 0.06)</td><td>AT CCAT</td><td>0.35 0.43</td><td>15.1 8.7</td><td>93.6 91.5</td><td>0.3 0.4</td><td>16.6 10.1</td><td>93.7 99.2</td></tr><tr><td>PGD Conf (L∞, = 0.06)</td><td>AT CCAT</td><td>0.37 0.49</td><td>15.1 8.7</td><td>92.1 66.8</td><td>0.3 0.4</td><td>16.6 10.1</td><td>92.2 97.3</td></tr><tr><td>PGD CE (L∞, = 0.06)</td><td>AT CCAT</td><td>0.40 0.98</td><td>15.1 8.7</td><td>93.6 10.4</td><td>0.3 0.4</td><td>16.6 10.1</td><td>93.7 100.0</td></tr><tr><td>Black-Box (L∞, = 0.06)</td><td>AT CCAT</td><td>0.50 0.78</td><td>15.1 8.7</td><td>87.1 87.0</td><td>0.3 0.4</td><td>16.6 10.1</td><td>87.2 99.5</td></tr><tr><td>Worst-Case(L2, = 1)</td><td>AT CCAT</td><td>0.59 0.77</td><td>15.1 8.7</td><td>73.9 46.2</td><td>0.3 0.4</td><td>16.6 10.1</td><td>74.4 81.9</td></tr><tr><td>PGD Conf (L2, = 1)</td><td>AT CCAT</td><td>0.63 0.78</td><td>15.1 8.7</td><td>64.9 45.2</td><td>0.3 0.4</td><td>16.6 10.1</td><td>65.3 80.9</td></tr><tr><td>PGD CE (L2, = 1)</td><td>AT CCAT</td><td>0.61 0.95</td><td>15.1 8.7</td><td>73.9 18.9</td><td>0.3 0.4</td><td>16.6 10.1</td><td>74.6 100.0</td></tr><tr><td>Black-Box (L2, = 1)</td><td>AT CCAT</td><td>0.81 1.00</td><td>15.1 8.7</td><td>35.8 8.8</td><td>0.3 0.4</td><td>16.6 10.1</td><td>36.9 100.0</td></tr><tr><td>Worst-Case (L1, = 7.85)</td><td>AT CCAT</td><td>0.61 0.78</td><td>15.1 8.7</td><td>68.0 45.2</td><td>0.3 0.4</td><td>16.6 10.1</td><td>68.3 75.7</td></tr><tr><td>PGD Conf (L1, = 7.85)</td><td>AT CCAT</td><td>0.61 0.84</td><td>15.1 8.7</td><td>66.8 35.7</td><td>0.3 0.4</td><td>16.6 10.1</td><td>67.1 73.5</td></tr><tr><td>PGD CE (L1, = 7.85)</td><td>AT CCAT</td><td>0.71 0.96</td><td>15.1 8.7</td><td>58.5 17.4</td><td>0.3 0.4</td><td>16.6 10.1</td><td>60.5 100.0</td></tr><tr><td>Worst-Case (L0, = 10)</td><td>AT CCAT</td><td>0.80 0.98</td><td>15.1 8.7</td><td>74.1 20.9</td><td>0.3 0.4</td><td>16.6 10.1</td><td>75.9 55.7</td></tr><tr><td>PGD Conf (L0, = 10)</td><td>AT CCAT</td><td>0.74 0.98</td><td>15.1 8.7</td><td>44.3 12.5</td><td>0.3 0.4</td><td>16.6 10.1</td><td>45.1 34.8</td></tr><tr><td>PGD CE (L0, = 10)</td><td>AT CCAT</td><td>0.76 1.00</td><td>15.1 8.7</td><td>49.4 7.6</td><td>0.3 0.4</td><td>16.6 10.1</td><td>51.4 79.3</td></tr><tr><td>Black-Box (L0, = 10)</td><td>AT CCAT</td><td>0.91 0.99</td><td>15.1 8.7</td><td>69.6 17.8</td><td>0.3 0.4</td><td>16.6 10.1</td><td>75.9 99.1</td></tr></table>

Per-attack L ∞ and L 2 results on Cifar10. Per-attack results considering PGD-CE, as inMadry et al. (2018), our PGD-Conf and the remaining black-box attacks for all threat models, i.e., L ∞ , L 2 , L 1 and L 0 , see text. The used values are reported in the left-most column. For the black-box attacks, we take the per-example worst-case across all black-box attacks.

Table content: <table/>

Peer review (iclr2020):
*Edit after rebuttal* See the comment below for a response to the author*s rebuttal. I have updated my score to a weak accept. Summary ======== This paper proposes a more granular adversarial training scheme, where the model is trained to have decaying confidence as the size of the adversarial perturbations increase. The idea is quite natural and well motivated and presented. The paper suffers in its theoretical and evaluation sections though, which were both fairly confusing to me. I lean towards rejection for this paper but would consider changing my score if the presentation was improved. Comments ========= The observation that adversarial training enforces high confidence in the entirety of the epsilon ball is interesting, and the motivation for the proposed scheme is sound. Yet, while the intuition of CCAT for addressing the accuracy-robustness trade-off makes sense to me, the purported beneficial effect for perturbations outside the given epsilon ball are not as clear. The authors note in Section 3.1 that adversarial training gives no indications on how the model should behave outside the given epsilon ball, or on other threat models. While this is true, it is not clear how CCAT supposedly remedies this. By making the confidence decay within the epsilon ball, the training also doesn*t explicitly enforce any behavior outside this ball or on other threat models. So if CCAT indeed provides benefits in those settings, it isn*t clear why those gains should be expected. The exposition of the technique in Section 3.2. is very clear, and Figure 1 nicely illustrates the effect of CCAT on the class confidences during an attack. The theoretical toy example in Section 3.3. is a bit confusing, and I*m not convinced that the proof of Proposition 1 is correct. First, a classification problem parametrized by the class imbalance is a little weird in this setting. I have a few questions about the proof: - You seem to be assuming that the model*s confidence is calculated using a softmax. This assumption should be stated. - The expressions for hat{p}(y=2|x=x) and hat{p}(y=1|x=x) are confusing to me. Why are you taking a log over the softmax? These probabilities are now negative... Also, in the second step, an equality of the form e^x / (e^y + e^x) = 1+e^(y-x) is used. This is incorrect, it should be 1/(1+e^(y-x)). In the end, the values log(1+e^a) and log(1+e^(-b)) are not in [0, 1] so they cannot be probabilities. - From there on, it isn*t clear what the rest of the proof is really saying The experimental section is quite thorough and the results seem convincing. The considered adaptive attack, that optimizes (4) seems natural given that the defense thresholds on the model*s confidence. I wonder how this is optimized though: Do you try a targeted attack for every possible target other than the true class? I found the conflation of evaluation metrics quite confusing. It seems the authors know this and have done an effort to explain their choices, but I feel like this could still be improved further. For example, some of the metrics (e.g., AUC) should be maximized, while others (Err or RErr) should be minimized. This makes the reading of the result tables quite difficult.I would also suggest dropping one digit of precision in the Tables for better readability. Overall, I have some mixed feelings about this paper. The idea is simple and well explained. The motivation, theoretical analysis, and reporting of results are somewhat confusing though. Some extra work on the presentation could definitely help this paper. Some additional references ======================== - In the introduction, when mentioning the trade-off between robustness and accuracy, you should cite *Robustness May Be at Odds with Accuracy* by Tsipras et al. rather than Schmidt et al. (that paper shows that robustness may require more training data). - Also in the introduction, you could consider referencing the following papers on robustness to different threat models: * Engstrom et al., *Exploring the Landscape of Spatial Robustness* * Tramer and Boneh, *Adversarial Training and Robustness for Multiple Perturbations* - In the related work, you reference a number of works for adversarial training but not the original idea in the paper by Szegedy et al. - In Section 3.1., the observation that the epsilon-ball around training examples can cross class boundaries is further analyzed in *Excessive Invariance Causes Adversarial Vulnerability* and *Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness* by Jacobsen et al. Typos ===== p.6 enumerator -> numerator