CONFIDENCE-CALIBRATED ADVERSARIAL TRAINING and Detection: MORE ROBUST MODELS GENERALIZ-
Year: 2019

Abstract:
Adversarial training is the standard to train models robust against adversarial examples. However, especially for complex datasets, adversarial training incurs a significant loss in accuracy and is known to generalize poorly to stronger attacks, e.g., larger perturbations or other threat models. In this paper, we introduce confidence-calibrated adversarial training (CCAT) where the key idea is to enforce that the confidence on adversarial examples decays with their distance to the attacked examples. We show that CCAT preserves better the accuracy of normal training while robustness against adversarial examples is achieved via confidence thresholding, i.e., detecting adversarial examples based on their confidence. Most importantly, in strong contrast to adversarial training, the robustness of CCAT generalizes to larger perturbations and other threat models, not encountered during training. For evaluation, we extend the commonly used robust test error to our detection setting, present an adaptive attack with backtracking and allow the attacker to select, per test example, the worst-case adversarial example from multiple blackand white-box attacks. We present experimental results using L∞, L2, L1 and L0 attacks on MNIST, SVHN and Cifar10.

heading: 1 INTRODUCTION
text: Deep neural networks have shown tremendous improvements in various learning tasks including applications in computer vision, natural language processing or text processing. However, the discovery of adversarial examples, i.e., nearly imperceptibly perturbed inputs that cause mis-classification, has revealed severe security threats, as demonstrated by attacking popular computer vision services such as Google Cloud Vision (Ilyas et al., 2018a) or Clarifai (Liu et al., 2016; Bhagoji et al., 2017b). As the number of safety- and privacy-critical applications is increasing, e.g., autonomous driving or medical imaging, this problem becomes even more important.
In practice, adversarial training and its variants (Szegedy et al., 2013; Goodfellow et al., 2014; Madry et al., 2018), i.e., training on adversarial examples, can be regarded as the state-of-the-art to obtain models robust against adversarial examples. In contrast to many other defenses, and to the best of our knowledge, adversarial training has not been broken so far. However, adversarial training is known to increase test error significantly. Only on simple datasets such as MNIST (LeCun et al., 1998), adversarial training is able to preserve accuracy. This observation is typically described as a trade-off between robustness and accuracy (Tsipras et al., 2018; Stutz et al., 2019; Raghunathan et al., 2019; Zhang et al., 2019). Furthermore, while adversarial training leads to robust models for the employed threat model used during training, the obtained robustness does not translate to other threat models, e.g., other Lp-balls (Sharma & Chen, 2017; Song et al., 2018; Madry et al., 2018; Tramèr & Boneh, 2019; Li et al., 2019a; Kang et al., 2019), larger perturbations than the ones used during training or distal adversarial examples (Hein et al., 2019).
Contributions: We address both problems of adversarial training: the drop in accuracy on datasets such as Cifar10 (Krizhevsky, 2009) and the poor “generalization” of robustness to larger perturbations or unseen threat models. To this end, we introduce confidence-calibrated adversarial training (CCAT) based on the idea that the predicted confidence for an adversarial example should decrease with its distance to the attacked example. Specifically, we bias the network to predict a
convex combination of uniform and (correct) one-hot distribution for adversarial examples, which becomes uniform as the distance to the attacked test example increases. Thus, CCAT implicitly biases the network to predict the uniform distribution over classes beyond the adversarial examples seen during training. In contrast, standard adversarial training forces the network to classify adversarial examples correctly with maximal confidence but provides no guidance how to extrapolate beyond adversarial examples seen during training. Overall, CCAT allows to detect previously unseen adversarial examples, e.g., large perturbations, different Lp attacks or distal adversarial examples, by confidence thresholding, i.e., detection.
We also discuss a worst-case evaluation methodology using an adaptive attack to compare our detection-based approach with adversarial training. First, we generalize the commonly used robust test error (Madry et al., 2018; Schott et al., 2019) to our detection setting. Second, for each test example, the worst-case adversarial example, i.e., with highest confidence, is selected from multiple white- and black-box attacks. Using an adapted attack based on (Madry et al., 2018; Dong et al., 2018), augmented with a novel backtracking scheme, we show that CCAT produces models with improved accuracy on Cifar10 and robustness to various unseen adversarial examples: larger perturbations, L2, L1 and L0 attacks, distal adversarial examples and corrupted examples on MNIST(-C), (LeCun et al., 1998; Mu & Gilmer, 2019) SVHN (Netzer et al., 2011), Cifar10(-C) (Hendrycks & Dietterich, 2019). We will make our code for CCAT and evaluation publicly available.
Outline: After reviewing related work in Sec. 1.1, we discuss adversarial training (Madry et al., 2018) and introduce CCAT in Sec. 2. In Sec. 3 we present our confidence-thresholded robust test error and present experimental results in Sec. 4. Finally, we conclude in Sec. 5.

heading: 1.1 RELATED WORK
text: Adversarial Examples: Adversarial examples can roughly be divided into white-box attacks, i.e., with access to the models, its weights and gradients, e.g. (Goodfellow et al., 2014; Madry et al., 2017; Carlini & Wagner, 2017b), and black-box attacks, i.e., only with access to the output of the model, e.g. (Chen et al., 2017; Brendel & Bethge, 2017; Su et al., 2017; Ilyas et al., 2018b; Sarkar et al., 2017; Narodytska & Kasiviswanathan, 2017). Adversarial examples were also found to be transferable between similar models (Liu et al., 2016; Xie et al., 2018). While these attacks are intended to be (nearly) imperceptible, visible adversarial transformations, e.g., rotations or rotations (Engstrom et al., 2017; 2019; Dumont et al., 2018; Alaifari et al., 2018; Xiao et al., 2018), or patches (Liu et al., 2018; Lee & Kolter, 2019; Brown et al., 2017; Karmon et al., 2018; Zajac et al., 2019) have also been proposed and transferred to the physical world (Athalye et al., 2018b; Li et al., 2019b; Kurakin et al., 2016). We refer to recent surveys (Barreno et al., 2006; Yuan et al., 2017; Akhtar & Mian, 2018; Biggio & Roli, 2018a) for more details.. Recently, white-box attacks utilizing projected gradient ascent to maximize cross-entropy loss or surrogate objectives, e.g., (Madry et al., 2017; Dong et al., 2018; Carlini & Wagner, 2017b), have become standard. Instead, we directly maximize the confidence in any but the true class, similar to (Hein et al., 2019; Goodfellow et al., 2019), to attack our proposed training procedure, CCAT.
Adversarial Training: Many defenses against adversarial attacks have been proposed, see, e.g., (Yuan et al., 2017; Akhtar & Mian, 2018; Biggio & Roli, 2018b), of which some have been shown to be ineffective, e.g., in (Athalye et al., 2018a; Athalye & Carlini, 2018). Currently, adversarial training, i.e., training on adversarial examples, is the de-facto standard to obtain robust models. While adversarial training was proposed in different variants (Szegedy et al., 2013; Zantedeschi et al., 2017; Miyato et al., 2016; Huang et al., 2015; Shaham et al., 2018; Sinha et al., 2018; Madry et al., 2017; Wang et al., 2019), the formulation by Madry et al. (2017) received considerable attention and has been extended in various ways (Lee et al., 2017; Ye & Zhu, 2018; Liu et al., 2019). For example, in (Shafahi et al., 2018; Pérolat et al., 2018), adversarial training is applied to universal adversarial example, in (Cai et al., 2018), curriculum learning is used, and in (Tramèr et al., 2017; Grefenstette et al., 2018) ensemble adversarial training is proposed. Adversarial training is also used in provable/certified defenses (Kolter & Wong, 2017; Zhang & Evans, 2018). The increased sample complexity of adversarial training (Schmidt et al., 2018) has been addressed in (Lamb et al., 2019) by training on interpolated examples or in (Carmon et al., 2019; Uesato et al., 2019) using unlabeled examples. The dependence on the threat model used during training was addressed in (Tramèr & Boneh, 2019) by using multiple threat models during training. Finally, the frequently observed trade-off between accuracy and robustness has been studied both theoretically and empir-
ically (Tsipras et al., 2018; Stutz et al., 2019; Zhang et al., 2019; Raghunathan et al., 2019). CCAT differs from standard adversarial training in the used attack objective and the imposed distribution over the labels, which tends towards a uniform distribution for large perturbations.
Detection: Instead of correctly classifying adversarial examples, as intended in adversarial training, several works (Gong et al., 2017; Rouhani et al., 2018; Grosse et al., 2017; Feinman et al., 2017; Liao et al., 2018; Ma et al., 2018; Amsaleg et al., 2017; Metzen et al., 2017; Bhagoji et al., 2017a; Hendrycks & Gimpel, 2017; Li & Li, 2017; Pang et al., 2018) try instead to detect adversarial examples. However, several detectors have been shown to be ineffective against adaptive attacks, i.e., adversaries aware of the used detection mechanism (Carlini & Wagner, 2017a). Recently, the detection of adversarial examples based on their confidence, similar to our approach with CCAT, has also been discussed (Pang et al., 2018); however, the authors build on (Feinman et al., 2017) which has been shown to be ineffective (Carlini & Wagner, 2017a), as well. Goodfellow et al. (2019), instead, focus on evaluating confidence-based detection methods using adaptive, targeted attacks maximizing confidence. Our attack, introduced in Sec. 2.2, is similar in spirit, however, untargeted by nature and, thus, suitable for usage with CCAT.

heading: 2 CONFIDENCE CALIBRATION OF ADVERSARIAL EXAMPLES
text: Adversarial training, as described by Madry et al. (2018), has become standard to train robust models. However, accuracy on challenging datasets such as Cifar10 is reduced and robustness does not generalize to larger perturbations, e.g., with respect to the used L∞ constraint, or different threat models in other Lp norms. With CCAT, we intend to address both shortcomings: Instead of training models to correctly classify adversarial examples, we want to detect adversarial examples based on their confidence. In the following, we first review adversarial training, before discussing the modifications needed for our CCAT, thereby turning robustness against adversarial examples into a detection problem.
Notation: We consider a classifier f : Rd → RK where K is the number of classes and fk denotes the confidence for class k. While we assume that the cross-entropy loss L is used for training, our approach can also be used with other losses. Given x ∈ Rd, classified correctly as y = argmax k fk(x), an adversarial example x+ δ is defined as a “small” perturbation δ such that argmax k fk(x + δ) 6= y, i.e., the classifier changes its decision. The strength of the change δ is measured by some Lp-norm with p ∈ {0, 1, 2,∞}; p = ∞ is a popular choice in the literature as this leads to the smallest perturbation per feature/pixel.

heading: 2.1 ADVERSARIAL TRAINING
text: Standard adversarial training, as formulated by Madry et al. (2018), is given as the following minmax problem:
min w
E [
max ‖δ‖∞≤
L(f(x+ δ;w), y) ]
(1)
with w being the classifier’s parameters and L being the cross-entropy loss. During mini-batch training the inner maximization problem,
max ‖δ‖∞≤
L(f(x+ δ;w), y), (2)
is approximately solved. In addition to the L∞-constraint, a box constraint, i.e., x̃i = (x + δ)i ∈ [0, 1], is enforced for images. Note that maximizing the cross-entropy loss is equivalent to finding the adversarial example with minimal confidence in the true class. For neural networks, this is generally a non-convex optimization problem. In (Madry et al., 2018) the problem is tackled using projected gradient descent (PGD), which is typically initialized using a random δ with ‖δ‖∞ ≤ . During adversarial training, PGD is applied to the batch of every iteration in order to update the classifier’s parameters on the obtained adversarial examples. At test time one uses the best out of several random restarts to assess robustness.
In contrast to adversarial training as proposed in (Madry et al., 2018), which computes adversarial examples for the full batch in each iteration, others compute adversarial examples only for half the
0 0.02 0.04 0.06 0
0.2
0.4
0.6
0.8
1
Perturbation Distance
C o n
fi d
en ce
p er
C la
ss
0 0.02 0.04 0.06 0
0.2
0.4
0.6
0.8
1
Perturbation Distance
C o n
fi d
en ce
p er
C la
ss
0 0.02 0.04 0.06 0
0.2
0.4
0.6
0.8
1
Perturbation Distance
C o n
fi d
en ce
p er
C la
ss
0 0.01 0.02 0.03 0.04 0
0.2
0.4
0.6
0.8
1
Perturbation Distance
C o n
fi d
en ce
p er
C la
ss
, . . . Classes 1, . . . , 10
examples of each batch (Szegedy et al., 2013), i.e., instead of training only on adversarial examples, each batch is divided 50% into clean and 50% into adversarial examples. Compared to Eq. (1), 50%/50% adversarial training effectively minimizes
minw E [ max‖δ‖∞≤ L(f(x+ δ;w), y) ] ︸ ︷︷ ︸
50% adversarial training
+E [ L(f(x;w), y) ]︸ ︷︷ ︸ 50% “clean” training
(3)
This improves test accuracy on clean examples compared to 100% adversarial training, i.e., Eq. (1), but typically leads to worse robustness. Intuitively, by balancing both terms in Eq. (3), the trade-off between accuracy and robustness can already be optimized to some extent (Stutz et al., 2019).
There are two problems with adversarial training: First, the -ball around training examples might include examples from other classes. Then, the attack, i.e., the inner maximization in Eq. (1), will focus on these regions such that adversarial training for these examples gets “stuck”. This case is illustrated in our theoretical toy dataset in Sec. 2.3 and has also been considered in related work (Jacobsen et al., 2019b;a). Here, both 100% and 50%/50% adversarial training, cf. Eq. (1) and (3), are not able to find the Bayes optimal classifier in a fully deterministic problem, i.e., zero Bayes error. This might contribute to the observed drop in accuracy for adversarial training on, e.g., Cifar10 (Krizhevsky, 2009). Second, and more importantly, adversarial training as in Eq. (1) does not guide the classifier how to extrapolate beyond the used -ball during training. Even worse, it enforces high confidence predictions within the -ball, which clearly cannot be extrapolated to arbitrary regions. It is not surprising that adversarial examples can often be found right beyond the -ball, i.e., using larger perturbations than at training time or other threat models, e.g., other Lp norm balls.

heading: 2.2 CONFIDENCE-CALIBRATED ADVERSARIAL TRAINING
text: Addressing these problems requires only few but effective modifications as outlined in the description of our proposed confidence-calibrated adversarial training (CCAT) in Alg. 1: during training, we bias the network to predict a uniform distribution over the classes on adversarial examples that are sufficiently far away from the original training examples. Subsequently, during testing, adversarial examples can be detected by confidence thresholding – adversarial examples receive near-uniform confidence while test examples receive high-confidence. Thus, given an example x, the ideal attacker during training maximizes the confidence in any arbitrary other label k 6= y instead of minimizing the confidence in the true label y, as in Eq. (1):
max ‖δ‖∞≤ max k 6=y fk(x+ δ;w) (4)
where fk denotes the probability f assigns to class k (i.e., in practice, f is the output of the softmax layer). A similar objective but for targeted attacks has been used in (Goodfellow et al., 2019), whereas our goal is an untargeted attack and thus our objective is the maximal confidence over all
Algorithm 1 Pseudo-code of confidence-calibrated adversarial training (CCAT). The main changes compared to regular adversarial training as, e.g., described in (Madry et al., 2018) or (Szegedy et al., 2013), are in the attack (line 4) and the probability distribution over the classes (line 6,7), which becomes more uniform as distance ‖δ‖∞ increases.
1: while true do 2: choose random batch (x1, y1), . . . , (xB , yB). 3: for b = 1, . . . ,B/2 do 4: {maximize confidence in other classes than true one for adversarial example x̃b, Eq. (4):} 5: δb := argmax ‖δ‖∞≤ maxk 6=yb fk(xb + δ) 6: x̃b := xb + δb 7: {probability over classes of x̃b becomes more uniform as ‖δb‖∞ increases, Eq. (6):} 8: λ := e−ρ‖δb‖∞ or λ := (1−min(1, ‖δ‖∞/ ))ρ 9: {ỹb is convex combination of one hot and uniform distribution over the classes, Eq. (5):}
10: ỹb := λ one hot(yb) + (1−λ) K 1 11: end for 12: {corresponds to 50%/50% adversarial training, Eq. (3):} 13: update parameters using ∑B/2 b=1 L(f(x̃b), ỹb) + ∑B b=B/2 L(f(xb), yb) 14: end while
other classes. Then, during training, CCAT biases the classifier towards predicting uniform distributions on adversarial examples by using the following distribution as target in the cross-entropy loss:
p̂(k) = λpy(k) + (1− λ)u(k) k = 1, . . . ,K. (5)
Here, py(k) is the original “one-hot” distribution, i.e., py(k) = 1 iff k = y and py(k) = 0 otherwise with y being the true label, and u(k) = 1/K is the uniform distribution. Thus, we enforce a convex combination of the original label distribution and the uniform distribution which is controlled by the parameter λ. We choose λ to decrease with the distance ‖δ‖∞ of the adversarial example to the attacked example x with the intention to enforce uniform predictions when ‖δ‖∞ = δ. Then, the network is encouraged to extrapolate this uniform distribution beyond the used -ball. Even if extrapolation does not work perfectly, the uniform distribution is much more meaningful for extrapolation in the region between classes compared to high-confidence predictions as encouraged in standard adversarial training. We consider two variants of transitions, i.e., controlling the trade-off λ between one-hot and uniform distribution:
λ =e−ρ‖δ‖∞ (“exponential transition” (exp)) λ =(1−min(1, ‖δ‖∞/ ))ρ (“power transition” (pow))
(6)
This ensures that for δ = 0 we impose the original (one-hot) label. For growing δ, however, the influence of the original label decays proportional to ‖δ‖∞. The speed of decay is controlled by the parameter ρ. For the exponential transition, we always have a bias towards the true label as even for large ρ, λ will be non-zero. In case of the power transition, λ = 0 for ‖δ‖∞ ≥ , meaning a pure uniform distribution is enforced. It is important to note that in CCAT in Alg. 1 we train on 50% clean and 50% adversarial examples in each batch, as in Eq. (3). Training only on adversarial examples will not work as we the network has no incentive to predict correct labels.

heading: 2.3 CONFIDENCE-CALIBRATED ADVERSARIAL TRAINING YIELDS ACCURATE MODELS
text: Proposition 1 analyzes 100% adversarial training and its 50%/50% variant as well as our confidencecalibrated variant, CCAT, to show that there exist problems where both 100% and 50%/50% adversarial training are unable to reconcile robustness and accuracy, as recently discussed (Tsipras et al., 2018; Stutz et al., 2019; Raghunathan et al., 2019; Zhang et al., 2019). However, our CCAT is able to obtain both robustness and accuracy given that λ in Eq. (6) is chosen appropriately. Proposition 1. We consider a classification problem with two points x = 0 and x = in R with deterministic labels, that is p(y = 2|x = 0) = 1 and p(y = 1|x = ) = 1 and the problem is fully determined by the probability p0 = p(x = 0) as p(x = ) = 1 − p0. The Bayes error of this
classification problem is zero. The predicted probability distribution over the classes is p̃(y|x) = egy(x) eg1(x)+eg2(x) , where g : Rd → R2 and we assume that the function λ : R+ → [0, 1] is monotonically decreasing and λ(0) = 1. The Bayes optimal classifier for the cross-entropy loss of
• adversarial training on 100% adversarial examples, cf. Eq. (1), yields an error of min{p0, 1− p0}. • adversarial training with 50% adversarial and 50% clean examples per batch, cf. Eq. (3),
yields an error of min{p0, 1− p0}. • CCAT on 50% clean and 50% adversarial examples, cf. Alg. 1, yields zero error if λ( ) < min { p0
1−p0 , 1−p0 p0
} .

heading: 3 CONFIDENCE-THRESHOLDED ROBUST TEST ERROR (RERR)
text: CCAT runs in a two-stage process: First, examples are rejected by confidence thresholding. Ideally, all adversarial examples are rejected as CCAT encourages low-confidence adversarial examples. Second, we evaluate robustness and accuracy on the non-rejected (i.e., correctly or incorrectly classified) examples. In the first stage (i.e., confidence thresholding), we consider successful adversarial examples as negatives and correctly classified test examples as positives. Then, we report the area under the ROC curve, i.e., ROC AUC, which shows how well we can discriminate adversarial examples from correctly classified clean examples. In the second stage (i.e., after detection by confidence thresholding), we fix the threshold τ at a true positive rate (TPR) of 99% for correctly classified test samples, i.e., the network is allowed to reject at most 1% of correctly classified test examples. We note that this also improves accuracy as errors on test examples typically have lower confidence. Then, we intend to use test error (Err) and robust test error (RErr), as also used in (Madry et al., 2018), to evaluate adversarial training. However, extending RErr to take into account confidencethresholding as in our detection setting is non-trivial: for example, correctly classified examples can can have lower confidence than their corresponding adversarial examples. Thus, given confidence threshold τ , we define the confidence-thresholded robust test error RErr as follows:
RErr(τ) =
N∑ n=1 1f(xn)6=yn1c(xn)≥τ + N∑ n=1 1f(xn)=yn1f(x̃n)6=yn1c(x̃n)≥τ
N∑ n=1 1c(xn)≥τ + N∑ n=1 1c(xn)<τ1c(x̃n)≥τ1f(xn)=yn1f(x̃n)6=yn . (7)
Here, τ is the confidence-threshold fixed on a held-out validation set, {(xn, yn)}Nn=1 are test examples, c(xn) := maxk fk(xn) denotes the classifier’s confidence on xn, f(xn) := argmax k fk(xn) denotes the classifier’s decision, and x̃n are adversarial examples. The numerator counts the number of incorrectly classified test examples xn with c(xn) ≥ τ (first term) and the number of successful adversarial examples x̃n on correctly classified test examples with c(x̃n) ≥ τ (second term). The denominator counts test examples xn with c(xn) ≥ τ (first term) and the number of successful adversarial examples x̃n with c(x̃n) ≥ τ but where the corresponding test example xn has c(xn) < τ (second term). The latter takes care of the special case where adversarial examples have higher confidence than their corresponding test examples, as mentioned above and is encouraged by the objective of our attack in Eq. (4). In total this yields a correct fraction within [0, 1] and for τ = 0, Eq. (7) reduces to the “standard” RErr. The confidence-thresholded Err(τ) corresponds to taking only the first terms in both numerator and denominator.

heading: 4 EXPERIMENTS
text: We compare normal training, adversarial training (AT) and our CCAT on MNIST, (LeCun et al., 1998) SVHN (Netzer et al., 2011) and Cifar10 (Krizhevsky, 2009). We use ResNet-20 (He et al., 2016), implemented in PyTorch (Paszke et al., 2017), initialized following (He et al., 2015) and trained using stochastic gradient descent with batch size of 100 for 100 or 200 epochs (MNIST and SVHN/Cifar10, respectively). As detailed in Sec. 3, we report ROC AUC (higher is better) as well as our confidence-thresholded test error (Err; lower is better) and robust test error (RErr; lower is better); Err is reported on the full test sets, while ROC AUC and RErr are reported on the first 1000 attacked test examples. More details and experimental results can be found in Appendix B.
0 10 20 30 40 0
0.2
0.4
0.6
Iteration
E rr o r
0 10 20 30 40 0
0.2
0.4
0.6
Iteration
E rr o r
, , , . . . Attacks on Different Test Examples
Figure 2: Momentum and backtracking. Our L∞ PGD-Conf attack, cf. Sec. 4.1, with 40 iterations with momentum and backtracking (left) and without both (right). We plot the objective of Eq. (4) over iterations for 10 samples (different colors).

heading: 4.1 ATTACKS
text: White-box and adaptive attacks: We follow (Madry et al., 2018) and use projected gradient descent (PGD) to minimize the negatives of Eq. (2) and (4); we denote them as PGD-CE and PGD-Conf. The perturbation δ is initialized uniformly over direction and distance; for PGD-Conf, we additionally use δ = 0 as initialization. Different from (Madry et al., 2018), we run exactly T iterations (no early stopping) and take the perturbation corresponding to the best objective of the T iterations. In addition to momentum, as in (Dong et al., 2018), we propose to use an adaptive learning rate in combination with a backtracking scheme to improve the attacks: after each iteration, the computed update is only applied if it improves the objective; otherwise the learning rate is reduced. We also implemented PGD for L2, L1 and L0 attacks; note that for each Lp, p ∈ {∞, 2, 1, 0}, PGD-Conf has been explicitly designed to attack our CCAT. We train on L∞ attacks using T = 40 iterations with = 0.3 (MNIST) or = 0.03 (SVHN/Cifar10). For evaluation, we use T = 2000 iterations and 10 random retries for PGD-Conf; T = 200 with 50 random retries for PGD-CE. For L2, L1 and L0 attacks, we set to 3, 10, 15 (MNIST) or 1, 7.85, 10 (SVHN/Cifar10), respectively, partly following (Tramèr & Boneh, 2019). Learning rates have been tuned for each Lp and objective (i.e., CE or Conf) independently.
Black-box attacks: For the L∞ case, we additionally use random sampling, the attack by Ilyas et al. (2018a), adapted with momentum and backtracking optimizing Eq. (4) for T = 2000 iterations with 10 attempts, a variant of (Narodytska & Kasiviswanathan, 2017) with T = 2000 iterations and the “cube” attack (Andriushchenko, 2019) with T = 5000 iterations. We also consider the cube attack in the L2 setting and the attack of (Croce & Hein, 2019) in the L0 setting. The black-box attacks ensure that our defense avoids, e.g., gradient masking as described in (Athalye et al., 2018a).

heading: 4.2 WORST-CASE EVALUATION METHODOLOGY
text: Instead of reporting robustness against individual attacks, as commonly done in the literature, we use a worst-case evaluation scheme: For each individual test example, all adversarial examples from multiple attempts (i.e., multiple random restarts) and across all (white- and black-box) attacks are accumulated. Subsequently, per text example, only the adversarial example with highest confidence is kept; these are the worst-case adversarial examples for each test example. In practice, this scheme allows to accumulate results from different attacks and thus corresponds to a much stronger robustness evaluation than usual.
0 0.2 0.4 0.6 0.8 1 0
0.2
0.4
0.6
0.8
Test Confidence
AT
0 0.2 0.4 0.6 0.8 1 0
0.2
0.4
0.6
Adversarial Confidence
AT
0 0.2 0.4 0.6 0.8 1 0
0.2
0.4
0.6
0.8
1
Test Confidence
CCAT, ρpow = 10
0 0.2 0.4 0.6 0.8 1 0
0.1
0.2
0.3
0.4
Adversarial Confidence
CCAT, ρpow = 10
Figure 3: Confidence histograms on SVHN. For AT (left) and CCAT (right) with ρpow = 10 (right), we show confidence histograms corresponding to correctly classified test examples (top) and successful adversarial examples (bottom). We consider the worst-case adversarial examples across all testedL∞ attacks for = 0.03.
0 0.2 0.4 0.6 0.8 1 0
0.2
0.4
0.6
0.8
1
False Positive Rate (FPR)
T ru
e P o si ti v e R a te
(T P R )
0 0.2 0.4 0.6 0.8 1 0
0.2
0.4
0.6
0.8
1
Confidence Threshold τ
R o b
u st
T es
t E
rr o r
(R E
rr )
Normal AT CCAT, ρpow = 10
Figure 4: ROC and RErr curves on SVHN. Left: ROC curves, i.e., FPR against TPR when distinguishing correctly classified test examples from successful adversarial examples by confidence. Right: RErr against confidence threshold τ . For evaluation, we choose τ in order to obtain 99%TPR. As described in the text, RErr subsumes both Err and FPR. Curves based on worstcase examples across all tested L∞ attacks.

heading: 4.3 ABLATION STUDY
text: Momentum and Backtracking: Fig. 2 illustrates the advantage of momentum and the proposed backtracking scheme for PGD-Conf with T = 40 iterations on 10 test examples of SVHN. As shown in Fig. 2 and Tab. 1, better objective values can be achieved within fewer iterations and avoiding oscillation which is important at training time. However, also at test time, Tab. 1 shows that attacking our CCAT model effectively requires up to T = 2000 iterations and zero initialization so that RErr for τ@99%TPR stagnates. In contrast, PGD-Conf performs better against AT even for smaller T and without momentum or backtracking. Thus, finding high-confidence adversarial examples against CCAT is more difficult than for AT. Overall, this illustrates our immense effort put into attacking our proposed defense with an adapted attack, novel optimization techniques, and large number of iterations.
Evaluation Metrics: Fig. 4 shows ROC and RErr curves on SVHN, considering AT and CCAT with ρpow = 10, i.e., using the power transition from Eq. (6). The ROC curves, and the corresponding AUC value, quantify how well (successful) adversarial examples can be detected, i.e., distinguished from (correctly classified) test examples. Note our conservative choice to use the confidence threshold τ at 99%TPR, wrongly rejecting at most 1% correctly classified examples. We note that τ depends only on correctly classified test examples, not adversarial examples, and RErr implicitly includes FPR as well as Err. We focus on the confidence-thresholded variants of RErr and Err as confidence thresholding will naturally also improve results for AT, resulting in a fair comparison, i.e., AT is also allowed to benefit from our two-stage detection setting; for Err this can be seen in Tab. 2. On SVHN and Cifar10, we found the power transition with ρpow = 10 from Eq. (6) to work best. Up to ρpow = 10, performance regarding RErr for τ@99% TPR continuously improves and after ρpow = 10 performance stagnates, as shown in detail in the appendix. On MNIST, interestingly, exponential transition with ρexp = 7 performs best against L∞ attacks; we assume that the
slight bias towards the true label preserved in the exponential transition helps. However, even on MNIST, the power transition improves robustness against previously unseen attacks, e.g., L2, L1 or L0 attacks, such that we include results for both.

heading: 4.4 RESULTS
text: In Tab. 3, we report the main results of our paper, namely robustness across all evaluated L∞, L2, L1 and L0 attacks; for L∞ we include results for the used during training and an increased . While on MNIST, CCAT incurs a drop of roughly 6% in RErr, and on Cifar10 a drop of roughly 5%, both against L∞ with the same as during training, it significantly outperforms AT on SVHN, by more than 16%. On SVHN and Cifar10, Err is additionally improved – on Cifar10, the improvement is particularly significant with roughly 6%. For L∞ attacks with larger , robustness of AT degrades significantly, while CCAT is able to preserve robustness to some extent, especially on SVHN. Only on Cifar10, RErr degrades similarly to AT. In terms of generalization to previously unseen threat models, i.e., L2, L1 and L0 attacks, CCAT outperforms AT significantly. We note that on all datasets, the considered -balls for L2, L1 and L0 are not contained in the L∞-ball used during training. Overall, robustness of AT degrades significantly under threat models not seen during training, while CCAT generalizes to new threat models much better.
In Tab. 4 we also report results for detecting uniform noise and adversarial uniform noise (i.e., distal adversarial examples) based on their confidence. For the latter, we sample uniform noise and subsequently use PGD-Conf to maximize the confidence (without considering any true label in Eq. (4)) in the L∞-ball around the noise point. Although ROC AUC values are high on uniform noise, an FPR of 40% or higher shows that normal training and AT assigns high confidence to uniform noise. When
maximizing confidence on uniform noise, FPR approaches 100% on all datasets. In contrast CCAT allows to separate these attacks perfectly from test examples. Furthermore, Tab. 5 presents results on MNIST-C and Cifar10-C (Mu & Gilmer, 2019; Hendrycks & Dietterich, 2019), containing the MNIST and Cifar10 test examples with various corruptions (e.g., spatial transformations, brightness/contrast changes, noise, etc.). Again, we consider our two-stage detection approach, reporting ROC AUC for detecting corrupted examples based on confidence and confidence-thresholded Err on the corrupted examples. The results are averaged over all available corruptions (15 on MNIST, 19 on Cifar10). As can be seen, allowing confidence thresholding is beneficial in this scenario for AT and CCAT. However, CCAT, outperforms normal training and AT significantly.

heading: 5 CONCLUSION
text: We proposed confidence-calibrated adversarial training (CCAT) which addresses two problems of standard adversarial training: an apparent accuracy-robustness problem, i.e., adversarial training tends to worsen accuracy on difficult datasets; and, more importantly, the lack of “generalizable” robustness, i.e., obtaining robust models against threat models not encountered during training. CCAT achieves comparable or better robustness against the threat model at training time, i.e., L∞constrained adversarial examples, with better accuracy on Cifar10. However, in strong contrast to adversarial training, CCAT generalizes to stronger L∞ attacks as well as L2, L1, L0 attacks and distal adversarial examples by allowing detection based on confidence-thresholding. For our experiments, we further introduced a generalized robust test error adapted to our detection-setting and used a worst-case evaluation methodology, both of independent interest to the community.

heading: A PROOF OF PROPOSITION 1
text: Proposition 2. We consider a classification problem with two points x = 0 and x = in R with deterministic labels, that is p(y = 2|x = 0) = 1 and p(y = 1|x = ) = 1 and the problem is fully determined by the probability p0 = p(x = 0) as p(x = ) = 1 − p0. The Bayes error of this classification problem is zero. The predicted probability distribution over the classes is p̃(y|x) =
egy(x)
eg1(x)+eg2(x) , where g : Rd → R2 and we assume that the function λ : R+ → [0, 1] is monotonically decreasing and λ(0) = 1. The Bayes optimal classifier for the cross-entropy loss of
• adversarial training on 100% adversarial examples, cf. Eq. (1), yields an error of min{p0, 1− p0}. • adversarial training with 50% adversarial and 50% clean examples per batch, cf. Eq. (3),
yields an error of min{p0, 1− p0}. • CCAT on 50% clean and 50% adversarial examples, cf. Alg. 1, yields zero error if λ( ) < min { p0
1−p0 , 1−p0 p0
} .
Proof. First, we stress that we are dealing with three different probability distributions over the labels: the true one p(y|x), the imposed one during training p̂(y|x) and the predicted one p̃(y|x). We also note that p̂ depends on λ through Eq. (5), and λ itself is a function of the norm ‖δ‖ through Eq. (6); here, this dependence is made explicit by writing p̂(λ)(y|x). This makes the expressions for the expected loss of CCAT slightly more complicated. We first derive the Bayes optimal classifier and its loss for CCAT. We introduce
a = g1(0)− g2(0), b = g1( )− g2( ). (8)
and express the logarithm of the predicted probabilities (confidences) of class 1 and 2 in terms of these quantities.
− log p̃(y = 2|x = x) = − log ( eg2(x) eg1(x) + eg2(x) ) = log ( 1 + eg1(x)−g2(x) ) = { log ( 1 + ea ) if x = 0 log(1 + eb) if x = .
− log p̃(y = 1|x = x) = − log ( eg1(x) eg1(x) + eg2(x) ) = log ( 1 + eg2(x)−g1(x) ) = { log(1 + e−a) if x = 0 log ( 1 + e−b ) if x = .
For our confidence-calibrated adversarial training one first has to solve
δ∗x(j) = argmax ‖δ‖∞≤ max k 6=j p̃(y = k |x+ δ).
We get with the expressions above (note that we have a binary classification problem):
δ∗0(1) = argmax ‖δ‖∞≤
p̃(y = 2|0 + δ) = { 0 if a > b else. .
δ∗0(2) = argmax ‖δ‖∞≤
p̃(y = 1|0 + δ) = { if a > b 0 else. .
δ∗ (1) = argmax ‖δ‖∞≤
p̃(y = 2| + δ) = { − if a > b 0 else. .
δ∗ (2) = argmax ‖δ‖∞≤
p̃(y = 1| + δ) = { 0 if a > b − else. .
Note that in CCAT for 50% of the batch we use the standard cross-entropy loss and for the other 50% we use L ( p̂y ( λ(‖δ∗x(y)‖) ) (x), p̃(x) ) = −
2∑ j=1 p̂y ( λ(‖δ∗x(y)‖) ) (y = j |x = x) log ( p̃ ( y = j |x = x+ δ∗x(j) )) .
The corresponding expected loss is then given by E [ L ( p̂y ( λ(‖δ∗x(y)‖) ) (x), p̃(x) )] = E [ E [ L ( p̂y ( λ(‖δ∗x(y)‖) ) (x), p̃(x) ) |x ]]
=p(x = 0)E [ L(p̂y(λ(‖δ∗0(y)‖))(0), p̃(0))|x = 0 ] + p(x = )E [ L ( p̂Y ( λ(‖δ∗ (y)‖) ) ( ), p̃( ) ) |x = ] ,
where p(x = 0) = p0 and p(x = ) = 1−p(x = 0) = 1−p0. With the true conditional probabilities p(y = s |x = x) we get
E[L(p̂y(λ(δ))(x), p̃(x))|x = x] = 2∑ s=1 p(y = s |x = x)L(p̂s(λ(‖δ∗x(s)‖))(x), p̃(x))
=− 2∑ s=1 p(y = s |x = x) 2∑ j=1 p̂s(‖λ(δ∗x(s)‖))(y = j |x = x) log (p̃(y = j |x = x+ δ∗x(s)))
For our problem it holds with p(y = 2 |x = 0) = p(y = 1 |x = ) = 1 (by assumption). Thus, E[L(p̂y(λ(δ))(x), p̃(x))|x = 0] = − 2∑ j=1 p̂2(λ(‖δ∗0(2)‖))(y = j |x = 0) log (p̃(y = j |x = x+ δ∗0(2))) E[L(p̂y(λ(δ))(x), p̃(x))|x = ] = − 2∑ j=1 p̂1(λ(‖δ∗ (1)‖))(y = j |x = ) log (p̃(y = j |x = x+ δ∗ (1)))
As ‖δ∗x(y)‖ is either 0 or ‖ ‖ and λ(0) = 1 we use in the following for simplicity the notation λ = λ(‖ ‖). Moreover, note that
p̂y(λ)(y = j|x = x) = { λ+ (1−λ)K if y = j, (1−λ) K else ,
where K is the number of classes. Thus, K = 2 in our example and we note that λ+ (1−λ)2 = 1+λ 2 .
With this we can write the total loss (remember that we have half normal cross-entropy loss and half the loss for the adversarial part with the modified “labels”) as
L(a, b) =p0 [ log(1 + ea)1a≥b + 1a<b ( (1 + λ) 2 log(1 + eb) + (1− λ) 2 log(1 + e−b) )] + (1− p0) [ log(1 + e−b)1a≥b + 1a<b ( (1 + λ) 2 log(1 + e−a) + (1− λ) 2 log(1 + ea)
)] + log(1 + ea)p0 + log(1 + e −b)(1− p0),
where we have omitted a global factor 12 for better readability (the last row is the cross-entropy loss). We distinguish two sets in the optimization. First we consider the case a ≥ b. Then it is easy to see that in order to minimize the loss we have a = b.
∂aL = 2 ea
1 + ea p0 −
e−a
1 + e−a (1− p0)
This yields ea = 1−p0p0 or a = log ( 1−p0 p0 ) and the minimum for a ≥ b is attained on the boundary of the domain of a ≤ b. The other case is a ≤ b. We get
∂aL = [ (1 + λ)
2
−e−a
1 + e−a + (1− λ) 2 ea 1 + ea
] (1− p0) + p0 ea
1 + ea ∂bL = [ (1 + λ)
2
eb
1 + eb + (1− λ) 2 −eb 1 + e−b
] p0 + (1− p0) −e−b
1 + e−b
This yields the solution
a∗ = log
( 1+λ 2 (1− p0)
p0 + 1−λ 2 (1− p0)
) , b∗ = log ( 1−λ 2 p0 + (1− p0)
1+λ 2 p0
)
It is straightforward to check that a∗ < b∗ for all 0 < p0 < 1, indeed we have
1+λ 2 (1− p0)
p0 + 1−λ 2 (1− p0)
= 1+λ 2 (1− p0)
p0 1+λ 2 + 1−λ 2
= 1− p0 − (1−λ)2 (1− p0)
p0 1+λ 2 + 1−λ 2
< 1−λ 2 p0 + (1− p0)
1+λ 2 p0
if 0 < p0 < 1 and note that λ < 1 by assumption. We have a∗ < 0 and thus g2(0) > g1(0) (Bayes optimal decision for x = 0) if
1 > 1− p0 p0 λ,
and b∗ > 0 and thus g1( ) > g2( ) (Bayes optimal decision for x = ) if
1 > p0
1− p0 λ.
Thus we recover the Bayes classifier if λ < min {1− p0
p0 , p0 1− p0
} .
Now we consider the approach by Madry et al. (2018) with 100% adversarial training. The expected loss can be written as
E [ max‖δ‖≤∞ L(y, f(x+ δ) ] = E [ E [ max‖δ‖≤∞ L(y, f(x+ δ)|x ]] =p(x = 0)p(y = 2|x = x)max { − log ( p̃(y = 2|x = 0) ) ,− log ( p̃(y = 2|x = )
)} + (1− p(x = 0))p(y = 1|x = x)max { − log ( p̃(y = 1|x = 0) ) ,− log ( p̃(y = 1|x = )
)} =p(x = 0)max { − log ( p̃(y = 2|x = 0) ) ,− log ( p̃(y = 2|x = )
)} + (1− p(x = 0))max { − log ( p̃(y = 1|x = 0) ) ,− log ( p̃(y = 1|x = )
} This yields in terms of the parameters a, b the expected loss:
L(a, b) =max { log(1 + ea), log(1 + eb) } p0 +max { log(1 + e−a), log(1 + e−b) } (1− p0)
The expected loss is minimized if a = b as then both maxima are minimal. This results in the expected loss
L(a) = log(1 + ea)p0 + log(1 + e −a)(1− p0). The critical point is attained at a∗ = b∗ = log (
1−p0 p0
) . This is never Bayes optimal for all 0 < p0 <
1. Note that b∗ = a∗ > 0 if p0 < 1 − p0 and thus the error is given by p0, similar a∗ = b∗ < 0 if 1− p0 < p0 and thus the error is given by 1− p0. We get an error of min{p0, 1− p0} of the Bayes optimal solution of 100% adversarial training.
Next we consider 50% adversarial plus 50% clean training. The expected loss E [ max‖δ‖≤∞ L(y, f(x+ δ) ] + E [ L(y, f(x+ δ) ] ,
can be written as L(a, b) =max { log(1 + ea), log(1 + eb) } p0
+max { log(1 + e−a), log(1 + e−b) } (1− p0)
+ log(1 + ea)p0 + log(1 + e −b)(1− p0)
We make a case distinction. If a ≥ b, then the loss reduces to
L(a, b) = log(1 + ea)p0 + log(1 + e −b)(1− p0)
+ log(1 + ea)p0 + log(1 + e −b)(1− p0)
≥ L(a, a) = 2 log(1 + ea)p0 + 2 log(1 + e −a)(1− p0)
Solving for the critical point yields a∗ = log (
1−p0 p0
) = b∗. Next we consider the set a ≤ b. This
yields the loss
L(a, b) = log(1 + eb)p0 + log(1 + e −a)(1− p0)
+ log(1 + ea)p0 + log(1 + e −b)(1− p0) Solving for the critical point yields a∗ = log (
1−p0 p0
) = b∗ which fulfills a ≤ b. Actually, it
coincides with the solution found already for 100% adversarial training. One does not recover the Bayes classifier for any 0 < p0 < 1.
Moreover, we note that
a∗ = b∗ = { > 0 if p0 < 12 , < 0 if p0 > 12 .
Thus, we classify x = 0 correctly, if p0 > 12 and x = correctly if p0 < 1 2 . Thus the error is given by min{p0, 1− p0}.

heading: B EXPERIMENTS
text: We give additional details on our experimental setup, specifically regarding attacks, training and the used evaluation metrics. Afterwards, we include additional experimental results, including ablation studies, results for 98% true positive rate (TPR), results per attack, results per corruption on MNISTC (Mu & Gilmer, 2019) and Cifar10-C (Hendrycks & Dietterich, 2019) and a comparison with (Pang et al., 2018).
Algorithm 2 Pseudo-code for the used projected gradient descent (PGD) procedure to maximize Eq. (9) or Eq. (10) subject to the constraints x̃i = xi + δi ∈ [0, 1] and ‖δ‖∞ ≤ ; in practice, the procedure is applied on batches of inputs. The algorithm is also easily adapted to work with a L2-norm; only the projections on line 6 and 24 needs to be adapted.
input: example x with label y input: number of iterations T input: learning rate γ, momentum β, learning rate factor α input: initial δ(0), e.g., Eq. (11) or δ(0) = 0
1: v := 0 {best objective achieved} 2: x̃ := x+ δ(0) {best adversarial example} 3: g(−1) := 0 {accumulated gradients} 4: for t = 0, . . . , T do 5: {projection onto L∞ -ball and on [0, 1]:} 6: clip δ(t)i to [− , ] 7: clip xi + δ (t) i to [0, 1] 8: {forward and backward pass to get objective and gradient:} 9: v(t) := F(x+ δ(t), y)
10: g(t) := sign ( ∇δ(t)F(x+ δ(t), y) ) 11: {keep track of adversarial example resulting in best objective:} 12: if v(t) > v then 13: v := v(t) 14: x̃ := x+ δ(t) 15: end if 16: {iteration T is only meant to check whether last update improved objective:} 17: if t = T then 18: break 19: end if 20: {integrate momentum term:} 21: g(t) := βg(t−1) + (1− β)g(t) 22: {“try” the update step and see if objective increases:}

heading: 23: δ̂(t) := δ(t) + γg(t)
text: 24: clip δ̂(t)i to [− , ] 25: clip xi + δ̂ (t) i to [0, 1] 26: v̂(t) := F(x+ δ̂(t), y) 27: {only keep the update if the objective increased; otherwise decrease learning rate:} 28: if v̂(t) ≥ v(t) then 29: δ(t+1) := δ̂(t) 30: else 31: γ := γ/α 32: end if 33: end for 34: return x̃, ṽ
B.1 ATTACKS
Complementary to the description of the projected gradient descent (PGD) attack by Madry et al. (2018) and our adapted attack, we provide a detailed algorithm in Alg. 2. We note that the objective maximized in (Madry et al., 2018) is
F(x+ δ, y) = L(f(x+ δ;w), y) (9) where L denotes the cross-entropy loss, f(·;w) denotes the model and (x, y) is an input-label pair from the test set. Our adapted attack, in contrast, maximizes
F(x+ δ, y) = maxk 6=y fk(x+ δ;w). (10) Note that the maximum over labels, i.e., maxk 6=y is explicitly computed during optimization; this means that in contrast to (Goodfellow et al., 2019), we do not run K targeted attack and subse-
quently take the maximum-confidence one, where K is the number of classes. We denote these two variants as PGD-CE and PGD-Conf, respectively. Deviating from (Madry et al., 2018), we initialize δ uniformly over directions and norm (instead of uniform initialization over the volume of the -ball):
δ = u δ′
‖δ′‖∞ , δ′ ∼ N (0, I), u ∼ U(0, 1) (11)
where δ′ is sampled from a standard Gaussian and u ∈ [0, 1] from a uniform distribution. We also consider zero initialization, i.e., δ = 0. For random initialization we always consider multiple attempts, 10 for PGD-Conf and 50 for PGD-CE, with zero initialization, we use only 1 attempt.
Both PGD-CE and PGD-Conf can also be applied using the L2, L1 and L0 norms following the description above. Then, gradient normalization, i.e., the signum in Line 10 in Alg. 2 for the L∞ norm, the projection, and the the initialization need to be adapted. For the L2 norm, the gradient is normalized by dividing by the L2 norm; for the L1 norm only the 1% largest values (in absolute terms) of the gradient are kept and normalized by their L1 norm; and for the L0 norm, the gradient is normalized by the L1 norm. Projection is easily implemented for the L2 norm, while we follow the algorithm of Duchi et al. (2008) for the L1 project; for the L0 projection, only the -largest values are kept. Similarly, initialization for L2 and L1 are simple by randomly choosing a direction (as in Eq. (11)) and then normalizing by their norm. For L0, we randomly choose pixels with probability 2/3 /(HWD) and set them to a uniformly random values u ∈ [0, 1], whereH×W ×D is the image size. In experiments, we found that tuning the learning rate for PGD with L1 and L0 constraints (independent of the objective, i.e., Eq. (9) or Eq. (10)) is much more difficult. Additionally, PGD using the L0 norm seems to get easily stuck in sub-optimal local optima.
Alg. 2 also gives more details on the employed momentum and backtracking scheme. These two “tricks” add two additional hyper-parameters to the number of iterations T and the learning rate γ, namely the momentum parameter β and the learning rate factor α. After each iteration, the computed update, already including the momentum term, is only applied if this improves the objective. This is checked through an additional forward pass. If not, the learning rate is divided by α, and the update is rejected. Alg. 2 includes this scheme as an algorithm for an individual test example x with label y for brevity; however, extending it to work on batches, which is used in our paper, is straight-forward. In practice, for PGD-CE, with T = 200 iterations, we use β = 0.9 and α = 1.25; for PGD-Conf, with T = 2000 iterations, we use β = 0.9 and α = 1.1.
We also give more details on the used black-box attacks. For random sampling, we apply Eq. (11) T = 5000 times in order to maximize Eq. (10). We also implemented the black-box attack of Ilyas et al. (2018a) using a population of 50 and variance of 0.1 for estimating the gradient in Line 10 of Alg. 2; a detailed algorithm is provided in (Ilyas et al., 2018a). We use a learning rate of 0.001 (note that the gradient is signed, as in (Madry et al., 2018)) and also integrated a momentum with β = 0.9
and backtracking as described in Alg. 2 with α = 1.1 and T = 2000 iterations. We use zero and random initialization; in the latter case we allow 10 random retries. For the simple black-box attack we follow the algorithmic description in (Narodytska & Kasiviswanathan, 2017) considering only axis-aligned perturbations of size per pixel. We run the attack for T = 2000 iterations and allow 10 random retries. Finally, we use a variant of the cube attack proposed in (Andriushchenko, 2019). We run the attack for T = 5000 iterations with a probability of change of 0.05. We emphasize that, except for (Ilyas et al., 2018a), these attacks are not gradient-based and do not approximate the gradient.
B.2 TRAINING
As described in Sec. 4, we follow the ResNet-20 architecture by He et al. (2016) implemented in PyTorch (Paszke et al., 2017). For training we use a batch size of 100 and train for 100 and 200 epochs on MNIST and SVHN/Cifar10, respectively: this holds for normal training, adversarial training (AT) and confidence-calibrated adversarial training (CCAT). For the latter two, we use PGD-CE and PGD Conf, respectively, for T = 40 iterations, momentum and backtracking (β = 0.9, α = 1.5). For PGD-CE we use a learning rate of 0.05, 0.01 and 0.005 on MNIST, SVHN and Cifar10. For PGD-Conf we use a learning rate of 0.05. For training, we use standard stochastic gradient descent, starting with a learning rate of 0.1 on MNIST/SVHN and 0.075 on Cifar10. The learning rate is multiplied by 0.95 after each epoch. We do not use weight decay; but the network includes batch normalization (Ioffe & Szegedy, 2015). On SVHN and Cifar10, we use random cropping, random flipping (only Cifar10) and contrast augmentation during training. We always train on 50% clean and 50% adversarial examples per batch, i.e., each batch contains both clean and adversarial examples which is important when using batch normalization.
B.3 EVALUATION METRICS
For reproducibility and complementing the discussion in the main paper, we describe the used evaluation metrics and evaluation procedure in more detail. Adversarial examples are computed on the first 1000 examples of the test set; the used confidence threshold is computed on the last 1000 examples of the test set; test errors are computed on all test examples minus the last 1000. As we consider multiple attacks, and some attacks allow multiple random attempts, we always consider the worst case adversarial example per test example and across all attacks/attempts; the worst-case is selected based on confidence.
ROC AUC: To compute ROC curves, and the area under the curve, i.e., ROC AUC, we define negatives as successful adversarial examples (on correctly classified test examples) and positives as the corresponding correctly classified test examples. The ROC AUC as well as the curve itself can easily be calculated using (Pedregosa et al., 2011). Practically, the generated curve could be used to directly estimate a threshold corresponding to a pre-determined true positive rate (TPR). However, this requires interpolation; after trying several constant interpolation schemes, we concluded that the
results are distorted significantly, especially for TPRs close to 100%. Thus, we followed a simpler scheme as mentioned above: on a held out validation set of size 1000 (the last 1000 samples of the test set), we sorted the corresponding confidences, and picked the confidence threshold in order to obtain the desired TPR, e.g., 99%, on this set exactly.
MNIST (PGD-Conf, L∞ with = 0.3)
Robust Test Error: For clarity, we repeat our confidence-integrated definition of the robust test error:
RErr(τ) =
N∑ n=1 1f(xn)6=yn1c(xn)≥τ + N∑ n=1 1f(xn)=yn1f(x̃n)6=yn1c(x̃n)≥τ
N∑ n=1 1c(xn)≥τ + N∑ n=1 1c(xn)<τ1c(x̃n)≥τ1f(xn)=yn1f(x̃n)6=yn , (12)
As described in the main paper, Eq. (12) quantifies the performance of a classifier with reject-option at a specific confidence threshold τ on both clean and adversarial examples. The regular robust test error, corresponding to Eq. (12) at τ = 0, can also be written as
RErr(0) =
N∑ n=1 1f(xn)6=yn + N∑ n=1 1f(xn)=yn1f(x̃n)6=yn
N∑ n=1 1
. (13)
The robust test error is easy to handle as it quantifies overall performance, including generalization on clean examples (first term in Eq. (13)) and robustness on adversarial examples corresponding to correctly classified clean examples (second term in Eq. (13)). Additionally, the robust test error lies in [0, 1]. Generalizing Eq. (13) to τ > 0 is non-trivial due to the following considerations: First, when integrating a confidence threshold, the reference set (i.e., the denominator) needs to be adapted. Otherwise, the metric will not reflect the actual performance after thresholding, i.e., rejecting examples – specifically, the values are not comparable across different confidence thresholds τ . Then, Eq. (13) might be adapted as follows:
RErr(τ) =
N∑ n=1 1f(xn)6=yn1c(xn)≥τ + N∑ n=1 1f(xn)=yn1f(x̃n)6=yn1c(x̃n)≥τ
N∑ n=1 1c(xn)≥τ
. (14)
Note that in the nominator, we only consider clean and adversarial examples with confidence above the threshold τ , i.e., c(xn) ≥ τ and c(x̃n) ≥ τ . Similarly, the denominator has been adapted accordingly, as after rejection, the reference set changes, too. However, this formulation has a problem when adversarial examples obtain higher confidence than the original clean examples. Thus, second, we need to account for the case where c(x̃n) > τ ≥ c(xn), i.e., an adversarial example obtains a higher confidence than the corresponding clean example. Then, the nominator may exceed the denominator, resulting in a value larger than one. To mitigate this problem, we need to include exactly this case in the denominator. Formalizing this case, we see that it corresponds exactly to the second term in the denominator or Eq. (12):
N∑ n=1 1c(xn)<τ1c(x̃n)≥τ1f(xn)=yn1f(x̃n)6=yn (15)
Overall, with Eq. (12) we obtain a metric that lies within [0, 1], is comparable across thresholds τ and, for τ = 0, reduces to the regular robust test error as used in related work (Madry et al., 2018).
B.4 ABLATION STUDY
Complementing Sec. 4, we include ablation studies for MNIST, SVHN and Cifar10; for our attack in Tab. 6 and training in Tab. 9.
Regarding the proposed attack, i.e., PGD-Conf, using momentum and backtracking, Tab. 6 shows that the main observations for SVHN can be transferred to MNIST and Cifar10. Only the improvement of backtracking and momentum over just using momentum cannot be confirmed on MNIST. Note that for fair comparison, T iterations with backtracking are equivalent to 3/2T iterations without backtracking; which is why we include results for T = 60 and T = 300. However, the importance of using enough iterations, i.e., T = 2000, and zero initialization to attack CCAT is still clearly visible. Interestingly, against AT on SVHN, more iterations are also beneficial, while this is not required on MNIST and Cifar10.
Tab. 9 also reports results for CCAT with different transitions, cf. Eq. (6), and values for ρ. As mentioned before, on SVHN and Cifar10, power transition with ρpow = 10 works best; for larger ρ performance stagnates. It is also important to note that the power transition does not preserve a bias towards to true label, i.e., for the maximum possible perturbation (‖δ‖∞ = ), Eq. (6) forces the network to predict a purely uniform distribution. This is in contrast to the exponential transition, where the true one-hot distribution always receives a non-zero weight. On MNIST, we found this to work considerably better.
B.5 ANALYSIS
For further analysis, Fig. 5 shows confidence histograms for AT and CCAT on MNIST and Cifar10. The confidence histograms for CCAT reflect the expected behavior: adversarial examples are mostly successful in changing the label, which is supported by high RErr values for confidence threshold τ = 0, but their confidence is pushed towards a uniform distributions. For AT, in contrast, successful adversarial examples – fewer in total – generally obtain high confidence; this results in confidence
thresholding being not effective for AT. This behavior, however, is less pronounced on MNIST. Here, the exponential transition results in adversarial examples with confidence slightly higher than uniform confidence, i.e., 0.1. This might be the result of preserving a bias towards the true label through Eq. (6). In fact, for lower ρ, we found that this behavior is pronounced, until, for very small ρ, the behavior of AT is obtained.
In Fig. 7, we plot the probabilities for all ten classes along an adversarial direction. We note that these directions do not necessarily correspond to successful adversarial examples. Instead, we chose the first 10 test examples on SVHN. The adversarial examples were obtained using our L∞ PGDConf attack with T = 2000 iterations and zero initialization for = 0.03. For AT, we usually observe a change in predictions along these directions; some occur within ‖δ‖∞ ≤ , corresponding to successful adversarial examples, some occur for ‖δ‖∞ > , corresponding to unsuccessful adversarial examples (within ). However, AT always assigns high confidence. Thus, when allowing larger adversarial perturbations at test time, robustness of AT reduces significantly. For CCAT, in
contrast, there are only few such cases; more often, the model achieves a near uniform prediction for small ‖δ‖∞ and extrapolates this behavior beyond the -ball used for training. On SVHN, this behavior successfully allows to generalize the robustness to larger adversarial perturbations. Furthermore, these plots illustrate why using more iterations at test time, and using techniques such as momentum and backtracking, are necessary to find adversarial examples as the objective becomes more complex compared to AT.
Fig. 8 shows concrete adversarial examples against AT and CCAT on MNIST and SVHN. Among the first 200 test examples, we picked the first six examples where AT is mistaken and (non-overlapping) six examples where CCAT has difficulties, i.e., assigns high confidence. On SVHN, for example, CCAT is able to detect adversarial examples with low confidence where AT assigns incorrect classes. On MNIST, in contrast, CCAT, cannot recover mistakes made by AT. Similarly, there exist examples where AT assigns the correct class while CCAT assigns the wrong class with very high confidence. On MNIST, we additionally found adversarial examples against CCAT to represent more “localized” changes in theL∞ norm compared to AT. Overall, however, we did not find any systematic difference between strong adversarial against AT and CCAT.
In Fig. 9, on MNIST, we additionally illustrate the advantage of CCAT with respect to the toy example in Proposition 1. Here, we consider the case where the -balls of two training or test examples (in different classes) overlap. As we show in Proposition 1, adversarial training is not able to handle such cases, resulting in the trade-off between accuracy in robustness reported in the literature (Tsipras et al., 2018; Stutz et al., 2019; Raghunathan et al., 2019; Zhang et al., 2019). This is because adversarial training enforces high-confidence predictions on both -balls (corresponding to different classes), resulting in an obvious conflict. CCAT, in contrast, enforces uniform predictions throughout the largest parts of both -balls, resolving the conflict. Finally, Fig. 9 also shows that CCAT is indeed able to extrapolate the uniform predictions beyond the -ball used during training.
B.6 RESULTS
In the following, we present and discuss complementary results corresponding to the main results of our paper as presented in Tab. 3 and 4. To this end, we consider requiring only 98% TPR instead of 99% TPR, and most importantly, break down our analysis by the different white-box and black-box attacks used.
Main results for 98% TPR: Tab. 7 reports results (corresponding to Tab. 3) requiring only 98%TPR. This implies, that compared to 99%TPR, up to 1% more correctly classified test examples can be rejected. For relatively simple tasks such as MNIST and SVHN, where Err values are low, this is a significant “sacrifice”. However, as can be seen, robustness in terms of RErr only improves slightly. We found that the same holds for 95%TPR, however, rejecting more than 2% correctly classified examples seems prohibitive large for the considered datasets.
Per-Attack Results: Finally Tab. 10, 11 and 12 we break down the results of Tab. 3 regarding the used attacks. For simplicity we focus on PGD-CE and PGD-Conf while reporting the used blackbox attacks together, i.e., taking the worst-case adversarial examples across all black-box attacks. For comparison, we also include non-thresholded Err and RErr for the results from the main paper. On MNIST, where AT performs very well in practice, it is striking that for 4/3 = 0.4 even blackbox attacks are able to reduce robustness completely, resulting in high RErr. This observation also transfers to SVHN and Cifar10. For CCAT, black-box attacks are only effective on Cifar10, where they result in roughly 87% RErr with τ@99%TPR. For the L2, L1 and L0 attacks we can make similar observations, except on MNIST, where AT seems similarly robust against L1 and L0 attacks as CCAT. Across all Lp norms, it can also be seen that PGD-CE performs significantly worse against our CCAT compared to AT, which shows that it is essential to optimize the right objective to evaluate the robustness of defenses and adversarially trained models, i.e., maximize confidence against CCAT
Results on Corrupted MNIST/Cifar10: We also conducted experiments on MNIST-C (Mu & Gilmer, 2019) and Cifar10-C (Hendrycks & Dietterich, 2019). These datasets are variants of MNIST and Cifar10 that contain common perturbations of the original images obtained from various types of noise, blur or transformations; examples include zoom or motion blue, Gaussian and shot noise, rotations, translations and shear. Tab. 14 presents the per-corruption results on MNISTC and Cifar10-C. Here, all includes all corruptions and mean reports the average results across
all corruptions; note that, due to the thresholding, leaving different numbers of corrupted examples after detection, the distinction between all and mean is meaningful. Striking is the performance of CCAT on noise corruptions such as gaussian noise or shot noise. Here, CCAT is able to detect 100% of the corrupted examples (cf. the true negative rate (TNR) in Tab. 14), resulting in a thresholded Err of 0%. This is in stark contrast to AT, exhibiting a Err of roughly 15% after detection on Cifar10-C. On the remaining corruptions, CCAT is able to perform slightly better than AT, which is often due to higher detection rate, i.e., higher ROC AUC. On, Cifar10, the generally lower Err of CCAT also contributes to the results. Overall, this illustrates that CCAT is able to preserve the inductive bias of predicting near-uniform distribution on noise similar to L∞ adversarial examples as seen during training.
Comparison with (Pang et al., 2018): We also compare AT and CCAT with the approach by Pang et al. (2018). We implemented the reverse cross-entropy error in PyTorch following the official code1. The reverse cross-entropy loss encourages the non-maximum predictions (i.e., in the case of correct classification, the probabilities of the “other” classes) to be uniform. Then, adversarial examples are detected based on statistics of the predicted distributions. We used the non-maximum element entropy (nonME) detector instead of the K-density detector (Feinman et al., 2017) which has been shown to be ineffective against adaptive attacks in (Carlini & Wagner, 2017a). We note that (Pang et al., 2018) do not train on adversarial examples as AT or CCAT. Our evaluation metrics need not be changed; however, thresholding is done on the nonME detector instead of on the confidence, as for CCAT. In Tab. 8, it can be seen, that the approach works considerably well on MNIST, regarding both L∞ and L2 attacks. Again, we report the worst-case results across PGDCE, PGD-Conf and the considered black-box attacks. However, on SVHN and Cifar10, we are able to break to robustness of the approach, reaching 92% (thresholded) RErr on SVHN and 99% RTE on Cifar10. On SVHN, the models performs slightly better against L2 attacks, with 81%, but can not be considered robust anymore.
1https://github.com/P2333/Reverse-Cross-Entropy

Review summaries:
This paper proposes the "confidence-calibrated adversarial training (CCAT)" to train robust DNNs against adversarial examples. The idea is to perform adversarial training with gradually smoothed network predictions (in an exponential or power rate to perturbation size). This new approach seems can help detection and defense at the same time to some extent. While the idea is quite interesting, the paper is poorly written and many times hard to read.