AVOIDING ROBUST MISCLASSIFICATIONS
Year: 2021

Abstract:
While current methods for training robust deep learning models optimize robust accuracy, in practice, the resulting models are often both robust and inaccurate on numerous samples, providing a false sense of safety for those. Further, they significantly reduce natural accuracy, which hinders the adoption in practice. In this work, we address both of these challenges by extending prior works in three main directions. First, we propose a new training method that jointly maximizes robust accuracy and minimizes robust inaccuracy. Second, since the resulting models are trained to be robust only if they are accurate, we leverage robustness as a principled abstain mechanism. Finally, this abstain mechanism allows us to combine models in a compositional architecture that significantly boosts overall robustness without sacrificing accuracy. We demonstrate the effectiveness of our approach to both empirical and certified robustness on six recent state-of-the-art models and using several datasets. For example, on CIFAR-10 with ε∞ = 1/255, it successfully enhanced the robust accuracy of a state-of-the-art standard trained model from 26% to 86% while only marginally reducing its natural accuracy from 97.8% to 97.6%.

text: While current methods for training robust deep learning models optimize robust accuracy, in practice, the resulting models are often both robust and inaccurate on numerous samples, providing a false sense of safety for those. Further, they significantly reduce natural accuracy, which hinders the adoption in practice. In this work, we address both of these challenges by extending prior works in three main directions. First, we propose a new training method that jointly maximizes robust accuracy and minimizes robust inaccuracy. Second, since the resulting models are trained to be robust only if they are accurate, we leverage robustness as a principled abstain mechanism. Finally, this abstain mechanism allows us to combine models in a compositional architecture that significantly boosts overall robustness without sacrificing accuracy. We demonstrate the effectiveness of our approach to both empirical and certified robustness on six recent state-of-the-art models and using several datasets. For example, on CIFAR-10 with ε∞ = 1/255, it successfully enhanced the robust accuracy of a state-of-the-art standard trained model from 26% to 86% while only marginally reducing its natural accuracy from 97.8% to 97.6%.

heading: 1 INTRODUCTION
text: In recent years, there has been a significant amount of work that studies and improves both adversarial (Szegedy et al., 2013; Goodfellow et al., 2014; Carlini & Wagner, 2017; Croce & Hein, 2020; Madry et al., 2018) and certified robustness (Balunovic & Vechev, 2019; Cohen et al., 2019; Salman et al., 2019; Xu et al., 2020; Zhai et al., 2020; Zhang et al., 2019b) of neural networks. However, currently, there are two key limitations that hinder the wider adoption of robust models in practice.
Existing Models are Robustly Inaccurate First, despite substantial progress in training robust models, existing works usually only report robust accuracy, i.e., samples for which the model robustly predicts the correct label. Meanwhile, the issue of robust inaccuracy, i.e., samples that are robustly misclassified with a wrong label, is usually not even reported (we formally define robust inaccuracy in Equation 3). This is especially problematic for safety-critical models, where the robustness can be mistakenly used as a safety argument. We quantify the severity of this issue in Table 1, by evaluating recent state-of-the-art robust models. As can be seen, recent models contain up to 15% of robust inaccurate samples and the ratio of such samples worsens with smaller perturbation regions.
x1
x2 0 12
rob((F, SERI), D) = 91.1% rob((F, SERI), D) = 100.0%
std
0 1 2
x1
x2 0 1
rob((F, SERI), D) = 100.0% rob((F, SERI), D) = 98.4%
TRADES
0 1 2
x1
x2 0 1
rob((F, SERI), D) = 98.4% rob((F, SERI), D) = 100.0%
ERA (ours) 0 1 2
robust & accurate non-robust & accurate robust & inaccurate non-robust & inaccurate
robust & accurate
non-robust & accurate
robust & inaccurate
non-robust & inaccurate Lstd LTRADES (ours) LERA
Figure 1: Decision regions for models trained via standard trainingLstd, adversarial trainingLTRADES (Zhang et al., 2019a), and our training LERA (Equation 5). In this case, our training achieves the same robust accuracy as LTRADES but avoids all robust inaccurate samples by making them non-robust. Note that all three models predict over all three classes, however, the decision regions for class 2 of the LTRADES and LERA trained models happen to be too small to be visible. The considered model architecture and the training hyperparameters are provided in Appendix A.2.
Robustness vs Accuracy Tradeoff Second, existing robust training methods improve the model robustness, but they also typically degrade the standard accuracy on unperturbed inputs. To address this limitation, a number of recent works study this issue in detail and propose new methods to mitigate it (Mueller et al., 2020; Raghunathan et al., 2020; Yang et al., 2020; Stutz et al., 2019).
Our Work In this work, we advance the line of work that aims to boost robustness without sacrificing accuracy, but we approach the problem from a new perspective – by avoiding robust inaccuracy.
Concretely, we propose a new training method that jointly maximizes robust accuracy while minimizing robust inaccuracy. We illustrate the effect of our training on a synthetic dataset (described in Appendix A.2) in Figure 1, showing the decision boundaries of three models, trained using standard training Lstd, adversarial training LTRADES (Zhang et al., 2019a), and our training LERA (Equation 5). First, observe that while the Lstd trained model achieves 100% accuracy, only 91.1% of these samples are robust (and accurate). When using LTRADES, we can observe the robustness vs accuracy tradeoff – the robust accuracy improves to 98.4% at the expense of 1.6% (robust) inaccuracy. In contrast, using our LERA, we retain the high robust accuracy of 98.4% but avoid all robust inaccurate samples by appropriately shifting the decision boundary, rendering them non-robust.
Second, since our models are trained to be robust only if they are accurate, we leverage robustness as a principled abstain mechanism. This abstain mechanism then allows us to combine models in a compositional architecture that significantly boosts overall robustness without sacrificing accuracy. Concretely, in Figure 1, we would define a selector model that abstains on all non-robust samples. Then, the abstained (non-robust) samples are evaluated by the standard trained model Lstd, while the selected samples are evaluated using the robust model LERA. This allows us to achieve the best of both models – high robust accuracy (98.4%), high natural accuracy (100%), and no robust inaccuracy.
We show the practical effectiveness of our approach by instantiating over several datasets and existing robust models for both empirical and certified robustness. Our evaluations show that our method effectively reduces robust and inaccurate samples by up to 97.28%. Further, our approach significantly improves robustness for CIFAR-10, CIFAR-100, MTSD and SBB datasets by 60.3%, 38.8%, 29.2% and 37.7%, respectively, while simultaneously decreasing natural accuracy by only 0.2%.

heading: 2 RELATED WORK
text: There is a growing body of work that extends models with an abstain option, either by training a selector mechanism separately or jointly together with the model. The existing approaches include various selection mechanisms such as entropy selection (Mueller et al., 2020), selection function (Cortes et al., 2016; Mueller et al., 2020; Geifman & El-Yaniv, 2019), softmax response (Stutz et al., 2020; Geifman & El-Yaniv, 2017), or explicit abstain class (Laidlaw & Feizi, 2019; Liu et al., 2019). In our work, we explore an alternative selector mechanism that uses model robustness. The advantage of this formulation is that the selector provides strong guarantees for each sample and can never produce false-positive selections. The disadvantage is that it introduces a significant runtime overhead, compared to many other methods that require only a single forward pass.
At the same time, several recent works investigated the robustness and accuracy tradeoff both theoretically (Yang et al., 2020; Dobriban et al., 2020) and practically by proposing new methods to mitigate it. For example, Raghunathan et al. (2020) proposes robust self-training that leverages unlabeled data to regularize the model. Stutz et al. (2019) considers a new method based on onmanifold adversarial examples, which are more aligned with the true data distribution than the `p-norm noise models. Mueller et al. (2020) focuses on deterministic certification and also proposes using compositional models to control the robustness and accuracy tradeoff. In our work, we also take advantage of compositional models, but we focus on both empirical and probabilistic certified robustness. Further, our selector formulation is based on a new training that minimizes robust inaccuracy and can be used to fine-tune any existing robust model. Finally, we provide individual robustness at inference time, rather than distributional robustness considered in prior works.

heading: 3 PRELIMINARIES
text: Let fθ :Rd→Rk be a neural network which classifies inputs x∈X ⊆Rd to outputs Rk (e.g., logits or probabilities). The hard classifier induced by the network is given as Fθ(x) = arg maxi∈Y fθ(x)i, where fθ(x)i is the network output for the i-th class and Y, |Y| = k is the finite set of discrete labels.
Natural Accuracy Given a distribution over input-label pairs D and a classifier Fθ : X → Y , an input-label pair (x, y) is considered accurate iff the classifier Fθ predicts the correct label y for x:
Rnat(Fθ) = E(x,y)∼D 1{Fθ(x) = y} (1)
Robust Accuracy Given an input-label pair (x, y), we say that the classifier Fθ is robust and accurate iff it predicts the correct label y for all samples from a predefined region Bpε(x), such as a `p-norm ball centered at x with radius ε, i.e., Bpε(x) ..= {x′ : ||x′ − x||p ≤ ε}. Formally:
Raccrob(Fθ) = E(x,y)∼D 1{Fθ(x) = y} ∧ 1{∀x′ ∈ Bpε(x). Fθ(x′) = Fθ(x)} (2)
Robust Inaccuracy Similarly to robust accuracy, an input-label pair (x, y) is considered robustly inaccurate iff the classifier Fθ predicts an incorrect label Fθ(x) 6= y and Fθ is robust towards that misprediction for all inputs in Bpε(x). Formally, the robust inaccuracy is defined as:
R¬accrob (Fθ) = E(x,y)∼D 1{Fθ(x) 6= y} ∧ 1{∀x′ ∈ Bpε(x). Fθ(x′) = Fθ(x)} (3)

heading: 4 REDUCING ROBUST INACCURACY: ADVERSARIAL & CERTIFIED TRAINING
text: In this section, we present our training method that extends existing robust training approaches by also considering samples that are robust but inaccurate. We start by describing a high-level problem statement which we then instantiate for both empirical robustness as well as certified robustness.
Problem Statement Given a distribution over input-label pairs D, our goal is to find model parameters θ such that the resulting model maximizes robust accuracy, while at the same time minimizing robust inaccuracy. Concretely, this translates to the following optimization objective:
arg min θ E(x,y)∼D β · Lrob(x, y)︸ ︷︷ ︸ optimize robust accuracy + 1{Fθ(x) 6= y} · L¬accrob (x, y)︸ ︷︷ ︸ penalize robust inaccuracy
(4)
where β ∈ R+ is a regularization term, 1{Fθ(x) 6= y} is an indicator function denoting samples for which the model is inaccurate, and Lrob(x, y) with L¬accrob (x, y) are loss functions that optimize robust accuracy and penalize robust inaccuracy, respectively. Here, the first loss function Lrob(x, y) is standard and can be directly instantiated using existing approaches (see next). The main challenge comes in defining the second loss term, as well as ensuring that the resulting formulation is easy to optimize, e.g., by defining a smooth approximation of the non-differentiable indicator function.

heading: 4.1 ADVERSARIAL TRAINING
text: We instantiate the loss function from Equation 4 when training empirically robust models as follows: LERA(fθ,(x, y)) = β·LTRADES(fθ,(x, y))+(1−fθ(x)y) min
x′∈Bpε (x) `CE(fθ(x ′), arg max c∈Y\{Fθ(x)} fθ(x ′)c)
(5)
Below, we introduce each term in more detail and discuss the motivation behind our formulation.
Lrob To instantiate Lrob, we can use any existing adversarial training method. For example, considering standard adversarial training (Goodfellow et al., 2014) would result in the following loss:
Ladv ..= max x′∈Bpε (x) `CE(fθ(x ′), y) (6)
where `CE is cross-entropy loss of the worst example in the allowed perturbation region Bpε . Similarly, we can instantiate the loss using more sophisticated methods, such as TRADES (Zhang et al., 2019a):
LTRADES ..= `CE(fθ(x), y) + γ max x′∈Bpε (x) DKL(fθ(x), fθ(x ′)) (7)
where DKL is the Kullback-Leibler divergence (Kullback & Leibler, 1951).
1{Fθ(x) 6= y} Next, we consider the indicator function, which encourages learning on inaccurate samples. Since the indicator function is computationally intractable, we replace the hard qualifier by a soft qualifier 1− fθ(x)y. The soft qualifier will be small for accurate and large for inaccurate samples, thus providing a smooth approximation of the original indicator function.
L¬accrob Third, we define the loss that penalizes robust but inaccurate samples. This can be formulated similar to the adversarial training objective (Madry et al., 2018), however, instead of optimizing the prediction of the adversarial example fθ(x′) towards the correct label y, we optimize towards the most likely adversarial label arg maxc∈Y\{Fθ(x)} fθ(x ′)c. This leads to the following formulation:
min x′∈Bpε (x)
`CE(fθ(x ′), arg max
c∈Y\{Fθ(x)} fθ(x
′)c) (8)
Note that the purpose of the L¬accrob loss is to penalize robustness by making the model non-robust. As a result, it is sufficient to consider only a single non-robust example, thus the minimization (rather than maximization) in the loss objective1.

heading: 4.2 CERTIFIED TRAINING
text: Similarly to Section 4.1, we now instantiate the loss function from Equation 4 for probabilistic certified robustness via randomized smoothing (Cohen et al., 2019). Randomized smoothing constructs a smoothed classifier Gθ : X → Y from a base classifier Fθ, where Gθ(x) predicts the class which Fθ is most likely to return when x is perturbed under isotropic Gaussian noise. Our proposed instantiation of Equation 4 for probabilistic certified robustness is as follows:
LCRA(fθ, (x, y)) = β · Lnoise(fθ, (x, y)) + 1
k k∑ j=1 ( 1− fθ(x+ ηj)y ) CR(fθ, (x, y)) (9)
where η1, ...,ηk are k i.i.d. samples from N (0, σ2I). Note that, since the robustness guarantees provided by randomized smoothing hold for the smoothed classifier Gθ, the three loss components from Equation 4 need to be formulated with respect to the smoothed classifier Gθ.
Lrob To instantiate Lrob, we can use any existing certified training method for randomized smoothing, such as the methods defined by Cohen et al. (2019) or Zhai et al. (2020). Concretely, when using Cohen et al. (2019), the loss is defined using Gaussian noise augmentation:
Lnoise ..= `CE(fθ(x+ η), y), η ∼ N (0, σ2I) (10)
1{Fθ(x) 6= y} We again replace the computationally intractable hard qualifier by a soft qualifier Eδ∼N (0,σ2I)[1−fθ(x+δ)y], which encodes the misprediction probability of the smoothed classifier. In practice, we approximate expectations over Gaussians via Monte Carlo sampling, thus leading to the approximated soft inaccuracy qualifier 1/k ∑k j=1 1− fθ(x+ ηj)y .
1Naturally, this assumes that the method used to check robustness can correctly detect the non-robustness, even if it is caused by a single example. Note that, for a fair evaluation, we use a relatively weak 10-step PGD (Madry et al., 2018) attack during training and a strong 40-step APGD (Croce & Hein, 2020) for evaluation.
L¬accrob Finally, we instantiate the L¬accrob loss term, which encourages the model toward non-robust predictions on robust but inaccurate samples. We propose to minimize robustness by directly minimizing the certified radius of the smoothed classifier Gθ. The certified radius formulation by Cohen et al. (2019) involves a sum of indicator functions, which is not differentiable. However, Zhai et al. (2020) have recently proposed the following differentiable certified radius formulation:
CR(fθ, (x, y)) = σ
2
[ Φ−1 (1 k k∑ j=1 fθ(x+ ηj ; Γ)y ) − Φ−1 ( max y′ 6=y 1 k k∑ j=1 fθ(x+ ηj ; Γ)y′ )] (11)
where Φ−1 is the inverse of the standard Gaussian CDF, Γ is the inverse softmax temperature multiplied with the logits of fθ, and η1:k are k i.i.d. samples from N (0, σ2I). Note that, by setting the loss term L¬accrob to CR(fθ, (x, y)), we directly penalize robustness of the smoothed classifier Gθ.

heading: 5 ROBUST ABSTAIN MODELS
text: In this section, we extend the models trained so far by leveraging robustness as a principled abstain mechanism. Further, we define an additional loss function based on the Deep Gamblers loss (Liu et al., 2019), which is specifically designed for training adversarially robust abstain models.
Abstain Model Consider an input space X ⊆ Rd and a label space Y . A model with an abstain option (El-Yaniv et al., 2010) is a pair of functions (Fθ, S), where Fθ : X → Y is a classifier and S : X → {0, 1} is a selection mechanism, which acts as a binary qualifier for Fθ. Let S(x) = 0 indicate that the model abstains on input x ∈ X , while S(x) = 1 indicates that the model commits to the classifier Fθ for input x and predicts Fθ(x).
Robustness Indicator Selector We now instantiate abstain models with a robustness indicator selector, that abstains on all non-robust samples. For adversarial robustness, the selector is defined as:
SERI(x) = 1{∀x′ ∈ B(x) : Fθ(x′) = Fθ(x)} (12) For certified robustness, the selector is defined as:
SCRI(x) = 1{∀x′ ∈ B(x) : Gθ(x′) = Gθ(x)} (13)
Robustness Guarantees: Robust Selection Similar to robust accuracy, the robustness of an abstain model needs to be evaluated with respect to a threat model. In our work, we consider the same threat model as for the underlying model Fθ, namely Bpε(x) ..= {x′ : ||x′ − x||p ≤ ε}, a `p-norm ball centered at x with radius ε. Then, we define the robust selection of an abstain model as follows:
Rselrob(S) = E(x,y)∼D 1{∀x′ ∈ Bpε(x). S(x′) = 1} (14) That is, we say that a model is robustly selecting x if the selector S would select all valid perturbations x′ ∈ Bpε(x). When used together with our definition of SERI, we obtain the following criterion:
Rselrob(SERI) = E(x,y)∼D 1{∀x′ ∈ Bpε(x). SERI(x′) = 1} = E(x,y)∼D 1{∀x′ ∈ Bpε(x). 1{∀x′′ ∈ Bpε(x′). Fθ(x′′) = Fθ(x′)}} = E(x,y)∼D 1{∀x′ ∈ Bp2·ε(x). Fθ(x′) = Fθ(x)}
In other words, to guarantee that the selector SERI is robust for all x′ ∈ Bpε(x), we in fact need to check robustness of the model Fθ to double that region x′ ∈ Bp2·ε(x). This is important in order to obtain the correct guarantees and is reflected in our evaluation in Section 7.
Note that when evaluating robust selection for certified training, it is sufficient to show that the smoothed model Gθ can be certified with a radius R ≥ ε. Then, the smoothed model guarantees that Gθ(x ′) = cA for all x′ ∈ Bpε(x), which is equivalent to our condition ∀x′∈B(x) : Gθ(x′) = Gθ(x).

heading: 6 BOOSTING ROBUSTNESS WITHOUT ACCURACY LOSS
text: Consider an abstain model (Fθ, S) and letD be a dataset to evaluate (Fθ, S). The selector S partitions D into two disjoint subsets – the set of abstained inputs D¬s and the set of selected inputs Ds for
which Fθ makes a prediction. However, depending on the application, it may be desirable to make a best-effort prediction on all samples, including D¬s. This insight leads to compositional architectures, already used by a number of prior works (Mueller et al., 2020; Wong et al., 2018).
Let H = ((Frobust, S), Fcore) be a 2-compositional architecture consisting of a selection mechanism S, a robustly trained model Frobust, and a core model Fcore. Given an input x ∈ X , the selector S decides whether the model is confident on x and commits to the robust model Frobust or whether the model should abstain and fall back to the core model Fcore. Formally: H(x) = S(x) · Frobust(x) + (1− S(x)) · Fcore(x) (15) Note that each model of a compositional architecture can be chosen arbitrarily, regardless of the model’s network architecture or its training. However, one benefit of using a compositional architecture is combining models that complement each other, resulting in overall improved model performance. In our work, we combine models trained via adversarial or certified training (which typically have lower natural accuracy), with models trained using standard training (which have high natural accuracy but low robustness). The compositional model performance then depends on the quality of the selector S, used to determine which model to evaluate for each sample.

heading: 7 EVALUATION
text: In this section, we present a thorough evaluation of our approach by instantiating it to four different datasets, six recent state-of-the-art models, for both adversarial and certified robustness, and including top-trained models from RobustBench (Croce et al., 2020). We show the following key results:
• Fine-tuning existing models with our proposed loss successfully decreases robust inaccuracy and provides a Pareto front of models with different robustness tradeoffs.
• Combining our proposed loss and robustness as an abstain mechanism leads to higher robust selection and accuracy compared to softmax response and selection network baselines.
• Our 2-compositional models significantly improve robustness by up to +60% while retaining the natural accuracy, causing only minor decrease of up to −0.2% (for B∞1/255 and B ∞ 2/255).
We perform all experiments on a single GeForce RTX 3090 GPU and use PyTorch (Paszke et al., 2019) for our implementation. The hyperparameters used for our experiments are provided in Appendix A.2.
Models Our proposed training method requires neither retraining classifiers from scratch nor modifications to existing classifiers, thus our approach can be applied to fine-tune a wide range of existing models2. To demonstrate this, we use the following robust pre-trained models:
For empirical robustness, we evaluate five existing models from Carmon et al. (2019), Gowal et al. (2020), Rebuffi et al. (2021), Sehwag et al. (2021) and Zhang et al. (2019a), where all but the last model are top models taken from RobustBench (Croce et al., 2020). The model by Sehwag et al. is trained for ε2 = 0.5, while the other models are trained for ε∞ = 8/255. In our evaluation, we fine-tune each model for 50 epochs for the considered threat model (ε∞ ∈ {1/255, 2/255, 4/255} and ε2 ∈ {0.12, 0.25}) using both the LTRADES loss (Zhang et al., 2019a) and our proposed LERA loss. For certified robustness, we use the σ = 0.12 Gaussian noise augmentation trained model by Cohen et al. (2019) and the ε2 = 0.50 adversarially trained model by Sehwag et al. (2021). Similarly to empirical robustness, we fine-tune the models for 50 epochs using either Gaussian noise augmentation training (Cohen et al., 2019) or the LCRA loss proposed in our work.
Datasets We evaluate our approach on two academic datasets – CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009), and two commercial datasets – Mapillary Traffic Sign Dataset (MTSD) (Ertler et al., 2020) and a Rail Defect Dataset kindly provided by Swiss Federal Railways (SBB). We provide full details, including example images and all preprocessings steps, in the Appendix A.1.
When training on the CIFAR-10 and CIFAR-100 datasets, we use the AutoAugment (AA) policy by Cubuk et al. (2018) as the image augmentation. For the MTSD and SBB datasets, we use standard image augmentations (SA) consisting of random cropping, color jitter, and random translation and rotation. For completeness, our evaluation also includes models trained without any data augmentations.
2Our method can also be used to train from scratch, in which case a scheduler for β should be introduced.
Metrics We use the natural accuracy, robust accuracy, and robust inaccuracy as our main evaluation metrics, as defined in Section 3, but evaluated on the corresponding test dataset D = {(xi, yi)Ni=1}. When evaluating the empirical robustness, we use 40-step APGD (Croce & Hein, 2020) to evaluate the robustness of the classifier Fθ. To evaluate certified robustness, we use the Monte Carlo algorithm for randomized smoothing from Cohen et al. (2019). We certify 500 test samples and use the same randomized smoothing hyperparameters as Cohen et al. (2019) (cf. Appendix A.2).

heading: 7.1 REDUCING ROBUST INACCURACY
text: Empirical Robustness For empirically robust models, the results in Figure 2 show the robust accuracy (Raccrob) and robust inaccuracy (R¬accrob ) of different existing models fine-tuned with ( ) and without ( ) data augmentations. At the same time, Figure 2 also shows the same models fine-tuned with our proposed loss with ( ) and without ( ) data augmentations. We can see that our approach improves over the existing models across all the datasets. For example, on CIFAR-10 and for B∞2/255, the model from Carmon et al. (2019) achieves 86.5% robust accuracy, but also robust inaccuracy of 1.34%. In contrast, using our loss LERA, we can obtain a number of models that reduce robust inaccuracy to 0.29%, while still achieving robustness of 83.8%. Similar results are obtained for other models, datasets, and perturbation regions (cf. Appendix A.5). We generally observe that our approach achieves consistently lower robust inaccuracy compared to adversarial training. Further, by varying the regularization term β, we obtain a Pareto front of optimal solutions.
Certified Robustness Similarly, we show the robust accuracy (Raccrob) and robust inaccuracy (R¬accrob ) for certifiably robust models, which were fine-tuned using Gaussian noise augmentation Lnoise and using our proposed loss function LCRA. In Table 2, we show results on CIFAR-10 for B20.12 and B20.25 perturbation regions. We can see that our approach achieves lower robust inaccuracy compared to existing models. For example, on CIFAR-10 and B20.25, the Cohen et al. (2019) model achieves 62% robust accuracy, but also 1% robust inaccuracy. In contrast, our approach reduces the robust inaccuracy to 0.4% while still achieving 53.8% robust accuracy. For the Sehwag et al. (2021) model, our approach even improves both the robust accuracy and the robust inaccuracy. For B20.25, our approach improves the robust accuracy by +4.8% and reduces the robust inaccuracy by −0.6%.

heading: 7.2 USING ROBUSTNESS TO ABSTAIN
text: Next, we evaluate using robustness as an abstain mechanism (Section 5) and how it benefits from the training proposed in our work. We compare the following abstain mechanisms:
Softmax Response (SR) (Geifman & El-Yaniv, 2017), which abstains if the maximum softmax output of the model fθ is below a threshold τ for some input x′ ∈ Bpε(x), that is: SSR(x) = 1{∀x′ ∈ Bpε(x) : maxc∈Y fθ(x′)c ≥ τ} (16) Similar to the robustness indicator (Section 5), to guarantee robustness of SSR, we need to check the maximum softmax output of the model fθ on double the region Bp2·ε(x). To evaluate robustness of SSR, we use a modified version of APGD called APGDconf (Appendix A.4). For each model considered in our work (e.g., Carmon et al. (2019)), we evaluate its corresponding abstain selector:
• ( , ) CARMONSR, GOWALSR, ZHANGSR, etc., all of which are fine-tuned using TRADES.
Robustness Indicator (RI) (our work), which abstains if the model Fθ is non-robust:
SRI(x) = 1{∀x′ ∈ Bpε(x) : Fθ(x′) = Fθ(x)} (17) Note that, unlike other selectors, our robustness indicator is by design robust against an adversary using the same threat model. For each model, we compare two instantiations of this approach:
• ( , ) TRADESRI, ( , ) ERARI (Equation 5).
Selection Network (SN), which trains a separate neural network sθ : X → R and selects if: SSN(x) = 1{sθ(x) ≥ τ} (18)
When considering the robustness of the abstain model, the robustness of both the classifier and the selection network have to be taken into account. We compare against two instantiations of this approach, both trained using certified training:
• ( ) ACE-COLTSN (Balunovic & Vechev, 2019; Mueller et al., 2020), and • ( ) ACE-IBPSN (Gowal et al., 2018; Mueller et al., 2020).
Empirical Robustness In Figure 3, we show the comparison of different abstain approaches using two metrics – the robust selection (Rselrob), and the ratio of non-abstained samples that are robust and accurate (Raccrob). Ideally, we would like both of these to be as high as possible, but typically there is a tradeoff between the two. This can clearly be seen in the results, where both our approach and softmax response can be used to obtain a Pareto front of optimal solutions.
Overall, the main results in Figure 3 show that as designed, our approach consistently improves robust accuracy. For example, on CIFAR-10 at ε∞ = 1/255 and Carmon et al. (2019) model, we
can successfully improve robust accuracy by +1.18% at the expense of -3.78% decrease in robust selection. This is close to optimal since increasing robust accuracy is typically achieved by correctly selecting for which samples to abstain. Interestingly, for some models and datasets, we can strictly improve over the baseline models by increasing both the robust accuracy and the non-abstained samples. On CIFAR-10 at ε∞ = 1/255 and Gowal et al. (2020) model, we increase the robust accuracy by +1.06% and robust selection by +1.61% (when training without data augmentations).
Compared to the other abstain methods, our approach, in general, improves both metrics while also providing much stronger guarantees. Concretely, our approach guarantees that selected samples are robust in the considered threat model. Softmax response only guarantees that all samples in the considered threat model have high confidence and is thus vulnerable to high confidence adversarial examples, and the selection network provides no guarantees with regards to the selector’s robustness.
Certified Robustness Applying our training for certified robustness LCRA with β = 1.0 consistently improves robust accuracyRaccrob of robustness indicator abstain models. In Table 3, we show our results on CIFAR-10, using `2 perturbations of radius 0.06 and 0.12. For instance, for the Cohen et al. (2019) model trained at σ = 0.12, we are able to improve the robust accuracy by +0.85% for ε2 = 0.25 perturbations, at the expense of−8.8% decrease in robust selection. On the other hand, for the Sehwag et al. (2021) model, our approach improves on both metrics. For ε2 = 0.25 perturbations, we increase robust accuracy by +0.82% and robust selection by +4.2%.

heading: 7.3 BOOSTING ROBUSTNESS WITHOUT ACCURACY LOSS
text: Finally, we present the results of combining the abstain models trained so far with state-of-the-art models trained to achieve high accuracy. Note that, as discussed in Section 5, when evaluating adversarial robustness for Bpε , we in fact need to consider B p 2·ε robustness of the abstain model.
A summary of the results is shown in Figure 4. Similar to the results shown so far, the 2-compositional architectures that use models trained by our method ( , ) improve over existing methods that optimize only for robust accuracy ( , ), as well as over models using softmax response ( , ) or selection network ( , ) to abstain. For example, for CIFAR-10 with ε∞=1/255 and the Carmon et al. (2019) model, we improve natural accuracy by +0.58% and +0.62%, while decreasing the robustness only by -2.75% and -2.82%, when training with and without data augmentations respectively.
More importantly, our approach significantly improves robustness of highly accurate noncompositional models, with minimal loss of accuracy. In fact, in half of the cases, the compositional architecture even slightly improves the overall accuracy, as summarized in Table 4. We provide full results, including additional models and perturbation bounds in Appendix A.7, and an evaluation of the considered highly accurate non-compositional models in Appendix A.8.

heading: 8 CONCLUSION
text: In this work, we address the issue of robust inaccuracy of state-of-the-art robust models. Ideally, models should be robust only if accurate, however, we show that existing robust models have nonnegligible amounts of robust but inaccurate samples. We present a new training method that jointly minimizes robust inaccuracy and maximizes robust accuracy. The key concept was extending an existing robust training loss with a term that minimizes robust inaccuracy, making our method widely applicable since it can be instantiated using various existing robust training methods. We show the practical benefits of our approach by both, using robustness as an abstain mechanism, and by leveraging compositional architectures to improve robustness without sacrificing accuracy.
However, there are also limitations and interesting extensions to consider in future work. First, while there are some cases where our training improves both robust accuracy and reduces robust inaccuracy (e.g., for certified robust Sehwag et al. (2021) model on CIFAR-10 or empirically robust Gowal et al. model on CIFAR-10), it does typically results in a trade-off between the two – reduced robust inaccuracy also leads to reduced robust accuracy. To address this issue, in practice we compute a Pareto front of optimal solutions, all of which can be used to instantiate the compositional model. An interesting future work would be to explore this trade-off further and develop new techniques to mitigate it. Second, given that we compute a Pareto front of the optimal solutions, a promising future work item is considering model cascades that consist of different models along this Pareto front, and progressively fall back to models with higher robust accuracy but also higher robust inaccuracy. Third, we observed that as the robust inaccuracy approaches zero (i.e., the best case), the training becomes much harder. This is both because these remaining robust inaccurate examples are the hardest to fix, as well as because there are only very few of them. In our work, we explored using data augmentation to address this issue, but more work is needed to make the training efficient in such a low data regime.

heading: A APPENDIX

heading: A.1 DATASETS
text: We ran our evaluations on four different datasets, namely on CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009), the Mapillary Traffic Sign Dataset (MTSD) (Ertler et al., 2020), and a rail defect dataset provided by Swiss Federal Railways (SBB). Additionally, we used a synthetic dataset consisting of two-dimensional data points. In the following, we explain the necessary preprocessing steps to create the publicly available MTSD dataset.
Mapillary Traffic Sign Dataset (MTSD) The Mapillary traffic sign dataset (Ertler et al., 2020) is a large-scale vision dataset that includes 52’000 fully annotated street-level images from all around the world. The dataset covers 400 known and other unknown traffic signs, resulting in over 255’000 traffic signs in total. Each street-level image is manually annotated and includes ground truth bounding boxes that locate each traffic sign in the image, as shown in Figure 5a. Further, each ground truth traffic sign annotation includes additional attributes such as ambiguousness or occlusion. Since the focus of this work is on classification, we convert the base MTSD dataset to a classification dataset (described below) by cropping to each ground truth bounding box. We show samples from the resulting cropped MTSD dataset in Figure 5b.
We convert the MTSD objection detection dataset into a classification dataset as follows:
1. Ignore all bounding boxes that are annotated as occluded (sign partly occluded), out-offrame (sign cut off by image border), exterior (sign includes other signs), ambiguous (sign is not classifiable at all), included (sign is part of another bigger sign), dummy (looks like a sign but is not) (Ertler et al., 2020). Further, we ignore signs of class other-sign, since this is a general class that includes any traffic sign with a label not within the MTSD taxonomy.
2. Crop to all remaining bounding boxes and produce a labeled image classification dataset. Cropping is done with slack, i.e. we crop to a randomly upsized version of the original bounding box. Given a bounding box BB = ([xmin, xmax], [ymin, ymax]), the corresponding upsized bounding box is given as
UBB = ( [xmin − λαx(xmax − xmin), xmax + λ(1− αx)(xmax − xmin)], [ymin − λαy(ymax − ymin), ymax + λ(1− αy)(ymax − ymin)]
) (19) where αx, αy ∼ U[0,1] 3 and λ is the slack parameter, which we set to λ = 1.0.
3. Resize cropped traffic signs to (64, 64). 3U[a,b] is the uniform distribution over the interval [a, b].
Rail Defect Dataset (SBB) The rail defect dataset (SBB) is a proprietary vision dataset collected and annotated by Swiss Federal Railways. It includes images of rails, each of which is annotated with ground truth bounding boxes for various types of rail defects. We note that all the models used in our work for this dataset are trained by the authors and not provided by SBB. In fact, for our work, we even consider a different type of task – classification instead of the original object detection. As a consequence, the accuracy and robustness results presented in our work are by no means representative of the actual models used by SBB.

heading: A.2 HYPERPARAMETERS
text: TRADES We use LTRADES (Zhang et al., 2019a) to both train models from scratch and fine-tune existing models. When training models from scratch, we train for 100 epochs using LTRADES, with an initial learning rate 1e-1, which we reduce to 1e-2 and 1e-3, once 75% and 90% of the total epochs are completed. When fine-tuning models, we train for 50 epochs using LTRADES, with an initial learning rate 1e-3, which we reduce to 1e-4 once 75% of the total epochs are completed. We use batch size 200, use 10-step PGD (Madry et al., 2018) to generate adversarial examples during training, and set the β parameter in LTRADES to βTRADES = 6.0.
Empirical Robustness Abstain Training We fine-tune for 50 epochs using LERA (Equation 5), with an initial learning rate 1e-3, which we reduce to 1e-4 once 75% of the total epochs are completed. We use batch size 200, use 10-step PGD (Madry et al., 2018) to generate adversarial examples during training, and set βTRADES = 6.0 for the loss term Lrob = LTRADES.
Certified Robustness Abstain Training We fine-tune for 50 epochs using LCRA (Equation 9), with an initial learning rate 1e-3, which we reduce to 1e-4 once 75% of the total epochs are completed. We use batch size 50, k = 16 i.i.d. samples fromN (0, σ2I), and set the inverse softmax temperature to Γ = 4.0 (cf. Section 4.2).
Probabilistic Certification via Randomized Smoothing We use the practical Monte Carlo algorithm by Cohen et al. (2019) for randomized smoothing, using the same certification hyperparameters as them. We use N0 = 100 Monte Carlo samples to identify the most probable class cA, N = 100, 000 Monte Carlo samples to estimate a lower bound on the probability pA, and set the failure probability to α = 0.001.
Synthetic Dataset In Figure 1, we illustrate the effect of our training on a synthetic three-class dataset, where each class follows a Gaussian distribution. We then use a simple four-layer neural network with 64 neurons per layer, and train it on N = 1000 synthetic samples, using Lstd, LTRADES (Zhang et al., 2019a), and LERA (Equation 5). For each loss variant, we train for 20 epochs, use a fixed learning rate 1e-1, and batch size 10. For LTRADES and LERA, we use 10-step PGD (Madry et al., 2018) to generate adversarial examples during training, and set βTRADES = 6.0.

heading: A.3 LOSS FUNCTION ABLATION STUDY
text: Additionally to the LERA loss from Equation 5, we consider an alternative loss formulation for training an empirical robustness indicator abstain model. The formulation is based on the Deep Gamblers loss (Liu et al., 2019), which considers an abstain model (Fθ, S) with an explicit abstain class a as a selection mechanism. Since we consider robustness indicator selection, we replace the output probability of the abstain class fθ(x)a with the output probability of the most likely adversarial label. This corresponds to the probability of a sample being non-robust and thus the probability of abstaining under a robustness indicator selector. Similar to LERA, we also add the TRADES loss (Zhang et al., 2019a) to optimize robust accuracy. The resulting loss is then defined as:
LDGA(fθ, (x, y)) = β · LTRADES(fθ, (x, y))− log ( fθ(x)y + maxc∈Y\{Fθ(x)} fθ(x ′)c ) (20)
We conduct an ablation study over the two loss functions, LERA and LDGA, for CIFAR-10 and a ε∞=8/255 TRADES (Zhang et al., 2019a) trained ResNet-50 model. We fine-tune the model for `∞ perturbations of radii 1/255 and 2/255, using both LERA and LDGA, training for 50 epochs each and setting the regularization parameter β = 1.0. For each loss variant, we train the base model once without data augmentations and once using the AutoAugment (AA) policy (Cubuk et al., 2018).
We show the robust accuracy and the robust selection of the resulting robustness indicator abstain models in Table 5. Observe that for all experiments, LERA trained models achieve consistently higher robust accuracy and higher robust selection, compared to LDGA trained models. For instance, when training for ε∞ = 1/255 perturbations without data augmentations, LERA achieves +1.71% higher robust accuracy and +1.33% higher robust selection, compared to LDGA. Similarly, when training with AutoAugment, LERA achieves +0.91% higher robust accuracy and +2.72% higher robust selection. Similar results hold for ε∞=2/255 perturbations.

heading: A.4 COMPARING ADVERSARIES FOR SOFTMAX RESPONSE (SR)
text: Recall from Section 7.2 that we evaluated the robustness of softmax response (SR) abstain models using APGDconf, which is a modified version of APGD (Croce & Hein, 2020) using the alternative adversarial attack objective by Stutz et al. (2020). This modified objective optimizes for an adversarial example x′ that maximizes the confidence in any label c 6= Fθ(x), instead of minimizing the confidence in the predicted label:
x′ = arg max x̂∈Bpε (x) max c6=Fθ(x) fθ(x̂)c (21)
The resulting adversarial attack finds high confidence adversarial examples, and thus represents an effective attack against a softmax response selector SSR.
In the following, we conduct an ablation study over APGD and APGDconf by evaluating the robust selection Rselrob and robust accuracy Raccrob of an SR abstain model (Fθ, SSR) using both APGD and APGDconf. We use the adversarially trained WideResNet-28-10 model by Carmon et al. (2019) (taken from RobustBench (Croce et al., 2020)), trained on CIFAR-10 for ε∞=8/255 perturbations. We then evaluate the classifier as an SR abstain model (Fθ, SSR) with varying threshold τ ∈ [0, 1), and report the robust selection and robust accuracy for varying `∞ perturbations in Figure 6. Observe that for small perturbations such as ε∞=1/255, APGD and APGDconf are mostly equivalent concerning robust selection and robust accuracy. However, for larger perturbations such as ε∞=4/255, the SR abstain model is significantly less robust to APGDconf than to standard APGD, showing the importance of choosing a suitable adversarial attack. High confidence adversarial examples are generally more likely to be found for larger perturbations, thus an SR selector is significantly less robust to APGDconf than to APGD for larger perturbations.

heading: A.5 ADDITIONAL EXPERIMENTS ON REDUCING ROBUST INACCURACY
text: In this section, we present additional experiments on reducing robust inaccuracy for empirical robustness.
Similar to the results in Figure 2, we show the robust accuracy (Raccrob) and robust inaccuracy (R¬accrob ) of different existing models fine-tuned with ( ) and without ( ) data augmentations, in Figure 7. At the same time, Figure 7 also shows the same models fine-tuned with our proposed loss with ( ) and without ( ) data augmentations. We again observe that our approach achieves consistently lower robust robust inaccuracy, compared to existing robust models. For example, on CIFAR-10 and for B∞1/255, the model from Carmon et al. (2019) achieves 91.7% robust accuracy but also 1.8% robust inaccuracy. Using our loss LERA and varying the regularization term β, we can obtain a number of models that reduce robust inaccuracy to 0.14% while still achieving robust accuracy of 75.8%.

heading: A.6 ADDITIONAL EXPERIMENTS ON USING ROBUSTNESS TO ABSTAIN
text: In this section, we present additional experiments on comparing different abstain approaches for empirical robustness.
We compare robustness indicator abstain models (F, SRI) using existing robust classifiers TRADESRI and classifiers fine-tuned with our proposed loss ERARI. Further, we again consider softmax response and selection network abstain models, as described in Section 7.2. Equivalent to Section 7.2, we use the robust selection (Rselrob), and the ratio of non-abstained samples that are robust and accurate (Raccrob) as our evaluation metrics.
We show the comparison of the different abstain models in Figure 8. Similar to the results in Section 7.2, we again show that, as designed, our approach consistently improves robust accuracy. For instance, consider the CIFAR-10 Zhang et al. (2019a) model at ε∞ = 1/255, trained without data augmentations ( ). The ERARI model with the highest robust selection Rselrob improves robust accuracy by +2.39% at the expense of -3.44% decrease in robust selection. This tradeoff is close to optimal since our approach increases robust accuracy by correctly abstaining from mispredicted samples, thus an increase in robust accuracy results in a corresponding decrease in robust selection. Further, we again observe that by varying the regularization parameter β, we can obtain a Pareto front of optimal solutions. Considering the CIFAR-10 Zhang et al. (2019a) model at ε∞ = 1/255, trained with data augmentations ( ), we can improve the robust accuracy up to 99.75%, an increase of +4.38% compared to the corresponding TRADESRI model ( ). However, this comes at the expense of a disproportionally large decrease of -42.27% lower robust selection. We observe similar results for other models, datasets, and perturbations regions, shown in Figure 8.
Further, we again note that our approach mostly improves both robust selection and robust accuracy when compared to softmax response and selection network abstain models.

heading: A.7 ADDITIONAL EXPERIMENTS ON BOOSTING ROBUSTNESS WITHOUT ACCURACY LOSS
text: In this section, we present additional results on combining abstain models with state-of-the-art models trained to achieve high natural accuracy.
Equivalent to Section 7.3, we put the abstain models trained so far in 2-composition (Section 6) with the standard trained core models discussed in Appendix A.8. We show the natural (Rnat) and adversarial accuracy (Raccrob) of the resulting 2-compositional architectures in Figure 9. We again observe that 2-compositional architectures using models trained by our method ( , ) improve over existing methods that solely optimize for robust accuracy ( , ). Further, our method mostly improves both the natural and robust accuracy, compared to 2-compositional architectures using softmax response ( , ) or selection network ( , ) to abstain. For example, on SBB and the Zhang et al. (2019a) model at ε∞ = 1/255, our approach ( ) improves natural accuracy by +0.68%, while decreasing the robust accuracy by only -1.54%.
Further, we show that 2-compositional architectures using models trained by our method achieve significantly higher robustness and mostly equivalent overall accuracy, compared to state-of-theart non-compositional models trained for high natural accuracy. In Table 6, we show the natural (Rnat) and adversarial accuracy (Raccrob) of our 2-compositional models and illustrate the accuracy improvement over the standard trained models discussed in Appendix A.8. For instance, consider CIFAR-10 at ε∞ = 2/255 and the 2-compositional architecture using the Gowal et al. (2020) model as robust model Frobust. Our model improves the robust accuracy by +75.3% and the natural accuracy
by +0.1%, compared to the standard trained model by Zhao et al. (2020). Similar results hold for other models, datasets, and perturbation regions.

heading: A.8 CORE MODELS
text: Recall from Section 6 that an abstain model (F, S) can be enhanced by a core model Fcore, which makes a prediction on all abstained samples, resulting in 2-compositional architectures. In Section 7.3, we presented an evaluation of 2-compositional architectures, where we used state-of-the-art standard trained models as core models. In Table 7, we show the natural and adversarial accuracy of core models used in Section 7.3, for varying `∞ perturbation regions, where we use 40-step APGD (Croce & Hein, 2020) to evaluate robustness.

heading: A.9 ROBUSTNESS/ACCURACY DATASET SPLITS
text: Consider a robustness indicator abstain model (Fθ, SRI) and a labeled dataset D = {(xi, yi)Ni=1} on which we evaluate the classifier Fθ : X → Y . Based on the robustness and accuracy of the classifier Fθ, we can partition D into four disjoint subsets D = {Dr∧aFθ , D ¬r∧a Fθ , Dr∧¬aFθ , D ¬r∧¬a Fθ }, where:
Dr∧aFθ = {(x, y) ∈ D : ∀x ′ ∈ Bpε(x). Fθ(x′) = Fθ(x) ∧ Fθ(x) = y} Dr∧¬aFθ = {(x, y) ∈ D : ∀x ′ ∈ Bpε(x). Fθ(x′) = Fθ(x) ∧ Fθ(x) 6= y} D¬r∧aFθ = {(x, y) ∈ D : ∃x ′ ∈ Bpε(x). Fθ(x′) 6= Fθ(x) ∧ Fθ(x) = y} D¬r∧¬aFθ = {(x, y) ∈ D : ∃x ′ ∈ Bpε(x). Fθ(x′) 6= Fθ(x) ∧ Fθ(x) 6= y}
We illustrate this dataset partitioning on the CIFAR-10 (Krizhevsky et al., 2009) dataset. We consider a TRADES (Zhang et al., 2019b) trained ResNet-50 and the WideResNet-28-10 models by Carmon et al. (2019); Gowal et al. (2020) (taken from Robustbench (Croce et al., 2020)), where each model is adversarially pretrained for ε∞= 8/255 and then fine-tuned via TRADES to the respective `∞ threat model illustrated Table 8. Further, we also consider a standard trained ResNet-50. We then evaluate the robustness and accuracy of each model using 40-step APGD (Croce & Hein, 2020). Considering Table 8, note that standard adversarial training methods do not necessarily eliminate the occurrence of robust inaccurate samples (x, y) ∈ Dr∧¬aFθ , and that the robust inaccuracy generally increases for smaller perturbation regions. Further, we note that while standard trained models have low robust inaccuracy, they also have low overall robustness, resulting in low overall robust accuracy.
Further, we also illustrate the robustness-accuracy dataset partitioning on CIFAR-100 (Krizhevsky et al., 2009). We consider a standard trained WideResNet-28-10 and the adversarially trained WideResNet-28-10 by Rebuffi et al. (2021). Again, the model by Rebuffi et al. (2021) was pretrained for ε∞=8/255 perturbations and then TRADES fine-tuned for the respective threat model indicated in Table 9. We again evaluate the robustness-accuracy dataset partitioning for varying `∞ perturbations using 40-step APGD (Croce & Hein, 2020), and list the exact size of each data split in Table 9.
Notably, we observe that on the model by Rebuffi et al. (2021), 15.24% of all test samples are robust but inaccurate for ε∞ = 1/255 perturbations, which is a significantly larger fraction compared to similar models on CIFAR-10.

Review summaries:
This submission aims to reduce the prevalence of samples for which a neural network might predict the wrong answer, not just for the sample but also its nearby points in the input space. Authors call this metric robust inaccuracy.  Next, after reducing the robust inaccuracy metric, they take advantage of their models to improve conventional robustness, by taking advantage of their more reliable robustness metric to abstain from samples for which model prediction is not robust (perhaps better to call this consistent, since it can refer to both correct and incorrect predictions). 

The paper addresses the issue of trained models being inaccurate but robust for some samples. To address this issue a flexible fine-tuning mechanism is proposed. A robustness-based ensembling method is introduced as well. 

To tackle the two limitations of the current robust training algorithms: 1). robust inaccuracy 2). the sacrifice of natural accuracy, this paper proposed a new training method that aims to maximize robust accuracy and minimize robust inaccuracy. Moreover, a robustness-based abstain mechanism is adopted to further boost overall robustness without sacrificing accuracy. Experiments show the effectiveness of the algorithm in terms of fewer robust and inaccurate samples and better robustness, with only marginally reduced natural accuracy.

This well presented work is motivated by the relevant issue that improving model robustness while maintaining high accuracy can result in robust inaccuracy, i.e. a non-negligible amount of samples get classified incorrectly, but the category remains consistent for their perturbations. Supporting this observation empirically, this work proposes an approach to address this issue.  There are two main technical contributions. The first complements the standard robustness objectives from the literature with a regularisation term penalising robust inaccuracy. The paper provides specific implementations of this term under two setups: empirical (adversarial) and certified robustness. The second contribution is an abstain model: a learned indicator function that decides which samples are to be processed by the robust (but less accurate) classifier, and which ones should default to the non-robust (but accurate) baseline. As a result, this two-stage approach achieves high robustness without compromising the classification accuracy. The experiments on standard benchmarks further confirm, that the robust model consistently decreases the fraction of robust inaccurate samples (albeit at the expense to robust accuracy).