DOCTOR: A Simple Method for Detecting Misclassification Errors
Year: 2021

Abstract:
Deep neural networks (DNNs) have shown to perform very well on large scale object recognition problems and lead to widespread use for real-world applications, including situations where DNN are implemented as “black boxes”. A promising approach to secure their use is to accept decisions that are likely to be correct while discarding the others. In this work, we propose DOCTOR, a simple method that aims to identify whether the prediction of a DNN classifier should (or should not) be trusted so that, consequently, it would be possible to accept it or to reject it. Two scenarios are investigated: Totally Black Box (TBB) where only the soft-predictions are available and Partially Black Box (PBB) where gradient-propagation to perform input pre-processing is allowed. Empirically, we show that DOCTOR outperforms all state-of-the-art methods on various well-known images and sentiment analysis datasets. In particular, we observe a reduction of up to 4% of the false rejection rate (FRR) in the PBB scenario. DOCTOR can be applied to any pre-trained model, it does not require prior information about the underlying dataset and is as simple as the simplest available methods in the literature.

heading: 1 Introduction
text: With the advancement of state-of-the-art Deep Neural Networks (DNNs), there has been rapid adoption of these technologies in a broad range of applications to critical systems, such as autonomous driving vehicles or industrial robots, including–but not limited to–classification and decision making tasks. Nevertheless, these solutions still exhibit unwanted behaviors as they tend to be overconfident even in presence of wrong decisions [17]. Developing methods and tools to make these algorithms
∗These authors contributed equally to this work †This paper is supported by the ERC project Hypatia under the European Unions Horizon 2020 research and innovation program. Grant agreement №835294. ‡This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement №792464.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
reliable, in particular for non-specialists who may treat them as “black boxes” with no further checks, constitutes a core challenge. Recently, the study of safety AI methods has gained ground, and many efforts have been made in several areas [7, 8, 9, 11, 26, 35, 36]. In this paper, we investigate a simple method capable of detecting whether a prediction of a classifier is likely to be correct, and therefore should be trusted, or it is not, and should be rejected.
Deep learning pursues the idea of learning effective representations from the data itself by training with the implicit assumption that the test data distribution should be similar to the training data distribution. However, when applied to real-world tasks, this assumption does not hold true, leading to a significant increase of misclassification errors. Although classic approaches to Out-Of-Distribution (OOD) detection [1, 12, 21, 23, 33] are not directly concerned with detecting misclassification errors, they are intended to prevent those errors indirectly by identifying potential drifts of the testing distribution. What the above ODD methods have in common with our work is that samples drawn from the in-distribution are more likely to be correctly classified than those from a different distribution. Indeed, the model’s soft-predictions for in-distribution samples tend to be generally peaky in correspondence to the correct class label while they tend to be less peaky for input samples drawn from a different distribution [12]. In general, most of these works consider white-box scenarios, where the hidden layers of the architecture are accessible or the corresponding weights are tuned during the training phase. A very effective approach to OOD detection is ODIN [23] which involves the use of temperature scaling and the addition of small perturbations to input samples. A related solution is introduced in [6] where the maximum soft-probability is called softmax response. Within this approach, the softmax response decides whether the classifier is confident enough in its prediction or not. A different approach to OOD detection is given by the use of the Mahalanobis distance [14, 22], which consists in calculating how much the observed out-distribution sample deviate from the indistribution ones but assuming the latter are given.

heading: 1.1 Summary of contributions
text: Our work tackles the problem of identifying whether the prediction of a classifier should or should not be trusted, no matter if they are made on out or in-distribution samples, and advances the state-of-the-art in multiple ways.
• From the theoretical point of view, we derive the trade-off between two types of error probabilities: Type-I, that refers to the rejection of the classification for an input that would be correctly classified, and Type-II, that refers to the acceptance of the classification for an input that would not be correctly classified (Proposition 3.1). The characterization of the optimal discriminator in eq. (10) allows us to devise a feasible implementation of it, based on the softmax probability (Proposition 3.2).
• From the algorithmic point of view, inspired by our theoretical analysis, we propose DOCTOR a new discriminator (Definition 2), which yields a simple and flexible framework to detect whether a decision made by a model is likely to be correct or not. We distinguish two scenarios under which DOCTOR can be deployed: Totally Black Box (TBB) where only the soft-predictions are available, hence gradient-propagation to perform input pre-processing is not allowed, and Partially Black Box (PBB) where we further allow method-specific inputs perturbations.
• From the experimental point of view, we show that DOCTOR outperforms comparable stateof-the-art methods (e.g., ODIN [23], softmax response [6] and Mahalanobis distance [22]) on datasets including both in-distribution and out-of-distribution samples, and different architectures with various expressibilities, under both TBB and PBB. A key ingredient of DOCTOR is to fully exploit all available information contained in the soft-probabilities of the predictions (not only their maximum).

heading: 1.2 Related works
text: Recent works have shown that the accuracy of a classifier and its ability to output soft-predictions that represent the true posteriors estimate can be totally disjointed [9, 19, 20]. Furthermore, models often tend to be overconfident about their decision even when their predictions fail [11, 17]. This motivates a novel research area that strives to develop methods to assess when decisions made by classifiers should or should not be trusted. Although the detection of OOD samples is a different
(domain) problem, it is naturally expected that samples from a distribution that is significantly different from the training one cannot be correctly classified. In [23], the authors propose a method which increases the peakiness of the softmax output by perturbing the input samples and applying temperature scaling [9, 13, 29] to the classifier logits in order to better detect in-distribution samples. It is worth noticing that this method requires additional information on the internal structure of the latent code of the model. A very different approach [14, 22] tackles OOD detection by using the Mahalanobis distance. Although this approach appears to be more powerful, it also requires additional samples to learn the mean by class and the covariance matrix of the in-distribution. In [4], classifiers are trained to output calibrated confidence estimates that are used to perform OOD detection. A related line of research is concerned with the problem of selective predictions (aka reject options) in deep neural networks. The main motivation for selective prediction is reducing the error rate by abstaining from prediction when in doubt, while keeping the number of correctly classified samples as high as possible [5, 6, 7]. The idea is to combine classifiers with rejection functions by observing the classifiers’ output, without using any supervision, to decide whether to accept or to reject the classification outcome. In [6], the authors introduce softmax response, a rejection function which compares the maximum soft-probability to a pre-determined threshold to decide whether to accept or reject the class prediction given by the model.

heading: 2 Main Definitions and Preliminaries

heading: 2.1 Basic definitions
text: We start by introducing some definitions and background; then, we describe our statistical model and some useful properties about the underlying detection problem. Let X ⊆ Rd be the (possibly continuous) feature space and let Y = {1, . . . , C} denote the concept of the label space related to some task of interest. Moreover, let pXY be the underlying (unknown) probability density function (pdf) over X × Y . Let Dn = { (x1, y1), . . . , (xn, yn) } ∼ pXY be a random realization of n i.i.d. samples according to pXY denoting the training set, where xi ∈ X is the input (feature), yi ∈ Y is the output class among C possible classes and n denotes the size of the training set. A predictor fDn : X → Y uses the inferred model PŶ |X ≡ PŶ |X(y|x;Dn) based on the training set,
fDn(x) ≡ fn(x;Dn) = ∆ arg max y∈Y PŶ |X(y|x;Dn),
and tries to approximate the optimal (Bayes) decision rule f?(x) =∆ arg max y∈Y PY |X(y|x). Notice that PŶ |X can be interpreted as the prediction of the class (label) posterior probability given a sample (e.g., PŶ |X(y |x) ≡ softmax(x)y), while PY |X is the true (unknown) probability. In several practical scenarios PŶ |X does not perfectly match PY |X and still fDn ≈ f ? (cf. [9]).

heading: 2.2 Error variable
text: Let E(x) =∆ 1 [Y 6= fDn(x)] denote the error variable for a given x ∈ X corresponding to fDn , i.e., where we denote with 1[E ] the indicator vector which outputs 1 if the event E is true and 0 otherwise. Similarly, we can define the self-error variable Ê(x) =∆ 1 [ Ŷ 6= fDn(x) ] also corresponding to the inferred predictor fDn but based on the prediction model PŶ |X of the class posterior probability.
Notice that Ê(x) is observable since the underlying distribution is known. However, E(x) cannot be observed and in general these binary variables do not coincide.
At this stage, it is convenient to introduce the notions of probability of classification error for a given x ∈ X w.r.t. both the true class posterior and the predicted probabilities:
Pe(x) =∆ E [E(x)|x] = 1− PY |X (fDn(x)|x) , (1) P̂e(x) =∆ E [ Ê(x)|x ] = 1− PŶ |X (fDn(x)|x) . (2)
Notice that P̂e(x) represents the probability of misclassification of the sample x with respect to the softmax probability PŶ |X , which can be interpreted as the model’s approximation of nature PY |X . Such approximation is close when the model is well-calibrated. Obviously, Pe?(x) ≤ Pe(x) for all x ∈ X , where Pe?(x) corresponds to the minimum error of the Bayes classifier:
Pe?(x) = 1− PY |X (f?(x)|x). It is worth mentioning that, by averaging (1) over the data distribution, we obtain the error rate of the classifier fDn . Although P̂e(x) provides a valuable candidate to infer the unknown error variable E(x), it is easy to check that
max { Pe(x), P̂e(x) } − Pr ( Ŷ = Y |x ) ≤ Pr { Ê(x) 6= E(x)|x } ≤ Pr ( Ŷ 6= Y |x ) , (3)
which in particular implies that the error incurred in using Ê(x) to predict E(x) is lower bounded by the classification error per sample (1). The proofs are in Supplementary material (Appendix A.3).
In this paper, we aim at identifying a discriminator capable of distinguishing between inputs x for which we can trust the predictions of the classifier fDn(x) (i.e., E(x) = 0) and those for which we should not trust predictions (i.e., E(x) = 1). In the next section, we will show that the function Pe(x) : X 7→ [0, 1] plays a central role in the characterization of the optimal discriminator. However, Pe(x) is not available in practical scenarios and the direct estimation (e.g., based on pairs of inputs and labels) of the true class posterior probability PY |X cannot be performed. Notice that it is not possible to sample the conditional pdf PY |X for each input x ∈ X . As a matter of fact, it is well-known that the application of direct methods for this estimation will lead to ill-posed problems, as shown in [32].

heading: 2.3 Statistical model for detection
text: Given a data sample x ∈ X and an unobserved random label y ∈ Y drawn from the unknown distribution pXY , we wish to predict the realization of the unobserved error variable E = ∆ 1[Y 6= fDn(X)]. To this end, we will model the data distribution as a mixture pdfs,
pXY (x, y) ≡ PE(1)pXY |E(x, y|1) + PE(0)pXY |E(x, y|0),
where pXY |E(x, y|1) denotes the pdf truncated to the error event {E = 1} (i.e., the hard decision fails) and pXY |E(x, y|0) is the pdf truncated to the success event {E = 0} (i.e., the hard decision succeeds). By taking the marginal of pXY over the labels, we obtain: pX(x) = PE(1)pX|E(x|1) + PE(0)pX|E(x|0). First, observe that the problem at hand is to infer E from (x, PŶ |X) since Y is not observed. Second, we further emphasize that in the present framework we assume that there are no available (extra) samples for training a discriminator to distinguish between pX|E(x|0) and pX|E(x|1). It is worth mentioning that a well-trained classifier would imply PE(1) PE(0), since in that case we should have very few classification errors. However, this also implies that it would be very unlikely to have enough samples available to train a good enough discriminator.

heading: 3 Performance Metrics and Discriminators

heading: 3.1 Performance metrics and optimal discriminator
text: We aim to distinguish between samples for which the predictions cannot be trusted and samples for which predictions should be trusted. We first state the optimal rejection region, given by (4), where we suppose the existence of an oracle who knows all the involved probability distributions. Definition 1 (Most powerful discriminator). For any 0 < γ <∞, define the decision region:
A(γ) =∆ { x ∈ X : pX|E(x|1) > γ · pX|E(x|0) } . (4)
The most powerful (Oracle) discriminator at threshold γ is defined by setting D(x, γ) = 1 for all x ∈ A(γ) for which the prediction is rejected (i.e., Ê = 1) and otherwise D(x, γ) = 0 for all x /∈ A(γ) for which the prediction is accepted.
In Proposition 3.1, we establish the characterization of the fundamental performance of the most powerful (Oracle) discriminator by providing a lower bound on the error achieved by any discriminator and show that this bound is achievable by setting γ = 1. Furthermore, we connect this result to the Bayesian error rate of this optimal discriminator. Proposition 3.1 (Performance of the discriminator). For any given decision region A ⊂ X , let
0(A) =∆ ∫ A pX|E(x|0)dx, and 1(Ac) =∆ ∫ Ac pX|E(x|1)dx , (5)
be the Type-I (rejection of the class prediction of an input x that would be correctly classified) and Type-II (acceptance of the class prediction of an input x that would not be correctly classified) error probability, respectively. Then,
0(A) + 1(Ac) ≥ 1− ∥∥pX|E=1 − pX|E=0∥∥TV (6)
= 1− 1 2 ∫ X |pX|E=1(x)− pX|E=0(x)|dx. (7)
Equality is achieved by choosing the optimal decision region A? ≡ A(1) in Definition 1. If the hypotheses are equally distributed, the minimum Bayesian error satisfies:
2 Pr {D(X) 6= E(X)} ≥ 1− ∥∥pX|E=1 − pX|E=0∥∥TV . (8)
Equality is achieved by using the optimal decision region.
Expressions (7) and (8) provide lower bounds for the total error of an arbitrary discriminator. The proof of this proposition is relegated to the Supplementary material (Appendix A.1). Using Bayes we can rewrite (4) via the posteriors as:
A(γ) = { x ∈ X : PE|X(1|x)PE(0) > γ · ( 1− PE|X(1|x) ) PE(1) } . (9)
From (9), it is easy to check that PE|X(1|x) = 1 − PY |X (fDn(x)|x) = Pe(x), and hence, the decision region A(γ) can be reformulated as:
A(γ′) = { x ∈ X : Pe(x)
1− Pe(x) > γ′
} = { x ∈ X : Pe(x) > γ ′
(γ′ + 1)
} , (10)
where γ′ =∆ γ · PE(1)PE(0) and 0 < γ ′ < ∞. According to (10) and Proposition (3.1), the optimal discriminator is given by D?(x, γ′) = 1, whenever x ∈ A(γ′), and D?(x, γ′) = 0, otherwise. The main difficulty arises here since the error probability function of an input: x 7→ Pe(x) is not known and in general cannot be learned from training samples.

heading: 3.2 DOCTOR discriminator
text: We start by deriving an approximation to the unknown function x 7→ Pe(x) which can be used to devise the decision region in expression (10). First, we state the following: Proposition 3.2. Let ĝ(x) be defined by
1− ĝ(x) =∆ ∑ y∈Y PŶ |X(y|x) Pr ( Ŷ 6= y|x ) = 1− ∑ y∈Y P 2 Ŷ |X(y|x), (11)
for each x ∈ X , which indicates the probability of incorrectly classifying a feature x if it was randomly labeled according to the model distribution PŶ |X trained based on the dataset. Then,
(1− √ ĝ(x))−∆(x) ≤ Pe(x) ≤ (1− ĝ(x)) + ∆(x), (12)
where ∆(x) =∆ 2 √ 2 KL ( PY |X(·|x)‖PŶ |X(·|x) ) and denotes the Kullback–Leibler (KL) divergence (further details are provided in Supplementary material Appendix A.2).

heading: 3.3 Discussion
text: It is worth emphasizing that expressions in (12) provide bounds to the unknown function x 7→ Pe(x) using a known statistics x 7→ 1− ĝ(x), which is based on the soft-probability of the predictor. On the other hand, 0 ≤ ĝ(x) ≤ √ ĝ(x) ≤ 1, for all x ∈ X , which simply follows using the subadditive of the function t 7→ √ t and the definition of ĝ(x). By Markov’s inequality,
Pr ( ∆(X) ≥ ε(η) ) ≤ η with ε(η) = 2 √ 2EXY [ − logPŶ |X(Y |X) ] /η, (13)
for any η > 0, where EXY [ − logPŶ |X(Y |X) ] in (13) is the cross-entropy risk. The latter is expected to be small provided that the model generalizes well. Thus, ε(η) can be expected to be small for a desired confidence η > 0. Interestingly, (11) turns out to be related to the uncertainty of the classifier via the quadratic Rényi entropy [31]: − log2 ( ĝ(x) ) = 2H2(Ŷ |x) ≤ 2H(Ŷ |x), where the latter is the Shannon entropy, i.e., the self-uncertainty of the classifier.

heading: 3.4 From the theory to a practical discriminator
text: Our previous discussion suggests that P̂e(x) in (2) may be a valuable candidate to approximate Pe(x) in the definition of the optimal discriminator (10). On the other hand, Proposition 3.2 suggests that 1 − ĝ(x) can also be a valuable candidate yielding another discriminator. These discriminators, referred to as DOCTOR, are introduced below. Definition 2 (DOCTOR). For any 0 < γ <∞ and x ∈ X , define the following discriminators:
Dα(x, γ) = ∆ 1 [1− ĝ(x) > γ · ĝ(x)] , Dβ(x, γ) =∆ 1 [ P̂e(x) > γ · (1− P̂e(x)) ] . (14)
Notice that because of Definition 2 and (11), Dα(x, γ) = 1[1 − ∑ y∈Y softmax 2(x)y >
γ · ∑ y∈Y softmax
2(x)y]; similarly because of Definition 2 and eq. (2), Dβ(x, γ) = 1[1 − maxy∈Y softmax(x)y > γ ·maxy∈Y softmax(x)y]. The performance of these discriminators will be investigated and compared to state-of-the-art methods in the next section. In the supplementary material (Appendix B), we illustrate how DOCTOR and the optimal discriminator (Definition 1) work on a synthetic data model that is a mixture of two spherical Gaussians with one component per class.

heading: 4 Experimental Results
text: In this section we present a collection of experimental results to investigate the effectiveness of DOCTOR, by applying it to several benchmark datasets. We provide publicly available code1 to reproduce our results, and we give further details on the environment, the parameter setting and the experimental setup in the Supplementary material (Appendix C). We propose a comparison with state-of-the-art methods using similar information. Though we are not concerned with the OOD detection problem, we are confident it is appropriate to compare DOCTOR to methods which use soft-probabilities or at most the output of the latent code, e.g., ODIN [23], softmax response (SR) [6] and Mahalanobis distance (MHLNB) [22]. Since we are focusing on misclassification detection, it is expected that OOD samples should be also detected as classification errors.
Totally Black Box (TBB) and Partially Black Box (PBB). We address two different scenarios with respect to the available information about the network. In the TBB only the output of the last layer of the network is available, hence gradient-propagation to perform input pre-processing is not allowed. In the PBB we allow method-specific inputs perturbations. When considering DOCTOR in PBB, for each testing sample x, we calculate the pre-processed sample x̃ by adding a small perturbation:
x̃α = x− × sign [ −∇x log ( 1− ĝ(x)
ĝ(x)
)] , and x̃β = x− × sign [ −∇x log ( P̂e(x)
1− P̂e(x)
)] .
We will write directly x̃ when it is clear from the context which input pre-processing we are referring to. In Supplementary material (Appendix C.2) we further analyze the equations above. When ODIN or MHLNB are used, we pre-process the inputs as in [23] and in [22], respectively.

heading: 4.1 Review of related methods
text: PBB. We compare DOCTOR (using input pre-processing and temperature scaling) with ODIN and MHLNB. ODIN [23] comprises temperature scaling and input pre-processing via perturbation. Temperature scaling is applied to its scoring function, which has fi(x̃) for the logit of the i-th class. Formally, given an input sample x:
SODIN(x̃) = max i=[1:C] exp(fi(x̃)/T )∑C j=1 exp(fj(x̃)/T )
, ODIN(x̃; δ, T, ) = {
out, if SODIN(x̃) ≤ δ in, if SODIN(x̃) > δ,
where x̃ represents a magnitude perturbation of the original x; T is the temperature scaling parameter; δ ∈ [0, 1] is the threshold value; in indicates the acceptance decision while out indicates the rejection decision. Notice, however, γ in DOCTOR and δ in ODIN, respectively, are defined over two different domains: if δ denotes a probability, γ is a ratio between probabilities. Although ODIN
1https://github.com/doctor-public-submission/DOCTOR/
originally required tuning the hyper-parameter T with out-of-distribution data, it was also shown that a large value for T is generally desirable, suggesting that this gain is achieved at 1000. Anyway, in this framework, we notice an improvement of ODIN in performance for low values of T . Thus we report the best results obtained by ODIN considering the range of hyper-parameters values tested also for DOCTOR (cf. section 4.3). ENERGY [24] comprises the denominator of the softmax activation:
ES(x) = −T · log C∑ j=1 exp(fj(x)/T ), ENERGY(x; ξ, T ) = { out, if − ES(x) ≤ ξ in, if − ES(x) > ξ,
where ξ ∈ R is the threshold value. Unlike all the methods considered in this paper, MHLNB [22] requires the knowledge of the training set Dn which the pre-trained network was trained on to compute its empirical class mean µ̂c for each class c and its empirical covariance Σ̂:
µ̂c = 1
nc ∑ i: yi=c f(x̃i); Σ̂ = 1 n ∑ c∈Y ∑ ∀i: yi=c (f(x̃i)− µ̂c)(f(x̃i)− µ̂c)>,
where nc denotes the number of training samples with label c and f(x̃) the logits vector. As MHLNB directly uses the vector of logits, it does not comprise temperature scaling. Given an input sample x:
M(x̃) = max c∈Y
− (f(x̃)− µ̂c)>Σ̂−1(f(x̃)− µ̂c), MHLNB(x̃; ζ, ) = {
out, if M(x̃) > ζ in, if M(x̃) ≤ ζ,
as mentioned above, x̃ represents a magnitude perturbation of the original x; ζ ∈ R+ is the threshold value; in indicates the acceptance decision while out indicates the rejection decision.
TBB. We compare DOCTOR (without input pre-processing and temperature scaling) with MHLNB (without input pre-processing and with the softmax output layer in place of the logits) and SR. Although both DOCTOR and SR have access to the softmax output of the predictor, a fundamental difference is that, while the former utilizes the softmax output in its entirety, the latter only uses the maximum value, therefore discarding potentially useful information. As it will be shown, this leads to better results for DOCTOR on several datasets (see table 1). We emphasize that, by setting T = 1 and = 0, ODIN reduces to softmax response [6] since SR(x) ≡ SODIN(x).

heading: 4.2 Detection of misclassification errors, experimental setup and evaluation metrics
text: Before digging into the detailed discussion of our numerical results, we present an empirical analysis of the behavior of DOCTOR, ODIN, SR and MHLNB when faced with the task of choosing whether to accept or reject the prediction of a given classifier for a certain sample. In Figure 1, we propose a graphical interpretation of the discrimination performance, considering the labeled samples in the dataset TinyImageNet and the ResNet network as the classifier. We separate correctly and incorrectly classified samples according to their true labels in blue and in red, respectively. We remind that the label information is not necessary for the discriminators to define acceptance and rejection regions. Then, for each sample we compute the corresponding discriminators’ output. These values are binned and reported on the horizontal axis of Figure 1a and Figure 1b for Dα, Figure 1c and Figure 1d for Dβ , Figure 1e for SR, Figure 1f for ODIN, Figure 1g and Figure 1h for MHLNB. In each each plot, and according to the corresponding discriminator, the bins’ heights represent the frequency of the samples whose value falls within that bin. The intuition is that, if moving along the horizontal axis it is possible to pick a threshold value such that, w.r.t. this value, blue bars are on one side of the plot and red bars on the other, this threshold would correspond to the optimal discriminator, i.e. the discriminator that chooses the optimal acceptance and rejection regions.
In Figure 1g through Figure 1h, we observe that, for MHLNB, no matter how well we choose the threshold value, it is hard to fully separate red and blue bars both in TBB and PBB, i.e. the discriminator fails at defining acceptance and rejection regions so that all the hits can be assigned to the first one and all the mis-classification to the second one. The samples distribution for SR and ODIN in Figure 1e and Figure 1f, respectively, does not look significantly different from the one related to Dα and Dβ in TBB (Equation (14)). However, the discrimination between samples becomes evident in PBB. This is shown in Figure 1d for Dβ (eq. (14)) and even more in Figure 1b for Dα (eq. (14)) where, quite clearly, rightly classified samples are clustered on the left-end side of the plot and incorrectly classified samples tend to cluster on the right-end side. This intuition is supported by the results in Table 1.
Datasets and pre-trained networks. We run experiments on both image and textual datasets. We use CIFAR10 and CIFAR100 [18], TinyImageNet [16] and SVHN [27] as image datasets; IMDb [25], AmazonFashion and AmazonSoftware [28] as textual datasets. Note that, for all the aforementioned datasets, we consider only the test set since we rely on pre-trained models. Along the same lines of [23], we use the pre-trained DenseNet models [15] for CIFAR10, CIFAR100 and SVHN. In addition, we use a pre-trained ResNet model [10] for TinyImageNet, and BERT [3, 34] for the Amazon datasets and IMDb. The accuracy achieved by the aforementioned networks on the test sets is showed in Table 1. According to the invariant properties of the discriminator (see Def. 2) with respect to the soft-probability of the underlying model, permutations of the posterior probabilities vector, due different initialization of the models before the training, do not change the output of Eq. (10), as it is a sum of squared values of the softmax probabilities. This variety of models/datasets characterizes the performance of the proposed method in scenarios with different accuracy levels.
Evaluation metrics. We will evaluate the performance according to Proposition (3.1) via the empirical estimates of Type-I and Type-II errors in expressions (5). Throughout this section, when the model’s decision for a sample is correct (hit) but is rejected by the discriminator, we refer to such event as false rejection; when the model’s decision for a sample is not correct (miss) and is rejected by the discriminator, we refer to such event as true rejection. Similarly, we refer to a false acceptance when a miss is not rejected and to a true acceptance when a hit is not rejected. More specifically, let Tm = {(x1, y1), . . . , (xm, ym)} ∼ pXY be the testing set, where xi ∈ X is the input sample, yi ∈ {1, . . . , C} is the true class of xi, and m denotes the size of the testing set. With j ∈ {α, β}:
FRj(γ) = {(x, y) ∈ Tm : y = fDn(x), Dj(x, γ) = 1}, (15) T Rj(γ) = {(x, y) ∈ Tm : y 6= fDn(x), Dj(x, γ) = 1}, (16) FAj(γ) = {(x, y) ∈ Tm : y 6= fDn(x), Dj(x, γ) = 0}, (17) T Aj(γ) = {(x, y) ∈ Tm : y = fDn(x), Dj(x, γ) = 0} . (18)
We measure the performance of the test in terms of:
• FRR versus TRR. The false rejection rate (FRR) represents the probability that a hit is rejected, while the true rejection rate (TRR) is the probability that a miss is rejected.
• AUROC. The area under the Receiver Operating Characteristic curve (ROC) [2] depicts the relationship between TRR and FRR. The perfect detector corresponds to a score of 100%.
• FRR at 95 % TRR. This is the probability that a hit is rejected when the TRR is at 95 %.

heading: 4.3 Experimental results: comparison between different discriminators
text: DOCTOR: comparison between Dα and Dβ . We compare the discriminators Dα and Dβ introduced in (14) to show how the AUROCs for CIFAR10, CIFAR100, TinyImageNet and SVHN change when
varying the parameters T and . It is observed that Dα is less sensitive to the selection of T : for all the datasets, Dα outperforms Dβ achieving the best AUROCs by setting T = 1. Contrary to Dα, Dβ is more sensitive to the value selected for T in the sense that small changes may result in very different values for the measured AUROCs (cf. Appendix C.4.1). In contrast,the best results are obtained for the same epsilon values of Dα and Dβ across all the datasets.
Comparison in TBB. We compare DOCTOR with MHLNB (without input pre-processing and with the softmax output in place of the logits) and SR. It is worth to emphasize that Dα does not coincide (in general) with SR since the former consists in the sum of squared values of all probabilities involved in the softmax. To complete the comparison, we include the results for both methods in Table 1.
Comparison in PBB. We compare DOCTOR with ODIN, MHLNB and ENERGY. We keep the same parameter setting for all the methods. In the case of DOCTOR and ODIN where temperature scaling is allowed, we test, for each dataset, 24 different values of for each of the 11 different values of T , see (Appendix C.4.2) for the set of ranges. In the case of MHLNB, which directly uses the logits, we keep T = 1 and we vary for each dataset. In the case of ENERGY, where no perturbation is allowed, we keep = 0 and we maintain T = 1 (as in [24]). According to our framework, no validation samples are available; consequently, in order to be consistent across the datasets, we only report the experimental settings and values for which, on average, we obtain favorable results for all the considered domains (cf. Figure 2). In order to be fair, we update ODIN’s parameters from those in [23] to new values which are more suitable to the task at hand (cf. plots in Appendix C.4.2).
DOCTOR’s performance compared to ODIN’s, MHLNB’s and ENERGY’s, are collected in Table 1 and in Figure 2. The results in the table show that noise further improves the performance of DOCTOR (cf. PBB) up to 1% over our previous experiments without noise (cf. TBB) in terms of AUROC. The improvement is even more significant in terms of FRR at 95% TRR: a 4% decrease is obtained in terms of predictions incorrectly rejected for DOCTOR when passing from TBB to PBB. Note that only the softmax output is available when we consider the pre-trained models for AmazonFashion, AmazonSoftware and IMDb datasets; therefore, we cannot access any internal layer and test DOCTOR for values of T which differ from the default value T = 1. Consequently, temperature scaling and input pre-processing cannot be applied in these cases and thus these datasets cannot be tested in PBB. Moreover, even in TBB, these datasets cannot be tested through MHLNB and ENERGY since the dataset on which the network was trained is not available. We provide simulations on how the range of interval for the different thresholds can affect the results in Appendix C.3.
Misclassification detection in presence of OOD samples. We evaluate DOCTOR’s performance in misclassifcation detection considering a mixture of both in (DATASET-IN) and out-of-distribution (OOD) samples (DATASET-OUT), i.e. input samples for which the decision should not be trusted. The results are compared with ODIN. We test the two methods when one sample to reject out of five (♣), three (♦) or two (♠) is OOD. The details of the simulations, the considered dataset, and the complete experimental results are relegated Appendix C.4.3. In Table 2 we report an extract of the results for the PBB scenario in terms of mean / standard deviation: DOCTOR achieves, and most of the time outperforms ODIN’s performance. We emphasize that, even though DOCTOR is not tuned for the OOD detection problem, it represents the best choice for deciding whether to accept or reject the prediction of the classifier also on mixed data scenarios where the percentage of OOD samples, as long as it is not dominant, can sensitively vary.

heading: 5 Summary and Concluding Remarks
text: We introduced a simple and effective method to detect misclassification errors, i.e., whether a prediction of a classifier should or should not be trusted. We provided theoretical results on the optimal statistical model for misclassification detection and we presented our empirical discriminator DOCTOR. Experiments on real (textual and visual) datasets–including OOD samples and comparison to state-of-the-art methods– demonstrate the effectiveness of our proposed methods. Whilst methods for ODD frameworks do not necessarily perform well in predicting misclassification errors, our result advances the state-of-the-art, and the main takeaway is that DOCTOR can be applied to both partially black-box (PBB) setups and totally black-box (TBB) ones. In the latter, information about the model’s architecture may be undisclosed for security reason when dealing with sensitive data). DOCTOR uses all the information in the softmax output, which results in equal or better performance with respect to the other methods: the results in PBB, where we observe a reduction up to 4% in terms of predictions incorrectly rejected with respect to the ones in TBB are particularly promising. Moreover, DOCTOR does not require training data and, thanks to its flexibility, it can be easily deployed in real-world scenarios. Currently, DOCTOR does not exploit information across the layers yet. Only the soft-predictions are used. Besides, the most important obstacle is the calibration of the threshold (γ) between the desired fault rejection and acceptance rates, which would require additional validation samples. However, quite often, the cost of collecting data for this operation can be prohibitive, making it difficult or too expensive to perform such calibration. As future work, we shall combine DOCTOR with other related lines of research such as: the extension to white-box incorporating additional information across the different latent codes of the model. Moreover, we shall investigate the possibility of combining the two proposed discriminators.

Review summaries:
The paper proposes a method for detecting misclassified samples in neural networks. The proposed method aims to approximate to error function of true probability density (which is unknown) of labels given data by using known statistics. To achieve this, known statistics that bounds the true error function are derived in the paper which are used to define discriminators to determine correctly and incorrectly classified samples. The first discriminator denoted as D_\alpha uses sum of squared softmax probabilities and the second discriminator is based on the predicted model for the posterior class probability denoted as D_\beta. The methods is tested in 2 settings: 1) Partially Black Box (PBB) where it is assumed that gradients of the network w.r.t input is available for input perturbation which was shown to be useful for OOD detection in earlier work, 2) Totally Black Box (TBB) where it is assumed that only softmax probabilities are available. The proposed discriminators are compared with well-known OOD detection methods: ODIN and Mahalanobis on multiple image and text datasets. The results demonstrate that the proposed framework (DOCTOR) outperforms OOD detection methods on detecting misclassification error.  

Motivated by the tendency of DL models to be overconfident, this paper introduces DOCTOR, a framework for determining when to accept/trust model decisions. The authors derive the tradeoffs of accepting an incorrect classification and rejecting a correct classification to motivate the formulation of DOCTOR, a set of discriminators which takes into account the full softmax values of the model to minimize the occurrence of type 1 and type 2 errors. DOCTOR is able to match or outperform prior works on both in-domain and out-of-domain problems for both totally black box and partially black box models. 

The paper addresses the important problem of detecting misclassifications in a classification system. The authors consider two scenarios: total black box setting in which only the softmax predictions are available, and a partial black box setting that assumes access to the model parameters. First, the authors derive a bound to approximate the unknown classification error using a combination of known statistics. Based on this, the authors propose a simple method that can detect misclassifications. The method is shown to work well empirically on some benchmark datasets.