DOCTOR: A Simple Method for Detecting Misclassification Errors
Authors: Federica Granese
Marco Romanelli
Daniele Gorla
Catuscia Palamidessi
Pablo Piantanida
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: AI Safety
Algorithms Evaluation
Deep Learning
misclassification detection
out-of-distribution

Abstract: Deep neural networks (DNNs) have shown to perform very well on large scale object recognition problems and lead to widespread use for real-world applications, including situations where DNN are implemented as “black boxes”.  A promising approach to secure their use is to accept decisions that are likely to be correct while discarding the others.  In this work, we propose DOCTOR, a simple method that aims to identify whether the prediction of a DNN classifier should (or should not) be trusted so that, consequently, it would be possible to accept it or to reject it. Two scenarios are investigated: Totally Black Box (TBB) where only the soft-predictions are available and Partially Black Box (PBB) where gradient-propagation to perform input pre-processing is allowed. Empirically, we show that DOCTOR outperforms all state-of-the-art methods on various well-known images and sentiment analysis datasets. In particular, we observe a reduction of up to 4% of the false rejection rate (FRR) in the PBB scenario. DOCTOR can be applied to any pre-trained model, it does not require prior information about the underlying dataset and is as simple as the simplest available methods in the literature.

DOCTOR: A Simple Method for Detecting
Misclassification Errors
Federica Granese"
Marco Romanelli L2S, CentraleSupélec, CNRS, Université Paris Saclay
Lix, Inria, Institute Polytechnique de Paris,
Sapienza University of Rome
federica. granese@inria.ff
marco www.ooommmmeell11mmmm..m.. hammmaneell@ceetrraeesusseeeee
Daniele Gorla
Catuscia Palamidessi!
Sapienza University Rome gorla@di uniromal.it
Lix, Inria, Institute Polytechnique de Paris,
catuscia@lix catusciaellxxoooyyeeeellgeg polytechnique. fr
Pablo Piantanida
L2S, CentraleSupélec,
CNRS, Université Paris Saclay
Abstract
Deep neural networks (DNNs) have shown to perform very well on large scale object recognition problems and lead to widespread use for real-world applications, including situations where DNN are implemented as "black boxes". A promising approach secure their use is to accept decisions that are likely be correct while discarding the others. In this work, we propose DOCTOR, a simple method that aims to identify whether the prediction of a DNN classifier should (or should not) be trusted that, consequently, it would be possible to accept it or to reject it. Two scenarios are investigated: Totally Black Box (TBB) where only soft-predictions are available and Partially Black Box (PBB) where gradient-proge propagation to perform input pre- preeprocessing is allowed. Empirically, we show that DOCTOR outperforms all state-of-the-an methods various well-known images and sentiment analysis datasets. In particular, we observe a reduction of up to 4% of the false rejection rate (FRR) in the PBB scenario. DOCTOR can be applied any pre- tre-trained model, it does not require prior information about the underlying dataset and is as simple the simplest available methods in the literature.
Introduction
With the advancement of state-of-the-art Deep Neural Networks (DNNs), there has been rapid adoption these technologies a broad range of applications to critical systems, such as autonomous driving vehicles or industrial robots, including-bur not limited to-classification and decision making tasks. Nevertheless, these solutions still exhibit unwanted behaviors as they tend to be overconfident in presence of wrong decisions [17]. Developing methods and tools to make these algorithms
"These authors contributed equally to this work This paper is supported by the ERC project Hypatia under the European Unions Horizon 2020 research and innovation program. Grant agreement X>835294.
*This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska Ceroossk---rieeee- grant agreement No792464.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

reliable, in particular for non-specialists who may treat them as "black boxes" with no further checks, constitutes a core challenge. Recently, the study of safety Al methods has gained ground, and many efforts have been made in several areas [7, 8, 9, 11, 26, 35, 36]. In this paper, we investigate simple method capable of detecting whether prediction of a classifier is likely to be correct, and therefore should be trusted, or it is not, and should be rejected.
Deep learning pursues the idea of learning effective representations from the data itself by training with the implicit assumption that the test data distribution should be similar to the training data distribution. However, when applied to real-world tasks, this assumption does not hold true, leading to a significant increase of misclassification errors, Although classic approaches to out-Of-Distributte (OOD) detection [1. 12. 21, 23, 33] are not directly concerned with detecting misclassffication errors, they are intended to prevent those emrrrs indirectly by identifying potential drifts of the testing distribution. What the above ODD methods have in common with our work is that samples drawn from the in-distribution are more likely to be correctly classified than those from a different distribution. Indeed, the model's s soft-predictions for in-distribution samples tend to be generally peaky in correspondence to the correct class label while they tend to be less peaky for input samples drawn from different distribution [12]. In general, most of these works consider white-box scenarios, where the hidden layers of the architecture are accessible or the corresponding weights are tuned during the training phase. A very effective approach to OOD detection is ODIN [23] which involves the use of temperature scaling and the addition of small perturbations to input samples. A related solution is introduced in [6] where the maximum soft- protbbrobyyility is called softmax response. Within this approach, the softmax response decides whether the classifier is confident enough in its prediction or not. A different approach to OOD detection is given by the use of the Mahalanobis distance [14, 22), which consists in caleulating how much the observed outriistrinution sample deviate from the in- distribution ones but assuming the latter are given.
1.1 Summary of contributions
Our work tackles the problem of identifying whether the prediction of a classifier should or should not be trusted, no matter if they are made on out or in-distribution samples, and advances the state-of-the-art in multiple ways.
From the theoretical point of view, we derive the trade-off between two types of error probabilities: Type-I, that refers to the rejection of the classification for an input that would be correctly classified, and Type-II, that refers to the acceptance of the classification for an input that would not be correctly classified (Proposition 3.1). The characterization of the optimal discriminator in eq. (10) allows to devise feasible implementation of it, based on the softmax probability (Proposition 3.2).
From the algorithmic point of view, inspired by our theoretical analysis, we propose DOCTOR new discriminator (Definition 2), which yields simple and flexible framework to detect whether a decision made by a model is likely to be correct or not. We distinguish two scenarios under which DOCTOR can be deployed: Totally Black Box (TBB) where only the coft-predictions are available, hence not allowed, and Partially Black Box (PBB) where we further allow method-specifie inputs perturbations.
to perform input pre-processing is
From the experimental point of view, we show that DOCTOR outperforms comparable state- of-the-art methods (e.g.. ODIN [23], softmax response [6] and Mahalanobis distance [22]) on datasets including both in-distribution and out-of-distribution samples, and different architectures with various expressibilities, under both TBB and PBB. A key ingredient of DOCTOR is to fully exploit all available information contained in the soft soft-probabilities of the predictions (not only their maximum).
1.2 Related works
Recent works have shown that the accuracy of classifier and its ability to output soft- coft-predictions that represent the true posteriors estimate can be totally disjointed [9, 19, 20). Furthermore, models often tend to be overconfident about their decision even when their predictions fail [11, 17). This motivates a novel research area that strives to develop methods to assess when decisions made by classifiers should should not be trusted. Although the detection of OOD samples is different
2

---

(domain) problem, it is naturally expected that samples from a distribution that is significantly different from the training one cannot be correctly classified. In [23]. the authors propose a method which increases the peakiness of the softmax output by perturbing the input samples and applying temperature scaling [9, 13, 29] to the classifier logits in order better detect in-distribution samples. It is worth noticing that this method requires additional information on the internal structure of the latent code of the model. A very different approach [14, 22] tackles OOD detection by using the Mahalanobis distance. Although this approach appears to be more powerful, it also requires additional samples to learn the mean by class and the covariance matrix of the in-distribution. In [4], classifiers trained to output calibrated confidence estimates that used to perform OOD detection. A related line of research is concerned with the problem of selective predictions (aka reject options) in deep neural networks. The main motivation for selective prediction is reducing the error rate by abstaining from prediction when in doubt, while keeping the number of correctly classified samples as high possible [5, 6, 7]. The idea is combine classifiers with rejection functions by observing the classifiers' output, without using any supervision, to decide whether to accept or to reject the classification outcome. In [6], the authors introduce softmax response, rejection function which compares the maximum soft- oot--robabbilty to a pre-determined threshold to decide whether to accept or reject the class prediction given by the model.
2 Main Definitions and Preliminaries
2.1 Basic definitions
We start by introducing some definitions and background; then, we describe our statistical model and some useful properties about the underlying detection problem. Let X' C IR be the (possibly continuous) feature space and let y {I,..., C'} denote the concept of the label space related to some task of interest. Moreover, let pxy be the underlying (unknown) probability density function (pdf) over X y. Let D = {x11 ,1.,.... (Xn: Mn)) Pxy be a random realization of n i.i.d. samples according to PXY denoting the training set, where x; E X' is the input (feature), y/s E y the output class among C/ possible classes and denotes the size of the training set. A predictor fD.. X + )) uses the inferred model PpIx- = Ppix (V x; D,) based on the training set,
SD., (x) = (x(x; A arg max yEY
Ppix (vlx;D.))
and tries to approximate the optimal (Bayes) decision rule (x(x arg max Prix(ulx) Notice yEy
that Ppix can be interpreted as the prediction of the class (label) posterior probability given a sample (c.g., P P.g...PIx (y xx) = softmax(x)y), while Pyyx is the true (unknown) probability. In several practical
scenarios Pg x does not perfectly match Pyix and still SD. s I* (cf. [9]).
2.2 Error variable
Let E(x) 2 [ + [DD denote the error variable for a given x E X' corresponding to SD... i.e. where we denote with [[]] the indicator vector which outputs 1 if the event & is true and 0 otherwise. Similarly, we can define the self-error variable E(x) e [[ * fo, (x)] also corresponding to the inferred predictor fD., but based on the prediction model Ppix of the class posterior probability.
Notice that Exx) is observable since the underlying distribution is known. However, E(x) cannot be observed and in general these binary variables do not coincide.
At this stage, it is convenient to introduce the notions of probability of classificution error for a given x E X w.r.t. both the true class posterior and the predicted probabilities:
Pe(x) e E
E
x x = 1 - Prix (x))x),
Pe(x) A E
[x)]xx 1 Ppix (x))x).
Notice that Pe(x) represents the probability of misclassification of the sample with respect to the softmax probability Prix- which can be interpreted as the model's approximation of nature
Prix- Such approximation is close when the model is well-calibrated. Obviously, Pe*(x) < Pe(x) for all x X, where Pe" (x) corresponds to the minimum error of the Bayes classifier:
3

---

(x) =1 Pyjx (x)x)) It is worth mentioning that, by averaging (1) over the data distribu- tion, we obtain the error rate of the classifier fD.. Although Pe(x) provides valuable candidate to infer the unknown enror variable Exx), it is easy to check that
max (eeex) Pr (Y
x|x) < Pr (E(x) + S Pr (8 +Y|x),
(3)
which in particular implies that the error incurred in using E(x) to predict E(x) is lower bounded by the classification enror per sample (1). The proof' are in Supplementary material (Appendix A.3).
In this paper, we aim at identifying discriminator capable of distinguishing between inputs for which we can trust the predictions of the classifier JD., (x) (i.e.. F(() = 0) and those for which we should not trust predictions (i.e., F((x) 1). In the next section, we will show that the function Pe(x) : X' + [0, 1] plays a central role in the characterization of the optimal discriminator. However, Pe(x) is not available in practical scenarios and the direct estimation (e.g., based on pairs of inputs and labels) of the true class posterior probability PYix cannot be performed. Notice that it is not possible sample the conditional pdf Py for each input x X. As matter of fact, it well-known that the application of direct methods for this estimation will lead to ill- ill-posed problems, as shown in [32].
2.3 Statistical model for detection
Given a data sample x X and an unobserved random label y y drawn from the unknown distribution pxy, we wish to predict the realization of the unobserved error variable E' 2 1[Y o fD. (X)]. To this end, we will model the data distribution mixture pdfs,
px) (x())xxxjj(xx, y/1 + (0))xxx1)) y/0),
where P 1 denotes the pdf truncated the error event {E = 1} (i.e., the hard decision
fails) and P (x, y0) is the pdf truncated to the success event {F 0} (i.e., the hard decision succeeds). By taking the marginal of pxy over the labels, we obtain: px (x(x) PE (1)pxx ( (x
(=(())x(((((())) First, observe that the problem at hand is to infer E from (x, P since Y is not observed. Second, we further emphasize that in the present framework are no available (extra) samples for training a discriminator to distinguish between p. P (x(0) and (x))) It is worth mentioning that a well-trained classifier would imply (E(1) < (0), since in that case we should have very few classification errors. However, this also implies that it would be very unlikely to have enough samples available to train a good enough discriminator.
assume that there
3 Performance Metrics and Discriminators
3.1 Performance metries and optimal discriminator
We aim to distinguish between samples for which the predictions cannot be trusted and samples for which predictions should be trusted. We first state the optimal rejection region, given by (4), where we suppose the existence of an oracle who knows all the involved probability distributions.
Definition (Most powerful discriminator). For any 0 < y < co, define the decision region:
A(7) {x E X ( > 7 Px| (x
The most powerful (Oracle) discriminator at threshold 7 is defined by setting D 7) = for all x E A(r) for which the prediction is rejected (i.e., E 1) and otherwise D(x, 7) = 0 for all x & A(r) for which the prediction is accepted.
In Proposition 3.1, we establish the characterization of the fundamental performance of the most powerful (Oracle) discriminator by providing lower bound on the error achieved by any discriminator and show that this bound is achievable by setting 1. Furthermore, we connect this result to the Bayesian error rate of this optimal discriminator.
Proposition 3.1 (Performance of the discriminator). For any given decision region C X, let
Eo(A) / (x)xxx and (1 a / A (x]x(x|[)dx,
(5)

---

be the Type-I (rejection of the class prediction of an input x that would be correctly classified) and Type-II II (acceptance of the class prediction of an input x that would not be correctly classified) error probability, respectively. Then,
eo (A + (A9) ) 2 - |XI|== - PX|E=O
(6)
= =1- =1- (Xx|E=I(X) - Px P
(7)
Equality is achieved by choosing the optimal decision region A' = A(1) in Definition the hypotheses equally distributed, the minimum Bayesian enror satisfies:
Pr (D(X) * E(X) 2 1 ||px|==1 Pxj
(8)
Equality is achieved by using the optimal decision region.
Expressions (7) and (8) provide lower bounds for the total error of an arbitrary discriminator. The proof this proposition is relegated to the Supplementary material (Appendix A.1). Using Bayes we can rewrite (4) via the posteriors as:
A(r) = {x E X' (EIx(1 > 7- (x( PE(1))
(9)
From (9), it is easy to check that PEIX (1xx) 1 - Prix (x)|x) -Pe(x), and hence. the decision region A(r) can be reformulated as:
A(7) x X X: 1- Pe(x) { 1-(xEX X Pe(x) :e(x)
Pe(x)
(10)
( + +)
where 7' 7 (a(1) and 0 <2' < co. According to (10) and Proposition (3.1), the optimal (U)
7
discriminator is given by D D* ) whenever A(r'), and D* 7²) 0, otherwise. The
main difficulty arises here since the error probability function of an input: - Pe(x) is not known and in general cannot be learned from training samples.
3.2 DOCTOR discriminator
We start by deriving an approximation to the unknown function x + Pe(x) which can be used to devise the decision region in expression (10). First, we state the following: Proposition 3.2. Let g(x) be defined by
(rix (v # u)x) = 1 E
(11)
VEY
VEY
for each E X, which indicates the probability of incorrectly classifying a feature x if it was randomly labeled according to the model distribution Pvzx trained based on the dataset. Then,
v e - A(x) < Pe(x) < (1 g(x)) Axx),
(12)
where A(x) 2 2 ( (-1x) and denotes the Kullback-Leibler (KL) divergence (further details are provided in Supplementary material Appendix A.2).
3.3 Discussion
It is worth emphasizing that expressions in (12) provide bounds to the unknown function x + Pe(x) using a known statistics x + 1-g(x). which based on the soft- poft-probability of the predictor. On the other hand, 0 < @(x) S (g(x) < 1, for all x E X, which simply follows using the subadditive of the function f + Vi and the definition of gx)). (.x. By Markov's 's inequality.
Pr (A 2 s()) S1 with (1)) = 2 x
log Ppix (YIX) /n.
(13)
for any 77 O, where Exr log Py XY] in (13) is the cross-entropy risk. The latter is expected to be small provided that the model generalizes well. Thus, s(n) can be expected to be small for desired confidence 7) > O. Interestingly, (11) turns out be related the uncertainty of the classifier via the quadratic Rényi entropy [31]: log, (g(x)) = 2H (P|()) S (f)x). where the latter is the Shannon entropy, i.c., the self-uncertainty of the classifier.
5

---

3,4 From the theory to practical discriminator
Our previous discussion suggests that Pe(x) in (2) may be valuable candidate to approximate Pe(x) in the definition of the optimal discriminator (10). On the other hand, Proposition 3.2 suggests that 1 - g(x) can also be a valuable candidate yielding another discriminator. These discriminators, referred to as DOCTOR, are introduced below.
Definition 2 (DOCTOR). For any 0 << oo and x E X, define the following discriminators:
Da(x, 7) - - g(x) > y g(x)), (x(x, e 11 [P((x) 7 -Pe(x))] (14)
Notice that because of Definition and (II), 7x,x,)) = 1[1 Eyey softmax (x), > Eyey (x)))] similarly because of Definition 2 and eq. (2), D2(x, = 1[1 maxyey softmax(x), 7 maXyey softmax(x)y)- The performance of these discriminators will be investigated and compared to state-of-the-ar methods in the next section. In the supplementary material (Appendix B), illustrate how DOCTOR and the optimal discriminator (Definition 1) work on a synthetic data model that is a mixture of two spherical Gaussians with one component per class.
4 Experimental Results
In this section we present collection of experimental results to investigate the effectiveness of DOCTOR, by applying it to several benchmark datasets. We provide publicly available code! to reproduce our results, and we give further details on the environment, the parameter setting and the experimental setup in the Supplementary material (Appendix C). We propose comparison with state-of-the-art methods using similar information. Though we are not concerned with the OOD detection problem, we are confident it is appropriate to compare DOCTOR to methods which use soft soft-probabee cett-probabilitiss or at most the output of the latent code, e.g., ODIN [23], softmax response (SR) [6] and Mahalanobis distance (MHLNB) [22]. Since are focusing misclassification detection, it is expected that OOD samples should be also detected as classification errors.
Totally Black Box (TBB) and Partially Black Box (PBB). We address two different scenarios with respect the available information about the network. In the TBB only the output of the last layer of the network is available, hence In the PBB we allow method-specific inputs perturbations. When considering DOCTOR in PBB, for each testing sample x, we calculate the pre-processed sample x by adding small perturbation:
to perform input pre- pre-processing is not allowed.
- {(x)
)| and X = - sign
Pe(()
]
N - c x sign
V log
Vx log
1 - Pe(x))
We will write directly X when it is clear from the context which input pre- prepcrecessnng we are referring to. In Supplementary material (Appendix C.2) we further analyze the equations above. When ODIN or MHLNB are used, we pre-process the inputs as in [23] and in [22], respectively.
4.1 Review of related methods
PBB. We compare DOCTOR (using input pre-processing and temperature scaling) with ODIN and MHLNB. ODIN [23] comprises temperature scaling and input pre-processing via perturbation. Temperature scaling applied to its scoring function, which has fi(x) for the logit of the i-th class.
Formally, given an input sample x:
exp(F((X)TT) max v=[1:C
out, in. if SODIN(X) > 6,
if SODIN(X) S 6
SODIN(X)
D (x)(T 1
where represents a magnitude e perturbation of the original x; T is the temperature scaling parameter; å E [0, 1] is the threshold value; in indicates the acceptance decision while out indicates the rejection decision. Notice, however, 7 in DOCTOR and ő ODIN, respectively, are defined over two different domains: if å denotes a probability, is a ratio between probabilities. Although ODIN

---

originally required tuning the hyper-parameete parameter T' with out-of out-o--distribution data, it was also shown that a large value for T' generally desirable, suggesting that this gain is achieved at 1000. Anyway, in this framework, we notice an improvement of ODIN in performance for low values of T. Thus we report the best results obtained by ODIN considering the range of syper-parameters values tested also for DOCTOR (cf. section 4.3). ENERGY [24] comprises the denominator of the softmax activation:
ES = -lgg Σ exp(fx(x)/T), E) out, if - ES(x) S E
in, if - ES(x) > &.
where & E R is the threshold value. Unlike all the methods considered in this paper. MHLNB [22] requires the knowledge of the training set D,, which the pre-trained network was trained on to compute its empirical class mean Pe for each class and its empirical covariance E:
Pcc
Σ = The R.
f(x))
Σ E (f(x)) - -
nEY VE: yi=c
where n. denotes the number of training samples with label c and f(x) the logits vector. As MHLNB directly uses the vector of logits, it does not comprise temperature scaling. Given an input sample x:
{ outt, if if > S
M ((X)
max cEY
(J(x) - T Pe). CH, = in, if S.
as mentioned above, X represents magnitude e perturbation of the original x; Ç E R2 is the threshold value; in indicates the acceptance decision while out indicates the rejection decision.
We compare DOCTOR (without input pre-processing and temperature scaling) with MHLNB (without input pre-processing and with the softmax output layer in place of the logits) and SR. Although both DOCTOR and SR have access to the softmax output of the predictor, fundamental difference is that, while the former utilizes the softmax output in its entirety, the latter only uses the maximum value, therefore discarding potentially useful information. As it will be shown, this leads to better results for DOCTOR on several datasets (see table 1). We emphasize that, by setting T = 1 and 0, ODIN reduces softmax response [6] since SR(x) SODIN(x).
TBB.
4.2 Detection of misclassification errors, experimental setup and evaluation metries
Before digging into the detailed discussion of our numerical results, we present an empirical analysis of the behavior of DOCTOR, ODIN, SR and MHLNB when faced with the task of choosing whether to accept or reject the prediction of a given classifier for a certain sample. In Figure 1, we propose a graphical interpretation of the discrimination performance, considering the labeled samples in the dataset Tiny Imageee and the ResNet network as the classifier. We separate correctly and incorrectly classified samples according to their true labels in blue and in red, respectively. We remind that the label information is not necessary for the discriminators to define acceptance and rejection regions. Then, for each sample we compute the corresponding discriminators' output. These values are binned and reported on the horizontal axis of Figure la and Figure lb for D.. Figure Ic and Figure Id for D3. Figure le for SR, Figure If for ODIN, Figure Ig and Figure 1h for MHLNB. each each plot, and according to the corresponding discriminator, the bins' heights represent the frequency of the samples whose value falls within that bin. The intuition that, if moving along the horizontal axis it is possible to pick threshold value such that, w.r.t. this value, blue bars are one side of the plot and red bars on the other, this threshold would correspond to the optimal discriminator; i.e. the discriminator that chooses the optimal acceptance and rejection regions.
In Figure ig through Figure Ih, we observe that, for MHLNB, no matter how well we choose the threshold value, it is hard to fully separate red and blue bars both in TBB and PBB, i.c. the discriminator fails defining acceptance and rejection regions so that all the hits can be assigned to the first one and all the mis-classiffcation to the second one. The samples distribution for SR and ODIN in Figure le and Figure 1f, respectively, does not look significantly different from the one related to D. and D3 in TBB (Equation (14)). However, the discrimination between samples becomes evident in PBB. This shown in Figure 1 for D, (eq. (14)) and even more in Figure Ib for D. (eq. (14)) where, quite clearly, rightly classified samples are clustered on the left-end side of the plot and incorrectly classified samples tend to cluster on the right-end side. This intuition supported by the results in Table 1.
7

---

16:1
Wrong Correct
1.4
Wrong Correct
2.0
Wrong Corfect
Wrong Correct
Somple free
Sample free
-3
Sample free
1,00
1.0
0.75
1.0
0.25
1- g1x0
1-
-19)
Rota]
Peli)
(a) Da -TBB
(b) D -BBB
(c) D TBB
(d) Dy- PBB
L33
Correct
Sample free.
2.2
Correct
Sample freq
Correct
Sample freq
Correct
Samge
0.32
3.5
25
SRUK1
SODINO
MXX
MIX
(c) SR TBB
(f) ODIN PBB
MHLNB TBB
(h) MHLNB PBB
Figure 1: DOCTOR, ODIN, SR and MHLNB to split data samples in TinylmageNet both under TBB
and PBB: (a) (b) show the results for expressions (2); (c) (d) show the results for (11); (e) shows
the results for SR; (f) shows the results for ODIN; (g) - (h) show the results for MHLNB. Histograms
for wrongly classified samples (red) and correctly classified samples (blue).
Datasets and pre-trained networks. We run experiments on both image and textual datasets. We use CIFARIO and CIFAR 100 [18], TinyImageNee [16] and SVHN [27] as image datasets; IMDb [25], AmazonFashion and AmazonSoftware [28] as textual datasets. Note that, for all the aforementioned datasets, we consider only the test set since we rely on pre-trained models. Along the same lines of [23]. use the pre-trained DenseNet models [15] for CIFARIO. CIFAR 100 and SVHN. In addition, we use a pre-trained ResNet model [10] for TinylmageNet, and BERT [3, 34] for the Amazon datasets and IMDb. The accuracy achieved by the aforementioned networks on the test sets is showed in Table 1. According to the invariant properties of the discriminator (see Def. 2) with respect to the soft-probability of the underlying model, permutations of the posterior probabilities vector, due different initialization of the models before the training, do not change the output of Eq. (10), as it is a sum of squared values of the softmax probabilities. This variety of models/datasets characterizes the performance of the proposed method in scenarios with different accuracy levels.
Evaluation metries. We will evaluate the performance according to Proposition (3.1) via the empirical estimates of Type-I and Type-II errors in expressions (5). Throughout this section, when the model's decision for sample is correct (hit) but is rejected by the discriminator, we refer such event as false rejection; when the model's decision for a sample is not correct (miss) and is rejected by the discriminator, we refer to such event as true rejection. Similarly, we refer to a false acceptance when a miss is not rejected and to a true acceptance when a hit is not rejected. More specifically, let Ti {(x1, v1),... (Xm: }nm)}
pxy be the testing set, where x; E X' is the input sample, y: E {1...., C} is the true class of x,, and m denotes the size of the testing set. With j E {a,
FR₂(7) {xx 3) E Tm y fD. (x), Dy(x, 7) 1}.
(15)
(R2(7) = {(x.u) E Tm : # fo. (x), Dj(x, 7) = 1},
(16)
FA2(7) ((x,)) E Tm : + la. (x), D5(x,7) 0},
(17)
(A,(7) = [((x,)) E Tm = fo. (x), Dy(x, 7) = =0}-
(18)
We measure the performance of the test in terms of:
FRR versus TRR. The false rejection rate (FRR) represents the probability that a hit is rejected, while the true rejection rate (TRR) is the probability that a miss is rejected.
AUROC. The area under the Receiver Operating Characteristic curve (ROC) [2] depicts the relationship between TRR and FRR. The perfect detector corresponds to a score of 100%.
FRR at 95 % TRR. This the probability that hit is rejected when the TRR at 95 %.
4.3 Experimental results: comparison between different discriminators
DOCTOR: comparison between D., and Da. We compare the discriminators D. and D3 introduced in (14) to show how the AUROCs for CIFARIO, CIFAR TinyImageNet and SVHN change when
8

---

varying the parameters T and It is observed that Da is less sensitive to the selection of T; for all the datasets, D, outperforms D achieving the best AUROCs by setting T 1. Contrary Da. D3 is more sensitive to the value selected for T in the sense that small changes may result in very different values for the measured AUROCs (ef. Appendix C.4.1). In contrast,the best results are obtained for the same epsilon values of Da and D3 across all the datasets.
Comparison in TBB. We compare DOCTOR with MHLNB (without input pre-processing and with the softmax output in place of the logits) and SR. It is worth to emphasize that Da does not coincide (in general) with SR since the former consists in the sum of squared values of all probabilities involved in the softmax. To complete the comparison, we include the results for both methods in Table 1.
-00
1833
3.00
3995-
2313
1915
1950
1953
1853
1955
115
255
235
COW
3005
PHUM
1835
MHUN
AALM
CIERCY
EVENCY
MM
MR
HM
MR
(a) CIFARIO PBB
(b) CIFARIOO PBB (c) TinyImageNet PBB
(d) SVHN PBB
Figure 2: ROC curves. Comparison between Dr (Ta, 1 and e., 0.00035), D3 (T,3 1.5 and
fa = 0.00035), ODIN (TODIN = 1.3 and EODIN = 0), MHLNB (TMHLNB = and <M = 0.0002)
and ENERGY (TeNeRGY 1 and cENERGY = 0). Red dashed lines mark the 95% threshold of TRR.
Comparison in PBB. We compare DOCTOR with ODIN, MHLNB and ENERGY. We keep the same parameter setting for all the methods. In the case of DOCTOR and ODIN where temperature scaling is allowed, we test, for each dataset, 24 different values of for each of the 11 different values of T, see (Appendix C.4.2) for the set of ranges. In the case of MHLNB, which directly uses the logits, we keep T = and we vary for each dataset. In the case of ENERGY. where no perturbation allowed, we keep E and we maintain T 1 (as [24]). According our framework, no validation samples are available; consequently, in order to be consistent across the datasets, we only report the experimental settings and values for which, on average. we obtain favorable results for all the considered domains (cf. Figure 2). In order to be fair, we update ODIN's parameters from those in [23] to new values which are more suitable to the task at hand (cf. plots in Appendix C.4.2).
DOCTOR's 's performance compared to ODIN's, S, MHLNB's s and ENERGY's, are collected in Table 1 and in Figure 2. The results in the table show that noise further improves the performance of DOCTOR (cf. PBB) up to 1% over our previous experiments without noise (cf. TBB) in terms of AUROC. The improvement is even more significant in terms FRR at 95% TRR: 4% decrease is obtained in terms of predictions incorrectly rejected for DOCTOR when passing from TBB to PBB. Note that only the softmax output is available when we consider the pre-trained models for AmazonFashion. AfazonSoftware and IMDb datasets; therefore, we cannot access any internal layer and test DOCTOR for values of T which differ from the default value = 1. Consequently, temperature scaling and input pre-processing cannot be applied in these cases and thus these datasets cannot be tested in PBB. Moreover, even in TBB, these datasets cannot be tested through MHLNB and ENERGY since the dataset on which the network was trained is not available. We provide simulations on how the range of interval for the different thresholds can affect the results in Appendix C.3.
Misclassification detection in presence of OOD samples. We evaluate DOCToR's performance in misclassifcation detection considering a mixture of both in (DATASET-IN) and out-of-distribtiio (OOD) samples (DATASET-OUT). i.c. input samples for which the decision should not be trusted. The results are compared with ODIN. We test the two methods when one sample reject out of five (+)) three (4) or two (4) is OOD. The details of the simulations, the considered dataset, and the complete experimental results are relegated Appendix C.4.3. In Table 2 we report an extract of the results for the PBB scenario in terms of mean / standard deviation: DOCTOR achieves, and most of the time outperforms ODIN's performance. We emphasize that, even though DOCTOR not tuned for the OOD detection problem, it represents the best choice for deciding whether to accept or reject the prediction of the classifier also on mixed data scenarios where the percentage of OOD samples, as long as it is not dominant, can sensitively vary.

---

Table 1: For all methods, in TBB, we set T and 0: in PBB we set ca CB 0.00035.
To = 1, Tp = 1.5, CODIN = 0 and TODIN = 1.3, EHI NB = 0.0002 and THHLN = 1, CENERGY = 0 and TENERGY = 1. In TBB, ODIN and SR coincide (T and c = 0).
DATASET
METHOD
AUROC'S %
FRR 9 195
TKRI
D.
TBB
94
PBB
95.2
TB8
17.9
PBB
13.9
DATASET
METHOD
TBB
AUROC %
PBB
FRR (95 TRR)
TBB
PBB
CIFARIO 95%
D2
68.5
94.8
18.6
13.4
92.3
38.6
ODIN
SR
93.8
93.8
94.2
18.2
18.2
18.4
SVIN
Acc. 96%
92.2
92.3
92.3
92.8
39.7
38.6
MHLNB
92.2
84.4
30.8
44.6
92.3
38.6
ENERGY
91.1
34.7
MHLNB
87.3
85.8
Da
87
88.2
40.6
35.7
ENERGY
83.9
49.4
CIFARIO0
D;
84.2
87.4
40.6
36.7
89.7
78%
ODIN
SR
86,9
86.9
87.1
40.5
40.5
40.7
Acc. 85%
ASAZON
FASHIDN
89.7
26.3
MELNB
82.6
66.7
94
D.
68.8
73.2
ENERGY
78.7
65.4
AMAZON SOFTWARE
68.8
73.2
45.8
Acc. 75%
67.3
86.6
TINY
Dy
84.9
$5.3
45.8
45.1
Acc. 635%
IMAGENET
ODIN
84.9
84.9
45.8
45.8
Acc. 90%
IMDn
n1
Da
84.4
84.4
54.2
54.4
SR
83.7
61.7
MULNB
78.4
82.3
ENERGY
78.2
63.7
Table 2: Same parameter setting as in table 1 (PBB) for D Dg. ODIN, ENERGY; as in [23] for ODINOOD and as in [22] for wB. Results presented in terms of mean / standard deviation.
BATASET-
DATASEE-
ALBUC
FRR
0g
DDIN
ODINeo
UNBRGY
0g
ODIN
OntNue
ENTIGY
CIPARIO
ISUN
15.470.1
15.1/0.1
94..0.11
19570
92.71L1
34.56.11
14/05
13.5/04
17.2/0.3
35.970
32.27011
92/0.1
Toox
95.2/0.1
94.970
92.3/01
56,270
14/0.55
17,8/0.4
35,0/0
32.2/01
90.3
CIRARIO
ISUS
95.5
95.3/9.1
94.976.1
91.570
92.910
54.576.1
14.7766
13.4/02
15.
34/0.1
27/1
92/022
Tooe (165)
95.470.1
35/00.1
94.875.1
91.470
01.870
36.2/0.1
15/0.1
14/5/0
17/05
34.370.
28.8/19
90/8.33
I50%
95.670
15.60
95:470
99.570
95.670
546/0
15119
136705
16.
30.600
251209
95/0.3
Teoe (RES)
95.670.1
95.2//.1
95.176.1
03.3
03.510
56.2/0.2
147/0.
14.370.5
17.1
3120
25.6703
00.2
5 Summary and Concluding Remarks
We introduced a simple and effective method to detect misclassification errors, i.e., whether a prediction of classifier should or should not be trusted. We provided theoretical results the optimal statistical model for misclassification detection and we presented our empirical discriminator DOCTOR. Experiments real (textual and visual) datasets-including OOD samples and comparison to state-of-the-art methods- demonstrate the effectiveness of our proposed methods. Whilst methods for ODD frameworks do not necessarily perform well in predicting misclassification errors, our result advances the ofte--of-the-ar and the main takeaway is that DOCTOR can be applied to both partially black-box (PBB) setups and totally black-box (TBB) ones. In the latter, information about the model's 's architecture may be undisclosed for security reason when dealing with sensitive data). DOCTOR uses all the information in the softmax output, which results in equal better performance with respect to the other methods: the results in PBB, where we observe a reduction up to 4% in terms of predictions incorrectly rejected with respect to the ones in TBB are particularly promising. Moreover, DOCTOR does not require training data and, thanks to its flexibility, it can be easily deployed in real-world scenarios. Currently, DOCTOR does not exploit information across the layers yet. Only the soft- noft-predictions are used. Besides, the most important obstacle is the calibration of the threshold (7) between the desired fault rejection and acceptance rates, which would require additional validation samples. However, quite often, the cost of collecting data for this operation can be prohibitive, making it difficult or too expensive to perform such calibration. As future work, we shall combine DOCTOR with other related lines of research such as: the extension to white-box incorporating additional information across the different latent codes of the model. Moreover, we shall investigate the possibility of combining the two proposed discriminators.
10

---

References
[1] J. Chen, Y. Li, X. Wu, Y. Liang, and S. Jha. Robust out-of-distribution detection in neural networks. arXiv preprint 1,Xiv:2003.0771,, 2020.
[2] Davis and M. Goadrich. The relationship between precision-recall and ROC curves. In Machine Learning, Proceedings of the Twenty-Thirl International Conference (ICMI. 2006), Pittsburgh, Pennsylvania, USA, June 25-29, 2006, volume 148, pages 233-240, 2006.
[3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and
Short Papers), pages 4171-4186, 2019.
[4] T. DeVries and G. W. Taylor. Learning confidence for out-of-distribution detection in neural networks. CoRR, abs/ 00118002..4.6. 2018.
[5] A. Gangrade, A. Kag, and V. Saligrama. Selective classification via one-sided prediction. In A. Banerjee and K. Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statisties, AISTATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of Machine Learning Research, pages 2787-187 PMLR, 2021.
[6] Y. Geifman and R. El-Yaniv. Selective classification for deep neural networks. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Gamett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4878-4887, 2017.
[7] Y. Geifman and R. Yaniv. Selectivenet: A deep neural network with integrated reject option. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97, pages 2151-2155 2019.
[8] Y. Geifman, G. Uziel, and R. Elniv... Reduced uncertainty estimation for deep neural classifiers. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
[9] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference Machine Learning, 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70, pages 1321-1330, 2017.
[10] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[11] M. Hein, M. Andriushchenko, and J. Bitterwolf. Why networks yield high-confidence predic- tions far away from the training data and how to mitigate the problem. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 41-50. Computer Vision Foundation / IEEE, 2019.
[12] D. Hendrycks and K. Gimpel. bascline for detecting misclassified and out-of-distribution examples in neural networks. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
[13] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
[14] Y. Hsu, Y. Shen, H. Jin, and Z. Kira. Generalized ODIN: detecting out-off-istrrbution image without learning from duits--dist data. In 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10948-10955 2020.
[15) G. Huang, Z. Liu, and K. Q. Weinberger. Densely connected convolutional networks. CoRR, htt//6088.6600 2016.
11

---

[16] Q. Z. Jiayu Wu and G. Xu. Tiny imagenet challenge. Technical report, 2017.
[17] A. Kristiadi, M. Hein, and P. Hennig. Being bayesian, even just bit, fixes overconfidence in relu networks. In Proceedings the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5436-5446. PMLR, 2020.
[18] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[19] V. Kuleshov and S. Ermon. Reliable confidence estimation via online learning. CoRR, abs/1 607.003994 2016.
[20] V. Kuleshov and P. Liang. Calibrated structured prediction. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Gameet, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 3474-3482, 2015.
[21] K. Lee, H. Lee, K. Lee, and J. Shin. Training confidence-calibrate classifiers for detecting ditti--dittiittionn samples. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, 2018.
[22] K. Lee, K. Lee, H. Lee, and J. Shin. A simple unified framework for detecting distri-dittiitttinnn samples and adversarial attacks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 7167-7177, 2018.
[23] Liang, Y. Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018
[24] W. Liu, X. Wang, D. Owens, and Y. Li. Energy-based out-of-distribution detection. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[25] A. L. Maas. R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, June 2011.
[26] A. Meinke and M. Hein. Neural networks that provably know when they don't 't know. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
[27] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. NIPS, 01 2011.
[28] J. Ni, J. Li, and J. J. McAuley. Justifying recommendations using distantly- reviews and fine- fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 188-197, 2019.
[29] J. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv. Large Margin Classif., 10, 06 2000.
[30] A. B. Tsybakov. Introduction to Nonparametric Estimation. Springer Publishing Company, Incorporated, 1st edition, 2008
[31] T. van Erven and P. Harremos. Rényi Divergence and Kullback-Leibler Divergence. IEEE Transactions on Information Theory, 60(7):3797-3820 2014.
[32] V. Vapnik and R. Izmailov. Complete statistical theory of learning: learning using statistical invariants. In Proceedings of the Ninth Symposium on Conformal and Probahilistic Prediction and Applications, volume 128, pages 4-40, 2020.
12

---

[33] A. Vyas, N. Jammalamadaka. X. Zhu, D. Das. B. Kaul, and T. L. Willke. oitoof-distribution detection using an ensemble of self supervised leave-out classifiers. Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedinggs, Part VIII, volume 11212, pages 560-574, 2018.
[34] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowiez, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jermite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Transformers: Sf natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 38,45, oct 2020.
[35] C. Xing, S. Arik, Z. Zhang, and T. Pfister. Distance-based learning from errors for confidence calibration. In International Conference Learning Represenlations, 2020.
[36] Z. Zhang, A. Dalca, and M. R. Sabuneu. Convolutional neural networks using structured dropout. CoRR, 10ss/9906,0951,, 2019.
13


Reviews:
[{"review_id": "y2iFGuzcvz", "reviewer": "Reviewer_4WST", "summary": "The paper proposes a method for detecting misclassified samples in neural networks.\nThe proposed method aims to approximate to error function of true probability density (which is unknown) of labels given data by using known statistics.\nTo achieve this, known statistics that bounds the true error function are derived in the paper which are used to define discriminators to determine correctly and incorrectly classified samples.\nThe first discriminator denoted as D_\\alpha uses sum of squared softmax probabilities and the second discriminator is based on the predicted model for the posterior class probability denoted as D_\\beta.\nThe methods is tested in 2 settings: 1) Partially Black Box (PBB) where it is assumed that gradients of the network w.r.t input is available for input perturbation which was shown to be useful for OOD detection in earlier work, 2) Totally Black Box (TBB) where it is assumed that only softmax probabilities are available.\nThe proposed discriminators are compared with well-known OOD detection methods: ODIN and Mahalanobis on multiple image and text datasets.\nThe results demonstrate that the proposed framework (DOCTOR) outperforms OOD detection methods on detecting misclassification error.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "** Strenghts **\n\n- The proposed discriminators are very simple yet seems very effective based on the results presented in the paper.\n- The paper presents statistical derivations of the proposed discriminators which supports why the proposed discriminators work.\n- Extensive experimental evaluations on both image and text dataset demonstrate that DOCTOR can detect misclassification error on the dataset they are trained on (see Table 1)\n- The paper also presents experiments when OOD samples are given in test time along with the in-distribution samples. In this case, networks can make mistakes on both in-distribution and OOD samples. Experimental results show that DOCTOR can detect misclassified samples in this setting as well. (see Table 2)\n\n** Weaknesses **\n- The paper defines 2 settings for the experiments: PBB and TBB. Most of the state-of-the-art OOD detection methods (e.g. Mahalanobis) focus on fully transparent setting where they assume that intermediate layer representations of the networks are available. I understand the motivation behind using PBB and TBB that they are more realistic settings. However, I believe it is crucial to compare the method with the fully transparent setting of the existing algorithms to better position the method within the related literature.\n\n- The experiments of detecting misclassification error in presence of OOD samples are a bit limited. The results are only compared with ODIN in this setting. I would expect to see more comparisons with some of the recent and more powerful methods such as Mahalanobis, Energy-Based OOD detection [1], Generalized ODIN [2].\n\n[1] Liu et al. Energy-based Out-of-distribution Detection\n[2] Hsu et al. Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data\n\n-  I believe a general problem with the OOD detection literature is that in-distribution and OOD datasets are quite dissimilar and most of the recent OOD detection methods perform quite well on these benchmark. A more challenging problem that needs to be solved is detecting OOD samples when OOD images are quite similar to each other. There are some works focuses on such settings e.g. where CIFAR100 is in-distribution and CIFAR10 is OOD [3] which achieve AUROC around 0.85. I think it is important to perform experiments on such more challenging setting to see the limits of the OOD detection methods.\n\n[3] Zhang et al. Hybrid Models for Open Set Recognition", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Please see weaknesses section about for the limitations\n\n- I don't see any potential negative impact of the work.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "iT2FaeOmBDm", "reviewer": "Reviewer_Psjq", "summary": "The paper addresses the important problem of detecting misclassifications in a classification system. The authors consider two scenarios: total black box setting in which only the softmax predictions are available, and a partial black box setting that assumes access to the model parameters. First, the authors derive a bound to approximate the unknown classification error using a combination of known statistics. Based on this, the authors propose a simple method that can detect misclassifications. The method is shown to work well empirically on some benchmark datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First, in Section 3, the authors derive a bound to approximate the misclassification error using known model statistics. This section was a bit hard to parse, and it took me a while to fully understand. The authors could work on explaining this section better. For instance, the explanation of quantity $\\hat{Pe}(x)$ in Equation 2 can be improved. But other than this, this section was interesting to read.\n\nI took a quick glance at the proofs. They seem to be correct, although I did not review them very carefully.\n\nThe experiments were conducted well. It was interesting to see that the method consistently improved in most settings compared to softmax response and MHLNB. Overall, I like the results. In all experiments, I feel running them multiple times and reporting aggregate statistics might be beneficial.\n\nSome comments:\n\nIs it possible to combine D_\\alpha and D_\\beta into a single discriminator?\n\nHow to choose which one of D_\\alpha or D_\\beta to use?\n\nCan the method be used just for OOD detection? I see that authors reported results of misclassification detection in the presence of a fraction of OOD samples in Table 2. But I would be interested to see the performance of just the OOD detection, and compare it with some standard benchmarks.\n\nIt would have been good if authors could discuss the failure modes of the approach.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors could include a discussion on the limitations and the failure modes of the method. In addition, if there is a way to combine D_alpha and D_beta into a single method, that would be nice as well.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "QP4Ff2iqXF", "reviewer": "Reviewer_EKmb", "summary": "Motivated by the tendency of DL models to be overconfident, this paper introduces DOCTOR, a framework for determining when to accept/trust model decisions. The authors derive the tradeoffs of accepting an incorrect classification and rejecting a correct classification to motivate the formulation of DOCTOR, a set of discriminators which takes into account the full softmax values of the model to minimize the occurrence of type 1 and type 2 errors. DOCTOR is able to match or outperform prior works on both in-domain and out-of-domain problems for both totally black box and partially black box models. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I appreciate the authors concern with addressing the issue of whether or not to trust model, as this is incredibly important when applying ML to things like healthcare or finance. The theoretical analysis seems very well thought out and a good motivation for DOCTOR, and the explanations of previous works was very helpful in the experiments. However, the method section is hard to follow due to the sheer amount of notation and minimal verbal intuition; I am still unsure of exactly how the softmax is used within the DOCTOR discriminators. This could be simply because I am not as familiar with this specific line of work, but if the authors could add a bit more intuition similar to what was done in the ODIN paper it would increase the ease of understanding significantly. \n\nMy other main concern is the results; it seems like DOCTOR does not achieve significant gains over ODIN or SR, with most gains being <0.5%. Since y'all are using pertained models, is it possible that if these models were trained with a few different initializations that DOCTOR would not outperform the others? Again I think the impact of DOCTOR could be great as it is lightweight and can be applied to many different setups, but I would need more empirical evidence that it is in fact the best choice. \n\nOther nits:\n* Why is ODIN not bolded for TinyImagenet in Table 1? It seems to have the same TBB AROC and TBB FRR of D_a, D_b, and SR. \n\n\n\nUpdate: I appreciate the authors clarification on DOCTORs use of the softmax and their explanation as to why as lower FRR while maintaining the same AUROC can be quite useful in practice. Due to this and the fact that DOCTOR has significant theoretical backing and seems relatively easy to use, I have changed my score to a weak reject. Again I want to thank the authors for their thoughtful response!", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did explain that \"practitioners who lack the necessary knowledge and skills may incur misinterpretation of the results obtainable with the proposed method\", but it would be helpful to go more in depth on how exactly these results could be misinterpreted or under what situations DOCTOR is not suitable for use. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission suggest a new method for detecting out-of-distribution samples for object detection and the reviewers seem to be in agreement that the submission has sufficient novelty and potential impact for an acceptance. Given the detailed new experimental comparisons with other methods, I suggest that paper is accepted for a spotlight presentation at NeurIPS.
Decision: Accept (Spotlight)