DOCTOR: A Simple Method for Detecting Misclassification Errors

1 Introduction . With the advancement of state-of-the-art Deep Neural Networks ( DNNs ) , there has been rapid adoption of these technologies in a broad range of applications to critical systems , such as autonomous driving vehicles or industrial robots , including–but not limited to–classification and decision making tasks . Nevertheless , these solutions still exhibit unwanted behaviors as they tend to be overconfident even in presence of wrong decisions [ 17 ] . Developing methods and tools to make these algorithms ∗These authors contributed equally to this work †This paper is supported by the ERC project Hypatia under the European Unions Horizon 2020 research and innovation program . Grant agreement №835294 . ‡This project has received funding from the European Union ’ s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement №792464 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . reliable , in particular for non-specialists who may treat them as “ black boxes ” with no further checks , constitutes a core challenge . Recently , the study of safety AI methods has gained ground , and many efforts have been made in several areas [ 7 , 8 , 9 , 11 , 26 , 35 , 36 ] . In this paper , we investigate a simple method capable of detecting whether a prediction of a classifier is likely to be correct , and therefore should be trusted , or it is not , and should be rejected . Deep learning pursues the idea of learning effective representations from the data itself by training with the implicit assumption that the test data distribution should be similar to the training data distribution . However , when applied to real-world tasks , this assumption does not hold true , leading to a significant increase of misclassification errors . Although classic approaches to Out-Of-Distribution ( OOD ) detection [ 1 , 12 , 21 , 23 , 33 ] are not directly concerned with detecting misclassification errors , they are intended to prevent those errors indirectly by identifying potential drifts of the testing distribution . What the above ODD methods have in common with our work is that samples drawn from the in-distribution are more likely to be correctly classified than those from a different distribution . Indeed , the model ’ s soft-predictions for in-distribution samples tend to be generally peaky in correspondence to the correct class label while they tend to be less peaky for input samples drawn from a different distribution [ 12 ] . In general , most of these works consider white-box scenarios , where the hidden layers of the architecture are accessible or the corresponding weights are tuned during the training phase . A very effective approach to OOD detection is ODIN [ 23 ] which involves the use of temperature scaling and the addition of small perturbations to input samples . A related solution is introduced in [ 6 ] where the maximum soft-probability is called softmax response . Within this approach , the softmax response decides whether the classifier is confident enough in its prediction or not . A different approach to OOD detection is given by the use of the Mahalanobis distance [ 14 , 22 ] , which consists in calculating how much the observed out-distribution sample deviate from the indistribution ones but assuming the latter are given . 1.1 Summary of contributions . Our work tackles the problem of identifying whether the prediction of a classifier should or should not be trusted , no matter if they are made on out or in-distribution samples , and advances the state-of-the-art in multiple ways . • From the theoretical point of view , we derive the trade-off between two types of error probabilities : Type-I , that refers to the rejection of the classification for an input that would be correctly classified , and Type-II , that refers to the acceptance of the classification for an input that would not be correctly classified ( Proposition 3.1 ) . The characterization of the optimal discriminator in eq . ( 10 ) allows us to devise a feasible implementation of it , based on the softmax probability ( Proposition 3.2 ) . • From the algorithmic point of view , inspired by our theoretical analysis , we propose DOCTOR a new discriminator ( Definition 2 ) , which yields a simple and flexible framework to detect whether a decision made by a model is likely to be correct or not . We distinguish two scenarios under which DOCTOR can be deployed : Totally Black Box ( TBB ) where only the soft-predictions are available , hence gradient-propagation to perform input pre-processing is not allowed , and Partially Black Box ( PBB ) where we further allow method-specific inputs perturbations . • From the experimental point of view , we show that DOCTOR outperforms comparable stateof-the-art methods ( e.g. , ODIN [ 23 ] , softmax response [ 6 ] and Mahalanobis distance [ 22 ] ) on datasets including both in-distribution and out-of-distribution samples , and different architectures with various expressibilities , under both TBB and PBB . A key ingredient of DOCTOR is to fully exploit all available information contained in the soft-probabilities of the predictions ( not only their maximum ) . 1.2 Related works . Recent works have shown that the accuracy of a classifier and its ability to output soft-predictions that represent the true posteriors estimate can be totally disjointed [ 9 , 19 , 20 ] . Furthermore , models often tend to be overconfident about their decision even when their predictions fail [ 11 , 17 ] . This motivates a novel research area that strives to develop methods to assess when decisions made by classifiers should or should not be trusted . Although the detection of OOD samples is a different ( domain ) problem , it is naturally expected that samples from a distribution that is significantly different from the training one can not be correctly classified . In [ 23 ] , the authors propose a method which increases the peakiness of the softmax output by perturbing the input samples and applying temperature scaling [ 9 , 13 , 29 ] to the classifier logits in order to better detect in-distribution samples . It is worth noticing that this method requires additional information on the internal structure of the latent code of the model . A very different approach [ 14 , 22 ] tackles OOD detection by using the Mahalanobis distance . Although this approach appears to be more powerful , it also requires additional samples to learn the mean by class and the covariance matrix of the in-distribution . In [ 4 ] , classifiers are trained to output calibrated confidence estimates that are used to perform OOD detection . A related line of research is concerned with the problem of selective predictions ( aka reject options ) in deep neural networks . The main motivation for selective prediction is reducing the error rate by abstaining from prediction when in doubt , while keeping the number of correctly classified samples as high as possible [ 5 , 6 , 7 ] . The idea is to combine classifiers with rejection functions by observing the classifiers ’ output , without using any supervision , to decide whether to accept or to reject the classification outcome . In [ 6 ] , the authors introduce softmax response , a rejection function which compares the maximum soft-probability to a pre-determined threshold to decide whether to accept or reject the class prediction given by the model . 2 Main Definitions and Preliminaries . 2.1 Basic definitions . We start by introducing some definitions and background ; then , we describe our statistical model and some useful properties about the underlying detection problem . Let X ⊆ Rd be the ( possibly continuous ) feature space and let Y = { 1 , . . . , C } denote the concept of the label space related to some task of interest . Moreover , let pXY be the underlying ( unknown ) probability density function ( pdf ) over X × Y . Let Dn = { ( x1 , y1 ) , . . . , ( xn , yn ) } ∼ pXY be a random realization of n i.i.d . samples according to pXY denoting the training set , where xi ∈ X is the input ( feature ) , yi ∈ Y is the output class among C possible classes and n denotes the size of the training set . A predictor fDn : X → Y uses the inferred model PŶ |X ≡ PŶ |X ( y|x ; Dn ) based on the training set , fDn ( x ) ≡ fn ( x ; Dn ) = ∆ arg max y∈Y PŶ |X ( y|x ; Dn ) , and tries to approximate the optimal ( Bayes ) decision rule f ? ( x ) =∆ arg max y∈Y PY |X ( y|x ) . Notice that PŶ |X can be interpreted as the prediction of the class ( label ) posterior probability given a sample ( e.g. , PŶ |X ( y |x ) ≡ softmax ( x ) y ) , while PY |X is the true ( unknown ) probability . In several practical scenarios PŶ |X does not perfectly match PY |X and still fDn ≈ f ? ( cf . [ 9 ] ) . 2.2 Error variable . Let E ( x ) =∆ 1 [ Y 6= fDn ( x ) ] denote the error variable for a given x ∈ X corresponding to fDn , i.e. , where we denote with 1 [ E ] the indicator vector which outputs 1 if the event E is true and 0 otherwise . Similarly , we can define the self-error variable Ê ( x ) =∆ 1 [ Ŷ 6= fDn ( x ) ] also corresponding to the inferred predictor fDn but based on the prediction model PŶ |X of the class posterior probability . Notice that Ê ( x ) is observable since the underlying distribution is known . However , E ( x ) can not be observed and in general these binary variables do not coincide . At this stage , it is convenient to introduce the notions of probability of classification error for a given x ∈ X w.r.t . both the true class posterior and the predicted probabilities : Pe ( x ) =∆ E [ E ( x ) |x ] = 1− PY |X ( fDn ( x ) |x ) , ( 1 ) P̂e ( x ) =∆ E [ Ê ( x ) |x ] = 1− PŶ |X ( fDn ( x ) |x ) . ( 2 ) Notice that P̂e ( x ) represents the probability of misclassification of the sample x with respect to the softmax probability PŶ |X , which can be interpreted as the model ’ s approximation of nature PY |X . Such approximation is close when the model is well-calibrated . Obviously , Pe ? ( x ) ≤ Pe ( x ) for all x ∈ X , where Pe ? ( x ) corresponds to the minimum error of the Bayes classifier : Pe ? ( x ) = 1− PY |X ( f ? ( x ) |x ) . It is worth mentioning that , by averaging ( 1 ) over the data distribution , we obtain the error rate of the classifier fDn . Although P̂e ( x ) provides a valuable candidate to infer the unknown error variable E ( x ) , it is easy to check that max { Pe ( x ) , P̂e ( x ) } − Pr ( Ŷ = Y |x ) ≤ Pr { Ê ( x ) 6= E ( x ) |x } ≤ Pr ( Ŷ 6= Y |x ) , ( 3 ) which in particular implies that the error incurred in using Ê ( x ) to predict E ( x ) is lower bounded by the classification error per sample ( 1 ) . The proofs are in Supplementary material ( Appendix A.3 ) . In this paper , we aim at identifying a discriminator capable of distinguishing between inputs x for which we can trust the predictions of the classifier fDn ( x ) ( i.e. , E ( x ) = 0 ) and those for which we should not trust predictions ( i.e. , E ( x ) = 1 ) . In the next section , we will show that the function Pe ( x ) : X 7→ [ 0 , 1 ] plays a central role in the characterization of the optimal discriminator . However , Pe ( x ) is not available in practical scenarios and the direct estimation ( e.g. , based on pairs of inputs and labels ) of the true class posterior probability PY |X can not be performed . Notice that it is not possible to sample the conditional pdf PY |X for each input x ∈ X . As a matter of fact , it is well-known that the application of direct methods for this estimation will lead to ill-posed problems , as shown in [ 32 ] .

Summary: The paper addresses the important problem of detecting misclassifications in a classification system. The authors consider two scenarios: total black box setting in which only the softmax predictions are available, and a partial black box setting that assumes access to the model parameters. First, the authors derive a bound to approximate the unknown classification error using a combination of known statistics. Based on this, the authors propose a simple method that can detect misclassifications. The method is shown to work well empirically on some benchmark datasets.