Detecting Misclassification Errors in Neural Networks with a Gaussian Process Model

1 INTRODUCTION . Classifiers based on Neural Networks ( NNs ) are widely deployed in many real-world applications ( LeCun et al. , 2015 ; Anjos et al. , 2015 ; Alghoul et al. , 2018 ; Shahid et al. , 2019 ) . Although good prediction accuracies are achieved , lack of safety guarantees becomes a severe issue when NNs are applied to safety-critical domains , e.g. , healthcare ( Selişteanu et al. , 2018 ; Gupta et al. , 2007 ; Shahid et al. , 2019 ) , finance ( Dixon et al. , 2017 ) , self-driving ( Janai et al. , 2017 ; Hecker et al. , 2018 ) , etc . One way to estimate trustworthiness of a classifier prediction is to use its inherent confidence-related score , e.g. , the maximum class probability ( Hendrycks & Gimpel , 2017 ) , entropy of the softmax outputs ( Williams & Renals , 1997 ) , or difference between the highest and second highest activation outputs ( Monteith & Martinez , 2010 ) . However , these scores are unreliable and may even be misleading : high-confidence but erroneous predictions are frequently observed ( Provost et al. , 1998 ; Guo et al. , 2017 ; Nguyen et al. , 2015 ; Goodfellow et al. , 2014 ; Amodei et al. , 2016 ) . In a practical setting , it is beneficial to have a detector that can raise a red flag whenever the predictions are suspicious . A human observer can then evaluate such predictions , making the classification system safer . In order to construct such a detector , quantitative metrics for measuring predictive reliability under different circumstances are first developed , and a warning threshold then be set based on users ’ preferred precision-recall tradeoff . Existing such methods can be categorized into three types based on their focus : error detection , which aims to detect the natural misclassifications made by the classifier ( Hendrycks & Gimpel , 2017 ; Jiang et al. , 2018 ; Corbière et al. , 2019 ) ; out-of-distribution ( OOD ) detection , which reports samples that are from different distributions compared to training data ( Liang et al. , 2018 ; Lee et al. , 2018a ; Devries & Taylor , 2018 ) ; and adversarial sample detection , which filters out samples from adversarial attacks ( Lee et al. , 2018b ; Wang et al. , 2019 ; Aigrain & Detyniecki , 2019 ) . Among these categories , error detection , also called misclassification detection ( Jiang et al. , 2018 ) or failure prediction ( Corbière et al. , 2019 ) , is the most challenging ( Aigrain & Detyniecki , 2019 ) and underexplored . For instance , Hendrycks & Gimpel ( 2017 ) defined a baseline based on maximum class probability after softmax layer . Although the baseline performs reasonably well in most testing cases , reduced efficacy in some scenaria indicates room for improvement ( Hendrycks & Gimpel , 2017 ) . Jiang et al . ( 2018 ) proposed Trust Score , which measures the similarity between the original classifier and a modified nearest-neighbor classifier . The main limitation of this method is scalability : the Trust Score may provide no or negative improvement over the baseline for high-dimensional data . ConfidNet ( Corbière et al. , 2019 ) builds a separate NN model to learn the true class probablity , i.e . softmax probability for the ground-truth class . However , ConfidNet itself is a standard NN , so its confidence scores may be unreliable or misleading : A random input may generate a random confidence score , and ConfidNet do not provide any information regarding uncertainty of these confidence scores . Moreover , none of these methods can differentiate natural classifier errors from risks caused by OOD or adversarial samples ; if a detector could do that , it would be easier for practitioners to fix the problem , e.g. , by retraining the original classifier or applying better preprocessing techniques to filter out OOD or adversarial data . To meet these challenges , a new framework is developed in this paper for error detection in NN classifiers . The main idea is to utilize RIO ( Residual , Input , Output ; Qiu et al. , 2020 ) , a regression model based on Gaussian Processes , on top of the original NN classifier . Whereas the original RIO is limited to regression problems , the proposed approach extends it to misclassification detection by modifying its components . The new framework not only produces a calibrated confidence score based on the original maximum class probability , but also provides a quantitative uncertainty estimation of that score . Errors can therefore be detected more reliably . Note that the proposed method does not change the prediction accuracy of the original classification model . Instead , it provides a quantitative metric that makes it possible to detect misclassification errors . This framework , referred to as RED ( RIO for Error Detection ) , is compared empirically to existing approaches on 125 UCI datasets and on a large-scale deep learning architecture . The results demonstrate that the approach is effective and robust . A further case study with OOD and adversarial samples shows the potential of using RED to diagnose the sources of mistakes as well , thereby leading to a possible comprehensive approach for improving trustworthiness of neural network classifiers in the future . 2 RELATED WORK . In the past two decades , a large volume of work was devoted to calibrating the confidence scores returned by classifiers . Early works include Platt Scaling ( Platt , 1999 ; Niculescu-Mizil & Caruana , 2005 ) , histogram binning ( Zadrozny & Elkan , 2001 ) , isotonic regression ( Zadrozny & Elkan , 2002 ) , with recent extensions like Temperature Scaling ( Guo et al. , 2017 ) , Dirichlet calibration ( Kull et al. , 2019 ) , and distance-based learning from errors ( Xing et al. , 2020 ) . These methods focus on reducing the difference between reported class probability and true accuracy , and generally the rankings of samples are preserved after calibration . As a result , the separability between correct and incorrect predictions is not improved . In contrast , RED aims at deriving a score that can differentiate incorrect predictions from correct ones better . A related direction of work is the development of classifiers with rejection/abstention option . These approaches either introduce new training pipelines/loss functions ( Bartlett & Wegkamp , 2008 ; Yuan & Wegkamp , 2010 ; Cortes et al. , 2016 ) , or define mechanisms for learning rejection thresholds under certain risk levels ( Dubuisson & Masson , 1993 ; Santos-Pereira & Pires , 2005 ; Chow , 2006 ; Geifman & El-Yaniv , 2017 ) . In contrast to these methods , RED assumes an existing pretrained NN classifier , and provides an additional metric for detecting potential errors made by this classifier , without specifying a rejection threshold . Designing metrics for detecting potential risks in NN classifiers has also become popular recently . While most approaches focus on detecting OOD ( Liang et al. , 2018 ; Lee et al. , 2018a ; Devries & Taylor , 2018 ) or adversarial examples ( Lee et al. , 2018b ; Wang et al. , 2019 ; Aigrain & Detyniecki , 2019 ) , work on detecting natural errors , i.e. , regular misclassifications not caused by external sources , is more limited . Ortega ( 1995 ) and Koppel & Engelson ( 1996 ) conducted early work in predicting whether a classifier is going to make mistakes , and Seewald & Fürnkranz ( 2001 ) built a meta-grading classifier based on similar ideas . However , these early works did not consider NN classifiers . More recently , Hendrycks & Gimpel ( 2017 ) and Haldimann et al . ( 2019 ) demonstrated that raw maximum class probability is an effective baseline in error detection , although its performance was reduced in some scenaria . More elaborate techniques for error detection have also been developed recently . Mandelbaum & Weinshall ( 2017 ) proposed a confidence score based on the data embedding derived from the penul- timate layer of a NN . However , their approach requires modifying the training procedure in order to achieve effective embeddings . Jiang et al . ( 2018 ) introduced Trust Score to measure the similarity between a base classifier and a modified nearest-neighbor classifier . Trust Score outperforms the maximum class probability baseline in many cases , but negative improvement over baseline can be observed in high-dimensional problems , implying poor scalability of local distance computations . ConfidNet ( Corbière et al. , 2019 ) learns to predict the class probability of true class with another NN , while Introspection-Net ( Aigrain & Detyniecki , 2019 ) utilizes the logit activations of the original NN classifier to predict its correctness . Since both models themselves are standard NNs , the confidence scores returned by them may be arbitrarily high without any uncertainty information . Moreover , existing approaches for error detection can not differentiate natural misclassification error from OOD or adversarial samples , making it difficult to diagnose the sources of risks . In contrast , RED explicitly reports its uncertainty about the estimated confidence score , providing more reliable error detection . The uncertainty information returned by RED may also be helpful in clarifying the cause of classifier mistakes , as will be demonstrated in this paper . 3 METHODOLOGY . This section gives the general problem statement , introduces the basic idea of original RIO , on which RED is built , and describes the technical details of RED . 3.1 PROBLEM STATEMENT . Consider a training dataset D = ( X , y ) = { ( xi , yi ) } Ni=1 , and a pretrained NN classifier that outputs a predicted label ŷi and class probabilities for each class σi = [ p̂i,1 , p̂i,2 , . . . , p̂i , K ] given xi , where N is the total number of training points and K is the total number of classes . The problem is to develop a metric that can serve as a quantitative indicator for detecting natural misclassification errors made by the pretrained NN classifier . 3.2 RIO . The original RIO ( Qiu et al. , 2020 ) was developed to quantify point-prediction uncertainty in regression models . More specifically , RIO fits a Gaussian Process ( GP ) to predict the residuals , i.e . the differences between ground-truth and original model predictions . It utilizes an I/O kernel , i.e . a composite of an input kernel and an output kernel , thus taking into account both inputs and outputs of the original regression model . As a result , it measures the covariances between data points in both the original feature space and the original model output space . For each new data point , a trained RIO model takes the original input and output of the base regression model , and predicts a distribution of the residual , which can be added back to the original model prediction to obtain both a calibrated prediction and the corresponding predictive uncertainty . In the original RIO work , SVGP ( Hensman et al. , 2013 ; 2015 ) was used as an approximate GP to improve the scalability of the approach . Both empirical results and theoretical analysis showed that RIO is able to consistently improve the prediction accuracy of base model as well as provide reliable uncertainty estimation . Moreover , RIO can be directly applied on top of any pre-trained models without retraining or modification . It therefore forms a promising foundation for improving reliability of error detection metrics as well .

Summary: This paper introduces RED, a new methodology to produce reliable confidence scores to detect missclassification errors in neural networks. The idea is to combine kernels based on both input and output spaces (as in RIO) to define a (sparse) GP that estimates the residual between the correctness of the original prediction and the maximum class probability. The authors show enhanced performance against other related methods and the ability of RED to detect OOD and adversarial data through the variance of the confidence score.