Title: A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
Authors: Dan Hendrycks
Kevin Gimpel
Venue: ICLR 2017
Keywords: Computer vision

Abstract: We consider the two related problems of detecting if an example is misclassified or out-of-distribution. We present a simple baseline that utilizes probabilities from softmax distributions. Correctly classified examples tend to have greater maximum softmax probabilities than erroneously classified and out-of-distribution examples, allowing for their detection. We assess performance by defining several tasks in computer vision, natural language processing, and automatic speech recognition, showing the effectiveness of this baseline across all. We then show the baseline can sometimes be surpassed, demonstrating the room for future research on these underexplored detection tasks.

TL;DR: Methods to Detect When a Network Is Wrong

Review discussion:
It would be interesting to also include a generative baseline for out-of-domain classification.
For example, on the TIMIT dataset, one could train a generative GMM-HMM system where phone durations are modeled using a Hidden-Markov-Model and output probabilities are modeled using Gaussian Mixture Models. One could then estimate a in/out domain classification score by computing the data-likelihood on test examples. Have the authors also considered other/similar generative model baselines in addition to using the softmax probabilities directly?
Thank you for the question. Using generative models as a baseline is a great idea, but we never pursued this avenue since we wanted a baseline simple to compute from prominent model designs from several fields. The intractability of estimating the normalizing sum for the probabilities [1] restricts the usable deep generative models to approaches like Deep Boltzmann Machines for some vision and NLP tasks. ASR seems exceptional because generative models like HMM-GMMs perform competitively, and our impression is that generative solutions are overall not as prominent in other fields due to less accuracy, simplicity, or fragility [2].
Consequently, we sought a baseline which works for discriminative classifiers and can be easily computed even as architectures evolve.
However, if a company uses a deep system, they could still use the probabilities from a DBM or a HMM-GMM as auxiliary information for error detection, and that seems like an interesting avenue.
[1] This paper describes the issue and tries to partially alleviate the problem: Taesup Kim, Yoshua Bengio. Deep Directed Generative Models with Energy-Based Probability Estimation. https://arxiv.org/pdf/1606.03439.pdf
[2] Risks of Semi-Supervised Learning: How Unlabeled Data Can Degrade Performance of Generative Classifiers http://sites.poli.usp.br/p/fabio.cozman/Publications/Chapter/cozman-cohen-ssl2006.pdf
Did you try different network structures / settings?
Thank you for the question. We found that higher-accuracy structures improved AUROCs and AUPRs.
For example, we switched from a 40-2 WideResNet to a 40-4 WideResNet for CIFAR-100 and the testing error went from 25.1% to 20.7%.
At the same time, the CIFAR-100/SUN AUROC went from 80 to 91, and the CIFAR-100/Gaussian AUROC went from 84 to 88.
However, for most experiments we simply used standard architecture settings and default optimizer hyperparameters.
The authors present results on a number of different tasks where the goal is to determine whether a given test example is out-of-domain or likely to be mis-classified. This is accomplished by examining statistics for the softmax probability for the most likely class;
although the score by itself is not a particularly good measure of confidence, the statistics for out-of-domain examples are different enough from in-domain examples to allow these to be identified with some certainty.
My comments appear below:
1. As the authors point out, the AUROC/AUPR criterion is threshold independent.
As a result, it is not obvious whether the thresholds that would correspond to a certain operating point (say a true positive rate of 10%) would be similar across different data sets.
In other words, it would be interesting to know how sensitive the thresholds are to different test sets (or different splits of the test set). This is important if we want to use the thresholds determined on a given held-out set during evaluation on unseen data (where we would need to select a threshold).
2. Performance is reported in terms of AUROC/AUPR and models are compared against a random baseline. I think it’s a little hard to look at the differences in AUC/AUPR to get a sense for how much better the proposed classifier is than the random baseline. It would be useful, for example, if the authors could also report how strongly statistically significant some of these differences are (although admittedly they look to be pretty large in most cases).
3. In the experiments on speech recognition presented in Section 3.3, I was not entirely clear on how the model was evaluated. In Table 9, for example, is an “example” the entire utterance or just a single (stacked?) speech frame.
Assuming that each “example” is an utterance, are the softmax probabilities the probability of the entire phone sequence (obtained by multiplying the local probability estimates from a Viterbi decoding?)
4. I’m curious about the decision to ignore the blank symbol’s logit in Section 3.3. Why is this required?
5. As I mentioned in the pre-review question, at least in the speech recognition case, it would have been interesting to compare performance obtained using a simple generative baseline (e.g., GMM-HMM). I think that would serve as a good indication of the ability of the proposed model to detect out-of-domain examples over the baseline.
Thank you for your analysis of our paper.
1. A common pattern is that the clean examples have a high mean and small standard deviation, and erroneous and out-of-distribution examples have a low mean with high standard deviation.
For example, for a previous CIFAR-10 model the prediction probability of correctly classified examples is 0.98 with a standard deviation of 0.066, and the erroneously classified examples have a mean of 0.784 with standard deviation 0.19, and the SUN images have a mean prediction confidence of 0.786 with a standard deviation of 0.19. We will consider adding new out-of-distribution data to test a threshold on for the final paper.
2. We made a note about statistical significance in the updated paper thank to this comment. The null hypothesis that AUROC = 0.5 is tested by the Wilcoxoon rank-sum test, and we reject the null with high statistical significance (p < 1e-3) for all AUROCs computed.
For example, the smallest test set has 500 examples, and the smallest AUROC is 0.61, so a loose upper bound on the p-values is 1 - Phi(500*500(0.61-0.5)/sqrt(500*500(500+500+1)/12)) ~ 1E-9, where Phi is the CDF of the standard normal. Error detection on CIFAR-10, for example, has a p-value which is approximately 1 - Phi(34) ~ 1E-253. We could not find a statistical test for AUPRs, however.
3. The bidirectional LSTM has access to the entire sequence of MFCCs. Each probability used in detection is from the softmax computed at each timestep of the sequence. No decoding is necessary for obtaining a probability useful for detection.
4.
When performing detection, we compute the softmax probabilities while ignoring the blank symbol's logit.
However, in training we leave the blank symbol alone. With the blank symbol's presence, the softmax distributions at most time steps predict a blank symbol with high confidence, but without the blank symbol we can better differentiate between normal and abnormal distributions. We now added this elaboration to the paper.
5. We trained a GMM/HMM with HTK and used the average log likelihood per frame for detection outputted from HVite. We compared the AUROC values of this generative model with the maximum softmax probabilities of the bidirectional LSTM in the paper. The results are as follows.
TIMIT/TIMIT+Airport
GMM/HMM Log Likelihood AUROC: 88
LSTM w/ Softmax Prob AUROC:   99
TIMIT/TIMIT+Babble
GMM/HMM Log Likelihood AUROC: 75
LSTM w/ Softmax Prob AUROC:   100
TIMIT/TIMIT+Car
GMM/HMM Log Likelihood AUROC: 92
LSTM w/ Softmax Prob AUROC:   98
TIMIT/TIMIT+Exhibition
GMM/HMM Log Likelihood AUROC: 99
LSTM w/ Softmax Prob AUROC:   100
TIMIT/TIMIT+Restaurant
GMM/HMM Log Likelihood AUROC: 81
LSTM w/ Softmax Prob AUROC:   98
TIMIT/TIMIT+Street
GMM/HMM Log Likelihood AUROC: 95
LSTM w/ Softmax Prob AUROC:   100
TIMIT/TIMIT+Subway
GMM/HMM Log Likelihood AUROC: 96
LSTM w/ Softmax Prob AUROC:   100
TIMIT/TIMIT+Train
GMM/HMM Log Likelihood AUROC: 97
LSTM w/ Softmax Prob AUROC:   100
The largest AUROC improvement of the softmax probability over the GMM/HMM is 25%.
The authors propose the use of statistics of softmax outputs to estimate the probability of error and probability of a test sample being out-of-domain. They contrast the performance of the proposed method with directly using the softmax output probabilities, and not their statistics, as a measure of confidence.
It would be great if the authors elaborate on the idea of ignoring the logit of the blank symbol.
It would be interesting to see the performance of the proposed methods in more confusable settings, ie., in cases where the out-of-domain examples are more similar to the in-domain examples. e.g., in the case of speech recognition this might correspond to using a different language's speech with an ASR system trained in a particular language. Here the acoustic characteristics of the speech signals from two different languages might be more similar as compared to the noisy and clean speech signals from the same language.
In section 4, the description of the auxiliary decoder setup might benefit from more detail.
There has been recent work on performance monitoring and accuracy prediction in the area of speech recognition, some of this work is listed below.
1. Ogawa, Tetsuji, et al. "Delta-M measure for accuracy prediction and its application to multi-stream based unsupervised adaptation." Proceedings of ICASSP. 2015.
2. Hermansky, Hynek, et al. "Towards machines that know when they do not know." Proceedings of ICASSP, 2015.
3. Variani, Ehsan et al. "Multi-stream recognition of noisy speech with performance monitoring." INTERSPEECH. 2013.
Thank you for your analysis of our paper.
When performing detection, we compute the softmax probabilities while ignoring the blank symbol's logit.
However, in training we leave the blank symbol alone. With the blank symbol's presence, the softmax distributions at most time steps predict a blank symbol with high confidence, but without the blank symbol we can better differentiate between normal and abnormal distributions. We now added this elaboration to the paper.
We tested the model in confusable settings for vision and NLP (CIFAR-10 and SUN, MNIST and Omniglot, IMDB and Movie Reviews, WSJ and Webblog). Your comment made us realize we should add a test for speech. We used Chinese speech and found that we could still detect the speech reliably (but not as easily), and the abnormality module still generalized to detecting this speech better than softmax probabilities alone. These results are in the updated paper.
We updated the paper to provide more detail of the abnormality module per your suggestion.
Finally, thank you for the links. We can differentiate between our work and their work if you want.
The paper address the problem of detecting if an example is misclassified or out-of-distribution. This is an very important topic and the study provides a good baseline.
Although it misses strong novel methods for the task, the study contributes to the community.
Thank you!
We have updated the paper to make mention of the statistical significance of the AUROCs. We also added Chinese speech from THCHS-30 as out-of-distribution examples.
Finally, we added more information about the logit of the blank symbol, the training of the abnormality module, and we revised Appendix B.

Meta-review:
role: PC
data: comment: The paper presents an approach that uses the statistics of softmax outputs to identify misclassifications and/or outliers. The reviewers had mostly minor comments on the paper, which appear to have been appropriately addressed in the revised version of the paper.

Decision: Accept (Poster)