Detecting Misclassification Errors in Neural Networks with a Gaussian Process Model
Authors: Xin Qiu
Risto Miikkulainen
Venue:  (2021)
Keywords: Neural Network Classifier
Error Detection
AI safety

Abstract: As neural network classifiers are deployed in real-world applications, it is crucial that their predictions are not just accurate, but trustworthy as well. One practical solution is to assign confidence scores to each prediction, then filter out low-confidence predictions. However, existing confidence metrics are not yet sufficiently reliable for this role. This paper presents a new framework that produces more reliable confidence scores for detecting misclassification errors. This framework, RED, calibrates the classifier's inherent confidence indicators and estimates uncertainty of the calibrated confidence scores using Gaussian Processes. Empirical comparisons with other confidence estimation methods on 125 UCI datasets demonstrate that this approach is effective. An experiment on a vision task with a large deep learning architecture further confirms that the method can scale up, and a case study involving out-of-distribution and adversarial samples shows potential of the proposed method to improve robustness of neural network classifiers more broadly in the future.

Under review as a conference paper at ICLR 2021
DETECTING MISCLASSIFICATION ERRORS IN NEURAL NETWORKS WITH A GAUSSIAN PROCESS MODEL
Anonymous authors Paper under double-blind review
ABSTRACT
As neural network classifiers are deployed in real-world applications, it is crucial that their predictions are not just accurate, but trustworthy as well. One practi- cal solution is assign confidence scores each prediction, then filter out low- confidence predictions. However, existing confidence metrics are not yet suffi- ciently reliable for this role. This paper presents a new framework that produces more reliable confidence scores for detecting misclassification errors. This frame- work, RED, calibrates the classifier's inherent confidence indicators and estimates uncertainty of the calibrated confidence scores using Gaussian Processes. Empir- ical comparisons with other confidence estimation methods on 125 UCI datasets demonstrate that this approach is efféctive. An experiment large deep learning architecture further confirms that the method can scale up, and case study involving out-of-distributie and adversarial samples shows potential of the proposed method to improve robustness of neural network classifiers more broadly in the future.
vision task with a
INTRODUCTION
Classifiers based on Neural Networks (NNs) are widely deployed in many real-world applications
(LeCun et al. 2013 Anjos al., 2015 Alghoul al. 2018 Shahid et 2019)
Although good
prediction accuracies are achieved, lack of safety guarantees becomes a severe issue when NNs are
applied to safety-criticall domains, e.g., healthcare (Selişteanu et al. 20188
Gupta et al. 2007] Shahid
et al. 2019), finance (Dixon et al. 2017), self-driving (Janai et al. 2017 Hecker et al. 2018). etc.
One way to estimate trustworthiness of classifier prediction is to use its inherent confidence-related score, e.g., the maximum class probability (Hendrycks & Gimpel| (017). entropy of the softmax outputs (Willamms & Renals 1997), or difference between the highest and second highest activation outputs (Monteith & Martinez 2010). However, these scores are unreliable and may even be mis- leading: high-confidence but erroneous predictions are frequently observed (Provost et al. 1998) Guo et al.] [017 Nguyen et al. 2015) Goodfellow et al.) [014 Amodei et al. 20166. In practi- cal setting, it is beneficial to have a detector that can raise a red flag whenever the predictions are suspicious. A human observer can then evaluate such predictions, making the classification system
safer.
In order to construct such detector, quantitative metrics for measuring predictive reliability under different circumstances are first developed, and a warning threshold then be set based on users' preferred precision-recall tradeoff. Existing such methods can be categorized into three types based on their focus: eтror detection, which aims to detect the natural misclassifications made by the classifier (Hendrycks & Gimpell [017 Jiang et al. 2018 Corbière al. 2019): outto--distribution (OOD) detection, which reports samples that are from different distributions compared to training data (Liang et al. [2088 [ee et al. [ Devrres Taylor 2018); and adversarial sample detection,
which filters out samples from adversarial attacks (Lee et al. 2018b Wang et al. 2019 Aigrain & Detyniecki 2019).
Among these categories, error detection, also called misclassification detection (Jiang et al. 2018 or failure prediction (Corbière et al. 2019). 9). is the most challenging (Aigrain & Detyniecki 2019) and underexplored. For instance, Hendryykk & Gimpel (2017) defined baseline based on maximum class probability after softmax layer. Although the baseline performs reasonably well in most test- ing cases, reduced efficacy in some scenaria indicates room for improvement (Hendryeks & Gimpel

---

Under review as a conference paper at ICLR 2021
2017). Jiang al. (2018) proposed Trust Score, which measures the similarity between the original classifier and a modified nearest- neighbor classifier. The main limitation of this method is scalabil- ity: the Trust Score may provide or negative improvement over the baseline for high- dighddimensional data. ConfidNet (Corbière et al. 2019) builds a separate NN model to learn the true class proba- blity, i.e. softmax probability for the ground-truth class. However, ConfidNet itself is standard NN, so its confidence scores may be unreliable or misleading: A random input may generate ran- dom confidence score, and ConfidNet do not provide any information regarding uncertainty of these confidence scores. Moreover, none of these methods can differentiate natural classifier errors from risks caused by OOD or adversarial samples; if detector could do that, would be easier for prac- titioners to fix the problem, e.g., by retraining the original classifier or applying better preprocessing techniques to filter out OOD or adversarial data.
To meet these challenges, new framework is developed in this paper for error detection in NN classifiers. The main idea is to utilize RIO (Residual, Input, Output: Qiu et al. 2020). regression model based on Gaussian Processes, on top of the original NN classifier. Whereas the original RIO is limited to regression problems, the proposed approach extends to misclassification detection by modifying its components. The new framework not only produces calibrated confidence score based on the original maximum class probability, but also provides a quantitative uncertainty esti- mation of that score. Errors can therefore be detected more reliably. Note that the proposed method does not change the prediction accuracy of the original classification model. Instead, it provides a quantitative metric that makes possible detect misclassification errors. This framework, referred to as RED (RIO for Error Detection), is compared empirically to existing approaches on 125 UCI datasets and on large-scale deep learning architecture. The results demonstrate that the approach is effective and robust. A further case study with OOD and adversarial samples shows the potential of using RED to diagnose the sources of mistakes as well, thereby leading to a possible comprehensive approach for improving trustworthiness of neural network classifiers in the future.
2 RELATED WORK
In the past two decades, a large volume of work was devoted to calibrating the confidence scores returned by classifiers. Early works include Platt Scaling (Plati 1999 Niculescu- Mizii & Caruana
2005), histogram binning (Zadrozny & Elkan 2001). isotonic regression (Zadrozny & Elkan 2002).
with recent extensions like Temperature Scaling (Guo al. (017). Dirichlet calibration (Kull al.]
2019), and distance-based learning from errors (Xing et al. 2020). These methods focus on reducing the difference between reported class probability and true accuracy, and generally the rankings of samples are preserved after calibration. As result, the separability between correct and incorrect predictions is not improved. In contrast, RED aims at deriving a score that can differentiate incorrect predictions from correct ones better.
A related direction of work is the development of classifiers with rejection/abstention option. These approaches either introduce new training pipelines/loss functions (Bartlett & Wegkamp/ (2088] Yuan & Wegkamp, 2010; Cortes et al., 2016), or define mechanisms for learning rejection thresholds under certain risk levels (Dubuisson & Masson, 1993, Santos-Pereira & Pires] 2005) [how 2006
Geifmann & EI- Yaniv 2017). In contrast to these methods, RED assumes an existing pretrained NN classifier, and provides an additional metric for detecting potential errors made by this classifier, without specifying a rejection threshold.
Designing metrics for detecting potential risks in NN classifiers has also become popular recently.
While most approaches focus on detecting OOD (Liang et al. 2018 Lee et al. 2018a Devries
& Taylor 2018) or adversarial examples (Lee et al., [0188]] Wang et al.) 2019 Aigrain & De-
[yniecki] 2019), work on detecting natural errors, i.e., regular misclassifications not caused by exter- nal sources, is more limited. [Otteaa (1995) and KKoppe & Engelson (1996) conducted early work in predicting whether classifier is going to make mistakes, and (Seewald & Fürnkranz) (2001) built meta-grading classifier based on similar ideas. However, these early works did not consider NN clas- sifiers. More recently. Hendryckks& & Gimpel| (2017) and Haldimann et al.) (2019) demonstrated that raw maximum class probability is an effective baseline in error detection, although its performance was reduced in some scenaria.
More elaborate techniques for error detection have also been developed recently. Mandelbaum & Weinshall (2077) proposed confidence score based on the data embedding derived from the penul-
2

---

Under review as a conference paper at ICLR 2021
timate layer of NN. However, their approach requires modifying the training procedure in order to achieve effective embeddings. Jiang et al. (2018) introduced Trust Score to measure the similarity between a base classifier and modified nearest-neighhor classifier. Trust Score outperforms the maximum class probability baseline in many cases, but negative improvement over baseline can be observed in high-dimensional problems, implying poor scalability of local distance computations. ConfidNet (Corbière et al., 2019) learns to predict the class probability of true class with another NN, while Introspection-Net (Aigrain & Detyniecki, (019) utilizes the logit activations of the orig- inal NN classifier to predict its correctness. Since both models themselves are standard NNs, the confidence scores returned by them may be arbitrarily high without any uncertainty information. Moreover, existing approaches for error detection cannot differentiate natural misclassificatios error from OOD or adversarial samples, making difficult to diagnose the sources of risks. In contrast, RED explicitly reports its uncertainty about the estimated confidence score, providing more reliable error detection. The uncertainty information returned by RED may also be helpful in clarifying the cause of classifier mistakes, will be demonstrated in this paper.
3 METHODOL.OGY
This section gives the general problem statement, introduces the basic idea of original RIO, on which RED is built, and describes the technical details of RED.
3.1 PROBLEM STATEMENT
Consider training dataset D y) = ((x,, D:) and pretrained NN classifier that outputs a predicted label y and class probabilities for each class o; [ . given x;, where N is the total number of training points and K is the total number of classes. The problem is to develop metric that can serve as a quantitative indicator for detecting natural misclassification errors made by the pretrained NN classifier.
3.2 RIO
The original RIO (Qiu et al., 2020) was developed to quantify point-prediction uncertainty in re- gression models. More specifically, RIO fits Gaussian Process (GP) to predict the residuals, i.c. the differences between ground-truth and original model predictions. It utilizes an I/O kernel, i.c. composite of an input kernel and output kernel, thus taking into account both inputs and out- puts of the original regression model. As a result, it measures the covariances between data points in both the original feature space and the original model output space. For each new data point, trained RIO model takes the original input and output of the base regression model, and predicts a distribution of the residual, which can be added back to the original model prediction to obtain both a calibrated prediction and the corresponding predictive uncertainty.
In the original RIO work, SVGP (Hensman et al. 2013 2015) was used as an approximate GP to improve the scalability of the approach. Both empirical results and theoretical analysis showed that RIO is able to consistently improve the prediction accuracy of base model as well as provide reliable uncertainty estimation. Moreover, RIO can be directly applied on top of any pre-trained models without retraining or modification. It therefore forms a promising foundation for improving reliability of error detection metrics well.
3.3 RIO FOR ERROR DETECTION (RED)
Although RIO performs robustly in wide variety of regression problems, cannot be directly applied to classification models. A new framework, namely RED, is proposed to utilize RIO for error detection in classification domains.
Building on the fact that the original maximum class probability is strong baseline for error detec- tion (Hendrycks & Gimpel, 2017; Haldimann et al. 2019). y the main idea of RED is to derive a more reliable confidence score by stacking RIO top of the original maximum class probability. Since RIO was designed for single-output regression problems, it contains an output kernel only for scalar outputs. In RED, this original output kernel is extended to multiple outputs, i.e. vector outputs
3

---

Under review as a conference paper at ICLR 2021
softmax
1p1.22
RIO
var (f)
NN Classifier
max (o)
argmax(())
Error
Uncertaintty
Datection
The RED Training and Deployment Processes. The solid pathways are active in both training and deployment phase, while the dashed pathways are active only the training phase. During the training phase, target confidence score is assigned to each training sample according to whether it correctly predicted by the original NN classifier or not. A RIO model is then trained predict the residual between the target confidence score and the original maximum class proba- bility é. The I/O kernel in RIO utilizes both the raw feature x and softmax outputs a to predict the residuals. In the deployment phase, given sian distribution of estimated residual i defined by the mean f and variance var(f). Addition of and forms a calibrated confidence score for error detection, and var(f) indicates the corresponding uncertainty.
Figure 1:
data point, the trained RIO model provides Gaus-
Algorithm RED Training and Deployment Procedures
Require:
y) = (x,, Bi) training data
ÿ = { labels predicted by original NN classifier on training data
o (o, = ................................ Pr. 1 softmax outputs original NN classifier on training data
{é= = max(o;)) maximum class probability returned by original NN classifier on
training data
x.: data to be predicted
σ: softmax outputs of original NN classifier
2=: maximum class probability returned by original NN classifier on x.
Ensure:
E ~ ~N. + F., var(f-)): a + F_ can be used as confidence score for error detection, and var(f.) represents the uncertainty of returned confidence score
Training Phase:
obtain target confidence score
{c; (oy..g. = i = ú.. otherwise dg..0r = 0)
8y.88 1N where 6 the Kronecker delta
2: calculate residuals r = {r: C 2..
3: for each optimizer step do
4: calculate covariance matrix Ko((K' ([,(),)),(XX)))
where each entry is given by ko((x,, oi), (x, = Aim (x,, x,) + houut (o,, for i,j = 1..j=1,2,... 1,2,...,
5: optimize GP hyperparameters by maximizing log marginal likelihood logp(r|{,,o)
-3rr" (K.(KK (X, ) + I -r--- log K. ((X, (X, + 0 - log
Deployment Phase:
6: calculate residual mean F. = K, (K. ((
(( σ), (X, + 1 and residual variance var(f.) ke,((,,; a.), (x., 0.)) k. (XX, .) (X, - where k, denotes the vector of kernel-based covariances (i.e., (x,,, x))) between x, and all the training points
7: return distribution of calibrated confidence score Níê, + f., var(f-))
such as those of the final softmax layer of a NN classifer, representing estimated class probabilities for each class. This modification allows RIO to access more information from the classifier outputs.
The targets for RIO training need to be redesigned as well. The raw targets of a classification problem are the ground-truth labels: they are in categorical space, while RIO works in continuous space. To solve this issue, RED constructs a different problem: Instead of predicting the labels directly, it predicts whether the original prediction is correct
A target confidence score is
4

---

Under review as a conference paper at ICLR 2021
assigned to each training data point accordingly. The residual between this target confidence score and the originally returned maximum class probability is calculated, and a RIO model is trained to predict these residuals. Given new data point, the trained RIO model combined with the original NN classifier thus provides a calibrated confidence score for detecting misclassification errors.
Figure I iillustrates the RED training and deployment processes conceptually, and Algorithm spec- ifies them in detail. In the training phase, the first step is to define a target confidence score C; for each training sample (x; Mi). For simplicity, all training samples that are correctly predicted by the original NN classifier receive as the target confidence score, and those that are incor- rectly predicted receive The validation dataset during the original NN training included in the training dataset for RED. After the target confidence scores are assigned, a regression problem formulated for the RIO model: Given the original raw features {x,] and the corresponding softmax outputs of the original NN classifier {o; [8,1 Pi,2: ................................ ,.......,...... pr.k]]== prediet the residuals
- (ra C: & ]=1 between target confidence scores c {c J and the original maximum class probabilities è = {2. max(r,))
The RIO model relies on an I/O kernel consisting of two components: the input kernel (x,, x;), which measures covariances in the raw feature space, and the modified multi-output kernel Kout (0..oy) which calculates covariances in the softmax output space. The hyperparameters of the 1/O kernel are optimized to maximize the log marginal likelihood log p(r| or). In the de- ployment phase, given a new data point x., the trained RIO model provides a Gaussian distribu- tion for the estimated residual A. ~ (N(f,, var(f.)). By adding the estimated residual back to the original maximum class probability é., a distribution of calibrated confidence score is obtained as d' NC, + fr, var(f-)). The mean é, + f. can be directly used as a quantitative metric for error detection, and the variance var(f.) represents the corresponding uncertainty of the confidence score.
4 EMPIRICAL EVALUATION
In this section, the error detection performance of RED is evaluated comprehensively on 125 UCI datasets, comparing it to other related methods. Its generality is then evaluated by applying it to two other base models, and its scale-up properties measured in larger deep learning architecture and in a larger task. Finally, RED's 's potential to improve robustness more broadly is demonstrated in a case study focusing OOD and adversarial samples.
4.1 COMPARISONS WITH RELATED APPROACHES
As comprehensive evaluation of RED, an empirical comparison with seven related approaches on 125 UCI datasets (Dua & Graff 2017) was performed. These approaches include maximum class probability baseline (MCP: Hendrycks & Gimpel, (0177), three state-of-the-ard approaches, namely Trust Score (Jiang et al. 2018). ConfidNet (Corbière et al. 2019), and Introspection-Ner (Aigrain & Detyniecki| 2019), as well as three earlier approaches, i.e. entropy of the original softmax outputs (tteinhardt & Liang 2016), DNGO (Snoek et al. 2015). and the original SVGP (Hensman et al.) 2013 2015). The 125 UCI datasets include 121 datasets used by Klambauer et al. (2017) and four more recent ones. Full details about the datasets and parametric setup of all tested algorithms, and a downloadable link for source codes are provided in Appendix A.
Following the experimental setup of Hendrycks & Gimpel (2017); Corbière et al. (0119) Aigrain & Detyniecki (2019), the task for each algorithm is to provide a confidence score for each testing point. An error detector can then use a predefined fixed threshold on this score to decide which points are probably misclassified by the original NN classifier. For RED, the mean of calibrated confidence score é + F. was used as the reported confidence score.
Five indeeeddn performance metrics were used to compare the methods: AUPR-Error, which computes the area under the Precision-Recal (AUPR) Curve when treating incorrect pre- dictions as positive class during the detection; AUPR-Success, which is similar to AUPR-Error but uses correct predictions positive class; AUROC, which computes the area under receiver operating characteristic (ROC) curve for the eтror detection task; A.-EError, which computes the average preci- sion (AP) under different thresholds treating incorrect predictions as positive class; and AP Auccess, which is similar to AP-Enro but uses correct predictions as positive class. AUPR and AUROC are commonly used in prior work (Hendrycks & Gimpel 2017
Corbière al. 2019
Aigrain & De-
5

---

Under review as a conference paper at ICLR 2021
Table 1:
Mean Rank on UCI datasets
AP-Error
AUPR-Error std
AP-Success med
AUPR-Success
AUROC
Method
mean:Estd
mean-l std
mean:istd
RED
1.39++0.61 *
1.49+
1.74+
1.80-+1.00.
1.65+
MCP Baseline
2.93+ 0.89
3.06+00.22
2.77+11007
2.75+1.111
2.80++.008
Trust Score
3.92:12..45
3.86112.50
3.64
3.6112.225
3.76.12.31
ConfidNet
6.13
6.33
6.07
6,0711.44
5.97:11.45
Introspection-Net
5.34.11.65
5.38
5.83
5.8911.51
5.71
Entropy
3.47=1.08
3.59+1.19
1.19
1.23+1.32
3.26+1.28
DNGO
1.19=1.51
5.46+1882
1.84
1.80=1.44
1.57+1147
SVGP
1.59#1.60
1.80
1.89
1.83+1.49
6.24+1.61
The symbols indicates that the differences between the marked ontry and all other counterparts are statitically significani at the 5% significance level for both paired t rest and Wilcoson tesi. The best entries that are significantly betier than all the others under buth statistical less are marked in boldface.
Table 2: A Pairwise Comparison between RED and Other Methods on UCI datasets
AP-Error +/=/-
AUPR-Error
AP-Success +/=/-
AUROC
Method
+/=/-
+ +/=/-
+/=/-
MCP Baseline
87 /5/5 /
90/32/0
58 63 1
56 65
/ 60
Trust Score
53/44// 16
49/47/17
50/ 47 16
48 08/49/10 16
59/ 17
ConfidNet
100/22/0
100/22/0
106 16
106/16/0
109 13/0
Introspection-Ne
/9/29/0
90/32/0
98/24/0
98/24/0
101 /2//0100
Entropy
4
75/46/1
53/68/1
53/ /
69/69/1
DNGO
92 17
73/31/5
99 / 10/0
97 12/0
98 /0
SVGP
93/23/1
98 28/23/1
/5/25/0
97 / 25 /0
102 19
The columns labeled number on
significanlly at significunce paired t-test, Wileoxen those labeled represent the contrary case; those labeled represent no statistical significunce.
tyniecki 2019), but discussed by Davis & Goadrich (2006) and Flach & Kull 2015), AUPR may provide opplyy-ottimistic measurement of performance. To compensate for this issue, AP-Error and AP-Success included as additional metrics. Since the target for the confidence metrics is to detect misclassification errors, the following discussion will focus more on AP-Error and AUPR- Error.
Ten independent runs were conducted for each dataset. During each run, the dataset was randomly split into training dataset and testing dataset, and standard NN classifier trained and evaluated on them. The same dataset split and trained NN classifier was used to evaluate all methods. Full details of the experimental setup are provided in Appendix I.
Table shows the ranks of each algorithm averaged over all 125 UCI datasets. The rank of each algorithm on each dataset is based on the average performance over 10 independent runs. RED performs best on all metries; the performance differences between RED and all other methods are statistically significant under paired t-test and Wilcoxon test. Trust Score has the highest standard deviation, suggesting that its performance varies significantly across different datasets.
As more detailed comparison, Table 2 shows how often RED performs statistically significantly better, how often the performance is not significantly different, and how often performs signifi- cantly worse than the other methods. RED most often significantly better, and very rarely worse. In a handful of datasets Trust Score is better, but most often it is not.
To illustrate the performance of RED further compared the bascline and the three state-of-the- art approaches, Figure 2 shows the distribution of the relative rank of RED, MCP baseline, Trust Score, ConfidNet and Introspection-Nee features in the dataset. These plots are based on the AP-Error metrie; other metrics provide similar results. RED performs consistently well over different dataset sizes and feature dimensionalities. Trust Score performs best in several datasets, but occasionally also worst in both small and large datasets, making it rather unreliable choice. ConfidNet generally exhibits worse performance on datasets with large dataset sizes and high feature dimensionalities, i.c. it does not scale well to larger problems.
function of the number of samples and the number of

---

Under review as a conference paper at ICLR 2021
RED
MCP Baseline
Trust Score
Conditktt
Intrespection-te Net
AP EIE Baack
AP-5m
29
1 30
20 13
1p
30 12
RED
MCP Bataline
Trust Score
CondifNet
Introspaction-Net
APtree Back
APArT
Number Featurea
Number Pastures
Number Features
Number Festures
Humber of Features
Figure 2: Performance Ranks Across Dataset Sizes and Dimensionalities on UCI Datasets. Each plot represents the distribution of relative ranks for one method (each column) as a function of the dataset size (top row) and the feature dimensionality (bottom row). Each dot in each plot represents the relative rank in one dataset. RED performs consistently well over datasets of different sizes and dimensionalities. Trust Score performs inconsistently, and ConfidNet performs poorly on larger datasets.
Table 3: A Pairwise Comparison between RED and Other Methods on UCI datasets
AP-Error +/=/-
AUPR-Emor +/=/-
AP-Success +/=/-
AUPR- Succe-S
AUROC
Method
+ +/=/-
+/=/- =-
BNN MCP
102 / 20
104 / 18 /
95 / /6
88 / 33 /1
95 /
BNN Entropy
67 /3//2
68/ /2/2
48 / 66 8
48 / 66 8
53 65
MC-Dropout MCP
/
88 /4//
70/ 52/0
67 / /5
71/51/0
MC-Dropout Entropy
58/68/0
/ /0
37/77/7
38/ /6/
4
The colamas labeled show the mimber of datasets on which RED performs significnnly beter at the 5% signiffiance level in paired 1-test, Wilcoron test, or both: those labeleed represent the contrary ease; those labeled represent stalistical significunce.
4.2 GENERALITY WRT. BASE MODELS
To evaluate generality and robustness of RED, was applied two other base models: an NN classifier using MC-dropout technique (Gal & Ghahramani 2016) and Bayesian Neural Network (BNN) classifier (Wen et al.) 2018). They were each traindd as base classifiers, and RED was then applied to each of them (implementation details are provided in Appendix A.I]] Experiments analogous to those in Table 2 were performed on 125 UCI datasets in both cases. Table shows the pairwise comparisons between RED and the internal confidence scores returned by the base models. MCP and Entropy represent the maximum class probability and entropy of softmax outputs, respectively, after averaging over 100 test-time samplings. RED significantly improves MC-dropout and BNN classifier in most datasets, demonstrating that it is a general technique that can be applied to variety of models.
4.3 SCALING UP TO LARGER ARCHITECTURES
To confirm that the RED approach scales up large deep learning architectures, VGG16 model
(Simonyan& & Zisserman 2015) was trained on the CIFAR- 10 dataset using a state-of-the-ar training
pipeline (see Appendix A.2 for details). In order remove the influence of feature extraction in image preprocessing and to make the comparison fair, all approaches used the same logit outputs of the trained VGG16 model as their input features. 10 independent runs are performed. During each run, a VGG16 model is trained, and all the methods are evaluated based on this VGG16 model.
Table 4 shows the results on the two main error detection performance metries (note that the table lists absolute values instead of rankings along each metric). Trust Score performs much better than in previous literatures (Corbière et al. 2019). This difference may be due to the fact that logit outputs are used as input features here, whereas Corbière et al. (2019) utilized a higher dimensional feature space for Trust Score. Based on the results, RED significantly outperforms all the counterparts in both metrics. This result demonstrates the advantages of RED in scaling up to larger architectures.
7

---

Under review as a conference paper at ICLR 2021
Table 4: A Comparison based on the VGG16 Network Architecture on the CIFAR-10 Task
RED
MCP Raseline
Trust Score
ConfiNNet
IntrospectionNe. mean-ssdd
Entropy
DNGO
SVGP
Metric
mttn:istd
med
mean sttd
mean std
meanii-sd
micm std
meunttatd
AP- (P-Eroo(())
49,88 1,99*
47.09:2.119
48.7612228
45.80=2885 45.70
42.11+:1.98 1.98
47.91
33,91 22177
40.71
AUPR Emro(())
49.79.12.00
46.99 12.21
48.63 63.68.02 2.29
42.01 1.98
47.81 2.19
34,43
40.60.1..34
symnhol indicttes differences between
eany and other counterparts are statitically significant paired test and Wilcoson test, The best entnes that are significanll better than all the others under both statestical test are marked in boldfaace.
significance level
mean/variance of RED confidence score for annealing
1.2
1.0
0.6
0.4
correct
incorrect
0.2
OOD
adversarial
0.075
0,080
0.085
0.090
0.095
0.100
0.105
0.110
0.115
variance
Figure 3: Identifying OOD and Adversarial Samples Based on Mean and Variance of Confi- dence Scores. Each dot represents one sample the testing set in the UCI Annealing task. The horizontal axis denotes the variance of RED-calibrated confidence score, and the vertical axis de- notes the mean. If an in-distribution sample is correctly classified by original NN classifier, it is marked "correct", otherwise it is marked "incorrect". Mean is a good separator of correct and incorrect classifications. High variance, on the other hand, indicates that RED is uncertain about its confidence score. which can be used to identify OOD and adversarial samples. In this manner, RED can serve as foundation for improving robustness of classifiers more broadly in the future.
4.4 A CASE STUDY WITH OOD AND ADVERSARIAL SAMPLES
In all experiments so far, the mean of calibrated confidence score è, + F. was used as RED's con- fidence score. Although good performance is observed in error detection by only using the mean, the variance of calibrated confidence score var(f.) may be helpful if the scenario is more complex, adversarial data.
e.g. the dataset includes some OOD data,
preliminary investigation of RED in such scenaria was performed by manually adding OOD and adversarial data into the test set of the UCI "annealing" dataset, on which RED detected errors well. The synthetic OOD and adversarial samples were created to be highly deceptive, aiming to evaluate the performance of RED under difficult cireumstances. The OOD data were sampled from Gaussian distribution with mean and variance 1, and the number of added OOD data was the same as the number of samples in the original test set. Note that all data points from original dataset are normalized to have mean () and variance for each feature dimension during preprocessing, so the OOD data and in-distribution data have similar scales. The adversarial data was created by adding negligible modifications to training samples that the original NN classifier predicted incorrectly with highest confidence. This process mimics an adversary that can arbitrarily alter the output of the NN classifier with minuscule changes to the input (Goodfellow al. 2014).
Figure shows the distribution of mean and variance of calibrated confidence scores for testing samples, including correctly and incorrectly labeled actual samples, well the synthetic OOD and adversarial samples. The mean is a good separator for correctly classified and incorrectly clas- sified samples, which tend to cluster the top and bottom half of the image, respectively. On the
8

---

Under review as a conference paper at ICLR 2021
other hand, variance is promising indicator of OOD and adversarial samples. RED's confidence scores of in-distribution samples have low variance because they covary with the training samples. The variance thus represents RED's " confidence in its confidence score. Samples with large variance indicate that RED is uncertain about its confidence score, which can be used as a basis for detecting OOI) and adversarial samples.
Thus, although the main focus this paper is to demonstrate RED on misclassification detection, the preliminary results in this subsection show that provides a promising foundation for detecting other error types as well.
DISCUSSION AND FUTURE WORK
One interesting observation from the experiments is that RED almost never performs worse than the MCP baseline. This result suggests that there is almost no risk in applying RED on top of an existing NN classifier. Since RED is based on a GP model, the estimated residual f. is close to zero if the predicted sample is far from the distribution of the original training samples, resulting in no change to the original MCP. In other words, RED does not make random changes to original MCP if it is very uncertain about the predicted sample, and this uncertainty is explicitly represented in the variance of the estimated confidence score. This property makes RED a particularly reliable technique for error detection.
Another interesting observation is that the variance is also helpful in detecting OOD and adversarial samples. This result follows from the design of the RIO uncertainty model. Since RIO in RED has an input kernel and an output kernel, lower estimated variance requires that the predicted sample close to training samples in both the input feature space and the classifier output space. This requirement is difficult for OOD and adversarial attacks to achieve, providing a basis for detecting them.
In real-world deployment, is necessary to define threshold for triggering error warning based on RED's confidence scores. A practical way is use a validation dataset determine how the precision/recall tradeoff changes over different thresholds. The user can then select threshold based on their preference.
The most compelling direction of future work is to extend this capability of RED further. Instead of using a single dimensional confidence score for error detection, it is possible to use mean and variance simultancously, leading to two dimensional detection space. Further separation between OOD and adversarial samples may be possible by adding one more dimension, such as the ratio between input kernel output and output kernel output. Also, instead of using hard target confidence score (i.e. either 0 or 1), may be possible to define soft target confidence that may be more informative. Further, RED may be used to calibrate other existing confidence metrics, such as the Trust Score, which may lead to a further improvement in detection performance.
6 CONCLUSION
This paper introduced new framework, RED, for error detection neural network classifiers that can produce more reliable confidence score than previous methods. RED is able to not only provide a calibrated confidence score, but also report the uncertainty of the estimated confidence score. Experimental results show that RED's scores consistently outperform state-of-theeart methods in separating the misclassified samples from correctly classified samples. Preliminary experiments also demonstrate that the approach scales up large deep learning architectures, and can form basis for detecting OOD and adversarial samples as well. is therefore a promising foundation for improving robustness of neural network classifiers.
REFERENCES
Jonathan Aigrain and Marein Detyniecki. Detecting adversarial examples and other misclassi- fications in neural networks by introspection. CoRR, abs/1905.09186, 2019. URL http:
//axxiv.org/aba.t 09186

---

Under review as a conference paper at ICLR 2021
Ahmed Alghoul, Sara Al Ajrami, Ghada Al Jarousha, Ghayda Harb, and Samy S. Abu-Naser. Email classification using artificial neural network. International Journal of Academic Engineering Research (IJAER), 2(11):8-144 14, 2018.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man. Con- crete problems in ai safety. 06 2016.
Ofelia Anjos, Carla Iglesias, Fátima Peres, Javier Martínez, Ángela García, and Javier Taboada. Neural networks applied to discriminate botanical origin of honeys. Food Chemistry, 175:128 136, 2015. ISSN 0308-8 -006. doi: https://bboorr//100000ffffoooodd //www. www.sccienedireec..
htth:
Peter L. Bartlett and Marten H. Wegkamp. Classification with reject option using hinge loss. J.
Mach. Learn. Res., 9: 9:18238840, June 2008. ISSN 1532-4435.
C. Chow. On optimum recognition error and reject tradeoff. IEEE Trans. Inf. Theor, 16(1):4146,
September 2006. ISSN 0018-9448. doi: 10.1109/TT.1990.10544466 URL hhttps://doi.
oTg/10.1109/Tm
1970. 1054406
Charles Corbière, Nicolas THOME, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure prediction by learning model confidence. In Advances in Neural Information Processing Systems 32, pp. 2902-2913. Curran Associates, Inc., 2019.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In Algorithmic Learning Theory, pp. 67-82, Cham, 2016. Springer International Publishing. ISBN 978-3-319- 46379-7.
Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. vol- ume 06, 2006. doi: 10.
Terrance Devries and Graham W. Taylor. Learning confidence for out-of-distribution detection in neural networks. ArXiv, abs/ 802./880..0465 2018
Matthew Dixon, Diego Klabjan, and Jin Bang. Classification-bass financial markets prediction using deep neural networks. Algorithmic Finance, 12 2017. doi: 10.2 10.2139/ssm.2755333
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL jRttp:////art ics. uci. edu/ml
Bernard Dubuisson and Mylne Masson. A statistical decision rule with incomplete knowledge about classes. Pattern Recognition, 26(1):155 165, 1993. ISSN 0031-3203. doi: https///doi.or org/
10.10166000113320330099900000000 URL http:: /ttp://wwww... sci encedi rect. wwm.coomo...oo.oo. ence/
a
Peter Flach and Meelis Kull. Precision-reeall-ghh curves: Pr analysis done right. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances Neural Information Processing Systems 28, pp. 838-846. Curran Associates, Inc., 2015.
Yarin Gal and Zoubin Ghahramani. Dropout a bayesian approximation: Representing model un- certainty in deep learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 16, pp. 1050-1059) JMLR.org, 2016. URL http:://l. acm. org/ / citation..... fw..immmiic... 3045390 3045502
Yonatan Geifman and Ran El-Yaniv, Selective classification for deep neural networks. In Proceed- ings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 48854894, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv 1412.6572, 12 2014.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 1321 1330. JMLR.org, 2017.
10

---

Under review as a conference paper at ICLR 2021
Cota Navin Gupta, Ramaswamy Palaniappan, Sundaram Swaminathan, and Shankar M. Krish- nan. Neural network classification of homomorphic segmented heart sounds. Applied Soft Computing, 7(1):286 297, 2007. ISSN 1568-4946. doi: http:///bi.org/11.11166// 2005.06.006. URL http:// http::/www.ett htw.//wwwcto..oooo encedi rect com com//science//ttticle/....
61568494605000699
David Haldimann, Hermann Blum, R. Siegwart, and C. Cadena. This is not what imagined: Error detection for semantic segmentation through visual dissimilarity. ArXiv, abs/1909.00676, 2019.
Simon Hecker, Dengxin Dai, and Luc Van Gool. Failure prediction for autonomous driving, 05 2018.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of -ifdiistritttonn examples in neural networks. Proceedings of International Conference on Learning Representa- lions, 2017.
James Hensman, Nicolò Fusi, and Neil D. Lawrence. Gaussian processes for big data. In Pro- ceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, UAI' pp. 282-290, Arlington, Virginia, United States, 2013. AUAI Press. UtLpttp: acm. .am.arr/../.o................... org/
ci craation.cfmmi--330
3023667
James Hensman, Alexander Matthews, and Zoubin Ghahramani. Scalable Variational Gaussian Process Classification. Guy Lebanon and S. N. Vishwanathan (eds.). Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Pro- ceedings of Machine Learning Research, pp. 351-360, San Diego, California, USA, 09-12 May 2015. PMLR. htLhhtt::// http::::ppocceeiiiig..
mir press /33/hhnsmann 1.hmm.mm..mmmmmmmmmmmmmm
Joel Janai, Fatma Güney, A. Behl, and Andreas Geiger. Computer vision for autonomous vehicles: Problems, datasets and state-of-the-ard ArXiv, abs/ 700105444955999 19,5 2017.
Heinrich Jiang, Been Kim, and Maya Gupta. To trust or not trust classifier, 05 2018.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 972981, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 108669644
Moshe Koppel and Sean P. Engelson. Integrating multiple classifiers by finding their areas of exper- tise. In In: AAAI-96 Workshop On Integrating Multiple Learning Models, pp. 53-58, 1996.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In Advances in Neural Information Processing Systems 32, pp. 12316 12326.
Curran Associates, Inc., 2019.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521:436 EP -, 05 2015. URL http:/ttsss//o...org///////
http//aturlllll 14539
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence--aliibaate classifiers for detecting out-of out-of--iitribuutin samples. In International Conference on Learning Representations, 2018a. URL
net. wtt..eee.ooouuunndrr.alll/aa
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting duttof-diittibutiin samples and adversarial attacks. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. desa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems 3/, pp. 7167-7177. Curran Associates, Inc., 2018b
Shiyu Liang. Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018. URL htt.:ttttp:://ppeeeeee
wet//fouumt forum? id
Amit Mandelbaum and D. Weinshall. Distance- based confidence score for neural network classi- 2017.
fiers. ArXiv,
11

---

Under review as a conference paper at ICLR 2021
Kristine Monteith and Tony R. Martinez. Using multiple measures to predict confidence in instance classification. The 2010 International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2010.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. pp. 427-436, 06 2015. doi: 10.1109/CVPR.2011 7298640.
Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning, ICML '05. pp. 625632, New York, NY, USA, 2005, Association for Computing Machinery. ISBN 1595931805. doi: 141145/1102351.11010
URL https://dci.ore 1145/
11C2351. 1102430
Julio Ortega. Exploiting multiple existing models and learning algorithms. In In AAAI 96 Workshop in Induction of Multiple Learning Models, pp. 239-266, 1995.
John C. Platt. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. In ADVANCES IN LARGE MARGIN CLASSIFIERS, pp. 61-74. MIT Press, 1999.
Foster J. Provost, Tom Fawcett, and Ron Kohavi. The case against accuracy estimation for compar- ing induction algorithms. In Proceedings of the Fifteenth International Conference Machine Learning, ICML '98, pp. 445453, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc. ISBN 1558605568.
Xin Qiu, Elliot Meyerson, and Risto Mikkulainen. Quantifying point-prediction uncertainty in neu- ral networks via residual estimation with an i/o kernel. In International Conference on Learning Representations, 2020.
Carla M. Santos-Pereine and Ana M. Pires. On optimal reject rules and roc curves. Pattern Recogn. Lett. 26(7):943952, May 2005. ISSN 0167-8655.
Alexander K. Seewald and Johannes Fümkranz.. An evaluation of grading classifiers. In Proceed- ings of the 4th International Conference on Advances in Intelligent Data Analysis, IDA '01, pp. 115124, Berlin, Heidelberg, 2001. Springer-Verlag. ISBN 3540425810.
Dan Selişteanu, Vladimir A. Maksimenko, Semen A. Kurkin, Elena N. Pitsik, Vyacheslav Yu. Musatov, Anastasia F. Runnova, Taryana Yu. Efremova, Alexander F. Hramov, and Alexan- der N. Pisarchik. Artificial neural network classification of motor-related ceg: An increase in classification accuracy by reducing signal complexity. Complexity, 2018:9359947, 2018. doi:
011555/2018/9385944 1BLLEEtte.///......1011111151//0
Nida Shahid, Tim Rappon, and Whitney Berta. Applications of artificial neural networks in health care organizational decision-making: A scoping review. PLOS ONE, 14(2):1-22, 02 2019. doi: http:/oonall.poe.002222223 pone. 0212356)
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference Learning Representations, 2015.
Jasper Snock, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat Prabhat, and Ryan P. Adams. Scalable bayesian optimization using deep neural networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML' 15, pp. 21712180 JMLR.org. 2015.
Jacob Steinhardt and Percy Liang. Unsupervised risk estimation using only conditional indepen- dence structure. Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 36643672, Red Hook, NY, USA, 2016. Curran Associates Inc.
ISBN 97815 10838898
Nalika Ulapane, Karthick Thiyagarajan, and Sarath Kodagoda. squared exponential kernel-based gaussian process regression, 03 2020.
initialization for
12

---

Under review as a conference paper at ICLR 2021
J. Wang, G. Dong. J. Sun, X. Wang, and P. Zhang. Adversarial sample detection for deep neural network through model mutation testing. In 2019 IEEE/ACM 41 International Conference on Software Engineering (ICSE), pp. 1245-1256, 2019.
Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse. Flipout: Efficient pseudo- independent weight perturbations on mini-batches. In International Conference on Learning Representations, 2018. URL
Gethin Williams and Steve Renals. Confidence measures for hybrid hmm/ann speech recognition. In In Proceedings of EuroSpeech, pp. 1955-1958, 1997.
Chen Xing, Sercan Arik, Zizhao Zhang, and Tomas Pfister. Distance-based learning from errors for confidence calibration. In International Conference Learning Representations, 2020. URL
htttps://openrev/ http://annaaieeee.e.eooooooannea
net/
Ming Yuan and Marten Wegkamp. Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research, 11(5):111-33,, 2010. URL http://
jm mall..orghe wwt..rog/pppppp//////e
/yy/ynnnn 0a. ht.m
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees
and naive bayesian classifiers. In Proceedings of the Eighteenth International Conference Machine Learning, ICML '01, pp. 609616, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558607781.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass prob- ability estimates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '02, pp. 694699, New York, NY, USA, 2002. Association for Computing Machinery. ISBN 158113567X doi: 10.1 10.11447715047.7755511 URL Tttps:://6i..rg/I0..11 htt.o///....11/711111 775151
A APPENDIX
A.1 EXPERIMENTAL SETUP FOR SECTION 4.1 AND SECTION 4.2
General Information 10 indenpendent runs are performed for each dataset. During each run, the dataset is randomly split into a training set (80%) and a testing set (20%). then a fully connected feed- forward NN classifier with 2 hidden layers, each with 64 hidden neurons, are trained on the training set. The activation function is ReLU for all the hidden layers. The maximum number of epochs for training is 1000. 20% of the training set used as validation set, and the split is random each inde- pendent run. An early stop is triggered if the loss on validation set has not be improved for 10 epochs. The optimizer is Adam with learning rate 0.001, 8 0.9, and B/ 0.999. The loss function is cross entropy loss. During each independent run, the same random dataset split and trained base NN classifier is used for evaluating algorithms. All source codes for reproducing the experimental results can be downloaded from lhtpps://drive.googgs wwt/dd////fooooerss///////oo 1Itess//XXXa (XR66s6kjiuuR7BBAA rY6QnCvosóa wtsposshrnnppsssiingggg
Dataset Description In total, 125 UCI datasets used in the experiments, among which 121 are from Klambauer et al. (2017). and 4 are recent datasets released in Dua & Graff (2017). All features in all datasets normalized to have mean 0 and standard deviation 1. Full details regarding the number of samples N, number of features M, and number of classes K for each dataset are shown in T
Parametric Setup for Algorithms
RED: SVGP (Hensman et al.) 2013 2015) is used as an approximator to original GP. The number of inducing points are 50. RBF kernel is used for both input and multi-output ker- nel. Automatic Relevance Determination (ARD) feature is turned on. The signal variances and length scales of all the kernels plus the noise variance the trainable hyperparame- ters. The optimizer is L-BFGS- B with default parameters as in htggs..ttm. and the tation http:////eccccooooooooooooooeeed
documen-
13

---

Under review as a conference paper at ICLR 2021
maximum number of iterations is set as 1000. The optimization process runs until the L-BFGS-B optimizer decides to stop. To overcome the sensitivity of GP optimization to initialization of the hyperparameters (Ulapane et al.] 2020), 20 random initialization of the hyperparameters are tried for each independent run. For each random initialization, the signal variances are generated from uniform distribution within interval [0, 1,, and the lengthscales are generated from a uniform distribution within interval [0, 10). For 10 ini- tializations, the hyperparameters of input kernel are first optimized while the multi-output kernel is temporarily turned off, then after the optimizer stops, the multi-output kernel is turned on, and both the two kernels are optimized simultaneously. For the other 10 initial- izations, both kernels are optimized simultaneouly from the start. The average performance of the 3 best optimized model in terms of corresponding metrics are used as the final per- formance of RED on each independent run. During our preliminary investigation, several statistic metrics on training set is effective in picking the true best-performing model out of these 20 trials, c.g., the gap between average estimated confidence scores of correctly classified training samples and incorrectly classified training samples, the scale of opti- mized noise variance of SVGP model, the ratio between sum of signal variances and noise variance after optimization, etc. Since improving initialization and optimization of GP hy- perparameters is out of the scope of this work, we simply use average performance of the best 3 models (top 15%) in comparison.
The maximum class probability of softmax outputs of the base NN classifier is used as the confidence score of MCP baseline. The setup of the base NN classifier is provided above.
MCP baseline:
Trust Score: k=10, a = 0. without filtering. This is the same as the default setup in
ConfidNet: During training. the input to ConfidNet is the raw feature, and the target is the class probability of the ground-truth class returned by base NN classifier. The architecture of ConfidNet is fully connected feed-forward NN regressor with 2 hidden layers, each with 64 hidden neurons. The activation function is ReLU for all the hidden layers. The maximum number of epochs for training is 1000. An early stop is triggered if the loss on validation data has not be improved for 10 epochs. The optimizer is RMSprop with learning rate 0,001, and the loss function mean squared error (MSE).
Introspection-Net During training, the input to Introspection-Ner is the logit outputs of base NN classifier, and the target is for correctly classified sample, and 0 for incorrectly classified sample. The architecture of ConfidNet a fully connected feed-forward NÑ regressor with 2 hidden layers, each with 64 hidden neurons. The activation function is ReLU for all the hidden layers. The maximum number of epochs for training is 1000. An early stop triggered if the loss validation data has not be improved for 10 epochs. The optimizer is RMSprop with learning rate 0.001, and the loss function is mean squared error (MSE).
Entropy:
The entropy of softmax outputs of the base NN classifier is used as the confidence score of Entropy. THe setup of the base NN classifier is provided above.
DNGO: A Bayesian linear regression layer similar to that of Snoek et al. (2015) is added after the logits layer of the original NN classifier to predict whether an original predic- tion is correct or not (1 for correct and 0 for incorrect). Default parametric setup, in mwwstrr/pbbbm//mmgg..pp is used.
The original SVGP without output kernel is used to predict directly whether pre- diction made by the base NN classifier is correct or not (1 for correct and 0 for incorrect). All other parameters are identical to those in RED.
SVGP:
BNN MCP: The standard dense layers in the base NN classifier described in RED setup above is replaced with Flipout layers (Wen et al. 2018). All other parameters are identical with those in RED. The maximum class probability averaging over 100 test-time samplings is used as the confidence score for error detection.
BNN Entropy: The same setup with BNN MCP, except now the entropy of softmax outputs averaging over 100 test-time samplings is used as the confidence score for error detection.
14

---

Under review as a conference paper at ICLR 2021
MC-Dropout MCP: A dropout layer with dropout rate of 0.5 is added after each dense layer the base NN classifier described in the RED setup. All other parameters are identical with those in RED. The maximum class probability averaging over 100 test-time Monte- Carlo samplings is used as the confidence score for error detection.
MC-Dropout Entropy: The same setup as with MC-Dropout MCP, except now the entropy of softmmx outputs is averaged over 100 test-time Monte-Carlo samplings and used con- fidence score for error detection.
A.2 EXPERIMENTAL SETUP FOR SECTION
Setup of VGG16 Training The standard VGGI6 architecture (Simonyan & Zisserman 2015) is used. The training pipeline is based on the default setup described in www.e///eeiaannnaaaaagggoooooooo
(Ovgg--pffr For the CIFAR-10 10 dataset, 40,000 samples are used as the training set, 10,000 as the validation set, and 10,000 as the test- ing set.
Parametric Setup for Algorithms All algorithms use the logit outputs of the trained VGG16 model as input features. The maximum class probability of softmax outputs of the trained VGG16 model is used the confidence score of MCP baseline. The parameters for RED, Trust Score, Entropy, DNGO and SVGP are identical to those in the UCI experiments. For ConfidNet and Introspection-Ner, all parameters are the same in the UCI experiments, except for that the number of hidden neurons for all hidden layers is increased to 128.
A.3 DETAILED RESULTS FOR SECTION 4.1 AND SECTION
This subsection shows all the detailed results for experiments performed in Section 4.1 and Sec-
tion 4.2] The results are averaged over 10 independent runs in terms of AP-Error, AP-Success.
AUPR-
and AUROC. Detailed results for Section 4.1 are shown in Table
Table 6. Table 7 Table 8 and Table9 Detailed results for Section 4.2) are shown in Table [O] Ta-
ble 1 Table 12 Table 13, and Table Each table is corresponding to one performance metric. The column N "M" and "K" denotes the number of samples, number of features, and number of classes for corresponding datasets. To save space, "MCP". "Intro-Net", "MC-D" stands for "MCP Baseline", "IItrospection-Nor
and "CC-Dropout" respectively. For datasets that the original NN classifier achieves 100% accuracy, the entries are marked as "NA". For dataset splits that the number of samples in one particular class is too small for Trust Score calculate neighborhood distance, the entries are marked as "NA"
Comparison RED Counterparts Using Emar
Dataset
RED
MCP
Score
ContidNet Intro-Net
Entropy
DNGO
abalone
4177
3
0.546
0.539
0.480
0,518
0.528
0.529
0.496
0.482
cntteinnflmmation
120
2
NA
NA
NA
NA
NA
NA
NA
NA
acute-nophritis
120
6
2
NA
NA
NA
NA
NA
NA
NA
NA
adult
48842
0.419
0.411
0.391
0.358
annealing
898
31
5
0.558
0.392
0.453
0.350
0.399
0.368
0.314
0.324
anhythmia
0.583
0.409
0.475
audiology-std
196
59
18
0.779
0.721
NA
0.428
0.518
0.731
0.590
0,556
balanco-scabl
625
0.888
0.816
0.540
0.046
0.155
0.730
0.319
0.128
balloons
16
0,840
0.806
0,704
0.694
0,806
0.806
0.615
0.583
bank
4521
16
2
0.397
0.393
0,390
0.372
0.339
0.393
0.108
0.383
bioconcentration
779
0.461
0.385
blood
748
2
0.390
0.389
0,272
0.382
0.372
0.389
0,319
0.362
cencer-
286
0.503
0.381
0.467
0.431
0.503
0.471
brenst-canee-wwis
699
2
0.423
0.393
0,405
0.313
0.245
0.393
0.175
0.152
Deeat--ancer-wisee wise-diag
569
30
0.626
0.599
0.500
0.089
0.374
0.601
0.089
0.119
hecsst-onner-wwis--rrog
198
33
0.408
0.403
0.296
0. 0.343
0.373
0.403
0.288
0.266
breast-lissue
106
9
6
0.813
0.743
0,800
0.696
0.601
0.749
0.744
0.722
car
1728
cardiotocography. 10clases
2126
21
1D
0.513
0.492
0.400
0.310
0.302
0.472
0,303
0.290
cardiotooography-3a0
2126
21
3
0.439
0.386
0.393
0.273
0.282
0.382
0.125
0.284
cbess-krvk
28056
18
0.550
0.523
0,715
0.486
0.449
0.515
0.386
0.405
chess-krvkp
3196
36
0.347
0.339
0.163
0.037
0.087
0.339
0.038
0.054
climate
540
20
0.456
0.428
0.276
0.093
0.394
0.095
0.132
congressional-voot vootng
435
16
2
0.468
0.465
0.429
0.474
0.459
0.464
0.478
0.470
conn hench sonar mines rocks
0.531
0.296
0.531
0.304
conn-hench- vowel deterding
990
11
11
0.689
0.523
0.875
0.148
0.170
0.493
0.271
0.200
Contiened next page.
15

---

Under review as a conference paper at ICLR 2021
Datasct
N
M
K
RED
MCP
Trust Soore
ConfidNet
Intro-Net
Entropy
DNGO
SVGP
connect-4
67557
42
2
0,406
0.390
0.326
0.259
0.395
0.390
0.321
0.155
contrac
1473
9
3
0.596
0.590
0.529
0.552
0.584
0.577
0.573
0,548
approval
690
0.365
0.359
0.336
0.264
0.331
0.359
0.302
0.336
cylinder-bands
512
35
2
0.494
0.482
0,506
0.345
0.422
0,482
0,383
0,304
dermatology
366
34
0.690
0.648
0.605
0.072
0.253
0.686
0.327
0.130
echocardiogram
131
2
0.510
0.413
0.325
0.353
0.407
0,413
0.306
0.375
ecoli
336
8
0.433
0.421
0.472
0.283
0.284
0.374
0.278
0.286
energy-yl
768
3 0,472
0,380
0,289
0.195
0.187
0,364
0,144
0.244
caorgy-y2
768
8
3
0.498
0.434
0.498
0.322
0.278
0,421
0.302
0.328
fertility
100
0.267
0.238
0. 193
0.217
0.238
0.169
0.160
flags
194
28
8
0.767
0.717
0,784
0,666
0.664
0.712
0,649
0.659
glass
214
9
0.583
0.525
0.657
0.507
0.412
0.489
0.498
0.414
habernan survival
306
0.452
0.446
0.386
0.412
0.425
0.446
0,360
0.499
hayes-roth
160
3
0.688
0.576
0.728
0.603
0.572
0.576
0.657
0.536
beart-cleveland
303
0.759
0.748
0,707
0.689
0.705
0.732
0.627
0.730
beart-bungarian
294
12
2
0.436
0.418
0,442
0.261
0.397
0.418
0.356
0.351
beart-switzerland
123
0.735
0.692
0.726
0.697
0.662
0.700
0,705
0.649
beart-va
200
12
5
0.757
0.732
0.676
0.678
0.732
0.746
0,677
0.639
bepatitis
155
19
2
0.542
0.441
0.495
0.423
0.330
0.441
0.408
0.369
hill valley
1212
LOD
0.657
0.546
0.478
0.508
0.594
0.546
0.554
0,546
horse-colic
368
25
2
0.444
0.431
0.426
0.263
0. 0.394
0.431
0.304
0.247
iidd-indian-liver
583
0,470
0.459
0,427
0.434
0.443
0,459
0.375
0.445
image senge-segmettaiion
2310
18
7
0,539
0.455
0.471
0.251
0.214
0.430
0.218
0.139
ionosphere
351
33
2
0.543
0.493
0.355
0.158
0,405
0.493
0,300
0.171
iris
150
3
0,686
0.633
0,747
0.154
0.202
0,605
0,360
0.227
led display
1030
10 0.572
0.571
0.301
0.553
0.512
0.547
0.508
0.470
lenses
24
3
0.875
0.900
0.786
0.665
0.790
0.900
0.622
0.587
lotter
20000
16
26
0.495
0.461
0.785
0.245
0. 0.081
0.465
0.109
0.164
libras
360
0.602
0.512
0.307
0.301
0.498
0.279
low res spect
531
100
9
0.552
0.515
0.454
0.304
0.305
0.504
0.297
0,343
lung-canser
32
56
3
0.887
0.819
0.695
0.690
0.841
0.859
0.641
0.582
lymphography
148
18
4
0,660
0.546
0.619
0.278
0.386
0.545
0,353
0,363
magic
19020
0.366
0.355
0.294
0.309
0.350
0.355
0,240
0.233
mammographic
961
0.471
0.468
0,288
0.414
0.454
0.468
0.386
0.402
messidor
1151
19
2
0.465
0.449
0.359
0,364
0.426
0.449
0.411
0,350
miniboone
130064
0.380
0.361
0.299
0.371
0.162
molec biol promnoter
106
57
2
0.550
0.541
0.786
0.330
0.505
0,541
0.432
0.437
molec-biol-splice
3190
60
3
0.461
0.455
0.375
0.193
0,208
0.464
0.396
0.174
monks-1 1
556
2
0.618
0.444
0,167
0.242
0.343
0.444
0,110
0,222
monks-2
601
0.524
0.466
0.611
0.417
0.426
0.466
0.389
0,387
monks-3
554
0.648
0.561
0,369
0.319
0,453
0.561
0.395
0,175
mushroom
8124
21
2
NA
NA
NA
NA
NA
NA
NA
NA
musk-1
476
0.518
0.508
0,501
0.101
0.354
0,508
0,295
0.103
musk-2
6598
166
2
0.409
0.372
0
0.009
0.077
0.372
0.006
0.007
nursery
12960
5
0,586
0.448
0.074
0.035
0.127
0.447
0.135
0.173
oncyles. www.umcius.....oom nucleus.44 4d
1022
41
0.442
0.401
0,277
0.296
0.383
0.401
0.385
0.330
oneytes merlaccius states 2
1022
25
3
0.505
0.488
0.413
0.256
0.358
0.477
0.214
0.262
oncytes. oocytes.trisopeens trssstroooouuuuuucuuuuu mucleus.. 2f
912
25
0.481
0.451
0,342
0.275
0.460
0,451
0.354
0.300
Srsyltestisopteruss states._bb
912
32
3
0.445
0.423
0,346
0.155
0.291
0.441
0.316
0.114
optical
5620
0.452
0.582
0.048
0.036
0.422
0.079
0.068
nzone
2536
72
2
0.308
0.293
0,270
0.207
1.666
0.293
0.032
0.025
page-blocks
5473
0.450
0.386
0.399
0.330
0.275
0.371
0.254
0.186
parkinsons
195
22
0.562
0.516
0.541
0.155
0.239
0.516
0.276
0.316
pendigits
10992
16
10
0.491
0,362
0.684
0.8
0. 0.132
0.363
0.
0.059
phishing
1353
3
0.453
0.433
0.369
0.347
0.371
0,412
0.338
0.334
pina
768
8
2
0.501
0.492
0.422
0.412
0.486
0.492
0.426
0.336
pittsburg bridges MATERIAL
106
0.600
0.501
0.360
D.511
0.506
0.392
Brobuugg-riddes-REEE.
103
3
0.593
0,536
0.654
0.522
0.549
0.529
0,501
0.498
pirtsbug--iidgss-PAA.
92
3 0,582
0.550
0.543
0.367
0.507
0.563
0.472
0.449
pirtshurg- Wwsbbugghridges---OOR.
102
2
0.472
0.397
0.456
0.391
0.506
0.397
0.375
0.352
pitsburg-brigge--YPE
105
7
6
0.809
0.736
0.758
0.684
0.662
0.692
0.702
0.737
planning
182
0.366
0.364
0.387
0.348
0.331
0.312
0.259
plani murgin
1600
64
100
0.586
0,572
0
0.331
0.266
0.569
0.305
0.183
plant-shupe
1600
100
0.693
0.666
0.759
0.524
0.477
0.643
0.517
0.477
plant-texture
1599
64
100
0.607
0,596
0,708
0,308
0.291
0.610
0,282
0.221
post-operative
90
3
0.428
0.416
0.289
0,377
0,413
0.374
0.441
0.418
primary-tumor
330
15
0.794
0.783
0.693
0.725
0.721
0.779
0.683
0.715
ringnorm
7400
20
2
0.348
0,344
0,275
0.042
0.116
0.344
0.112
0.027
seeds
210
0.583
0.514
0.526
0.344
0.333
0.397
0.345
semeion
1593
256
10
0.541
0,527
0.704
0.092
0.237
0.564
0,180
0.142
soyhean
683
0.573
0.516
0.448
0.317
1.191
0.505
0.234
0.319
spanbase
4601
57
2
0,316
0.292
0,299
0.179
0.238
0.292
0,203
0,139
spect
265
22
2
0.520
0.506
0.384
0.443
0.471
0.506
0.419
0.434
spectf
267
0.478
0.427
0.429
0.251
0.428
0.427
0.258
0.346
690
14
2
0.473
0.455
0.418
0.417
0.416
0.455
0.407
0.430
statlo german credit
1030
0.423
0.422
0,383
0.329
0.389
0,422
0.354
0.254
statlng-heart
270
13
2
0.471
0.438
0.388
0.322
0.462
0.438
0.350
0.285
Contiened on next page.
16

---

Under review as a conference paper at ICLR 2021
Datasct
N
M
K
RED
MCP
Trust Soore
ConfidNet
Intro-Net
Entropy
DNGO
SVGP
starlog-image
2310
18
7
0.561
0.464
0.484
0.240
0.114
0.473
0.274
0.183
statlog- landsat
6435
36
6
0.430
0.39%
0.554
0.322
0.312
0.409
0.288
0.255
statlog shuttle
58000
0.551
0.318
0.547
0.233
0.157
0.310
0.139
0.247
statlog-vehicle
846
18
4
0.542
0.526
0,319
0.392
0.361
0.536
0,346
0,364
stecl-plates
1941
27
0.529
0.517
0.533
0.452
0.437
0.514
0.437
0.404
synthetic-contee
600
60
0.647
0.541
0,462
0.052
0.117
0.498
0.199
0,031
teaching
151
0.646
0.604
0.540
0.632
0.593
0.612
0.676
0.569
thyroid
7200
21
3
0,442
0.413
0,160
0.126
0.190
0,409
0.015
0.114
tic-tac-toe
958
9
2
0.633
0.538
1,000
0.160
0.513
0.538
0.385
0.480
ritanic
2201
0.314
0.313
0,202
0.310
0.315
0.313
0,227
0.321
trains
10
29
2
0.778
0.750
0,917
0.833
0.750
0.750
0,625
0.750
twonorm
7400
20
2
0.371
0.357
0.404
0.160
0.2900
0.357
0.176
0.037
centebral-columee 2elases
310
0.501
0.490
0.346
0.378
0.432
0.490
0.389
0.273
vertebral- cow.mm- Jeluses
310
3
0.542
0.518
0.308
0.489
O. 0.396
0.529
0.439
0.307
wall-following
5456
24
0,418
0.378
0,304
0.208
0.248
0,370
0.261
0.139
waveform
5000
21
3
0.465
0.423
0,331
0.357
0.394
0.423
0.200
0.162
waveform-noise
5000
40
0.428
0.398
0.322
0.230
0.326
0.397
0.215
0.167
wine
178
13
3
0.894
0.742
0.661
0.101
0.458
0.825
0.613
0,284
qual--uulity-red
1599
11
6
0.534
0.522
0.540
0.494
0.491
0.496
0.523
0.453
wine quality white
4898
11
7
0.541
0.529
0.626
0.534
0.523
0.516
0.524
0.480
yeast
1484
8
10
0.570
0.563
0.510
0.506
0 0.498
0.548
0.410
0.912
Z00
101
7
0.910
0.804
0.793
0.297
0.546
0.893
0.443
0.694
Continued from previous page.
Table 6: Comparison between RED and Counterparts Using AP-Success
Dataset
N
M
K
RED
MCP
Trust Score
ConfidNet
Intro-Net
Entropy
DNGO
SVGP
abalone
4177
8
3
0.853
0.850
0.821
0.845
0.844
0.849
0,814
0.840
acute- inlleminlammation
120
NA
NA
NA
NA
acute nephritis
120
2
NA
NA
NA
NA
NA
NA
NA
NA
adult
48842
2
0.968
0.967
0,944
0.964
0.961
0.967
0.927
0,961
annealing
898
31
5
0.975
0.899
0,943
0.909
0.916
0.902
0,898
0.913
arthythmia
452
262
13
0.817
0.810
0.809
0.654
0.668
0.803
0.673
0.778
audiology-std
196
59
18
0,946
0.926
0.625
0.740
0.927
0.756
0.782
balance-scale
625
3
0.999
0.999
0,997
0.972
0.983
0.999
0.985
0.976
ballooms
0.843
0.806
0.769
0.806
0.667
bank
4521
16
2
0.977
0.980
0.976
0.968
0.937
0.980
0.892
0.979
bioconoentration
779
3
0.746
0.728
0.782
0.717
0.726
0.721
0.693
0.698
blood
748
2
0.892
0.892
0.873
0.902
0
0.892
0.869
0.900
breast-cancer
286
9
2
0.820
0.820
0,765
0. 796
0.775
0.820
0.767
0.785
becast-cance-wiis
699
0.998
0.997
0.998
0.994
0.985
0.997
0.986
breast-cancer-wee wiseer-wsse-diag
569
30
2
0.996
0.996
0,997
0.970
0
0.998
0.973
0.981
breast cancer wisc prog
198
0.856
0.860
0.797
0.824
0.835
0.860
0.787
0.795
breast-fissue
106
9
6
0.909
0.878
0.898
0.863
0.767
0.872
0,800
0.849
car
1728
4
0.999
0.999
0.999
0.992
0.988
0.999
0.984
0.990
curdiotocography-1tee
2126
21
10
0.959
0.959
0,927
0.890
0.872
0.958
0.885
0.901
cardiotooography--llee
2126
21
3
0,990
0.991
0
0.972
0. 0.964
0.991
0.935
0.979
cbess-Krvk
28056
18
0.900
0.867
0.782
0.801
chess krvkp
3196
36
2
1.000
1.000
0.995
0.990
0.994
1.000
0.993
0.996
climate
540
0.989
0.938
0.976
0.987
0.945
0.950
congressionall-vorie
435
16
2
0.693
0.690
0.672
0.692
O 0.684
0.689
0.691
0.689
comn-bench-sonna-miine mine--oossssss
208
60
2
0.962
0.963
0.980
0.815
0.901
0.963
0.874
0.897
conn- benchbenhhhhwee vowel- deteerdng
990
11
1,000
1.000
1.000
0.980
0.984
1.000
0.989
0.989
connect-4
67557
42
2
0.969
0.969
0.938
0.942
0.958
0.969
0.927
0.868
contrac
1473
0.717
0.581
0.682
0.675
0.658
0.669
credit approval
690
15
2
0.943
0.941
0,916
0.909
0.936
0.941
0.896
0.928
cylinder-hands
512
0.865
0.862
7.7
0.809
0.862
0.767
0.758
dermatology
366
34
6
0.999
0.999
0.997
0.975
0.981
0.999
0.977
0.951
cchocardiogram
131
0.944
0.936
0.925
0.916
0.924
0.936
0.888
0.929
ecoli
336
8
0.959
0.956
0.969
0.911
0.879
0.954
0,886
0.919
caergy-y1
768
8
3
0.992
0.994
0,992
0.975
0.974
0.994
0.959
0.983
energy-y2
768
0
0.986
0.985
0.962
0.946
0.963
0.978
fertility
100
2
0.953
0.928
0,948
0.920
0.929
0.928
0.903
0.939
flags
194
28
0.739
0.716
0.817
0,573
0.617
0.720
0.580
0.599
glass
214
9
6
0.854
0.811
0,915
0.779
0.738
0,810
0.741
0.783
survival
306
0.850
0.849
0.854
0.821
0.835
0.849
0.818
0.835
roth
160
0.931
0.881
0.959
0.887
0.864
0.887
0,881
0.837
cleve-cle
303
13
5
0.856
0.859
0,836
0.800
0.822
0.861
0.696
0.838
heart hungarian
294
0.925
0.908
0.847
0.915
0.844
0.869
heart- switzerad
123
12
5
0.500
0.480
0.478
0.451
0.452
0.490
0.471
0.446
beart-va
200
12
0.526
0.499
0.462
0.439
0.504
0.511
0.417
0.392
bepatiti
155
19
2
0.965
0.965
0,975
0.930
0.896
0.965
0.897
0.915
hill valley
1212
100
0.763
0.704
0.559
0.653
0.741
0.704
0.698
0.693
horse-colic
368
25
0.927
0.918
0.920
0.833
0.913
0.918
0,836
0.871
Continued on ned page.
17

---

Under review as a conference paper at ICLR 2021
Datasct
N
M
K
RED
MCP
Trust Soore
ConfidNet
Intro-Net
Entropy
DNGO
SVGP
ipd Indian-liver
583
9
2
0.878
0.874
0.814
0.860
0.863
0.874
0,800
0.871
image segrmentation
2310
18
7
0.997
0.997
0,998
0.983
0.984
0.997
0.981
0.985
sonosphere
0.990
0.988
0.976
0.900
0.973
0.958
0
0.914
iris
150
4
3
0.992
0.991
0.997
0.938
0.917
0.991
0,950
0.955
lod-display
1030
10
0.883
0.883
0.759
0.874
0,850
0.880
0.835
0.847
lenses
24
3
0.982
0.960
0.944
0.871
0.883
0,960
0,859
0.844
letter
20000
16
26
0.995
0.996
0.999
0.980
0.947
0.996
0.963
0.982
libras
360
90
15
0.956
0.943
0,956
0.808
0.807
0.941
0.863
0.850
low-res-spect
531
100
9
0.977
0.968
0,953
0.908
0. 0.892
0.968
0.902
0,950
lung- cancer
32
56
0.771
0.734
0.650
0.547
0.743
0.739
0.496
0.459
tymphography
148
18
4
0.941
0.925
0,958
0.809
0.861
0.924
0,865
0.826
magic
19020
10
2
0.966
0.966
0.954
0.961
0.959
0.966
0.907
0.946
mammoographic
961
0.922
0.919
0.882
0.910
0.911
0.919
0,882
0.926
messidor
1151
19
2
0.863
0.861
0.764
0.806
0.830
0.861
0.808
0.806
miniboone
130064
50
0,992
0.992
0,983
0,988
0,986
0.992
0.972
0.981
molec-biol-premoter
106
57
2
0.892
0.888
0,953
0.724
0.880
0.888
0.784
0.822
molec-biol-splice
3190
3
0.953
0.953
0
0.833
0.846
0.952
0.901
0.838
monks-1
556
2
0.998
0.997
0,984
0.988
0.989
0.997
0.981
0.995
monks-2
601
2 0.885
0.857
0.918
0.809
0.810
0.857
0.761
0.808
monks. 3
554
2
0.989
0.999
0.993
0.981
0.993
0.999
0.981
0.991
mushroom
8124
21
2
NA
NA
NA
NA
NA
NA
NA
NA
musk-1
476
166
0,984
0,983
0,986
0.912
0.957
0,983
0,942
0.934
musk-2
6598
166
2
1,000
1.000
0,998
0.995
0.995
1.000
0.994
0.994
pursery
12960
5
1,000
1.000
1,000
0,999
0,999
1.000
0.999
1.000
socyees.mrluecciu aucleuss_dd .4d
1022
41
2
0.939
0.930
0
0.892
0.904
0,930
0,890
0.907
oncyles, deeeeseelacccus..... states.25
1022
25
3
0.991
0
0.991
0.967
0.970
0.991
0.950
0.975
oncytes riioopeersspuuclle 2f
912
25
0.940
0.931
0.908
0.872
0.923
0.931
0.873
0.876
oocytes eacytes.trisopterus www.oortorussssttoooo statess55
912
32
3
0.991
0.990
0,985
0,952
0. 0.967
0.990
0.955
0.945
optical
5620
0.999
0.999
0
0
0,999
0.982
0.992
osome
2536
72
2
0.994
0.996
0,993
0.988
0.966
0.996
0.968
0.951
page-blocks
5473
10
5
0.998
0.997
0.993
0.993
0,989
0.997
0.989
0.994
parkinsons
195
22
2
0.994
0.992
0,995
0.939
0.944
0,992
0.962
0.964
pendigits
10992
16
10
0.999
0,999
1.000
0.992
0.992
0.999
0.994
0,993
phishing
1353
3
0.975
0.973
0.958
0.959
0.946
0.972
0.929
0.952
pima
768
8
2
0.898
0.894
0,875
0.848
0.873
0.894
0.857
0.870
pitisburg- beiiges-MATERRIE MATERIAL
106
0.976
0.969
0,921
0.933
0.939
pittsburg bridges L.
103
7
3
0.790
0.780
0.845
0.766
0.744
0,783
0.735
0.773
Miisbuug-bridee-SSPA.
92
3
0.851
0.808
0.785
0.678
0.756
0.805
0.782
0.716
pilisburg- T-OR-D
102
2
0.957
0.943
0,924
0,926
0.946
0.943
0,932
0.920
pittsburg bYiP
105
7
6
0.818
0.692
0.747
0.651
0.635
0.664
0.737
0.766
planning
182
2
0.729
0.722
0.737
0.730
0.728
0.722
0.734
0.692
plant-margin
1600
64
100
0.951
0.950
0,966
0.828
0.817
0.950
0.816
0.731
plant-shupe
1600
100
0.849
0.718
0.671
0.838
0.697
0.673
plant fextt-exxture
1599
64
100
0.953
0.950
0,958
0.809
0.804
0.950
0.789
0.791
post- operatpee
90
3
0.683
0.673
0.739
0.684
0.685
0.670
0.727
0.687
primary- tumer
330
17
15
0.762
0.752
0.654
0.640
0.633
0.750
0.546
0.642
ringnorm
7400
20
2
0.999
0.999
0
0.982
0.977
0.999
0.981
0.976
seods
210
3
0.988
0.986
0.992
0,964
0.961
0.985
0.935
0.947
senacion
1593
256
10
0.994
0.994
0,996
0.927
0.950
0.994
0.943
0.963
suyheun
0
0.995
0
0.970
0.942
0.995
0,944
0.974
spumhase
4601
57
2
0.989
0.988
0.984
0.965
0.966
0.988
0.962
0.974
spect
265
22
2
0.811
0.802
0.767
0.731
0.788
0.801
0.686
0.733
spectf
267
44
2
0.955
0.951
0,942
0.913
0.919
0.953
0.879
0.949
narlog-australian-cred
690
14
2
0.701
0.692
0.656
0. 6.668
0.678
0.692
0.629
0.662
1030
0.890
0.891
0.873
0.837
0.850
0.891
0.821
0.782
statlog-beart
270
13
2
0.940
0.935
0.906
0.882
0.917
0.935
0.853
0.863
statlog image
2310
0.998
0.998
0
0.986
0.980
0.998
0,984
0.990
starlog-landsat
6435
36
6
0.986
0.987
0
0.964
0.959
0.987
0.953
0.971
statlog-shuitle
58000
9
7
1,000
1.000
1.000
0.998
0.998
0.999
0.999
0.999
varlog-vehicle
846
4
0.966
0.967
0.927
0.929
0.913
0.968
0,870
0.922
steel-plates
1941
27
7
0.919
0.917
0.928
0.882
0 0,850
0.916
0.827
0.870
syuthetic-coatool
600
1,000
1.000
0.986
0.988
1,000
0.988
0.986
teaching
151
5
3
0.686
0.648
0.598
0.657
0.640
0.634
0.682
0.632
thyroid
7200
21
0.999
0.999
0
0.992
0.984
0.999
0.985
0.996
tic-tac-toe
958
9
2
1,000
0.999
1.000
0.999
0.998
0.999
0.994
0.999
titanic
2201
2
0.877
0.876
0.805
0.874
0.873
0.876
0.821
0.877
trains
10
29
2
0.778
0.750
0.917
0.833
0.750
0.750
0.625
0.750
twonorm
7400
20
2
0.998
0.999
0.998
0.991
0.991
0.999
0.981
0.973
vertebral Zelases
310
0.977
0.977
0.952
0.942
0.917
0.937
vertebral column. columm.3clases
310
3
0.978
0.977
0,943
0.959
0.933
0.978
0.912
0.953
wall- followigg
5456
0.980
0.980
0.972
0.928
0.949
0.980
0.945
0.933
waveform
5000
21
3
0.968
0.966
0,947
0.947
0.941
0.966
0,866
0.848
waveform--nose
5000
40
3
0.962
0.962
0.935
0.881
0.900
0.962
0.872
0.844
wine
178
3
0.999
0.999
0.998
0.948
0.990
0.999
0.969
0.978
wine-quality-red
1599
11
6
0.735
0.727
0.751
0.705
0.696
0.719
0.695
0.697
wine whiite
4898
0.673
0.663
0.745
0.650
0.655
0.656
0.652
0.611
yeast
1484
8
10
0.756
0.751
0,706
0.718
0.692
0.745
0.601
0.579
Contieued on next page.
18

---

Under review as a conference paper at ICLR 2021
Datasct
N
M
K
RED
MCP
Trust Soore
ConfidNet
Intro-Net
Entropy
DNGO
SVGP
101
16
7
0.999
0.998
0.997
0.928
0.963
0.999
0.963
0.950
Continued from previous page.
Table 7: Comparison between RED and Counterparts Using AUPR-Error
Datasct
K
RED
MCP
Trust Soore
ConfidNet
Intro-Net
Entropy
DNGO
SVGP
abalone
4177
8
3
0.543
0.537
0.478
0,516
0.526
0.527
0.498
0.480
acute-inflammaiion
120
NA
NA
NA
NA
NA
NA
acute-uephriti
120
6
2
NA
NA
NA
NA
NA
NA
NA
NA
adult
48842
0.419
0.410
0.390
D.412
0.410
0.357
anncaling
898
31
5
0.547
0.375
0,440
0.331
0.385
0.350
0,347
0,352
arthythmia
452
262
13
0.573
0.570
0.606
0.396
0.399
0.547
0.412
0.465
aadiology-sti
196
59
18
0.766
0.704
NA
0.401
0.494
0.712
0.564
0.532
balanco-scall
625
3
0.877
0.794
0.501
0.030
0.124
0.685
0.396
0.085
balloons
16
0.785
0.736
0.569
0.736
0.458
bank
4521
16
2
0.391
0.387
0.385
0.367
0.333
0.387
0.554
0.377
hinconcentration
779
0.485
0.461
0.501
0.418
0.433
0,431
0.398
0.376
blood
748
2
0.377
0.375
0,312
0.367
0.357
0.377
0.345
0,348
breast cancer
286
0.518
0.483
0.354
0.447
0.405
0.483
0.421
0.452
meast--anncr-wwiee
699
0.374
0.341
0.347
0.253
0.184
0.341
0.367
0.121
brcnst-canccr-wise-de
569
30
2
0,594
0.561
0.438
0.070
0.348
0.562
0,170
0,100
meeast-cmcer-wise-pe
198
0.310
0.236
breast tissue car
106
9
6
0.795
0.717
0,771
0.660
0.556
0.722
0.717
0.690
1728
0.537
0.419
0.111
0.041
0,413
0.317
0.081
cuntiolocography- 10clases
2126
21
10
0.507
0.486
0.394
0.301
0.295
0.465
0,297
0,284
cundiotocograhhy-ellee
2126
21
3
0.429
0.372
0.377
0.259
0.268
0.369
0.493
0.272
chess-krvk
28056
18
0,550
0.523
0,715
0.485
0,448
0.514
0,386
0.404
cless-krvkp
3196
36
2
0.308
0.299
0,141
0.024
0.067
0.299
0.245
0.036
climate
540
0.081
0.362
0.296
0.110
congressional vooting
435
16
2
0,503
0,503
0.659
0.505
0.543
0.488
0.503
0.511
208
2 0,507
0.487
0.666
0,260
0,305
0.487
0.352
0.264
conn bench vowel deterding
990
1I 0.636
0.455
0.865
0.123
0.144
0.431
0,250
0.152
connect-4
67557
42
2
0,405
0.389
0.326
0.258
O. 0.394
0.389
0.371
0.155
contrac
1473
3
0.591
0.585
0.525
0,546
0.579
0.573
0.569
0.544
credit-sppreval
690
15
2
0,344
0.337
0.316
0.241
0.311
0.357
0.289
0.316
cylinder bands
512
D.406
0.288
dermatology
366
34
6
0.603
0.572
0.573
0.046
0.194
0.607
0.317
0.121
cchocardiogram
131
0.4 0.463
0.350
0,274
0.
0.357
0,350
0,236
0.325
ccoli
336
8
0.409
0.396
0,443
0.244
0.259
0,344
0,251
0.251
energy-yl
768
8
0.442
0.333
0.258
0. 163
0. 165
0.312
0.565
0.211
caargy-y2
768
3
0.481
0.409
0.298
0.260
0.396
0.307
fertility
100
2
0.218
0.200
0,180
0.153
0.165
0.200
0.121
0.098
flags
194
0.760
0.703
0.651
0.649
0.697
0.632
0.647
glass
214
9
6
0.564
0,501
0.637
0,480
0.385
0.462
0.471
0,384
habervan-survival
306
0.426
0.420
0.363
0.389
0.400
0,420
0.354
0.477
bayes-roth
160
0.672
0.560
0.796
0.569
0.537
0,553
0.649
0,510
beart-cleveland
303
13
S
0.752
0.739
0.694
0.675
0. 0.694
0.719
0.612
0.720
beart-bungarian
294
0.406
0.234
0.360
0.317
bearl switzerland
123
12
0.720
0.672
0.711
0.677
0.639
0.680
0.685
0.618
heart-va
200
0.748
0.720
0.664
0.721
0.736
0.621
hopatitis
155
19
2
0.499
0,383
0.435
0,375
0.273
0,383
0.365
0.317
hill- hill-valley
1212
100
2
0.654
0.540
0.472
0.502
0.591
0.541
0.545
0.540
burse-colic
368
25
2
0.422
0.407
0.399
0.245
D.369
0.407
0,289
0.225
ilpd-indian-liver
583
9
2
0.458
0.445
0.410
0.419
0.428
0.445
0.433
0.429
2310
0.229
0.123
ionospbere
351
33
2
0.513
0.452
0.303
0.128
D.364
0.452
0
0.148
iris
150
0.623
0.553
0.684
0. 105
0.1
0.511
0.337
0.189
lod-display
1000
7
10
0,574
0.573
0,566
0,557
0.512
0.548
0,504
0.469
Jenses
24
3
0.833
0.887
0.798
0.572
0.759
0.887
0.530
0.499
letter
20000
26
0.493
0.459
0.784
0.242
0.125
0.463
0.110
0.162
Libras
360
90
15
0.586
0.485
0.639
0.287
0.282
0.449
0.482
0.258
res spect
0.493
0.314
lung- cancer
32
56
3
0.870
0.784
0.609
0.633
0.812
0.836
0.572
0.479
lymphography
148
0.641
0.512
0.582
0.228
0.339
0.514
0.302
0.323
magic
19020
2
0,364
0.353
0,293
0.307
0.348
0.353
0.293
0.232
manmoggaphiic
961
0.458
0.456
0.277
0.399
0.441
D.456
0.402
0.384
messidor
11.51
0.458
0.442
0.350
0.356
0.417
0.442
0.404
0.341
miniboone
1300664
S0
2
0.379
0.361
0,190
0.299
0.370
0.361
0,349
0.162
molec biol. promoter
106
0,506
0.497
0.292
D.436
0.497
0.399
0.396
mnlle--hiol-spiee
3190
60
3
0.457
0.450
0.372
089
0.203
0.459
0.398
0.171
monks-1
556
0,576
0.359
0.126
0.199
0.305
0.359
0.153
0.149
monks-2
601
2
0.512
0.452
0.622
0.405
0.412
0.452
0.376
0,374
monks-3
534
0.603
0.509
0,291
0.269
0.403
0.509
0.450
0.123
mushroom
8124
21
NA
NA
NA
NA
NA
NA
NA
NA
Continued on nex page.
19

---

Under review as a conference paper at ICLR 2021
Datasct
N
M
K
RED
MCP
Trust Soore
ConfidNet
Intro-Net
Entropy
DNGO
SVGP
musk-1
476
166
2
0,482
0.470
0.476
0.081
0.319
0.470
0.273
0.089
musk-2
6598
166
2
0.371
0.339
0.064
0.007
0.061
0.339
0.503
0.006
nursery
12960
0.550
0.388
0.027
0.129
0.386
0.368
0.131
oncytes, merluecius, nncleus, .d
1022
41
2
0.431
0.389
0.264
0.284
0.368
0.389
0.378
0.318
statees.f
1022
25
3
0.486
0.466
0.387
0.234
0.338
0.453
0.304
0.244
912
25
0,470
0.439
0.329
0.262
0.447
0,439
0.349
0.285
oncyles, sunylesstrioptenss www.sooooessssstee Staates__s
912
32
3
0.420
0.394
0.321
0.137
0.266
0.417
0.316
0.101
optical
5620
62
10
0.437
0.411
0,572
0.043
0.037
0,402
0.070
0.067
ozone
2536
72
2
0.288
0.269
0,253
0.189
0.127
0.269
0,516
0.023
page-bloch
5473
0.439
0.374
0,388
0.317
0.263
0.359
0,255
0.173
parkinsons
195
22
2
0.490
0.456
0,477
0.108
0.197
0.456
0,264
0.277
pendigits
10992
16
10
0.480
0.344
0.677
0.168
0.124
0.343
0.102
0.050
phishing
1353
3
0.439
0.416
0.357
0.332
0.355
0.399
0.330
0.322
pima
768
2
0.489
0.480
0.408
0. 0.399
0.477
0.480
0.414
0.322
Hrisbugg-bidges-MATEE
106
3
0.553
0.444
0,405
0. 303
0,459
0,449
0.272
0.336
Biidburg-riigee-RELLLL
103
7
3
0.565
0.498
0.624
0.485
0.519
0.491
0.464
0.452
bittshug-bridges-SPP
92
0,536
0.504
0.494
0.319
D.461
0.521
0.416
0.412
pittsburg-brigge----o0
102
2
0.388
0.299
0.376
0.320
0.462
0.299
0.298
0.274
105
7
6
0.797
0.713
0.741
0.658
0.634
0.664
0.668
0.716
plamning
182
0.339
0.337
0.362
0.315
0.295
0.337
0.281
0.229
plant-margin
1600
64
100
0.580
0.565
0.692
0.323
0.258
0.562
0.297
0.1
plant-shape
1600
64
100
0.690
0.662
0.757
0,519
0.475
0.639
0.513
0.472
plani-texture
1599
64
100
0.602
0.590
0.704
0.300
0.323
0.605
0.275
0.216
post-operative
90
3
0,391
0.375
0.230
0.529
0.368
0.334
0.398
0.368
primary-tunee
330
17
15
0.789
0.777
0,683
0.717
0.712
0.772
0.673
0.704
ringnorm
7400
20
0.334
0.329
0.269
0.040
0.111
0.329
0.285
0.025
seeds
210
0.515
0.447
0.453
0,282
0.284
0.406
0.340
0.314
somcion
1593
256
10
0,530
0,514
0.698
0.083
0.224
0.554
0.166
0.132
soybean
683
18
0.546
0.476
0.282
0.164
0,469
0.289
spumbase
4601
57
2
0.307
0.282
0,290
0.170
0.228
0.282
0,211
0.131
spect
265
22
2
0,503
0.487
0.381
0.425
0,451
0.487
0.397
0.419
speet!
267
44
2
0,447
0.386
0.345
0.216
0.392
0.387
0,286
0.304
stattog mustralian credit
690
0.465
0.446
0,408
0,408
0.406
0.446
0.395
0.421
statlog-german-cnn german-credit
1000
24
2
0,413
0.410
0.373
0.320
0.378
0,410
0,352
0.244
statlog-bcart
270
13
2
0.443
0.406
0.351
0,288
0.435
0.406
0.329
0.255
statlog-image
2310
0.541
0.220
0.100
0.260
0.166
statlog landsat
6435
36
6
0.425
0.393
0.551
0.317
0.306
0.404
0.291
0.251
statlog- shuttlle
58000
9
7
0.534
0.294
0.527
0.214
0.183
0.281
0.347
0.229
aatlog-vehice
846
18
4
0.531
0.515
0.304
0.376
0.347
0.524
0.335
0.350
steel plates
1941
27
7
0.523
0.511
0.528
0.446
0,430
0.508
0.422
0.397
synthetic-control
600
60
0,590
0.453
0.396
0.031
0.076
0,402
0,209
0.020
teaching
151
5
3
0.626
0.578
0,503
0.614
0.568
0.586
0.653
0.544
thyroid
7200
21
0.428
0.116
0.178
0.101
tic- toe
958
9
2
0.562
0.473
1,000
0.132
0.461
0.473
0.423
0.418
titanic
2201
2
0.386
0.385
0.599
0.372
0.386
0.385
0.437
0.370
trains
10
29
2
0.667
0.625
0.875
0.750
0.625
0.625
0.438
0.625
twonorm
7400
20
2
0.359
0.345
0.394
0.1
0.278
0.345
0.270
0.034
vertebral-colunn22oll
310
0.468
0.456
0,302
0.345
0,402
0.456
0.349
0.243
310
6
3
0.505
0.478
0,279
0.460
0.366
0.495
0.411
0.271
wall following
5456
0.373
0.299
0.204
0.243
0.364
0.263
0.135
waveform
5000
21
3
0.462
0.419
0.327
0.353
0.390
0.419
0.537
0,159
waveform-noise
5000
40
3
0.424
0.394
0.318
0.227
0.322
0.394
0.511
0.165
wine
178
0.857
0.669
0.606
0.056
0.413
0.794
0.602
0,181
wine-quality-red
1599
11
6
0,530
0.518
0.537
0.489
0.486
0.491
0.519
0.448
wine-qualitywwiie yeast
4898
0.539
0.625
0.532
0.521
0.523
0.478
1484
8
10
0.565
0,557
0,506
0.502
0.493
0.542
0.405
0.911
200
101
0.888
0.744
0.752
0.183
0.473
0.842
0.385
0.657
Continued from previous page.
Table &: Comparison hetween RED and Counterparts Using AUPR-SSuceess
Datasct
N
M
K
RED
MCP
Trust Soore
ConfidNet
Intro-Ner
Entropy
DNGO
SVGP
ahalome
4177
8
3
0.852
0.850
0.821
0.844
0.844
0.849
0,806
0.840
acute-inlammattion
120
2
NA
NA
NA
NA
NA
NA
NA
acute-nopharits
120
2
NA
NA
NA
NA
NA
NA
NA
NA
adult
48842
14
2
0.968
0.967
0.944
0.964
0.961
0.967
0.897
0.961
annealing
898
31
5
0.975
0.903
0,942
0.907
0.914
0.899
0.908
0.919
arthythmia
452
262
13
0.814
0.806
0
0.645
0.659
0.799
0.665
0.775
audiology-au std
196
0.945
0.609
0.725
0.741
0.769
halance-scale
625
4
3
0.999
0.999
0.997
0.971
0.983
0.999
0.986
0.976
balloons
16
2
0.773
0.736
0.861
0.690
0.736
0.736
0,516
0.542
bank
4521
16
2
0.977
0.980
0,976
0.968
0.936
0,980
0.946
0.979
biscuncentration
779
3
0.742
0.723
0.779
0.712
0.722
0.716
0.687
0.693
bload
748
0.891
0.891
0.872
0.901
0.882
0.891
0.881
0.899
Continued on ned page.
20

---

Under review as a conference paper at ICLR 2021
Datasct
N
M
K
RED
MCP
Trust Soore
ConfidNet
Intro-Net
Entropy
DNGO
SVGP
hreast-cancer
286
2
0.815
0.814
0.757
0.788
0.767
0.814
0.758
0.777
699
9
2
0.998
0.997
0.998
0.994
0.984
0.997
0.992
0.986
breast cancer seneeerwwis--iie wisc-diag
569
0.996
0.997
0.997
0.969
0.980
0.998
0,970
0.981
ceast-cancer-wiss-prre
198
33
2
0.850
0.855
0.787
0.817
0.828
0.855
0.779
0.785
breast-lissue
106
9
6
0.902
0.869
0.893
0.855
0.751
0.863
0.782
0.838
1728
4
0.989
0.959
0,939
0.992
0.988
0.999
0,989
0.989
cundioleeography-10aae -00llsss 10clases
2126
21
10
0.959
0.959
0.927
0.889
0.871
0.958
0.884
0.901
cardiotooograhhy-llles
2126
21
3
0.990
0.991
0,988
0.971
0.963
0.991
0.963
0.979
chess-krvkk
28056
6
18
0.900
0.890
0.934
0.867
0.837
0.889
0.782
0.801
chess-krvkp
3196
2
1,000
1,000
0,995
0.990
0.994
1.000
0.972
0.996
climate
540
20
2
0.991
0.989
0,976
0.937
0.975
0.986
0.958
0,949
congressionallvoiing
435
16
2
0.701
0.701
0.706
0.704
0.694
0.701
0.713
0.698
conn bench sonar mines rocks
208
60
2
0.961
0.963
0,980
0.806
0.898
0.963
0,852
0.894
comn-hench-vood sechh--owlllleee vowel-deterding
990
11
II
1,000
1.000
1,000
0.979
0. 0.984
1.000
0.991
0.989
connect-4
67557
42
0,969
0.969
0,938
0,942
0.958
0.969
0,883
0.867
contrac
1473
9
3
0.716
0.713
0,578
0.680
0.672
0.704
0.654
0.667
credit-approval
690
0.942
0.941
0,915
0.
0.936
0.941
0.888
0.927
eylinder-bands
512
35
2
0. 864
0.860
0,870
0.784
0.804
0,860
0,769
0.752
dermstology
366
34
6
0.999
0.999
0.991
0.975
0.980
0.999
0.976
0,950
echocardiogram
131
2
0.941
0.934
0,922
0.913
0.920
0.934
0
0.925
ccoli
336
7
8
0.958
0.955
0.969
0.908
0.874
0.953
0.881
0,916
cuergy-yl
768
3 0,992
0.993
0,992
0.975
0.973
0.993
0,960
0.983
energy-y2
768
3
0.986
0.986
0
0.961
0.945
0.986
0.960
0.978
fortility
100
9
0.951
0.923
0,946
0.914
0.924
0.923
0.893
0.936
flags
194
28
8
0.731
0.707
0,812
0.551
0.599
0.710
0,560
0.585
glass
214
9
6
0.850
0.804
0.912
0.767
0.725
0.802
0.729
0.775
Sahermnanssurvvve
306
2
0.844
0.843
0.851
0.815
0.828
0.843
0,813
0.829
hayes-roth
160
3
3
0.932
0.880
0,965
0.882
O. 0.854
0.885
0.864
0.821
beart-cleveland
303
5
0.853
0.856
0.792
0.818
0,858
0.835
beart-hungarian hunguriam
294
12
2
0.923
0.905
0,926
0.841
0.912
0.905
0.838
0.862
beart-switzerland
123
12
5
0.464
0.440
0.429
0.399
0,414
0.448
0.435
0.408
beart-va
200
12
5
0,500
0.470
0.438
0.413
0.480
0,483
0,390
0.364
hepatitis
155
19
0.964
0.964
0.974
0.925
O 0.888
0.964
0.891
0.910
hill-valley
1212
100
2
0.761
0.702
0.554
0.650
0.739
0.702
0.702
0.690
borse-colic
368
25
2
0.926
0,916
0,917
0.827
0.911
0.916
0.825
0.869
uldd-indian--ive
583
0.878
0.873
0.859
0.862
0.873
0.869
image segmentatiion
2310
18
7
0.997
0.997
0
0.983
0.984
0.997
0.977
0.985
ionesphere
351
33
2
0.990
0.988
0.976
0.897
0,973
0.988
0.928
0,933
iris
150
3
0,992
0.991
0,997
0,935
0.912
0.990
0.945
0.954
led display
1000
7
1D
0.888
0.888
0.773
0.877
0.849
0.885
0,824
0.845
Jenses
24
3
0.979
0.947
0,944
0
0.845
0.947
0.818
0.815
letter
20000
16
26
0.995
0.996
0,999
0.980
0. 8888
0.996
0.964
0.982
libras
360
15
0.955
0.942
0.802
0.800
0,940
0.839
0.848
low res- spect
531
100
9
0.977
0.968
0,952
0.905
0.889
0.968
0.898
0.949
lung-canoer
32
56
3
0.728
0.687
0.572
0.461
0.703
0.688
0.343
0.356
lymphography
148
18
4
0.939
0.922
0,956
0.797
0.851
0.921
0,856
0.817
magic
19020
10
2
0.966
0.966
0.954
0.960
0 0.959
0.966
0.838
0,946
manmographic
961
0.922
0.918
0.881
0,909
0.910
0.918
0.865
0.925
messidor
1151
19
2
0.862
0.861
0.762
0.805
0.829
0.861
0.795
0.805
miniboone
130064
0,992
0.992
0
0
0.992
0.933
0.981
molec-hiol proonnpr
106
57
2
0.886
0.882
0,951
0.702
0.873
0.882
0.766
0.811
molec-biol-splicc
3190
3
0.953
0.953
0.926
0.832
0.845
0.952
0.892
0.837
monks- I
556
2
0.998
0.997
0,982
0.988
0.989
0.997
0.983
0.995
monks-2
601
2 0.884
0.855
0.925
0.806
0 807
0.855
0.751
0.806
monks-3
554
0.999
0.999
0.992
0.980
0.993
0.999
0.977
0.991
mushroom
8124
21
2
NA
NA
NA
NA
NA
NA
NA
NA
musk-1
476
0,984
0.983
0
0.910
0.956
0.983
0.909
0.934
musk-2
6598
166
2
1.000
1.000
0.998
0.995
0.995
1.000
0.997
0.994
oursery
12960
5
1,000
1.000
1.000
0.999
0.999
1.000
0.999
1.000
oncytes merluscius nucleus 4d
1022
41
2
0.939
0.929
0.831
0.891
0.903
0.929
0.831
0.907
Stttuecciusssstts.222 2f
1022
25
3
0.991
0.991
0.991
0.967
0.970
0.991
0.952
0.975
oocytes. wrw.sstsssoeeuuuuuuuuee 2f
912
0.939
0.931
0.907
0.870
0.922
0.838
0.874
oncytes, wwt.soopeeussslattssss... Stb
912
32
3
0.991
0.990
0.985
0.951
0.967
0.990
0.932
0.945
optical
5620
0.999
0.999
0,999
0
0.978
0.999
0,980
0.992
ozone
2536
72
2
0.994
0.996
0,993
0.988
0.966
0.996
0.984
0.951
page-blocks
5473
10
S
0.998
0.997
0.993
0.993
0.989
0.997
0.966
0.994
parkinsons
195
22
2
0.994
0.992
0.995
0.936
0.942
0.992
0.960
0.963
pendigits
10992
16
10
0.999
0.999
1.000
0.992
0.992
0.999
0.994
0.993
phishing
1353
3
0.975
0.973
0.959
0.945
0.919
0.952
pima
768
2
0.897
0.893
0,873
0.845
0.871
0.893
0.839
0.869
pirtshurg- wwridges-MATERAAL
106
3
0.975
0.968
0,950
0.916
0.928
0.968
0.881
0.936
pittsbugstrrdges-RELLL
103
7
3
0.776
0.766
0.832
0.741
0.725
0.769
0.715
0.755
pitisburg- Seidues--AddggSAPAAA
92
7
3
0.842
0.793
0.766
0.649
0.734
0.790
0.767
0.695
brobuug-brigge---OORDD
102
2
0.956
0.941
0.919
0.924
0.945
0.941
0.931
0.916
Sutsburg-biidgs-TYPP
105
7
6
0.805
0.658
0.718
0.612
0.596
0.631
0.713
0.748
planning
182
0.715
0.709
0.723
0.714
0.713
0.709
0.720
0.677
plant- mart-n
1600
64
100
0.951
0.949
0.966
0.827
0.816
0.950
0.815
0.729
Contiened on next page.
21

---

Under review as a conference paper at ICLR 2021
Datasct
N
M
K
RED
MCP
Trust Soore
ConfidNet
Intro-Net
Entropy
DNGO
SVGP
plant-shupe
1600
64
100
0.860
0.849
0.879
0.715
0.668
0.837
0.693
0.669
pluni-texture
1599
64
100
0.953
0.950
0,958
0.807
0.780
0.950
0.786
0.790
post- operatpeee
90
0.653
0.642
0.719
0.656
0.660
0.639
0.702
0.654
primary-tumor
330
17
15
0.757
0.746
0,646
0.627
0.623
0.745
0,531
0.629
ringnorm
7400
20
2
0.999
0.999
0.989
0.982
0.977
0.999
0.959
0.976
seeds
210
0.987
0.986
0,991
0.962
0.960
0.985
0.929
0.945
semeiion
1593
256
10
0.994
0.994
0
0.926
0.949
0.994
0.943
0.963
soyhean
683
35
18
0.996
0.995
0.993
0.970
0.940
0.995
0,942
0.973
spambase
4601
57
2
0.989
0
0,983
0.965
0.966
0
0.941
0.974
spect
265
2
0,805
0.795
0.762
0.721
0.783
0.795
0.673
0.724
speetf
267
44
2
0.954
0.951
0,941
0.911
0.917
0.952
0,880
0,948
690
14
2
0.696
0.687
0.649
0.662
0.672
0.687
0.621
0.657
statlog german credit
1000
24
0.889
0.890
0,872
0.835
0.848
0.890
0,771
0.779
statlog-heart
270
13
2
0.939
0.934
0.904
0.878
0.914
0.934
0.829
0.860
statlog-image
2310
0.998
0.998
0,998
0,986
0.979
0.998
0,984
0.990
statlog-landiat
6435
36
6
0.986
0.987
0,991
0.963
0.959
0.987
0.948
0.971
statlog- shuttle
58000
9
7
1,000
1,000
1,000
0.998
0.907
0.999
0.999
0.999
statlo--vehclle
846
18
4
0.965
0.967
0.927
0.928
0.912
0,968
0,868
0.921
plattss
1941
27
7
0.919
0.916
0.928
0.881
0.848
0.915
0.824
0.870
synthetic yonhetic-contml
600
60
1,000
1.000
0.999
0.985
0.988
1.000
0.987
0.986
teaching
151
5
3
0.668
0.625
0.569
0.630
0.625
0.611
0.662
0.615
thyroid
7200
21
3
0,999
0.999
0,996
0.992
0.984
0.999
0.992
0.996
tic-tac-toe
958
9
2
1,000
0.999
1,0000
0.999
0.998
0.999
0.994
0.999
titanic
2201
2
0.895
0.894
0.889
0.890
0.891
0.894
0,878
0.895
trains
10
29
2
0.667
0.625
0,875
0.750
0.625
0.625
0,438
0.625
twonomn
7400
20
0.998
0,999
0.998
0.991
0.991
0.999
0.924
0,973
cittebra-coluum- 2elases
310
0.977
0.977
0.952
0.952
0.940
0.977
0.894
0.936
vertebral-column- colulm-ou-ssssssess
310
3
0.977
0.977
0,943
0.958
0.931
0.977
0.908
0,952
wall-following
5456
24
0,980
0.980
0,928
0.949
0.932
waveform
5000
21
3
0.968
0.966
0,947
0.947
0.940
0.966
0.928
0.847
waveform-noise wine
5000
40
3
0.962
0.962
0.935
0.880
0,900
0.962
0.926
0.844
178
13
0.999
0.999
0,998
0.945
0.990
0,999
0.968
0.978
wine quality- red
1599
11
6
0.734
0.726
0.750
0.702
0.692
0.717
0.692
0.695
wine-quality-whic. yeast
4898
11
7
0.672
0.662
0.745
0.648
0.654
0.655
0.650
0.610
1484
8
10
0.754
0.749
0,704
0.716
0. 0.689
0.742
0.598
0.571
20O
101
0.999
0.997
0.919
0.958
0.999
0.958
0.943
Continued from previous page.
Table 9: Compurison between RED and Counterparts Using AUROC
Dataset
N
M
K
RED
MCP
Trust Score
ConfidNet
Intro
Entropy
DNGO
SVGP
abalone
4177
8
3
0.734
0.730
0.680
0.719
0.726
0.726
0.697
0.700
acute inflammation
120
NA
NA
NA
NA
ncute-nephritis
120
6
2
NA
NA
NA
NA
NA
NA
NA
NA
adult
48842
2
0.841
0.837
0.759
0.828
0,830
0.837
0.751
0.814
annealing
898
31
5
0.865
0.699
0,776
0.724
0.723
0.694
0.619
0.675
arthythmia
452
262
13
0.712
0.715
0.727
0,505
0.517
0.703
0.528
0.639
audiology-std
196
18
0.883
0.439
0.6U5
0.666
0.672
balance-scal
625
3
0.986
0.987
0,909
0.510
0.621
0.985
0.712
0.536
halloons
0.741
0.667
0.556
0.667
0.667
0.312
0.333
bank
4521
16
2
0.854
0.861
0,840
0.822
0.750
0.861
0,500
0.854
bioconoentration
779
3
0.621
0.607
0.660
0.572
0.595
0.588
0.547
0.541
blood
748
2
0.705
0.705
0.634
0.716
0.687
0.705
0.665
0.704
breast-cancer
286
9
2
0.682
0.677
0.587
0.637
0.609
0.677
0.607
0.620
breast-cancer-wil
699
0.944
0.936
0.873
0.759
0.797
0.727
breast cancer wisc diag
569
30
2
0.925
0.925
0.921
0.558
0.682
0.940
0.551
0.575
198
0.640
0.644
0.521
0.
0.595
0.644
0.514
0.484
breast-lissue
106
9
6
0.864
0.815
0,850
0.788
0.651
0.811
0,761
0.785
car
1728
4
0.974
0.968
0.934
0.717
0.615
0.968
0.653
0.691
candioloecoraphyy 10clases
2126
21
10
0.839
0.835
0.756
0.670
0.622
0.829
0.660
0.663
cardiotooograbby-Slaee chess-Krv.
2126
21
3
0.902
0.901
0.882
0.804
0.761
0.900
0.563
0.813
28056
18
0.778
0.759
0.722
0.685
0.753
0.642
chess-krvkp
3196
36
2
0.967
0.963
0.777
0.573
0.642
0.963
0.568
0.688
climate
540
20
2
0.898
0.858
0.760
0.449
0.744
0.854
0.554
0.572
congressional-voiia
435
16
2
0.603
0.600
0,562
0.607
0.595
0.598
0.619
0.612
conn- comn-berch--oone sonar rocks
208
60
2
0.833
0.833
0.912
0.488
0.643
0.833
0.634
0.625
conn bench. vowel deterding
990
11
1I
0.950
0.979
0.995
0.582
0.624
0.979
0.716
0.759
connect-4
67557
42
2
0.836
0.831
0.744
0.727
0.816
0.831
0.736
0.522
contrac:
1473
0.663
0.658
0.556
0.619
0.642
0.626
0.608
credit-approval
690
1.5
2
0.741
0.741
0.683
0.649
0.703
0.741
0.625
0.694
cylinder-bands
512
35
2
0.703
0.702
0.713
0.579
0.637
0.702
0.561
0.516
dermatology
366
34
0.980
0.974
0,934
0.589
0.766
0.976
0.750
0.333
echocandiogram
131
10
0.758
0.709
0.655
0.634
0.687
0.709
0.597
0.659
eanli
336
0.791
0.785
0,806
0.670
0.556
0.773
0.592
0.667
Continued on nex page.
22

---

Under review as a conference paper at ICLR 2021
Datasct
N
M
K
RED
MCP
Trust Soore
ConfidNet
Intro-Net
Entropy
DNGO
SVGP
energy-yl
768
8
3
0.896
0.886
0.848
0.691
0.650
0.886
0.604
0.777
energy-y2
768
8
3
0.892
0.880
0.874
0.777
0.668
0.880
0.802
0.834
fertility
100
0.588
0.469
0.563
0.382
0.485
0.469
0.413
0.508
flags
194
28
8
0.727
0.696
0,791
0.603
0.618
0.698
0.598
0,574
glass
214
9
6
0.719
0.659
0.823
0.648
0.560
0.651
0.603
0.581
nubernan-surrival
306
2
0.694
0.692
0.651
0.645
0.671
0.692
0,636
0.684
hayes roth
160
3
3
0.846
0.746
0.909
0.785
0.749
0.759
0.811
0.674
heart-eleveland
303
0.806
0.811
0,783
0.758
0.769
0.811
0.680
0.786
bearl-bungarian
294
12
2
0.750
0.724
0,742
0.555
0.730
0.724
0,599
0.650
switzzrwizz
123
0.584
0.545
0,580
0.545
0.496
0.554
0.554
0.499
beart-va
200
12
5
0.629
0.606
0,544
0.522
0.603
0.618
0,505
0.455
bepatiti
155
19
2
0.822
0.797
0.845
0.703
0.619
0.797
0.624
0.677
hill vall-y
1212
10D
0.712
0.636
0.512
0.585
0.669
0.636
0.636
0.627
horse- colic
368
25
0.743
0.731
0.749
0.530
0.717
0.731
0.563
0.568
ipddindian--iver
583
0.719
0,709
0.650
0,686
0.695
0,709
0.614
0,709
maage-egmenuaaion
2310
18
7
0.935
0.936
0,946
0.770
0.742
0.935
0.734
0.749
ionosphere
351
33
2
0.909
0.886
0.779
0.504
0.786
0.886
0.630
0.535
iris
150
3
0.942
0.928
0,963
0.553
0.471
0.926
0.693
0.612
led-display
1030
7
10
0.762
0.762
0.545
0.749
0.717
0.753
0.706
0.700
lenses
24
3
0.917
0.875
0.863
0.677
0.740
0.875
0.625
0.583
letter
20000
16
26
0.932
0,940
0.977
0. 0.795
0,526
0.940
0.646
0.775
libras
360
15
0.845
0.865
0.524
0.535
0,803
0.681
0.562
Low-res-spec
531
100
9
0.864
0.837
0.791
0.618
0.559
0.836
0.616
0.717
lung-canoer
32
56
3
0.759
0.686
0.573
0.444
0.700
0.725
0.378
0.304
lymphography
148
18
4
0,805
0.753
0,825
0.503
0.622
0.748
0.623
0,566
magic
19020
10
0.810
0.807
0.753
0.788
0.796
0.807
0.639
0.720
mammographiic
961
2
0.764
0.761
0.650
0.729
0.742
0.761
0.693
0.755
messidor
1151
19
2
0.697
0.690
0.565
0.609
0. 0.662
0.690
0.646
0.601
miniboone
130064
0,905
0.903
0.878
0
0,903
0.783
molec-bioo protnoter
106
57
2
0.735
0.727
0
0.414
0.719
0.727
0,505
0.576
molcc-biol-splic
3190
60
3
0.810
0.810
0.722
0.517
0.547
0.811
0.717
0.501
monks-1
556
2
0.959
0.936
0.619
0.715
0.779
0,916
0.597
0.862
monks 2
601
0.751
0.698
0.814
0.616
0.632
0.698
0.602
0.598
monks.3
554
2
0.972
0.966
0.845
0.750
0.872
0.966
0.820
0.825
mushroom
8124
21
2
NA
NA
NA
NA
NA
NA
NA
NA
musk-1
476
U.863
0.860
0.863
0.449
0.724
0.503
musk-2
6598
166
2 0.984
0
0.794
0.513
0.627
0,982
0,500
0.417
nursery
12960
5
0.993
0.955
0.925
0.601
0.603
0.965
0.674
0.832
oncyles werpleesmeruuciuu.oo. mucleus..44 .d
1022
41
2
0.779
0.758
0.627
0.652
0.717
0.758
0.708
0.693
oncyles merluccius states 2
1022
25
3
0.907
0.909
0.907
0.763
0.795
0.908
0.706
0.793
oncytes, trisopterus.mcceee nucleus. 2f
912
25
0.786
0.766
0.693
0.619
0.759
0.766
0.665
0.638
states_5b
912
32
3
0.898
0.895
0.834
0.638
0.744
0.895
0.704
0,546
optical
5620
0,970
0.964
0,561
0.534
0.573
0.703
ozone
2536
72
2
0.899
0.905
0,851
0.773
0.568
0.906
0.500
0.315
page- blockblocks
5473
10
5
0.952
0.942
0.905
0.906
0.855
0.941
0.792
0.861
parkinsons
195
22
0.918
0.881
0,928
0.574
0.561
0.881
0.729
0.666
pendigits
10992
16
10
0.951
0.948
0.972
0.681
0.635
0.947
0.650
0.660
phishing
1353
3
0.859
0.852
0.788
0.798
0.762
0.847
0.726
0.766
pima
768
8
2
0.754
0.751
0,700
0.673
0.730
0.751
0.697
0.653
pittsburg bridges MATERIAL
106
0.848
0.771
0,586
D.706
0.772
0.692
Srabbugg-hridgss-a2 -T
103
3
0.643
0.611
0.739
0.612
0.587
0.614
0.569
0.604
pirtsburg-bridees-SPP
92
3
0.
0.634
0.631
0.403
0.574
0.631
0.579
0.488
pittshurg TOR
102
2
0.778
0.740
0.694
0.669
0.750
0.740
0.712
0.653
pittsuug-bridges--YV
105
7
6
0.799
0.713
0.743
0.657
0 0.636
0.673
0.723
0.728
planning
182
0.479
0.468
0.492
0.473
0.468
0.474
0.400
plant-mann mungin
1600
64
100
0.846
0.844
0.888
0.605
0.559
0.844
0.583
0
shupe
1600
100
0.794
0.780
0.647
0.587
0.760
0.626
0.586
plant-texture
1599
64
100
0.854
0.848
0,883
0,580
0.560
0.850
0.534
0.491
post-operative
90
3
0.443
0.435
0.414
0.433
0,446
0,410
0,517
0.480
primary-tumen
330
15
0.769
0.765
0.676
0.689
0.673
0.764
0.621
0.689
ringnorm
7400
20
2
0.953
0.952
0,708
0.587
0. 0.565
0.952
0.623
0.507
seeds
210
3
0.913
0.890
0.746
0.724
0,881
0.717
0.610
semeiio
1593
256
10
0.930
0.926
0,953
0.499
D. 0.662
0.928
0.615
0.657
soyhean
683
0.945
0.942
0,916
0.765
0.598
0.937
0.614
0.807
spanbase
4601
57
2
0,864
0.856
0.809
0.710
0.724
0.856
0.692
0.713
spect
265
22
2
0. 0.668
0.660
0.581
0.562
0.620
0.660
0.524
0.558
speetf
267
44
2
0.798
0.788
0,740
0.650
0.727
0.789
0.621
0,760
aadlog-australinn-ccedit
690
14
2
0.585
0.569
0.543
0.537
0.539
0.569
0.510
0.525
statog--erman--ce german- credit
1030
0.715
0.720
0.620
0.676
0.636
0.513
statlog heart
270
13
2
0.762
0.746
0.693
0.624
0.731
0.746
0.580
0.565
statlgg-image
2310
7
0.943
0.947
0,941
0.818
0.683
0.947
0.777
0.784
staclog-landsat
6435
36
6
0.888
0.887
0,919
0.795
0.770
0.889
0,755
0.787
matlog-shuttle
58000
9
7
0.995
0.919
0.998
0.763
0.589
0.911
0.599
0.778
statlog-vehicle
846
0.854
0.856
0.719
0.759
0.716
0.858
0.682
0.730
steel-plates
1941
27
7
0.795
0.790
0.805
0.736
0.695
0.787
0.688
0.692
synihetic-contul)
600
0.984
0.980
0.956
0.497
0.547
0.977
0.671
0.497
teaching
151
5
0.645 3
0.596
0.555
0.618
0.574
0.585
0.662
0.553
Continned on next page.
23

---

Under review as a conference paper at ICLR 2021
Dotaset
N
M
K
RED
MCP
Trust Score
ConfidNet
Intro-Net
Entropy
DNGO
SVGP
thyroid
7200
21
3
0.970
0.963
0.849
0.782
0.646
0.968
0.500
0.809
ticttec-toe
958
9
2
0.986
0.933
1,000
0.876
0.894
0.933
0.684
0.942
0.671
trains
10
29
2
0.556
0.500
0.833
0.667
0.500
0.500
0,250
0,500
twonorm
7400
0.952
0.952
0.941
0.817
0.812
0.952
0.662
0.499
vertebral-colume 2elases
310
0.860
0.857
0.770
0.771
0.765
0.857
0.735
0.707
vertebral- column Jcluses
310
0.877
0.873
0.715
0.819
0.710
0.877
0.726
0.750
wall-following
5456
24
0.850
0.847
0.793
0.641
0.706
0.846
0.700
0.603
waveform
5000
21
0.846
0.834
0,761
0.773
0.791
0.833
0.563
0,500
waveform-noise
0.692
0.505
wine
178
13
0.991
0.975
0,954
0.345
0.799
0.980
0,751
0.646
wine-quality-eed
1599
11
0.647
0.638
0.644
0.615
0.6U7
0.617
0.625
0.573
wine quality-white
4898
11
0.621
0.612
0.677
0.604
0.605
0.597
0.600
0.553
yeast
1484
10
0.683
0.678
0.620
0.627
0.611
0.665
0.486
0.792
7
0,986
0.970
0.958
0.689
0.779
0.987
0.594
0.783
Continued from previous page.
Table 10: Comparison between RED Counterparts Using AP-Error
Dutaset
K
RED+BNN
BNN MCP
BNN Entropy
RED+MC-D
MC-D MCP
MC-D Entropy
abalone
4177
8
3
0.529
0.522
0.513
0.516
0,509
0.503
acute-inlammation
120
NA
acule nephritis
120
2
NA
NA
NA
NA
NA
NA
adult
annealing
898
31
0,788
0.424
0.338
0.590
0.421
0.375
arthythmii
452
262
13
0.719
0.614
0.734
0.551
0.543
0.540
mudiology-std
196
59
0,768
0.684
0.756
0.760
0.659
0.698
balance-scale
625
3
0,820
0.532
0,581
0.685
0,548
0.621
balloons
bank
4521
16
0.527
0.443
0.448
0.431
0,410
0.413
hinconcentration
779
0.487
0.486
0.418
blood
748
0.515
0.471
0.498
0.441
0,441
0.443
hreast-cancer
286
0.504
0.396
0.421
0.518
0.486
0.497
bccast-cance-wwise
699
0.473
0.447
0.454
0.425
0.418
reeut-cancer-wis-eee
569
30
2
0.430
0.362
0.446
0.545
0.476
0.543
cancer wisc prog
0.491
hreast-fissue
106
0.847
0.777
0.797
0.885
0.841
0.832
car
1728
0.604
0.495
0.339
0.656
0,480
0.326
Bunditoocorrabyy1Oeke
2126
21
10
0,548
0.515
0.463
0.529
0,488
0.464
cadiooooography--dlee
2126
21
3
0.499
0.424
0.421
0.468
0.400
0.40S
cbess-krvk
28056
18
0.501
0.474
0.675
0.648
cbess- chess-krvkp
3196
36
2
0.392
0,369
0.385
0.443
0.417
0.424
climate
0.472
congressionall-vviin
435
0,517
0.466
0.472
0.486
0.485
0.496
hoa--beuch--omar-miiie---o
208
0.497
0.487
0.509
0.537
0.555
conn- conn-bench- vewbech--vool-deedddgg
990
11
0.672
0.466
0.365
0.683
0,359
0.471
connect-4
67557
42
2
0.408
0.402
0.401
0.490
0,410
0.407
contrac
1473
0.622
0.594
credit approval
690
15
0.388
0.359
0.365
0.414
0.402
0.409
Hainder-hands
0.422
dermatology
366
34
0.633
0.637
0.510
0.757
0.726
0.645
cchocardiogram
131
0.440
0.407
0.443
0.533
0.488
0.507
ecoli
336
0.520
0.484
0.496
0.416
0,410
0.411
caargy-yl
768
8
0.868
0.419
0.367
0.571
0.388
0.394
energy-y2
fertility
100
0.213
0.178
0.205
0.329
0,299
0.301
flags
194
0,746
glass
214
0.664
0,511
0.499
0.541
0,480
0.476
hubernan-suurival
306
0.476
0.337
0.349
0.492
0.489
0.482
noth
160
0.671
0.568
0.560
0.713
0.572
0.594
cleve-dd
303
13
5
0,772
0.744
0.760
0.709
0.687
0.695
heart-hungarian
0.463
0.411
heart- switzeradd
123
12
0.755
0.700
0.656
0.647
0.619
0.585
heart-va
200
0.812
0.828
0.715
0.733
hepatitis
155
19
2
0.610
0.559
0.585
0.616
0,582
0.564
hill valley
1212
100
0.513
0.497
0.533
0.570
0.565
0.548
horse colic
368
25
0.393
0.343
0.365
0.379
0.372
0.397
ilpd-indian-liver
$83
9
2
0.499
0.395
0.428
0.481
0.470
0.472
image segmentation
0.371
0.406
ionosphere
351
33
0.367
0.316
0.448
0.436
0.360
0.394
iris
150
3
0.821
0.675
0.466
0.694
0.705
dod-display
1030
10
0,577
0.566
0.521
0.574
0,571
0.523
lemses
24
3
0.805
0.662
0.674
0.979
0.875
0.854
letter
20000
26
0.461
0.455
0.401
0.708
0.627
0.563
Continued ned page.
24

---

Under review as a conference paper at ICLR 2021
Dotaset
N
M
K
RED+BNN
BNN MCP
BNN Eatropy
RED+MC-D
MC-D MCP
MC-D Entropy
libras
360
90
15
0.698
0.587
0.507
0.584
0.433
0.377
low-ren res spect
531
100
9
0.623
0,545
0.604
0.575
0,541
0.563
lung cancer
tymphography
148
18
4
0,456
0.427
0.412
0.507
0.492
0.467
magic
19020
0.358
0.354
0.356
0.358
0.350
0.353
mammoographic
961
0.444
0.434
0.425
0.435
0,431
0.432
messidor
1151
0.472
0.455
0.443
0,414
0.401
0.399
miniboone
130064
50
0.373
0,360
0.360
0.374
0,364
0.366
molee-bio--romooter
106
57
2
0.642
0,520
0.569
0.560
0.491
0.500
smlle--boo-splicc
3190
0.399
monks-1
556
2
0,853
0,402
0.430
0.811
0,538
0.542
monks-2
601
0.896
0.389
0.429
0.527
0.416
0.427
monks.3
554
0.626
0.486
0.485
0.661
0.519
0.515
mushroom
8124
21
NA
NA
NA
NA
NA
musk-1
476
0.651
0.481
0.549
0.454
0.421
0.382
musk-2
6598
166
2
0,540
0.389
0.447
0.456
0,351
0.406
nursery
12960
0.235
0.306
eocytes.merluecte nucleus.4d
1022
41
0,813
0.487
0.470
0.407
0,387
0.387
states.21
1022
25
0.464
0.404
0.358
0.449
0.426
0.468
oncyles, www../ooooeesspneeeeeee nucleuss
912
25
0.572
0.443
0.436
0.384
0.370
0.386
912
32
0.481
0.374
0.420
0.408
0.382
0.383
optical
5620
0.307
0.313
0.363
ozone
2536
72
2
0,321
0.240
0.285
0.350
0,351
0.337
page-blocks
5473
0.446
0.365
0,320
0.354
parkinsons
195
22
0.785
0,562
0.533
0.538
0,453
0.509
pendigits
10992
10
0.328
0.309
0.314
0.397
0.371
0.281
phishing
1353
0.604
0.451
0.454
0.589
0.471
0.444
pima
768
2
0.488
0.472
0.465
0.469
0,466
0.464
pitisburg- butsburg-bridges-aee bridges-MATERIAL
106
0.460
0,500
pittsburg- L REL--L
103
0.642
0,569
0.525
0.590
0.535
0.531
pittsbug-bridges-S55
92
0.593
0.579
0.622
0.545
0.498
0.520
pitisburg- bridges-T-ORDD
102
0,585
0.462
0.526
0.466
0.437
0.468
pitisburg bridges.TYPE
105
0.834
0.758
0.755
0.784
0.725
0.720
planning
182
0.331
0.315
0.350
0.331
0.329
0.323
plant-margin
1600
64
100
0.623
0.603
0,509
0.614
0.603
0.476
plant-shape
plant texture
1599
64
100
0.659
0.650
0.556
0.654
0.624
0.503
post-operative
90
3
0.345
0.326
0,308
0.360
0.352
0.366
primary tumer
330
15
0,880
0.854
0.845
0.809
0.803
0.801
ringnorm
7400
20
0.381
0.344
0.304
0.415
0.350
0.377
seods
210
0.662
0.654
0.734
0.717
0.654
0.657
senccion
1593
256
10
0.552
0.538
0.479
0.538
0,514
0.499
soybeun
spumhase
4601
57
2
0.332
0,318
0.331
0.359
0.355
0.377
spect
265
22
0.473
0.427
0.418
0.518
0.513
0.518
spectf
267
44
0.784
0.581
0.575
0.430
0.362
0.355
690
0.471
0.360
0.347
0.446
0.414
0.421
statlog- geroo---rman--redit
1000
0.602
0.547
0.557
0.422
statlog-beart
270
13
2
0.459
0.432
0.400
0,402
0.376
0.395
statlog image
mutlog-landen
6435
36
0.491
0.457
0.447
0.445
0.415
0.380
statlog-shurtle shuttle
58000
0,541
0.284
0.448
0.447
0.483
statlog vehicle
846
0.546
0.532
0.439
0.526
0.493
0.450
steel-plates
1941
27
7
0.605
0.539
0.481
0.546
0.516
0.491
syuthetic-coetrol
0.713
0.463
teaching
151
5
0.727
0.694
0.693
0.699
0.650
0.626
thyroid
0.478
tic-tac-toe
958
9
0,863
0.257
0.398
0.693
0.458
0.558
titanic
2201
0.367
0.346
0.338
0.344
0.344
0.346
trains
10
29
1.000
0.833
0.917
0.979
0.812
0.812
twonorm
7400
20
2
0,376
0.351
0,360
0.379
0.373
0.374
vertebral-columee Zelases
0.509
veriebral column Jclases
310
3
0.679
0.632
0.599
0.476
0.472
0.472
wall-folle fall-following
waveform
5000
21
0.418
0.395
0.368
0.432
0.424
0.392
waveform--nosse
5000
40
0.441
0.430
0.398
0.440
0.429
0.406
wine
178
0.675
0.648
0.732
0.843
0.820
0.695
wine-quality-red
1590
11
6
0,578
0.532
0,521
0.521
0.502
0.492
wine quae-qualtyy-hiit white
0.475
yeast
1484
8
10
0.605
0.597
0.576
0.596
0,590
0.549
700
0.821
0.808
Continued from previous page.
25

---

Under review as a conference paper at ICLR 2021
Table 11: Comparison between RED and Counterparts Using AP-Success
Datasel
N
M
K
RED+BNN
BNN MCP
BNN Eniropy
RED+MC.D
MC.D MCP
MC.D Entropy
abalone
4177
3
0,855
0.854
0.849
0.854
0.852
0.851
acule inflammation
120
NA
NA
NA
NA
NA
NA
neste-nephritis
120
NA
NA
NA
NA
NA
NA
adult
48842
14
2
0,970
0.969
0.970
0.970
0,969
0.970
annealing
898
0,983
0.950
0.947
0.981
0,971
arthythmia
452
262
13
0,805
0.756
0.811
0.831
0.814
0.827
audiology-std
196
59
18
0.920
0.881
0.939
0.943
0
0.927
bulance-scull
625
0,999
0.996
0.997
0.999
0,998
0.999
hallooms
16
0.869
0.845
0.845
0.880
0.861
0.861
bank
4521
0,985
0.980
0.981
0.981
0,982
0.983
binconcentration
779
9
0.767
0.752
0.746
0.757
0,748
0.738
blood
748
0.885
0.905
0.902
0,903
0.905
breast-cancer
286
2
0,834
0.779
0.828
0.829
0.826
0.838
meast-cuncer-wisg
699
9
0.998
0,992
0.996
0.998
0.998
0.997
breast wisc diag
569
30
0,997
0.990
0.997
0.998
0,994
0.998
meaast-cance--wisc...
198
33
0.879
0.856
0.840
0.863
0.862
0.864
breast-lissue car
106
0,932
0.909
0.912
0.935
0,914
0.925
1728
4
0,997
0.998
0.998
0.999
1.000
0.999
cardiotosography- (0clases
2126
21
0,960
0.954
0.952
0.962
0,956
0.955
cardiotoogrrabby-allee
2126
21
3
0,990
0.988
0.989
0.992
0,991
0.992
chess-krvk
28056
18
0,923
0.919
0.912
0.806
0.753
0.738
chess. krikp
3196
36
0,999
0.998
1.000
1.000
0.999
1.000
climate
$40
20
2
0.989
0
0.993
0.987
0.987
0.990
congressionall-voinng
0,749
0.704
0.725
0.720
0,719
0.724
conn-hench-sonn caaa---ono----ii miness---nsss
208
60
2
0,940
0.931
0.953
0.955
0,951
0.957
comn-bench--voeee wowoboodoooooll-dddrdiinggg
990
11
0.998
0.994
0.990
0.998
0
0.996
connect-4
67557
42
2
0,975
0.975
0.975
0.959
0,954
0.954
contrac
1473
9
0.744
0.741
0.739
0.741
0.740
0.731
credit- appiitaaprooval
690
0
0.937
0.936
0.945
0,942
0.943
cylinder-bands
512
35
2
0.848
0.822
0.832
0.871
0.865
0.874
dermatology
366
0.992
0.999
0.999
cchocardiogram
131
10
2
0,947
0.935
0.923
0.951
0,946
0.950
ccoli
336
0.958
0.950
0.964
0.957
0
0.959
energy-y1
768
0,997
0.978
0.979
0.996
0,993
0.993
energy-y2
768
0.993
0.973
0.973
0.990
0
0.987
fertility
100
0,959
0.923
0.928
0 0.962
0.935
0.939
flags
194
28
8
0,743
0.694
0.716
0.756
0,708
0.728
glass
0.747
habernan-survival
306
2
0,873
0.718
0.801
0.851
0.847
0.848
bayes-roth
160
0.974
0.969
0,963
0.955
0.904
0.905
beart-cleveland
303
0.893
0.881
0.886
0.867
0.865
0.869
heart hungarian
294
12
0.934
0.913
0.929
0.926
0.908
0.911
seizt-switzerradd
123
0,588
0.539
0.517
0.502
0.476
0.469
beart-va
200
12
0.415
0,349
0.357
0.468
0.438
0.436
hepatitis
0.972
0.968
hill valley
1212
1OD
2
0,523
0.509
0.525
0.551
0,548
0.543
borse-colic
368
25
0.927
0.893
0.911
0.906
0
0.903
ipd indian liver
583
0.873
0.845
0,853
0.880
0.876
0.876
image-sege seggeeettgmttttioon
2310
18
7
0.997
0.993
0.997
0.998
0.997
0.998
ionosphere
351
33
0,984
0.965
0.982
0.991
0.991
iris
150
4
3
0,997
0.994
0.993
0.999
0.996
0.997
led display
1030
0,870
0.879
lenses
24
4
3
0.893
0.820
0.832
0.990
0.945
0.935
letter
20000
16
26
0.996
0.997
0.997
0.988
0.985
0.980
libras
360
90
15
0.879
0.847
0.732
0.952
0.939
0.925
low-res-spect
531
100
9
0.973
0.966
0.979
0.979
0.974
0.984
lung-canser
32
0.685
0.695
0.592
yymhhgrrphy
148
18
4
0.950
0.933
0.943
0.932
0.936
0.934
magic
19020
0.968
0.968
0.968
0,967
0.967
mammographic
961
2
0,925
0.909
0.916
0.922
0,919
0.920
messidor
1151
0.812
0.809
0,810
0.872
0.871
0.870
miniboone
130064
50
0.993
0.992
0.993
0.992
0.992
0.993
molec-biol-promotee
106
57
2
0,934
0.838
0.908
0.928
0,901
0.901
molec-biol- biol--iio--
3190
0.975
0.972
monks: 1
556
2
1.000
0.996
0.997
1.000
0,999
0.999
monks..2
0.818
0.822
monks-3
554
2
0.997
0.996
0.996
0.999
0.997
0.997
mushroom
8124
21
NA
NA
NA
NA
NA
NA
musk-1
476
166
0,938
0.915
0.919
0.980
0.976
0.980
musk-2
6598
166
2
0,999
0.998
0.998
1.000
0.999
1.000
nursery
12960
1.000
oncytes www.umeiusuuuuoommmm mucleus, 4d
1022
41
2
0,955
0.781
0.782
0.942
0.936
0.940
oocytes, mww.uoeeeeeeeeeeeemmeeee states,2f 2
1022
0.987
0.983
0.985
0.991
0.988
0.991
oocytes trisopeerus nuclous.22 2
912
25
0,914
0.850
0.866
0.936
0.926
0.934
oncytes trisopterus states 5b
912
32
3
0.985
0.977
0.980
0.993
0.991
0.993
Continued on next page.
26

---

Under review as a conference paper at ICLR 2021
Dotaset
N
M
K
RED+BNN
BNN MCP
BNN Eatropy
RED+MC-D
MC-D MCP
MC-D Entropy
optical
5620
62
10
0.997
0.997
0.999
0.999
0.998
0.999
uoome
2536
72
2
0,996
0.993
0.995
0.995
0,993
0.996
page bagesblocks
5473
0.998
0.998
0.998
parkinsons
195
22
2
0,988
0.975
0.977
0.986
0,987
0.987
pendigits
10992
10
0.999
0.999
0.999
0.999
0.999
0.999
phishing
1353
3
0,974
0.967
0.967
0.979
0,976
0.974
pima
768
0.897
0.890
0,906
0.901
0.899
0.904
106
0,978
0.963
0.971
0.976
0.973
0.973
pitsbbug-brigge-RRLLLL
103
7
0.809
0.772
0.731
0.798
0.786
0.791
pirtshurg- SPAN
0.837
0.813
pitssbus-bridgg---OOR-D
102
0,956
0.910
0.937
0.965
0,944
0.945
pitssuug-bridges-111
105
0,800
0.719
0.764
0.783
0.725
0.713
plamning
182
0.715
0.761
0.774
0.713
0.707
0.699
plant-murgin
1600
64
100
0.935
0.934
0.909
0.954
0.952
0.919
plant-shape
1600
100
0.714
0.685
0.622
0.874
0.857
0.804
plant-texture
1599
64
100
0,947
0.945
0.921
0.947
0,940
0.902
post-operative
90
3
0.685
0.664
0.066
0.692
0.688
0.688
primary-tumee
330
17
15
0,782
0.748
0.755
0.759
0.751
0.753
ringnorm
7400
20
0.999
0.999
0.999
0.998
0.997
0.998
seeds
210
0,980
0.976
0.991
0.990
0,986
0.985
semcion
1593
256
I0
0.992
0
0.992
0.992
0.993
0.993
soybcan
683
35
18
0,996
0.993
0.993
0.996
0.992
spumbase
4601
57
2
0,987
0.986
0.990
0.989
0,988
0.992
spect
265
0.753
0.777
0.792
0,780
0.782
speet!'
267
44
2
0,965
0.951
0.955
0.959
0,949
0,950
statlog mustralian credit
690
0.744
0.669
0.664
0.721
0.705
0.705
statlog- serrggggemannnne credi
1000
0,904
0.899
0,900
0.892
0.897
0.899
statlog-beart
270
13
2
0,937
0.926
0.930
0.933
0,926
0.931
statlog-image
2310
0.997
0.997
0.998
statlog- landsat
6435
36
6
0,984
0.982
0.981
0.985
0,985
0.985
statlog-shuttle
58000
9
1,000
0.999
1,000
1.000
1,000
1.000
statlog-vehicle
846
18
0,944
0,940
0.914
0,970
0,967
0.965
steel plates
1941
27
0.923
0.911
0,900
0.923
0,913
0.916
synthetic-contmi)
600
0,998
0.994
1.000
1,000
0.999
0.999
teaching
131
5
0.616
0,580
0,548
0,709
0.648
0.640
thyroid
7200
1,000
tic tac toe
958
9
1.000
0.998
0.999
1,000
0,999
0.999
titanic
2201
0.890
0.884
0.884
0.890
0.890
0.891
trains
10
29
1,000
0.833
0.917
0.979
0.812
0.812
twonorm
7400
20
0.999
0.999
0.999
0.999
0.998
0.999
Soreebaa-coluum-221e
2olases
310
0,976
0.962
0.967
0.976
0.975
0.977
310
6
0,971
0.965
0.965
0.976
0,976
0.976
wall- following
5456
0.990
0.994
waveform
5000
21
0,973
0.970
0.970
0.973
0,973
0.972
waveform-noise
5000
40
0.975
0.974
0.973
0.971
0,971
0.971
wine
178
0,997
0,990
0.999
0.999
0,998
0.999
wine-quality-red
1599
11
0.726
0.699
0.713
0.737
0.729
0.727
wine-quality-whine yeast
4698
7
0.626
0.609
0.665
0.654
0.635
1484
8
10
0.724
0.719
0.717
0.749
0,744
0.721
200
0.997
Continued from previous page.
Table 12; Comparison hetween RED and Counterparts Using AUPR- Frror
Datasct
N
M
K
RED+BNN
BNN MCP
BNN Entropy
RED+MC-D
MC-D MCP
MC-D Entropy
abalome
4177
8
3
0.526
0,519
0,510
0.513
0.506
0.501
acote-inflmmattion
120
NA
acutc-nephritis
120
2
NA
NA
NA
NA
NA
NA
adult
48842
0.414
0.406
0.405
0.415
0.405
0.404
anneuling
898
31
0.785
0.414
0.327
0.580
0.401
0.353
artyythii
452
262
13
0,713
0.603
0.726
0.539
0,531
0.529
audiologysttd
196
0.744
halance-scan scale
625
0,790
0.474
0.518
0.656
0,519
0.580
balloons
16
0,752
0.744
0,744
0.759
0.759
bank
4521
16
2
0,523
0.438
0.442
0.426
0,403
0.405
bioconcentration
779
0.557
0.477
0.477
0.454
0.427
0.408
bload
748
0,505
0.457
0.485
0.429
0.428
0.430
breast-cancer
286
9
2
0.488
0.372
0.394
0.498
0.459
0.476
cancer
0.360
hreast-cancer- waneeee---i--edddaae
569
30
2
0.362
0.333
0.379
0.511
0.441
0.500
breast-cance-wiise breast-lissue car
198
0,528
0.460
0.500
0.477
0.469
0.460
106
0,830
0.751
0.772
0.877
0.825
0.813
1728
0.591
0.475
0.316
0.628
0.418
0.277
curdiolocography- O0clases
2126
21
10
0.542
0.509
0.457
0.524
0.482
0.457
Continued on ned page.
27

---

Under review as a conference paper at ICLR 2021
Dotaset
N
M
K
RED+BNN
BNN MCP
BNN Eatropy
RED+MC-D
MC-D MCP
MC-D Entropy
candiotocographyykkhee
2126
21
3
0.490
0.413
0.409
0.458
0.388
0.392
chess krvk
28056
6
18
0.514
0,500
0.473
0.751
0.674
0.648
chess-krvkp
3196
0.312
0.378
climate
540
20
2
0,523
0.486
0.489
0.406
0,374
0.431
congressionaal-ooingg
435
2
0.503
0.450
0.457
0.471
0,470
0.481
comn benchhsonnen sont--oooo-------eeee mocks
208
60
2
0,463
0,460
0.466
0.494
0.496
0.510
conn bench vowel deterding
990
11
11
0.648
0.439
0.337
0.665
0.541
0.435
connect-4
67557
0.407
0.401
0.401
0,490
0,409
0.407
contrac
1473
9
3
0.631
0.626
0.619
0.601
0,595
0.590
credit-approval
690
0.344
0.386
cylinder-bands
512
35
2
0,506
0.423
0.406
0.443
0,447
0.454
dermstology
366
34
0.593
0.646
0,423
0.682
0.639
0.535
echocandiogram
131
0.394
0.360
0.378
0.477
0.435
0.452
ccoli
336
0.491
0.454
0,462
0.389
0.383
0.371
energy-yl
768
0.861
0.398
0.346
0.552
0,353
0.365
eaergy-y2
768
8
0.698
0.392
0.330
0.521
0.445
0.410
fertility
100
0,156
0.125
0.165
0.270
0.253
0.256
flags
194
28
0.785
0.737
0.732
0.754
0,705
0.715
glass
214
0.651
0.493
0.478
0.522
0.456
0.447
habermann survival
306
0.455
0.315
0.321
0.468
0.465
0.457
hayes-roth
160
0.638
0.518
0,520
0.686
0.535
0.555
beart-lleveland
303
0.764
0.730
0.750
0,700
0.674
0.683
beart- bungarian
294
12
2
0.463
0.431
0.437
0.387
0,356
0.380
beart-switzerladd
123
0.741
0.680
0.633
0.628
0.597
0.561
beart-va
200
0,827
0,803
0.822
0.744
0,703
0.720
hepatitis
155
19
0.575
0.523
0,536
0.583
0.539
0.508
hill-valley
1212
10D
0.508
0.491
0.529
0.566
0.561
0.543
horse-colic
368
25
2
0.365
0.314
0.339
0.354
0.347
0.372
iidd-iniin-liier
0.382
0.415
0.457
image segrenttation
2310
18
0.527
0.352
0.352
0.517
0.432
0.382
ionosphere
351
33
0.314
0.283
0,387
0.380
0.287
0.314
iris
150
3
0,772
0.624
0.381
0.686
0.631
0.619
led display
1000
1D
0.571
0.559
0.513
0.568
0.565
0.513
Jenses
24
0,757
0.553
0.559
0.974
0.833
0.80 807
letter
20000
16
26
0.458
0.452
0.397
0.705
0.626
0.562
libras
360
0.492
0.347
low res spect
531
100
9
0.610
0,523
0.588
0.557
0,517
0.543
lung-canser
32
56
0.730
0.686
0.663
0.840
0.768
0.725
lymphography
148
0.411
0.381
0.363
0.460
0.442
0.423
magic
19020
0.356
0.353
0.354
0.357
0.348
0.351
manmographic
961
0.432
0.420
0.409
0.419
0.415
0.417
messidor
1151
19
2
0.466
0.447
0.436
0.405
0.391
0.389
miniboone
130064
0,360
0.366
molec: hiol promoteer
106
57
2
0.608
0,518
0.526
0.506
0.435
0.450
molec-biol-splice
3190
60
0.435
0.420
0,392
0.465
0.426
0.439
monks- 1
556
0.827
0.354
0.377
0.788
0.474
0.477
monks-2
601
0.895
0.379
0.419
0.515
0.401
0.413
monks-3
554
0.596
0.446
0.446
0.627
0.468
0.462
mustroom
8124
21
2
NA
NA
NA
NA
NA
NA
musk-1
0.534
0.340
musk-2
6598
166
2
0,533
0.378
0.438
0.435
0.328
0.381
oursery
12960
0,340
0.166
0.162
0.578
0.318
0.275
oncyles merluscius nucleus 4d
1022
41
0,810
0.478
0.461
0.394
0.373
0.374
cocytes.merluecin merlaecius states.
1022
25
0,445
0.387
0.339
0.426
0.402
0.446
oocytes www.rrooooruusuuclcoooooo
912
25
0.432
0.426
0.373
trw.oossstsopperuus..o states__b
912
32
3
0.466
0.352
0.400
0.378
0,348
0.346
optical
0.289
ozone
2536
72
2
0,303
0.221
0.265
0.330
0.336
0.311
page-blocks
5473
0.436
0.352
0,308
0.3 0.394
0.360
0.342
parkinsons
195
22
0.769
0.521
0.482
0.465
0.371
0.454
pendigits
10992
16
10
0,309
0.287
0.292
0.382
0.355
0.261
phishing
1353
pima
768
2
0.477
0.459
0.451
0.457
0.454
0.452
pirtshurg MATERIAL.
106
0.379
0.435
pitshung--ridges-RELLL
103
3
0.618
0,535
0.486
0.558
0,501
0.492
pittsbug--beigee--PP4
92
0.550
0.534
0,582
0.499
0.449
0.463
pitshuug--rriges---OO-.)
102
0,530
0.399
0.466
0. 0.387
0.355
0.397
pitssbug--trigee-TYPE
105
7
6
0,825
0.737
0.738
0.770
0,704
0.694
planning
182
plant- margin
1600
64
100
0.619
0,597
0.502
0.610
0.598
0.469
plant-shape
1600
100
0.787
0,740
0.566
plant-texture
1599
64
100
0.655
0.646
0.550
0.650
0.620
0.496
opst-operative
90
3
0.301
0.277
0.253
0.307
0.300
0.323
primary-tumor
330
15
0,878
0.850
0.841
0.804
0.797
0.794
ringnorm
7400
20
2
0,370
0.331
0.290
0.405
0.341
0.367
weeds
0.644
0.711
semeinn
1593
256
10
0,543
0,527
0.465
0.523
0.496
0.483
Contirued next page.
28

---

Under review as a conference paper at ICLR 2021
Dataset
N
M
K
RED+BNN
BNN MCP
BNN Eatropy
RED+MC-D
MC-D MCP
MC-D Entropy
soyheun
683
35
18
0.524
0.487
0.354
0.498
0.482
0.368
spumbase
4601
57
2
0.324
0.311
0.323
0.351
0.347
0.369
spect
265
0.404
0.394
0.498
0.492
0.497
speetf
267
44
2
0,775
0,560
0.545
0.396
0.322
0.314
natlog-asstrlian-ccedit
690
0.460
0.350
0.335
0.435
0.403
0.409
caullogggrmann credit
1030
24
0.595
0,540
0.547
0.407
0,400
0.412
statlog heart
270
13
0.440
0.409
0.367
0.366
0.338
0.359
statlog-imags
2310
0,532
0,404
0.330
0.491
0,439
0.373
statlog-landsar
6435
36
6
0.488
0.453
0.444
0.441
0.411
0.375
statlog shuttle
58000
0.261
0.427
0,451
0.465
statlo--vchicle
846
18
4
0,536
0.522
0.426
0.513
0,478
0.437
steel-plates
1941
27
0.601
0.533
0,475
0.541
0.510
0.485
control
600
60
0.391
0.233
0.684
0.510
0.340
0.386
teaching
151
5
0.710
0.672
0.673
0.680
0.632
0.605
thyroid
7200
21
0.552
0.439
0.473
0.536
0,476
0,461
tic-tac-toe
958
9
2
0.851
0.187
0,344
0.642
0,379
0.505
ritanic
2201
0,361
0.340
0.332
0.338
0.340
trains
10
29
2
1,000
0.750
0.875
0.969
0.719
0.719
twonorn
7400
20
0.364
0.338
0.348
0.367
0.361
0.361
vertebral column 2elases
310
0,523
0.465
0.475
0.440
0.422
0.434
vertebral-colume ceaal--oou-----eeeeee
310
0.659
0.613
0,570
0.436
0.431
0.437
wall-following
5456
24
0,422
0.373
0.365
0,432
0,388
0,406
waveform
5000
21
3
0.414
0.391
0,363
0.428
0.419
0.387
waveform-noise
5000
40
0.437
0.425
0.393
0.436
0.425
0.401
wine
178
13
0.635
0.626
0.623
0.822
0,810
0.648
wine qualityyareeee
1599
11
0.574
0.527
0,516
0.517
0.497
0.487
wine caaneuullityytti white
4898
11
7
0.507
0.495
0.473
0.508
0.502
0.475
yeast
1484
8
10
0.602
0,594
0,573
0.592
0.586
0.544
Z0O
0.771
0.764
0.819
Continued from previous page.
Table 13: Comparison between RED and Counterparts Using AUPR-Suceess
Datasel
N
M
K
RED+BNN
BNN MCP
BNN Entropy
RED+MC.D
MC-D MCP
MC.D Entropy
abalone
4177
8
3
0.855
0.853
0,849
0,853
0.852
0.851
acute inflammation
120
NA
nephritiss
120
2
NA
NA
NA
NA
NA
NA
adult
48842
0,970
0.969
0.970
0.970
0.969
0.970
anncaling
898
31
5
0,983
0.951
0:947
0.981
0,972
0.974
anthythmia
452
262
13
0.801
0.750
0.806
0.827
0,809
0.823
audiology-std
196
18
0.912
0.938
0,902
0.925
balance-scale
625
3
0,999
0.996
0.997
0.999
0,998
0.999
ballooms
0.780
bank
4521
16
2
0.985
0.980
0.981
0.981
0,982
0.983
bioconenntration
779
0.764
0.749
0.741
0.753
0.744
0.734
blood
748
2
0,902
0.884
0.905
0.901
0,902
0.904
breast-cancer
286
9
2
0,829
0.770
0.823
0.823
0,821
0.834
brcast-cancer-wiss
0.995
0.995
0.997
breast cancer-ccereeeeee wisc diag
569
30
2
0,997
0.995
0.997
0.998
0
0.998
breast-cancer-we wisc-prrs proc
198
0.833
0.860
breast-tissue car
106
0,929
0.904
0.906
0.931
0,907
0.920
1728
4
0.997
0.998
0.998
0.999
1.000
0.999
candiolocooraphyy 10clases
2126
21
10
0,960
0.954
0.952
0.962
0.956
0.955
cardiotooggrabyy-olee
2126
21
3
0,990
0.988
0.989
0.992
0,991
0.992
cbess-Arvk
28056
chess krvkp
3196
36
2
0,999
0.999
1.000
1.000
0.999
1.000
climate
540
0.991
0.993
0.989
0.990
congressionall-ooinng
435
16
2
0,745
0.700
0.719
0.713
0,713
0.717
comn-bench-soone soneech---oa---eee mincs-rocks
208
60
0.939
0.948
0.952
0.954
0.951
0.956
bench vowel deterding
990
11
0,998
0.994
0.990
0.998
0.996
0.996
connect-4
67557
42
2
0,975
0.975
0.975
0.959
0,954
0.954
contrac
1473
0.737
0.729
credit approval
690
15
2
0,953
0.936
0.935
0.944
0,942
0.943
cylinder-bands
512
0.846
0.820
0.830
0.869
0.865
0.872
dermatology
366
34
0.998
0.996
0.999
1.000
1.000
0.999
echocardiogram
131
0,945
0.934
0.918
0.949
0,944
0.949
ecoli
336
0,958
0.954
0.963
0.956
0.955
0.959
energy-y1
768
8
3
0,997
0.978
0.979
0.996
0,983
0.993
energy-yd y2
0.973
fertility
100
2
0.957
0.917
0.921
0.960
0.928
0.933
flags
194
28
0.734
0.679
0.702
0.747
0.697
0.718
glass
214
0,832
0.729
0.741
0.833
0,786
0.824
habermaa survival
306
0.869
0.769
0.795
0.845
0,540
0.841
hayes-roth
160
0.973
0.969
0.962
0.94
0.901
0.903
Continued on ned page.
29

---

Under review as a conference paper at ICLR 2021
Datasct
N
M
K
RED+BNN
BNN MCP
BNN Eatropy
RED+MC-D
MC-D MCP
MC-D Entropy
heart- clatt-cleeelad
303
13
5
0.891
0.879
0.884
0.865
0.863
0.866
beart hunguriam
294
12
2
0,933
0.913
0.927
0.924
0,904
0.908
seitt-swibzerland
123
0,562
0.505
0.489
0.470
0.437
heart-va
200
12
5
0,379
0.312
0.323
0.443
0,410
0.407
bepatiti
155
19
2
0.968
0.958
0.972
0.957
0.966
0.967
hill-valley
1212
100
0.517
0.504
0.519
0.546
0.542
0.538
horse colic
368
25
0.926
0.894
0,909
0.904
0.895
0.901
upd-indian--iver
583
0.872
0.845
0.852
0.879
0.875
0.875
imago-sgmmcutation
2310
18
7
0,997
0.995
0.997
0.998
0,998
0.998
ionosphere
351
0,984
0.977
0.982
0.991
0
0.991
iris
150
4
3
0,997
0.994
0.993
0.997
0,996
0.997
led-display
1000
7
10
0.886
0.881
0.870
0.887
0
0.878
lenses
24
0,860
0.783
0.798
0.988
0,934
0.922
letter
20000
16
26
0.996
0.997
0.997
0.988
0
0.980
libras
360
90
15
0,878
0.845
0.728
0.952
0,938
0.924
low-res-spect
531
100
9
0,973
0.973
0.979
0.978
0,978
0,984
lung-cancer
32
56
0.645
0.613
0.628
0.666
0,582
0.519
tymphography
148
18
0,948
0.941
0.942
0.929
0,934
0.932
magic
19020
10
0.969
0.968
0.968
0.968
0.967
0.967
mammographiic
961
0,924
0.908
0.915
0.921
0
0.919
messidor
1151
19
2
0,811
0.808
0.809
0.871
0.871
0.870
miniboone
130064
50
0,993
0.992
0.993
0.992
0,992
0.993
molec-biol-pee bio--porortoeeeeer
106
57
2
0,932
0.896
0.904
0.925
0.897
0.897
molcc-biol-splice
3190
0.977
0.976
0.975
0.970
0,970
0.972
monks-1
556
2
0,999
0.996
0.997
1.000
0,999
0.999
monks-2
601
0.964
0.808
0,816
0.870
0.821
0.820
monks.3
554
0.997
0.996
0.996
0.999
0.997
0.997
mushroom
8124
21
2
NA
NA
NA
NA
NA
NA
musk-1
476
166
0.914
0.918
0,979
0,980
musk-2
6598
166
2
0,999
0.998
0.998
1,000
0,999
1.000
nursery
12960
1.000
1.000
1,000
1.000
1,000
1.000
muyyeesmmrlugccia aucleus..d 4d
1022
41
2
0,955
0.779
0.780
0,942
0,936
0.939
oncytes, merluccius states. 2
1022
25
0.987
0.984
0.985
0.991
0,989
0.991
oncytes trisopterus nucleus 2
912
25
0,914
0.849
0,866
0.936
0.925
0.933
oocytes. wttpsssstroooouuussooo statess_
912
32
3
0,984
0.977
0,980
0.993
0,992
0.993
optical
5620
0,997
0.988
0.999
0.999
ozone
2536
72
2
0,996
0.995
0.995
0.995
0,995
0.996
page-blocks
5473
10
0.998
0.997
0.998
0.998
0.998
0.998
parkinsons
195
22
0,988
0.976
0.977
0.985
0,987
0.987
pendigits
10992
16
10
0.999
0.999
0.999
0.999
0.999
0.999
phishing
1353
3
0,974
0.967
0.967
0.979
0.975
0.974
pima
768
8
2
0.896
0.889
0.906
0,900
0
0.903
pitisburg- briiges-ddMMe MATERIAL
106
0.969
0.972
pittshurg bridges REL.I RE
103
3
0.798
0.756
0.709
0.784
0,771
0.775
pilisburg-brian Bulsburg-bridges-SS bPiN
92
3
0.845
0.835
0,822
0.843
0.794
0.799
pittsburg bridges TOR
102
0,954
0.913
0.933
0.964
0,941
0.940
hrinurggrriges-TYPPP
105
7
6
0.785
0.691
0.744
0.765
0.697
0.683
planning
182
0.766
0.750
0.765
0.699
0.692
0.682
plant-margin
1630
64
100
0,935
0.933
0.908
0.954
0,952
0.919
plant-shupe
1600
100
0.683
0.620
plant-texture
1599
64
100
0,947
0.945
0.921
0.947
0.940
0.902
post-operative
90
3
0.659
0.633
0.635
0.667
0.663
0.662
primary- tumor
330
15
0.777
0.740
0.747
0.753
0.744
0.747
ringnorm
7400
20
2
0,999
0.999
0.999
0.998
0.997
0.998
seeds
210
3
0.984
0.990
0.990
0.985
0.985
semeiin
1593
256
10
0.992
0.992
0.992
0.992
0,993
0.993
soyheun
0
0.994
0.993
0.996
0
0.992
spanbase
4601
57
2
0.987
0.986
0.990
0.989
0.989
0.992
spect
265
22
0.769
0.743
0.769
0.785
0.772
0.773
speetf
267
44
2
0.965
0.956
0.954
0.958
0.951
0.949
aedog--asttalian-ccedit
690
14
2
0,739
0.663
0.659
0.716
0,700
0.700
statlg-german--reee gredin-credit
1030
0.899
0.899
0.898
statlog heart
270
13
2
0.936
0.928
0.929
0.931
0.924
0.930
statlng- -mmgge
2310
0
0.997
0.997
0.998
0
0.998
staclog-landsat
6435
36
6
0,984
0.982
0.981
0.985
0
0.985
statlog-shuttle
58000
9
1.000
0.999
1,000
1.000
1.000
1.000
staclog-vehicle
846
0,944
0.940
0.914
0.970
0.967
0.964
stecl-plates
1941
27
7
0,923
0.912
0.900
0.923
0.913
0.916
synthetic-control
600
0.997
1.000
0.999
teaching
151
5
0.589
0.555
0.517
0.694
0.627
0.618
thyroid
7200
21
1,000
0.999
1.000
0.999
0.999
1.000
tic-tac-toc
958
9
2
1.000
0.998
0.999
1.000
0.999
0.999
titanic
2201
0.890
0.884
0.885
0.8%0
0.889
0.891
trains
10
29
1,000
0.750
0.875
0.969
0.719
0.719
twonorm
7400
20
2
0,999
0.999
0.999
0.999
0,998
0.999
certebrl-columee Zelases
310
0.964
0.967
0.976
vertebral. www.mllooooonnccce
310
3
0,971
0.967
0.965
0.976
0.975
0.975
Contieued on next page.
30

---

Under review as a conference paper at ICLR 2021
Dotaset
N
M
K
RED+BNN
BNN MCP
BNN Eatropy
RED+MC-D
MC-D MCP
MC-D Entropy
wall followoll
5456
24
4
0,990
0.987
0.990
0.992
0.992
0.994
waveform
5000
21
3
0,973
0.970
0.970
0.973
0,973
0.972
waveform-noise
0.973
wine
178
13
3
0,997
0.994
0.999
0.999
0,998
0.999
iine--uality-red
1599
0.725
0.697
0.711
0.736
0.728
0.725
wine saane-quallyy-aaaee
4898
11
0.639
0.624
0.607
0.664
0.653
0.634
yeast
1484
10
0.721
0.716
0.715
0.747
0.741
0.718
101
7
0.997
0.996
0.997
0.998
0.996
0.995
Continued from previous page.
Table Comparison between and Counterparts Using AUROC
Dutaset
K
RED+BNN
BNN MCP
BNN Eatropy
RED+MC-D
MC-D MCP
MC-D Entropy
abalone
4177
8
3
0,731
0.727
0.718
0.725
0,723
0.716
acute-infammation
120
NA
acule nephariiss
120
2
NA
NA
NA
NA
NA
NA
adult
0.843
0.841
annealing
898
31
0,928
0.798
0.764
0,883
0.850
0.836
arthythmii
452
262
13
0.766
0.703
0.784
0.709
0.698
0.700
madiologysstd
196
59
18
0,850
0.814
0,865
0.874
0.814
0.851
balanco-scalc
625
3
0,982
0,946
0.965
0.977
0,955
0.968
balloons
bank
4521
16
2
0,900
0.867
0.876
0.868
0.872
0.877
hinconcentration
blood
748
0.757
0.723
0.753
0.731
0.731
0.735
cancer
286
0.662
0.581
0.652
0.672
0.668
0.678
breast-cance--wise
699
0,944
0.876
0.911
0.943
0.938
0.936
breast-canccr-wis-dee
569
30
2
0.898
0.769
0.896
0.941
0.893
0.929
breast cancer- wisc-prog
breast tissue car
106
0.891
0,860
0,862
0.909
0.883
0.888
1728
0.956
0,942
0.965
candiolocography.
Satrephhyy-llllaaaa 10clases
2126
21
10
0.848
0.835
0,810
0.845
0.831
0.818
ouddnnoooorapphy-elee
2126
21
3
0.912
0.896
0.901
0.913
0.905
0.910
chess-krvk
28056
18
0.792
0.781
0.762
0.777
0.716
0.696
chess-krvkp
3196
36
2
0.917
0.883
0.954
0.968
0.927
0.956
climate
congressionall-vviin
435
16
0.632
0.594
0.605
0.616
0.615
0.617
homn-bench-sona-minne---
208
0,780
0.794
0.825
0.836
comn-beach- wewpech--ooll--deeedee
990
11
0,965
0.916
0.863
0.969
0,936
0.928
connect-4
67557
42
2
0.852
0.851
0.851
0.832
0.807
0.808
contrac
1473
0.690
0.683
0.665
appdit-approval
690
15
2
0.773
0.723
0.735
0.753
0,750
0.753
cylinder hands
0.653
dermatology
366
34
0.937
0.882
0.969
0.988
0.987
0.982
cchocardiogram
131
0.737
0.703
0.719
0.798
0.776
0.786
ccoli
336
0.817
0.807
0.835
0.780
0.777
0.800
eoorgy-yl
768
0,980
0.849
0.849
0.933
0.888
0.888
eaergy-y2
fertility
100
2
0.594
0.439
0.471
0.707
0.609
0.608
flags
194
0.730
glass
214
9
0,739
0,578
0.589
0.668
0.612
0.660
habernan-survival
306
0.712
0.544
0.559
0.700
0.699
0.703
bayes hayes-roth
160
0
0.863
0.837
0.880
0,786
0.787
beart-cleveland
303
13
S
0.838
0.827
0.834
0.795
0,795
0.800
beart-hungarian
0.766
heart switzerland
123
12
0.637
0,592
0.543
0.521
0.487
0.444
heart-va
0.590
0.561
hopatitis
155
19
2
0,853
0.812
0.859
0.843
0.849
0.850
hill valley
1212
100
0.511
0.495
0.528
0.560
0.559
0.543
horse colic
368
25
0,733
0.664
0.688
0.702
0.693
0.708
uldd-indaan-lier
$83
9
2
0,723
0.655
0.671
0.723
0.716
0.717
image segmenntttion
2310
0.952
ionosphere
351
33
0,853
0.777
0,851
0.882
0,874
0.878
iris
150
0.931
0.908
0.961
lod-display
1030
7
10
0,760
0.753
0.728
0.760
0,756
0.738
Jemses
3
0.798
0.667
0.702
0.979
0.385
0.865
letter
20000
26
0,940
0.950
0.943
0.955
0.920
0.895
libras
360
90
15
0.804
0.739
0.594
0.818
0.767
0.720
res spect
lung-cancer
32
56
0.669
0.601
0.596
0.722
0,613
0.564
tymhhggaphyy
148
0.741
0.742
0.758
0.748
magic
19020
0,816
0.814
0.816
0.813
0.811
0.811
mammographic
0.753
0.739
0,740
0.757
0.754
0.755
messidor
1151
0.661
0.655
0.648
0.691
0.689
0.686
Continued on ned page.
31

---

Under review as a conference paper at ICLR 2021
Dotasct
N
M
K
RED+BNN
BNN MCP
BNN Entropy
RED+MC-D
MC-D MCP
MC-D Entropy
miniboone
130064
50
2
0,906
0.903
0.904
0.908
0.904
0.905
molec-bine biol promooeer
106
57
2
0.803
0.676
0.745
0.779
0.720
0.713
molec: hiol splice
3190
0.845
monks-1
556
2
0,989
0.877
0.918
0.991
0,963
0.975
monks-2
601
0.934
0.581
0.611
0.742
0.639
0.641
monks-3
554
0,959
0.930
0.934
0.970
0,950
0.950
mushroom
8124
21
NA
NA
NA
NA
NA
NA
musk-1
476
166
0.838
0.769
0.792
0.835
0.821
0.828
musk-2
6598
166
2
0,968
0.957
0.961
0.971
0,937
0.971
nursery
12960
0.967
oocytes_mee morloecias nucleus.44 4d
1022
41
2
0,910
0.649
0.641
0.771
0.761
0.764
oocytes.mertunee states.21
1022
25
0.891
0.873
0.872
0.902
0.892
0.905
oncyles, wwt.:rirrpeuuuuucceeooooe 2u
912
25
0.793
0.677
0.697
0.752
0.735
0.750
wwtssrsssesssssiii St
912
32
0.875
0.832
0.854
0.903
0.898
0.914
optical
5620
0.897
0.907
0.963
0,941
0.960
ozone
2536
72
2
0.899
0.866
0.882
0.889
0.872
0.909
page-blocks
5473
0.932
0.947
0.948
0,942
0.950
parkinsons
195
22
0,932
0.851
0.865
0.890
0.869
0.873
pendigits
10992
10
0.936
0.934
0.947
0.946
0.937
0.941
phishing
1353
0.884
0.841
0.840
0.876
0.863
0.855
pima
768
0.755
0.750
0,752
0.751
0.750
0.750
pittsburg- Missuug-briges-MMTTRIAAL
106
0.775
0.797
0.813
pittsbug--brddee REL-L
103
7
0.682
0.625
0,576
0.656
0.624
0.629
Sitshugg-bridee-SPAAA
92
0.716
0.698
0.702
0.675
0.618
0.643
Hlssburg-bridge-----O
102
2
0,793
0.715
0.742
0.780
0,730
0.740
pilisburg bYidges/TYPE
105
0.791
0.728
0,750
0.771
0.715
0.712
planning
182
0.518
0.496
0.519
0.452
0.446
0.441
plant-margin
1600
64
100
0.835
0.829
0.767
0.855
0.850
0.768
plant-shape
1600
0.677
0.706
plani texture
1599
64
100
0.856
0.853
0.796
0,853
0,840
0.760
post-operative
90
3
0.380
0.364
0.357
0.428
0.415
0.414
primary tummrr
330
15
0,831
0.811
0.803
0.781
0,777
0.777
ringnorm
7400
20
0.953
0,943
0,952
0.940
0.915
0.946
seeds
210
0.875
0.869
0.932
0.929
0.896
0.902
soncion
1593
256
10
0,916
0,915
0.903
0.913
0.917
0.914
soybean
spumbase
4601
57
2
0,853
0.847
0,873
0.861
0,858
0.889
spect
265
22
0.616
0.576
0.592
0.656
0.646
0.649
spectf
267
44
0.895
0.845
0.848
0.777
0,751
0.744
statlog mustralian-erede credit
690
0.628
0.502
0.490
0.582
0.557
0.561
statlog-gent nanlog-german-creit
1000
0,793
0.781
0.786
0.717
0.720
0.722
statlog-beart
270
13
2
0.739
0.701
0.703
0.736
0.712
0.724
statlog image
2310
0.954
stating landsat
6435
36
0,890
0.876
0.874
0.884
0.881
0.877
statlog shuttele
58000
0.999
0.823
0.9%8
0.999
0.905
0.998
statlog vehicle
846
0.829
0.818
0.741
0.861
0.857
0.837
steel-plater
1941
27
7
0.824
0.788
0.757
0.802
0.786
0.773
syothetic-cottool
600
0.861
0.738
0,980
0.974
0.965
0.966
teaching
151
5
0.656
0.618
0.614
0.690
0.624
0.611
thyroid
7200
0.985
ric-tac-tne
958
9
2
0.990
0.875
0.961
0.985
0.951
0.975
titanic
2201
0.677
0.664
0.065
0.669
0.668
0.674
trains
10
29
1.000
0.667
0,833
0.958
0.625
0.625
twonorm
7400
20
2
0.958
0.956
0.957
0.950
0.945
0.955
vertebral-oolum-e 2olasss
310
0.836
0.846
0.858
vertebral coloo-ooomnnncce Jclases
310
3
0.888
0.877
0.870
0.861
0,860
0.859
following
waveform waveform-noose
5000
21
0,845
0.834
0.833
0.850
0,849
0.845
5000
0,858
0.856
0.849
0.850
0.850
0.849
wine
178
0,923
0.857
0.976
0.979
0.949
0.954
wine-quality-red
1599
11
6
0.657
0.629
0.622
0.640
0.626
0.611
wine-qualitywwinn yeast
0.544
1484
8
10
0.675
0.667
0.648
0.690
0.687
0.645
700
0.965
Continued from previous page.
32


Reviews:
[{"review_id": "zH59rd_2BGQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, their goal is to improve calibration and accuracy by augmenting a classification model with a GP. They base their model off RIO (ICLR 2020) which targets regression problems and tries to predict the residual between predicted value and true value. They propose a model, RED, which instead tries to predict the residual between the predicted confidence score for the true class and 1 — the true class target confidence score using a GP. They show strong improvements over the methods they compare to for 125 UCI datasets and CIFAR-10 dataset.\n\nI find the approach interesting though the novelty is incremental over the RIO paper. My main concern is that I think some additional methods need to be compared with. For example [1] uses a bayesian last layer which is something that should be compared with. Using an ensemble of single layer NNs for the last layer or using MC-dropout at test time (which is known to approximate Bayesian inference under certain conditions) would also be interesting.\n\n[1] “Scalable Bayesian Optimization Using Deep Neural Networks” by Snoek et al. \n[2] “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning” by Gal et al.\n\nEdit: Based on the author response in terms of adding additional experiments, I'm raising my score to a 6.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "yaY5db__tMY", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper solves an interesting problem of predicting uncertainty in NN without re-raining/modifying the existing NN. The authors propose a framework to calculate a confidence score for detecting misclassification errors by calibrating the NN classifier’s confidence scores and estimates uncertainty around the calibrated scores using Gaussian processes. This framework is called RED (Residual i/o Error Detection). \n\nThis paper is also technically sound and to the best of my knowledge is novel and relevant to the community. \n\nIt would be good to apply SVGP directly to some of these datasets and compare the results against NN+SVGP results.\n\nYou use the term “calibrated” confidence score/prediction. Could you explain what do you mean by calibrated?\n\nI find the presentation of results very confusing. For example, in Table 1, AP-Error is smallest for the RED method and in Table 3 AP-error is the largest for the RED method. In both cases, it is mentioned that the RED method outperforms other methods. \n\nYou mentioned ConfidNet outperformed the MCP baseline by a margin of 0.42. I do not see this number on the table.\n\nIt would be good if the authors could mention in the paper what is RIO short for.\n\nYou mentioned that you need to extend the kernel to multiple output kernel. Could you explain a bit more about that and how you build it?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WBMKrZ7hKvz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: Following the authors' clarifications and additional experimental work, I'm increasing my rating to 6.\n\nThis paper proposes RED, a framework for detecting misclassification errors, based on regression of target confidence scores and application of a Gaussian process for uncertainty in predicted confidence scores. It builds upon RIO, a framework for predicting residuals of regression models and their uncertainties using GPs. Compared with other confidence metrics, RED aims for greater separability between correct and incorrect predictions.\n\nThe method is straightforward to implement and performs well against the baselines considered on classification tasks for 125 UCI datasets. However, I question whether the baselines are sufficient; it is not demonstrated whether RED would outperform other confidence scoring and OOD detection methods mentioned in the related work section, such as temperature scaling (or the related method ODIN, proposed in Liang, S., Li, Y., and Srikant, R., 2017. Enhancing the reliability of out-of-distribution image detection in neural networks.) or simply the entropy of the softmax predictions. Unless there is a good justification for the limited set of baselines, I believe the paper's claims to generality are limited.\n\nAdditionally, for the OOD detection results shown in Figure 3, why were AUROC and AUPRC not reported? While the scatterplots show separability of OOD data visually, these metrics (used elsewhere in the paper) would give a better indication of performance (and again, I think a greater range of baselines and tasks would be necessary to make any firm claims about OOD detection).", "extra_scores": {}, "extra_text": {}}, {"review_id": "8y86mQBwfLy", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#######################################################\nSUMMARY\n\nThis paper introduces RED, a new methodology to produce reliable confidence scores to detect missclassification errors in neural networks. The idea is to combine kernels based on both input and output spaces (as in RIO) to define a (sparse) GP that estimates the residual between the correctness of the original prediction and the maximum class probability. The authors show enhanced performance against other related methods and the ability of RED to detect OOD and adversarial data through the variance of the confidence score. \n\n#####################################################\nPROS\n\n1) Obtaining confidence scores for neural network predictions is a timely and very relevant topic for the ICLR community, since it is one of the main limitations of real-world applications of current neural nets.\n\n2) The related literature review is clear and, to the best of my knowledge, the proposed metholody based on Gaussian Processes is novel. \n\n3) The experimental validation of the proposed method on the UCI datasets is strong. It uses a wide range of datasets and several statistical tests, and RED obtains superior performance.\n\n4) The idea of using the variance of the proposed confidence score to identify OOD and adversarial data is interesting and promising.\n\n######################################################\nCONS\n\n1) My main concern is that the contribution in RED can be regarded somehow incremental given the RIO approach. It utilizes the same rationale behind RIO, and just adapts the necessary components so that it works in classification. The adaptation of these components is also straightforward: the output kernel now works on several dimensions (instead of the scalar dimension of regression) and the target is now the correctness of the original prediction. \n\n2) The experimental validation focuses on several competitors which can be considered \"of the same family\" as the proposed approach. Namely, all of them calibrate the predictions of a pre-trained neural network. I think it would be interesting to also compare to a different \"family\" of methods. For instance, (Functional) Bayesian Neural Networks are meant to obtain calibrated predictions by leveraging epistemic uncertainty (that coming from the model parameters). \n\n3) I do not fully understand the relevance of the experiment with the large deep learning architecture given by the VGG16 model. Since the proposed method works on the pre-trained neural network, my understanding is that the complexity of the neural network itself is not relevant for the performance of the proposed approach. Also, in this experiment I miss several independent runs to assess the results variability. \n\n####################################\nAdditional questions/feedback:\n\n1) In the second paragraph of section 4.2., there seems to be a typo when reporting the margin. It is said 0.42 and 0.55 for ConfidNet and RED respectively, but I think it should be 0.042 and 0.055 by looking at Table 3.\n\n2) It is not entirely clear to me why the process described in section 4.3. (second paragraph) produces proper OOD and adversarial data. For instance, some of the intended OOD data could be similar to training data (specially because the latter is being normalized to mean 0 and std 1). And similarly for the adversarial case. I think this could be better explained.\n\n3) When it comes to real practice, a key decision is to set a threshold on the confidence score to decide what instances should be supervised by an expert. Is there any recommendation on this?\n\n####################################### \nAFTER REBUTTAL\n\nThe new baselines added make the experimental validation more convincing. Therefore, I have raised my rating to 6 (Marginally above the acceptance threshold). However, I still believe that the contribution is incremental, and I think the paper would gain in terms of novelty if it focused more on the detection of OOD data and adversarial attacks (which right now is more like a preliminary test).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors use a GP classifier to detect if the output of a NN classifier has been decided correctly. The GP takes as input the original input vector x and the output of the NN, i.e. the calibrated posterior probabilities given by the NN. It uses that as an input vector for the GP classifier to decide if the sample was correctly decided. The output of the GP will serve as confidence in the output of the NN. The results are comparable/superior with the state-of-the-art and the authors have repeated the experiments with over 125 different datasets. The reviewers of this paper were all cautiously positive about the paper, but all of them pointed towards the reduced novelty of the paper. Also, none of the reviewers were willing to champion this paper as a must-have at ICLR 2021.
 
For my reading of the paper, I would tend to agree with the reviewers’ comments. Also, I find that using the same NN, rather shallow, with the same configuration for all the datasets seems rather limited. Given that this method is independent of the underlying classifier and that the databases used are low dimensions and a low number of training examples, I would have liked to see what a random forest or a GP can accomplish. Also, I would have used bigger NNs that can be trained to overfit the sigmoid outputs for classification of higher accuracy. I believe that having a diversity of underlying classifiers is more relevant than having 125 datasets. We need to find the best classifier or ensemble and then apply the different mechanisms for estimating if the output is the correct one. Otherwise, the proposed method might only be workable for this specific NN configuration. In the tables, it can be hinted that this might be happening, as about 80% of the cases MCP and RED are indistinguishable in the AUROC values.
 
Also, for all of these datasets a GP could be used as an underlying classifier, and given the premises of this paper, the authors could check how well calibrate a GP classifier is. Also, there has been considerable work on calibrating NNs when they are trained to overfit. Comparing with those methods should be straightforward, as they provide more information than just a confidence score. This is probably the most influential paper: https://arxiv.org/abs/1706.04599 (1000+ references), but there are some recent papers too.  
 
Finally, if the goal is to use a GP to detect if the classification done by the NNs is accurate, using a GP might be an overkill, as the complexity of the GP, especially for large datasets might end up being larger than the underlying classifier.

Decision: Reject