What is Flagged in Uncertainty Quantification?  Latent Density Models for Uncertainty Categorization
Authors: Hao Sun
Boris van Breugel
Jonathan Crabbé
Nabeel Seedat
Mihaela van der Schaar
Venue: NeurIPS 2023 poster (2023)
Keywords: Uncertainty Explaination
Uncertainty Quantification
Interpretability
TL;DR: In this work, we propose a framework for categorizing uncertain examples flagged by uncertainty quantification methods.

Abstract: Uncertainty quantification (UQ) is essential for creating trustworthy machine learning models. Recent years have seen a steep rise in UQ methods that can flag suspicious examples, however, it is often unclear what exactly these methods identify. In this work, we propose a framework for categorizing uncertain examples flagged by UQ methods. We introduce the confusion density matrix---a kernel-based approximation of the misclassification density---and use this to categorize suspicious examples identified by a given uncertainty method into three classes: out-of-distribution (OOD) examples, boundary (Bnd) examples, and examples in regions of high in-distribution misclassification (IDM). Through extensive experiments, we show that our framework provides a new and distinct perspective for assessing differences between uncertainty quantification methods, thereby forming a valuable assessment benchmark.

What is Flagged in Uncertainty Quantification? Latent Density Models for Uncertainty Categorization
Hao Sun Boris van Breugel', Jonathan Crabbé, Nabeel Seedat, Mihaela van der Schaar Department of Applied Mathematics and Theoretical Physics University of Cambridge
Abstract
Uncertainty Quantification (UQ) is essential for creating trustworthy machine learning models. Recent years have seen steep rise in UQ methods that can flag suspicious examples, however, it is often unclear what exactly these methods identify. In this work, we propose framework for categorizing uncertain examples flagged by UQ methods in classification tasks. We introduce the confusion density matrix kernel-based approximation of the misclassification density- and use this to categorize suspicious examples identified by given uncertainty method into three classes: uut-of-distributioo (OOD) examples, boundary (Bnd) examples, and examples in regions of high in-distribution misslassificatton (IDM). Through extensive experiments,
show that our framework provides and distinct perspective for assessing differences between uncertainty quantification methods, thereby forming valuable assessment benchmark.
1 Introduction
Black-box parametric models like neural networks have achieved remarkably good performance on variety of challenging tasks, yet many real-world applications with safety concerns e.g. healthcare (1]. finance 2 3 and autonomous driving [4 neceesittatee reliability. These scenarios
require trustworthy model predictions to avoid the high cost of erroneous decisions.
Uncertainty quantification 16-10) addresses the challenge of trustworthy prediction through inspecting the confidence of a model, enabling intervention whenever uncertainty is too high. Usually, however, the cause of the uncertainty is not clear. In this work, we go beyond black-box UQ in the context classification tasks and aim answer two questions:
1. How do we provide a more granular categorization of why uncertainty methods to identify certain predictions as suspicious?
2. What kind examples do different UQ methods tend to mark as suspicious?
Categorizing Uncertainty We propose a Density- based Approach for Uncertainty Categorization (DAUC): model-agnostic framework that provides post-hoc categorization for model uncertainty. We introduce the confusion density matrix, which captures the predictive behaviors of a given model. Based such confusion density matrix, we categorize the model's uncertainty into three classes:
(1) OOD uncertainty caused by OOD examples, i.c. test-time examples that resemble no training-tine -imm sample [TH44). Such uncertainty can be manifested by a low density in the confusion density matrix; (2) Bnd uncertainty caused by neighboring decision boundaries, i.e., non-conformal predictions due to confusing training-time time resemblances from different classes or inherent ambiguities making it challenging to classify the data 15:17: Such uncertainty can be manifested by the density of
hhs789@cam..c equal contributions
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

00p Baample
Iepeove Moed
Boundary
Boundry
Hoad
Production
(010
Boundary & IDM Eeample
Traiel Proficion
Bouples
Impeove
IDM Examppe
Madi
Decruttdd Dro.etio
Medal Loost Cangeean
Bon' Eimple
Prodict
Figure 1: Given prediction and UQ algorithm, our method divides flagged test time examples into different classes. Class OOD identifies outliers, which are mistrusted because they do not resemble training data; class IDM indicates examples lying in regions with high misclassification; class Bnd indicates examples that lie near the decision boundary.
diagonal elements in the confusion density matrix; (3) IDM uncertainty caused by imperfections in the model-as as manifested by high misclassification at validation time-such that similar misclassification be expected during testing [18]. Such uncertainty
be manifested by the density of off-diagonal elements in the confusion density matrix. Figure Iillustrates the different classes and in Section 4 we provide visualisations using real data. We show how DAUC can be used to benchmark broad class of UQ methods, by categorizing what each method tends to flag as suspicious.
Our contributions can be summarized follows:
1. Formally, we propose the confusion density matrix the heart of DAUC -tht links the training time error, decision boundary ambiguity, and uncertainty with latent representation density.
2. Practically, we leverage DAUC unified framework for uncertain example categorization. DAUC offers characterisation uncertain examples at test time.
3. Empirically, we use DAUC to benchmark existing UQ methods. We manifest different methods' sensitivity to different types of uncertain examples, this provides model insight and aids UQ method selection.
2 Related Work
Uncertainty Quantification Uncertainty quantification methods are used to assess the confidence
in model's predictions. In recent years the machine learning community has proposed many methods which broadly fall into the following categories: (1) Ensemble methods (i.e. Deep Ensembles (6), considered state of the art hat-have2a high computational burden; (2) Approximate Bayesian methods (i.e. Stochastic Variational Inference (7-9), however work by (1921 suggests these methods do not yield high-quality uncertainty estimates; and (3) Dropout-based methods such as Monte Carlo Dropout [10), which are simpler as they do rely on estimating a posterior, however the quality of the uncertainty estimates is linked to the choice of parameters which need to be calibrated to match the level of uncertainty and avoid suboptimal performance (22). For completeness we note that whilst conformal prediction 23 is another UQ method, the paradigm is different from the other aforementioned methods, as returns predictive sets to satisfy coverage guarantees rather than a value-based measure of uncertainty.
In practice, the predictive uncertainty for a model prediction arises from the lack relevant training data (epistemic uncertainty) or the inherent non-separable property of the data distribution (aleatoric uncertainty) (24) This distinction in types of uncertainty is crucial as it has been shown that samples with low epistemic uncertainty why epistemic uncertainty has been used for OOD/outlier detection [10]. Moreover, ambiguous instances close to the decision boundary typically result in high aleatoric uncertainty (27)
more likely under the data distribution, hence motivating
This suggests that different sets of uncertain data points are associated with different types of uncertainties and, consequently, different types of misclassifications. Furthermore, it is to be expected that different uncertainty estimators are more able/prone to capturing some types of uncertainty than others. To understand these models better, we thus require more granular definition to characterize

---

Table Comparison with related work in UQ We aim to provide flexible framework for inspecting mistrusted examples identified by uncertainty estimators. Furthermore, this framework enables us to improves the prediction performance on a certain type of flagged uncertain class.
Model
Uncertainty Estimation
Categorize Uncertainty
Improve Prediction
Method
Examples
Structure
BNNs
Bayesian Layers
g 39
GP
Gaussian Processes
MC-Dropout
Drop-Out Layers
10
Deep-Ensemble
Multiple Models
6
ICP
Model "Wrapper"
P3
40
Performance Prediction
Multiple Predictive Models
3y
18
DAUC
Assumption
(Ours)
uncertain data points. We employ three classes: outliers (OOD), boundary examples (Bnd) and examples of regions with high in-distribution misclassification (IDM), see Fig T
Out-of-distribution (OOD) detection Recall that UQ methods have been used to flag OOD examples. For completeness, we highlight that other alternative methods exist for OOD detection. For example, Lee al. 28 detects OOD examples based the Mahalanobis distance, whilst Ren et al. [29] uses the likelihood ratio between two generative models. Besides supervised learning, OOD is an essential topic in offline reinforcement learning [30 Sun et al. [36] detects the OOD state in the context of RL using confidence intervals. We emphasize that DAUC's aim is broader- a unifying framework for categorizing multiple types of uncertainty- could be used to replace DAUC's OOD detector.
existing OOD methods
Accuracy without Labels We contrast work to the literature which aims determine model accuracy without access to ground-truth labels. Methods such as [37 388 propose a secondary regression model an accuracy predictor given data sample. Ramalho and Miranda [8]] combine regression model with latent nearest neighbors for uncertain prediction. This is different from our setting which is focused on inspecting UQ methods on sample level by characterizing it as an outlier, boundary, or IDM example. We contrast DAUC with related works in Table [
3 Categorizing Model Uncertainty via Latent Density
3.1 Preliminaries
We consider a typical classification setting where X' C Rd is the input space and y [0, 1] is the set of class probabilities, where dx is the dimension of input space and C E No the number of classes. We are given a prediction model f X + y that maps E X to class probabilities (()) E y. An uncertainty estimator u X + [0, 1] quantifies the uncertainty of the predicted outcomes. Given some threshold т, the inference-time predictions can be separated into trusted predictions (x E X()(()) < r} and untrusted predictions {a E X)(()) 2 T). We make the following assumption on the model architecture of
Assumption 1 (Model Architecture). Model f can be decomposed as f= s ooooooooooooooooooooo.oooooooo where g : X - H C (Rda is a feature extractor that maps the input space to a du dx dimensional latent (or representation) space, 1: + RRC is a linear map between the latent space to the output space and xp: C + y is a normalizing map that converts vectors into probabilities.
Remark 1. This assumption guarantees that the model is endowed with lower-dimensional representation space. Most modern uncertainty estimation methods like MCD, Deep-En -neembee,,,,,, BNN satisfy this assumption. In the following, we use such space to categorize model uncertainty.
We assume that the model and the uncertainty estimator have been trained with a set of N E N* training examples Darain {(x", ,y") n E [N]]. At inference time the underlying model f and uncertainty method u predict class probabilities f(x) E y and uncertainty u(x) E [0, respectively. We assign a class to this probability vector f(x) with the map class y [C| that maps probability vector y to the class with maximal probability class[y] = arg max Ve. While uncertainty estimators flag examples to be trustworthy or not, those estimators do not provide fine-grained reason for what certain prediction should not be mistrusted. Our aim is to use the
3

---

model's predictions and representations of a corpus of labelled examples- which we will usually take to be the training (Drriin) or validation vDi sets- categorize inference-time uncertainty predictions. To that aim, we distinguish two general scenarios where a model's predictions should be considered with skepticism.
3.2 Flagging OOD Examples
There is limit to model generalization. Uncertainty estimators should be skeptical when the input E X' differs significantly from input examples that the model was trained on. From an UQ perspective, the predictions for these examples are expected to be associated with a large epistemic uncertainty.
natural approach to flagging these examples is to define a density p Derain): X R$ over the input space. This density should be such that p pxx Dirain) is high whenever the example x E resembles one or several examples from the training set Dtrain Conversely, a low value for p (x: (rrriin) ) indicates that the example differs from the training examples. Of course, estimating the density (x (Drainn is a nontrivial task. At this stage, it is worth noting that this density does not need to reflect the ground-truth data generating process underlying the training Prain- For the problem at hand, this density (<: 'Durain) need only measure how close the example x is to the training data manifold. A common approach build kernel density estimation with the training set Draain Further, we note that Cosuupttoon[prvvviiiii learned for the classification task on Dusin- In Appendix A.I1 we argue that this latent space is suitable for our kernel density estimation. This motivates the following definition for (- Deraan).
a representation space H that was specifically
Definition (Latent Density), Let f X 3 be a prediction model, let g X + H. be the feature extractor from Assumption [ and let H H IR+ be kernel function. The latent density p. ( DD X + IR+ is defined over a dataset D as:
Σ K xlo(z),g(T))
N
RED
Test examples with low training density are likely to be underfitted for the model - thus should not be trusted.
Definition 2 (OOD Score). The OOD Score TooD is defined as
TooD(x) p( Perain)
For test example x E Drest, if (TToo(((Duuall 2 TOOD the example suspected to be an outlier with respect to the training set. We set TOOD c p(x' [Prran)) i.c. new sample's 's training density is smaller than the minimal density of training examples.
3.3 Flagging IDM and Boundary Examples
Samples that are not considered outliers, yet given high uncertainty scores, divide up further into two von-exclusionary categories. The first category consists of points located near the boundary between two or more classes, the second consists of points that are located in regions of high misclassification.
For achieving this categorization, we will use a separate validation set D We first partition the vali- dation examples according to their true and predicted label. More precisely, for each couple of classes (c1, 02) [C]2, we define the corpus Cor = {(x, y) E Dval class [y] C₁ A class [f (x)) 02) of validation examples whose true class is ej and whose predicted class is c2. In the case where these two classes are different 6₁ * C22 this corresponds to corpus of misclassified examples. Clearly, some example x E resembles one or several examples of those misclassification corpus, is legitimate to be skeptical about the prediction f(x) for this example. In fact, keeping track of the various misclassification corpora from the validation set allows us to have an idea of what the misclassification is likely to be.
In order to make this detection of suspicious examples quantitative, we will mirror the approach from Definition Indeed, we can define kernel density p (- Cor+oe)): X + B for each corpus

---

Carto Again, this density will be such that (- Cc,+te2) ) is high whenever the representation of the example æ E X' resembles the representation of one or several examples from the corpus -re If this corpus is corpus of misclassified examples, this should trigger our skepticism about the model's prediction f' f())). By aggregating the densities associated with each of these corpora, we arrive the following definition.
Definition 3 (Confusion Density Matrix), Let f x + +V be a prediction model, ler g g.:. X - H be the feature extractor from Assumption [ and let N. H. x H. + Rt be kernel function. The confusion density matrix P-- Dval) + (R (B+))xc )CCC defined as
Per,ez (x (vaa)) p (z) Cerrre2)
(lg()). [C]?
(3)
BEC 102
The name confusion density chosen to make a parallel with confusion matrices. Like confusion matrices, our confusion density indicates the likelihood of each couple C2), where eı is the true class and 02 is the predicted class. Unlike confusion matrices, our confusion density provides an instance-wise (i.e. for each x E 2) likelihood for each couple.
Remark
3.3.1 Bnd examples
By inspecting the confusion matrix, we can quantitatively distinguish two situations where the example x E X' is likely to be mistrusted. The first situation where high uncertainty arises, is when g(x) is close latent representations of validation examples that have been correctly assigned label that differs from the predicted one class((()). This typically happens when g(z) is located close to decision boundary in latent space. In our validation confusion matrix, this likelihood that x is related to validation examples with different labels is reflected by the diagonal elements. This motivates the following definition.
Definition 4 (Boundary Score). Let P(x [x] Dval) ) be the confusion density matrix for an example X with predicted class = class| [ [x)]. We define the boundary score as the sum of each density of well-classfied examples from a different class:
TBnd (I)
Po.o (x Dval).
Points are identified as Bnd when TBad -Bad see Appendix A.2
3.3.2 IDM examples
The second situation is the one previously mentioned: the latent representation g(x) is close to latent representations of validation examples that have been misclassified. In our validation confusion matrix, this likelihood that x is related to misclassified validation examples is reflected by the off-diagonal elements. This motivates the following definition.
Definition (IDM Score). (xx vat be the confusion density matrix for an example E X. We define the IDM score as the sum of each density corresponding to misclassification of the predicted class = class f(x in the confusion density matrix:
C
TIDM(x) E Pa.e (o Dval).
(5)
Points are identified as IDM when TIDM TIDM- We choose TDMM such that the proportion of IDM points in the validation set equals the number of misclassified examples. Details for definitions and choices of thresholds are provided in Appendix A.2
Remark Note that the definitions of Bnd examples and IDM examples do not exclude each other, therefore, uncertain example can be flagged as Bnd example, an IDM example, flagged as both Bnd and IDM (B&I). To make this distinction clear, we will refer to the disjoint classes as:
SBad {x|x Drest, TBnd(x) (x) TBud, TIDM(x) TIDM}
SIDM = {x/1 E Drest, TBad (I) < Tudd TIDM (x) > TIDM}
SBLI {x|x Drest; Tind (x) Tund TiDM(I) TIDM]
Test examples that are flagged by uncertainty method v. w-yet not meet any the thresholds are marked as Other.
5

---

In a nutshell DAUC uses the OOD, IDM and Bnd classes to categorize model uncertainty see Table 2 for an overview. Better predictions may be possible for IDM samples, in case different classifier is used. For samples that are also labelled as Bnd, fine-tuning the existing model-possine after gathering more data-may be able to separate the different classes better. IDM samples that are not in the Bnd class are harder, and may only be classified correctly if a different latent representation found, or an different training set is used. We explore the idea of improving the performance on uncertain examples in Appendix B
In Section 4.1 we explore the distinction between classes further.
Table 2: Summary of different uncertainty types
Type
Definition
Description
0OD
(Pran)) TOOD
Samples that not resemble the training data. Additional la- belled data that covers this part of the input space is required to improve performance on these samples. Samples near the boundaries in the latent space. Predictions on these samples are sensitive to small changes in the predictor, and fine-tuning the prediction model may yield better predictions. Samples that are likely to be misclassified, since similar examples were misclassified in the validation set.
Bnd
Cape (x.o((DDea)) > TAnd
IDM
Enpa Po >
7IDM
Experiments
In this section, we demonstrate our proposed method with empirical studies. Specifically, we use two experiments Proof-of-Conee Concept, and two experiments as Use Cases. Specifically, in Sec. 4.
we
visualize the different classes of flagged examples on a modified Two-Moons dataset; in Sec. 4.2 we
quantitatively assess DAUC's categorization accuracy on the Dirty-MNIST dataset [41]: in Sec.
4.3
we present use case of DAUC MAAUC-comparing existing uncertainty estimation benchmarks; in Sec. we demonstrate another important use case of DAUC-improving uncertain predictions.
Our selection of the DiNIT dataset for empirical evaluation was motivated by the pursuit of better reproducibility. As an existing publicly available resource, Dirty-MNIST provides gold labels for boundary classes and OOD examples, making particularly suitable for benchmarking the performance of DAUC.
Recognizing the importance of demonstrating the broader applicability of DAUC. we have extended our evaluation to include results on the Dirty-CIFAR dataset. Details of this additional evaluation available in Appendix C.4, where we also describe how we created the dataset. This dataset will also be made publicly available. Additional empirical evidence that justifies DAUC is provided in Appendix C
4.1
Visualizing DAUC with Two-Smiles
4.1.1
Experiment Settings
To highlight the different types of uncertainty, we create a modified version of the Two-Moons dataset, which we will call "Two-Smiles". We use scikit-learn's 's datasets package to generate 6000 two-moons examples with noise rate of 0.1. In addition, we generate two Gaussian clusters of 150 examples centered at (0, 1.5) and (1, -1) for training, validation and test, and mark them as the positive and negative class separately. The data is split into training, validation and test set. We add additional 1500 OOD examples to the test set, that are generated with a Gaussian distribution centered at (2, 2) and (-1, 1.5). Overall, the test set counts 4500 examples, out of which 1500 are positive examples, 1500 are negative examples and 1500 are OOD- see Figure (a).
4.1.2 Results
We demonstrate the intuition behind the different types of uncertainty, by training a linear model to classify the Two-Smiles data-i.e. using a composition of an identity embedding function and linear classifier, see Assumption T Figure shows the test-time misclassified examples. Since the identity function does not linearly separate the two classes and outliers are unobserved at training

---

(a) Test Dataset
(b) Test Errors
(c) Flagged OOD
(d) Flagged Bnd
(e) Flagged IDM
Figure Visualization of our proposed framework Two-Smiles dataset with linear model.
time, the classifier makes mistakes near class boundaries, in predicting the OOD examples, as well that DAUC correctly flags the OOD examples, boundary examples and the IDM examples. Also note the distinction between SB&I and SIDM- Slightly fine-tuning the prediction model may yield a decision boundary that separates the SB&I samples from the incorrect class. On the other hand, this will not suffice for the SIDM examples-i.e. the clusters at (0, 1.5) and (1,-1)-whicl require significantly different representation/predictoonn model to be classified correctly.
the clusters (0, 1.5) and -1) In Figure 2 (c)-e),
4.2 Verifying DAUC with Dirty-MNIST:
4.2.1
Experiment Settings
We evaluate the effectiveness of DAUC the Dirty-MNIST dataset [41] The training data set is composed of the vanilla MNIST dataset and Ambiguous-MNIST. containing ambiguous artificial examples-e.g, digits that look like 9s. The test set is similar but also contains examples from Fashion-MNIST as OOD examples. The advantage of this dataset is that the different data types roughly correspond the categories want to detect, and such we verify whether DAUC's classification corresponds to these "ground-truthe labels. Specifically, we deem all examples from MNIST be non-suspicious, associate MNIST with class OOD (There are 1, 000 OOD examples out of the 11, 000 test examples). See 4]] for more details on the dataset. We use 3-layer CNN model with ReLU activation and MaxPooling as the backbone model in this section.
Babbiuoous-MNISS with class Bnd, and associate Fashion-
4.2.2 Results
(a) OOD
(b) Bnd
(c) IDM
(d) Flag Bnd
(e) Flag IDM
Figure 3: (a-c) Box- Box-plots of different data classes and corresponding scores. All values are scaled by being divided by the maximal value. (d-e) Precision- curves for different choices of thresholds in flagging Bnd examples and IDM examples.
assume the ground-truth label of Fashion-MNIST data is OOD. Com- paring these the classification by DAUC, of DAUC in identifying outliers. We compute precision, recall and Fi-scores for different classes, Figure 3 (a) and Table 3 All Fashion-MNIST examples are successfully flagged OOD by
Flagging Outliers
able to quantitatively evaluate the performance
DAUC. In Appendix we include more OOD experiments, including comparison to OOD detector baselines.
Table 3: Quantitative results on the Dirty-MNIST dataset. Our proposed method can flag all three classes of uncertain examples with high Fl-Score. The results presented in the table are based on repeated runs.
Category
Precision
Recall
F1-Score
OOD
1.000 + 0.000
1.000 + 0.000
1.000 + 0.000
Bnd
0.959 + 0.008
0.963 + 0.006
0.961 + 0.003
IDM
0.918 0.039
0.932 0.024
0.924 0.012

---

Class
Clsss
Cas
Cass
Cars
7
)
8
9
9
outlier
Class
Class
Class
Class
Closs
Class 5
7
9
6
9
BAA DNNTERN
Class:
Class
Class
Class
Class
Clacs
7
7
4
9
f
Figure 4: Examples of different uncertainty classes SOOD. SIDM. SBnd and SBAI. For better visualization, we only plot some classes including the outliers. t-SNE [42 is leveraged generating low-dim visualizations.
Flagging Bnd Examples We expect most of the boundary examples to belong to the Ambiguous- MNIST class, these have been synthesised using linear interpolation of two different digits in latent space 41). Figure (b) shows that DAUC's boundary scores are indeed significantly higher in Mubiguous-MNNIS compared to vanilla MNIST. Figure 3 (d) shows the precision-
curve of DAUC, created by varying threshold Bad- Most boundary examples are correctly discovered under a wide range of threshold choices. This stability of uncertainty categorization is desirable, since TBnd is usually unknown exactly.
Flagging IDM Examples In order to quantitatively evaluate the performance of DAUC flagging IDM examples, we use a previously unseen hold-out set, which is balanced to consist of 50% misclas- sified and 50% correctly classified examples. We label the former test-time IDM examples and the latter as non-IDM examples, and compare this to DAUC's categorization. Figure 3 shows DAUC successfully assigns significantly higher IDM scores to the examples that are to-be misclassified.
Varying TDMM we create the precision recall curve for the IDM class in Figure (c), which fairly stable w.r.t. the threshold TIDM- In practice, recommend to use the prediction accuracy the validation dataset for setting TIDM- see Appendix A.2
Visualizing Different Classes Figure4sshows examples from the SOOD, SBnd SIDM and Sp&I sets. The first row shows OOD examples from the Fashion-MNIST dataset. The second row shows boundary examples, most of which indeed resemble examples, which DAUC thinks are likely to be misclassified since mistakes were made nearby on the validation set. Indeed, these examples look like they come from the "dirty" part -irIT-MNIST, and most digits are not clearly classifiable. The last row contains B&I examples, which exhibit both traits.
than one class. The third row shows IDM
4.3 Benchmark Model Uncertainty Categorization
In this section, we demonstrate how DAUC categorizes existing UQ model uncertainty. We compare UQ methods MC-Dropout [T] (MCD), Deep-Ensemble (6) (DE) and Bayesian Neural Networks (BNNs). These methods output predictions and uncertainty scores simultaneously. we follow the traditional approach to mark examples as uncertain or trusted according to their uncertain scores and specified thresholds. To demonstrate how DAUC categorizes all ranges of uncertain examples, present results from top 5% to the least 5% uncertainty.
Figure 5 compares the proportion of different classes of flagged examples across the three UQ methods. The first and second rows show the total number and proportion of flagged examples for each class, respectively. There is significant difference in what the different UQ methods identify. There significant difference between types of classes that the different UQ methods identify as discussed in the literature [19 which have shown that some UQ methods might not yield high quality uncertainty estimates due to the learning paradigm or sensitivity to parameters. Let look at each column more carefully:

---

000
Brd
12M
01544
100
10p Uicettiryy
Uncenalrty
Uncortaryy
200 Hbygle Uicotan
+3n2/2 Urcomain
alpWaFiugee Unco 1a
#00hus2 Unooraa
0.2
MC0
M:0
PCD
MD
102
Txp Urcuriaryy
Unsoriainty
Uncotaariy
Figure 5: Results of applying method in categorizing different uncertainty estimation methods. First row; comparisons the numbers in different classes of examples. Sccond row: comparisons on the proportion of different classes flagged examples to total number of identified uncertain examples. Different methods tend to identify different certain types of uncertain examples. The results presented are based on 8 repeated runs with different random seeds.
1. DE tends to identify the OOD examples as the most uncertain examples. Specifically, looking at the bottom figure we see that the top 5% untrusted examples identified by DE are almost all OOD examples, which is not the case for the other UQ methods. By contrast, MCD is poor at identifying OOD examples; it flags some of the OOD samples as the most certain. This is explained by MCD's mechanism. Uncertainty is based on the difference in predictions across different drop-outs, however this could lead to outliers always having the same prediction- due to correlation between different nodes in the MCD model, extreme values of the OOD examples may always saturate the network and lead to the same prediction, even if some nodes are dropped out. DE is most apt at flagging the OOD class. This confirms the finding by (20) who showed that DE outperforms other methods under dataset shift which effectively what OOD represents.
2. After the OOD examples have been flagged, the next most suspicious examples are the Bnd and IDM classes see columns 2 and 3. The number of these examples increases almost linearly with the number of flagged examples, until at about 88% no more examples are flagged as IDM and Bnd. This behaviour is explained by the Vanilla MNIST examples- which are generally distinguishable and relatively casily classified correctly corcetly-accounting for about 15% of the test examples.
3. As expected, the number of examples belonging to the Other class increases when more examples are flagged as uncertain. This makes sense, as the Other class indicates we cannot flag why the methods flagged these examples uncertain, i.e. maybe these predictions should fact be trusted.
4.4 Improving Uncertain Predictions
In this section, we explore the inverse direction, i.e. employing DAUC's categorization for creating better models. We elaborate the practical method in Appendix B We experiment on UCT's Covtype, Digits and Spam dataset [3] with linear models (i.c. = Id) and experiment on DMNIST with ResNet- learning the latent representation.
Our empirical studies (Figure Appendix have shown that the B&I examples are generally hardest to classify, hence we demonstrate the use case of DAUC for improving the predictive performance on this class. We vary the proportion q of training samples that we discard before training new prediction model fB&I-only saving the training samples that resemble the B&I dataset most see Figure 6 We find that retraining the linear model with filtered training data according to Eq. 5 significantly improves the performance. We observe that performance increases approximately linearly proportional to q. until the amount of training data becomes too low. The latter depends on the dataset and model used.

---

Prediction Acc.
Al Rhaned Tha
0.
Barnd Tobiin
5.5
Kf Elmee Tranng Sir
53
040
5.0
5.9
6/
Qua
Quantle
Quantll
Quartik
Figure 6: Improved uncertain predictions. Left to right: Covtype, Digits, Spam, DMNIST. Experiment are performed with different proportion of training samples discarded: c.g., with = 0.0, all examples in the training set used; while with g 0.9, only top 10% examples most resembling the test data are used for training. The results presented are based 10 repeated runs with different random seeds.
Conclusion and Future Work
We have proposed DAUC, a framework for model uncertainty categorization. DAUC categorizes uncertain examples identified by UQ benchmarks into three classes-OOD, Bnd and IDM. These classes correspond different causes for the uncertainty and require different strategies for possibly better predictions. We have demonstrated the power of DAUC by inspecting three different UQ highighs-t that each one identifies different examples. We believe DAUC can aid the development and benchmarking of UQ methods, paving the way for more trustworthy ML models.
In future work, DAUC has great potential be extended more general tasks, such the regression setting, and reinforcement learning setting, where uncertainty quantification is essential. The idea of separating the source of uncertainty improves not only exploration [44 but also exploitation in the offline settings 35
In the of Large Language Models (LLMs) 45 46
uncertainty quantification is essential in
evaluating the task performance of LLMs [77. and holds great potential for AI aligmment [48 49)
understanding the ability boundary of LLMs is essential, and identifying the suspicious outputs of LLMs can be potentially addressed by extending the framework of DAUC to the LLMs' setting.
Acknoledgement
HS acknowledges and thanks the funding from the Office of Naval Research (ONR). We thank the van der Schaar lab members for reviewing the paper and sharpening the idea. We thank all anonymous reviewers, ACs. SACs, and PCs for their efforts and time the reviewing process and in improving our paper.
10

---

References
[1] Ravi Aggarwal, Viknesh Sounderajah, Guy Martin, Daniel SW Ting, Alan Karthikesalingam, Dominic King, Hutan Ashrafian, and Ara Darzi. Diagnostic accuracy of deep learning in medical imaging: systematic review and meta-analysis. NPJ digital medicine, 4(1):1-23, 2021.
[2] Alisa Kim, Y Yang, Stefan Lessmann, Tiejun Ma, M-C Sung, and Johnnie EV Johnson. Can deep learning predict risky retail investors? a case study in financial risk behavior forecasting. European Journal of Operational Research, 283(1):217-235
2020.
[3] Qianggang Ding, Sifan Wu, Hao Sun, Jiadong Guo, and Jian Guo. Hierarchical multi-scale gaussian transformer for stock movement prediction. In IJCAI, pages
2020.
[4] Jinkyu Kim and John Canny. Interpretable learning for self-driving cars by visualizing causal attention. In Proceedings of the IEEE international conference on computer vision, pages 2942 2950,, 2017.
[5] Jiankai Sun, Hao Sun, Tian Han, and Bolei Zhou. Neuro-symbolic program search for au- tonomous driving decision module design. In Jens Kober, Fabio Ramos, and Claire Tom- lin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Pro- ceedings of Machine Learning Research, pages 21-30. PMLR, 16-18 Nov 2021. URL https://proceeet
mlr pptps//15//ee/p/ppt v155/v1555//// sun21a.
[6] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 6405-6416, 2017.
[7] Charles Blundell, Julien Coreeiise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pages 1613-1622. PMLR, 2015.
[8] Alex Graves. Practical variational inference for neural networks. Advances in neural information processing systems, 24, 2011.
[9] Soumya Ghosh, Jiayu Yao, and Finale Doshi-Velez Structured variational learning of bayesian neural networks with horseshoe priors. In International Conference Machine Learning, pages 1744 1753. PMLR, 2018.
[10] Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059, PMLR, 2016
[11] Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. arXiv preprint 1802: 2018
[12] Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. In International Conference on Machine Learning, pages 9690-9700, PMLR, 2020.
[13] Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 2021.
[14] Xuming Ran, Mingkun Xu, Lingrui Mei, Qi Xu, and Quanying Liu. Detecting out-of-distribution samples via variational auto-encoder with reliable uncertainty estimation. Neural Networks, 145:199-208 208, 2022.
[15] Dimitris Tsipras. Shibani Santurkar, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. From imagenet to image classification: Contextualizing progress on benchmarks. In Interna- tional Conference on Machine Learning, pages 9625-9635. PMLR, 2020.
11

---

[16] Ikki Kishida and Hideki Nakayama. Empirical study of easy and hard examples in enn training. In International Conference on Neural Information Processing, pages 179-188. Springer, 2019.
[17] Rajmadhan Ekambaram, Dmitry Goldgof, and Lawrence O Hall. Finding label noise ex- amples in large scale datasets. In 2017 IEEE International Conference Systems, Man, and Cybernetics (SMC), pages 2420-2424. IEEE, 2017.
[18] Tiago Ramalho and Miguel Miranda. Density estimation in representation space to predict model uncertainty. In International Workshop on Engineering Dependable and Secure Machine Learning Systems, pages 84-96. Springer, 2020.
[19] Jiayu Yao, Weiwei Pan, Soumya Ghosh, and Finale Velhi-Velez. Quality of uncertainty quantification for bayesian neural network inference. arXiv preprint
2019.
[20] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's 's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems, 32:13991- 1-9902- 2019.
[21] Andrew YK Foong, David R Burt, Yingzhen Li, and Richard E Turner On the expressiveness of approximate inference in bayesian neural networks. arXiv preprint XXXv:1909000719, 2019.
[22] Francesco Verdoja and Ville Kyrki. Notes on the behavior of me dropout. arXiv preprint arXiv:2008.02627, 2020.
[23] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in random world. Springer Science & Business Media, 2005.
[24] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 30:5574-5584, 2017.
[25] Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243-297, 2021. ISSN 1566-
2535. doi: 10. 10.1016/j.mm wt..00666..nfffs.220. 20ff .5..000.
[26] Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning 2021 110:3, 110(3): 457-506, 2021. ISSN 1573-0565. doi: 10. 100777110999--1---99966.3
[27] Lisa Schut, Oscar Key, Rory Mc Grath, Luca Costabello, Bogdan Sacaleanu, Yarin Gal, et al. Generating interpretable counterfactual explanations by implicit minimisation of epistemic and aleatoric uncertainties. In International Conference on Artificial Intelligence and Statistics, pages 1756-1764. PMLR, 2021.
[28] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting outtof--iitributtin samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
[29] Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. Advances in Neural Information Processing Systems, 32: 1770777114,7,8 2019.
[30] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint urXiv:2005.01643 2020.
[31] Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking coal-conddtioned supervised learning and its connection to offline rl. arXiv preprint arXiv:2202.04448, 2022.
[32] Xiaoyu Wen, Xudong Yu, Rui Yang, Chenjia Bai, and Zhen Wang. Towards robust offline-to- online reinforcement learning via uncertainty and smoothness. arXiv preprint arXiv:2309. 16973, 2023.
12

---

[33] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Ro- bust offline reinforcement learning via conservative smoothing. Advances in Neural Information Processing Systems, 33:23851-238666 2022.
[34] Rui Yang, Lin Yong, Xiaoteng Ma, Hao Hu, Chongjie Zhang, and Tong Zhang. What is essential for unseen goal generalization of offline gonl-condi rl? In International Conference on Machine Learning, pages 19543-39577 PMLR, 2023.
[35] Zhihan Liu, Yufeng Zhang, Zuyue Fu, Zhuoran Yang, and Zhaoran Wang. Learning from demon- stration: Provably efficient adversarial policy imitation with linear function approximation. In International Conference on Machine Learning, pages 14094-14138, PMLR, 2022.
[36] Hao Sun, Alihan Hüyük, Daniel Jarrett, and Mihaela van der Schaar. Accountability in offline reinforcement learning: Explaining decisions with corpus of examples. arXiv preprint arXiv:2310.07747, 2023.
[37] Sindhu Ghanta, Sriram Subramanian, Lior Khermosh, Harshil Shah, Yakov Goldberg, Swami- nathan Sundararaman, Drew Roselli, and Nisha Talagala. {MPP): Model performance predictor. In 2019 {USENIX} Conference on Operational Machine Learning (OpML 19), pages 23-25, 2019.
[38] Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15069-15078, 2021.
[39] Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning. pages 63-71. Springer, 2003.
[40] Vineeth Balasubramanian. Shen-Shyang Ho, and Vladimir Vovk. Conformal prediction for reliable machine learning: theory, adaptations and applications. Newnes, 2014.
[41] Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal. De- terministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty. arXiv preprint arXiv: 102. 1582, 2021.
[42] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (11). 2008
[43] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. Uttp::::
ics wui. www.uuuum/li....c.
[44] Daniel Jarrett, Corentin Tallec, Florent Altché, Thomas Mesnard, Rémi Munos, and Michal Valko. Curiosity in hindsight. arXiv preprint arXiv:2211.10513. 2022.
[45] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[46] OpenAI. Gpt-4 technical report. arXiv, pages 2303-08774 2023.
[47] Hao Sun. Offline prompt evaluation and optimization with inverse reinforcement learning. arXiv preprint (tt.::20000655333 2023.
[48] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2244522204.05862. 2022.
[49] Hao Sun. Reinforcement learning in the era of lmm: What is essential? what is needed? an rl perspective on rlhf, prompting, and beyond. arXiv preprint arXiv:2310.06144 2023.
[50] Jonathan Crabbé, Zhaozhi Qian, Fergus Imrie, and Mihaela van der Schaar. Explaining latent representations with a corpus of examples. NeurIPS 2021, 2021.
13

---

[51] Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00805. 2017.
[52] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv:1412.6572, 2014.
[53] Arnaud Van Looveren, Janis Klaise, Giovanni Vacanti, Oliver Cobb, Ashley Scillitoe, and Robert Samoilescu. Alibi detect: Algorithms for outlier, adversarial and drift detection, 2019.
[54] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining, pages 413-422. IEEE, 2008.
[55] Simon J Sheather and Michael C Jones. A reliable kernel density estimation. Journal of the Royal Statistical Society: Series (Methodological), 53(3):683-690, 1991.
bandwidth selection method for
[56] David W Scott. Multivariate density estimation and visualization. In Handbook computational statistics, pages 549-569 Springer, 2012.
[57] Bernard W Silverman. Density estimation for statistics and data analysis. Routledge, 2018.
[58] Soumya Ghosh, Q. Vera Liao, Karthikeyan Natesan Ramamurthy, Jiri Navratil, prasanna Sattigeri, Kush Varshney, and Yunfeng Zhang. Uncertainty quantification 360: holistic toolkit for quantifying and communicating the uncertainty of ai, 2021.
[59] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2832 2011.
14

---

A Missing Details
A.1 Motivations for working with model latent space
In Section 3 we introduced the confusion density matrix that allows us to categorize suspicious examples at testing time. Crucially, this density matrix relies on kernel density estimations in the latent space H. associated with the model f through Assumption [ Why are we performing a kernel density estimation in latent space rather than in input space X The answer is fairly straightforward: we want our density estimation be coupled to the model and its predictions.
Let us now make this point more rigorous. Consider two input examples 21, E X. The model assigns representations g(x1), (12)) E H. and class probabilities fxx1), fxx22 y. If we define our kernel A: in latent space H. this often mean that x([((1)), (x2()) grows as (9(x)) - g(x2) |l H decreases. Hence, examples that are assigned a similar latent representation by the model are related by the kernel. Since our whole discussion revolves around model predictions, we would like to guarantee that two examples related by the kernel are given similar predictions by the model f. In this way, we would be able to interpret large kernel density k[g(z1). (x2)] as hint that the predictions (x11 and f(r)) are similar. We will now show that, under Assumption such a guarantee exists. Similar to [50], we start by noting that
[I o (21) - (I (ogg (z
111 [0(21)) - g )ll S 1141 (l)())) ) - g222))
where I|- is a norm on RC and |4]pp is the operator norm of the linear map l. In order to extend this inequality to black-box predictions, we note that the normalizing map in Assumption i often Lipschitz function with Lipschitz constant À E R. For instance, Softmax function with inverse temperature constant X-1 is X-Lipschitz [51]. class probabilities:
We use this fact to extend our inequality to predicted
(((a)) = ol - (volo (12))ly
-
S S [[l g) - (I (12))le
S 1141 (19(x1) g(22)l
This crucial inequality guarantees that examples Iı, T2 E X' that are given a similar latent representa- tion ((11) = g(222 will also be given similar prediction f(11) = f(2)). In short: two examples that are related according to a kernel density defined in the model latent space H. are guaranteed to have similar predictions. This is the motivation wanted support the definition of the kernel x in latent space.
An interesting question remains: is it possible to have similar guarantees if we define the kernel in input space? When we deal with deep models, the existence of adversarial examples indicates the opposite [52). Indeed, if 12 is an adversarial example with respect to I1, we have 21 x2 (and hence
||I| x2 xx small) with two predictions f(x1) and (x2) that are significantly different. Therefore, defining the kernel x in input space might result in relating examples that are given significantly
different prediction by the model. For this reason, we believe that the latent space is more appropriate in our setting.
A.2 Details: Flagging IDM and Bnd Examples with Thresholds
In order to understand uncertainty, it will be clearer to map those scores into binary classes with thresholds. In our experiments, we use empirical quantiles as thresholds. e.g., to label an example as IDM, we specify an empirical quantile number g, and calculate the corresponding threshold based on
5d(3) s(IP...s!) where s denotes the the order statistics of IDM Scores for test examples: IDDM IDM
n-th smallest IDM score out of (Dresul testing-time time examples. Then, the threshold given quantile number g is
fl Pr l-1))
( =
IDDM
"This is the case for all the kernels that rely on a distance (e.g. the Radial Basis Function Kernel, the Matern kernel even Polynomial kemnlss (39).
15

---

Similarly, we can define quantile- quantile-based based threshold in flagging Bnd examples based on the order statistics of Bnd Scores for test examples, such that for given quantile 9,
L/ -9)) Bud
Bud (g)
Practically, natural choice of q is to the validation accuracy: when there are 1 g examples misclassified in the validation set, we also expect the testing-time in distribution examples with the highest g be marked Bnd or IDM examples.
B Improving Predicting Performance of Uncertain Examples
Knowing the category that a suspicious example belongs to, can we improve its prediction? For case of exposition, we focus improving predictions for SB&1-
Let ( SB&) be the latent density be defined as Definition We can improve the prediction performance of the model on SB&I examples by focusing on the part of examples in the training set that are closely related those suspicious examples. We propose to refine the training dataset Dirain by only keeping the examples that resembles the latent representations for the specific type of test-time suspicious examples, and train another model on this subset of the training data:
Dirain = (x Dirain |p (æ: (S&r)) 2 T
(6)
where Twst is threshold that be adjusted to keep prespecified proportion q of the related training data. Subsequently, new prediction model fB&1 is trained Drrain-
Orthogonal to ensemble methods that require multiple models trained independently, and improve overall prediction accuracy by bagging boosting, our method is targeted at improving the model's performance on a specified subclass of test examples by finding the most relevant training examples. Our method is therefore more transparent and can be used in parallel with ensemble methods if needed.
Threshold (g) For every training example x E Dirain. we have the latent density p(a: DR&T) over the B&I class of the test set. With their order statistics P(1) (a(BaB))) .... P(] 1) (a) DB&I). Given quantile number g, our empirical quantile based threshold Trest is chosen as
Trest (4) = (Leeill) (r))).
During the inverse training time, we train our model to predict those B&I class of test examples only with the training data with higher density than (q). We experiment with different choices of q in the experiment (Figure 6 Sec. 4.4)
C
Additional Experiments
C.1 Categorization of Uncertainty under Different Thresholds
In the main text, we provide results with TBd = TIDM = 0.8, which approximates the accuracy on validation set-as natural choice. In this section, vary these thresholds and show in Figure that changing those thresholds does not significantly alter the conclusions drawn above.
Figure 8 looks more closely into the top 25% uncertain examples for each method, and the accuracy on each of the uncertainty classes. As expected, the accuracy of the B&T examples always lower than that of the trusted class, meaning that those examples are most challenging for the classifier. And the accuracy of flagged classes are always lower than the other class, verifying the proposed categorization of different classes.
C.2 Inverse Direction: More Results
In the main text, show the results on improving prediction performance the B&I class with training example filtering (On the Covtype, Digits dataset). More results on other classes examples provided in this section.
We experiment on three UCI datasets: Covtype, Digits, and Spam. And experiment with three classes we defined in this work:
16

---

000
CN
Cther
100
MD3
500
M70
Uncerta
Top Unioetaiiyy
(a) THD (0.5), (0.5(
100
1100
100
110
1880
Uncertaiity
Uncertaity
10p Unie6tairy
(b) (BD(0.6), (DM(())
000
Other
8
MG3
110
MCD
305 Shoertality
Top Uncarta my
Tap Incorta mly
Top Un Catcy
(c) TBD (0.7), (0.7)
Other
800
230
30
201
250
180
130
(d) (0.(0.8), (0.8)
o0n
Other
NG
MDD
305 Shoortaitty
Top Uncaraaityy
Top Incoriarty
Top Un Coteaay
(c) (0.9), TIDM (0.9)
Figure 7: Experiments on different choices of thresholds. The results presented are based on 8 repeated runs with different random seeds.
1. B&I class (Figure 9. As have discussed in our main text, the prediction accuracy on the B&I class are always the lowest among all classes. By training with filtered examples in Dirain rather than the entire training set, the B&I class of examples can be classified with remarkably improved accuracy.
2. Bnd class (Figure 10). This class of examples are located at boundaries in the latent space of validation set, but not necessarily have been misclassified. Therefore, their performance baseline (training with the entire D Pussin is relatively high. The improvement is clear but not as much as on the other two classes
3. IDM class (Figure TTI. For this class of examples, similar mistakes have been make in the validation set, yet those examples are not necessarily located in the boundaries the misclassification may be caused by ambiguity decision boundary, imperfectness of either the model or the dataset. The primal prediction accuracy on this class of examples is lower than the Bnd class but higher than the B&I class, training with filtered Derain also clearly improve the performance on this class of examples.
17

---

56.53%
02.028
65.03%
Acc. 52.07%
Act. 61.41%
Act. 58 151
Other Acc. $4.9%
Other Acc. /4.42%
Other Acc. 68.994
(a) BNNs
MCD
(c) DE
Figure 8: The top 25% uncertain examples identified by different methods. Legend of each figure provide the accuracy and proportion of cach class. As the classifier can not make correct predictions the 0OD examples, it's always better for uncertainty estimators flag more OOD examples.
Table 4: DAUC is not the only choice in identifying OOD examples. On the Dirty-MNIST dataset, DAUC, Outlier-EE and the IForest can identify most outliers in the test dataset. (Given threshold 1.0 for those two benchmark methods).
Method
Precision
Recall
F1-Score
DAUC
1.0000 + 0.0000
1.0000 = 0.0000
1.0000 + 0.0000
Outlier-AF AE
1.0000 0.0000
1.0000 = 0.0000
1.0000 0.0000
IForest 54
0.9998 + 0.0004
1.0000 + 0.0000
0.9999 + 0.0002
C.3 Alternative Approach in Flagging OOD
As have mentioned in the main text, although DAUC has unified framework in understanding all three types uncertainty the uncertainty caused by OOD examples can also be identified by off-the-shelf algorithms. We compare DAUC to two existing outlier detection methods in Table 4 where methods achieve good performance on the Dirty-MNIST dataset. Our implementation is based on Alibi Detect [53]
0.45
Training
Trairing
0.00
W/ Filtered Training Set
Prediction Acc.
Baseline
Baseline
0.79
0.55
0.05
0.30
0.00
0.5
0.00
6.10
0.20
0.30
8.30
10
0.70
0.00
0.10
0.20
0.70
0,93
030
40
0.60
0.70
0.50
ag3o
Quantile
Quantile
Quantile q
(a) Covtype
(b) Digits
(c) Spam
Figure 9: Experiments the B&I class (reported in the main text). The results presented are based on 10 repeated with different random seeds.
C.4 Experiments on Dirty-CIFAR-10
Dataset Discription In this experiment, we introduce revised version of the CIFAR-10 dataset to test DAUC's scalability. Similar to the DirST datset [4I), we use linear combinations of the latent representation to construct the "boundary" class. the original CIFAR-10 Dataset, each of the 10 classes of objects has 6000 training examples. We split the training set into training set (40%), validation set (40%) and test set (20%). verify the performance of DAUC in detecting OOD examples, randomly remove one of those 10 classes (denoted with class-i) during training
18

---

0.34
Training
Trairing
0.05
Filtered Training Set
Prodiction AcC.
Baseline
Baseline
0.02
0.00
0.46
6.2
0.1
0.78
6.11
0.00
0.10
0.20
0.70
0.00
0.10
0.00
6.10
0.30
40
60
0.80
Quantile
Quantile
Quantile q
(a) Covtype
(b) Digits
(c) Spam
Figure 10: Experiments the Bnd class. The results presented are based 10 repeated runs with different random seeds.
Prediction Acc.
0.48
Baseline
wr Fitered Training Set
w/ Fitered Traringg Set
0.00
W Filtered Training Set
Baseline
0.40
0.70
0:38
0.40
6.2
0.00
0.00
0.0
0.30
0.00
0.30
8
0.00
0.10
0.20
0.30
6.
og.
Quantile
Quantile
Quantile q
(a) Covtype
(b) Digits
(c) Spam
Figure 11: Experiments on the IDM class. The results presented are based on 10 repeated runs with different
random seeds.
and manually concatenate OOD examples with the test dataset, with label i. In our experiment, we 1000 MNIST digits the OOD examples, with vero-padding to make those digits share the same input shape as the CIFAR-10 images. Combining those boundary examples, OOD examples and the vanilla CIFAR-10 examples, we get new benchmark, dubbed Dirty-CIFAR-10, for quantitative evaluation of DAUC.
Quantify the performance of DAUC Dirty-CIFAR-10 Quantitatively, evaluate the per- formance of DAUC in categorizing all three classes of uncertain examples. Results of averaged performance and standard deviations based on repeated runs are provided in Table 5
Table 5: Quantitative results on the Ditty-IIFRR-I dataset. DAUC scales well and is able to categorize all three classes of uncertain examples. Results presented in the table are based on repeated runs with different random seeds.
Category
Precision
Recall
Fl F1-Score
OOD
0.986 0.003
0.959 + 0.052
0.972 0.027
Bnd
0.813 + 0.002
0.975 + 0.000
0.887 + 0.001
IDM
0.688 0.041
0.724 + 0.017
0.705 + 0.027
Categorize Uncertain Predictions Dirty- CIFA- Similar Sec. 4.3 and Figure 5 we can categorize uncertain examples flagged by BNNs, MCD, and DE using DAUC see Figure T2 We find that the experiment with CIFAR- 10, DE tends discover more OOD examples as top uncertain examples. Differently, although BNNs flag fewer OOD examples as top-uncertain, they continuously discover those OOD examples and are able find most of them for top 50% uncertainty. On the contrary, MCD performs the worst among all three methods, similar to the result is drawn from the DMNIST experiment. On the other hand, while BNN is good identifying OOD examples, flags less uncertain examples in the Bnd and IDM classes. DE is the most apt at flagging both Bnd and
19

---

IDM examples and categorizes far fewer examples into the Other class. These observations are well aligned with the experiment results we had with DMNIST Sec. 4.3 showing the scalability of DAUC to large-scale image datasets.
000
Brd
IDM
Other
1130
135
V7D
M0
PTD
MC
Top Untersaity
10p Unceraiity
10p Uncenaaity
Uncora
Hbgge Uncotan
+andloRayged Urcorain
alpWaFiugee Uncertah
+c0hso2Raggod Unoorga
MC0
MCD
PCD
M0
Top to
Uncoriantte
Uncerarity
Figure 12: Experiments the CIFAR- 10 datasct. Results of applying DAUC in categorizing different uncertainty estimation methods. First row: comparisons of the numbers in different classes examples. Second row: comparisons on the proportion of different classes of flagged examples the total number identified uncertain examples. Different methods tend to identify different certain types of uncertain examples. The results presented are based on 8 repeated runs with different random seeds.
D Implementation Details
D.1
Code
Our code is available at httppe//gghaaooooo////agecche///
D.2
Hyperparameters
D.2.1
Bandwidth
In our experiments, we use (z-score) normalized latent representations and bandwidth 1.0. In the inverse direction, as the sample sizes much smaller, bandwidth of 0.01 is used as the recommended setting. There is a vast body of research on selecting a good bandwidth for Kernel Density Estimation models [55-57] and using these to adjust DAUC's bandwidth a more informed choice may further improve performance.
D.3 Inverse Direction: Quantile Threshold
As depicted in Appendix A.2 natural choice of g is to use the validation accuracy. We use this heuristic approach in our experiments for the inverse direction.
D.4
Model Structure
In our experiments, we implement MCD and DE with 3-layer-CNNs with ReLU activation. Our experiments BNNs are based the IBM UQ360 software (58). More details of the convolutional network structure are provided in T
D.5 Implementation of Kernel Density Estimation and Repeat Runs
Our implementation of KDE models based on sklearn's 's KDE package [59]. Gaussian kernels are used as default settings. We experiment with 8-10 random seeds and report the averaged results and standard deviations. In our experiments, we find using different kernels in density estimation provides highly correlated scores. We calculate the Spearman's 's p correlation between scores DAUC gets over runs with Gaussian, Tophat, and Exponential kernels under the same bandwidth. Changing the kemel brings highly correlated scores (all above 0.86) for DAUC and, hence, has minor impact
20

---

Table 6: Network Structure
Layer
Unit
Activation
Pooling
Conv
(1,32,3, 3,1,1 1.
ReL.0
MaxPool2d(2)
Cony
(32, 64, 1,
ReLUU
MaxPool2d(2)
Conv
(64, 3,1,,
ReLUO
MaxPool2d(2)
FC
(64 x 3 x 3,40)
ReLUO
Out
(40, NNlass)
SoftMax()
DAUC's performance. We preferred KDE since the latent representation is relatively low-dimensional. We found that low-dim latent space (e.g., 10) works well for all experiments (including CIFAR-10).
D.6
Hardware
All results reported our paper are conducted with a machine with 8 Tesla K80 GPUs and 32 Intel(R) E5-2640 CPUs. The computational cost is mainly in density estimation, and for low-dim representation space, such an estimation can be efficient: running time for DAUC on the Dirty-MNIST dataset with KDE is approximately 2 hours.
Assumptions and Limitations
In this work, we introduced the confusion density matrix that allows us to categorize suspicious examples at testing time. Crucially, this density matrix relies kernel density estimations in the latent space H associated with the model f through Assumption holds for most modern uncertainty estimation methods.
We note this assumption generally
While the core contribution this work is to introduce the concept of confusion density matrix for uncertainty categorization, the density estimators leveraged in the latent space can be further improved. We leave this to future work.
Broader Impact
While previous works uncertainty quantification (UQ) focused on the discovery of uncertain examples, in this work, we propose a practical framework for categorizing uncertain examples that flagged by UQ methods. We demonstrated that such a categorization selection - different UQ methods are good at figuring out different uncertainty sources. Moreover, we show that for the inverse direction, uncertainty categorization can improve model performance.
be used for UQ method
With our proposed framework, many real-world application scenarios can be potentially benefit, c.g.. in Healthcare, patient marked uncertain that categorized OOD preferably identified by Deep Ensemble, as we have shown - should be treated carefully when applying regular medical experience; and uncertain case marked IDM preferably identified by MCD can be carefully compared with previous failure cases for tailored and individualized medication.
21


Reviews:
[{"review_id": "9oW2sasuE5", "reviewer": "Reviewer_sVgx", "summary": "Paper introduces a framework to detect and categorize different model uncertainty types in classification setting. Proposer model-agnostic (with some assumptions of model structure) uncertainty quantification (UQ) relies on kernel density estimation on latent space representation by defining scores for OOD, Bnd, and IDM examples. Proposed approach is empirically evaluated using toy data as well as real MNIST and UCI datasets. Some of the limitations of existing UQ are examined as well as ability to use new uncertainty categorisation as flagging method to filter out training example to learn more accurate models.", "questions": "- Table 1: Could you ellaborate why other related approaches could not be used to improve the prediction? (if threshold is determined for the uncertainty estimates these methods are utilising). I see prediction improvement more related to use case than general property of particular method directly. It might be not give better predictive accuracy than proposed method, but is doable.\n- Figure 4: What is clustering/projecting method used? Maybe mention in the text.\n- Figure 5 & 6: How error bars are calculated? Maybe mention in the text.\n\nMinor\n- DAUC is developed for classification tasks in specific. Maybe that could be mentioned in the abstract.\n", "limitations": "Some of the assumptions and limitations of proposed framework are listed in the supplementary material in relation to model structure and latent space density estimation. Furthermore, the usefulness of UQ are broader sense are mention as an important part of safety critical AI applications. No negative impact identified.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "To my knowledge, paper presents a novel approach to categorise different predictive uncertainty types in classification setting. The proposal is well-structured with appropriate theoretical background, and especially practical evaluation which includes both toy illustration of properties as well as real data evaluation of MNIST variants and UCI datasets. Proposed framework shows simple yet effective solution that seems to improve existing approaches in practice and is bringing new knowledge for analysing existing UQ methods.\n\nSummary of strengths\n- Novel view on uncertainty quantification\n- Work in practice, at least in given classification tasks\n- Well-made empirical evaluation (see also weaknesses)", "weaknesses": "There are some weaknesses that could further improve the presentation and usefulness of proposed framework. Analysis of different model architectures (i.e. latent space) with proposed density estimation could be further extend to show how these behave with different level of uncertainty in the datasets to justify the modelling choices. Also, the analysis of verifying DAUC concentrates only on MNIST variants, but it could be useful to examine different types of data with different noise and uncertainty levels.\n\nSummary of weaknesses\n- Limited analysis of model assumptions and density estimation\n- Somewhat limited type of datasets and model architectures evaluated\n- Some polishing of text here and there (see questions)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's thoughtful insights and constructive feedback. We will respond to each of the questions in turn.\n\n--- \n### Q1: Dataset and model architecture choices.\nA1: \nOur primary directive in selecting the Dirty-MNIST dataset was to anchor our work in a realm of reproducibility. Dirty-MNIST, beyond its public accessibility, is uniquely positioned with gold labels for boundary classes and OOD examples. This distinction positions it as an ideal choice in evaluating DAUC.\n\nIn regard to our model architecture choices, we use a prevailing structure referring to the implementation of MCD, BNNs, and DE — the uncertainty quantification methods we benchmarked in our paper. We would like to emphasize, as highlighted in Table 1, that DAUC is designed to be **flexible in terms of model structure**. It functions effectively under conditions where Assumption 1 can be met, a scenario commonly satisfied when using prevailing neural network architectures.\n\n**[Supplementary Experiments in Appendix]** Recognizing the importance of demonstrating the broader applicability of DAUC, we enriched our evaluation with the Dirty-CIFAR dataset. Limited by the space, we postponed details of this additional evaluation in Appendix C.4 for your reference. We would also make this dataset publicly available.\n\n**[Extended Analysis in the Attached PDF]** In order to verify the scalability of DAUC, we conduct **additional experiments on the CIFAR-100 dataset** where the number of classes scales up to 100. Given the limited time and the non-trivial efforts in creating boundary examples and corresponding gold labels, we would focus on the inverse direction as a downstream application in demonstrating DAUC’s scalability.\n\nThe results are shown in **Figure 1 in the attached 1-page pdf response**. For those experiments, we use ResNet-18 to learn representation and identify the IDM and Bnd examples marked as suspicious, and then improve the prediction on those subsets. DAUC scales well on the CIFAR-100 dataset — prediction performance of the classifiers can be improved through filtering similar examples in the training dataset.\n\n### Q2: Why other related approaches could not be used to improve the prediction?\nA2: \nWe would like to note that **DAUC’s capability of improving prediction is benefited by the uncertainty categorization**: To make this clear, we could consider a counter-example of why may other methods could fail in improving prediction: some methods may prioritize the predictions on the OOD examples as the most uncertain. E.g., the DE method tends to flag OOD examples as the most uncertain (as depicted in Figure 5).\n\nIn order to improve the predictions, we could use the categorization of uncertainty provided by DAUC: Since the original classification model is trained with the entire dataset and minimizes overall prediction error, it may perform poorly on the minor examples. And DAUC is able to identify the source of uncertainty in such cases. We use this insight to further improve the prediction of those examples with a specialized focus on them.\n\n### Q3: Figure 4: What is the clustering/projecting method used? \nA3: \nWe used the t-SNE clustering method in generating the low-dim projection for the ease of visualization in Figure 4. We have updated the manuscript accordingly to make it clear.\n\n### Q4: How error bars are calculated?\nA4: \nWe used 10 repeated runs in our experiments and reported the mean and standard deviation in tables and figures. We have updated the captions to explicitly explain how error bars are generated in the manuscript.\n\n### Q5: DAUC is developed for classification tasks in specific. Maybe that could be mentioned in the abstract.\nA5: \nIt is true that DAUC is designed for uncertainty categorization in classification tasks. We have updated the abstract and introduction to make it clearer. \n\n---\nWe hope that these clarifications address your concerns, and we are happy to have further discussions would they remain unclear. \n\n"}, {"review_id": "x97rl3Vl7k", "reviewer": "Reviewer_hHZH", "summary": "I have read the other reviews and all rebuttals. The other reviewers are positive overall, and the authors provided very detailed and thorough rebuttals. I have increased my score from \"5: Borderline accept\" to \"6: Weak Accept\".\n***\n***\n***\n***\n\n\nThe authors propose an approach for categorizing examples which are flagged as uncertain by uncertainty estimation methods (e.g., deep ensembles, MC-dropout) into three classes: out-of-distribution examples (OOD), boundary examples (Bnd) and examples in regions of high in-distribution misclassification (IDM). They propose  a method for post-hoc uncertainty categorization, utilizing latent space / feature-space based density models.", "questions": "1. Can the proposed method be applied to datasets with a lot more classes (e.g. 1000 classes)? Does the confusion density matrix become expensive to compute? What happens if the corpus C_{c1 --> c2} is empty for some pairs of classes?\n\n2. Does the validation set have be large, to ensure that the corpus C_{c1 --> c2} contains at least a few examples for each pair of classes? Does it have to be much bigger for e.g. 1000 classes than 10 classes?\n\n3. For the use case in Section 4.3, could you clarify how the Bnd/IDM categorization is useful? What type of conclusions does it enable? Could it be used as evidence that uncertainty estimation method X is better than method Y?\n\n4. Would it be possible to repeat the experiment in Section 4.4 on a larger dataset with more classes?\n\n\nMinor things:\n- Figure 1 caption: \"into four classes\" --> \"into three classes\"?\n- Line 104: \"for what a\" --> \"for why a\"?\n- Line 197: \"an different\" --> \"a different\".\n- In Section 3.1 the term \"inference-time\"/\"inference time\" is used, but in the rest of the paper \"test-time\"/\"test time\" is used?\n- In Definition 3 - 5 the full term \"confusion density matrix\" is used, but in the text in Section 3.3.1 and 3.3.2 just \"confusion matrix\" is used. Would it be more clear to always use the full term?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is very well written overall, I enjoyed reading it. More or less everything is clearly described and explained. The authors definitely seem knowledgeable, they have done a thorough job putting together the paper. I liked the color coding in Section 4. \n\nSection 3.3 is interesting. The proposed confusion density matrix is novel (I think) and makes some intuitive sense, it is a pretty neat idea.\n\nThe example in Section 4.1 is quite neat, it provides good intuition.\n\nThe use case in Section 4.4 seems potentially useful.", "weaknesses": "I find it quite difficult to judge the technical contribution and impact of the proposed approach. Doing OOD detection based on feature-space density is not new (see e.g. Sun et al., \"Out-of-distribution detection with deep nearest neighbors\", ICML 2022), and I am not sure how useful the Bnd/IDM categorization actually is in practice.\n\nThe authors show two use cases in Section 4.3 and 4.4. However, in Section 4.3 the only concrete conclusion seems to be that DE is most apt at flagging the OOD class, but this is not really a new finding? And while the results in Section 4.4 do seem quite promising, the method is here only applied to MNIST?\n\nIn fact, the proposed method is never applied to any dataset with more than ~10 classes? Can it be difficult to apply to a dataset with e.g. 1000 classes? What happens to the confusion density matrix computation, does it become expensive? And what happens if the corpus C_{c1 --> c2} is empty for some pairs of classes?\n\n\nSmall thing:\n- I found Section 3.3.1 a bit confusing at first, I had to get through Definition 4 before I understood the paragraph above. In particular, I was a bit confused by \"The first situation where high uncertainty arises, is when g(x) is close to latent representations of validation examples that have been correctly assigned a label that differs from the predicted one\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging comments and appreciation of our paper's clarity, novelty, and presentation. We now address each specific question in turn:\n\n---\n### Q1: Working with more classes, computation, scalability, and additional experiments for Sec. 4.4.\nA1: Technically, there's **no strict computational barrier** preventing our method's extension to tasks with 1000 classes. While extending our method in those tasks would intensify computational demands, we would like to note that such a demand is tractable with modern computation resources, especially with our parallelized implementation: this is because the calculations of different elements in the confusion density matrix can be highly parallelized. On the other hand, the demand for interpretability often associates with high-stake tasks that typically present fewer classes, as evidenced in healthcare (e.g., treat or not treat) and finance decisions (e.g., buy, sell, hold). \n\nTo further address the scalability concerns, we conducted **additional experiments** on the CIFAR-100 dataset. Given the limited time and the non-trivial efforts in creating boundary examples with corresponding gold labels, we would focus on the inverse direction.\n\nThe results are shown in Figure 1 in our attached 1-page pdf response. First, we use ResNet-18 to learn representation and identify the suspicious IDM and Bnd examples. We are then able to improve the prediction on those subsets. We observe that DAUC scales well on the CIFAR-100 dataset — prediction performance of the classifiers is improved through filtering similar examples in the training dataset.\n### Q2: What happens if the C_{c1 --> c2} is empty for some c1 c2?\nA2: In our work, C_{c1 --> c2} being empty means no validation examples from class c1 have been mistaken for class c2. This will lead to a zero value in the confusion density matrix. However, such zero values are not an issue and could be regularly observed. For instance, in the MNIST dataset, while class 1 examples might be misclassified as class 7, none from class 0 are mislabeled as class 7. In fact, we wish to note that DAUC's design ensures that its performance will not be critically affected by the absence of specific misclassification types. This is because the **categorizations of DAUC are based on the sums over the off-diagonal and diagonal values of the confusion density matrix** respectively (cf. Eqn. 4 and 5).\n### Q3: Does the validation set have to be large? Sensitivity to the number of classes.\nA3: As explained in the last response, zero values in the confusion density matrix can be common in practice. We would not impose additional conditions on the validation set more than the conventional requirements: the validation set should be representative of the test set.\n\nTo further verify the intuition, we present **additional experiments** to evaluate the effectiveness of DAUC across various settings. With the results presented in Figure 2-3 of the attached PDF file, several things can be deduced:\n\n- Figure 2: Increase the **number of classes** in the Dirty-MNIST dataset\n  1. The performance of DAUC in OOD identification remains consistently high across varying numbers of classes. This demonstrates that DAUC is resilient in identifying OOD examples, even when the in-distribution examples span fewer classes.\n  2. DAUC's ability to identify Bnd examples also shows good performance across different class counts. The F1-score indicates an upward trend as the number of classes increases, illustrating DAUC's adaptability in complex scenarios. \n  3. The accuracy of IDM identification appears to be more dependent on the number of classes, with a decrease in performance when the class count is reduced. This sensitivity can be explained by the accompanying decrease in the validation set size.\n- Figure 3: changing the **training-validation split** in the Dirty-MNIST dataset. \n\n  We varied the proportions of the training set, allocating between 50% to 90% of the data for training, with the remaining designated for validation. Across these configurations, DAUC consistently demonstrated high performance, underscoring its resilience to changes in the training-validation distribution. Particularly noteworthy is DAUC's ability to reliably identify both Bnd and OOD examples, regardless of the data split. However, as the validation set size decreased, we observe a decline in the accuracy of IDM identification. This trend mirrors our earlier observations where the performance dipped as the number of classes decreased.\n### Q4: How is the Bnd/IDM categorization useful? Compare UQ methods.\n\nA4: In Section 4.3, we explored the types of examples that various uncertainty methods identify as suspicious, focusing on interpretation rather than ranking different methods. Our findings provide insights for further enhancements:\n- OOD Category: If a prediction is uncertain and falls into the OOD category, gathering more training data and labels can enhance predictive performance on similar examples.\n- Bnd Category: Uncertain predictions in the Bnd category signal that minor changes in the latent representation may alter the prediction. This characteristic is important for high-stakes scenarios like healthcare where rapid changes in decisions can occur.\n- IDM Category: This reveals that the current model struggles with accurate predictions for specific examples. In this case, collaboration with other models or expert assessments are practical ways to improve performance.\n\nWhile we demonstrated DAUC’s ability using the DE, MCD and BNNs methods, we wish to note that our design is generally compatible with any UQ method. As a model-agnostic approach, **DAUC paves the way for selecting the appropriate UQ method according to user specifications and preferences.**\n\n---\nWe hope these clarifications address your concerns and further illuminate our ideas. Should there be any additional questions or concerns, we are more than willing to provide further explanations. "}, {"review_id": "irTIXc30vx", "reviewer": "Reviewer_7E4k", "summary": "In this paper, the authors present a unique approach for classifying uncertainty into distinct categories, providing insights into the reasons behind labeling a particular sample as suspicious or highly uncertain. They develop a kernel density-based confusion density matrix for any neural network which separates suspicious data samples from the uncertainty estimator into three categories: out-of-distribution (OOD), samples near the decision boundary, and misclassified samples. The paper outlines the methodology for constructing this density-based matrix and compares the performance of existing uncertainty estimators using this proposed categorization. Empirical studies and results are provided, demonstrating how this classification can in fact improve the training of predictive models and highlight the reasons for model failure. ", "questions": "* In my opinion, at a high level, this method basically computes the kernel similarity scores between the test data and the samples in the train/val distribution in the latent space to find out whether the test sample is OOD, bnd etc. How does the choice of the layer chosen impact the categorization ? Does this change ? This is because, the earlier layers can capture only the basic semantics and the discrepancies between inliers and the suspicious samples may not be captured.\n\n* How exactly is uncertainty from the model computed ? Is epistemic uncertainty alone computed for first filtering the suspicious vs safe samples, or the total uncertainty ? Writing a brief section about the same either in the main/supplement will be beneficial for the reader.\n\n* Can the authors comment on how effective the proposed approach will be to handle open-set data or OOD/anamolous data samples for e.g,  classes unseen during training ? Consider that there is an unc. estimator that is sensitive to open-set data and flags the sample as suspicious. Will the kernel density estimate score that categorizes OOD data in your formulation be able to capture such subtle shifts ? These open-set data need not share the same labels as that of your training data, but share a large amount of semantic information ?\n\n* For computing the confusion density matrix elements for the bnd and IDM samples, how does the properties of the validation data impact the estimation ? For e.g., A certain class c1 can offer no misclassification while another class c2 offers a greater amount of misclassification. \n\n* Can the authors comment on the scalability of the algorithm ? For instance, if the number of classes exceeds K (K>=100),  it is required to obtain a K x K matrix.\n", "limitations": "Yes, the limitations have been listed by the authors in the supplementary.  However, in my opinion,  I am skeptical about the scalability of the approach as well as its ability to handle data from open-set recognition conditions.  ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "In my opinion, this categorization process of determining what uncertainties actually flag (OOD, bnd etc.) is the main contribution. Importantly, the paper shows that this is a general framework applicable to any classification model (having a reasonably sized no. of classes). The paper demonstrates experiments on a variety of benchmarks contributing to its efficacy. Also, the paper is well written and simple to follow.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer's time and effort in evaluating our paper. We will now answer each of the questions:\n\n---\n### Q1: Choice of the latent space\nA1: We compute kernel similarity scores within the latent space, as opposed to the input space or basic semantics space, to ensure that the density estimation is directly tied to the model's predictions.\n\nSince our whole discussion revolves around model predictions, we would like to guarantee that two examples related to the kernel are given similar predictions by the model $f$. In this way, we would be able to interpret a large kernel density $\\kappa[g(x_1), g(x_2)]$ as a hint that the predictions $f(x_1)$ and $f(x_2)$ are similar. We will now show that, under Assumption 1, such a guarantee exists. First, we note that\n\n$||(l \\circ g)(x_1)-(l \\circ g)(x_2)||_{\\mathbb{R}^C}$  \n\n$=||l[g(x_1)-g(x_2)]||_{\\mathbb{R}^C}$\n\n$\\le||l|| \\cdot ||g(x_1)-g(x_2)||_\\mathcal{H}$\n\nwhere $||\\cdot||_{\\mathbb{R}^C}$ is a norm on $\\mathbb{R}^C$ and $||l||$ is the operator norm of the linear map $l$. In order to extend this inequality to black-box predictions, we note that the normalizing map in Assumption 1 is often a Lipschitz function with Lipschitz constant $\\lambda \\in \\mathbb{R}$. For instance, a Softmax function with inverse temperature constant $\\lambda^{-1}$ is $\\lambda$-Lipschitz. We use this fact to extend our inequality to predicted class probabilities:\n\n$||f(x_1)-f(x_2)||_\\mathcal{Y}$ \n\n$||\\phi\\circ l \\circ g(x_1)- \\phi\\circ l \\circ g(x_2)||_\\mathcal{Y}$\n\n$\\le\\lambda||l\\circ g(x_1)-l\\circ g(x_2)||_{\\mathbb{R}^C}$\n\n$\\le\\lambda||l||\\cdot||g(x_1)-g(x_2)||_\\mathcal{H}$\n\nThis crucial inequality guarantees that examples $x_1,x_2\\in\\mathcal{X}$ that are given a similar latent representation $g(x_1)\\approx g(x_2)$ will also be given a similar prediction $f(x_1)\\approx f(x_2)$. \n\nIn short: **two examples that are related according to a kernel density defined in the model latent space $\\mathcal{H}$ are guaranteed to have similar predictions.** This is the motivation we wanted to support the definition of the kernel $\\kappa$ in latent space.\n### Q2: Computation of uncertainty\nA2: DAUC **categorizes the uncertainty** through the confusion density matrix we introduced in Section 3. The categorization does not require any specific uncertainty computation.\n\nIn general, DAUC is able to categorize uncertain examples **flagged by any UQ method**. In our paper, we benchmark with MCD, DE and BNNs as 3 typical UQ methods. Given the uncertain scores predicted by those models and thresholds, we are able to mark different proportions of examples as uncertain. To demonstrate how DAUC categorizes all ranges of uncertain examples, we present results on the top $5\\%$ to the least $5\\%$ uncertainty.\n\nWe added an additional section in our appendix to explicitly explain the workflow to improve clarity.\n### Q3: OpenSet and anomalous detection\nA3: To inspect the relation to open-set learning and recognition [1-3]: while DAUC provides a unified framework for understanding different types of uncertainty, including that caused by OOD examples, we would clarify that existing off-the-shelf algorithms [4] can also identify such uncertainty. What distinguishes DAUC is its ability to categorize uncertainty originating from various sources, such as ambiguous boundary examples, similar misclassified examples, and OOD examples, in a **unified framework**. Specialized algorithms could potentially further improve each element. We agree that this is an important area for future exploration and hope the reviewer concurs with our prioritized scope of the current work.\n\nWe’ve added the discussion and relation on the open-set learning to our manuscript.\n### Q4: Impact of validation data\nA4: The imbalance in misclassification will not affect the categorizations of DAUC, as DAUC leverages the confusion density matrix rather than individual elements of the matrix.\nThe situation of *A certain class c1 … amount of misclassification* could be common in practice. For instance, in MNIST, there could be no examples from class 0 misclassified as class 7.\n\nDAUC’s categorizations are based on the sums over the off-diagonal and diagonal values of the confusion density matrix, respectively (refer to Eqn. (4) and (5)). Therefore, the **absence of specific misclassification instances does not directly impact DAUC's performance.** Additional empirical evidence is provided in **Figure 3 of the attached PDF file**.\n### Q5: Scalability\nA5: \nWhile it is true that experimentation with more classes can be more computationally demanding, we wish to note that the scenarios the UQ methods primarily target often involve a limited number of high-stakes decisions (e.g., in healthcare, treat or not treat; in finance, buy, sell, or hold). In addition, the confusion density matrix calculation can be **highly parallelized**, such that it is always tractable based on modern computation resources even for more complex settings.\n\nTo show the scalability explicitly, we **additionally experiment on the CIFAR-100 dataset**, with a special focus on the inverse direction. We use ResNet-18 to learn representations and identify the IDM and Bnd examples. We are then able to improve the prediction on those subsets. As we can observe in **Figure 1 of the attached PDF file**, DAUC scales effectively. By filtering similar examples in the training dataset, the prediction performance of the classifiers is enhanced.\n\n---\nWe hope our response can be helpful, and we are more than willing for any further clarification and discussion. \n\n---\n*References*\n\n[1] *Zhang, Jingyang, et al. \"OpenOOD v1. 5: Enhanced Benchmark for Out-of-Distribution Detection.\"*\n\n[2] *Yang, Jingkang, et al. \"Openood: Benchmarking generalized out-of-distribution detection.\"*\n\n[3] *Yang, Jingkang, et al. \"Generalized out-of-distribution detection: A survey.\"*\n\n[4] *Djurisic, Andrija, et al. \"Extremely simple activation shaping for out-of-distribution detection.\"*"}, {"review_id": "4COROvjSRb", "reviewer": "Reviewer_AvJq", "summary": "This paper proposes a model-agnostic framework named DAUC to categorize uncertain examples flagged by UQ methods, which introduces the confusion density matrix and provides post-hoc categorization for model uncertainty. Besides, this paper categorizes suspicious examples identified by a given uncertainty method into OOD, Bnd and IDM three classes.", "questions": "1. Checking some details in the paper, such as illustration, etc, corresponding to the paper.\n2. Adding the basis of classifying uncertain samples into three categories, or relevant explanations.\n3. Adding the explanation of DAUC approach model-agnostic.\n", "limitations": "Is DAUC a model-agnostic approach that can be extended to other tasks or domains?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper focuses on samples that remain suspect in uncertain methods, and proposes a model-agnostic density-based approach to mitigate this problem.\n2. The overall writing structure of this paper is clear, and the core ideas expressed are clear too.\n", "weaknesses": "1. Why are the uncertain samples of the model divided into three categories? It is only mentioned in the abstract (i.e., a kernel-based approximation of the misclassification density), but there is no relevant description in the paper. \n2. The four classes in Figure 1 should be changed to three classes, corresponding to the paper.\n3. In the Categorizing Uncertainty part of the Section introduction, it is mentioned that DAUC is a model-agnostic method, but in the experimental part, we do not see the model-agnostic of DAUC.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's efforts and constructive comments in improving our paper. We will respond to each of the questions in turn:\n\n---\n### Q1: The categorization of uncertain examples.\nA1: The categorization of uncertainty into three non-exclusive classes (OOD, Bnd, and IDM) is a fundamental aspect of our approach. These categories enable us to address various types of uncertainty, **each with unique characteristics that can be discerned through the confusion density matrix**.\n- OOD: This class represents examples unlike any seen during training. The kernel density of inference examples with respect to the training dataset identifies this uncertainty. Additional labeled data encompassing this region of the input space would be necessary for performance enhancement on such samples.\n- Bnd: Arising from confusion near decision boundaries, this class is recognized by the diagonal kernel density of inference examples relative to the validation dataset. This form of uncertainty indicates misclassifications stemming from inherent ambiguities in the data. Fine-tuning the prediction model might enhance predictions on these sensitive samples.\n- IDM: This category, identified by the off-diagonal kernel density of inference examples compared to the validation dataset, pertains to the model's intrinsic errors leading to misclassification. Recognizing IDM uncertainty is crucial for understanding the model's inherent biases or limitations.\n\nIn a nutshell, these classes correspond to three properties that can be **manifested by the confusion density matrix**: OOD is represented by the **overall density**, Bnd by the **diagonal elements**, and IDM by the **off-diagonal elements**. \n\nTo address your concern in Figure 1, we have made it clearer by noting explicitly in our updated manuscript that an example can simultaneously be categorized to the Bnd class and the IDM class, e.g., an example located in the decision boundary can have similar misclassified neighbors. This does not contradict our 3-type categorization according to the confusion density matrix.\n\nWe agree that a more extensive explanation would facilitate understanding, and we have therefore **refined and highlighted our introduction to bridge any gap**. \n\n### Q2: Assumption on model: Model-Agnosticity\nA2: We would start by clarifying the definition of **Model-Agnostic** does not indicate the neural network model used in DAUC and the confusion density matrix. Instead, we would say DAUC is a model-agnostic uncertainty categorization method to highlight its ability to categorize suspicious examples identified by **different UQ models**. The model-agnostic refers to the general applicability of DAUC w.r.t. UQ models, in contrast to neural network models. We illustrate such property by benchmarking different UQ models: MCD, DE, and BNNs.\n\nWe have updated our manuscript accordingly to make the definition of Model-Agnosticity in our context clearer.\n\nOn the other hand, for the requirement on the neural network model used in DAUC, we would like to emphasize, as highlighted in Table 1, that **DAUC is designed to be flexible in terms of model structure**. It functions effectively under conditions where Assumption 1 can be met, a scenario commonly satisfied when using general neural network architectures.\n\n### Q3: Other tasks or domains.\nA3: We wish to demonstrate the general applicability of DAUC through the following empirical observations, in addition to the main results presented in the paper.\n\nTo demonstrate the broader applicability of DAUC, we **extended our evaluation with the Dirty-CIFAR dataset**. Details of this additional evaluation are available in Appendix C.4, We also described how the dataset is created, and **this dataset will also be made publicly available for future research.**\n\nTo stress-test and show the scalability of DAUC, we **experiment with the CIFAR-100 dataset as another additional domain**. Given the limited time and the non-trivial efforts in creating boundary examples and corresponding gold labels, we focus on the inverse direction, as a downstream application, in demonstrating DAUC’s scalability.\n\nThe results are shown in **Figure 1 in the attached 1-page PDF file**. For those experiments, we use ResNet-18 to learn representation and identify the IDM and Bnd examples marked as suspicious, and then improve the prediction on those subsets. DAUC scales well on the CIFAR-100 dataset — prediction performance of the classifiers can be improved through filtering similar examples in the training dataset.\n\nTo sum up, if we could add up experiments in our main text, appendix, and the updated supplementary response, we used **1 synthetic dataset, 3 tabular datasets, and 3 image datasets in verifying the effectiveness of DAUC**. All of those results support that DAUC provides a new and distinct perspective for identifying uncertainty sources and enables downstream prediction improvement based on the categorization.\n\n---\nIf there are leftover concerns, please let us know and we will do our utmost to address them. Thank you once again for your insightful comments, time, and efforts in reviewing our paper.\n\n"}]

Author rebuttal:
We extend our sincere gratitude to all reviewers for their insightful comments, valuable suggestions, time, and efforts in evaluating and improving our paper. 
We thank all reviewers for their affirmation of our work’s **novelty** (reviewers: sVgx, hHZH, 7E4k), **presentation** (reviewers: sVgx, hHZH, 7E4k, AvJq), **evaluation** (reviewers: sVgx, 7E4k), and **contribution** (reviewers: sVgx, 7E4k, AvJq). 

----

To address the concerns raised by reviewers, we would respond to each of their questions respectively. Below, as a general response, we aim to outline the **key revisions and additional experimentation conducted by far**:

#### **Supplementary Experimental Evaluation** [Please refer to the attached PDF]

1. To stress-test the scalability of DAUC, we carried out additional experiments using the CIFAR-100 dataset.
2. To highlight the efficacy of DAUC across different number of classes, we conducted incremental studies with the Dirty-MNIST dataset.
3. We demonstrated the robustness of DAUC through a study varying the sizes of the validation dataset.

#### **Revised Manuscript for Clarity** (Include but not limited to)

1. We updated our introduction, Figure 1, and method sections to illustrate the motivation of our three-class categorization. This motivation stems from distinct characteristics manifested by the confusion density matrix.
2. We additionally discussed the relationship between DAUC, OOD detection algorithms, and open-set problems in our related work section.
3. We added a section in our appendix that explicitly explains the workflow of DAUC to enhance clarity: DAUC categorizes uncertain examples flagged by plug-in UQ algorithms.
4. We updated our manuscript to make the definition of Model-Agnosticity in our context clearer: it is used to express the applicability of DAUC to diverse UQ methods.


----

We hope our clarification and additional empirical studies could address the concerns raised by reviewers. Should there be any leftover questions, please let us know and we will make every effort to address them during the subsequent discussion period.



Meta-review: All reviewers agree that this paper makes an interesting and substantial contribution to uncertainty quantification. A few issues have been raised in the reviews, but these could mostly be clarified in the rebuttal. During the review phase, the authors made quite some changes in the paper, which is not uncritical, because these changes cannot be verified (unlike for journal papers, there is no official revision or possibility of a re-review). That said, their effort to comply with the reviewers' comments and suggestions is of course appreciated. Overall, all reviewers are on the positive side and in favour of acceptance.
Decision: Accept (poster)