SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hao Sun†; Boris van Breugel†, Jonathan Crabbé, Nabeel Seedat, Mihaela van der Schaar  
Department of Applied Mathematics and Theoretical Physics  
University of Cambridge
Uncertainty Quantification (UQ) is essential for creating trustworthy machine learning models. Recent years have seen a steep rise in UQ methods that can flag suspicious examples, however, it is often unclear what exactly these methods identify. In this work, we propose a framework for categorizing uncertain examples flagged by UQ methods in classification tasks. We introduce the confusion density matrix—a kernel-based approximation of the misclassification density—and use this to categorize suspicious examples identified by a given uncertainty method into three classes: out-of-distribution (OOD) examples, boundary (Bnd) examples, and examples in regions of high in-distribution misclassification (IDM). Through extensive experiments, we show that our framework provides a new and distinct perspective for assessing differences between uncertainty quantification methods, thereby forming a valuable assessment benchmark.
Black-box parametric models like neural networks have achieved remarkably good performance on a variety of challenging tasks, yet many real-world applications with safety concerns—e.g. healthcare [1], finance [2, 3] and autonomous driving [4, 5]—necessitate reliability. These scenarios require trustworthy model predictions to avoid the high cost of erroneous decisions.
Uncertainty quantification [6-10] addresses the challenge of trustworthy prediction through inspecting the confidence of a model, enabling intervention whenever uncertainty is too high. Usually, however, the cause of the uncertainty is not clear. In this work, we go beyond black-box UQ in the context classification tasks and aim to answer two questions:
1. How do we provide a more granular categorization of why uncertainty methods to identify certain predictions as suspicious?
2. What kind of examples do different UQ methods tend to mark as suspicious?
Categorizing Uncertainty We propose a Density-based Approach for Uncertainty Categorization (DAUC): a model-agnostic framework that provides post-hoc categorization for model uncertainty. We introduce the confusion density matrix, which captures the predictive behaviors of a given model. Based on such a confusion density matrix, we categorize the model's uncertainty into three classes: (1) OOD uncertainty caused by OOD examples, i.e. test-time examples that resemble no training-time sample [11-14]. Such uncertainty can be manifested by a low density in the confusion density matrix; (2) Bnd uncertainty caused by neighboring decision boundaries, i.e., non-conformal predictions due to confusing training-time resemblances from different classes or inherent ambiguities making it challenging to classify the data [15-17]; Such uncertainty can be manifested by the density of
*hs789@cam.ac.uk, †: equal contributions
Figure 1: Given a prediction and UQ algorithm, our method divides flagged test time examples into different classes. Class OOD identifies outliers, which are mistrusted because they do not resemble training data; class IDM indicates examples lying in regions with high misclassification; class Bnd indicates examples that lie near the decision boundary.
diagonal elements in the confusion density matrix; (3) IDM uncertainty caused by imperfections in the model—as manifested by high misclassification at validation time—such that similar misclassification is to be expected during testing [18]. Such uncertainty can be manifested by the density of off-diagonal elements in the confusion density matrix. Figure 1 illustrates the different classes and in Section 4 we provide visualisations using real data. We show how DAUC can be used to benchmark a broad class of UQ methods, by categorizing what each method tends to flag as suspicious.
1. Formally, we propose the confusion density matrix—the heart of DAUC—that links the training time error, decision boundary ambiguity, and uncertainty with latent representation density.
2. Practically, we leverage DAUC as a unified framework for uncertain example categorization. DAUC offers characterisation of uncertain examples at test time.
3. Empirically, we use DAUC to benchmark existing UQ methods. We manifest different methods' sensitivity to different types of uncertain examples, this provides model insight and aids UQ method selection.
Uncertainty Quantification Uncertainty quantification methods are used to assess the confidence in a model's predictions. In recent years the machine learning community has proposed many methods which broadly fall into the following categories: (1) Ensemble methods (i.e. Deep Ensembles [6]), which—while considered state of the art—have a high computational burden; (2) Approximate Bayesian methods (i.e. Stochastic Variational Inference [7-9]), however work by [19-21] suggests these methods do not yield high-quality uncertainty estimates; and (3) Dropout-based methods such as Monte Carlo Dropout [10], which are simpler as they do rely on estimating a posterior, however the quality of the uncertainty estimates is linked to the choice of parameters which need to be calibrated to match the level of uncertainty and avoid suboptimal performance [22]. For completeness we note that whilst conformal prediction [23] is another UQ method, the paradigm is different from the other aforementioned methods, as it returns predictive sets to satisfy coverage guarantees rather than a value-based measure of uncertainty.
In practice, the predictive uncertainty for a model prediction arises from the lack of relevant training data (epistemic uncertainty) or the inherent non-separable property of the data distribution (aleatoric uncertainty) [24-26]. This distinction in types of uncertainty is crucial as it has been shown that samples with low epistemic uncertainty are more likely under the data distribution, hence motivating why epistemic uncertainty has been used for OOD/outlier detection [10]. Moreover, ambiguous instances close to the decision boundary typically result in high aleatoric uncertainty [27].
This suggests that different sets of uncertain data points are associated with different types of uncertainties and, consequently, different types of misclassifications. Furthermore, it is to be expected that different uncertainty estimators are more able/prone to capturing some types of uncertainty than others. To understand these models better, we thus require a more granular definition to characterize
Table 1: Comparison with related work in UQ. We aim to provide a flexible framework for inspecting mistrusted examples identified by uncertainty estimators. Furthermore, this framework enables us to improves the prediction performance on a certain type of flagged uncertain class.
Method	Model Structure	Uncertainty Estimation	Categorize Uncertainty	Improve Prediction	Examples
BNNs	Bayesian Layers	✓	·	·	[8, 9]
GP	Gaussian Processes	✓	·	·	[39]
MC-Dropout	Drop-Out Layers	✓	·	·	[10]
Deep-Ensemble	Multiple Models	✓	·	·	[6]
ICP	Model “Wrapper”	✓	·	·	[23, 40], [37, 18]
Performance Prediction	Multiple Predictive Models	✓	·	·
DAUC	Assumption 1	✓	✓	✓	(Ours)
uncertain data points. We employ three classes: outliers (OOD), boundary examples (Bnd) and examples of regions with high in-distribution misclassification (IDM), see Fig 1.
Out-of-distribution (OOD) detection Recall that UQ methods have been used to flag OOD examples. For completeness, we highlight that other alternative methods exist for OOD detection. For example, Lee et al. [28] detects OOD examples based on the Mahalanobis distance, whilst Ren et al. [29] uses the likelihood ratio between two generative models. Besides supervised learning, OOD is an essential topic in offline reinforcement learning [30-35]. Sun et al. [36] detects the OOD state in the context of RL using confidence intervals. We emphasize that DAUC's aim is broader—creating a unifying framework for categorizing multiple types of uncertainty—however, existing OOD methods could be used to replace DAUC's OOD detector.
Accuracy without Labels We contrast our work to the literature which aims to determine model accuracy without access to ground-truth labels. Methods such as [37, 38] propose a secondary regression model as an accuracy predictor given a data sample. Ramalho and Miranda [18] combine regression model with latent nearest neighbors for uncertain prediction. This is different from our setting which is focused on inspecting UQ methods on a sample level by characterizing it as an outlier, boundary, or IDM example. We contrast DAUC with related works in Table 1.
3 Categorizing Model Uncertainty via Latent Density
We consider a typical classification setting where  $\mathcal{X} \subseteq \mathbb{R}^{d_X}$  is the input space and  $\mathcal{Y} = [0,1]^C$  is the set of class probabilities, where  $d_X$  is the dimension of input space and  $C \in \mathbb{N}^*$  is the number of classes. We are given a prediction model  $f: \mathcal{X} \to \mathcal{Y}$  that maps  $x \in \mathcal{X}$  to class probabilities  $f(x) \in \mathcal{Y}$ . An uncertainty estimator  $u: \mathcal{X} \to [0,1]$  quantifies the uncertainty of the predicted outcomes. Given some threshold  $\tau$ , the inference-time predictions can be separated into trusted predictions  $\{x \in \mathcal{X} | u(x) < \tau\}$  and untrusted predictions  $\{x \in \mathcal{X} | u(x) \geq \tau\}$ . We make the following assumption on the model architecture of  $f$ .
Assumption 1 (Model Architecture). Model  $f$  can be decomposed as  $f = \varphi \circ l \circ g$ , where  $g: \mathcal{X} \to \mathcal{H} \subseteq \mathbb{R}^{d_H}$  is a feature extractor that maps the input space to a  $d_H < d_X$  dimensional latent (or representation) space,  $l: \mathcal{H} \to \mathbb{R}^C$  is a linear map between the latent space to the output space and  $\varphi: \mathbb{R}^C \to \mathcal{Y}$  is a normalizing map that converts vectors into probabilities.
Remark 1. This assumption guarantees that the model is endowed with a lower-dimensional representation space. Most modern uncertainty estimation methods like MCD, Deep-Ensemble, BNN satisfy this assumption. In the following, we use such a space to categorize model uncertainty.
We assume that the model and the uncertainty estimator have been trained with a set of  $N \in \mathbb{N}^*$  training examples  $\mathcal{D}_{\mathrm{train}} = \{(x^n, y^n) \mid n \in [N]\}$ . At inference time the underlying model  $f$  and uncertainty method  $u$  predict class probabilities  $f(x) \in \mathcal{V}$  and uncertainty  $u(x) \in [0,1]$ , respectively. We assign a class to this probability vector  $f(x)$  with the map class:  $\mathcal{V} \to [C]$  that maps a probability vector  $y$  to the class with maximal probability class  $[y] = \arg \max_{c \in [C]} y_c$ . While uncertainty estimators flag examples to be trustworthy or not, those estimators do not provide a fine-grained reason for what a certain prediction should not be mistrusted. Our aim is to use the
model's predictions and representations of a corpus of labelled examples—which we will usually take to be the training  $(\mathcal{D}_{\mathrm{train}})$  or validation  $(\mathcal{D}_{\mathrm{val}})$  sets—to categorize inference-time uncertainty predictions. To that aim, we distinguish two general scenarios where a model's predictions should be considered with skepticism.
3.2 Flagging OOD Examples
There is a limit to model generalization. Uncertainty estimators should be skeptical when the input  $x \in \mathcal{X}$  differs significantly from input examples that the model was trained on. From an UQ perspective, the predictions for these examples are expected to be associated with a large epistemic uncertainty.
A natural approach to flagging these examples is to define a density  $p(\cdot \mid \mathcal{D}_{\mathrm{train}}): \mathcal{X} \to \mathbb{R}^+$  over the input space. This density should be such that  $p(x \mid \mathcal{D}_{\mathrm{train}})$  is high whenever the example  $x \in \mathcal{X}$  resembles one or several examples from the training set  $\mathcal{D}_{\mathrm{train}}$ . Conversely, a low value for  $p(x \mid \mathcal{D}_{\mathrm{train}})$  indicates that the example  $x$  differs from the training examples. Of course, estimating the density  $p(x \mid \mathcal{D}_{\mathrm{train}})$  is a nontrivial task. At this stage, it is worth noting that this density does not need to reflect the ground-truth data generating process underlying the training set  $\mathcal{D}_{\mathrm{train}}$ . For the problem at hand, this density  $p(x \mid \mathcal{D}_{\mathrm{train}})$  need only measure how close the example  $x$  is to the training data manifold. A common approach is to build a kernel density estimation with the training set  $\mathcal{D}_{\mathrm{train}}$ . Further, we note that Assumption 1 provides a representation space  $\mathcal{H}$  that was specifically learned for the classification task on  $\mathcal{D}_{\mathrm{train}}$ . In Appendix A.1, we argue that this latent space is suitable for our kernel density estimation. This motivates the following definition for  $p(\cdot \mid \mathcal{D}_{\mathrm{train}})$ .
Definition 1 (Latent Density). Let  $f: \mathcal{X} \to \mathcal{Y}$  be a prediction model, let  $g: \mathcal{X} \to \mathcal{H}$  be the feature extractor from Assumption 1 and let  $\kappa: \mathcal{H} \times \mathcal{H} \to \mathbb{R}^+$  be a kernel function. The latent density  $p(\cdot \mid \mathcal{D}): \mathcal{X} \to \mathbb{R}^+$  is defined over a dataset  $\mathcal{D}$  as:
$$
p (x \mid \mathcal {D}) \equiv \frac {1}{N} \sum_ {\tilde {x} \in \mathcal {D}} \kappa [ g (x), g (\tilde {x}) ] \tag {1}
$$
Test examples with low training density are likely to be underfitted for the model — thus should not be trusted.
Definition 2 (OD Score). The OOD Score  $T_{\mathrm{OD}}$  is defined as
$$
T _ {\mathrm {O O D}} (x) \equiv \frac {1}{p (x | \mathcal {D} _ {\text {t r a i n}})} \tag {2}
$$
For a test example  $x \in \mathcal{D}_{\mathrm{test}}$ , if  $T_{\mathrm{OOD}}(x|\mathcal{D}_{\mathrm{train}}) \geq \tau_{\mathrm{OOD}}$  the example is suspected to be an outlier with respect to the training set. We set  $\tau_{\mathrm{OOD}} = \frac{1}{\min_{x' \in \mathcal{D}_{\mathrm{train}}} p(x'| \mathcal{D}_{\mathrm{train}})}$ , i.e., a new sample's training density is smaller than the minimal density of training examples.
3.3 Flagging IDM and Boundary Examples
Samples that are not considered outliers, yet are given high uncertainty scores, we divide up further into two non-exclusionary categories. The first category consists of points located near the boundary between two or more classes, the second consists of points that are located in regions of high misclassification.
For achieving this categorization, we will use a separate validation set  $\mathcal{D}_{\mathrm{val}}$ . We first partition the validation examples according to their true and predicted label. More precisely, for each couple of classes  $(c_{1}, c_{2}) \in [C]^{2}$ , we define the corpus  $\mathcal{C}_{c_{1} \mapsto c_{2}} \equiv \{(x, y) \in \mathcal{D}_{\mathrm{val}} \mid \text{class}[y] = c_{1} \wedge \text{class}[f(x)] = c_{2}\}$  of validation examples whose true class is  $c_{1}$  and whose predicted class is  $c_{2}$ . In the case where these two classes are different  $c_{1} \neq c_{2}$ , this corresponds to a corpus of misclassified examples. Clearly, if some example  $x \in \mathcal{X}$  resembles one or several examples of those misclassification corpus, it is legitimate to be skeptical about the prediction  $f(x)$  for this example. In fact, keeping track of the various misclassification corpora from the validation set allows us to have an idea of what the misclassification is likely to be.
In order to make this detection of suspicious examples quantitative, we will mirror the approach from Definition 1. Indeed, we can define a kernel density  $p\left(\cdot \mid \mathcal{C}_{c_1 \mapsto c_2}\right): \mathcal{X} \to \mathbb{R}^+$  for each corpus
$\mathcal{C}_{c_1 \mapsto c_2}$ . Again, this density will be such that  $p(\cdot \mid \mathcal{C}_{c_1 \mapsto c_2})$  is high whenever the representation of the example  $x \in \mathcal{X}$  resembles the representation of one or several examples from the corpus  $\mathcal{C}_{c_1 \mapsto c_2}$ . If this corpus is a corpus of misclassified examples, this should trigger our skepticism about the model's prediction  $f(x)$ . By aggregating the densities associated with each of these corpora, we arrive at the following definition.
Definition 3 (Confusion Density Matrix). Let  $f: \mathcal{X} \to \mathcal{Y}$  be a prediction model, let  $g: \mathcal{X} \to \mathcal{H}$  be the feature extractor from Assumption 1 and let  $\kappa: \mathcal{H} \times \mathcal{H} \to \mathbb{R}^+$  be a kernel function. The confusion density matrix  $P(\cdot | \mathcal{D}_{\mathrm{val}}): \mathcal{X} \to (\mathbb{R}^+)^{C \times C}$  is defined as
$$
P _ {c _ {1}, c _ {2}} (x \mid \mathcal {D} _ {\text {v a l}}) \equiv p (x \mid \mathcal {C} _ {c _ {1} \mapsto c _ {2}}) = \frac {1}{\left| \mathcal {C} _ {c _ {1} \mapsto c _ {2}} \right|} \sum_ {\tilde {x} \in \mathcal {C} _ {c _ {1} \mapsto c _ {2}}} \kappa [ g (x), g (\tilde {x}) ], \forall (c _ {1}, c _ {2}) \in [ C ] ^ {2} \tag {3}
$$
Remark 2. The name confusion density is chosen to make a parallel with confusion matrices. Like confusion matrices, our confusion density indicates the likelihood of each couple  $(c_{1}, c_{2})$ , where  $c_{1}$  is the true class and  $c_{2}$  is the predicted class. Unlike confusion matrices, our confusion density provides an instance-wise (i.e. for each  $x \in \mathcal{X}$ ) likelihood for each couple.
3.3.1 Bnd examples
By inspecting the confusion matrix, we can quantitatively distinguish two situations where the example  $x \in \mathcal{X}$  is likely to be mistrusted. The first situation where high uncertainty arises, is when  $g(x)$  is close to latent representations of validation examples that have been correctly assigned a label that differs from the predicted one class  $[f(x)]$ . This typically happens when  $g(x)$  is located close to a decision boundary in latent space. In our validation confusion matrix, this likelihood that  $x$  is related to validation examples with different labels is reflected by the diagonal elements. This motivates the following definition.
Definition 4 (Boundary Score). Let  $P(x \mid \mathcal{D}_{\mathrm{val}})$  be the confusion density matrix for an example  $x \in \mathcal{X}$  with predicted class  $\hat{c} = \operatorname{class}[f(x)]$ . We define the boundary score as the sum of each density of well-classified examples from a different class:
$$
T _ {\mathrm {B n d}} (x) = \sum_ {c \neq \hat {c}} ^ {C} P _ {c, c} (x \mid \mathcal {D} _ {\text {v a l}}). \tag {4}
$$
Points are identified as Bnd when  $T_{\mathrm{Bnd}} > \tau_{\mathrm{Bnd}}$  see Appendix A.2.
3.3.2 IDM examples
The second situation is the one previously mentioned: the latent representation  $g(x)$  is close to latent representations of validation examples that have been misclassified. In our validation confusion matrix, this likelihood that  $x$  is related to misclassified validation examples is reflected by the off-diagonal elements. This motivates the following definition.
Definition 5 (IDM Score). Let  $P(x \mid \mathcal{D}_{\mathrm{val}})$  be the confusion density matrix for an example  $x \in \mathcal{X}$ . We define the IDM score as the sum of each density corresponding to a misclassification of the predicted class  $\hat{c} = \mathrm{class} f(x)$  in the confusion density matrix:
$$
T _ {\mathrm {I D M}} (x) = \sum_ {c \neq \hat {c}} ^ {C} P _ {c, \hat {c}} \left(x \mid \mathcal {D} _ {\text {v a l}}\right). \tag {5}
$$
Points are identified as IDM when  $T_{\mathrm{IDM}} > \tau_{\mathrm{IDM}}$ . We choose  $\tau_{\mathrm{IDM}}$  such that the proportion of IDM points in the validation set equals the number of misclassified examples. Details for definitions and choices of thresholds are provided in Appendix A.2.
Remark 3. Note that the definitions of Bnd examples and IDM examples do not exclude each other, therefore, an uncertain example can be flagged as a Bnd example, an IDM example, or flagged as both Bnd and IDM (B&I). To make this distinction clear, we will refer to the disjoint classes as:
$$
\mathcal {S} _ {\mathrm {B n d}} = \{x | x \in \mathcal {D} _ {\text {t e s t}}, T _ {\mathrm {B n d}} (x) > \tau_ {\mathrm {B n d}}, T _ {\mathrm {I D M}} (x) \leq \tau_ {\mathrm {I D M}} \}
$$
$$
\mathcal {S} _ {\mathrm {I D M}} = \{x | x \in \mathcal {D} _ {\text {t e s t}}, T _ {\text {B n d}} (x) \leq \tau_ {\text {B n d}}, T _ {\text {I D M}} (x) > \tau_ {\text {I D M}} \}
$$
$$
\mathcal {S} _ {\mathrm {B} \& \mathrm {I}} = \{x | x \in \mathcal {D} _ {\text {t e s t}}, T _ {\text {B n d}} (x) > \tau_ {\text {B n d}}, T _ {\text {I D M}} (x) > \tau_ {\text {I D M}} \}
$$
Test examples that are flagged by uncertainty method  $u$  —yet do not meet any of the thresholds—are marked as Other.
In a nutshell DAUC uses the OOD, IDM and Bnd classes to categorize model uncertainty—see Table 2 for an overview. Better predictions may be possible for IDM samples, in case a different classifier is used. For samples that are also labelled as Bnd, fine-tuning the existing model—possibly after gathering more data—may be able to separate the different classes better. IDM samples that are not in the Bnd class are harder, and may only be classified correctly if a different latent representation is found, or an different training set is used. We explore the idea of improving the performance on uncertain examples in Appendix B. In Section 4.1 we explore the distinction between classes further.
Table 2: Summary of different uncertainty types
Type	Definition	Description
OOD	1/p(x|Dtrain) > τOOD	Samples that do not resemble the training data. Additional labelled data that covers this part of the input space is required to improve performance on these samples.
Bnd	∑c≠hatPc,c(x|Dval) > τBnd	Samples near the boundaries in the latent space. Predictions on these samples are sensitive to small changes in the predictor, and fine-tuning the prediction model may yield better predictions.
IDM	∑c≠hatPc,hat(x|Dval) > τIDM	Samples that are likely to be misclassified, since similar examples were misclassified in the validation set.
In this section, we demonstrate our proposed method with empirical studies. Specifically, we use two experiments as Proof-of-Concept, and two experiments as Use Cases. Specifically, in Sec. 4.1 we visualize the different classes of flagged examples on a modified Two-Moons dataset; in Sec. 4.2 we quantitatively assess DAUC's categorization accuracy on the Dirty-MNIST dataset [41]; in Sec. 4.3, we present a use case of DAUC—comparing existing uncertainty estimation benchmarks; in Sec. 4.4, we demonstrate another important use case of DAUC—improving uncertain predictions.
Our selection of the Dirty-MNIST dataset for empirical evaluation was motivated by the pursuit of better reproducibility. As an existing publicly available resource, Dirty-MNIST provides gold labels for boundary classes and OOD examples, making it particularly suitable for benchmarking the performance of DAUC.
Recognizing the importance of demonstrating the broader applicability of DAUC, we have extended our evaluation to include results on the Dirty-CIFAR dataset. Details of this additional evaluation are available in Appendix C.4, where we also describe how we created the dataset. This dataset will also be made publicly available. Additional empirical evidence that justifies DAUC is provided in Appendix C.
4.1 Visualizing DAUC with Two-Smiles
4.1.1 Experiment Settings