SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To highlight the different types of uncertainty, we create a modified version of the Two-Moons dataset, which we will call "Two-Smiles". We use scikit-learn's datasets package to generate 6000 two-moons examples with a noise rate of 0.1. In addition, we generate two Gaussian clusters of 150 examples centered at  $(0,1.5)$  and  $(1,-1)$  for training, validation and test, and mark them as the positive and negative class separately. The data is split into a training, validation and test set. We add an additional 1500 OOD examples to the test set, that are generated with a Gaussian distribution centered at  $(2,2)$  and  $(-1,-1.5)$ . Overall, the test set counts 4500 examples, out of which 1500 are positive examples, 1500 are negative examples and 1500 are OOD—see Figure 2 (a).
We demonstrate the intuition behind the different types of uncertainty, by training a linear model to classify the Two-Smiles data—i.e. using a composition of an identity embedding function and linear classifier, see Assumption 1. Figure 2 (b) shows the test-time misclassified examples. Since the identity function does not linearly separate the two classes and outliers are unobserved at training
(a) Test Dataset
(b) Test Errors
Figure 2: Visualization of our proposed framework on the Two-Smiles dataset with a linear model.
(c) Flagged OOD
(d) Flagged Bnd
(e) Flagged IDM
time, the classifier makes mistakes near class boundaries, in predicting the OOD examples, as well as the clusters at  $(0,1.5)$  and  $(1,-1)$ . In Figure 2 (c)-(e), we see that DAUC correctly flags the OOD examples, boundary examples and the IDM examples. Also note the distinction between  $S_{\mathrm{B}\& \mathrm{I}}$  and  $S_{\mathrm{IDM}}$ . Slightly fine-tuning the prediction model may yield a decision boundary that separates the  $S_{\mathrm{B}\& \mathrm{I}}$  samples from the incorrect class. On the other hand, this will not suffice for the  $S_{\mathrm{IDM}}$  examples—i.e. the clusters at  $(0,1.5)$  and  $(1,-1)$ —which require a significantly different representation/prediction model to be classified correctly.
4.2 Verifying DAUC with Dirty-MNIST:
4.2.1 Experiment Settings
We evaluate the effectiveness of DAUC on the Dirty-MNIST dataset [41]. The training data set is composed of the vanilla MNIST dataset and Ambiguous-MNIST, containing ambiguous artificial examples—e.g., digits 4 that look like 9s. The test set is similar but also contains examples from Fashion-MNIST as OOD examples. The advantage of this dataset is that the different data types roughly correspond to the categories we want to detect, and as such we can verify whether DAUC's classification corresponds to these "ground-truth" labels. Specifically, we deem all examples from MNIST to be non-suspicious, associate Ambiguous-MNIST with class Bnd, and associate Fashion-MNIST with class OOD (There are 1,000 OOD examples out of the 11,000 test examples). See [41] for more details on the dataset. We use a 3-layer CNN model with ReLU activation and MaxPooling as the backbone model in this section.
(a) OOD
Figure 3: (a-c) Box-plots of different data classes and corresponding scores. All values are scaled by being divided by the maximal value. (d-e) Precision-Recall curves for different choices of thresholds in flagging Bnd examples and IDM examples.
(b) Bnd
(c) IDM
(d) Flag Bnd
(e) Flag IDM
Flagging Outliers We assume the ground-truth label of all Fashion-MNIST data is OOD. Comparing these to the classification by DAUC, we are able to quantitatively evaluate the performance of DAUC in identifying outliers. We compute precision, recall and F1-scores for different classes, see Figure 3 (a) and Table 3. All Fashion-MNIST examples are successfully flagged as OOD by DAUC. In Appendix C.3 we include more OOD experiments, including comparison to OOD detector baselines.
Table 3: Quantitative results on the Dirty-MNIST dataset. Our proposed method can flag all three classes of uncertain examples with high F1-Score. The results presented in the table are based on 8 repeated runs.
Category	Precision	Recall	F1-Score
OOD	1.000 ± 0.000	1.000 ± 0.000	1.000 ± 0.000
Bnd	0.959 ± 0.008	0.963 ± 0.006	0.961 ± 0.003
IDM	0.918 ± 0.039	0.932 ± 0.024	0.924 ± 0.012
Figure 4: Examples of different uncertainty classes  $S_{\mathrm{OOD}}$ ,  $S_{\mathrm{IDM}}$ ,  $S_{\mathrm{Bnd}}$  and  $S_{\mathrm{B&I}}$ . For better visualization, we only plot some classes including the outliers. t-SNE [42] is leveraged in generating low-dim visualizations.
Flagging Bnd Examples We expect most of the boundary examples to belong to the Ambiguous-MNIST class, as these have been synthesised using a linear interpolation of two different digits in latent space [41]. Figure 3 (b) shows that DAUC's boundary scores are indeed significantly higher in Ambiguous-MNIST compared to vanilla MNIST. Figure 3 (d) shows the precision-recall curve of DAUC, created by varying threshold  $\tau_{\mathrm{Bnd}}$ . Most boundary examples are correctly discovered under a wide range of threshold choices. This stability of uncertainty categorization is desirable, since  $\tau_{\mathrm{Bnd}}$  is usually unknown exactly.
Flagging IDM Examples In order to quantitatively evaluate the performance of DAUC on flagging IDM examples, we use a previously unseen hold-out set, which is balanced to consist of  $50\%$  misclassified and  $50\%$  correctly classified examples. We label the former as test-time IDM examples and the latter as non-IDM examples, and compare this to DAUC's categorization. Figure 3c shows DAUC successfully assigns significantly higher IDM scores to the examples that are to-be misclassified.
Varying  $\tau_{\mathrm{IDM}}$  we create the precision-recall curve for the IDM class in Figure 3 (e), which is fairly stable w.r.t. the threshold  $\tau_{\mathrm{IDM}}$ . In practice, we recommend to use the prediction accuracy on the validation dataset for setting  $\tau_{\mathrm{IDM}}$ —see Appendix A.2.
Visualizing Different Classes Figure 4 shows examples from the  $S_{\mathrm{OOD}}$ ,  $S_{\mathrm{Bnd}}$ ,  $S_{\mathrm{IDM}}$  and  $S_{\mathrm{B&I}}$  sets. The first row shows OOD examples from the Fashion-MNIST dataset. The second row shows boundary examples, most of which indeed resemble more than one class. The third row shows IDM examples, which DAUC thinks are likely to be misclassified since mistakes were made nearby on the validation set. Indeed, these examples look like they come from the "dirty" part of dirty-MNIST, and most digits are not clearly classifiable. The last row contains B&I examples, which exhibit both traits.
4.3 Benchmark Model Uncertainty Categorization
In this section, we demonstrate how DAUC categorizes existing UQ model uncertainty. We compare UQ methods MC-Dropout [10] (MCD), Deep-Ensemble [6] (DE) and Bayesian Neural Networks [9] (BNNs). These methods output predictions and uncertainty scores simultaneously, we follow the traditional approach to mark examples as uncertain or trusted according to their uncertain scores and specified thresholds. To demonstrate how DAUC categorizes all ranges of uncertain examples, we present results from top  $5\%$  to the least  $5\%$  uncertainty.
Figure 5 compares the proportion of different classes of flagged examples across the three UQ methods. The first and second rows show the total number and proportion of flagged examples for each class, respectively. There is a significant difference in what the different UQ methods identify. There is a significant difference between types of classes that the different UQ methods identify as discussed in the literature [19-21], which have shown that some UQ methods might not yield high quality uncertainty estimates due to the learning paradigm or sensitivity to parameters. Let us look at each column more carefully:
Figure 5: Results of applying our method in categorizing different uncertainty estimation methods. First row: comparisons on the numbers in different classes of examples. Second row: comparisons on the proportion of different classes of flagged examples to the total number of identified uncertain examples. Different methods tend to identify different certain types of uncertain examples. The results presented are based on 8 repeated runs with different random seeds.
1. DE tends to identify the OOD examples as the most uncertain examples. Specifically, looking at the bottom figure we see that the top  $5\%$  untrusted examples identified by DE are almost all OOD examples, which is not the case for the other UQ methods. By contrast, MCD is poor at identifying OOD examples; it flags some of the OOD samples as the most certain. This is explained by MCD's mechanism. Uncertainty is based on the difference in predictions across different drop-outs, however this could lead to outliers always having the same prediction—due to correlation between different nodes in the MCD model, extreme values of the OOD examples may always saturate the network and lead to the same prediction, even if some nodes are dropped out. DE is most apt at flagging the OOD class. This confirms the finding by [20] who showed that DE outperforms other methods under dataset shift—which is effectively what OOD represents.
2. After the OOD examples have been flagged, the next most suspicious examples are the Bnd and IDM classes—see columns 2 and 3. The number of these examples increases almost linearly with the number of flagged examples, until at about  $88\%$  no more examples are flagged as IDM and Bnd. This behaviour is explained by the Vanilla MNIST examples—which are generally distinguishable and relatively easily classified correctly—accounting for about  $15\%$  of the test examples.
3. As expected, the number of examples belonging to the Other class increases when more examples are flagged as uncertain. This makes sense, as the Other class indicates we cannot flag why the methods flagged these examples as uncertain, i.e. maybe these predictions should in fact be trusted.
4.4 Improving Uncertain Predictions
In this section, we explore the inverse direction, i.e. employing DAUC's categorization for creating better models. We elaborate the practical method in Appendix B. We experiment on UCI's Covtype, Digits and Spam dataset [43] with linear models (i.e.  $g = Id$ ) and experiment on DMNIST with ResNet-18 learning the latent representation.
Our empirical studies (Figure 8, Appendix C) have shown that the B&I examples are generally hardest to classify, hence we demonstrate the use case of DAUC for improving the predictive performance on this class. We vary the proportion  $q$  of training samples that we discard before training new prediction model  $f_{\mathrm{B\& I}}$  —only saving the training samples that resemble the B&I dataset most—see Figure 6. We find that retraining the linear model with filtered training data according to Eq. 6 significantly improves the performance. We observe that performance increases approximately linearly proportional to  $q$ , until the amount of training data becomes too low. The latter depends on the dataset and model used.
Figure 6: Improved uncertain predictions. Left to right: Covtype, Digits, Spam, DMNIST. Experiment are performed with different proportion of training samples discarded: e.g., with  $q = 0.0$ , all examples in the training set are used; while with  $q = 0.9$ , only top  $10\%$  examples most resembling the test data are used for training. The results presented are based on 10 repeated runs with different random seeds.
We have proposed DAUC, a framework for model uncertainty categorization. DAUC categorizes uncertain examples identified by UQ benchmarks into three classes—OOD, Bnd and IDM. These classes correspond to different causes for the uncertainty and require different strategies for possibly better predictions. We have demonstrated the power of DAUC by inspecting three different UQ methods—highlighting that each one identifies different examples. We believe DAUC can aid the development and benchmarking of UQ methods, paving the way for more trustworthy ML models.
In future work, DAUC has great potential to be extended to more general tasks, such as the regression setting, and reinforcement learning setting, where uncertainty quantification is essential. The idea of separating the source of uncertainty improves not only exploration [44] but also exploitation in the offline settings [30-35].
In the era of Large Language Models (LLMs) [45, 46], uncertainty quantification is essential in evaluating the task performance of LLMs [47], and holds great potential for AI alignment [48, 49] — as understanding the ability boundary of LLMs is essential, and identifying the suspicious outputs of LLMs can be potentially addressed by extending the framework of DAUC to the LLMs' setting.
HS acknowledges and thanks the funding from the Office of Naval Research (ONR). We thank the van der Schaar lab members for reviewing the paper and sharpening the idea. We thank all anonymous reviewers, ACs, SACs, and PCs for their efforts and time in the reviewing process and in improving our paper.
[51] Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00805, 2017.
[52] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
[53] Arnaud Van Looveren, Janis Klaise, Giovanni Vacanti, Oliver Cobb, Ashley Scillitoe, and Robert Samoilescu. Alibi detect: Algorithms for outlier, adversarial and drift detection, 2019. URL https://github.com/SeldonIO/alibi-detect.
[54] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining, pages 413-422. IEEE, 2008.
[55] Simon J Sheather and Michael C Jones. A reliable data-based bandwidth selection method for kernel density estimation. Journal of the Royal Statistical Society: Series B (Methodological), 53(3):683-690, 1991.
[56] David W Scott. Multivariate density estimation and visualization. In Handbook of computational statistics, pages 549-569. Springer, 2012.
[57] Bernard W Silverman. Density estimation for statistics and data analysis. Routledge, 2018.
[58] Soumya Ghosh, Q. Vera Liao, Karthikeyan Natesan Ramamurthy, Jiri Navratil, Prasanna Sattigeri, Kush R. Varshney, and Yunfeng Zhang. Uncertainty quantification 360: A holistic toolkit for quantifying and communicating the uncertainty of ai, 2021.
[59] Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine learning research, 12:2825-2830, 2011.
A Missing Details
A.1 Motivations for working with model latent space
In Section 3, we introduced the confusion density matrix that allows us to categorize suspicious examples at testing time. Crucially, this density matrix relies on kernel density estimations in the latent space  $\mathcal{H}$  associated with the model  $f$  through Assumption 1. Why are we performing a kernel density estimation in latent space rather than in input space  $\mathcal{X}$ ? The answer is fairly straightforward: we want our density estimation to be coupled to the model and its predictions.
Let us now make this point more rigorous. Consider two input examples  $x_{1}, x_{2} \in \mathcal{X}$ . The model assigns a representations  $g(x_{1}), g(x_{2}) \in \mathcal{H}$  and class probabilities  $f(x_{1}), f(x_{2}) \in \mathcal{Y}$ . If we define our kernel  $\kappa$  in latent space  $\mathcal{H}$ , this often means² that  $\kappa[g(x_{1}), g(x_{2})]$  grows as  $\|g(x_{1}) - g(x_{2})\|_{\mathcal{H}}$  decreases. Hence, examples that are assigned a similar latent representation by the model  $f$  are related by the kernel. Since our whole discussion revolves around model predictions, we would like to guarantee that two examples related by the kernel are given similar predictions by the model  $f$ . In this way, we would be able to interpret a large kernel density  $\kappa[g(x_{1}), g(x_{2})]$  as a hint that the predictions  $f(x_{1})$  and  $f(x_{2})$  are similar. We will now show that, under Assumption 1, such a guarantee exists. Similar to [50], we start by noting that
$$
\begin{array}{l} \| (l \circ g) (x _ {1}) - (l \circ g) (x _ {2}) \| _ {\mathbb {R} ^ {C}} = \| l [ g (x _ {1}) - g (x _ {2}) ] \| _ {\mathbb {R} ^ {C}} \\ \leq \left\| l \right\| _ {\mathrm {o p}} \left\| g (x _ {1}) - g (x _ {2}) \right\| _ {\mathcal {H}}, \\ \end{array}
$$
where  $\| \cdot \|_{\mathbb{R}^C}$  is a norm on  $\mathbb{R}^C$  and  $\| l\|_{\mathrm{op}}$  is the operator norm of the linear map  $l$ . In order to extend this inequality to black-box predictions, we note that the normalizing map in Assumption 1 is often a Lipschitz function with Lipschitz constant  $\lambda \in \mathbb{R}$ . For instance, a Softmax function with inverse temperature constant  $\lambda^{-1}$  is  $\lambda$ -Lipschitz [51]. We use this fact to extend our inequality to predicted class probabilities:
$$
\begin{array}{l} \left\| f \left(x _ {1}\right) - f \left(x _ {2}\right) \right\| _ {\mathcal {Y}} = \left\| \left(\varphi \circ l \circ g\right) \left(x _ {1}\right) - \left(\varphi \circ l \circ g\right) \left(x _ {2}\right) \right\| _ {\mathcal {Y}} \\ \leq \lambda \| (l \circ g) (x _ {1}) - (l \circ g) (x _ {2}) \| _ {\mathbb {R} ^ {C}} \\ \leq \lambda \| l \| _ {\mathrm {o p}} \| g (x _ {1}) - g (x _ {2}) \| _ {\mathcal {H}}. \\ \end{array}
$$
This crucial inequality guarantees that examples  $x_{1}, x_{2} \in \mathcal{X}$  that are given a similar latent representation  $g(x_{1}) \approx g(x_{2})$  will also be given a similar prediction  $f(x_{1}) \approx f(x_{2})$ . In short: two examples that are related according to a kernel density defined in the model latent space  $\mathcal{H}$  are guaranteed to have similar predictions. This is the motivation we wanted to support the definition of the kernel  $\kappa$  in latent space.
An interesting question remains: is it possible to have similar guarantees if we define the kernel in input space? When we deal with deep models, the existence of adversarial examples indicates the opposite [52]. Indeed, if  $x_{2}$  is an adversarial example with respect to  $x_{1}$ , we have  $x_{1} \approx x_{2}$  (and hence  $\| x_{1} - x_{2} \|_{\mathcal{X}}$  small) with two predictions  $f(x_{1})$  and  $f(x_{2})$  that are significantly different. Therefore, defining the kernel  $\kappa$  in input space might result in relating examples that are given a significantly different prediction by the model. For this reason, we believe that the latent space is more appropriate in our setting.
A.2 Details: Flagging IDM and Bnd Examples with Thresholds
In order to understand uncertainty, it will be clearer to map those scores into binary classes with thresholds. In our experiments, we use empirical quantiles as thresholds. e.g., to label an example as IDM, we specify an empirical quantile number  $q$ , and calculate the corresponding threshold based on the order statistics of IDM Scores for test examples:  $S_{\mathrm{IDM}}^{(1)}, \ldots, S_{\mathrm{IDM}}^{(|D_{\mathrm{test}}|)}$ , where  $S_{\mathrm{IDM}}^{(n)}$  denotes the  $n$ -th smallest IDM score out of  $|\mathcal{D}_{\mathrm{test}}|$  testing-time examples. Then, the threshold given quantile number  $q$  is
$$
\tau_ {\mathrm {I D M}} (q) \equiv S _ {\mathrm {I D M}} ^ {(\left| \left| \mathcal {D} _ {\mathrm {t e s t}} \right| \cdot q \right\rfloor)}.
$$
2This is the case for all the kernels that rely on a distance (e.g. the Radial Basis Function Kernel, the Matern kernel or even Polynomial kernels [39]).
Similarly, we can define quantile-based threshold in flagging Bnd examples based on the order statistics of Bnd Scores for test examples, such that for given quantile  $q$ ,
$$
\tau_ {\mathrm {B n d}} (q) \equiv S _ {\mathrm {B n d}} ^ {(\left| \left| \mathcal {D} _ {\text {t e s t}} \right| \cdot q \right\rfloor)}.
$$
Practically, a natural choice of  $q$  is to use the validation accuracy: when there are  $1 - q$  examples misclassified in the validation set, we also expect the testing-time in distribution examples with the highest  $1 - q$  to be marked as Bnd or IDM examples.
B Improving Predicting Performance of Uncertain Examples
Knowing the category that a suspicious example belongs to, can we improve its prediction? For ease of exposition, we focus on improving predictions for  $S_{\mathrm{B\&I}}$ .
Let  $p(x \mid S_{\mathrm{B\&I}})$  be the latent density be defined as in Definition 1. We can improve the prediction performance of the model on  $S_{\mathrm{B\&I}}$  examples by focusing on the part of examples in the training set that are closely related to those suspicious examples. We propose to refine the training dataset  $D_{\mathrm{train}}$  by only keeping the examples that resembles the latent representations for the specific type of test-time suspicious examples, and train another model on this subset of the training data:
$$
\tilde {\mathcal {D}} _ {\text {t r a i n}} \equiv \left\{x \in \mathcal {D} _ {\text {t r a i n}} \mid p (x \mid \mathcal {S} _ {\mathrm {B} \& \mathrm {I}}) \geq \tau_ {\text {t e s t}} \right\}, \tag {6}
$$
where  $\tau_{\mathrm{test}}$  is a threshold that can be adjusted to keep a prespecified proportion  $q$  of the related training data. Subsequently, new prediction model  $f_{\mathrm{B\&I}}$  is trained on  $\hat{\mathcal{D}}_{\mathrm{train}}$ .
Orthogonal to ensemble methods that require multiple models trained independently, and improve overall prediction accuracy by bagging or boosting, our method is targeted at improving the model's performance on a specified subclass of test examples by finding the most relevant training examples. Our method is therefore more transparent and can be used in parallel with ensemble methods if needed.
Threshold  $\tau_{\mathrm{test}}(q)$  For every training example  $x\in \mathcal{D}_{\mathrm{train}}$  we have the latent density  $p(x|\mathcal{D}_{\mathrm{B\& I}})$  over the B&I class of the test set. With their order statistics  $p_{(1)}(x|\mathcal{D}_{\mathrm{B\& I}}),\dots,p_{(|\mathcal{D}_{\mathrm{train}}|)}(x|\mathcal{D}_{\mathrm{B\& I}})$ . Given quantile number  $q$  our empirical quantile based threshold  $\tau_{\mathrm{test}}$  is chosen as
$$
\tau_ {\text {t e s t}} (q) \equiv p _ {(\lfloor q \cdot | \mathcal {D} _ {\text {t r a i n} | \rfloor})} (x | \mathcal {D} _ {\mathrm {B} \& \mathrm {I}}).
$$
During the inverse training time, we train our model to predict those B&I class of test examples only with the training data with higher density than  $\tau_{\mathrm{test}}(q)$ . We experiment with different choices of  $q$  in the experiment (Figure 6 in Sec. 4.4).
C.1 Categorization of Uncertainty under Different Thresholds
In the main text, we provide results with  $\tau_{\mathrm{Bnd}} = \tau_{\mathrm{IDM}} = 0.8$ , which approximates the accuracy on validation set—as a natural choice. In this section, we vary these thresholds and show in Figure 7 that changing those thresholds does not significantly alter the conclusions drawn above.
Figure 8 looks more closely into the top  $25\%$  uncertain examples for each method, and the accuracy on each of the uncertainty classes. As expected, the accuracy of the B&I examples is always lower than that of the trusted class, meaning that those examples are most challenging for the classifier. And the accuracy of flagged classes are always lower than the other class, verifying the proposed categorization of different classes.
C.2 Inverse Direction: More Results
In the main text, we show the results on improving prediction performance on the B&I class with training example filtering (On the Covtype, Digits dataset). More results on other classes of examples are provided in this section.
We experiment on three UCI datasets: Covtype, Digits, and Spam. And experiment with three classes we defined in this work:
(a)  $\tau_{\mathrm{BD}}(0.5),\tau_{\mathrm{IDM}}(0.5)$
(b)  $\tau_{\mathrm{BD}}(0.6),\tau_{\mathrm{IDM}}(0.6)$
(c)  $\tau_{\mathrm{BD}}(0.7),\tau_{\mathrm{IDM}}(0.7)$
(d)  $\tau_{\mathrm{BD}}(0.8),\tau_{\mathrm{IDM}}(0.8)$