SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tsai Hor Chan
Department of Statistics and Actuarial Science
The University of Hong Kong
hchanth@connect.hku.hk
Kin Wai Lau
TCL AI Lab
stevenlau@tcl.com
Jiajun Shen
shenjiajun90@gmail.com
Guosheng Yin
guosheng.yin@imperial.ac.uk
Lequan  $\mathbf{Y}\mathbf{u}^{*}$
lqyu@hku.hk
Uncertainty estimation aims to evaluate the confidence of a trained deep neural network. However, existing uncertainty estimation approaches rely on low-dimensional distributional assumptions and thus suffer from the high dimensionality of latent features. Existing approaches tend to focus on uncertainty on discrete classification probabilities, which leads to poor generalizability to uncertainty estimation for other tasks. Moreover, most of the literature require seeing the out-of-distribution (OOD) data in the training for better estimation of uncertainty, which limits the uncertainty estimation performance in practice because the OOD data are typically unseen. To overcome these limitations, we propose a new framework using data-adaptive high-dimensional hypothesis testing for uncertainty estimation, which leverages the statistical properties of the feature representations. Our method directly operates on latent representations and thus does not require retraining the feature encoder under a modified objective. The test statistic relaxes the feature distribution assumptions to high dimensionality, and it is more discriminative to uncertainties in the latent representations. We demonstrate that encoding features with Bayesian neural networks can enhance testing performance and lead to more accurate uncertainty estimation. We further introduce a family-wise testing procedure to determine the optimal threshold of OOD detection, which minimizes the false discovery rate (FDR). Extensive experiments validate the satisfactory performance of our framework on uncertainty estimation and task-specific prediction over a variety of competitors. The experiments on the OOD detection task also show satisfactory performance of our method when the OOD data are unseen in the training. Codes are available at https://github.com/HKU-MedAI/bnn Uncertainty.
Deep neural networks (DNNs) have demonstrated state-of-the-art (SOTA) performances on many problem domains, such as computer vision [30, 31], medical diagnosis [24, 16, 38, 2], and recommendation systems [1, 3]. Despite their successes, most of the existing DNN designs can only recognize in-distribution samples (i.e., samples from training distributions) but cannot measure the confidence level (i.e., the uncertainties) in the prediction, especially for out-of-distribution (OOD) samples. Estimation of prediction uncertainties plays an important role in many machine learning applications [25, 37]. For instance, uncertainty estimation can help to detect OOD samples in the data and inspect anomalies in the system. Further, the uncertainty estimates can indicate the distributional shifts in the environment and thus facilitate learning in a non-stationary environment.
In response to the high demand, several attempts have been made to obtain a better estimate of the uncertainty, which can be roughly divided into two categories — Bayesian and non-Bayesian methods. Bayesian methods [4, 19] mainly operate with a Bayesian neural network (BNN), which introduces a probability distribution (e.g., multivariate Gaussian) to the neural network weights. This enables the model to address model-wise (i.e., epistemic) uncertainty by drawing samples from the posterior distribution. Non-Bayesian methods [32, 27, 10], on the other hand, assume a distribution on the model outputs (e.g., classification probabilities). The uncertainty scores can be obtained by evaluating a pre-determined metric (e.g., classification entropy) on the derived distribution.
However, most of the aforementioned methods are subject to several drawbacks: (1) They rely on strong assumptions (e.g., parametric models) on low-dimensional features and thus suffer from the curse of high dimensionality, leading to poor performance when the dimension of the output is high. (2) They are limited to classification problems, and existing methods explicitly make assumptions about the classification probabilities. This leads to poor generalizability in uncertainty estimation for other tasks, such as regression and representation learning. (3) Their performances heavily rely upon the feature encoder, which can be compromised when the features are of poor quality (e.g., the number of samples used to train the encoder is small). (4) They require a modification of the training loss, and thus additional training is needed when applying the methods to a new problem. Pretrained features cannot be directly applied to the OOD tasks without retraining for their proposed loss.
Observing the above limitations, we propose a new framework for uncertainty estimation. Our framework comprises two key components: one Bayesian deep learning encoding module and one uncertainty estimation module using the adaptable regularized Hotelling  $T^2$  (ARHT) [26]. Our contributions are summarized as: (1) We formulate uncertainty estimation as a multiple high-dimensional hypothesis testing problem, and adopt a Bayesian deep learning module to better address the aleatoric and epistemic uncertainties when learning feature distributions. (2) We propose using the ARHT test statistic for measuring uncertainty and demonstrate the advantages of ARHT over the existing uncertainty measures, including its consistency and robustness properties. This enables us to design a data-adaptive detection method so that each individual data point can be assigned an optimal hyperparameter. Because it relaxes the strong assumptions on latent feature distributions, ARHT less sensitive to the feature quality. As result, our method can be interpreted as a post-hoc method as it works on feature distributions generated by any encoder (e.g., a pre-trained encoder). (3) We adopt the family-wise testing procedure to determine the optimal threshold of OOD detection, which minimizes the false discovery rate (FDR). (4) We perform extensive experiments on standard and medical image datasets to validate our method on OOD detection and image classification tasks compared to SOTA methods. Our proposed method does not require prior knowledge of the OOD data, while it yields satisfactory performance even when the training samples are limited.
Deep Neural Network: A deep neural network (DNN) with  $L$  layers can be defined as
$$
f _ {l} (\boldsymbol {x}; W _ {l}, b _ {l}) = \frac {1}{\sqrt {D _ {l - 1}}} \Big (W _ {l} \phi \big (f _ {l - 1} (\boldsymbol {x}; W _ {l - 1}, b _ {l - 1})) \Big) + b _ {l}, \quad l \in \{1, \dots , L \},
$$
where  $\phi$  is a nonlinearity activation function,  $\pmb{x}$  is the input,  $D_{l-1}$  is the dimension of the input,  $b_{l} \in R^{D_{l}}$  is a vector of bias parameters for layer  $l$ , and  $W_{l} \in R^{D_{l} \times D_{l-1}}$  is the matrix of weight parameters. Let  $\pmb{w}_{l} = \{W_{l}, b_{l}\}$  denote the weight and bias parameters of layer  $l$ , and the entire trainable network parameters are denoted as  $\pmb{\theta} = \{\pmb{w}_{l}\}_{l=1}^{L}$ .
Bayesian neural network (BNN): A BNN specifies a prior  $\pi (\pmb {\theta})$  on the trainable parameters  $\pmb{\theta}$ . Given the dataset  $\mathcal{D} = \{\pmb {x}_i,\pmb {y}_i\}_{i = 1}^N$  of  $N$  pairs of observations and responses, we aim to estimate the posterior distribution of  $\pmb{\theta}$ ,  $p(\pmb {\theta}|\mathcal{D}) = \pi (\pmb {\theta})\prod_{i = 1}^{N}p(y_i|f(\pmb {x}_i;\pmb {\theta})) / p(\mathcal{D})$ , where  $p(y_{i}|f(\pmb{x}_{i};\pmb{\theta}))$  is the likelihood function and  $p(\mathcal{D})$  is the normalization term.
Pooled Sample Covariance Matrix: Let  $\{X_{1j}\}_{j = 1}^{n_1}$  be the  $p$ -dimensional embeddings from the training images and  $\{X_{2j}\}_{j = 1}^{n_2}$  be the  $p$ -dimensional embeddings from the testing images. The pooled sample covariance is defined as
$$
\boldsymbol {S} _ {n} = \frac {1}{n - 2} \sum_ {k = 1} ^ {2} \sum_ {j = 1} ^ {n _ {k}} \left(\boldsymbol {X} _ {k j} - \bar {\boldsymbol {X}} _ {k}\right) \left(\boldsymbol {X} _ {k j} - \bar {\boldsymbol {X}} _ {k}\right) ^ {\top}, \tag {1}
$$
where  $n = n_1 + n_2$ ,  $n_k$  is the sample size and  $\bar{X}_k$  is the sample mean of the  $k$ -th set  $(k = 1, 2)$ .
High-Dimensional Test Statistics: The Hotelling  $T^2$  test statistic [15] is given by
$$
T = \frac {n (n - p)}{p (n - 1)} \left(\bar {\boldsymbol {X}} _ {1} - \bar {\boldsymbol {X}} _ {2}\right) ^ {\top} \boldsymbol {S} _ {n} ^ {- 1} \left(\bar {\boldsymbol {X}} _ {1} - \bar {\boldsymbol {X}} _ {2}\right). \tag {2}
$$
The Hotelling  $T^2$  test assumes  $T \sim F(p, n - p)$  under the null hypothesis. To resolve the potential singularity of the covariance matrix, the regularized Hotelling  $T^2$  (RHT) test statistic [5] loads an identity matrix  $I_p$  to  $T$ ,
$$
\mathrm {R H T} (\lambda) = \frac {n _ {1} n _ {2}}{n _ {1} + n _ {2}} \left(\bar {\boldsymbol {X}} _ {1} - \bar {\boldsymbol {X}} _ {2}\right) ^ {\top} \left(\boldsymbol {S} _ {n} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \left(\bar {\boldsymbol {X}} _ {1} - \bar {\boldsymbol {X}} _ {2}\right), \tag {3}
$$
where  $\lambda$  is a tuning parameter. The ARHT test statistic, which will be formally introduced by Eq. 4 in Section 3.3, standardizes the RHT and addresses the skewness of Hotelling's  $T^2$  when the feature dimension is high.
Our uncertainty estimation framework comprises a Bayesian neural network encoder and an OOD testing module with ARHT as the uncertainty measure. Figure 1 provides an overview of our proposed framework, with the detailed algorithm given in the appendix. Ablation studies on key components of our framework are provided in Section 5.
3.1 Problem Definition and Test Hypothesis
Suppose that we have a set of samples  $\{I_1, \ldots, I_n\}$  as the training set, and a set of testing samples  $I_1', \ldots, I_n'$  containing both in-distribution and OOD data. We aim to develop an uncertainty estimation framework that can accurately classify the test samples as either in-distribution or OOD, without seeing the OOD data during the training. Moreover, the framework can still perform well on its predictive task (e.g., classification) without sacrifice in uncertainty estimation. We set the null hypothesis  $H_0: \pmb{\mu}_1 = \pmb{\mu}_2$  and the alternative hypothesis as  $H_1: \pmb{\mu}_1 \neq \pmb{\mu}_2$ , where  $\pmb{\mu}_1$  and  $\pmb{\mu}_2$  are the mean representations of the training and the testing samples, respectively.
3.2 Bayesian Neural Network as Encoder
We adopt a BNN encoder and train it with stochastic variational inference (SVI) [29] to learn the representation of the input. In SVI, a posterior distribution  $p(\pmb{\theta}|\pmb{y})$  is approximated by a distribution  $q$  selected from a candidate set  $\mathcal{Q}$  by maximizing an evidence lower bound (ELBO):  $\max_{q\in \mathcal{Q}}\mathbb{E}_{\pmb{\theta}\sim q}[\log p(\pmb{y}|\pmb{\theta})] - \mathrm{KL}(q\| \pi)$ , where  $\mathrm{KL}(q\| \pi)$  is the Kullback-Leibler divergence between the variational posterior distribution  $q$  and the prior distribution  $\pi$ ,  $p(\pmb{y}|\pmb{\theta})$  is the likelihood, and  $\mathbb{E}_{\pmb{\theta}\sim q}[\log p(\pmb{y}|\pmb{\theta})]$  represents the learning objective. This can be supervised (e.g., cross-entropy loss) or unsupervised (e.g., contrastive learning loss) learning. The KL divergence of two multi-variate Gaussian distributions is provided in the appendix. We use gradient descent to optimize the ELBO, i.e., to minimize the distance between prior and variational posterior distributions. We obtain the trained BNN encoder once the optimization step is completed. Since BNNs operate on an ensemble
Figure 1: Workflow of our proposed uncertainty estimation framework. Our framework contains a BNN encoding module and an uncertainty estimation module using high-dimensional testing. The training objective  $\mathcal{L}_{\mathrm{obj}}$  can be either supervised (e.g., cross-entropy) or unsupervised (e.g., contrastive learning). The ELBO represents the evidence lower bound used for training BNN.
of posterior model weights, they can capture the epistemic uncertainties in the posterior predictive distribution  $p(\pmb{y}|\mathcal{D})$ . Hence, using BNNs instead of frequentist architectures can better approximate the posterior distribution of the feature embeddings.
To obtain the uncertainty scores, we need to compute the sample means  $\bar{X}_1$ ,  $\bar{X}_2$  and covariance matrices  $\Sigma_1, \Sigma_2$  for the training data (containing in-distribution data only) and the testing data (containing both in-distribution and OOD data). First, we sample  $s$  weights for every data point in the training data, and generate representations for each data point using the sampled weights. In total,  $n_1$  representations are generated from the training data. We can compute the mean  $\bar{X}_1$  and covariance matrix  $\Sigma_1$  of the training data using these  $n_1$  representations. For each testing data point  $x_t$ , we sample  $n_2$  weights from the variational posterior distribution, and generate  $n_2$  representations  $\{\hat{x}_{tj} = f(x_t; \boldsymbol{\theta}_j)\}_{j=1}^{n_2}$ , where  $\boldsymbol{\theta}_j$  is the  $j$ -th weight sample drawn from the trained posterior distribution  $p(\boldsymbol{\theta} | \mathcal{D})$ . We obtain the mean  $\bar{X}_2 = \sum_{j=1}^{n_2} \hat{x}_{tj} / n_2$  and the covariance matrix  $\Sigma_2 = \sum_{j=1}^{n_2} (\hat{x}_{tj} - \bar{X}_2)(\hat{x}_{tj} - \bar{X}_2)^\top / n_2$ . Hence, we can compute the pooled covariance matrix  $S_n$  by Eq. (1).
3.3 High-Dimensional Testing as Uncertainty Measure
With the pooled sample covariance matrix, we can compute the ARHT test statistics by Eq. (3) and Eq. (4), where  $\bar{X}_1$  is the mean of training samples and  $\bar{X}_2$  is the mean of  $n_2$  embeddings of each testing sample. To solve the skew  $F$  distribution of RHT when  $n \gg p$  for large  $n$  and  $p$  (as shown in Figure 2), we adopt the ARHT statistic to perform a two-sample test which has robust regularization of the covariance matrix [26]. In particular, the ARHT is given by
$$
\operatorname {A R H T} (\lambda) = \sqrt {p} \frac {p ^ {- 1} \operatorname {R H T} (\lambda) - \hat {\Theta} _ {1} (\lambda , \gamma)}{\left\{2 \hat {\Theta} _ {2} (\lambda , \gamma) \right\} ^ {\frac {1}{2}}}, \tag {4}
$$
where  $\hat{\Theta}_1(\lambda, \gamma) = \{1 - \lambda m_F(-\lambda)\} / \{1 - \gamma (1 - \lambda m_F(-\lambda))\}$ ,
$$
\hat {\Theta} _ {2} (\lambda , \gamma) = \frac {1 - \lambda m _ {F} (- \lambda)}{\left[ 1 - \gamma (1 - \lambda m _ {F} (- \lambda)) \right] ^ {3}} - \lambda \frac {m _ {F} (- \lambda) - \lambda m _ {F} ^ {\prime} (- \lambda)}{\left[ 1 - \gamma (1 - \lambda m _ {F} (- \lambda)) \right] ^ {4}},
$$
$$
m _ {F} (z) = \frac {1}{p} \operatorname {t r} \left\{\boldsymbol {R} _ {n} (z) \right\}, \quad m _ {F} ^ {\prime} (z) = \frac {1}{p} \operatorname {t r} \left\{\boldsymbol {R} _ {n} ^ {2} (z) \right\}, \quad \boldsymbol {R} _ {n} (z) = \left(\boldsymbol {S} _ {n} - z \boldsymbol {I} _ {p}\right) ^ {- 1}, \quad \gamma = \frac {p}{n}.
$$
As a result, we have  $\mathrm{ARHT}(\lambda) \sim \mathcal{N}(0,1)$  [26], which prevents the catastrophic skewness of  $F$ -distribution in high dimensions (see Figure 2) and yields smoother uncertainty scores. The ARHT can
Table 1: The OOD detection performance (in %) of our method, BNN-ARHT, compared to various competitors, using the LeNet [22] architecture. Standard deviations are given in brackets.
Model	In-Distrib.	Fashion-MNIST	OOD Datasets	SVHN
AUC	AUPR	AUC	AUPR	AUC	AUPR
MC Dropout [10]	MNIST	99.33 (0.3)	99.27 (0.3)	99.85 (0.09)	99.88 (0.07)	99.96 (0.007)	99.96 (0.002)
Deep Ensembles [21]	MNIST	90.70 (8.4)	91.08 (7.7)	99.70 (8.4)	91.08 (7.7)	99.21 (0.9)	99.68 (0.4)
Kendall and Gal [19]	MNIST	92.54 (2.6)	92.77 (1.9)	94.11 (4.9)	93.40 (5.7)	99.60 (0.5)	99.13 (1.0)
EDL [32]	MNIST	73.43 (16.0)	80.22 (11.1)	72.61 (8.6)	81.42 (7.0)	63.43 (1.2)	85.09 (3.4)
DPN [27]	MNIST	99.41 (0.2)	99.37 (0.3)	99.96 (0.03)	99.96 (0.03)	99.96 (0.01)	99.96 (0.003)
PostNet [4]	MNIST	98.59 (0.4)	94.70 (0.5)	—	—	—	—
Detector [12]	MNIST	75.57 (15.2)	83.75 (29.9)	95.71 (11.0)	85.00 (30.0)	77.92 (12.4)	83.75 (37.3)
BNN-ARHT (Ours)	MNIST	99.51 (0.4)	99.47 (0.3)	99.98 (0.01)	99.98 (0.004)	99.97 (0.007)	99.96 (0.004)
MC Dropout [10]	CIFAR10	76.23 (5.6)	74.21 (5.3)	77.15 (2.2)	79.0 (1.9)	78.09 (1.2)	84.35 (1.1)
Deep Ensembles [21]	CIFAR10	71.25 (3.0)	75.32 (2.3)	86.77 (3.6)	90.35 (3.1)	76.15 (5.3)	82.62 (13.1)
Kendall and Gal [19]	CIFAR10	77.41 (17.3)	77.00 (18.2)	89.08 (17.8)	90.93 (14.8)	67.40 (3.1)	71.44 (10.1)
EDL [32]	CIFAR10	67.81 (12.1)	71.81 (11.5)	77.53 (14.4)	80.50 (11.7)	69.57 (4.7)	83.74 (3.4)
DPN [27]	CIFAR10	57.54 (1.7)	68.29 (3.4)	62.34 (3.7)	70.49 (8.0)	57.48 (4.4)	77.76 (6.2)
PostNet [4]	CIFAR10	—	—	—	—	76.04 (1.6)	69.30 (1.7)
Detector [12]	CIFAR10	76.46 (15.3)	71.63 (21.5)	76.99 (15.9)	91.00 (24.4)	76.01 (13.6)	90.00 (22.9)
BNN-ARHT (Ours)	CIFAR10	77.78 (5.0)	79.06 (6.8)	92.77 (1.8)	93.74 (1.0)	82.01 (1.2)	91.61 (0.3)
be interpreted as a more robust distance measure on embedding distributions compared to existing metrics such as Mahalanobis distance.
We select  $\lambda$  from a predefined set using a data-adaptive method [26]. The set for grid search is chosen as  $\{\lambda_0, 5\lambda_0, 10\lambda_0\}$  given the hyperparameter  $\lambda_0$ . For a testing data point  $x_t$ , we compute  $Q(\lambda, \gamma; \pmb{\xi}) = \sum_{k=0}^{2} \xi_k \hat{\rho}_k(-\lambda, \gamma) / \{\gamma \hat{\Theta}_2(\lambda, \gamma)\}^{1/2}$ , for each  $\lambda$  in the candidate set, where  $\pmb{\xi} = (\xi_0, \xi_1, \xi_2) \in \mathbb{R}^3$  is a pre-specified weight vector set as  $\pmb{\xi} = (0, 1, 0)$ ,  $\hat{\rho}_0(-\lambda, \gamma) = m_F(-\lambda)$ ,  $\hat{\rho}_1(-\lambda, \gamma) = \hat{\Theta}_1(\lambda, \gamma)$ , and  $\hat{\rho}_2(-\lambda, \gamma) = \{1 + \gamma \hat{\Theta}_1(\lambda, \gamma)\} \{p^{-1} \mathrm{tr}\{S_n\} - \lambda \hat{\rho}_1(-\lambda, \gamma)\}$ . The optimal  $\lambda$  is then selected by maximizing the  $Q$  function, i.e.,  $\arg \max_{\lambda} Q(\lambda, \gamma; \pmb{\xi})$ . For each testing data point, we can proceed with the above selection process to determine the optimal  $\lambda$  for the individual input. This makes the determination of uncertainty scores more flexible and data-adaptive.
3.4 Optimal Threshold Adjusting for Family-Wise Discovery Error
We obtain the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPR) using different thresholds. To determine the optimal threshold, we adopt a family-wise testing procedure to compute the  $p$ -values.
It is necessary to balance the tradeoff between the power and the type I error rate of the test. We adopt the Benjamini-Hochberg (BH) procedure [35] procedure to adjust for multiple tests. The threshold of  $p$ -values is calibrated by the BH procedure, and we have a rejection set  $\hat{R}$  of indices to the samples whose  $p$ -values are below the threshold,  $\hat{R} = \{i \in \mathcal{I} : \hat{p}_i \leq \alpha \hat{k} / (mH_m)\}$ , where  $\hat{k} = \max \{k \in \mathcal{I} : \hat{p}_i \leq \alpha k(mH_m)\}$ ,  $\mathcal{I} = \{1, \dots, m\}$  is the set of indices corresponding to the  $m$  tests, and  $H_m = \sum_{j=1}^{m} 1/j$ . The samples in the rejection set are then classified as the OOD sample. The BH procedure is shown to achieve a more powerful hypothesis testing performance [13].
Figure 2: Comparison of the density functions of  $\mathcal{N}(0,1)$  and  $F$ -distribution under low (left) and high (right) dimensions.
4.1 Datasets and Experiment Setting
We design OOD detection tasks on both standard and medical image datasets to demonstrate the application of our framework. We also evaluate our framework on the image classification
Table 2: The OOD detection performance (in %) on DRD [8], with the LeNet [22] architecture.
Model	AUROC	AUPR
MC Dropout [10]	59.52 (1.1)	60.95 (6.0)
Kendall and Gal [19]	91.06 (9.8)	92.71 (7.8)
Deep Ensembles [21]	59.67 (1.3)	56.58 (2.3)
DPN [27]	60.57 (1.1)	65.32 (1.3)
EDL [32]	53.01 (1.9)	58.22 (9.6)
Detectron [12]	90.74 (9.9)	46.00 (31.4)
BNN-ARHT (Ours)	93.44 (3.8)	95.44 (2.2)
Table 3: Classification performance (in %) on MNIST, with the LeNet [22] architecture.
Model	Accuracy	F-1 Score
MC Dropout [10]	98.48	98.89
Kendall and Gal [19]	98.78	98.65
Deep Ensembles [21]	97.90	97.89
DPN [27]	98.89	98.83
EDL [32]	21.24	12.88
PostNet [4]	99.12	99.06
BNN-ARHT (Ours)	99.26	99.06
task in comparison with the baseline methods to show that our framework does not sacrifice classification performance. For the image classification task, we benchmark the classification performance of the encoder trained on a holdout set of the in-distribution dataset (i.e., MNIST).
For the OOD detection task, we treat CIFAR 10 and MNIST as the in-distribution datasets, and Fashion-MNIST, OMNIGLOT, and SVHN as the OOD datasets. To validate the advantage of BNN encoding on limited observations, we compose a medical image benchmark using samples from the Diabetes Retinopathy Detection (DRD) dataset [8], with samples shown in Figure 3. We treat healthy
Figure 3: Healthy (left) and unhealthy (right) samples of the DRD dataset.
samples as in-distribution data and unhealthy samples as OOD data. Distinct from the settings in some existing works [27] which include OOD data in the training, we exclude the OOD data when training the feature encoder. We use the AUROC and AUPR as the evaluation metrics for OOD detection tasks, and adopt accuracy and the macro F1-score to evaluate the classification performance. Detailed definitions of the evaluation metrics and further descriptions of the datasets can be found in the appendix.
4.2 Competitive Methods
We compare our proposed framework, named as BNN-ARHT, with seven competitors — (1) Deep ensembles [21]: an ensemble of neural networks to address the epistemic uncertainties, and the number of models in the ensemble is set as 5; (2) MC Dropout [10]: a non-Bayesian method using dropout on trained weights to produce Monte Carlo weight samples [27]; (3) Kendall and Gal [19]: the first work addressing the aleatoric and epistemic uncertainties in deep learning; (4) EDL [32]: it estimates uncertainty by collecting evidence from outputs of neural network classifiers by assuming a Dirichlet distribution on the class probabilities; (5) DPN [27]: it assumes a prior network with Dirichlet distributions on the classification outputs; (6) PostNet [4]: it uses normalizing flow to predict an individual closed-form posterior distribution over predicted class probabilities; (7) Detector [12]: it detects the change in distribution with the discordance between an ensemble of classifiers trained to agree on training data and disagree on testing data. Since Detector operates on small samples from each dataset for OOD detection (e.g., 50 over 60,000 for CIFAR 10), we use a larger number of runs (i.e., 100) for a fair comparison. Five-fold cross-validation is applied to each of the competitive methods. We report the means and standard deviations over the runs for each metric.
4.3 Predictive and Uncertainty Estimation Performance