SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A related ablation experiment comparing the frequentist and Bayesian architecture is presented in the main text. We additionally conduct an experiment with the Bayesian model architecture scaled up to ResNet50. We further conduct an experiment using the frequentist ResNet50 (the hypothesis test reduces to a one-sample test) and the testing AUROC is 72.77. These results validate the scalability of our method to large and modern vision architectures.
C.3 Encoder Quality
To assess the effect of the encoder quality, we fix the encoder (e.g., LeNet) when comparing our framework with the current SOTA methods. Since the baseline methods are trained on classification problems (e.g., the CIFAR 10 image classification and DRD auxiliary task), we use the training accuracy at different epochs to measure the quality of the encoder. We observe that the OOD detection performance is monotonously improved with the increase in the training accuracy (i.e., the encoder quality). Figure 9 presents an example of the OOD detection experiment on CIFAR10 and TinyImageNet.
C.4 Influence of  $n_2$ .
Figure 11 presents the OOD detection results under a more complex setting (In-distribution: CIFAR10, OOD: SVHN). We observe a similar pattern to that in Figure 5 in the main text. This shows that the sample covariance is more influenced by the  $n_1$  training/in-distribution samples, making the test statistics reflect more the training distribution (hence the overall consistent pattern). Future work on variance-adjusted test statistics may put more weight on the feature distributions of testing samples, so that we would more easily observe performance improvement as  $n_2$  increases in this case.
C.5 Under the Regression Setting.
We choose OOD as the benchmark task since it is the most common benchmark for uncertainty estimation. We additionally construct a regression setting with two multivariate Gaussian distributions of different means and variances indicating different distributions. Most of the uncertainty estimation frameworks cannot be applied to this setting because they only work under the classification settings.
Figure 9: The OOD detection performance in terms of AUROC with respect to the training accuracy of the encoder. We adopt CIFAR 10 as the training/in-distribution dataset and TinyImageNet as the OOD data.
Figure 10: Ablation study with respect to  $s$ . We Figure 11: Ablation study with respect to  $n_2$ . We use CIFAR10 as the in-distribution dataset and use CIFAR10 as the in-distribution dataset and SVHN as the OOD dataset. SVHN as the OOD dataset.
The results in Table 10 show that our method also achieves satisfactory performance under the regression settings, demonstrating its generalizability to other tasks.
C.6 Comparison with More Baselines
More baselines on uncertainty estimation are added for comparison: (1) I-EDL [6]: use the Fisher information matrix to measure the informativeness of evidence carried by each sample; and (2) RKL-PN [28]: prior networks trained with the reverse KL divergence. Table 11 presents the additional results, from which we observe that our method still achieve the state-of-the-art performance with more baselines included.
D Multivariate Gaussian Distribution
The multivariate Gaussian distribution is crucial for the approximation of a vanilla BNN [17]. We provide the formal definition and its important properties in this section. The density of a multivariate
Table 10: The OOD detection performance (in %) of our method, BNN-ARHT, compared with various competitors, using the ResNet50 architecture. We constructed a simulated regression setting where  $\mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma})$  is the distribution for in-distribution data and  $\mathcal{N}(-\boldsymbol{\mu},\boldsymbol{\Sigma})$  is the distribution for OOD data. The auxiliary regression task is to predict the norm of the sampled vector using an MLP with two layers. We choose  $\boldsymbol{\mu} = [0.5,\dots,0.5]^{\top}$ ,  $\boldsymbol{\Sigma} = 9\boldsymbol{I}_p$ , and  $p = 128$ .
Model	AUROC	AUPR
MC Dropout [10]	62.12	63.35
Deep Ensembles [21]	73.18	70.45
Kendall and Gal [19]	67.00	70.00
BNN-ARHT (Ours)	73.52	72.99
Table 11: The OOD detection performance (in %) of our method, BNN-ARHT, compared with various competitors, using the LeNet architecture. Standard deviations are given in brackets.
Model	In-Distrib.	Fashion-MNIST	In-Distrib.	SVHN
AUC	AUPR	AUC	AUPR
MC Dropout	MNIST	99.33 (0.3)	99.27 (0.3)	CIFAR10	78.09 (1.2)	84.35 (1.1)
Deep Ensembles	MNIST	90.70 (8.4)	91.08 (7.7)	CIFAR10	76.15 (5.3)	82.62 (13.1)
Kendall and Gal	MNIST	92.54 (2.6)	92.77 (1.9)	CIFAR10	67.40 (3.1)	71.44 (10.1)
EDL	MNIST	73.43 (16.0)	80.22 (11.1)	CIFAR10	69.57 (4.7)	83.74 (3.4)
DPN	MNIST	99.41 (0.2)	99.37 (0.3)	CIFAR10	57.48 (4.4)	77.76 (6.2)
PostNet	MNIST	98.59 (0.4)	94.70 (0.5)	CIFAR10	76.04 (1.6)	69.30 (1.7)
Detector	MNIST	75.57 (15.2)	83.75 (29.9)	CIFAR10	76.01 (13.6)	90.00 (22.9)
RKL-PN	MNIST	—	78.45 (3.1)	CIFAR10	57.89 (1.8)	61.41 (2.8)
I-EDL	MNIST	98.49 (0.3)	98.89 (0.3)	CIFAR10	—	83.26 (2.4)
BNN-ARHT (Ours)	MNIST	99.51 (0.4)	99.47 (0.3)	CIFAR10	82.01 (1.2)	91.61 (0.3)
Gaussian distribution is defined as
$$
p (\boldsymbol {x}; \boldsymbol {\mu}, \boldsymbol {\Sigma}) = \frac {1}{(2 \pi) ^ {\frac {n}{2}} | \boldsymbol {\Sigma} | ^ {\frac {1}{2}}} \exp \left\{- \frac {1}{2} (\boldsymbol {x} - \boldsymbol {\mu}) ^ {\top} \boldsymbol {\Sigma} ^ {- 1} (\boldsymbol {x} - \boldsymbol {\mu}) \right\},
$$
where  $\pmb{\mu} \in \mathbb{R}^p$  is a  $p$ -dimensional mean vector and  $\pmb{\Sigma} \in \mathbb{R}^{p \times p}$  is the covariance matrix. The KL divergence between two multivariate normal distributions  $\mathcal{N}(\pmb{\mu}_1, \pmb{\Sigma}_1)$  and  $\mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)$  is given by
$$
\operatorname {K L} \left(\mathcal {N} \left(\boldsymbol {\mu} _ {1}, \boldsymbol {\Sigma} _ {1}\right) \| \mathcal {N} \left(\boldsymbol {\mu} _ {2}, \boldsymbol {\Sigma} _ {2}\right)\right) = \frac {1}{2} \left[ \log \frac {\left| \boldsymbol {\Sigma} _ {2} \right|}{\left| \boldsymbol {\Sigma} _ {1} \right|} - p + \operatorname {t r} \left\{\boldsymbol {\Sigma} _ {2} ^ {- 1} \boldsymbol {\Sigma} _ {1} \right\} + \left(\boldsymbol {\mu} _ {2} - \boldsymbol {\mu} _ {1}\right) ^ {\top} \boldsymbol {\Sigma} _ {2} ^ {- 1} \left(\boldsymbol {\mu} _ {2} - \boldsymbol {\mu} _ {1}\right) \right].
$$
E Baseline Methods and Implementation Details
Implementation Details. We present additional implementation details and hyperparameter settings. We first provide the key settings and adaptations applied to the baseline methods for reproducibility. We follow the default settings for other fine-grained parameters (e.g., learning rates).
The proposed method is implemented in Python with Pytorch library on a server equipped with four NVIDIA Tesla V100 GPUs. The dropout ratio of each dropout layer is selected as 0.2. All models are trained with 100 epochs with possible early stopping. We use the Adam optimizer to optimize the model with a learning rate of  $5 \times 10^{-5}$  and a weight decay of  $1 \times 10^{-5}$ . Data augmentations such as color jittering and random cropping and flipping are applied as a regularization measure.
Hyperparameter Settings. The hyperparameter settings for BNN training and ARHT testing are given as follows:
- Prior mean of weights — sampled from  $\mathcal{N}(-3,0.01)$
$s = 5$
$n_2 = 300$
-  $\lambda_0 = 0.01$
Additional Settings of Baseline Methods. We further introduce the experimental settings of baseline methods:
- Deep ensembles [21]: Set the number of ensembles as 5.
- MCDropout [10]: Set the dropout ratio as 0.2 for both training and inference.
- Kendall and Gal: Set the number of inference weight samples as 20.
- Detector [12]: Set the number of runs as 100.
- PostNet [4]: Because the original codes operate on the features extracted from the standard datasets, this method cannot be generalized to new datasets (e.g., SVHN) due to unavailability of the data processing codes.
F Uncertainty Measures
We describe the uncertainty measures used in the OOD misclassification task in this section. These definitions are well-known and summarized in Malinin and Gales [27],
- Entropy:
$$
H [ p (\boldsymbol {\mu} | \mathcal {D}) ] = - \sum_ {c = 1} ^ {K} p (w _ {c} | \mathcal {D}) \ln p (w _ {c} | \mathcal {D}),
$$
where  $P(w_{c}|\mathcal{D})$  is the predictive probability of class  $c$ , and  $K$  is the number of classes for classification.
- Maximum probability: we take the maximum predicted probability  $\mathcal{P}$  from all classes as the confidence score,
$$
\mathcal {P} = \max  _ {c} P (w _ {c} | \mathcal {D}).
$$
- Differential entropy:
$$
I [ y, \boldsymbol {\mu} | \mathcal {D} ] = - \int_ {S ^ {K - 1}} p (\boldsymbol {\mu} | \mathcal {D}) \ln p (\boldsymbol {\mu} | \mathcal {D}) d \boldsymbol {\mu}
$$
where  $S^{K - 1}$  is the supporting set, and  $\mu$  is the predictive class probability assumed to follow a Dirichlet distribution.
- Accuracy: the fraction of correct predictions to the total number of ground truth labels.
- F-1 score: The F-1 score for each class is defined as
$$
F - 1 \text {s c o r e} = 2 \cdot \frac {\text {p r e c i s i o n} \cdot \text {r e c a l l}}{\text {p r e c i s i o n} + \text {r e c a l l}}
$$
where 'recall' is the fraction of correct predictions to the total number of ground truths in each class and 'precision' is the fraction of correct predictions to the total number of predictions in each class.
AUROC: the area under the receiver operating curve (ROC) which is the plot of the true positive rate (TPR/Recall) against the false positive rate (FPR).
- AUPR: the area under the precision-recall curve. Note that the AUPR for binary classification is sensitive to the distribution of positive and negative classes. Hence, the higher AUPR does not necessarily imply a better model performance.
G Algorithm
Algorithm 1 gives the detailed workflow of our proposed uncertainty estimation framework.
Algorithm 1 Our proposed uncertainty estimation framework
Input: The prior distribution of weights of BNN encoder  $\pi (\theta)\sim \mathcal{N}(0,I)$  Training data  $\mathcal{D}_{tr} = \{\pmb {x}_i,\pmb {y}_i\}_{i = 1}^{N_{tr}}$  Testing data  $\mathcal{D}_{te} = \{\pmb {x}_i,\pmb {y}_i\}_{i = 1}^{N_{te}}$  Hyperparameters  $\mu_0,\rho_0,n_2$  Initial variational posterior distribution  $q(\theta)\sim \mathcal{N}(\pmb {\mu},\log (1 + \exp (\pmb {\rho})))$  with initial parameters  $\pmb {\mu} = \mu_0\mathbf{1}$  and  $\pmb {\rho} = \rho_0\mathbf{1}$  Output: The uncertainty scores   
1: for  $(x_{i},y_{i})$  in  $\mathcal{D}_{tr}$  do Train BNN encoder   
2: Draw weight sample  $\pmb{\theta}$  from  $q(\pmb {\theta})$    
3:  $\hat{y}_i = f_\theta (x_i)$  Forward propagation   
4: Compute task-specific loss  $\mathcal{L}_{\mathrm{obj}}$    
5: Compute KL(q|π) and hence the ELBO   
6: Backpropagate the ELBO to update  $\pmb{\mu}$  and  $\pmb{\rho}$    
7: end for   
8: Compute  $\mu_{1}\in \mathbb{R}^{p},\Sigma_{1}\in \mathbb{R}^{p\times p}$  Obtain summary statistics of training   
9: for  $(x_{i},y_{i})$  in  $\mathcal{D}_{tr}$  do OOD Detection   
10: Compute  $\mu_{2}\in \mathbb{R}^{p},\Sigma_{2}\in \mathbb{R}^{p\times p}$    
11: Compute the pooled sample covariance matrix by Eq. (1)   
12: Compute ARHT by Eq. (4) as the uncertainty score   
13: Detect OOD samples using the uncertainty score under family-wise adjusted threshold   
14: end for