SlowGuess/NeurIPS_2023_OCR huggingface dataset:
over the class logits, which is cast into class probabilities via a mean-field approximation. These class probabilities are then trained under a CE loss.  $\beta$  is a learnable parameter matrix,  $\phi (x) =$ $\cos (Wh(x) + b)$  is a feature embedding based on frozen random parameters  $W$  and  $b$ , and  $\Phi^{\top}\Phi$  is the empirical covariance matrix of the feature embeddings over the training dataset. The method also applies spectral normalization to the hidden weights in each layer in order to satisfy input distance awareness. We treat whether to apply spectral normalization through the network and whether to use layer normalization in the last layer as hyperparameters.
B.9 Direct Loss Prediction (Losspred)
Losspred trains a classifier under a cross-entropy loss and uses its uncertainty module  $\kappa$ , a 3-layer MLP attached to the embedding space, to predict the value of the cross-entropy loss. Both components are balanced by the hyperparameter  $\lambda \in [0.01, 0.99]$ :
$$
\mathcal {L} = \mathcal {L} _ {\text {c r o s s - e n t r o p y}} (x, y) + \lambda (\kappa (x) - \mathcal {L} _ {\text {c r o s s - e n t r o p y}} ^ {\text {d e t a c h e d}} (x, y)) ^ {2}. \tag {8}
$$
The gradients of  $\mathcal{L}_{\mathrm{cross - entropy}}^{\mathrm{detached}}$  inside the  $L_{2}$  loss are detached to prevent fitting it to  $\kappa (x)$ , instead of the other way around. Besides  $\lambda$ , a second hyperparameter is whether or not to warm up  $\kappa$  to 0.001.
B.10 Deep Ensemble
Ensemble (Lakshminarayanan et al., 2017) has 10 classifier heads attached to the embedding space. Their logits are transformed into probabilities by a softmax and then averaged. This average is trained under a CE loss. Ensemble has no hyperparameters other than the learning rate.
B.11 MCDropout
MCDropout (Srivastava et al., 2014) trains with a dropout rate of  $[0.05, 0.25]$ , which is a hyperparameter. During inference time, it keeps the dropout activated to sample 10 logits and averages them like Ensemble.
C.1 Pretrained models already close the gap in terms of  $\mathbf{R}@\mathbf{1}$
This section compares the models in terms of their R@1. To this end, we hyperparameter-tuned them with respect to R@1, and not R-AUROC. Similar to the main text, we also add an additional baseline that was trained on the downstream classes, where the original test split was split into equal sized train and test splits.
Fig. 6 shows these performances. As opposed to the R-AUROC, we can see that the gap between pretrained zero-shot and many-shot models is much tighter in terms of the  $\mathbf{R}@\mathbf{1}$ . The best pretrained ResNet-50 has an average  $\mathbf{R}@\mathbf{1}$  of 0.48 vs. 0.54 when training on the downstream data.
Surprisingly, when it comes to R@1, CE is among the best two approaches both for ResNet and ViT backbones. In fact, three of the four best approaches on ViTs and two of the four on ResNets rely on CE as part of their loss. The approaches that do not rely on CE are probabilistic embeddings – nivMF on ResNet and ViT and ELK on ResNet.
C.2 R@1 and R-AUROC correlate negatively in most models
In extension to the plot that compared the best hyperparameter setup for R-AUROC to the best for R@1, Fig. 7 and Fig. 8 present the trade-offs for all tested hyperparameters.
The general picture is the same as in the comparison of best vs best: Most models show a tradeoff between achieving the best R-AUROC and the best  $\mathbf{R}@\mathbf{1}$ . This is indicated by a negative rank correlation in 15 out of 22 models. Still, there are some models where the uncertainty estimation and prediction performances correlate moderately positively (0.4 ≤ Rank  $\mathrm{Corr.}\leq 0.72$ ). These are InfoNCE and MCInfoNCE on ResNets, and HET-XL, nivMF, and HIB on ViTs.
Figure 6: Pretrained models achieve an almost as good R@1 on unseen downstream data as models trained on the downstream data. Minimum, average, and maximum R@1 per model. Model hyperparameters were optimized w.r.t. R@1.
As mentioned in Appendix B, each approach was tested on 10 hyperparameters, with 2 additional runs on other seeds for the best hyperparameters. The reason that some plots show less than 12 points is that those had a  $\mathrm{R}@\mathrm{1} < 0.1$  on the downstream datasets and were excluded from the analysis. Some plots also show more than 12 points. This is because their best hyperparameter for  $\mathrm{R}@\mathrm{1}$  was unlike that for R-AUROC, adding another 2 runs on different seeds for the  $\mathrm{R}@\mathrm{1}$ . Further, some approaches had optimal hyperparameters close to the original search bounds, such that the search was extended, leading to additional points in the plots.
Figure 7: R@1 and R-AUROC are negatively correlated on seven out of ten model classes with ResNet backbones. Plot shows all tested hyperparameter combinations.
Downstream R@1
Downstream R@1
Figure 8: R@1 and R-AUROC are negatively correlated on seven out of ten model classes with ViT backbones. Plot shows all tested hyperparameter combinations.
C.3 Further correlation of up- and downstream metrics
This section reports further metrics for each model. These include a new metric (top-1 accuracy on upstream ImageNet) and all downstream metrics (R@1, R-AUROC, percentage where cropped image has higher uncertainty) on the upstream dataset.
Fig. 9 shows the pairwise correlations between all metrics, for every tested hyperparameter setting. Let us first consider the new metric, ImageNet top-1 accuracy, on its own. The best-performing models are a CE and a nivMF ViT, with an accuracy of 0.84 each. Other than them, Losspred and HET-XL on ViTs also have a high accuracy, similar to the results on the R@1 benchmark.
Regarding correlations between metrics, accuracy and R@1 are highly correlated, as expected. Further, models with a high accuracy or R@1 on ImageNet also have a high R@1 on the downstream data, resulting in the small gap explained in Appendix C.1. While up- and downstream R-AUROC do not correlate, up- and downstream percentage of cropped images having a higher uncertainty correlate nearly linearly. This reinforces that the percentage can be considered as a general notion of uncertainty. It should, however, be noted that ELK ViTs already achieve a performance of 0.99 on this metric, even on downstream data, such that it is not able to guide the field as well as R-AUROC.
Figure 9: Correlations between several up- and downstream metrics across all models and hyperparameter choices.
C.4 Class-entropy is useful for OOD detection
Figure 10: Cross-entropy and embedding norm-based uncertainty estimators have the best OOD detection capabilities. Averaged AUROC of distinguishing ImageNet vs CUB, ImageNet vs CARS, and ImageNet vs SOP. Error bars indicate minimum/maximum performance across seeds.
Figure 11: Embeddings of OOD images tend to have smaller  $L_{2}$ -norms than embeddings of ID images. Embedding norms are from InfoNCE models that use the (inverse) norm as uncertainty estimate (without using them during training) and reach an OOD AUROC of 0.73.
In this section, we study if uncertainties on downstream data are generally higher than for the upstream data. To this end, we perform out-of-distribution detection experiments: Using the same pretrained models as before, we calculate their uncertainties on downstream dataset samples and on an equally sized set of upstream samples. We quantify how well the predicted uncertainties distinguish whether the sample was from a downstream dataset (1) or not (0) by calculating the AUROC on ImageNet vs CUB, ImageNet vs CARS, and ImageNet vs SOP, and averaging them across the datasets.
Fig. 10 shows the average result across all seeds, along with minimum and maximum performances. Generally, pretrained models perform differently than in the URL benchmark in the main text. This is because the OOD task benchmarks epistemic uncertainty (i.e., OOD data just has to have generally high uncertainties). On the other hand, the URL benchmark tests predictive uncertainties on OOD data. There, it is not enough to predict high values on OOD data but the models need to differentiate within them, which lays more focus on aleatoric uncertainty. In more detail, models that directly predict variances or losses do not provide as good OOD performance. Models that use class-entropy
as uncertainty estimates (SNGP, HET-XL, CE, Ensemble, and, only on ViTs, MCDropout) and also InfoNCE, which uses the norm of the embedding vector, do work well.
There is an intuitive explanation for this. The latter models implicitly provide epistemic uncertainty estimates by construction: ResNets and ViTs embed inputs with less known features closer to the origin. As an example, Fig. 11 shows that for InfoNCE, the embedding norms of OOD samples are smaller. Note that this is no trained behaviour; InfoNCE only trains on normalized embeddings, so the norms occur naturally. In InfoNCE, we use this embedding norm directly as uncertainty estimator. But the same happens in models that use the class entropy: Embeddings with small norm lay close to the origin, where they lead to uniform distributions over the classes, i.e., a high entropy.
In summary, the OOD experiment reveals that probabilistic embeddings and loss prediction methods provide aleatoric uncertainty estimates, whereas models that explicitly or implicitly use the embedding norm provide good epistemic uncertainty estimates. Some models, like SNGP, provide a mixture and are good in both tasks.
C.5 Uncertainties on mixtures of in- and out-of-distribution data
Figure 12: Cross-entropy based uncertainties provide the best mix of OOD detection capability and ordering within ID and OOD uncertainties. R-AUROC on mixed ImageNet+CUB, ImageNet+CARS, and ImageNet+SOP data. Error bars indicate minimum/maximum performance across seeds.
In some applications, models might encounter a mixture of in-distribution and out-of-distribution data. In this case, the model both needs to assign higher uncertainties to the OOD data and it needs to differentiate the uncertainties within both the ID and the OOD sets. This blends the OOD detection of the previous experiment with traditional ID calibration and the OOD calibration of the URL benchmark.
Fig. 12 measures the R-AUROC on a mixture of upstream and downstream data. As in the previous experiment, we use 50/50 splits of ImageNet+CUB, ImageNet+CARS, and ImageNet+SOP, and average across those combinations. We find that HET-XL and SNGP, which previously performed well on both OOD detection and the URL benchmark, also perform well on this mixed task. The remaining models tend to follow the ranking of the OOD benchmark rather than that of URL. This indicates that good epistemic uncertainty estimation outweighs aleatoric uncertainty estimation in this task. This is intuitive, because the R-AUROC measures how likely it is that a wrong prediction has a higher uncertainty than a correct one. In such a 50/50 split of ID and OOD data, the capability to distinguish ID from OOD data, and thus data with generally less errors from data with generally more errors, thus leads to a higher R-AUROC.
This serves to demonstrate that the quality of an uncertainty estimate always depends on the task and setup at hand. While OOD detection and ID calibration are ideally both reflected within the very same predictive uncertainty value, both are of different importance depending on the data mixture.
C.6 Few-shot uncertainties starting from pretrained models
Figure 13: Cross-entropy training on the downstream test data reaches the zero-shot AUROC of 0.6 at 2-5 samples. R-AUROC of pretrained models when finetuned on the downstream test classes.
In this section, we provide an initial attempt on a few-shot experiment. We use both the best three pretrained models (ELK, MCInfoNCE, and nivMF for ResNet 50; ELK, Losspred, and SNGP for ViT Medium) along with the CE model as pretrained checkpoints and continue to train under the same hyperparameters and losses on  $k \in \{1,2,5,10\}$  samples of each class of the downstream datasets. This is done on the test classes of the downstream classes, which were separated into disjoint train and test samples for this experiment. We train on each CUB/CARS/SOP separately and average their performance.
Fig. 13 shows the minimum, maximum and average performance across the seeds. First, we can see that the normal cross entropy training reaches the zero-shot R-AUROC of the currently best pretrained models at around 2 samples per class (totaling in 200 samples on CUB, 196 on CARS, and 22636 on SOP). This again demonstrates the point that the challenge URL addresses is unsolved yet. It also shows that knowledge of the specific downstream task can increase the uncertainty estimators quality even at only a few samples. Second, we find that most pretrained models increase their performance as well. This, however, happens not for all models and is highly noisy. We attribute this to the simple setup of our experiment, which uses the same hyperparameters as for the pretraining and performs standard training as opposed to specialized few-shot methods. Finding best practices to tune pretrained uncertainties to downstream few-shot tasks is a promising undertaking for future research.
C.7 Uncertainty estimation does not add significant computational costs
Table 1 reports the computational complexity during all benchmarks. It shows the number of parameters as proxy for RAM usage, the duration of the first training epoch and evaluation, and the time needed for each sample during evaluation. These results were collected on the go and there are possible confounders such as network storage workload. We thus recommend to interpret them as rough indicators. Note also that ViTs were run on NVIDIA A100 GPUs, and ResNets on NVIDIA RTX2080TIs (except HET-XLon ResNet, due to RAM usage).
First, we see that explicit uncertainty estimation does not come at a high RAM cost. The 3-layer MLP serving as uncertainty head for MInfoNCE, ELK, nivMF, HIB, and Losspred adds 4.2M parameters to a ResNet or 2.6M to a ViT. The difference is due to the ViT's lower-dimensional embedding space. Bigger increases occur only for ensembles, which uses additional classifiers with  $10 \cdot 2.1\mathrm{M}$  parameters on ResNets and each  $10 \cdot 0.5\mathrm{M}$  on ViTs.
Model	Parameters (Millions)	Epoch Time (s)	Inference time per sample (ms)
ResNet 50 on RTX 2080TI	CE	25.6	5971	3.7
InfoNCE	23.5	6703	3.7
MInfoNCE	27.7	6656	3.8
ELK	29.8	6703	3.8
nivMF	29.8	6256	3.8
HIB	29.8	7279	3.8
HET-XL (on A100)	33.9	(4189)	(2.3)
Losspred	29.8	6526	3.7
MCDropout	25.6	7105	13.1
Ensemble	46.0	6002	3.7
SNGP	28.7	7632	3.7
ViT Medium on A100	CE	38.9	4922	2.8
InfoNCE	38.3	7804	2.7
MInfoNCE	41.0	9883	2.4
ELK	41.5	4838	2.7
nivMF	41.5	4121	2.4
HIB	41.5	3950	2.5
HET-XL	39.4	4205	2.4
Losspred	41.5	3814	2.6
MCDropout	38.9	4721	9.6
Ensemble	44.0	4107	2.5
SNGP	42.0	4811	2.9
Table 1: Computational costs of all approaches. Epoch times include training on ImageNet as well as evaluating on all eight downstream datasets.
Training times should be interpreted with caution due to the aforementioned network storage. However, in general uncertainty estimates do not seem to exceedingly increase the train time, with between  $-22\%$  and  $+28\%$  over the CE baseline. Taking multiple samples during training (MCIinfoNCE, nivMF, HIB, HET-XL) also does not systematically increase runtime. This is likely due to their efficient sampling implementations (Kirchhof et al., 2022; Davidson et al., 2018; Ulrich, 1984). The only consistent runtime cost occurs when training unsupervised models (InfoNCE, MCIinfoNCE), which need to forward propagate two augmentations of each sample to obtain have positive pairs.
Similarly, providing an uncertainty estimate at inference takes only up to 0.1 additional milliseconds on ResNets, because the uncertainties are all calculated within a single forward pass, and sampling was only required for the losses during training. The only exception here is MCDropout, which requires making 10 full forward passes during inference, which increases the time by a factor of roughly 3.6. The factor is not 10, because loading the data and storing the results is part of the measured elapsed time.
In summary, we find that uncertainty estimates only have small computational costs if they are implemented in a forward fashion.