URL: A Representation Learning Benchmark for Transferable Uncertainty Estimates
Authors: Michael Kirchhof
Bálint Mucsányi
Seong Joon Oh
Enkelejda Kasneci
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Representation Learning
Uncertainty
Zero-shot
Transfer
Generalization
Downstream
Benchmark
TL;DR: URL evaluates uncertainty estimates of large pretrained models on unseen downstream data, thereby extending representation learning benchmarks.

Abstract: Representation learning has significantly driven the field to develop pretrained models that can act as a valuable starting point when transferring to new datasets. With the rising demand for reliable machine learning and uncertainty quantification, there is a need for pretrained models that not only provide embeddings but also transferable uncertainty estimates. To guide the development of such models, we propose the Uncertainty-aware Representation Learning (URL) benchmark. Besides the transferability of the representations, it also measures the zero-shot transferability of the uncertainty estimate using a novel metric. We apply URL to evaluate ten uncertainty quantifiers that are pretrained on ImageNet and transferred to eight downstream datasets. We find that approaches that focus on the uncertainty of the representation itself or estimate the prediction risk directly outperform those that are based on the probabilities of upstream classes. Yet, achieving transferable uncertainty quantification remains an open challenge. Our findings indicate that it is not necessarily in conflict with traditional representation learning goals. Code is available at [https://github.com/mkirchhof/url](https://github.com/mkirchhof/url).

URL: A Representation Learning Benchmark for Transferable Uncertainty Estimates
Michael Kirchhof
Bálint Muesányi University of Tübingen
University of Tübingen
michael. kirchhof@unn Senhoffuuii-- www.ootuiititnnnnnnttmmm tuw.ingen..nndee..
Seong Joon Oh University of Tübingen, Tübingen AI Center
Enkelejda Kasneci
TUM University
Abstract
Representation learning has significantly driven the field to develop pretrained models that can act as a valuable starting point when transferring to new datasets. With the rising demand for reliable machine learning and uncertainty quantification, there is need for pretrained models that not only provide embeddings but also transferable uncertainty estimates. To guide the development of such models, we propose the Uncertainty-aware Representation Learning (URL) benchmark. Besides the transferability of the representations. it also meaExamplessures the zero-shot transferability of the uncertainty estimate using novel metric. We apply URL to evaluate eleven uncertainty quantifiers that are pretrained on ImageNet and transferred eight downstream datasets. We find that approaches that focus the uncertainty of the representation itself or estimate the prediction loss directly outperform those that are based the probabilities of upstream classes. Yet, achieving transferable uncertainty quantification remains an open challenge. Our findings indicate that is not necessarily in conflict with traditional representation learning goals. Code is available at https:/ //t/tttbt../t//////// com/ mkirchhoff /url.
Introduction
Pretrained models are a vital component of many machine learning applications. The driving force behind their development has been representation learning benchmarks, e.g. Roth et al. (2020); Chen et al. (2020): They task models to output representations e(x) of input data x that generalize across datasets in zero-shot manner. These pretrained representations provide valuable starting point for downstream applications, requiring less supervised data to be fine-tuned for specific tasks.
At the same time, uncertainty quantification remains major challenge in the recent efforts towards reliable machine learning (Collier al., 2023; Tran et al., 2022). Uncertainty quantification refers to estimating the degree of uncertainty risk u(z) E R in model's prediction. This is particularly important high-stakes applications such as medical image classification. Here, the model can refrain from making predictions if the uncertainty, e.g., u(xx)= ;= 1 - max, P(Y y/2),, is too high (Zou et al., 2023; Bouvier et al., 2022). Beyond classification, uncertainty is an inherent property of vision and language (e.g., low image resolution ambiguous text inputs) that cannot be learned away even with large amounts of data (Chun et al., 2022; Kendall and Gal, 2017). Consequently. recent literature suggests representing images not points e(x), but as probabilistic embeddings (Kirchhof et al., 2023; Collier et al., 2023; Chun et al., 2021). Here, u(x) ) is the variance parameter of distribution around e(x:) in the embedding space, representing the input's inherent ambiguity. This can then be utilized for uncertainty-awart retrieval.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

A major hurdle on the way to reliable uncertainty estimates is that u(z) needs to be trained from the ground up for each specific task, requiring substantial labeled data. Replicating the successes of representation learning promises to reduce this burden by pretraining a u(r) which can be transferred to downstream tasks datasets has not been tested in literature, with previous benchmarks evaluating on the same datasets they trained on (Detommaso et al., 2023; Nado et al., 2021). Thus, we propose a novel Uncertainty- aware Representation Learning (URL) benchmark. Models that output both embeddings e(x) and uncertainty estimates u(æ) of any form are pretrained on large collections of upstream data and evaluated on unseen downstream datasets. The transferability of their embeddings e(x: is evaluated in terms of the Recall @ (R@ 1), as in established representation learning benchmarks (Roth et al., 2020; Chen et al., 2020). The transferability of their uncertainty estimates u(x) is evaluated with a novel metric, the Recall @1 1 AUROC (R-AUROC). It naturally extends R@ 1-based benchmarks and can be seamlessly integrated in as little as four lines of code, without requiring any new ground- truth labels. Nonetheless, is not only an abstract metric but has practical significance: Models with higher R-AUROC are also more aligned with human uncertainties and react better to uncertainty- inducing interventions like image cropping.
zero- or finetuned few-shot manner. Yet, this transferability of u(x) to new
On this benchmark, we reimplement and train eleven state-of-the-art uncertainty estimators, from plass-entropy baselines over probabilistic embeddings to ensembles, with ResNet (He et al., 2016) and ViT (Dosovitskiy et al., 2021) backbones on ImageNet-IK 1k (Deng al., 2009). Our main findings are:
1. Transferable uncertainty estimation is an unsolved challenge (Section 4.2),
2. MCInfoNCE and direct loss prediction generalize best (Section 4.3),
3. Uncertainty estimation is not always in conflict with embedding estimation (Section 4.4),
4. Models with good uncertainties upstream not necessarily good downstream (Section 4.5),
5. URL captures how aligned a model is with human uncertainty (Section 4.6).
These findings demonstrate that pretraining models for downstream uncertainty estimation is an important yet unsolved challenge. We hope that our benchmark will serve as a valuable resource in guiding the field towards pretrained models with reliable and transferable uncertainty estimates.
2
Related work
Our benchmark connects recent uncertainty quantification benchmarks with representation and zero-shot learning for unseen data, which we introduce below. Specific datasets and methods for uncertainty quantification are described in the experiments section when they are benchmarked.
Uncertainty benchmarks. Uncertainty quantification has become an essential consideration for reliable machine learning, and so several libraries have been recently developed to guide its advance- ment (Detommaso et al., 2023; Nado et al., 2021). These libraries provide various metrics for eval- uating and improving uncertainty estimates
in-distribution data. Galil et al. (2023b) and Galil et al. (2023a) benchmarked over 500 large vision models trained on ImageNet from the timm (Wight- man, 2019) library and reported that Vision Transformers (ViT) provide the best uncertainty esti- mates. Further, scaling of these ViT's to up to 22B parameters and pretraining on a large corpus of upstream data results in very accurate uncertainty estimates (Dehghani et al., 2023; Tran et al., 2022). However, when moving away from in-distribution data, the quality of uncertainty estimates deteri- orates (Tran al., 2022) and we can only expect that they will be generally higher and allow for out-of-distte iitoo-ddistribution detection (Ovadia et al., 2019). This motivates our benchmark: We aim to de- velop pretrained models that can generalize their uncertainty estimates and discriminate certain from uncertain examples even within unseen datasets. While some works applied their uncertainty esti- mates to unseen datasets (Cui et al., 2023; Collier et al., 2023; Ardeshir and Azizan, 2022; Karpukhin et al., 2022), their downstream evaluations focused on embeddings, leaving the uncertainty estimates untested. Our benchmark intends to bridge this gap and assess the transferubility of uncertainty es- timates, with the goal of enhancing large pretrained models towards zero-shot uncertainty estima- tion. To design benchmark for transferability, we connect the upper benchmarking techniques to paradigms from representation and zero-shot learning below.

---

daf url berchnanch (pratrained_ modelll dounstrean_ londat) Predict embeddinga uncertainties downstream data labels embeddings
list()
for (inage, label) in dosnatrean_
enbededding. uncertainty peetrained_nodee (ingo)
(labels, embeddings, uncertainties) append (label, enbedding, uncertainty)
# Calculate Re1 and R-AUROC next mext__niihbbr_tte is_same_clare labels =* labels [next_ next__niighor__dt1 r_at_1 mean mean(1s_same_clase
(embeddige))
10
12
F_auroc conрata... auroc (uncertainties, nat sans_ [ana_clasa
13
14
retura r_at_1, I auroc
Algorithm Adding URL to existing representation learning benchmarks takes only the four highlighted lines of code.
Representation and zero-shot learning. Transferability is generally evaluated by testing whether models can make sensible decisions on unseen data. In zero-shot learning (Xian et al., 2017), the model is tasked to give class-predins class-predictions new downstream classes. This requires both learning transferable representation space on upstream data and creating classifier heads for the new classes from auxiliary information. Representation learning benchmarks (Roth et al., 2020; Khosla et al., 2020; Bengio et al., 2013) focus on the former. To this end, they use a metric similar to class accuracy, the Recall @ (R @ 1) (Mikolov et al., 2013). It calculates the model's embeddings of all unseen downstream data and compares whether each embedding's s next neighbor is in the same class or not. This tells whether the embeddings
semantically meaningful, such that the pretrained model can be successfully transferred to downstream tasks. We extend representation learning benchmarks to additionally judge the transferability of uncertainty estimates. To this end, we propose a metric that can be implemented on top of the R@ in four lines of code in the next section.
3 Uncertainty-awee -wwaee representation learning (URL) benchmark
3.1 Evaluating uncertainty about representations
To quantify its uncertainty, a model f x' + x U4 assumed to predict both an embedding e(x) E and a scalar uncertainty value u(x) u C IR for each input image X. We do not impose restrictions on how u(x) is calculated, e.g., could the negative maximum probability of softmax classifier, predicted variance from dedicated uncertainty module, the disagreement between ensemble members. The predicted uncertainty u(x) is commonly benchmarked in terms of its expected calibration eπror (ECE), negative lieelied, area under the receiver-operator characteristics curve (AUROC), or abstained prediction curves. All of these measures are w.r.t. the correctness of classification decision. Hence, u(z) can only be evaluated in-distribution with known classes. Our setup involves unseen datasets and classes, so we need to develop a fitting measure.
To this end, let take Lahlou et al. (2023)'s decision-theoretic perspective on uncertainty quantifica- tion: Uncertainty quantification loss prediction. The uncertainty expresses the expected loss of a model's is decision on specific datapoint. In Gaussian regression with I.2 loss, the expected loss is the target's 's variance, so an uncertainty quantifier u(z) should be proportional to it. classification with 0-1 loss, u(x) should be proportional to the probability of returning the correct class.
In representation learning, the model's decision is the embedding e(x) and the loss is the R@ The uncertainty quantifier's 's goal is then to report the loss attached to the embedding, i.e., u(x) should be proportional whether the R R@] will be correct or not. This demonstrates the use case of u(x): Telling whether an embedding e(x) can be trusted or could be misplaced in the embedding space. This is an important property as models the form + e + y have an information bottleneck in the quality of the embedding e(x) due to the data-processing inequality (Cover, 1999). For every downstream task, higher uncertainty u(ar) about e(x) monotonically increases the loss of y(e(x)). In other words, if the embedding is wrong, then the prediction in any downstream task will be wrong.

---

We measure whether the uncertainty quantifier u(z) is proportional to the correctness of the embed- ding e(x) via the AUROC with respect to whether the R@ correct (one) or not (zero), named R-AUROC. As the R@1 1 is 0-1 loss, the R-AUROC can be interpreted as the probability that an in- correct embedding will receive higher uncertainty score than a correct embedding (Fawcett, 2006). An R-AUROC close to 1 means that u(r) clearly separates correct from incorrect embeddings, while an R-AUROC of 0.5 means that it has no more predictive power than a random guess.
A positive trait of the R-AUROC is that it is indicative of how well-aligned the model is with human uncertainties and how well the model reacts uncertainty-indeee
interventions (see Fig. 5 and Section 4.6). It also does not require uncertainty ground-truth and takes only four lines of code to implement into existing representation learning benchmarks, as shown in Algorithm 1. We choose the AUROC over other calibration measures such as the ECE because it accepts uncertainties u(z) E R (instead of u((x) E [0. 1)) and because it avoids some loopholes of the ECE. We discuss these and more design choices behind this metric in Appendix A.
3.2 URL benchmark protocol
The R-AUROC can be evaluated on any downstream dataset that allows calculating the R@ 1, i.e., has class labels. Yet, in order to keep future results comparable, we propose a benchmark protocol for uncertainty-an aware representation learning (URL). Our code is based on timm (Wightman, 2019) and available at https://git github. wtcluu..co///ii com/mkirchhof.
Datasets. We train each model on ImageNet-1k 1k (Deng et al., 2009) as upstream dataset. We note that future works may use larger-scale upstream datasets (Collier et al., 2023; Tran et al., 2022) or auxiliary information (Han et al., 2023; Ortiz-Jimenez al., 2023), as long as they stay disjoint the downstream datasets. As downstream datasets, we follow the standardized representation learning protocol of Roth et al. (2020) and use FUB-200-2011 (Wah et al., 2011), CARS196 (Krause et al., 2013), and Stanford Online Products (Song et al., 2016). We follow the original splits that divide their classes into equally sized train and test sets. Following Roth et al. (2020), we further divide the classes in the train set equally into a train and validation split. In our zero-shot transfering setup, all models are trained only on the upstream ImageNet dataset and do not use the downstream train splits. All results report the performance on the test sets, averaged across the three datasets and three seeds.
Hyperparameters. We use the downstream validation split to select the best learning rate, early stopping, and further hyperparameters of each model individually, see also Appendix B. Each model is tuned for 10 search iterations via Bayesian Active Learning (Biewald, 2020). The best model is chosen based on the AURUOO on validation data, where models with an R @@@ below 0.1 on the validation splits filtered out. The best model is replicated on three seeds.
Architectures. Following uncertainty quantification and representation learning benchmarks (Wen et al., 2021; Dusenberry et al., 2020; Roth et al., 2020), we use a ResNet-50 (He et al.. 2016) with an embedding space dimension of 2048 as backbone. We further study ViT-Medium (Dosovitskiy et al., 2021) backbones due to their performance (Galil et al., b023a,b) and increasing number of large- scale uncertainty quantifiers built on top of them (Collier et al., 2023; Tran al., 2022). Methods that predict uxx: with explicit modules use 3-layer MLP head attached to the embeddings.
Training infrastructure. Each model is trained with an aggregated batch size of 2048, recent studies indicate higher batch sizes might benefit uncertainty quantification (Galil et al., 2023b). We use the Lamb optimizer (You et al., 2020) with cosine annealing learning rate scheduling (Loshchilov and Hutter, 2017) for all models since it performed best in preliminary experiments. The ResNets and ViTs are trained on NVIDIA RTX 2080 Ti and A00 GPUs, respectively, for 32 epochs from checkpoint pretrained on ImageNet to reduce the computational costs. In total, the experiments took 3.2 GPU years of runtime.
Further metries. Uncertainty estimates aim to assess the errors made by individual models, so that they are necessarily model- and performance-dependee To provide comprehensive view, we not only evaluate the quality of the uncertainty estimate using R-AUROC but also consider the model's representation learning performance using R@I.

---

Experiments
4.1 Uncertainty estimators
We apply URL to benchmark two baselines (CE, InfoNCE), five probabilistic embeddings approaches (MCInfoNCE, ELK,, nivMF, HIB, HET-XL), two direct variance models (Losspred, SNGP), and two ensembles (Ensemble, MCDropout). We introduce each approach below and explain further details on their reimplementations and hyperparameters in Appendix and runtimes Appendix C.7.
Cross Entropy (CE) a supervised baseline which trains under a cross-entropy loss. It uses the entropy of the upstream class probabilities u(x) = HY [z)) as uncertainty estimate.
InfoNCE (Oord et al., 2018) is an unsupervised baseline. Following SIMCLR (Chen et al.. 2020), takes two random transforms of each image and pulls their embeddings towards each other and repels them from the remaining batch. InfoNCE itself does not estimate (a)), so we use the embedding norm u(x) == |]e( as a heuristic (Kirchhof et al., 2022; Scott et al., 2021; Li et al., 2021).
MCInfoNCE (Kirchhof al.. 2023) follows the unsupervised setup of InfoNCE, but predicts certainty (<x) =: 1/u(z) along with each embedding to define a distribution in the embedding space, so called probabilistic embeddings. It draws samples from them and applies InfoNCE on each.
Expected Likelihood Kernel (ELK) (Kirchhof et al., 2022; Shi and Jain, 2019) also predicts certainties <(()) =: 1/u(x) to define probabilistic embeddings. The probabilistic embeddings compared to class distribution via an expected likelihood distribution toniriionnttiiisiibutttionn kernel (Jebara and Kondor, 2003). This makes it supervised probabilistic embedding-base based loss.
Non-isotropic von Mises-Fisher (nivMF) (Kirchhof et al., 2022) is analoguous to ELK, but models class distributions as non-isotropic von Mises-Fisher distributions, thereby allowing different vari- ances along each embedding space axis. Image certainties are still scalars x(x) =: 1/u(z)).
Hedged Instance Embeddings (HIIB (Oh et al., 2019) predicts variances (xx) =: u(z) of proba- bilistic embeddings. Samples are drawn to compute match probabilities between two images. It aims to increase the match probabilities of same-class pairs and decrease that of different-clas, ones.
Heteroscedastic Classifier (HET-XL) (Collier al., 2023) differs from the above probabilistic embeddings approaches in that it predicts full covariance matrices E(r) in the embedding space. draws samples from these probabilistic embeddings to caleulate the expected (() We test both uxx)= == det (x)) and the class entropy u(x)= == (((((()))) as possible uncertainty estimates.
Neural Gaussian Processes (SNGP) (Liu et al., 2020) model class logits as Gaussian Processes with a predicted mean and a heteroscedastie variance. They are pooled into class probabilities [x)] and trained under CE loss. The entropy of these probabilities serves as uncertainty value u(x)= == H(P(Y(z)).
Loss Prediction (Losspred) approaches (Upadhyay et al., 2023; Lahlou et al., 2023; Levi et al., 2022; Laves et al., 2020; Yoo and Kweon, 2019) in regression treat uncertainty quantification as secondary regression task. We apply the same principle to classification, where we task the uncertainty module u(x) to predict the (gradient-detached) CE loss at each sample via an L2 loss added to the train loss.
Deep Ensembles (Lakshminarayanan et al., 2017) train multiple randomly initiated networks under a CE loss obtain several predictions. They are pooled one class distribution P(Y (z).. We define the uncertainty either via its entropy u uxx) := H (((((())) or the Jensen-Shannow divergence between the ensemble members' class probability distributions. Following (Lee et al., 2015), we only train multiple output heads and share the backbone to reduce computational complexity.
MCDropout (Gal and Ghahramani, 2016) applics Dropout (Srivastava et al., 2014) at inference time. This gives multiple predictions per input, imitating the upper Ensemble. We use both the entropy u(x) H(P(Y|(:) and the Jensen-Shannon Shannon divergence between the ensemble members' class probability distributions as possible uncertainty metrics.
4.2 Transferable uncertainty estimation is an unsolved challenge
Fig. 1 presents the URL benchmark results, i.e., the R-AUROC calculated for all above approaches on ResNet and ViT backbones. The barplot shows the minimum, average, and maximum performance across three seeds of each hyperparameter--une approach.
5

---

0.700
ResNet 50 trained downstream test classes
Downstream R-AUROC
0.650
0.625
0.600
0.375
0.530
0.525
0.500
MCIafeNCE
RivMF
ELK
Lowpred
N1B
Ensemble
InfoNCE
MCDropont
HET-XL
SNCP
CE
Lowpped
ELK
SNOW
nivMF
HIB
HET-XL
CE
Ensemble
MCDropont
InfoNCE
MCInfaNCE
ResNet 50
VI Mediam
Figure 1: Zero-shot uncertainty estimates of pretrained models (bars) do not reach the performance of many-shot models yet (dashed line). The URL benchmark aims to guide the field to close this gap. Minimum, average, and maximum R-AUROC across three seeds.
Before comparing the models, we first investigate whether the transferability problem URL addresses already solved by any ResNet 50 with standard cross-entropy loss and entropy of the class probabilities uncertainty prediction on the downstream test classes (split into a train and test split for this experiment only). This many-shot performance of 0. 0.68 is not reached by any of the methods that transfer their uncertainty in a zero-shot way, marking URL as an open challenge. In the standard R@ I, this gap has already been closed in representation learning (see Appendix C.I) and we hope that URL guides the field towards the same for transferable uncertainty estimation.
existing methods. obtain an upper reference, we additionally train
4.3 MCInfoNCE and direct loss prediction generalize best
To compare the approaches in detail, in addition Fig. Fig. 2 reports both the downstream uncertainty and R@ performance. The overall best method is Losspred, with the second-best average R-AUROC of 0.568, close the best method, MCInfoNCE, with an average R-AUROC of 0.569, while maintaining the second-best average R of 0.53, close to the best R nivMF. MCInfoNCE marks the best performance both metrics within the ResNet models, closely followed by nivMF. This is remarkable as it is the only unsupervised method aside from the InfoNCE baseline. One final noteworthy mention ELK which provides decent uncertainty estimates on both ResNets and ViTs, whereas most other models vary in their performance depending on the backbone.
@ of 0.57 achieved by
When grouping the approaches, those that directly model the variance (Losspred, SNGP) appear to have an edge on the ViTs, especially Losspred, which is the only method that disentangles variance estimation from how the class logits are calculated. Such disentanglement via having two losses could be added to other approaches in future works. Probabilistic embeddings, especially MCInfNNE,, nivMF and ELK, also show promising performance both the bigger ViTs and the smaller ResNets. Ensembles fail to provide transferable uncertainty estimates. The baselines unsurprisingly fail, indicating that well-calibrated class probabilities on the upstream dataset do not serve as good uncertainties on downstream data. We investigate this further in Section 4.5.
4.4 Uncertainty estimation is not always in conflict with embedding estimation
commonly raised concern is whether or not uncertainty quantification deteriorates the prediction, or, in the representation learning setup, the embedding quality. In the previous section, have

---

0.58
CE
InfoNCE
Downstream R- R-AUROC
0.36
MCInfoNCE
ELK
nivMF
HIB
0.34
HET-XI.
Lasspred
MCDrropout
Ensemble
SNGP
0.50
small ResNet 50
big ViT-Medium
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0.53
Downstream R@1
Figure 2: Among ViTs and ResNets, respectively, Losspred and MCInfoNCE transfer best both in terms uncertainty estimates (y-axis), measured by our R-AUROC, and embedding quality (x-axis), measured by Recall @ I. Three seeds per model and architecture
0.58
CE
InfoNCE
Downstream R- AUROC
0.36
MCInfoNCE
nivMF
HIR
0.34
HET-XL
Losapeed
MCDrupoul
Easemble
SNGP
small RosNet
big
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0.53
Downstream R@D
Figure 3: Best hyperparameters chosen for R@1 and for R-AUROC for each model. For some models, there is one best hyperparameter for both, resulting a point, but most have a large trade-off. Average performance across three seeds.
already seen that Losspred can achieve both with only a slight trade-off the best method in each category. In this section, we further detail this question within each model class.
Fig. 3 shows the performance of the best hyperparameters chosen according R-AUROC according to the @ 1. there was no trade-off, the points would lay at the same position or only have a short line connecting them. This the case for MCInfoNCE. ELK. nivMF, HET-XL, and Ensemble on ViTs and MCInfoNCE, MCDropout, and InfoNCE on ResNets. The remaining 14 of the 22 approaches show large tradeoffs, e.g., 0.21 @@ for +0.01 R-AUROC for HIB on ViTs. Whereas this comparison regards only the two extreme ends of the spectrum, Appendix C.2 measures the rank correlation across all tested hyperparameters. approaches trading off uncertainty and prediction. However, from another perspective, Losspred, nivMF, and MCInfoNCE are model classes that provide good performance both simultaneously. Hence, the question of whether there is general trade-off between uncertainty estimation and prediction is still up to debate. Studying these models and mitigating the model-internal trade-offs an interesting future work that we hope URL can enable.
shows similar conlusion, with 15 out of 22

---

0.58
Rank Correlation = 0.09
0.58
Rank Correlation = 0.54
0.56
0.36
Downstream R-AUROC
Downstream R-AUROC
0.54
0.54
0.52
0.52
0.50
0.30
0.48
0.46
0.46
0.44
0.44
0.3
0.4
0.5
0.6
0.7
0.8
0.0
0.2
0.4
0.6
0.8
1.0
Upstream R-AUROC
Upstream Pet. Cropped Image has Higher Uncertainty
Figure 4: The R-AUROC on upstream data does not indicate the performance on downstream data (left). Yet, the percentage of upstream images where a cropped version receives higher uncertainty is indicative (right). Plots show all hyperparameters, including non-optimal ones.
4.5 Models with good uncertainties upstream are not necessarily good downstream
We have seen that CE is unable to transfer its well-cali cell-calibrated upstream uncertainty estimates to downstream datasets. This brings up the question of how much the upstream and downstream uncertainty quantification abilities coincide in general. Fig. 4 (left) shows that the majority of models achieves an R-AUROC above 0.7 on the upstream seen classes. But this does not indicate a good downstream performance (Rank Corr. 0.09), neither across nor within model classes (unlike up- and downstream R @ 1, which transfers better, see Appendix C.3). This demonstrates that transferable uncertainty quantification will not solve itself by merely becoming better on the upstream data.
This also opens question about model choice: If the upstream performance cannot tell how well the model's uncertainty predictions will perform downstream, how should we select pretrained models? In this paper, we used downstream validation data. However, if we are limited to upstream data, we may test the uncertainty module in a more general task that also holds downstream. In Fig. (right), we calculate how often the model assigns a higher uncertainty value to a cropped version of an image than to the original image. The rank correlation of 0.54 with the downstream R-AUROC signals that models that perform well on this general uncertainty task also tend to generalize better to the downstream data. This means that general uncertainty tasks might be good heuristics to choose models, reinforcing practices in recent literature (Kirchhof et al., 2023).
4.6 URL captures how well-aligned model is with human uncertainty
While the R-AUROC is simple and theoretically founded, readers might still wonder why we want to drive the development of models based on this rather technical-seeming metric. In this section, we show that the R-AUROC reflects how well-aligned the model with human uncertainties.
To verify this, we use five additional downstream datasets from Schmarje et al. (2022): CIFAR-10H (Peterson al., 2019), Benthic (Langenkämper al., 2020; Schoening et al., 2020), Pig (Schmarje et al., 2022). Turkey (Volkmann et al., 2022, 2021), and Treeversity#1 (Arnold Arboretum, 2020). They present human annotators with naturally ambiguous images and record their uncertainty by collecting multiple class annotations per image. The entropy of this distribution measures the human uncertainty h(x)) H( ( (Yl=)) (x)=)). We can then measure the alignment of the model f's uncertainties with human uncertainties via rank correlation a(f) Rank ( (f(x), h2)))=). Fig. 5 (left) shows that this alignment metrie a(f) is positively correlated with the R-AUROC (Rank Corr. 0.80). Further, Fig. 5 (right) shows that the same holds for the correlation between R-AUROC and how well a model detects the uncertainty introduced synthetically via cropping (Rank Corr. 0.71), as in the previous section. This means that the R-AUROC is not just technical metric, but reveals

---

Pet. Cropped Image has Higher Uncertainty
Aligmment with Human Uncertainties
0.100
Rank Correlation 0.80
Rank Correlation 0.71
0.075
0.050
0.6
0.000
0.4
0.002
0.2
-0.075
0.48
0.58
Downstream R-AUROC
Downstream R-AUROC
Figure 5: a model has a high R-AUROC, it is likely also well-aligned with human uncertainties (left). Further, it is likely able detect uncertainties induced via synthetical cropping (right), All results on five further downstream datasets from Schmarje et al. (2022).
how well model's uncertainty estimate is aligned with human and synthetical notions of uncertainty, despite not requiring access to human uncertainty ground truths.
4.7 URL is no out-of-distribution detection benchmark
Last, we want to clarify how URL is different from out-of distrrbbibtt (OOD) detection benchmarks like (Ovadia al., 2019). While both test uncertainty estimates OOD data, the goal is different: In OOD detection benchmarks, the uncertainty estimates are tasked to be generally higher for OOD than for in-distribution (ID) samples. In URL, we look only the OOD data and uncertainties within this data are correctly sorted. This is because our use-case is not to build OOD or anomaly detectors, but pretrained models whose uncertainty estimates generalize new datasets. In Appendices C.4 and C.5 we show that methods with good OOD detection abilities are not necessarily good in URL or vice versa. This demonstrates that URL is concerned with predictive uncertainty estimation (and generalization), which is largely driven by aleatoric uncertainty, rather than epistemic uncertainty estimation, which tested in OOD benchmarks.
the
Conclusion
Summary This paper proposes the uncertainty- representation learning (URL) benchmark. On top of the Recall @1 1, URL adds easy-to- easy-to-implement metric that evaluates how well models estimate uncertainties on unseen downstream data. Besides having a theoretical foundation, also behaves similarly to practical metrics like the alignment with human uncertainties. In benchmarking eleven state-of-the-art approaches on ResNet and ViT backbones, we found that the challenge URL poses is far from being solved. We hope that URL guides the field to overcome this challenge and yield models with reliable pretrained uncertainty estimates.
Outlook We gathered some insights that might guide future developments: Both unsupervised and supervised methods can learn transferable uncertainty estimates. This is not necessarily stakes with the embedding and prediction quality. However, many methods have internal trade-offs their hyperparameters. A deeper analysis of the reasons for this trade-off could allow control and mitigate it. Loss prediction and probabilistic embedding methods are currently the most promising approaches. They may be combined to enhance each other and define state-of-the-art.
I Limitations Although URL allows using any upstream benchmark, we have focused on ImageNet- 1k to train all current methods on the same ground. We leave the investigation of further scaled datasets forthcoming research. Further, we hyperparameter-tuned each model individually with the

---

same budget, but the vast number of hyperparameters in some models, like SNGP, means that our active learning search may have missed some fruitful combinations. Finally, our study concentrates on zero-shot uncertainty estimates. It will be an interesting endeavour to see if pretrained models with good zero-shot estimates also accelerate learning uncertainties in few-shot scenarios.
Acknowledgements
This work was funded by the Deutsche Forschungsgermeinsccatt (DFG, German Research Foundation) under Germany's Excellence Strategy EXC number 2064/1 Project number 390727645. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Michael Kirchhof.
References
Shervin Ardeshir and Navid Azizan. Uncertainty in contrastive learning: On the predictability of downstream performance. arXiv preprint arXiv:2207.09336. 2022.
Arnold Arboretum. The Vrreitrsity dataset, 2020. URL htt.://rbbrrtuu...araaddd harvard edu/ researchhlttt--resorees
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions Paltern Analysis and Machine Intelligence, 3798):1798-1828, 2013.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https::/twwttt wandb. com/
Victor Bouvier, Simona Maggio, Alexandre Abraham, and Léo Dreyfus-Schmidt Towards clear expectations for uncertainty estimation. arXiv preprint arXiv:2207. 133441 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross- retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Sanghyuk Chun, Wonjae Kim, Song Park, Minsuk Chang Chang, and Seong Joon Oh. ECCV caption: Correcting false negatives by collecting seahhin--nnd-uuma---riiiiie image-caption associations for ms-coco. In European Conference on Computer Vision (ECCV), 2022.
Mark Collier, Rodolphe Jenatton, Basil Mustafa, Neil Houlsby, Jesse Berent, and Effrosyni Kokiopoulou. Massively scaling heteroscedastic classifiers. arXiv preprint arXiv:2301.12866, 2023.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Peng Cui, Dan Zhang, Zhijie Deng, Yinpeng Dong, and Jun Zhu. Learning sample difficulty from pre-trained models for reliable prediction. arXiv preprint ar arXiv:2304.1011 2023.
Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomczak. Hyperspherical variational auto-encoders. 34th Conference on Uncertainty in Artificial Intelligence (UAI), 2018.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa. Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, al. Scaling vision transformens to 22 billion parameters. arXiv preprint 2rX2v.2302.054442 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In Computer Vision and Pattern Recognition (CVPR), 2009.
10

---

Nicki Skafte Detlefsen, Jiri Borovec, Justus Schock, Ananya Harsh Jha, Teddy Koker, Luca Di Liello, Daniel Stancl, Changsheng Quan, Maxim Grechkin, and William Falcon. Torchmetrics measuring reproducibility in pytorch. Journal of Open Source Software, 7(70):4101, 2022.
Gianluca Detommaso, Alberto Gasparin, Michele Donini, Matthias Seeger, Andrew Gordon Wilson, and Cedric Archambeau. Fortuna: library for uncertainty quantification in deep learning. arXiv preprint arXiv:2.302.044 04019, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snock, Katherine Heller, Balaji Lakshninarayanan, and Dustin Tran. Efficient and scalable Bayesian neural nets with rank-1 factors. In International conference on machine learning (ICML), 2020.
Tom Fawcett. An introduction to ROC analysis. Puttern recognition letters, 27(8):611-874, 2006.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning (ICML), 2016.
Ido Galil, Mohammed Dabbah, and Ran El-va..v. A framework for benchmarking class-out-of- distribution detection and its application to imagenet. In The Eleventh International Conference Learning Representations (ICLR), 2023a.
Ido Galil, Mohammed Dabbah, and Ran El-Ya.iv. What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers? In International Conference on Learning Representations (ICLR), 2023b.
Dongyoon Han, Junsuk Choe, Seonghyeok Chun, John Joon Young Chung, Minsuk Chang, Sangdoo Yun, Jean Y. Song, and Seong Joon Oh. Neglected free lunch- - learning image classifiers using annotation byproducts. arXiv preprint arXiv:2303. 17595, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR), 2016.
Tony Jebara and Risi Kondor. Bhattacharyya and expected likelihood kernels. In Learning Theory and Kernel Machines. 2003.
Ivan Karpukhin, Stanislav Dereka, and Sergey Kolesnikov. Probabilistic embeddings revisited. arXiv preprint 0rXiv:2202.06766 2022.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sama,, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems (NeurIPS), 33:1866:111 101118666117777 2020.
Michael Kirchhof, Karsten Roth, Zeynep Akata, and Enkelejda Kasneci. A non-isotropic probabilistic take on proxy-based deep metric learning. In European Conference on Computer Vision (ECCV), 2022.
Michael Kirchhof, Enkelejda Kasneci, and Seong Joon Oh. Probabilistic contrastive learning recovers the correct aleatoric uncertainty of ambiguous inputs. International Conference on Machine Learning (ICML), 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In Conference on Computer Vision and Pattern Recognition (CVPR) Workshop, 2013.
Salem Lahlou, Moksh Jain, Hadi Nekoei, Victor I Butoi, Paul Bertin, Jarrid Rector-Brooks Maksym Korablyov, and Yoshua Bengio. DEUP: Direct epistemic uncertainty prediction. Transactions on Machine Learning Research (TMLR), 2023, ISSN 2835-8856.
11

---

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Daniel Langenkämper, Robin Van Kevelaer, Autun Purser, and Tim W Nattkemper. Gear-induced concept drift in marine images and its effect on deep learning classification. Frontiers in Marine Sciiece, 7:506, 2020.
Max-Heinrich Laves, Sontje Ihler, Jacob F Fast, Lüder A Kahrs, and Tobias Ortmaier. Well-calibrated regression uncertainty in medical imaging with deep learning. In Medical Imaging with Deep Learning, pages 393 493-412. PMLR, 2020.
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint 15Xiv:5511.063 14, 2015.
Dan Levi, Liran Gispan, Niv Giladi, and Ethan Fetaya. Evaluating and calibrating uncertainty prediction in regression tasks. Sensors, 22(15)55540, 2022.
Shen Li, Jianqing Xu, Xiaqing Xu, Pengcheng Shen, Shaoxin Li, and Bryan Hooi. Spherical confidence learning for face recognition. In Proceedings the Vision and Pattern Recognition (CVPR), 2021.
the IEEE/CVF Conference on Computer
Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance aware- ness. Advances Neural Information Processing Systems (NeurIPS), 2020.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICLR), 2017. 2017.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. International Conference on Learning Representations (ICLR), 2013.
Zachary Nado, Neil Band, Mark Collier, Josip Djolonga, Michael Dusenberry, Sebastian Farquhar, An- gelos Filos, Marton Havasi, Rodolphe Jenatton, Ghassen Jerfel, Jeremiah Liu, Zelda Mariet, Jeremy Nixon, Shreyas Padhy, Jie Ren, Tim Rudner, Yeming Wen, Florian Wenzel, Kevin Murphy, D. Scul- ley, Balaji Lakshminarayanan, Jasper Snoek, Yarin Gal, and Dustin Tran. Uncertainty Baselines: Benchmarks for uncertainty & robustness in deep learning, arXiv preprint arXiv:2106 106004015 2021.
Seong Joon Oh, Andrew C. Gallagher, Kevin P. Murphy, Florian Schroff, Jiyan Pan, and Joseph Roth. Modeling uncertainty with hedged instance embeddings. In International Conference Learning Representations (ICLR), 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807:03787. 2018
Guillermo Jimenez,e Mark Collier, Anant Nawalgaria, Alexander D' D'Amurr, Jesse Berent, Rodolphe Jenatton, and Effrosyni Kokiopoulou. When does privileged information explain away 2023.
label noise? arXiv preprint arXiv:2303.01
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Joshua Peterson, Ruairidh M Battleday, Thomas Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedingy of the IEFF/CVE International Conference on Computer Vision (CVPR), pages 9617-9626 2019.
Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Bjorn Ommer, and Joseph Paul Cohen. Revisiting training strategies and generalization performance in deep metric learning. In Interna- tional Conference on Machine Learning (ICML), 2020.
12

---

Lars Schmarje, Vasco Grossmann, Claudius Zelenka, Sabine Dippel, Rainer Kiko, Mariusz Oszust, Matti Pastell, Jenny Stracke, Anna Valros, Nina Volkmann. et al. Is one annotation enough? A data- centric image classification benchmark for noisy and ambiguous label estimation. arXiv preprint arXiv:2207.062 14, 2022.
Timm Schoening, Autun Purser, Daniel Langenkämper, Inken Suck, James Taylor, Daphne Cuvelier, Lidia Lins, Erik Simon-Lledó, Yann Marcon, Daniel OB Jones, et al. Megafauna community assessment of polymetallic-nodule fields with cameras: platform and methodology comparison. 2020.
Biogeosciences, 17(12):3115-33
Tyler R Scott, Andrew C Gallagher, and Michael Mozer. von Mises-Fisher loss: An exploration of embedding geometries for supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Yichun Shi and Anil K Jain. Probabilistic face embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Computer Vision and Pattern Recognition (CVPR), 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMIR), 191):1999-1958, 2014.
Dustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, et al. Plex: Towards reliability using pretrained large model extensions. arXiv preprint arXiv:2207.0744 2022.
Gary Ulrich. Computer generation of distributions the m-sphere. Journal of the Royal Statistical Sociity: Series C (Applied Statistics), 33(2):158-163, 1984.
Uddeshya Upadhyay, Jae Myung Kim, Cordelia Schmidt, Bernhard Schölkopf, and Zeynep Akata. Posterior annealing: Fast calibrated uncertainty for regression. arXiv preprint arXiv:2302.110012 2023.
Nina Volkmann, Johannes Brünger, Jenny Stracke, Claudius Zelenka, Reinhard Koch, Nicole Kemper, and Birgit Spindler. Learn to train: Improving training data for a neural network to detect pecking injuries in turkeys. Animals, 1 11(9):2655, 2021.
Nina Volkmann, Claudius Zelenka, Archana Malavalli Devaraju. Johannes Brünger, Jenny Stracke, Birgit Spindler, Nicole Kemper, and Reinhard Koch. Keypoint detection for injury identification during turkey husbandry using neural networks. Sensors, 22(14):51885 2022.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD birds-200-2011 dataset. Technical Report CNS-TR TR-201--111---- 1-011, California Institute of Technology. 2011.
Yeming Wen, Ghassen Jerfel, Rafael Muller, Michael W Dusenberry, Jasper Snoek, Balaji Lakshmi- narayanan, and Dustin Tran. Combining ensembles and data augmentation can harm your calibra- tion. In International Conference on Learning Representations (ICLR), 2021.
Ross Wightman. Pytorch image models, 2019.
Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning- the good, the bad and the ugly. In Computer Vision and Pattern Recognition (CVPR), 2017.
Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations (ICLR), 2020.
Ke Zou, Zhihao Chen, Xuedong Yuan, Xiaojing Shen, Meng Wang, and Huazhu Fu. A review of uncertainty estimation and application in medical imaging. arXiv preprint arXiv:2302 081 19, 2023.
13

---

A FAQ on URL benchmark and R-AUROC
A.1 Why AUROC with respect to @@1 and not accuracy?
First attempts in zero-shot learning literature measured the accuracy on unseen classes. This required learning not only an embedder, but also constructing classifier labels, usually through prototypes or attributes. Subsequent approaches in representation and deep metric learning seeked generally transferable embeddings, where no information is known about the test classes. So, it was impossible to give class logits and calculate an accuracy. Instead they measure the R@ 1. We have the same goal of transferring uncertainty estimates to unknown test conditions, so we built up the uncertainty benchmark on predicting whether the R will be correct or wrong. This measures the risk inherent to the embedding, rather than the (classification) downstream task, which is unknown in advance. However, the risk in the embedding is a bottleneck to the potential subsequent classification layer and we indeed observe in Appendix C.3 that R R@ I and accuracy are highly correlated, so that the difference is negligible.
A.2
Why ranking-basee based metric (AUROC) and not probability- calibration?
The AUROC measures if the predicted uncertainties are ranked such that the most uncertain have the most erroneous predictions. It is intriguing to use different metric, which is to predict the probability of error directly as a value in [0, 1j. However, this metric fundamentally cannot be calibrated before the downstream task is known: In classification, we do not know how many classes there will be and beyond classification, our uncertainty metric should indicate the prediction risk, but without knowing cannot calibrate the uncertainty estimates. Therefore, the best strategy is to give correctly ranked uncertainty metric, which is what AUROC measures. It can be calibrated into the [0, 1] region for the downstream task at hand once data is available. Another point is that [0, IJ estimate has the trivial solution of always giving a random prediction the certainty being the prior over the classes. ranking based metric cannot do that, that would have an AUROC of 0.5. The AUROC thus forces the model to quantify how uncertain it is, not only that it is uncertain.
the loss or its range in before,
A.3 Does URL measure predictive, alcatoric, or epistemic uncertainty?
The R-AUROC in our setup measures the predictive uncertainty, in other words, the overall expected risk of a prediction. We believe this overall uncertainty is most relevant when seeking a reliable model. Note, however, that the AUROC uses a rank-based relative comparison of the uncertainty estimates. This means that a constant high uncertainty on out-of-distribution data does not suffice; instead, the model needs quantify how uncertain it is, not just that it is uncertain. This is influenced both by the inherent ambiguity of the downstream sample (aleatoric) and how far it is from the upstream data (epistemic).
A.4 How is the AUROC implemented precisely?
We use the TorchMetr ics implementation (Detlefsen et al., 2022). It applies the trapezoidal rule and uses every uncertainty value as a possible threshold.
A.5 Why CUB200-2011, CARS196, and Stanford Online Products?
Our definition of URL is agnostic of the datasets and can be applied to any downstream dataset. For this paper, we use CUB200-2011 (Wah al., 201 I), CARS196 (Krause al., 2013), and Stanford Online Products (Song et al., 2016), because they are commonly used as zero-shot learning and representation learning datasets. We hope that this status prevents data-leakage to upstream pretraining datasets.
A.6 Can URL also be implemented for downstream datasets that don't have labels?
Throughout this paper, we measure the R@1 1 by seeing if the next neighbor of test image has the same class label. This ensure the compatibility with representation learning benchmarks. One can also define self-supervised R @@I (e.g., seeing if a crop of the same image is detected as belonging
14

---

to that image), In this case, R-AUROC automatically generalizes to this form of supervision, as it still measures errors R@ regardless of how it was computed.
A.7 Can URL be applied beyond vision tasks?
Yes. URL can be deployed whenever there is downstream dataset on which we measure a R@
B
Reimplementations and hyperparameterss
All methods benchmarked this paper are re-implemented and hyperparameter tuned to ensure consistent comparison. First, let explain the hyperparameters shared by all approaches: As backbones, we use ResNet-50 (He et al., 2016) architectures with 224x224 inputs and 2048-dim max- pooled embeddings, and ViT-Medium (Dosovitskiy et al., 2021) with an input size of 256x256 split into 16x16 patches. All models are tuned for 32 epochs with a Lamb optimizer (You et al., 2020). 16 minibatches of size 128 accumulated reach summed batch size of 2048, unless otherwise noted. The learning rate is fixed at the first epoch, then warmed up for five epochs and cooled down using a cosine scheduler (Loshchilov and Hutter, 2017). The learning rate is hyperparameter for all approaches, searched over range of [0.0001, 0.01]. The hyperparameter search is extended by more runs if the optimal value close to boundary. Let now describe the approaches and their additional hyperparameters in detail. All optimal hyperparameters are reported in the code appendix.
B.1 Cross-Entropy (CE)
The CE baseline applies cross-entropy loss the class-logits output by final linear layer appended to the embeddings. It has no hyperparameters except the learning rate.
B.2 InfoNCE
InfoNCE (Oord et al., 2018) uses random crops per input image that considered positive, whereas the remaining batch is considered negative. This results in a doubled VRAM usage, so that the batch size is reduced to 21 minibatches of 96 images each. InfoNCE uses an (inverse) temperature parameter f, tuned within [8, 64J. All embeddings e are normalized to lay on the unit sphere, where all negative embeddings. The final loss is
ef and ež denote the positive and e
= -t e + log
Σ exp (t (TTT (6-) + log
Σ
exp (t. (t) (1)
eBalhhch
EBBcch
B.3 MCInfoNCE
MCInfoNCE (Kirchhof et al., 2023) works similar to InfoNCE, except that it does not directly compare the embeddings, but samples from estimated posteriors st vMF(e k(e )). s vMF(e x(ez)) )), 87 ~ vMF(eT, x(e))). The concentration, i.e., inverse uncertainty, is estimated from 3-layer MLP attached to the embeddings. Its initial value is either randomly initialized or warmed up to 0.001, which is hyperparameter. The second hyperparameter is its (inverse) temperature [8, 64). Like InfoNCE, uses a reduced batch size of 21 times 96. The loss obtained by calculating the InfoNCE loss for 16 samples s; from the respective posteriors.
16
C
-t
81, 82,5 + log
Σ Eatch
exp (t- T (7) log
T Σ exp (t.
s $7
(2)
16
1=1
AT
*T ERatch
B.4 Expected Likelihood Kernel (ELK)
ELK uses ProxyNCA formulation, as proposed in Kirchhof et al. (2022). Like MCInfoNCE, it parametrizes posteriors C = vMF(e, k(e)) from each image's normalized embeddings and layer MI.P for Its initial value is either randomly initialized or warmed up to 0.001, which is hyperparameter. ELK is supervised and learns vMF class distributions Pe for each class e == I.....C C.
The concentrations of these classes are scaled up by the hyperparameter t e [8, 64], which takes
15

---

similar role to the inverse temperature in the previous losses. These are compared to the embedding posterions via cestributiontt-distiiiii not requiring sampling. With p" denoting the true class of the given sample, the loss can be written as
similarity function elk_sim. This is solved analytically,
= -elk _lk_ssim(s,p") + log exp foll_sim((s/pp
B.5 Non-isotropic von Mises-Fisher (nivMF)
nivMF (Kirchhof et al., 2022) has the same hyperparameters and loss as ELK, except that the class proxy distributions Pe are non-isotropic vMFs. Since the expected-like likelioood between the image embeddings" vMFs and the classes' non-isotropic vMFs has no analytical solution, it is Monte-Carlo approximated with 16 samples.
C = aapprox_ekk_sinneeee + log _ cmm
exp
(4)
B.6 Hedged Instance Embeddings (HIB)
HIB (Oh et al., 2019), like MCInfoNCE, takes samples from estimated posteriors Sn,i ~ vMF(en, x ((n))), where en are the image's T.2 normalized embeddings e, and estimated by 3-layer MLP. Its initial value is either randomly initialized or warmed up to 0.001, which is a hyper- parameter. HIB then calculates a matching probability by comparing the samples of every pair of images n. m in the batch: Pa.m = Eis sigmoid (t $m,; + b), where E [8, 64] is a hyperpa- rameter similar to the (inverse) temperature and b E [-8, 8] is a second hyperparameter, The match- ing probability should be high for images with the same label and low for images with different labels. Let Tume denote the pairs of images with the same label and Tdifferent the pairs with different labels, both without self-matches. Then the loss is
C
Σ
logPn,
Σ
log Pn.mm
[aame
[Taifferent/
(Z.
(n.m))Eanteen
where |- denotes the cardinality of the set. As opposed to the original implementation, we use cosine distances instead of L2 distances and remove the prior regularizer. This to make HIB more comparable to the other approaches in this paper. We also changed the second term from encouraging low log match probabilities for different labels (- log(1 (11 pn.m)) to discouraging high ones (+ log pn. which stabilized training. HIB requires additional VRAM and thus uses 21 batches of size 96 (43 of size 48 on ViTs).
B.7 Heteroscedastic Classifier (HET-XL)
HET-XL (Collier et al., 2023) predicts a distribution in the embedding space for each image. It then takes samples and calculates a Monte Carlo estimate of the expected model output under the embedding distribution. As opposed to the other probabilistic embedding approaches from above, operates in Euclidean space by predicting Gaussian distribution (o((; c 0). <'(x; 0 with low- rank approximation of the covariance matrix Ex;:; dow) V(x) V(x) diag (d())). V(x) and d(x:) are output by a linear layer attached to the embeddings. The number of columns in V increases the rank of the low-rank approximation, but also the number of parameters that the final linear layer has to predict, and thus the memory requirements. We thus set this hyperparameter to 1 (exploratory experiments with rank of 10 did not show increased performance). The final loss
Lemes-entropy (E,r [softmax, (W (o(x;; 0) + e' e((x)) .w) with e'(x) N (0, E' E'<x; 0cox)) (6)
where the softmax temperature is learnable parameter.
B.8 Speraizll Neural Gaussian Processes (SNGP)
SNGP (Liu al., 2020) predicts a Gaussian distribution N (o(x)) (b(x))B.<(()) (.)( (I + TT 5(xx)
16

---

over the class logits, which is cast into class probabilities via mean-field approximation. These class probabilities are then trained under CE loss. is learnable parameter matrix, Φ(z) cos (Wh(z) + b) is feature embedding based on frozen random parameters W and b, and $ 2 is the empirical covariance matrix of the feature embeddings over the training dataset. The method also applies spectral normalization the hidden weights in each layer in order to satisfy input distance awareness. We treat whether to apply spectral normalization through the network and whether to use layer normalization the last layer as hyperparameters.
B.9 Direct Loss Prediction (Losspred)
Losspred trains a classifier under a cross-entropy loss and uses its uncertainty module i, 3-layer MLP attached to the embedding space, to predict the value of the cross-entrope loss. Both components are balanced by the hyperparameter À E [0.01, 0.99]:
L entrs
entropy (r. y) + X (1<(2) -
cross-entropy (x. y))
Ldetached
(8)
The gradients Edetached enoes-entropy inside the L2 loss are detached to prevent fitting it to r(()), instead of the other way around. Besides A, second hyperparameter is whether warm up 0.001.
B.10 Deep Ensemble
Ensemble (Lakshminarayanan et al., 2017) has 10 classifier heads attached to the embedding space. Their logits are transformed into probabilities by a softmax and then averaged. This average is trained under CE loss. Ensemble has no hyperparameters other than the learning rate.
B.11 MCDropout
MCDropout (Srivastava et al., 2014) trains with dropout rate of [0.05, 0.25). which is hyperpa- rameter. During inference time, keeps the dropout activated to sample 10 logits and averages them like Ensemble.
C Additional results
C.1 Pretrained models already close the gap in terms of R@1
This section compares the models in terms their R@ To this end, we hyperparameter-tuned them with respect to @@ I, and not R-AUROC. Similar to the main text, we also add an additional baseline that was trained on the downstream classes, where the original test split was split into equal sized train and test splits.
Fig. 6 shows these performances. As opposed to the R-AUROC, we pretrained zero-shot and many-shot models is much tighter in terms of the R@ 1. The best pretrained ResNet-50 has average R @ @1 of 0.48 vs. 0.54 when training the downstream data.
that the gap between
Surprisingly, when it comes to @@l, CE among the best two approaches both for ResNet and ViT backbones. In fact, three of the four best approaches on ViTs and two of the four on ResNets rely on CE as part of their loss. The approaches that do not rely on CE are probabilistic embeddings nivMF on ResNet and VIT and ELK on ResNet.
C.2
@ and R-AUROC correlate negatively in most models
In extension the plot that compared the best hyperparameter setup for R-AUROC to the best for R@1 1, Fig. and Fig. 8 present the trade-offs for all tested hyperparameters.
The general picture is the same as in the comparison of best vs best: Most models show trade- off between achieving the best R-AUROC and the best R @@1 1. This indicated by a negative rank correlation in 15 out of 22 models. Still, there are some models where the uncertainty estimation and prediction performances correlate moderately positively (0.4 S Rank Corr. < 0.72). These are InfoNCE and MCInfoNCE on ResNets, and HET-XL, nivME, and HIB on ViTs.
17

---

0.6
ResNet 50 trained on downstream test classes
Downstream R@l
0.4
0.2
0.1
0.0
ELK
CE
aivME
Losspred
Ensamble
MCIafoNCE
SNGP
HIB
InfoNCE
HET-XL.
MCDmpout
Lesppee
nisMF
MCDaqoout
CR
HET-XL
MC
HBB
InfONCE
SNGP
Ensamble
ELK
ResNet 50
VIT Medium
Figure 6: Pretrained models achieve almost good R unseen downstream data models trained on the downstream data. Minimum, average, and maximum R@1 per model. Model hyperparameters were optimized w.r.t. R@1.
As mentioned in Appendix B, each approach was tested on 10 hyperparameters, with additional runs other seeds for the best hyperparameters. The reason that some plots show less than 12 points that those had a R@ 0.1 on the downstream datasets and were excluded from the analysis. Some plots also show more than 12 points. This is because their best hyperparameter for R was unlike that for R-AUROC, adding another 2 runs on different seeds for the R@ Further, some approaches had optimal hyperparameters close to the original search bounds, such that the search was extended, leading to additional points in the plots.
18

---

Rank Corr. = -0.34
Rom = -0882
Rank Corr. + 0.32
Downstream R-AUROC
0.515
Downstream R-AUROC
Downstrean R-AUROC
0.54
0.510
0.305
0.34
0.500
0.53
0.50
0.495
CE ResNeg 50
0.52
ELK ResNet 30
Ensamble ResNet 50
0.40
0.45
0.40
0.45
0.35
0.40
0.45
Downstream R@!
Downstream R@
Downstream R@
0.515
Rank Coti. -0.30
Ränk Cotr. -0.36
Rank Corr. 0.72
Downstream R E-AUROC
Downstream R-AUOO
Downstream RUROO
0.325
U.510
0.54
0.305
0.53
0.515
0.500
0.52
0.510
0.495
HELT-XL ResiNet 50
0.51
HIB ResNet 50
0.505
InfoNCE ResNet 50
0.25
0.30
0.35
0.30
0.35
0.40
0.45
0.25
0.30
0.40
Downstream R@1
Downstream R@)
Downstream R@1
Downstream R-A E-AUROC
0.56
Rank Corr. -0.60
Downstream R-AUROC
0.525
Rank Corr. 0.02
Downstream R-AUROC
0.573
Rank Corr. 0.60
0.550
0.520
0.52
0.515
0.525
0.510
0.505
0.475
0.48
Losspred ResNet 50
0.500
MCDropout 50
0.450
MCInfoNCE ResNet 50
0.30
Downstream R@
Downstream R @@)
Downstream R@!
Downstream R-AUROC
RanleCorr, -0.86
Downstream R-AUROC
Rank Corr. -0.58
0.510
0.505
0.300
0.54
U.495
0.52
0.490
0.485
SNGP ResNet 50
0.50
nivME ResNet 50
0.3
Downstream R@
Downstream R@1
Figure 7: R@ I and R-AUROC are negatively correlated on seven out of ten model classes with ResNet backbones. Plot shows all tested hyperparameter combinations.
19

---

0.540
Ránk Corr.
-0.26
0.57
Rank Cor
75
Rank Corr. 0.27
Downstream R AUROC
Downstream R- AUROOC
Downstream R-AUROC
0.530
0.533
0.56
0.530
0.525
0.55
0.525
0.54
0.520
0.515
0.515
CE VIT Me
0.52
DLK VIT Medium
Ens able ViT Medium
0.510
0.4
0.4
0.40
0.45
Downstream R@
Downstream R@1
Downstream R@!
Downstream R-AUROC
0.54
Rank Corr. 0..3
Downstream R-AUROC
Rank Corr. 0.49
Downstream R-AUROC
Rank Corr. -0.22
0.52
0.54
0.50
0.53
0.49
0.50
HET-XL VIT Mediam
0.52
1B VIT Medium
NNCE VIT Medium
0.48
0.4
Downstrearn R@
Downstream R@1
Downstream R@!
Rank Cor 40.39
Rank Corr. -0.46
Bank Corr. -0.42
Downstream R- AUROC
Downstream R-AUROC
Downstream R AUROC
0.57
0.53
0.56
0.52
0.50
0.55
0.51
0.49
0.54
Losspred VIT Medi
0.50
MC MCDropout ViT M
m
0.48
MCInfoN E VIT Med um
0.45
0.50
0.55
0.2
0.4
0.4
0.5
Downstream R@
Downstream R@
Downstream R@
0.58
Rank Corr.
-0.64
Rank Corr. 0.40
Downstream R AUROC
Downstream R AUROC
0.55
0.57
0.54
0.56
0.53
0.55
0.52
0.54
SNGP Medium
MF VIT Me
0.42
0.46
0.40
0.45
0.50
0.55
Downstream R@]
Downstream R@
Figure 8: R@ I and R-AUROC are negatively correlated on seven out of model classes with VIT backbones. Plot shows all tested hyperparameter combinations.
20

---

C.3 Further correlation of up- and downstream metrics
This section reports further metrics for each model. These include new metric (top-1 accuracy on upstream ImageNet) and all downstream metrics (R@ I, R-AUROC, percentage where cropped image has higher uncertainty) on the upstream dataset.
Fig. 9 shows the pairwise correlations between all metries, for every tested hyperparameter setting. Let us first consider the new metric, ImageNet top-1 accuracy, on its own. The best-performing models a CE and nivMF VIT, with an accuracy of 0.84 each. Other than them, Losspred and HET-XL on ViT' also have a high accuracy, similar to the results on the R@ benchmark.
Regarding correlations between metrics, accuracy and R@) are highly correlated, as expected. Further, models with a high accuracy or R@ on ImageNet also have a high R@ on the downstream data, resulting in the small gap explained in Appendix C.I. While up- and downstream R-AUROC do not correlate, up- and downstream percentage of cropped images having a higher uncertainty correlate nearly linearly. This reinforces that the percentage uncertainty. It should, however, be noted that ELK VIT's already achieve a performance this metric, downstream data, such that it able to guide the field as well as R-AUROC.
be considered
general notion of 0.99 on
Figure 9: Correlations between several up- and downstream metrics across all models and hyperpa- rameter choices.
21

---

C.4
Class-entropy is useful for OOD detection
0.8
0.6
0OD AUROC
0.5
0.4
0.2
0.1
InfoNCE
SNOP
HET-XL
CE
Ensemble
MCInfoNCE
Leaspred
MivMF
MCDmpout
HIB
ELK
SNGP
HET-XL
MCInfaNCE
MCDapoul
InfoNCE
CE
Easemble
Lospped
HiB
ELK
nivMF
niv
ResNet
VIT Medium
Figure 10: Cross-entropy and embedding norm-based uncertainty estimators have the best OOD detection capabilities. Averaged AUROC of distinguishing ImageNet vs CUB, ImageNet vs CARS, and ImageNet vs SOP. Error bars indicate minimum/maximun performance across seeds.
0.7
ImageNet (ID)
0.14
ImageNet (ID)
CUB (OOD)
CUB (OOD)
0.12
CARS (OOD)
(OOD)
0.3
SOP (OOD)
0.10
SOP (OOD)
Density
Density
0.4
0.06
0.2
0.04
0.1
0.02
0.00
10
40
L2 norm of embedding (ResNet 50)
L2 norm of embedding (ViT Medium)
Figure 11: Embeddings of OOD images tend to have smaller L2-norms than embeddings of ID images. Embedding norms are from InfoNCE models that use the (inverse) norm as uncertainty estimate (without using them during training) and reach an OOD AUROC of 0.73.
In this section, we study if uncertainties on downstream data are generally higher than for the upstream data. To this end, we perform out-of-distribution detection experiments: Using the same pretrained models as before, we calculate their uncertainties on downstream dataset samples and on equally sized set of upstream samples. We quantify how well the predicted uncertaintics distinguish whether the sample from downstream dataset (1) not (0) by calculating the AUROC on ImageNet vs CUB, ImageNet vs CARS, and ImageNet vs SOP, and averaging them across the datasets.
Fig. 10 shows the average result across all seeds, along with minimum and maximum performances. Generally, pretrained models perform differently than the URL benchmark in the main text. This is because the OOD task benchmarks epistemic uncertainty (i.e., OOD data just has to have generally high uncertainties). On the other hand, the URL benchmark tests predictive uncertainties on OOD data. There, it is not enough to predict high values on OOD data but the models need to differentiate within them, which lays more focus on aleatorie uncertainty. In more detail, models that directly predict variances losses do not provide good OOD performance. Models that class-entropy
22

---

as uncertainty estimates (SNGP, HET-XL, CE, Ensemble, and, only on ViTs, MCDropout) and also InfoNCE, which uses the norm of the embedding vector, do work well.
There is an intuitive explanation for this. The latter models implicitly provide epistemic uncertainty estimates by construction: ResNets and ViTs embed inputs with less known features closer to the origin. As an example, Fig. 11 shows that for InfoNCE, the embedding norms of OOD samples are smaller. Note that this is no trained behaviour; InfoNCE only trains on normalized embeddings, so the norms occur naturally. In InfoNCE, we use this embedding norm directly as uncertainty estimator. But the same happens in models that use the class entropy: Embeddings with small norm lay close to the origin, where they lead to uniform distributions over the classes, i.e., a high entropy.
In summary, the OOD experiment reveals that probabilistic embeddings and loss prediction methods provide aleatoric uncertainty estimates, whereas models that explicitly or implicitly use the embedding norm provide good epistemic uncertainty estimates. Some models, like SNGP, provide mixture and are good in both tasks.
C,5 Uncertainties on mixtures of in- and out-of-distribution data
0.7
0.6
Eval Time (
0.1
0.3
0.1
0.0
SNGP
HET-XX
InfoNCE
Ensemble
Losppee
MCIafoNCE
MCDropeul
HIB
ELK
SNGP
HET-XL
MCDropout
CE
Ensemble
MCInfoNCE
InfoNCE
EL
HIB
Losepred
BiMF
ResNer
VIT Medium
Figure 12: Cross-entropy based uncertainties provide the best mix of OOD detection capability and ordering within ID and OOD uncertainties. R-AUROC on mixed ImageNet+CUB, ImageNet+CARS. and ImageNet+SOP data. Error bars indicate minimun/maximum performance across seeds.
In some applications, models might encounter a mixture of in-distribution and data. In this case, the model both needs to assign higher uncertainties to the OOD data and needs to differentiate the uncertainties within both the ID and the OOD sets. This blends the OOD detection of the previous experiment with traditional ID calibration and the OOD calibration of the URL benchmark.
Fig. 12 measures the R-AUROC on a mixture of upstream and downstream data, As in the previous experiment, we use 50/50 splits of ImageNet+CUB, ImageNet+CARS, and ImageNet+SOP, and average across those combinations. We find that HET-XL and SNGP, which previously performed well on both OOD detection and the URL benchmark, also perform well on this mixed task. The remaining models tend follow the ranking of the OOD benchmark rather than that of URL. This indicates that good epistemic uncertainty estimation outweighs aleatoric uncertainty estimation in this task. This intuitive, because the R-AUROC measures how likely has higher uncertainty than a correct one. In such 50/50 split of ID and OOD data, the capability to distinguish ID from OOD data, and thus data with generally less errors from data with generally more errors, thus leads to higher RUAUOC.
that a wrong prediction
23

---

This serves to demonstrate that the quality of an uncertainty estimate always depends on the task and setup hand. While OOD detection and ID calibration are ideally both reflected within the very same predictive uncertainty value, both are of different importance depending on the data mixture.
C.6
Few-shot uncertainties starting from pretrained models
0.72
CE ResNet 50
CE VIT Medium
ELK ResNet
0.73
ELK VIT Medium
Downstream R-AUROC
MCInfoNCE ResNet 50
Downstream R RAAUROC
Losspred VIT Medium SNGP VIT Medium
0.68
nivMI ResNet 50
0.70
0.66
0.64
0.65
0.62
0.60
0.60
0.56
2
6
10
6
10
k-shot (ResNet 50)
k-shot (VTT Medium)
Figure 13: Cross-entropy training on the downstream data reaches the zero-shot AUROC of 0.6 at 2-5 samples. R-AURÓC of pretrained models when finetuned on the downstream test classes.
In this section, provide initial attempt few-shot experiment. We use both the best three pretrained models (ELK, MCInfoNCE, and nivMF for ResNet 50; ELK, Losspred, and SNGP for VIT Medium) along with the CE model as pretrained checkpoints and continue to train under the same hyperparameters and losses on E 21, 5, 10} samples of each class of the downstream datasets. This is done on the test classes of the downstream classes, which were separated into disjoint train and test samples for this experiment. We train on each CUB/CARS/SOP separately and average their performance.
Fig. 13 shows the minimum, maximum and average performance across the seeds. First, we can see that the normal cross entropy training reaches the zero-shot R-AUROC of the currently best pretrained models at around samples per class (totaling in 200 samples on CUB, 196 on CARS, and 22636 on SOP). This again demonstrates the point that the challenge URL addresses is unsolved yet. also shows that knowledge the specific downstream task can increase the uncertainty estimators quality even at only few samples. Second, we find that most pretrained models increase their performance as well. This, however, happens not for all models and is highly noisy. We attribute this to the simple setup of our experiment, which uses the same hyperparameters for the pretraining and performs standard training as opposed to specialized few-shot methods. Finding best practices to tune pretrained uncertainties downstream few-shot tasks is promising undertaking for future research.
C.7 Uncertainty estimation does not add significant computational costs
Table reports the computational complexity during all benchmarks. It shows the number of parameters as proxy for RAM usage, the duration the first training epoch and evaluation, and the time needed for each sample during evaluation. These results were collected the go and there possible confounders such as network storage workload. We thus recommend to interpret them as rough indicators. Note also that ViTs were run NVIDIA A100 GPUs, and ResNets NVIDIA RTX2080TIs (except HET-XLon ResNet, due to RAM usage).
First, we see that explicit uncertainty estimation does not come at a high RAM cost. The 3-layer MLP serving as uncertainty head for MCInfoNCE, ELK, nivMF, HIB, and Losspred adds 4.2M parameters to ResNet or 2.6M to a VIT. The difference is due to the VIT's lower-dimensional embedding space. Bigger increases occur only for ensembles, which uses additional classifiers with 10 M. parameters on ResNets and each 10 0.5M on ViTs.

---

Model
Parameters (Millions)
Epoch Time (s)
Inference time per sample (ms)
CE
25.6
5971
3.7
ResNet 50 on RTX 208011
InfoNCE
23.5
6703
3.7
MCInfoNCE
27.7
6656
3,8
ELK
29.8
6703
3.8
nivMF
29.8
6256
3.8
HIB
29.8
7279
3.8
HET-XL (on A100)
33.9
(4189)
(2.3)
Losspred
29.8
6526
3.7
MCDropout
25.6
7105
13.1
Ensemble
46.0
6002
3.7
SNGP
28.7
7632
3.7
CE
38.9
4922
2.8
InfoNCE
38.3
7804
2.7
VIT Medium on A100
MCInfoNCE
41.0
9883
2,4
ELK
41.5
4838
2.7
nivMF
41.5
4121
2.4
HIB
41.5
3950
2.5
HET-XL
39.4
4205
2.4
Losspred
41.5
3814
2.6
MCDropout
38.9
4721
9.6
Ensemble
44.0
4107
2.5
SNGP
42.0
4811
2.9
Table Computational costs of all approaches. Epoch times include training on ImageNet as well as evaluating on all eight downstream datasets.
Training times should be interpreted with caution due to the aforementioned network storage. How- ever, in general uncertainty estimates do not seem exceedingly increase the train time, with be- tween -22% and +28% over the CE baseline. Taking multiple samples during training (MCInfoNCE, nivMF, HIB, HET-XL.) also does not systematically increase runtime. This is likely due their efficient sampling implementations (Kirchhof et al., 2022; Davidson et al., 2018; Ulrich, 1984). The only consistent runtime cost occurs when training unsupervised models (InfoNCE, MCInfoNCE), which need to forward propagate two augmentations of each sample to obtain have positive pairs.
Similarly, providing uncertainty estimate at inference takes only up to 0.1 additional milliseconds on ResNets, because the uncertainties are all calculated within a single forward pass, and sampling was only required for the losses during training. The only exception here is MCDropout, which requires making 10 full forward passes during inference, which increases the time by a factor of roughly 3.6. The factor is not 10, because loading the data and storing the results is part of the measured elapsed time.
In summary, we find that uncertainty estimates only have small computational costs if they are implemented in a forward fashion.
25


Reviews:
[{"review_id": "qJwuQlOKtj", "reviewer": "Reviewer_HXyY", "summary": "", "questions": "", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "(1) **Contribution:** This paper introduces a novel URL benchmark to evaluate state-of-the-art uncertainty estimators on unseen data. The observations from the experimental results reveal the connections between uncertainty estimation and embedding estimation. It also points out that models with good uncertainties upstream might not be a solid indicator of downstream.\n\n(2) **Relevance:** The proposed benchmark demonstrates that transferable uncertainty estimation is challenging and unsolved for existing approaches. Thus, it provides some insights into generalizing the estimated uncertainty to unseen data. This might motivate the community to design universal uncertainty estimators in future work.\n\n(3) **Quality:** This paper is well written and the problem setting of transferable uncertainty estimation is well motivated. The main findings are supported by the experimental results.\n\n(4) **Ethical and social implications:** There is no explicit ethical concern for the proposed benchmark. Instead, transferable uncertainty estimation allows for exploring the trustworthiness of the model in unseen data and domains. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "UuLoi1IofF", "reviewer": "Reviewer_D238", "summary": "", "questions": "", "limitations": "The authors mentioned their limitations clearly.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper makes a contribution to the advancement of pretrained models that offer reliable uncertainty estimates, a complex yet crucial problem in the machine learning domain. It creates a pathway for a more nuanced understanding of how models can predict uncertainties, which is a key step towards more reliable AI.\n\nThe thoroughness of this study is exemplary, with an evaluation of ten distinct uncertainty quantifiers that have been pre-trained on ImageNet and transferred to eight different downstream datasets. This comprehensive analysis provides an extensive review of the existing state-of-the-art methodologies. This breadth of experimentation enriches the study and deepens our understanding of uncertainty quantification, thus marking a substantial stride in this field of research.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "4nZynQZEET", "reviewer": "Reviewer_qnX2", "summary": "", "questions": "", "limitations": "The authors discuss some limitations of their benchmark in the paper, e.g. with respect to the used datasets and hyperparameter tunings in the experiments. The weaknesses/limitations I see (cf. \"Opportunities for Improvement\") are not discussed.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The benchmark presented in the paper is very extensive. The authors provide a detailed analysis of the results. A diverse array of uncertainty estimation methods is evaluated.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "HLjaXxkUBQ", "reviewer": "Reviewer_JYVQ", "summary": "", "questions": "", "limitations": "The paper conducted some experiments on the same ground. TheThey did not touch the scaled data. ", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "(1) The paper tackled a vital and challenge problem, URL.\n(2) The proposed benchmarking method can help  to understand some uncertainty in representarion learning and measure it using the metric, R-AUROC；\n(3）The authors conducted some experiments to demonstrate how to benchmark the uncertainty estimator for representation learning.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The authors introduce a benchmark for representation learning that evaluates the transferability of uncertainty estimates to zero-shot downstream tasks. The authors provide an extensive benchmark of various kinds of models. The models are pre-trained on the ImageNet dataset. Eight datasets are used for the downstream evaluation. The results show that no model achieves a good R-AUROC score (all below 0.6), indicating much room for improvement.

If looking at the average score (6,5,6,5), the mean 5.5. doesn't place this paper competitively above the acceptance bar. However, I am recommending acceptance based on the following facts:

- Reviewer JYVQ (rate 5) provided literally no helpful information in his/her review, and did not participate in any follow-up discussion with either author or AC.  AC firmly believes counting on such an outlier is damaging to a fair conference. So, the rating from Reviewer JYVQ was excluded, leaving scores (6,6,5) as "factual ratings" that AC considers.

- Reviewer D23 (another rate 5) did not participate in followup discussion. AC read his/her comments and found his main questions, namely the variety of scenarios including OOD and the computational cost issues, have been thoroughly addressed by the authors. The former issue was same raised by Reviewer qnX2, and was addressed to satisfaction by authors (as confirmed by Reviewer qnX2)

- Reviewers HXyY and qnX2 (both rate 6) provide longer and informative review comments. Particularly, Reviewer qnX2 engaged in thorough, multi-round discussions with the authors, and they came to many consensus afterward. Reviewer qnX2 actually improved the rating after the discussion to show support.


Comparing all papers in AC' batch, it is believed that this paper has the highest quality, and the authors addressed the reviewers' comments to the best. For example, the most engaging reviewer (qnX2) has raised many constructive concerns and the authors resolved them all. Therefore, AC found it fair to consider this work a competitive, and ready contribution to NeurIPS D&B.

This decision has been made in agreement with the AC and SAC.
Decision: Accept (Poster)