# ViLU: Learning Vision-Language Uncertainties for Failure Prediction

Marc Lafon*,¹

Yannis Karmim*,¹

Julio Silva-Rodríguez³

Paul Couairon²

Clément Rambour¹,²

Raphaël Fournier-Sniehotta¹,²

Ismail Ben Ayed³

Jose Dolz³

Nicolas Thome²,⁴

¹Conservatoire National des Arts et Métiers, CEDRIC, F-75141 Paris, France

²Sorbonne Université, CNRS, ISIR, F-75005 Paris, France

³ETS Montreal, Canada

⁴Institut universitaire de France (IUF)

## Abstract

Reliable Uncertainty Quantification (UQ) and failure prediction remain open challenges for Vision-Language Models (VLMs). We introduce ViLU, a new Vision-Language Uncertainty quantification framework that contextualizes uncertainty estimates by leveraging all task-relevant textual representations. ViLU constructs an uncertainty-aware multi-modal representation by integrating the visual embedding, the predicted textual embedding, and an image-conditioned textual representation via cross-attention. Unlike traditional UQ methods based on loss prediction, ViLU trains an uncertainty predictor as a binary classifier to distinguish correct from incorrect predictions using a weighted binary cross-entropy loss, making it loss-agnostic. In particular, our proposed approach is well-suited for post-hoc settings, where only vision and text embeddings are available without direct access to the model itself. Extensive experiments on diverse datasets show the significant gains of our method compared to state-of-the-art failure prediction methods. We apply our method to standard classification datasets, such as ImageNet-1k, as well as large-scale image-caption datasets like CC12M and LAION-400M. Ablation studies highlight the critical role of our architecture and training in achieving effective uncertainty quantification. Our code is publicly available and can be found here: ViLU Repository.

## 1. Introduction

Vision Language Models (VLMs) [9, 25, 27, 36, 50] are highly popular foundation models pre-trained on large-scale image-text datasets, e.g., LAION [37]. They possess the appealing capability of performing zero-shot image classification, meaning they can classify samples into classes that were not seen during training.

Uncertainty quantification (UQ) is a fundamental challenge in deep learning and involves estimating the confidence of a model's predictions. This paper addresses the problem of reliable UQ for VLMs to detect their potential failures in downstream tasks. Reliable UQ is crucial in safety-critical domains and offers significant opportunities for various applications involving VLMs, such as failure prediction [17], out-of-distribution detection [46], active learning [47], and reinforcement learning [12], among others.

The vanilla method for UQ with VLMs is the Maximum Concept Matching (MCM) score [30], a direct extension of Maximum Class Probability (MCP) [17] for classification. Although MCM is a strong baseline, it suffers from fundamental drawbacks: by design, it assigns high confidence to failures and struggles with fine-grained concepts. As illustrated in Fig. 1, the VLM misclassifies the “Eskimo dog” image as a “Siberian husky”, and the high MCM score prevents the detection of the error.

In classification, learning-based methods have also been explored for failure prediction. In particular, a few deep learning models designed to predict the classifier's loss and dedicated to learning visual uncertainties (LVU) have been proposed [8, 19, 47]. However, when applied to VLMs, these methods do not model the relationships between downstream concepts, intrinsically limiting their failure prediction performance. The example in Fig. 1 still receives a high confidence score with LVU methods. Furthermore, although recent methods have proposed specific calibration techniques for VLMs [31, 34, 42, 44, 48], fewer works have focused on UQ solutions for VLMs in general, with the exception of the recent [2].

This paper introduces ViLU, a post-hoc framework for learning Vision-Language Uncertainties and detecting VLMs' failure. The core idea in ViLU is to define a con-

<footnote>* Equal contribution
Corresponding author: marc.lafon@lecnam.net</footnote>
<figure><img src="image_1.png" /><figcaption>Figure 1. Motivation of ViLU. In zero-shot classification with VLMs, uncertainty may arise from both the image and target concept definitions. Given an ambiguity between several concepts, the vanilla Maximum Concept Matching (MCM) [30], can assign a high confidence to a wrong prediction, e.g. an "American Eskimo dog" wrongly classified as a "Siberian husky". Previous methods based on Learning Visual Uncertainties (LVU) [8, 19, 47] do not account for ambiguity between concepts and often fail, whereas ViLU captures a fine-grained uncertainty by contextualizing the image within the spectrum of possible textual concepts.</figcaption></figure>

<figure><img src="image_2.png" /></figure>

fidence score depending both on the visual input but also on the set of concepts that defines the downstream task, e.g. classification or caption matching. By finely modeling the interaction between the image and the target concepts, ViLU assigns a low confidence score to the misclassified example – see Fig. 1.

We summarize our key contributions as follows:

* ViLU introduces a novel multi-modal uncertainty representation that integrates visual embeddings, predicted textual embeddings, and image-conditioned textual representations via a cross-attention module. This formulation enables the model to capture fine-grained ambiguities between the input image and candidate concepts, significantly improving failure prediction.

* We propose a dedicated uncertainty predictor that operates on this enriched representation and is trained to discriminate between correct and incorrect predictions using weighted binary cross-entropy (BCE). Unlike conventional loss-prediction-based UQ methods, ViLU is fully loss-agnostic, making it particularly well-suited for post-hoc uncertainty estimation in black-box VLMs.

We conduct an extensive experimental validation of ViLU on various downstream classification datasets, as well as image-caption datasets such as CC12M and LAION-400M. First, we highlight that state-of-the-art methods struggle to outperform the MCM baseline, illustrating the difficulty of the task. In contrast, ViLU delivers significant and consistent improvements over multiple baselines, including recent VLM-specific methods [2]. Thorough ablation studies further validate our architectural choices and training design for optimal performance.

## 2. Related Work

**Vision-language models (VLMs)** have gained popularity for aligning image and text representations [9, 25, 27, 36, 50], achieving unprecedented zero-shot performance by jointly learning a shared vision-text embedding space on large-scale web datasets [9, 24, 37]. They have been applied in a plethora of domains including image classification [22, 52, 54], open-vocabulary segmentation [23, 43, 49, 51] and cross-modal retrieval [1, 18]. Nevertheless, VLMs align deterministic text and image representations without accounting for uncertainty in zero-shot predictions [2, 42]. While simple uncertainty quantification methods exist, e.g. MCM [30], a robust approach for failure prediction remains lacking. We address this gap with an effective UQ method for failure prediction of VLMs.

### Uncertainty quantification (UQ) for failure prediction.

In classification, MCP [17] is the vanilla method for UQ. However, by only considering the maximum over the predicted probabilities, MCP tends to overestimate confidence in failure cases [8]. Another common approach is to use the Shannon entropy of the predicted softmax distribution [38], but its invariance to label permutations [14, 38] limits its effectiveness for failure detection. Recently, Doctor [14] was proposed to refine the Shannon entropy [38] using Rényi entropy, while Rel-U [13] further extended it by incorporating a learned distance matrix to model class relationships from classifier predictions. Such methods, however, suffer from a limited expressiveness in their UQ models, and struggle to capture finer-grained uncertainties. Another line of work is to learn the classifier's loss using deep neural networks [8, 19, 47]. [47] estimates model loss to improve active learning, while [8] learns the cross-entropy loss for classification and segmentation. More recently, [19] applies this approach for large-scale UQ with vision transformers. However, these methods are limited to learning visual uncertainties (LVU), and generalize poorly to VLMs as they do not account for ambiguity in the language modality. To overcome this limitation, we propose an efficient UQ method for frozen, pre-trained VLMs and show that directly predicting failures outperforms loss prediction.

**VLMs' UQ.** MCP can be easily adapted for UQ of VLMs by leveraging their zero-shot probabilities, leading to the Maximum Concept Matching (MCM) method [30]. However, MCM inherits MCP's limitations for failure prediction.
tion, especially its overconfidence for errors. However, VLMs' overconfidence is generally less pronounced than for standard classifiers [29, 41]. As a result, when the VLM's downstream accuracy is sufficiently high, MCM remains a strong baseline for failure prediction. Several works extend VLMs for UQ in cross-modal retrieval tasks [5, 6, 26, 32], often by learning probabilistic embeddings for each modality. However, most require retraining both visual and textual backbones, limiting their practicality [5, 6]. To address this, ProbVLM [42] learns distribution parameters over embeddings via adapters, but overlooks cross-modal similarity scores, which limits its effectiveness for failure prediction. BayesVLM [2] applies a Laplace approximation to model uncertainty over similarities post-hoc. While both capture vision-language uncertainty, their objectives are not tailored for failure prediction, which limits their performance compared to our approach.

# 3. Background

## 3.1. Contrastive vision-language models

**Zero-shot predictions.** Consider a particular dataset $\mathcal{D} = \{(x_i, t_i)\}_{i=1}^N$, where each image $x_i \in \mathcal{X}$ is paired with a class name or caption $t_i \in \mathcal{T}$. Contrastive VLMs are composed of a pre-trained vision encoder $f_v(\cdot)$ that maps an input image $x_i$ to a visual embedding $z_{v_i} = f_v(x_i) \in \mathbb{R}^d$, and a text encoder $f_\tau(\cdot)$ that embeds a textual input $t_j$ into a representation $z_{t_j} = f_\tau(t_j) \in \mathbb{R}^d$. This multi-modal joint embedding space is learned during pre-training by aligning paired concepts. Thus, it enables zero-shot image classification, by computing the probability $p(t_j|x_i)$ of image $x_i$ being associated to caption $t_j$ using the softmax of the similarities between the visual representations $z_{v_i}$ and the embeddings $z_{t_j}$ of a set of K candidate textual concepts:

$$p(t_j|x_i) = \frac{\exp(z_{v_i}^\top z_{t_j}/\tau)}{\sum_k \exp(z_{v_i}^\top z_{t_k}/\tau)} \quad (1)$$

where $\tau$ is a temperature parameter optimized during pre-training, and $z_{v_i}$ and $z_{t_j}$ are $\ell_2$-normalized embeddings.

## 3.2. VLMs' failure prediction with MCM

We aim to determine whether VLMs can recognize when their predictions are unreliable. This is captured by an uncertainty scoring function, $u(x)$, where higher values indicate a greater likelihood of misclassification.

**Maximum Concept Matching (MCM)** [30] corresponds to the probability of the predicted caption for image $x_i$, i.e., the one with the highest probability:

$$u_{\text{MCM}}(x_i, t_1, \dots, t_k) = 1 - \max_j p(t_j|x_i). \quad (2)$$

**Limitations.** Despite being a reasonable UQ method in coarse-grained classification tasks, $u_{\text{MCM}}$ has fundamental limitations. Firstly, the max operation in Eq. (2) by

design assigns an overestimated UQ for incorrect predictions. Therefore, MCM performances drop when the zero-accuracy of the classifier is low, as shown in the experiments. Also, in fine-grained classification or open-vocabulary settings, visual and textual alignments become more dispersed, reducing MCM's reliability. This limitation is particularly problematic for modern VLMs, which are trained on large-scale datasets for open-vocabulary tasks.

# 4. ViLU model

This section presents our ViLU framework for multi-modal failure prediction on VLMs, as illustrated in Fig. 2. We first introduce our general post-hoc methodology in Sec. 4.1, which enables UQ on VLMs without modifying their internal parameters. We then describe our architecture in Sec. 4.2, detailing the design of ViLU's embedding using image-text cross-attention, and of our failure classification head. Finally, we outline ViLU's training procedure in Sec. 4.3, including our weighted cross-entropy loss that directly aligns with the failure prediction task.

## 4.1. Methodology for UQ on VLMs

**Post-hoc Setting.** We aim to design a discriminative and reliable UQ measure for pre-trained VLMs. To achieve this, we adopt a post-hoc approach, relying solely on visual and textual representations, i.e., $\mathcal{D} = \{(z_{v_i}, z_{t_i})\}_{i=1}^N$. This allows the UQ model to be easily integrated on top of a pre-trained VLM, providing uncertainty estimates without modifying internal representations, requiring fine-tuning, or depending on the loss function used during training.

**Learning vision-language uncertainties.** We propose leveraging the interactions between the visual modality and the set of candidate concepts to estimate uncertainty for failure prediction. Uncertainty in model predictions can arise from visual patterns (e.g., low image quality, ambiguous features) or textual patterns, which define concept distinctions. Additionally, it is shaped by cross-modal interactions. To model these interactions, we learn a global uncertainty representation $u_\theta(\cdot)$ that captures visual-textual interactions and their inherent uncertainty. We adopt a data-driven approach, training the uncertainty module to predict VLMs' misclassifications. Specifically, we frame our objective as a binary classification task, where $u_\theta$ predicts whether an input will be misclassified by the VLM (see Sec. 4.3). Formally, let us define our uncertainty module as:

$$\begin{aligned} u_\theta: \mathbb{R}^d \times \mathbb{R}^{K \times d} & \rightarrow [0, 1] \\ (z_v, z_{t_1}, \dots, z_{t_K}) & \rightarrow u_\theta(z_v, z_{t_1}, \dots, z_{t_K}) \end{aligned} \quad (3)$$

Note that our uncertainty function in Eq. (3) can handle varying values of K, as the number of candidate concepts may vary during inference.
<figure><img src="image_3.png" /><figcaption>Figure 2. Overview of ViLU: A learning-based Vision-Language Uncertainty quantification framework for VLM failure prediction. The key strength of ViLU lies in its ability to contextualize uncertainty estimates by leveraging all textual representations relevant to the task (see Sec. 4.1). It constructs an uncertainty-aware representation by combining the visual embedding $z_v$, the predicted textual embedding $z_t$, and an image-conditioned textual representation $z_t^\alpha$ obtained via cross-attention (see Sec. 4.2). Instead of relying on loss prediction, ViLU trains an uncertainty predictor as a binary classifier to distinguish between correct and incorrect predictions (see Sec. 4.3). ViLU is a post-hoc approach that can be efficiently deployed on top of any pre-trained VLM without accessing its weights (black box) and supports both image-caption and image-label tasks.</figcaption></figure>

## 4.2. ViLU's architecture

This section details the architectural components of our uncertainty module $u_\theta$ and how a visual feature, $z_v$, for a given sample and textual embeddings of K target concepts, $Z_t = \{z_{t_j}\}_{1 \le j \le K}$, are combined to provide an expressive representation for failure detection.

**Vision-textual cross-attention.** To enable flexible representations of the visual and textual modalities, we employ a cross-attention module $h_{\theta_{\text{XA}}}$ that produces an image-specific textual representation $z_t^\alpha$ allowing to efficiently capture inherent cross-modal uncertainty:

$$z_t^\alpha = h_{\theta_{\text{XA}}}(z_v, z_{t_1}, \dots, z_{t_K}) \quad (4)$$

More specifically, the query is the visual representation $z_v$, and keys and values are the K textual embeddings $Z_t$. The cross-attention's output is obtained as:

$$\alpha = \mathrm{softmax} \left( (W_Q z_v)^\top (W_K Z_t) / \sqrt{d} \right) \\ z_t^\alpha = \sum_{j=1}^K \alpha_j (W_V z_{t_j}) \quad (5)$$

where $\alpha$ are the attention weights, and $W_Q$, $W_K$, and $W_V$ denote the projection matrices for the queries, keys, and values, respectively. This weighted textual embedding refines the textual context based on the model's predicted distribution over the candidate captions, allowing for a more accurate uncertainty assessment. This contextualized representation can be computed for any number of concepts, ensuring a generic uncertainty module that remains well-defined across different concept sets.

**ViLU embeddings.** To resolve vision-language ambiguity for failure prediction, our model must capture key information to distinguish correct from incorrect predictions. At a minimum, this requires the visual representation $z_v$ and the textual embedding of the predicted caption $z_t$, allowing the model to approximate MCM's uncertainty estimator. However, this limited representation overlooks ambiguous alternatives, making error detection unreliable. To address this, we construct a rich vision-language uncertainty embedding $z_{\text{ViLU}} = (z_v, z_t, z_t^\alpha)$ by concatenating $z_v$, $z_t$, and the cross-attention output $z_t^\alpha$.

**Learning complex patterns for failure prediction.** Detecting misclassifications among numerous fine-grained concepts is challenging, as it requires capturing complex cross-modal relationships. To overcome this, we apply a non-linear transformation on our ViLU embeddings via a multi-layer perceptron (MLP), $g_{\theta_{\text{MLP}}}$, which enhances feature expressiveness and produces a scalar uncertainty estimate. Formally, uncertainty quantification is expressed as the predicted failure score $\hat{y}_i$:

$$\hat{y}_i = \sigma(g_{\theta_{\text{MLP}}}(z_{\text{ViLU}})), \quad (6)$$

where $\sigma$ denotes the sigmoid function, ensuring that $\hat{y}_i \to 1$ when the predicted zero-shot label is likely incorrect. Importantly, $g_{\theta_{\text{MLP}}}$ in Eq. (6) boils down to the unnormalized MCM score when using a bilinear form on $z_{\text{ViLU}}$ (see Appendix B). Our failure predictor is thus a consistent generalization of MCM, incorporating its prior and learning to refine it for finer multi-modal UQ.
## 4.3. Training procedure

ViLU accommodates both image-caption and image-label tasks during training and inference.

**1) Image-Label Tasks:** Image-label classification considers a predefined set of K target categories, leading to a batch-independent predictive pipeline. Here, the textual representations of categories are obtained using text templates (e.g., "A photo of a [CLASS]"), resulting in a fixed set of textual captions {$t_j$}$_{j\in\{1,...,K\}}$. The predicted concept for an image is then determined as:

$$ \hat{t}_i = \arg \max_{j \in \{1,...,K\}} p(t_j | x_i). \qquad (7) $$

This setting is batch-independent, making it suitable for standard classification datasets with predefined labels.

**2) Image-Caption Tasks:** When a set of textual descriptions is available from an open-vocabulary domain, the goal is to assign the most similar caption to a given input image. It is typically validated using batches of paired images and text descriptions. Given a batch B of images with associated text descriptions, {($x_i, t_i$)}$_{i\in B}$, the predicted concept for an image is determined as:

$$ \hat{t}_i = \arg \max_{j \in B} p(t_j | x_i). \qquad (8) $$

Here, the performance is batch-dependent, with larger batch sizes increasing task complexity.

**Training objective.** Our uncertainty module $u_\theta$ is trained as a binary classifier to predict VLM zero-shot misclassifications. The parameters $\theta = \{\theta_{\text{XA}}, \theta_{\text{MLP}}\}$ of our uncertainty model are trained by mini-batch gradient descent to minimize the following weighted binary cross-entropy loss:

$$ \mathcal{L}_{\text{wBCE}} = -\frac{1}{B} \sum_{i} [wy_i \log \hat{y}_i + (1 - y_i) \log(1 - \hat{y}_i)] \quad (9) $$

where $y_i = 1_{\{\hat{t}_i \neq t_i\}}$ is the target label, and w is a weighting factor that mitigates the potential class imbalance between correctly and incorrectly classified examples. This weight is dynamically adjusted based on the empirical classification accuracy of the VLM within each mini-batch:

$$ w = \log \left( 1 + \frac{\sum_{i=1}^{B} (1 - y_i)}{\sum_{i=1}^{B} y_i} \right) \qquad (10) $$

Previous UQ methods predict the model's training loss [8, 19, 47], assuming a high loss value indicates misclassification. Applying this approach to VLMs would require predicting CLIP's [36] contrastive loss or SigLIP's [50] sigmoid loss, which is impractical in a post-hoc setting where the pre-training loss is unknown. In contrast, ViLU is loss-agnostic, relying only on whether a prediction is correct. As shown in the experiments Tab. 4, training it as a binary classifier consistently outperforms loss prediction for VLMs, naturally aligning with standard error detection metrics.

## 5. Experiments

This section presents experimental results validating our multi-modal uncertainty model. We first focus on predicting VLM failures in zero-shot classification across two setups: i) standard image-label classification datasets and ii) large-scale image-caption datasets. Next, we experimentally analyze the different components of our model and training procedure, conducting various assessments to evaluate its behavior. We provide qualitative results and visualizations illustrating the ability of ViLU to quantify uncertainty.

### 5.1. Experimental setup

**Datasets.** We conduct a benchmark across 16 commonly used datasets to evaluate the ability of VLMs to detect misclassification in image classification. As outlined in Sec. 4.3, we consider two settings: i) image-label datasets and ii) image-caption datasets. Image-label datasets are standard datasets used in CLIP's transfer learning [22, 52], covering general object recognition, fine-grained classification, and specialized domains (see Appendix A.1). We use official or CoOp [53] data splits for evaluation. Image-caption datasets, including CC3M [39], CC12M [4], and LAION-400M [37], contain free-text descriptions for each sample. From these datasets, we randomly hold out 1% of the data for testing.

**Implementation details.** We use CLIP ViT-B/32 as the default backbone in all experiments. Full implementation details, along with additional results using other backbones, are provided in Appendix A.2 and C.4, respectively.

**Baselines.** We explore relevant baselines for uncertainty estimation in vision, as well as recent approaches for VLMs. Further details on these baselines and implementation specifics can be found in Appendix A.3. **1) Measures of output distribution:** MCM [30], described in Sec. 3, along with its calibrated variant [15], provide robust uncertainty estimates without additional training. Similarly, Entropy [38] and Doctor [14] are commonly used as baselines for uncertainty estimation. **2) Data-driven predictors:** We implement the recent Rel-U [13], which incorporates cross-label uncertainties in the logit space. However, since Rel-U relies on label-based information, it does not apply to image-caption datasets. Additionally, we assess methods that leverage embedding representations to learn patterns related to uncertainty. Specifically, we compare against baselines that predict classifier loss, such as ConfidNet [8] and other vision-only estimators [19, 47], collectively referred to as 'Learning Visual Uncertainty' (LVU). Finally, we evaluate the most recent post-hoc UQ probabilistic modeling approach designed for VLMs, BayesVLM [2].

**Metrics.** To assess the performance of our uncertainty model, we rely on two standard metrics: **1) False Positive Rate** at 95% True Positive Rate (FPR95) and **2) Area Under
<table><caption>Table 1. Misclassification detection on image-label datasets. The evaluation leverages each dataset's labeled classes as textual queries, which are fixed for each batch. Values below dataset names denote CLIP zero-shot accuracy on the respective dataset.</caption><thead><tr><th rowspan="3"></th><th colspan="2">CIFAR-10</th><th colspan="2">CIFAR-100</th><th colspan="2">Caltech101</th><th colspan="2">Flowers102</th><th colspan="2">OxfordPets</th><th colspan="2">Food101</th><th colspan="2">ImageNet-1k</th></tr><tr><th colspan="2">88.3%</th><th colspan="2">68.6%</th><th colspan="2">91.4%</th><th colspan="2">64.0%</th><th colspan="2">85.1%</th><th colspan="2">78.9%</th><th colspan="2">62.0%</th></tr><tr><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th></tr></thead><tbody><tr><td>MCM [30]</td><td>89.9</td><td>52.1</td><td>82.7</td><td>67.3</td><td>88.1</td><td>68.7</td><td>86.6</td><td>68.0</td><td>87.2</td><td>59.9</td><td>86.4</td><td>63.3</td><td>80.8</td><td>71.3</td></tr><tr><td>TS [15] + MCM [30]</td><td>89.9</td><td>51.5</td><td>83.9</td><td>68.4</td><td>90.4</td><td>55.7</td><td>86.9</td><td>66.0</td><td>89.1</td><td><u>55.5</u></td><td>86.9</td><td>62.8</td><td>80.7</td><td>71.5</td></tr><tr><td>Entropy [38]</td><td>88.7</td><td>59.9</td><td>79.8</td><td>71.9</td><td>86.1</td><td>78.8</td><td>85.5</td><td>65.0</td><td>88.0</td><td>60.0</td><td>86.1</td><td>65.0</td><td>78.3</td><td>76.8</td></tr><tr><td>Doctor [14]</td><td>89.5</td><td>56.5</td><td>82.3</td><td>69.7</td><td>88.7</td><td>66.5</td><td>86.7</td><td>63.9</td><td>88.9</td><td>56.6</td><td>86.8</td><td>63.4</td><td>80.3</td><td>72.9</td></tr><tr><td>Rel-U [13]</td><td>86.2</td><td>54.4</td><td>81.0</td><td>68.2</td><td>90.2</td><td>58.5</td><td>90.0</td><td>47.3</td><td>83.5</td><td>59.3</td><td>81.8</td><td>73.4</td><td>75.1</td><td>85.0</td></tr><tr><td>LVU [8, 19, 47]</td><td>96.6</td><td><u>21.2</u></td><td>80.3</td><td>68.5</td><td>89.8</td><td>50.9</td><td>90.5</td><td><u>38.3</u></td><td>84.1</td><td>55.7</td><td>82.7</td><td>69.9</td><td>78.7</td><td>77.0</td></tr><tr><td>BayesVLM [2]</td><td>92.6</td><td>44.9</td><td><u>87.0</u></td><td><u>60.3</u></td><td><u>94.0</u></td><td><u>37.4</u></td><td>87.3</td><td>62.4</td><td><u>89.5</u></td><td>60.3</td><td><u>87.8</u></td><td><u>60.3</u></td><td><u>81.5</u></td><td><u>70.3</u></td></tr><tr><td><strong>ViLU (Ours)</strong></td><td><strong>98.3</strong></td><td><strong>7.7</strong></td><td><strong>91.5</strong></td><td><strong>35.4</strong></td><td><strong>96.7</strong></td><td><strong>18.2</strong></td><td><strong>98.7</strong></td><td><strong>5.1</strong></td><td><strong>94.4</strong></td><td><strong>24.5</strong></td><td><strong>94.8</strong></td><td><strong>28.5</strong></td><td><strong>89.5</strong></td><td><strong>47.4</strong></td></tr></tbody></table><table><thead><tr><th rowspan="3"></th><th colspan="2">FGVCAircraft</th><th colspan="2">EuroSAT</th><th colspan="2">StanfordCars</th><th colspan="2">DTD</th><th colspan="2">SUN397</th><th colspan="2">UCF101</th><th colspan="2">Average</th></tr><tr><th colspan="2">18.1%</th><th colspan="2">35.8%</th><th colspan="2">60.1%</th><th colspan="2">43.0%</th><th colspan="2">62.1%</th><th colspan="2">61.6%</th><th colspan="2">62.7%</th></tr><tr><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th></tr></thead><tbody><tr><td>MCM [30]</td><td><u>75.7</u></td><td>82.9</td><td>64.1</td><td>87.6</td><td>81.4</td><td>73.4</td><td>77.4</td><td>77.9</td><td>78.8</td><td>75.9</td><td>84.1</td><td>68.9</td><td>81.8</td><td>70.6</td></tr><tr><td>TS [15] + MCM [30]</td><td>74.9</td><td>82.9</td><td>63.0</td><td>88.1</td><td>81.6</td><td>71.9</td><td>76.9</td><td>78.3</td><td>79.0</td><td>75.5</td><td>84.5</td><td>69.7</td><td>82.0</td><td>69.5</td></tr><tr><td>Entropy [38]</td><td>74.1</td><td>83.6</td><td>61.0</td><td>92.1</td><td>79.4</td><td>77.5</td><td>76.4</td><td>80.2</td><td>75.8</td><td>78.6</td><td>83.5</td><td>72.9</td><td>80.2</td><td>74.0</td></tr><tr><td>Doctor [14]</td><td>74.8</td><td>82.9</td><td>62.4</td><td>89.5</td><td>80.9</td><td>73.1</td><td>76.9</td><td>82.6</td><td>78.2</td><td>77.2</td><td>84.5</td><td>70.2</td><td>81.6</td><td>71.2</td></tr><tr><td>Rel-U [13]</td><td>68.6</td><td><u>82.5</u></td><td>76.3</td><td>72.1</td><td>75.5</td><td>78.9</td><td>81.4</td><td>69.7</td><td>75.2</td><td>81.3</td><td>84.1</td><td>61.1</td><td>80.7</td><td>68.6</td></tr><tr><td>LVU [8, 19, 47]</td><td>74.8</td><td>83.5</td><td><u>95.9</u></td><td><u>19.3</u></td><td>78.4</td><td>75.4</td><td><u>87.2</u></td><td><u>55.9</u></td><td>76.7</td><td>76.9</td><td><u>88.6</u></td><td><u>53.6</u></td><td><u>85.0</u></td><td><u>57.4</u></td></tr><tr><td>BayesVLM [2]</td><td>70.9</td><td>84.3</td><td>74.3</td><td>86.6</td><td><u>87.7</u></td><td><u>63.4</u></td><td>77.6</td><td>77.0</td><td><u>80.3</u></td><td><u>73.4</u></td><td>84.6</td><td>66.2</td><td>84.2</td><td>65.1</td></tr><tr><td><strong>ViLU (Ours)</strong></td><td><strong>81.0</strong></td><td><strong>71.7</strong></td><td><strong>98.8</strong></td><td><strong>4.4</strong></td><td><strong>90.1</strong></td><td><strong>46.8</strong></td><td><strong>93.8</strong></td><td><strong>28.8</strong></td><td><strong>88.6</strong></td><td><strong>50.3</strong></td><td><strong>95.9</strong></td><td><strong>20.9</strong></td><td><strong>93.2</strong></td><td><strong>29.9</strong></td></tr></tbody></table>

the receiver-operating characteristic Curve (AUC). These are commonly used in failure prediction to quantify the model's ability to detect incorrect classifications [8, 13, 14].

## 5.2. Main results

**Image classification datasets.** In Tab. 1, we evaluate misclassification detection on CLIP's zero-shot predictions across 13 standard image-label datasets. Notably, MCM emerges as a strong baseline in this setting, even outperforming more complex methods such as Doctor and Rel-U. This advantage arises because these methods rely on task-specific vision classifiers, which are generally less well-calibrated than large-scale pre-trained models. Consequently, MCM achieves superior performance without requiring post-processing steps, such as temperature scaling [15], for failure prediction. On the other hand, expressive data-driven methods such as LVU, BayesVLM, and our proposed ViLU enable more effective failure detection. Our proposed ViLU ranks first across all datasets and metrics. Its novel architectural design leverages the class-semantic information embedded in MCM while also capturing uncertainty patterns specific to each sample's ambiguities. As a clear demonstration of its effectiveness, ViLU achieves remarkable improvements in FPR95, surpassing MCM, BayesVLM, and LVU by margins of -40.7, -35.2, and -27.5, respectively (Tab. 1, Average column).

CLIP's zero-shot accuracy for each dataset is reported in Tab. 1. MCM and BayesVLM performances closely follow CLIP's accuracy, struggling in low zero-shot accu-

racy settings: they achieve only 64.1% and 74.3% AUC in EuroSAT, a dataset on which CLIP's accuracy is 35.8%. This limitation makes them unreliable when zero-shot accuracy is low – an unpredictable scenario in real-world settings. In contrast, ViLU remains effective across all accuracy regimes. See Appendix C for further analysis.

**Large-scale image-caption datasets.** We now evaluate the ability of state-of-the-art methods to detect failures in open-vocabulary settings, where tasks vary based on the inference batch and its specific captions. Tab. 2 presents results for applicable baselines in this challenging setting, along with the proposed ViLU. Unlike in previous experiments on image-label datasets, LVU fails to outperform even the MCM baseline, underscoring the importance of considering target objectives alongside sample-related uncertainties in open-vocabulary scenarios. As a result, methods specifically designed for vision-language models, such as BayesVLM, achieve superior performance across all three evaluated datasets. Our proposed ViLU achieves the best performance, consistently surpassing BayesVLM with FPR95 improvements of -21.5, -28.1, and -5.2 on CC3M, CC12M, and LAION-400M, respectively. This advantage stems from our explicit modeling of misclassification errors, whereas BayesVLM focuses on the uncertainty of similarities between individual embeddings, which is more implicit.
<table><caption>**Table 2. Misclassification detection on image-caption datasets.** The evaluation uses the captions of each randomly-retrieved batch as textual queries. Hence, the textual queries vary for each batch. Results reported with a batch size of 1024 samples for inference.</caption><thead><tr><th rowspan="2"></th><th colspan="2">CC3M</th><th colspan="2">CC12M</th><th colspan="2">LAION-400M</th></tr><tr><th colspan="2">58.8%</th><th colspan="2">73.5%</th><th colspan="2">90.5%</th></tr><tr><th></th><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th></tr></thead><tbody><tr><th>MCM [30]</th><td>83.9</td><td>69.0</td><td>88.8</td><td>58.8</td><td>91.7</td><td>50.2</td></tr><tr><th>Entropy [38]</th><td>82.5</td><td>73.3</td><td>87.7</td><td>63.0</td><td>89.4</td><td>62.5</td></tr><tr><th>Doctor [14]</th><td>83.7</td><td>70.1</td><td>88.6</td><td>59.9</td><td>91.2</td><td>54.5</td></tr><tr><th>LVU [8, 19, 47]</th><td>69.3</td><td>82.5</td><td>74.4</td><td>76.5</td><td>80.2</td><td>72.3</td></tr><tr><th>BayesVLM [2]</th><td><u>87.1</u></td><td><u>62.6</u></td><td><u>90.9</u></td><td><u>53.3</u></td><td><u>95.1</u></td><td><u>26.4</u></td></tr><tr><th><strong>ViLU (Ours)</strong></th><td><strong>91.4</strong></td><td><strong>41.1</strong></td><td><strong>95.2</strong></td><td><strong>25.2</strong></td><td><strong>97.3</strong></td><td><strong>21.2</strong></td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>GT: "container ship"
Pred: "ocean liner"</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>GT: "-mailbox"
Pred: "birdhouse"</figcaption></figure>

<caption>**Figure 3. Two qualitative examples.** Misclassifications detected by ViLU, but not MCM [30] and visual-only baselines [8, 19].</caption>

## 5.3. Ablation studies

**Architectural design of ViLU.** Tab. 3 highlights the contributions of different components in our model, namely the cross-attention mechanism and the inclusion of the predicted caption as input. As observed in [8, 19], using **only visual information** (first row) is effective on CIFAR-10 but struggles on larger datasets with fine-grained or semantically similar concepts. For instance, on ImageNet-1k, visual information alone fails to outperform MCM. Incorporating the **predicted class textual embedding** (second row) significantly improves performance across all datasets, yielding a +10 AUC gain on ImageNet-1k and +14 AUC on CC12M. This additional input helps the model to handle class ambiguity, which is crucial for datasets where categories are easily confused. For example, as shown in Fig. 3, the class container ship is often misclassified as ocean liner, another type of boat. Finally, the **cross-attention module** (third row) enables the model to integrate contextual information from all candidate classes or captions. By re-contextualizing predictions among available textual inputs, this mechanism proves particularly beneficial for CC12M, where captions change dynamically across batches. Unlike CIFAR-10 and ImageNet-1k, where class sets are fixed, omitting cross-attention in CC12M results in an AUC plateau at 88.9, only slightly above MCM. Incorporating this mechanism enhances the model's ability to detect ambiguities and assess potential errors.

<table><caption>**Table 3. Ablation on different components of ViLU.**</caption><thead><tr><th rowspan="2">Visual embed.</th><th rowspan="2">Cross attention</th><th rowspan="2">Predicted caption</th><th colspan="2">CIFAR-10</th><th colspan="2">ImageNet-1k</th><th colspan="2">CC12M</th></tr><tr><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th></tr></thead><tbody><tr><td>✓</td><td>✗</td><td>✗</td><td>96.4</td><td>21.8</td><td>78.7</td><td>77.0</td><td>74.0</td><td>76.5</td></tr><tr><td>✓</td><td>✗</td><td>✓</td><td>97.9</td><td>10.8</td><td>88.8</td><td>50.1</td><td>88.9</td><td>48.9</td></tr><tr><td>✓</td><td>✓</td><td>✗</td><td>97.7</td><td>11.4</td><td>86.1</td><td>63.5</td><td>93.6</td><td>37.0</td></tr><tr><td>✓</td><td>✓</td><td>✓</td><td><strong>98.3</strong></td><td><strong>8.2</strong></td><td><strong>89.5</strong></td><td><strong>47.4</strong></td><td><strong>95.2</strong></td><td><strong>25.2</strong></td></tr></tbody><tfoot><tr><td colspan="4">MCM [30]</td><td>89.9</td><td>52.1</td><td>80.8</td><td>71.3</td><td>88.8</td><td>58.8</td></tr></tfoot></table>

**Loss function design for failure prediction.** In Sec. 4.3,

we discussed our optimization target and choice of loss function. Unlike prior works [8, 19, 47], which treat the problem as a regression task by directly predicting the test-time loss and optimizing it with MSE, we instead frame it as a binary classification task, distinguishing between errors and correct predictions. As shown in Tab. 4, this approach consistently outperforms MSE-based loss approximations, yielding a +3 AUC improvement on ImageNet-1k while reducing FPR95 by 16 points. Additionally, our automatic weighting of the two classes (misclassified and correctly classified) further enhances the performance, resulting in a +1 AUC gain on ImageNet-1k and a 2-point reduction in FPR95 on CIFAR-10.

<table><caption>**Table 4. Role of the proposed weighted loss.** Effect of loss function choice and adaptive weighting in Eq. (10) for failure detection.</caption><thead><tr><th rowspan="2"></th><th colspan="2">CIFAR-10</th><th colspan="2">ImageNet-1k</th></tr><tr><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th></tr></thead><tbody><tr><th>MSE [8, 19]</th><td>95.1</td><td>30.4</td><td>84.8</td><td>64.4</td></tr><tr><th>w/ Weighting</th><td>95.6</td><td>29.6</td><td>85.7</td><td>63.2</td></tr><tr><th>BCE</th><td>97.7</td><td>10.5</td><td>88.6</td><td>48.4</td></tr><tr><th>w/ Weighting (Ours)</th><td><strong>98.3</strong></td><td><strong>8.2</strong></td><td><strong>89.5</strong></td><td><strong>47.4</strong></td></tr></tbody></table>

## 5.4. In-depth analysis

**Classification complexity on image-caption datasets.** Fig. 4 explores the performance of ViLU in challenging image-caption classification tasks, where difficulty is determined by the number of concepts to be distinguished simultaneously, i.e., batch size used for inference. Naturally, increasing the batch size makes the classification more complex, making errors harder to detect for methods that rely on semantic relationships among query concepts, such as ViLU or MCM. However, ViLU consistently outperforms MCM across all batch sizes. Notably, this configuration does not affect prior LVU methods, which estimate vision-only uncertainty. However, they fall short in performance, particularly compared to the proposed ViLU.

**Requirements on training data.** As a data-driven uncertainty quantifier, the proposed ViLU requires a subset of image-caption or image-label examples. Fig. 5 illustrates
<figure><img src="image_6.png" /><figcaption>Figure 4. Robustness to image-caption task complexity. Inference batch size effect in failure detection for ViLU (CC12M).</figcaption></figure>

the data-efficiency of the proposed approach for uncertainty quantification. The results showcase the efficiency of ViLU, which *requires only a small amount of data to surpass the strong baseline MCM*, e.g., 2.5% for ImageNet and even less for specialized datasets such as EuroSAT.

<figure><img src="image_7.png" /><figcaption>Figure 5. Data-efficiency. Performance, in terms of FPR95↓, w.r.t. the available data for training ViLU.</figcaption></figure>

**Cross-datasets generalization.** To evaluate ViLU’s robustness to dataset shifts and its ability to generalize without per-dataset tuning, we report in Tab. 5 its zero-shot transfer performance when pre-trained on the large-scale CC12M dataset. ViLU consistently outperforms MCM across all 12 datasets in this pure transfer setting. The table also highlights ViLU’s clear advantage over LVU (also trained on CC12M), underscoring the benefit of explicitly modeling vision-language uncertainty for zero-shot generalization. While these results demonstrate strong transfer capabilities, generalization can still be improved. In Appendix C.6, we explore how smarter batch sampling strategies aimed at improving concept coverage during pre-training could further enhance performance.

**Qualitative assessment.** In Fig. 3, we present two qualitative examples from ImageNet where the VLM misclassified the input images. On the left, the model predicted *ocean liner* instead of *container ship*, likely due to their shared visual features as large vessels. On the right, a * mailbox* was misclassified as a * birdhouse*, possibly influenced by the surrounding vegetation and the mailbox’s opening, which resemble typical birdhouse features. ViLU effectively captures both sources of ambiguity: semantic

<table><caption>Table 5. FPR95↓ across datasets. Zero-shot performance when pre-trained on CC12M.</caption><thead><tr><th>Dataset</th><th>MCM</th><th>LVU</th><th>ViLU</th></tr></thead><tbody><tr><td>CIFAR-10</td><td><strong>52.1</strong></td><td>77.2</td><td>54.2</td></tr><tr><td>CIFAR-100</td><td>67.3</td><td>83.8</td><td><strong>59.9</strong></td></tr><tr><td>Caltech101</td><td>68.7</td><td>82.5</td><td><strong>48.8</strong></td></tr><tr><td>Flowers102</td><td>68.0</td><td>96.8</td><td><strong>67.4</strong></td></tr><tr><td>OxfordPets</td><td>59.9</td><td>93.1</td><td><strong>58.1</strong></td></tr><tr><td>Food101</td><td><strong>63.3</strong></td><td>87.2</td><td>67.4</td></tr><tr><td>FGVCAircraft</td><td>82.9</td><td>94.5</td><td><strong>82.3</strong></td></tr><tr><td>EuroSAT</td><td>87.6</td><td>88.2</td><td><strong>85.7</strong></td></tr><tr><td>DTD</td><td><strong>77.9</strong></td><td>93.1</td><td>78.2</td></tr><tr><td>SUN397</td><td>75.9</td><td>90.1</td><td><strong>72.7</strong></td></tr><tr><td>StanfordCars</td><td><strong>73.4</strong></td><td>92.6</td><td>84.1</td></tr><tr><td>UCF101</td><td>68.9</td><td>90.4</td><td><strong>63.8</strong></td></tr><tr><td><strong>Average</strong></td><td>70.5</td><td>89.1</td><td><strong>68.6</strong></td></tr></tbody></table>

confusion between visually similar classes and misinterpretations driven by contextual cues.

**Uncertainty distribution score.** Fig. 6 illustrates the effectiveness of different uncertainty estimation methods in distinguishing between correctly and incorrectly classified samples on ImageNet. In MCM and LVU, the uncertainty distributions for success and failure overlap significantly. In contrast, ViLU produces a more distinct separation. Indeed, misclassified samples receive high uncertainty (peaking near 1.0), while correct predictions have low uncertainty (peaking near 0.0). Its higher density at the extremes indicates a well-calibrated uncertainty measure.

<figure><img src="image_8.png" /><figcaption>Figure 6. Uncertainty score distribution. Predictions for correctly and incorrectly classified samples on ImageNet.</figcaption></figure>

## 6. Conclusion

We have presented our ViLU method, a new uncertainty quantification approach for failure prediction of pre-trained VLMs. ViLU learns an appropriate uncertainty embedding space including fine interactions between visual and concepts ambiguities to learn an effective binary failure predictor on the downstream task. Extensive experiments on several datasets show the significant gain of our method compared to state-of-the-art UQ methods for failure prediction. In addition, ablation studies clearly validate our architectural choices and training design. We also show that ViLU remains effective even in the low-performance regime of VLMs. Future works include adjusting ViLU in the context of domain adaptation and test-time adaptation of VLMs.
Acknowledgment

This work was supported by the French National Research Agency (Agence Nationale de la Recherche, ANR) under grants from project DIAMELEX (ANR-20-CE45-0026) and project RODEO (ANR-24-CE23-5886). It was granted access to the HPC resources of IDRIS under the allocation AD011013370R2 made by GENCI.

References

[1] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. A CLIP-hitchhiker's guide to long video retrieval. *arXiv preprint arXiv:2205.08508*, 2022. 2

[2] Anton Baumann, Rui Li, Marcus Klasson, Santeri Mentu, Shyamgopal Karthik, Zeynep Akata, Arno Solin, and Martin Trapp. Post-hoc probabilistic vision-language models. *arXiv preprint arXiv:2412.06014*, 2024. 1, 2, 3, 5, 6, 7, 13

[3] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. In *European Conference on Computer Vision (ECCV)*, 2014. 12

[4] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In *IEEE Conference on Computer Vision and Pattern Recognition, CVPR* 2021, virtual, June 19-25, 2021, pages 3558–3568. Computer Vision Foundation / IEEE, 2021. 5

[5] Sanghyuk Chun. Improved probabilistic image-text representations. In *The Twelfth International Conference on Learning Representations*, 2024. 3

[6] Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 8415–8424, 2021. 3

[7] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 3606–3613, 2014. 12

[8] Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure prediction by learning model confidence. *Advances in Neural Information Processing Systems*, 32, 2019. 1, 2, 5, 6, 7, 12, 13, 14, 15

[9] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. *arXiv preprint arXiv:2409.17146*, 2024. 1, 2

[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 248–255, 2009. 12

[11] Li Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental

bayesian approach tested on 101 object categories. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)*, pages 178–178, 2004. 12

[12] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: representing model uncertainty in deep learning. In *Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48*, page 1050–1059. JMLR.org, 2016. 1

[13] Eduardo Dadalto Câmara Gomes, Marco Romanelli, Georg Pichler, and Pablo Piantanida. A data-driven measure of relative uncertainty for misclassification detection. In *The Twelfth International Conference on Learning Representations*, 2024. 2, 5, 6, 12

[14] Federica Granese, Marco Romanelli, Daniele Gorla, Catuscia Palamidessi, and Pablo Piantanida. Doctor: A simple method for detecting misclassification errors. *Advances in Neural Information Processing Systems*, 34:5669–5681, 2021. 2, 5, 6, 7, 12

[15] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In *International Conference on Machine Learning (ICML)*, pages 1321–1330, 2017. 5, 6, 12

[16] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. In *IEEE International Geoscience and Remote Sensing Symposium (IGARSS)*, pages 3606–3613, 2018. 12

[17] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In *The Fifth International Conference on Learning Representations*. OpenReview.net, 2017. 1, 2

[18] Siteng Huang, Biao Gong, Yulin Pan, Jianwen Jiang, Yiliang Lv, Yuyuan Li, and Donglin Wang. Vop: Text-video cooperative prompt tuning for cross-modal retrieval. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)*, pages 6565–6574, 2023. 2

[19] Michael Kirchhof, Mark Collier, Seong Joon Oh, and Enkelejda Kasneci. Pretrained visual uncertainties. *arXiv preprint arXiv:2402.16569*, 2024. 1, 2, 5, 6, 7, 12, 13, 14

[20] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, page 3498–3505, 2012. 12

[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. (2009). 2009. 12

[22] Marc Lafon, Elias Ramzi, Clément Rambour, Nicolas Audebert, and Nicolas Thome. Gallop: Learning global and local prompts for vision-language models. In *European Conference on Computer Vision*, pages 264–282. Springer, 2024. 2, 5

[23] Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. In *European Conference on Computer Vision*, pages 70–88. Springer, 2024. 2
[24] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Sid-dharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. *Advances in Neural Information Processing Systems*, 36, 2024. 2

[25] Hugo Laurençon, Leo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024. 1, 2

[26] Hao Li, Jingkuan Song, Lianli Gao, Pengpeng Zeng, Haonan Zhang, and Gongfu Li. A differentiable semantic metric approximation in probabilistic embedding for cross-modal retrieval. In *Advances in Neural Information Processing Systems*, 2022. 3

[27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In *International conference on machine learning*, pages 19730–19742. PMLR, 2023. 1, 2

[28] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. In *ArXiv Preprint*, 2013. 12

[29] Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. *Advances in neural information processing systems*, 34:15682–15694, 2021. 3

[30] Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of-distribution detection with vision-language representations. *Advances in neural information processing systems*, 35:35087–35102, 2022. 1, 2, 3, 5, 6, 7, 12, 13, 14

[31] Balamurali Murugesan, Julio Silva-Rodríguez, Ismail Ben Ayed, and Jose Dolz. Robust calibration of large vision-language adapters. In *European Conference on Computer Vision*, pages 147–165, 2024. 1

[32] Andrei Neculai, Yanbei Chen, and Zeynep Akata. Probabilistic compositional embeddings for multimodal image retrieval. *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)*, pages 4546–4556, 2022. 3

[33] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In *Indian Conference on Computer Vision, Graphics and Image Processing*, 2008. 12

[34] Changdae Oh, Hyesu Lim, Mijoo Kim, Dongyoon Han, Sangdoo Yun, Jaegul Choo, Alexander Hauptmann, Zhi-Qi Cheng, and Kyungwoo Song. Towards calibrated robust fine-tuning of vision-language models. *Advances in Neural Information Processing Systems*, 37:12677–12707, 2024. 1

[35] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, page 3498–3505, 2012. 12

[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In *International conference on machine learning*, pages 8748–8763. PMLR, 2021. 1, 2, 5, 14

[37] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: open dataset of clip-filtered 400 million image-text pairs. CoRR, abs/2111.02114, 2021. 1, 2, 5

[38] Murat Sensoy, Lance M. Kaplan, and Melih Kandemir. Ev-idential deep learning to quantify classification uncertainty. In *Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada*, pages 3183–3193, 2018. 2, 5, 6, 7, 12

[39] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers*, pages 2556–2565. Association for Computational Linguistics, 2018. 5

[40] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. In *ArXiv Preprint*, 2012. 12

[41] Weijie Tu, Weijian Deng, and Tom Gedeon. A closer look at the robustness of contrastive language-image pre-training (clip). *Advances in Neural Information Processing Systems*, 36:13678–13691, 2023. 3

[42] Uddeshya Upadhyay, Shyamgopal Karthik, Massimiliano Mancini, and Zeynep Akata. Probvlm: Probabilistic adapter for frozen vision-language models. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 1899–1910, 2023. 1, 2, 3, 12

[43] Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. In *European Conference on Computer Vision*, pages 315–332. Springer, 2024. 2

[44] Shuoyuan Wang, Jindong Wang, Guoqing Wang, Bob Zhang, Kaiyang Zhou, and Hongxin Wei. Open-vocabulary calibration for fine-tuned CLIP. In *Proceedings of the 41st International Conference on Machine Learning*, pages 51734–51754, 2024. 1

[45] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 3485–3492, 2010. 12

[46] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey, 2024. 1

[47] Donggeun Yoo and In So Kweon. Learning loss for active learning. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 93–102, 2019. 1, 2, 5, 6, 7, 12, 13, 14

[48] Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark A Hasegawa-Johnson, Yingzhen Li, and Chang D Yoo. C-TPT: A transferable visual language model for action detection. In *arXiv preprint*, 2021. 12
Calibrated test-time prompt tuning for vision-language models via text feature dispersion. In *The Twelfth International Conference on Learning Representations*, 2024. 1

[49] Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. *Advances in Neural Information Processing Systems*, 36:32215–32234, 2023. 2

[50] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 11975–11986, 2023. 1, 2, 5, 12, 14

[51] Zhuowen Tu Zheng Ding, Jieke Wang. Open-vocabulary universal image segmentation with maskclip. In *International Conference on Machine Learning*, 2023. 2

[52] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)*, pages 16816–16825, 2022. 2, 5

[53] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. *International Journal of Computer Vision*, 130(9):2337–2348, 2022. 5

[54] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. *International Journal of Computer Vision*, 130(9):2337–2348, 2022. 2
# ViLU: Learning Vision-Language Uncertainties for Failure Prediction

## Supplementary Material

### A. Additional details on experimental setup

#### A.1. Datasets

In this section, we provide additional details on the image-label datasets used in the main experiments presented in the paper. These span general object recognition, fine-grained classification, and specialized domains. The datasets include ImageNet-1k [10], CIFAR-10 [21], CIFAR-100 [21], SUN397 [45], FGVCAircraft [28], EuroSAT [16], StanfordCars [20], Food101 [3], Oxford-Pets [35], Flowers102 [33], Caltech101 [11], DTD [7], and UCF101 [40].

#### A.2. Implementation details

As mentioned in the main paper, we use CLIP ViT-B/32 backbone in all our experiments. We provide additional results using other CLIP backbones and architectures, such as SigLIP [50] in Appendix C.4. Regarding the proposed ViLU, the MLP layer for misclassification prediction follows a four-layer architecture with dimensions [512, 256, 128, 1] and ReLU activations. Training is performed using SGD as the optimizer, with the cross-attention layers remaining frozen during the first epoch. We select the learning rate through a grid search over {10⁻¹, 10⁻², 10⁻³} and explore batch sizes among {128, 256, 512, 1024}.

#### A.3. Baselines & Implementation

* **MCM [30]:** Maximum Concept Matching (MCM) estimates uncertainty in VLMs by leveraging the softmax probability distribution over all classes or captions. It selects the most likely caption for an image based on the highest probability score, providing a natural measure of confidence in the model's predictions. No additional training is required since MCM directly uses the model's output probabilities.

* **MCM + TS [15]:** This method extends MCM by applying Temperature Scaling (TS) to adjust the softmax probabilities better. TS optimizes the temperature parameter to refine the confidence scores, leading to more calibrated uncertainty estimates. The multiplicative temperature parameter is learned using the whole training dataset to minimize expected calibration error, using LBFGS optimizer.

* **Entropy [38]:** This method quantifies uncertainty in neural network predictions by calculating the Shannon entropy of the output probability distribution. High entropy indicates more significant uncertainty, as the model assigns similar probabilities across multiple classes, reflecting ambiguity in its prediction. On the other hand, low entropy signifies confidence, with the model favoring a

specific class. Entropy-based uncertainty estimation does not require additional training.

* **DOCTOR [14]:** DOCTOR quantifies uncertainty by analyzing the confidence distribution of the model's predictions. It computes the Rényi entropy of order two, a measure based on the squared probabilities assigned to each class, emphasizing how concentrated or dispersed the probability mass is. A prediction with one dominant probability value will yield a low uncertainty score, while a more evenly spread distribution results in higher uncertainty. This method does not require additional training and operates directly on the model's softmax outputs.

* **Rel-U [13]:** Rel-U is a data-driven method that incorporates cross-label uncertainties directly in the logit space. Learning relationships between class logits provides a refined estimation of uncertainty beyond traditional confidence scores. Due to its reliance on a cross-label cost penalty matrix, Rel-U does not apply to image-text datasets where labels are absent. Rel-U's hyper-parameters are fixed to $\lambda = 0.15$ and $T = 0.5$ greedily, since they provided the best performance.

* **Learning Visual Uncertainties (LVU) [8, 19, 47]:** LVU refers to a class of models designed to predict the loss of a visual backbone as a means to estimate potential errors. ConfidNet [8] established that accurately predicting uncertainty is equivalent to estimating the model's loss—if a model can predict the loss of its visual backbone, it inherently quantifies its error. Another approach, Pretrained Visual Uncertainties [19], follows a similar principle by learning to predict backbone loss, leveraging pretraining on ImageNet-21k.

**Implementation:** To evaluate the LVU baseline, we use the same MLP architecture as our model but restrict the input to the visual token only. Additionally, following [8, 19], this baseline is trained with an MSE loss, in contrast to our method, which uses a BCE loss.

* **ProbVLM [42]:** ProbVLM introduces a probabilistic adapter that estimates probability distributions for embeddings of pre-trained VLMs. This is achieved through inter- and intra-modal alignment in a post-hoc manner. The goal is to capture the inherent ambiguity in embeddings, reflecting the fact that multiple samples can represent the same concept in the physical world. This method enhances the calibration of embedding uncertainties in retrieval tasks and benefits downstream applications like active learning and model selection.

**Implementation:** ProbVLM models probability distributions over the embeddings of image and text modal-
ities. However, it does not explicitly model the uncertainty in their interaction via cosine similarity. As a result, directly adapting the method for image classification is not straightforward. We attempted to include ProbVLM in our baseline comparison by using its proposed visual aleatoric uncertainty metric, but it resulted in nearly random failure prediction performance. Additionally, we explored using its cross-modal loss as an uncertainty logit, applying a softmax transformation, but this approach also proved ineffective. In contrast, BayesVLM addresses this limitation by modeling the uncertainty over the similarity computation, enabling a more principled approach to downstream tasks like image classification.

* **BayesVLM [2]:** BayesVLM is a training-free method for estimating predictive uncertainty. It employs a post-hoc approximation of the Bayesian posterior, allowing for analytic computation of uncertainty propagation through the VLM. By approximating the Bayesian posterior over model parameters, BayesVLM captures uncertainties inherent to the model itself (image and text encoders). These model uncertainties are then propagated through the VLM to produce uncertainty estimates for predictions. **Implementation:** To evaluate BayesVLM, we follow the implementation provided in its official Github repository https://github.com/AaltoML/BayesVLM

# B. Additional details on ViLU

## B.1. Bilinear interpretation of MCM

In Sec. 4.2 of the main paper, we mentioned that ViLU is a consistent generalization of MCM. More precisely, the uncertainty module $g_\theta$ can model the unnormalized MCM score by approximating the following bilinear form on $z_{\text{ViLU}} = (z_v, z_t, z_t^\alpha)$:

$$g_\theta(z_{\text{ViLU}}) = \frac{1}{2} z_{\text{ViLU}}^T A z_{\text{ViLU}} = z_v^\top z_t, \quad (11)$$

$$\text{with } A = \begin{pmatrix} 0 & I_d & 0 \\ I_d & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix} \in \mathbb{R}^{3d \times 3d}.$$

## B.2. ViLU variant for generalization experiments

For the generalization experiments presented in the main paper (cross-dataset transfer) and the supplementary material (domain generalization and concept coverage), we used a slightly modified version of ViLU. Specifically, the MCM score was explicitly provided as an additional input to the uncertainty module $g_\theta$, alongside the visual and textual embeddings. While the original design of ViLU allows $g_\theta$ to model this behavior implicitly through interactions between the modalities, we found that explicitly including the MCM score improves uncertainty generalization.

# C. Additional experimental results

## C.1. Impact of MLP depth on performance

The results in Fig. 7 show that ViLU is relatively robust to MLP depth variations, particularly on ImageNet, where performance remains stable across different configurations. Across all tested datasets, a depth of 4 layers consistently achieved strong results, suggesting that this architecture provides a good balance between expressiveness and generalization for failure prediction.

<figure><img src="image_9.png" /><figcaption>Figure 7. **Impact of MLP depth.** Performance of ViLU on ImageNet and CIFAR-10 for different MLP depths.</figcaption></figure>

## C.2. Robustness to image-text task complexity

We analyze in Tab. 6 how inference-time batch size affects failure detection performance for MCM, LVU, and ViLU on the CC12M dataset. As batch size increases, the number of candidate captions used during inference grows, introducing more semantic competition and making the task more complex. Despite this, ViLU consistently outperforms both MCM and LVU across all tested settings. Notably, even under very large batch sizes—16,384 and 32,768—ViLU maintains strong performance, with only moderate degradation in AUC and FPR95. These results confirm the robustness of our method to increased image-text ambiguity at test time.

<table><caption>Table 6. Numerical results corresponding to Fig. 4, showing the effect of inference batch size on failure detection for ViLU (on CC12M).</caption><thead><tr><th rowspan="2">Batch Size</th><th colspan="2">MCM [30]</th><th colspan="2">LVU [8, 19, 47]</th><th colspan="2">ViLU</th></tr><tr><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th></tr></thead><tbody><tr><td>128</td><td>92.7</td><td>15.6</td><td>75.2</td><td>76.2</td><td><strong>96.9</strong></td><td><strong>15.6</strong></td></tr><tr><td>512</td><td>90.1</td><td>54.6</td><td>74.8</td><td>76.5</td><td><strong>95.7</strong></td><td><strong>22.5</strong></td></tr><tr><td>1024</td><td>88.8</td><td>58.8</td><td>74.4</td><td>76.5</td><td><strong>95.2</strong></td><td><strong>25.2</strong></td></tr><tr><td>2048</td><td>87.5</td><td>61.5</td><td>74.3</td><td>76.8</td><td><strong>94.4</strong></td><td><strong>28.7</strong></td></tr><tr><td>4096</td><td>86.4</td><td>64.2</td><td>73.9</td><td>77.0</td><td><strong>93.6</strong></td><td><strong>31.8</strong></td></tr><tr><td>8192</td><td>85.3</td><td>65.3</td><td>73.6</td><td>77.0</td><td><strong>92.8</strong></td><td><strong>34.9</strong></td></tr><tr><td>16384</td><td>84.5</td><td>66.4</td><td>73.3</td><td>76.7</td><td><strong>91.9</strong></td><td><strong>37.8</strong></td></tr><tr><td>32768</td><td>83.7</td><td>67.0</td><td>73.2</td><td>76.6</td><td><strong>91.1</strong></td><td><strong>39.9</strong></td></tr></tbody></table>
### C.3. Reliability of misclassification detection

Fig. 8 illustrates the relationship between misclassification detection performance and the zero-shot accuracy of the vision-language model for each baseline. Each dot at a given x-coordinate represents the classification performance of different baselines on the same dataset. The results emphasize the superior reliability of the uncertainty estimates provided by our method, particularly in low zero-shot accuracy settings. Notably, the tendency curves indicate a strong correlation between model performance and uncertainty metrics for both MCM and BayesVLM. Specifically, as zero-shot accuracy decreases, these two methods exhibit the worst performance. This suggests that they are only reliable when the model’s zero-shot accuracy is high—an unpredictable scenario in real-world settings, where ground-truth labels are unavailable.

<figure><img src="image_10.png" /><figcaption>Figure 8. Reliability of misclassification detection. Our method, ViLU, enhances misclassification detection by providing more reliable uncertainty estimates, particularly when zero-shot accuracy is low.</figcaption></figure>

### C.4. Extension to different VLMs.

Tab. 7 presents the performance of ViLU when applied to different zero-shot vision-language backbones, including CLIP [36] and SigLIP [50], with both ViT-B and ViT-L variants. Across all settings, ViLU consistently outperforms MCM by a large margin in both AUC and FPR95, demonstrating strong and reliable failure detection. On CIFAR-10, the improvements are particularly pronounced: for example, using CLIP ViT-L/14, ViLU achieves an AUC of 99.0 compared to 93.6 for MCM, and reduces FPR95 from 31.5 to just 4.1. On ImageNet-1k, the gains remain substantial, with up to 30-point reductions in FPR95. Unlike LVU-based methods [8, 19, 47], which require access to the model’s pre-training loss, ViLU is trained solely from classification correctness, making it applicable to a broad range of frozen or proprietary VLMs. Overall, the consistent results across architectures confirm that ViLU generalizes effectively with minimal assumptions.

<table><caption>Table 7. Generalization across backbones. ViLU shows consistent performance gains on several VLMs compared to MCM.</caption><thead><tr><th rowspan="2">Backbone</th><th rowspan="2">Method</th><th colspan="2">CIFAR-10</th><th colspan="2">ImageNet-1k</th></tr><tr><th>AUC↑</th><th>FPR95↓</th><th>AUC↑</th><th>FPR95↓</th></tr></thead><tbody><tr><td rowspan="2">CLIP [36]</td><td rowspan="2">ViT-B/16</td><td>MCM [30]</td><td>90.9</td><td>47.3</td><td>81.0</td><td>73.0</td></tr><tr><td>ViLU</td><td><b>98.4</b></td><td><b>8.0</b></td><td><b>90.3</b></td><td><b>44.2</b></td></tr><tr><td rowspan="2">ViT-L/14</td><td rowspan="2">MCM [30]</td><td>93.6</td><td>31.5</td><td>82.9</td><td>68.9</td></tr><tr><td>ViLU</td><td><b>99.0</b></td><td><b>4.1</b></td><td><b>91.2</b></td><td><b>39.2</b></td></tr><tr><td rowspan="2">SigLIP [50]</td><td rowspan="2">ViT-B/16</td><td>MCM [30]</td><td>92.8</td><td>46.1</td><td>84.2</td><td>65.8</td></tr><tr><td>ViLU</td><td><b>97.6</b></td><td><b>13.3</b></td><td><b>90.7</b></td><td><b>44.4</b></td></tr><tr><td rowspan="2">ViT-L/16</td><td rowspan="2">MCM [30]</td><td>95.6</td><td>29.1</td><td>86.8</td><td>64.1</td></tr><tr><td>ViLU</td><td><b>98.4</b></td><td><b>7.8</b></td><td><b>91.3</b></td><td><b>41.4</b></td></tr></tbody></table>

### C.5. Domain generalization on ImageNet variants

To evaluate ViLU’s robustness under distribution shift, we consider a domain generalization setup in which ViLU is trained on the original ImageNet dataset and evaluated on two domain-shifted variants: ImageNet-V2 (IN-V2) and ImageNet-Sketch (IN-S). We first assess ViLU’s uncertainty estimates in a zero-shot transfer setting, where the model is applied directly to each variant without any adaptation. As shown in Tab. 8, ViLU achieves competitive performance, notably outperforming LVU on IN-S (FPR95 of 73.1 vs. 86.6) and remaining close to MCM (70.9). On IN-V2, ViLU performs even better, reaching an FPR95 of 54.8 compared to 71.7 for MCM and 77.3 for LVU. These results confirm that ViLU retains reliable uncertainty estimates even when evaluated on unseen domains.

We then explore a few-shot adaptation scenario, where ViLU is fine-tuned using only five labeled images per class from the target domain (IN-V2 or IN-S). On IN-S, this minimal supervision significantly reduces FPR95 from 73.1 to 54.4, outperforming both MCM (70.9) and LVU (72.3). On IN-V2, ViLU achieves similarly strong improvements, lowering FPR95 from 54.8 to 52.7, once again surpassing MCM (71.7) and LVU (68.7). These results highlight ViLU’s strong adaptability in low-data regimes and confirm that even minimal adaptation of the uncertainty head can lead to substantial gains in reliability under distribution shifts.

<table><caption>Table 8. FPR95↓ on ViLU’s domain generalization from ImageNet to ImageNet-V2 and ImageNet-Sketch.</caption><thead><tr><th>Dataset</th><th>MCM</th><th>LVU (zero-shot)</th><th>ViLU (zero-shot)</th><th>LVU (5-shot)</th><th>ViLU (5-shot)</th></tr></thead><tbody><tr><td>IN-V2</td><td>71.7</td><td>77.3</td><td><strong>54.8</strong></td><td>68.7</td><td><strong>52.7</strong></td></tr><tr><td>IN-S</td><td>70.9</td><td>86.6</td><td>73.1</td><td>72.3</td><td><strong>54.4</strong></td></tr></tbody></table>
## C.6. Impact of concept coverage in pre-training

In the main paper, we evaluated the zero-shot generalization ability of ViLU when pre-trained on CC12M and tested on 12 downstream datasets spanning various domains. In this section, we conduct a controlled experiment to assess whether better coverage of target concepts during pre-training improves zero-shot transfer. To this end, we construct a synthetic multi-dataset by combining the training sets of the 12 downstream datasets. Each image is paired with a pseudo-caption of the form “This is a photo of a ”, allowing us to train ViLU in the same image-caption setting as for CC12M. As shown in Tab. 9, this targeted pre-training leads to a substantial reduction in FPR95 across most datasets, with an average of 63.1 compared to 68.6 for the CC12M variant and 70.5 for MCM. These results confirm that more explicit coverage of the target classes during pre-training can significantly improve the quality of uncertainty estimates in zero-shot settings.

<table><caption>Table 9. FPR95↓ across datasets. Zero-shot performance when pre-trained on a Multi-datasets vs. CC12M.</caption><thead><tr><th>Dataset</th><th>MCM</th><th>CC12M<br>ViLU</th><th>Multi-datasets<br>ViLU</th></tr></thead><tbody><tr><td>CIFAR-10</td><td>52.1</td><td>54.2</td><td><strong>31.9</strong></td></tr><tr><td>CIFAR-100</td><td>67.3</td><td>59.9</td><td><strong>50.3</strong></td></tr><tr><td>Caltech101</td><td>68.7</td><td><strong>48.8</strong></td><td>70.8</td></tr><tr><td>Flowers102</td><td>68.0</td><td>67.4</td><td><strong>45.9</strong></td></tr><tr><td>OxfordPets</td><td>59.9</td><td><strong>58.1</strong></td><td>72.1</td></tr><tr><td>Food101</td><td>63.3</td><td>67.4</td><td><strong>36.2</strong></td></tr><tr><td>FGVCAircraft</td><td>82.9</td><td>82.3</td><td><strong>80.3</strong></td></tr><tr><td>EuroSAT</td><td>87.6</td><td><strong>85.7</strong></td><td>91.3</td></tr><tr><td>DTD</td><td>77.9</td><td>78.2</td><td><strong>75.2</strong></td></tr><tr><td>SUN397</td><td>75.9</td><td><strong>72.7</strong></td><td>81.0</td></tr><tr><td>StanfordCars</td><td><strong>73.4</strong></td><td>84.1</td><td>76.4</td></tr><tr><td>UCF101</td><td>68.9</td><td>63.8</td><td><strong>45.4</strong></td></tr><tr><td>Average</td><td>70.5</td><td>68.6</td><td><strong>63.1</strong></td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>Figure 9. Uncertainty score distribution. Prediction for correctly and incorrectly classified samples on CC12M, LAION400M, CIFAR100 and DTD.</figcaption></figure>

## C.7. Qualitative results

We provide additional visualizations on eight datasets in Fig. 9 and Fig. 10, illustrating the distribution of uncertainty scores for correctly and incorrectly classified validation samples. Our results demonstrate the consistency of ViLU in assigning high uncertainty scores to misclassified samples (red) and low uncertainty scores to correctly classified ones (blue).

Unlike visual uncertainty models such as ConfidNet [8], which rely solely on image features, our multimodal architecture leverages both visual and textual information to provide more reliable uncertainty estimates. Learning a cross-attention mechanism between image and text allows ViLU to better capture ambiguities in class definitions, leading to improved uncertainty calibration across diverse datasets.

<figure><img src="image_12.png" /><figcaption>Figure 10. Uncertainty score distribution. Prediction for correctly and incorrectly classified samples on EuroSAT, Food101, Flowers102 and UCF101.</figcaption></figure>