Post-hoc Probabilistic Vision-Language Models
Authors: Anton Baumann
Rui Li
Marcus Klasson
Santeri Mentu
Shyamgopal Karthik
Zeynep Akata
Arno Solin
Martin Trapp
Venue: ICLR 2026 Poster (2026)
Keywords: Uncertainty Quantification
Active Fine-Tuning
Bayesian Deep Learning
Vision-Language Models
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Vision-language models (VLMs), such as CLIP and SigLIP, have found remarkable success in classification, retrieval, and generative tasks. For this, VLMs deterministically map images and text descriptions to a joint latent space in which their similarity is assessed using the cosine similarity. However, a deterministic mapping of inputs fails to capture uncertainties over concepts arising from domain shifts when used in downstream tasks. In this work, we propose post-hoc uncertainty estimation in VLMs that does not require additional training. Our method leverages a Bayesian posterior approximation over the last layers in VLMs and analytically quantifies uncertainties over cosine similarities. We demonstrate its effectiveness for uncertainty quantification and support set selection in active learning. Compared to baselines, we obtain improved and well-calibrated predictive uncertainties, interpretable uncertainty estimates, and sample-efficient active learning. Our results show promise for safety-critical applications of large-scale models.

Published as a conference paper at ICLR 2026
POST-HOC PROBABILISTIC VISION-LANGUAGE MODELS
Anton Baumann 122 Rui Li
Mareus Klasson
Santeri Mentu
Shyamgopal Karthik
Zeynep Akata 1226,77
Arno Solin Martin Trapp 9.10
TTechnical University of Munich
"Helmholtz Munich *ELLIS Institute Finland Aalto University
"Finnish Center for Artificial Intelligence
University of Tübingen Munich Center for Machine Learning EErcssonn Research KKTH Royal Institute of Technology Digital Futures
7 Munich Data Science Institute
ABSTRACT
Vision-language models (VLMs), such as CLIP and SigLIP, have found remarkable success in classification, retrieval, and generative tasks. For this, VLMs determin- istically map images and text descriptions to a joint latent space in which their similarity is assessed using the cosine similarity. However, a deterministic mapping of inputs fails to capture uncertainties over concepts arising from domain shifts when used in downstream tasks. this work, we propose post-hoc uncertainty estimation in VLMs that does not require additional training. Our method leverages a Bayesian posterior approximation over the last layers in VLMSs and analytically cosine similarities. We demonstrate its effectiveness
quantifies uncertainties for uncertainty quantification and support set selection in active learning. Com- pared to baselines, we obtain improved and well-calibrated predictive uncertainties, interpretable uncertainty estimates, and sample-efficient active learning. Our results show promise for safety-critical applications large-scale models.
Post-huc Probabilistic Model
Uncerlainty Quantification
Active Learning
Laplace approcimation
Updatc mudel
Pre eranned VLM (CLPP SigLLM
VLy
Glassss 0455
Glases 0.99
Glasses: 0.2
Scissons 0:35
Scisces: 0.01
Scusoes:
Certsin
Figure 1: We introduce an efficient and effective post-hoc method to provide uncertainty estimates for langualanguugge models (e.g., CLIP, SigLIP) using a Laplace approximation. We demonstrate that uncertainty estimates derived from this approximation improve the calibration of these models on several zero-shot classification benchmarks (Sec. 4.1) and are effective active learning (Sec. 4.2).
INTRODUCTION
Pre-trained large-scale vision-language models (VLMs) (Bordes et al., 2024; Zhang et al., 2024). such as CLIP (Radford et al., 2021) and SigLIP (Zhai et al., 2023), have achieved remarkable success in tasks like zero-shot classification, retrieval, and generation, driven by training on billion-scale data sets (Gadre et al., 2023; Schuhmann et al., 2022). However, when employing large-scale machine learning models reliably in real-world settings and downstream applications, expect them not only to provide accurate predictions but also to enable us to quantify their predictive uncertainties. Obtaining efficient and effective uncertainty estimates is particularly relevant for safety-critical applications, as well as when making decisions based on those estimates, such as in active learning.
Previous work on uncertainty quantification for VLMs has primarily focused on calibration (Guo et al., 2017: Tu et al., 2023), test-time adaptation (Ayhan & Berens, 2018: Farina et al., 2024; Yoon al., 2024; Lafon et al., 2025), fine-tuning (Fort al., 2021; Tu al., 2023: Ju et al., 2025), or training probabilistic VLMs from scratch (Chun, 2024: Chun et al., 2025). However, each of those approaches has limitations regarding their applicability in real- world settings. For example. calibration methods cannot capture epistemic uncertainties, adapter and maaining-based methods
WWok partially done during internship at Aalto University.

---

Published as a conference paper at ICLR 2026
Before Active Learning (Zero-shol)
After Active Learning
Entropy (error > 0.5)
Error (entropy < (counts log log-scule)
0.5
1-5
0.2
0.6
0.8
(b)
(a)
(b)
(a)
Firre
Fmmor
0.5
0.5
(c)
(d)
(c)
Uncertainty (Entropy)
Uncertainly (Entropy)
Uncertainty (Entropp)
Uncertainly (Entropy)
Figure 2: Predictive error vs. uncertainty (entropy) on the EuroSAT data set (Helber et al., 2019) for the OpenCLIP ViT-H-14 14 model. The zero-shot comparison (left side) of the original model and its Bayesian counterpart (l) indicates that our Bayesian model exhibits better calibration and substantially reduces overconfident predictions. Active Learning results (right side) show that those improvements lead to a substantially reduced misclassification rate after adaptation; quadrant (b).
come with substantial computational demands and require retraining in streaming/active learning settings, and test-time adaptation methods significantly increase inference costs.
To manifest efficient and effective uncertainty quantification for the reliable application of VLMs, identify the following desiderata: The method should be applicable to any VLM architecture (model-agnostic), and uncertaintics should be obtained in a post-hoc manner without retraining the model from scratch. During inference, it should have low to computational overhead (efficient) and capture relevant sources of uncertainties (effective). Finally, the method should extract uncertainties from the original VLM without adding new layers or adapters that require training (training-free)
The Bayesian framework provides a principled way to model epistemic and aleatoric uncertainties, and has shown promise as a 'toolbox' for uncertainty quantification in deep learning (Papamarkou et al., 2024). Consider Fig. 2, which shows results on the EuroSAT data set (Helber et al., 2019), a land use and land cover classification task based on Sentinel-2 satellite images, for the popular OpenCLIP model We observe that the Bayesian counterpart (I) results in less overconfident predictions before active learning (compare quadrant l and a) and substantially reduces the error in the predictions after active learning, compared to the fine-tuned OpenCLIP model misclassification of the OpenCLIP model after active learning can be attributed to its overconfident behaviour before and after active learning, indicating the benefits of using a Bayesian formulation.
Much of the
This work proposes BLye an efficient and effective post-hoc uncertainty quantification method for pre-trained VI.Ms that adheres to the outlined desiderata. We leverage a Laplace approximation (MacKay, 1992) to the Bayesian posterior, thereby climinating the need for additional training, architectural changes, or modifications to the training objective. For this, we introduce independent probabilistic models for each modality, adhering to the i.i.d. assumption and enabling efficient posterior inference. Further, derive analytical expression for the distribution over cosine similarities for efficient uncertainty propagation. We evaluate our approach on zero-shot classification benchmarks and for uncertainty-aware active fine-tuning (Gal et al., 2017: Hübotter et al., 2024), finding improvements in performance over baselines in both scenarios. Lastly, we assess the efficiency and robustness of our approach (BayesVLM) and find that BayesVLM provides efficient, effective, and robust uncertainty estimates, even when the VLM is pre-trained on proprietary data. Contributions The overall contributions are illustrated in Fig. 1 and can be summarised as follows: (i) we propose VayesVLM, an efficient and effective post-hoc method for uncertainty quantification in pre-trained VLMs, without architecture changes or further training (Sec. 3); (ii) we present the first direct Bayesian formulation of vision-language models and derive an analytical expression of the distribution over cosine similarities for efficient uncertainty propagation (Sec. 3.2); (iii) we demonstrate the efficacy of Bayes VLM in both zero-shot and active learning settings, showing improvements over baselines in both settings. And we assess its efficiency and robustness, finding that Bayes VLM provides robust estimates while introducing little to computational overhead (Sec. 4).
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
Vision-language models Models like CLIP (Radford et al., 2021) and SigLIP (Zhai et al., 2023), trained on massive data sets such as LAION (Schuhmann et al., 2022), have become widespread in various applications, including zero-shot classification, generative modeling (Rombach et al., 2022; Podell et al., 2024), and retrieval (Saito et al., 2023; Karthik al., 2024). This work presents an effective post-hoc approach to uncertainty estimation for these pre-trained VLMs.
Uncertainty vision-language models Quantifying uncertainties in VL has observed increasing interest, with approaches involving learning probabilistic embeddings, for example, by learning additional probabilistic adapters (Chun et al., 2025; Lafon al., 2025) or through pre-training/fine- tuning with probabilistic loss (Chun, 2024; Ju al., 2025). In addition, recent approaches also explored training-fre uncertainty quantification, e.g., through test-time augmentation (Ayhan & Berens, 2018) zero-shot outto--distributiin detection (Fu et al., 2025). Another key approach is to solely focus on calibration through methods such as temperature scaling (Guo et al., 2017). Further related works are discussed in App. B. In contrast, we present training-free post-hoc approach that does not require architectural changes, but estimates the Bayesian posterior of a pre-trained model and efficiently propagates uncertainty arising from the Bayesian posterior to the VLM output.
Active learning The goal of active learning (Ren et al., 2021; Settles, 2009) is to improve model performance by 'actively' selecting additional informative data through an acquisition function (Holub et al., 2008; Sener & Savarese, 2018). A particularly relevant area is Bayesian active learning (MacKay. 1992: Gal et al., 2017), where acquisition functions leverage model uncertainties. Notable examples include the BALD score (Houlsby et al., 2011) and EPIG (Bickford Smith et al., 2023), both of which are functions of information gain. While such methods are gaining traction in large language models (Hübotter et al., 2025), they remain relatively underexplored for VLMs, where ad-hoc strategies are more prevalent. In our work, we bridge this gap.
3 METHODS
Notation We denote vectors by bold lower-case letters (e.g., x, a) and use bold upper-case letters for matrices (e.g., X, P). Further, sets denoted in upper-case calligraphic letters (e.g., D, I) and model parameters or hyperparameters are denoted using Greek letters (e.g., o, 0). In particular, let at J!MG E Ph and 2r !x! RRPII denote the zh image and jHh text description, respectively. Further, let Ø: 12PM + IR
dive and th: 1P + dd denote the image and text encoders of the VLM, where Pine and Pixr the respective input dimensionalities and disa drxt is the dimensionality of the respective feature space. Then, by denoting the linear image and text projections as P E R dx and Q E ddddr respectively, the feature embeddings in the joint space can be written as g (M) and h = Qrb(x:XT) We write G and H to denote the matrices of stacked image and text embeddings, respectively. whose rows correspond the individual gi and hy. Lastly, we use the hat symbol to denote unit-length normalised vectors, e.g., 9 = 9/ lall. The notation is listed in full in App. A.
3.1 BACKGROUND
Language-image pre-training We consider VLMs trained by minimising the InfoNCE loss (Oord al., 2018) (e.g., CLIP (Radford al., 2021)) and present additional experiments for the SigLIP loss (Zhai et al., 2023). Specifically, the InfoNCE loss is defined as the sum of two cross- entropy terms, one for each relational direction-image to text (LCE IMG IMG X XTXT)) and text image (XIT,
Ixr x IIM))). The total loss is defined as follows LINEANE X IMG X TXT
1 Ei log exp(to h4))
exp (th, 9:)
2n Ei log
2n
Ej exp(t)] h2)
Σ exp (th, 93)
IMG-TXXT, (00
IMG+-TXT, TAT,,
where is a learnable temperature parameter, n denotes the number of image-text pairs, and ğ and h the unit-length normalised embeddings. This contrastive loss function encourages embeddings for matching image-text pairs to be similar while simultancously pushing unrelated image-text pairs away from each other (Oord et al., 2018). In practice, evaluating this loss is infeasible on billions of
3

---

Published as a conference paper at ICLR 2026
Inguts
Encoders
Laplace Approx. (Bq. (6))
Embeddings Become Probabilistic (B4, (7))
Cosine Similarities Become
Probabilistic (Eq. (10))
N(P. Yoad]
An image of (reading glasses).
XIQQ Ers))
image (scissors)
An image of (rayon pencils).
Figure 3: Illustration of uncertainty propagation in BLyesVLMMs: We estimate uncertainties over the last layers of both encoders using a Laplace approximation, which induces probabilistic feature embeddings. We then approximate the distribution over cosine similarities by estimating the expected value and variance. The cosine similarity distribution is then propagated to the VLM output.
data points. The common practice adopted to evaluate on a sufficiently large batch. Recently, the SigI.IP loss (Zhai et al., 2023) was proposed as an alternative to the InfoNCE loss, binary classification loss over cosine similarities. Further details on SigLIP are given in App. B.2.
Laplace approximation Given data set D {a y1)) and denote the neural network parameters as 0. In Bayesian deep learning, we aim to estimate the posterior distribution, i.e.
p(0) IT = p(y: Fy 0)
prior x likelihood
0 (D)
(2)
So p(0) p(y: x,, 0) do marginal likelihood
Since the marginal likelihood involves an intractable nigh-dimensional integral, we approximate the posterior. We adopt the Laplace approximation (LA) (MacKay, 1992), a post-hoc method that has been increasingly used in the Bayesian deep learning community (Daxberger et al., 2021; Li et al., 2025; Meronen et al., 2024; Ritter et al., 2018; Roy et al., 2022; Scannell et al., 2024).
Specifically, LA fits Gaussian distribution the posterior, centred the MAP estimate of a pre- trained model, and is therefore 'post-hoc'. The prior is implicitly defined by the 1.2 regularisation (weight decay) commonly used during training (Radford al., 2021; Zhai al., 2023), and corre- sponds to diagonal Gaussian prior on the parameters, i.e., p(0) ( À I). The likelihood is defined by the training loss. The final approximate posterior given as p(0 D) ss (VMOAP, E) where OMAP is the MAP estimate and Σ = (-V2 p log p(D 0) 0==0MPP + AT) is the Hessian of the negative log joint evaluated MAPP. detailed derivation is given in App. B.3.
3.2 BLYESVLM: POST-HOC PROBABILISTIC VLMS
To estimate predictive uncertainties in a post-hoc fashion for VLMs, we independently estimate the posterior of the image projection P and text projection Q. For CLIP, we reformulate the contrastive loss to obtain tractable likelihoods for and Q, enabling separate posterior inference. We then approximate the Hessian of the log-likelihood and show how the resulting posteriors induce distribution over cosine similarities. Finally, we derive Gaussian approximation of this distribution for efficient downstream inference. Our Bayes VLM pipeline is illustrated in Fig. 3.
Estimate posterior:
Likelihood approximation The first step formulating our Bayesian model, BLy, is to define its likelihood function. When doing so, encounter the following key challenges: popular loss functions for VLMs, such as the InfoNCE loss (Eq. (I)), entangle modalities and data points. While this is desirable behaviour when learning multi-modal models, it breaks the usual i..d. assumption made in Bayesian models. Specifically, we have that
(x) JMG
TXT) non-i.id p(a] IMA xx! TXT
TvG TXT O).
which hinders straightforward application of the Bayesian framework. data is only conditionally independent. For that purpose, we are instead assuming two independent probabilistic models, one for each modality, with likelihood functions corresponding to the conditional probability for each
4

---

Published as a conference paper at ICLR 2026
modality rather than their joint, i.e.,
zt !MM
iid p(at
X TxT O).
x
ExT 1id
p( TXT
X XIMG 0).
(iid.. assumption)
Consequently, in case of the InfoNCE loss, each likelihood function is given by its respective mopality-specific sub-loss term, i.e., in case of the probabilistic model for the image modality, we have LCF (X ING X XIxT), and corresponds to a categorical distribution. A similar approximation is also applied to SigLIP. Crucially, defining independent probabilistic models for each modality additionally necessitates independence between the encoders. For example, when treating the projection layers P and Q probabilistically, we obtain that:
P IL Q-
(Consequence of i.i.d. assumption)
Following the i..dd assumption, the probabilistic model for the image modality is
image encoder (b()) and image projection layer
given text embeddings H
Po(a
g
Hg:
compute logits
and the likelihood becomes a categorical distribution (see App. C.2.1 for formulation)
log p( I-M
IMG
X TxT 0)
log ( IMG
X TXT
0) = log
(g.)).
The probabilistic model for text input can be obtained similarly. We can now apply the LA to this probabilistic model to estimate the approximate posterior.
Why is this still a reasonable approximation? For VLMs, it is important to capture interactions between modalities, and assuming independence seems problematic at first. However, a local post-hoc posterior estimation through the LA, we are effectively introducing an independence conditionally the MAP estimate of the (joint) contrastive loss. Thus, erucially, even though assume independence between modalities, we can still capture interactions between modalities. Note that this assumption is also important for computational reasons, helps derive computationally efficient approach. Our empirical assessment of the Hessian block structure, as discussed in App. F.13, shows that rross-modal curvature terms are moderate magnitude, indicating low to moderate cross- modal dependencies. A detailed discussion is given in Apps. C.1 and C.2.1.
are using
Hessian approximation Computing the full Hessian of the negative log- likelihood for the posterior covariance in the Laplace approximation is infeasible, as its size scales quadratically with the number of model parameters, making both its estimation and subsequent predictions computationally prohibitive. We, therefore, adopt the Generalised Gauss-Newton (GGN) approximation (Schraudolph, 2002), which requires the Jacobian of the outputs with respect to the parameters. For linear projection layers, this Jacobian can be derived in closed form. For the image and text encoders, however, the parameter count is prohibitively large, so we treat them as deterministic and approximate the posterior only over the projection matrices P and Q.
Estimate posterior:
To further reduce computational and memory costs, we use the Kronecker-factowe (KFAC) Gener- alised Gauss -Nwton (GGN) approximation (Ritter al., 2018; Martens & Grosse, 2015), which expresses the Hessian as a Kronecker product of two smaller matrices. This preserves a richer posterior structure than diagonal approximations. Following (Ritter et al., 2018), the KFAC GGN approximation for the Hessian of P is
(1/viR >. (00
IMG ) (0 !MG)
(1/v == JING JIMo(a) (Ma) T AIMo (x) IMG ))
(5)
A
Biso
where [I.
a1 %: /ag: and AIMC = (dia((T) मπ. with मे = exp(fo)/E exp(fe)) 9. h. =: fe. As estimating the Kronecker factors A and B over the training data set is infeasible, fol- lowing prior work (Ritter al., 2018), we leverage a subset of the data and include a pseudo-date count r to compensate for the reduced sample size. The posterior covariance over P is approximated as
ENN = (T(AING & BIMG) +X) ~ (VFAIMG + vXI)
@ (VFBING + vx1)
(6)
À
B B-1
two
5

---

Published as a conference paper at ICLR 2026
Note that the Kronecker factors A and B can be understood model statistics under the training data. After having the Gaussian posterior over P and Q, as Gaussians are closed under linear transformations, the distribution over g (and h) can be obtained analytically:
p(g D) =N (MMAPO@ (
(MG) T À A (aalo))
B
(7)
Analogous results hold for the text projection and text embedding h, which we omit here for brevity. See App. C.2.2 for detailed derivations, and Algorithm I outlines the steps described above.
Make predictions: Cosine similarities approximation Given posterior distribution over the model parameters, evaluating the VLM on an image-text pair yields random image and text embeddings rather than deterministic ones, inducing a distribution over their cosine similarity. While the cosine similarity remains well-defined, it becomes a random variable and is generally not Gaussian. The default prediction method, Monte Carlo estimation, requires costly sampling. To improve efficiency, we propose ProbCosine, Gaussian approximation of the cosine similarity distribution based on the first two moments of the image and text embeddings.
Let the Gaussian distributions for the probabilistic image and text embeddings have means Mg = (Pg,1 1. ................................
Mg.d) and fh = (pb,1)..... (h, and diagonal covariances Eg = diigg 1, o g.d) and Σ, i a.a)) Given the cosine similarity Scos (3r, y) x 1/||=|||lv|l between two vectors, the expected cosine similarity under the distribution of g and h is approximately:
Σi Pg.itn,i
(g,
(g, h)]
+ tog v EE Ei + t
g.
where we use the fact that E[ = 22 + and E ||||=|| S Ex He,t E..ee + +o by applying the triangle inequality. We can obtain the second moment (variance) Var[Scos(2 (g))) similarly, which is given as:
Var[Scos (g, h)] = Σ, og (o, + (h + H E. + 0g E, + oh
Henceforth, the local Gaussian approximation to the distribution over cosine similarities is: p(Scos(g, h)) .N (E|Scos(8) (g. h)], Var[Scos h)))
(10)
Finally, the predictive distribution p(y æ)) e.g., in a zero-shot classification setting, is calculated
with the probit approximation (Ghosal et al., 2022; Gibbs, 1998). Hence, approach allows for the direct propagation of model uncertainties to the class conditional. As shown in Fig. 10 (App. F),
compared ground truth, our approximation qualitatively results in a low approximation error. A detailed derivation can be found in App. C.3, and Algorithm 2 outlines the steps described above.
3.3 APPLICATION: PROBABILISTIC ACTIVE TEW-SHOT LEARNING
Active learning (Ren al., 2021; Settles, 2009) naturally evaluates uncertainty quality by selecting informative samples via predictive uncertainties. We assess BayesVLM with Bayesian acquisition functions and adaptive target-region selection. Given unseen test data Vest { Ji=1 with unknown labels, the goal is to choose a labeled subset {(z,, 1))} with æj pl x. v) that best reduces label uncertainty on Xest We first bias selection toward the query-set predictive distribution, then rank support candidates by influence or informativeness.
Target region selection Following Margatina et al. (2021): Hübotter et al. (2025), we first apply k-NN in feature space to pre-select support candidates near the test data, focusing on training points likely useful for the downstream task and reducing scquisition-function cost. Because features are stochastic, we compute either the expected cosine similarity (Eq. (8)) or the 2- 2-Wasserstein distance between aaage-feattree distributions. Details of the calculations are given in App. D.1.
Acquisition functions We consider the BALD (Gal et al., 2017) and EPIG (Bickford Smith et al., 2023) scores as acquisition functions and assess their viability on downstream tasks. Both acquisition functions can utilise model uncertainties estimated by the LA, but they differ conceptually in terms of which uncertainties are targeted. See App. D.2 for details.
Online Laplace approximation We maintain a Laplace posterior P and update it online by (i) a gradient step on P and (ii) updating the Kronecker factors. The prior precision can optionally be re-estimated after each step (Lin et al., 2023); see App. D.3.
the projeeprooection matrix

---

Published as a conference paper at ICLR 2026
Table 1: Does BLyesVLM provide useful uncertainty estimates in zero-shot settings? Yes. With the OpenCLIP ViT-B-32 model, our BayesVLM performs on par with CLIP and temp. scaling on ACC (%) and NLPD, while being better calibrated according to the ECE.
Metries
Methods
FLOWERS- 102
FOOD-101
CIFAR-10
CIFAR-100
IMAGENET-R
UCF101
SUN397
(LIPoRallord ct 3021)
20.
93:61=034
75.7620.209
11.5220.002
ACC1
CLIP scaling)
TTA (Farina sl., 2024)
688 0.00
81
80.2
93.66 40.20
88.51=03155
78. 76-0-66-00
65
T8 4700
74. 74.522003007
63. 77.07777747
50 80.8220.m77
68. 129
67. 1840 2333
BayeVVLM
40.4334.3144
93.62
73 a3.638
66.96:904u8 43s
CLIP at. 2021)
1 90
D. 1070000000
0,2150003
0.97an0028
Lovannee
1 39
10
NLPD
CLIP (temp. scaling)
67
007a
0:21=00000
0:
04410000
1.44.0000
1.11-00:30
TTA (Furina et al., 2024)
1. 18630007
0.67
0.35
9.90000000
1 1.50s0.00m
1.14
BayeVVLM
13.33000
0.68 100022
0:20 000007
0.96
00110/077
1 1444811
1. n2030
CLIP et 1021)
6.50
5.20
11.52
8.71
BCR
CLIP (tcmp. scaling)
TTA (Furina al., 2024)
9.63
5.51
4.18
4.74
2.02
1.88
3.07
5.27
2.88
1.80
11.75
3.61
9.92
2.67
BayesVI.M
4.22
1.69
0.72
1.78
3.57
2.06
Table 2: Can ProbCosine improve the zero-shot performance of pre-trained probabilistic models? Yes. Applying ProbCosine (Ours) to PCME++ (Chun, 2024) consistently improves zero- shot performance over its standard prediction (Mean) across classification benchmarks and metrics.
Metries
Methods
FLOWERS- 102
FOOD-101
CIFAR-10
CIFAR-100
IMAGENET-R
UCF101
SUN397
Mean
10.59
65.47=0.m00
75. 16
42.52=00009
42.87=0.057
45 75.97 20.00u
28. 50=00007
ACC 1
Ours
40.43+00ms
65.54+00000
75.12+00045
42.60=00089
42.83-0.0057
46 00+0.000000
28. 28.50+0.003
3.22+0:051
1.30+00125
0.77+00132
2.28+00216
2.77-00056
2.18-0.0ico
3.83+0.030
NLPD 4
Ours
3.04=i.047
1.25
0.75+n017
2.21400098
2.59:00mn
2.09=0.016
3.50=0,0722
Mean
8.81
6.78
4.79
10.78
17.38
12.62
26.03
ECE
Ours
2.79
1.54
2.02
4.89
10.82
5.61
19.41
4 EXPERIMENTS
We outline our setup and address three questions: (i) Uncertainty quantification: Does VL. pro- vide reliable uncertainty estimates? (ii) Active learning: Can we select informative data for fine-tuning using Bayes VLM uncertainty estimates? (ii) Efficiency and robustness: Does Bayes introduce overhead during inference, does it work in closed-soure data settings, and how sensitive is its perfor- mance to key hyperparameters? Further setup details and additional results appear in Apps. E and F.
Data sets We evaluate zero-shot classification on FLOWERS- 102 (Nilsback & Zisserman, 2008), FOOD-101 (Bossard et al., 2014). CIFAR- 10/100 (Krizhevsky & Hinton, 2009), IMAGENET- R (Hendrycks et al., 2021), UCF101 (Soomro et al., 2012), and SUN397 (Xiao et al., 2010). For active learning, we form a cross-domain setup with test data from a single domain and a training pool spanning all domains, using OfficeHome (Venkateswara et al., 2017) (Art, Clipart, Product) and an ImageNet variant with ImageNet-R and ImageNet-Sketch (Wang et al., 2019).
Network architectures In the zero-shot experiments, we used the OpenCLIP (Ilharco al., 2021) ViT-B-32 and ViT-L-14, and the SigLIP-B-16 model (Zhai et al., 2023). In the active learning experiments, we use either CLiP-Huge and SigLIP- -Base and fine-tune their projection layers.
Zero-shot baselines We compare with vanilla CLIP/SigLIP. CLIP/SigLIP with temperature scal- ing (Guo al., 2017; Nixon al., 2019), and test-time augmentation (TTA) (Farina et al., 2024). Tem- perature scaling uses the parameter minimising negative log predictive density (NLPD) (Quinonero- Candela et al., 2005) on the ImageNet validation set (Deng et al., 2009). We also show ProbCo- sine can pair with probabilistic VLMs trained from scratch, e.g., ProLIP (Chun et al., 2025) and PCME+ (Chun, 2024). Our focus is on training-free uncertainty estimation, not methods requiring extra adaptation (Upadhyay et al., 2023; Zhou et al., 2025).
Acquisition functions For active learning, we incorporate the uncertainties from VLyes into the acquisition functions BALD and EPIG and compare against random and entropy- -baodd selection. Both BALD and EPIG use target region selection with nearest neighbour (NN), which selects a test sample based on the uncertainty score, and then selects its 1-NN of the labelled training samples. We also combine the random and entropy baselines with this targeted selection strategy.
Hyperparameter settings mini-batches) from LAION-400M (Schuhmann et al., 2022), and used the same estimate across all ex- periments. The pseudo-data count r was selected via grid search to minimise NLPD on the ImageNet
We estimated the Hessian with 327k image-text pairs (10 CLIP
7

---

Published as a conference paper at ICLR 2026
OH--Art
OHI-CliArrt
OH-Product
IN-R
IN-Sketch
Weighed ACC
0.80
0.78
U.5.
0.83
0.50
on
0.30
1.15
NLPD
0.55
0:20
1.10
$4
8
9
4
10
Subeet
Subset
Subset Size
Subset Size
Figure 4: Can we select informative data for fine-tuning using BLyesVLM uncertainty estimates? Yes. On the OfficeHome data set (OH) and ImageNet variants (IN), when using uncertainty-based scores (EPIG (-) and BALD (-)) to select the fine-tuning data, achieve better performance compared with Entropy (targeted) (-), Entropy (-). Random selection (targeted) (-), ) and Random selection (-- Thus, highlighting the benefits of using uncertainties from Bayes VLM.
validation set, and the prior precision À was set by maximising the marginal likelihood (App. C.2). The same hyperparameters were used for both zero-shot and active learning experiments.
Evaluation metries For the zero-shot experiments, we report the mean and standard error of accuracy (ACC), NLPD Catnooner--Candele et al., 2005), and the expected calibration error (ECE) (Guo et al., 2017) computed over the test set. We use paired (-test with 0.05 to bold results with significant statistical difference. Active-learning results use weighte accuracy and NLPD.
4.1 UNCERTAINTY QUANTIFICATION: DOES BAYESVLM PROVIDE RELIABLE ESTIMATES?
We first test the uncertainty estimates of BayesVLLM in the zero-shot setting. In Table 1, report the zero-shot performance of the CLIP-base model using our post-hoe Bayes VLM approach, alongside baseline methods, with a focus on predictive quality and uncertainty calibration. Results for CLIP- Large are provided in Table 9 (App. F.7). We observe that BayesVLM achieves similar ACC but lower NLPD than the deterministic CLIP across all data sets, showing that BayesVLM is less overconfident when predicting the incorrect class. BayesVLM performs similarly to temp. scaling on ACC and NLPD, but outperforms all baselines on the ECE. Although TTA achieves higher ACC on some benchmarks, Bayes VLM is significantly better calibrated, which results in more useful uncertainty estimates. We conclude that VLy improves model calibration and uncertainty estimation without compromising performance, indicating the effectiveness of our post-hoc strategy.
To test ProbCosine (Sec. 3.2), we applied it to probabilistic embeddings from the pre-trained VLMs PCME++ (Chun, 2024) and ProLIP (Chun et al., 2025) (see Table 11). Zero-shot results for PCME+ (Table 2) show that PCME++ combined with ProbCosine keeps accuracy while consistently improving calibration, indicating ProbCosine can improve any VLM with Gaussian embeddings. Similarly, ProbCosine improves calibration in most cases when combined with ProLIP (Table 11).
4.2 ACTIVE LEARNING: CAN WE SELECT INFORMATIVE DATA USING BAYESVLM?
To further assess the utility of BayesVLM's uncertainty estimates, we evaluate in the active learning setting. We consider a cross-domaie setting where the unlabelled target data is from single domain while the labelled training samples are from multiple domains. The goal is select the most informative samples from the diverse pool for adapting to the target domain, given a maximum budget (subset size) of support set samples. We experiment with the OfficeHome (Venkateswara et al., 2017) (OH) dataset, where the domains are (Art, Clipart, Product), and ImageNet-vane variant (IN) with domains {R. Sketch). We incorporated the BayesVLM uncertainties into BALD and EPIG and compared against random and entropy- based selection, either from (i) the full training pool (Random, Entropy) or with (ii) selection from the test set followed by I-NN selection (targeted).
As shown in Fig. 4, both EPIG and BALD, with VayesVLM uncertainties for data selection, out- perform Random and Entropy across various subset sizes and target domains. On OH-Product and IN-Sketch, EPIG and BALD obtain similar weighted ACC Entropy-targetar However, EPIG
8

---

Published as a conference paper at ICLR 2026
consistently achieves lower NLPD than hatropp-targetdd, which shows that the finetuned model is less overconfident on incorrect predictions when trained with samples selected using Bayes VLM. Similar conclusions can be observed for ELIP-Huge and SigLIP-Bar (Figs. 13 and 14 in App. F).
In Fig. 2, we show the change in the predictive error (1 py = y² x)) and the predictive uncertainty (entropy) for Bayes before (zero-shot) and after active learning EuroSAT (Helber et al., 2019) using EPIG. We use 200 support points and compare against CLIP with entropy selection. Bayes VLM reduces overconfident predictions in the zero-shot setting (samples move (b) - (a)), and more effectively adapts to the new data set based on the support set (samples move to (c)).
4.3 EFFICIENCY AND ROBUSTNESS: HOW EFFICIENT AND ROBUST IS BAYESVLM?
Following the protocol for zero-shot experiments, we assessed the performance of Bayes VLM when estimating the Hessian in settings where the training data not available, an increasingly common setting for modern machine learning models. In particular, using the CC12M as a proxy dataset for CLIP models and used the LAION-4000 -400 dataset as a proxy for Google's ' SigLIP model. We find that BayesVLM provides robust uncertainty estimates for CLIP even when estimated on the proxy dataset, cf.., Table 3, and remains stable under mild distribution shifts in the proxy dataset (see App. F.11). Moreover, Bayes VLM provides competitive results for SigLIP, a VLM model trained on proprietary data (Table 10 in App. F.7), is robust w.r.t. the pseudo-data count T (App. F.8), provides interpretable uncertainties under corruptions (Fig. 5), and and maintains calibration under substantial distribution shift (App. F.12).
estimated the Hessian of BayesVLM
Table 3: Does BayesVLM work in closed-source data settings? Yes. With OpenCLIP ViT-B-32 trained on LAION-400M and Bayes VLM estimated on the proxy dataset CC12M, we find that results robust and show only slight degradation; statistically significant differences are bold (p 0.05).
Metrics
Dataset
FLOWERS-102
FOOD-101 101
CIFAR-10
CIFAR-100
IMAGENCT-R
UCF101
SUN397
LATON-40MM
68.87=0.009
80.43+0.396
33162-0.2111
73.63 13.6340.5105
-4.45
61.43+00s00
66.96 crus
1
CC12M
68. 12:0.m0
80.35
93.37=4
73.78aa.. aws
74.32:0.1m0
61.46200.907
60.81:0.470
LAION 400M
1.73+00020
0.68:00128
0.20+00077
0.95-00010
1.03+00177
1.4400018
1.12:0.05
NI.PD L
CC12M
1.77a0usm
0. 0.68
0.20sauwe
0.96
1.03
1.4-00015
1.13.0002
LAION 400M
4.22
1.69
0.72
1.92
1.78
3.77
2.06
ECE
CC12M
3.84
0.99
0.70
1.43
1.39
3.83
3.89
Computational overhead Compared to the deterministic CLIP, BayesVLM adds under 5% runtime for CL.IP-base and less than 1 1% for huge models (Table 7 in App. F.5). Inference cost rises only 0.11% GFLOPs for CLIP--bsee, whereas TTA needs an 80x increase, see Table 8 in App. F.5.
Probabilistic cosine similarities We qualita- tively assessed the distribution obtained by Prob- Cosine on a randomly selected test example from
Expecial Valae
Variane
the OfficeHome clipart domain, evaluating the
An inngge of
mean and variance of the cosine similarity un-
Amirg of sxp
der increasing corruption in both image and text domains. Text corruption was introduced by randomly replacing characters with 'x', and im- age corruption by randomly adding grey squares. Fig. 5 shows the mean and variance of cosine sim- ilarities as corruption increases. We observe that the expected cosine similarity generally decreases and variance increases with more corruption, indi- cating that our approximation effectively captures model uncertainties under distribution shift. Note that observe a slight increase in the cosine similarity after one character has been replaced, indicating that performing predictions solely on the expected cosine similarity can be problematic. In this case, the variance over cosine similarities can capture the change in the input, highlighting the importance of capturing and propagating the model uncertainties.
Ac of
As in cop
ins aspp
ararcoda aoces
Figure 5: Illustration of ProbCosine under in- creasing corruption. The mean similarity de- creases and variance increases with higher lev- els of comuption, demonstrating effective uncer- tainty estimation under distribution shift.
Number of data points for Hessian estimation We evaluated how the number of samples affects Hessian estimation by computing the trace over 10 random subsets of LAION-400M. As shown in

---

Published as a conference paper at ICLR 2026
Fig. 11 (App. F.4), the traces for both image and text projections quickly converge with low variance, suggesting that 10 bani-batches are sufficient for a stable estimate.
Number of negative samples We vary the batch size E (32768, 8192, )048} and estimate the posterior from 1-5 random batches, report- ing med over five trials. Since the poste- rior depends on negative samples only via the Hessian B (cf. Eq. (5)), we show the relative trace tr( (r(Bxx)) / tr (T(ss))), which is expected to be one. As observed in Figure 6 and Fig. 16 (App. F.9), a base batch size of 32768 stays near 1 across all batches with minimal variance, indi- cating stable estimates of the Hessian.
Br EICMOXK
# Datapoints (i Ki
Figure 6: Relative trace of the image Hessian B-factor for varying base batch sizes K (2048 (---), 8192 (-.-.. 32768 ( (-)) and 1-5 random
batches. Error bars show #1 std over five trials.
DISCUSSION & CONCLUSION
In this work, we introduced a novel approach for post-hoc uncertainty estimation and propagation for large-scale vision language models (VLMs) such as CLIP (Radford et al., 2021) and SigLIP (Zhai et al., 2023). For this, we first formulated probabilistic models admissible to Bayesian treatment and then utilised post-hoc posterior approximation over the last layer of each encoder. Moreover, derived an analytic approximation of the distribution over cosine similarities for efficient uncertainty propagation. Thus, approach allows efficient and effective uncertainty quantification without any BL. architectural changes or additional training. We demonstrated the effectiveness of Bayes VL.M zero- shot and active learning settings, showing improvements over baselines, and additionally assessed its robustness and efficiency, showing that Bayes VLM is valuable tool for reliable application of VLMs.
Beyond the settings considered this work, LLM VLM is also applicable several related problems. An interesting application is the detection of OOD or failure modes. Having access to uncertainties over the image and text embeddings directly enables such scenarios. For example, credible intervals obtained from our Bayesian treatment provide a principled signal for detecting distribution shift or unreliable predictions. Another promising direction uncertainty-awwi awaree retrieval. Since retrieval methods rely on similarity scores in shared embedding space, uncertainties over the embedding projections can be incorporated into the retrieval process. This allows retrieval systems to more reliably detect cases where inputs fall outside the model's training distribution.
Limitations The limitations of our approach are (i) we need access to training data to estimate the Hessian, (ii) require that embeddings are Gaussian distributed, (iii) method only utilises Bayesian projection layers, and (iv) we assume independence between image and text projection parameters in the local curvature approximation. Because training data for many VLMs closed- source, we also assessed potential performance degradation when estimating the Hessian on proxy datasets and found that BayesVLM yields robust estimates. However, further research is needed in closed-source settings.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we have provided detailed information on our method and experimental setups. We will discuss the respective details below.
BayesVLM method & algorithms In addition to the details presented in the main text (Sec. 3), we provided detailed derivations in App. of (i) the likelihood function approximation in App. C.I. (ïi) the Laplace approximation used in our method in App. C.2, and (iii) the distribution over cosine similarities App. C.3. Moreover, we provided algorithmic descriptions of our method in Algorithm I and Algorithm 2, outlining the precomputation of Bayes VLM and the forward inference. Lastly, we presented detailed descriptions of the active learning algorithm used in our work in App. D and provided specific details on (i) the targeted selection algorithm in App. D.I, (ii) the acquisition functions used in this work in App. D.2, and (iti) the online Laplace updates in App. D.3.
Experiments In addition to the details provided in the main text in Sec. 4, we provided extensive additional information in App. E. Specifically, we (i) detail information on the pre-traines models used in this work in App. 1. (ii) present detailed information hyper-paramedt in App. E.2, and in App. E.3, and (iii) present details on the hyperparameters and setup of the active learning experiments in App. E.4. We also presented additional experiments and experimental results that extend beyond those presented in the main text in App. F.
the Hessian estimation and respective
The code for the experiments is available at: https://aall .iithub. io/Bayes //BayeeVLMM/ Models and precomputed Hessian estimates can be accessed at: https:// huggingface, ct/ ceto/oollecte wtw.eettosocococcocoooooooootttt http:/aaaatommmmmaaaeeeaa
Implementation
ACKNOWLEDGEMENN
AS, RL, and SM acknowledge funding from the Research Council of Finland (grant number 339730 and 362408). MT acknowledges funding from the Research Council of Finland (grant number 347279) and support from the Wallenberg AI, Autonomous Systems, and Software Program (WASP), funded by the Knut and Alice Wallenberg Foundation. MK and SM acknowledge funding from the Finnish Center for Artificial Intelligence (FCAI). AB, SK, and ZA acknowledge partial funding by the ERC (853489- DEXIM) and the Alfried Krupp von Bohlen und Halbach Foundation. SK thanks the International Max Planck Research School for Intelligent Systems (IMPRS-IS) We acknow ledge CSC-IT Center for Science, Finland, for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through CSC. We acknowledge the computational resources provided by the Aalto Science- project. Finally, we thank Riccardo Mereu, Jonas Hübotter, and Omar Eldeeb for providing feedback on the manuscript.
REFERENCES
Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning. In European Conference on Computer Vision, pp. 137-153. Springer, 2020. 18
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations, 2020. 18
Murat Seckin Ayhan and Philipp Berens. Test-time data augmentation for estimation of heteroscedas- tic aleatoric uncertainty in deep neural networks. In Medical Imaging with Deep Learning, 2018 8. 3, 18
Jihwan Bang. Sumyeong Ahn, and Jae-Gil Lee. Active prompt learning in vision language models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp.
27004-2700 27014, 2024. 18
Freddie Bickford Smith, Andreas Kirsch, Sebastian Farquhar, Yarin Gal, Adam Foster, and Tom Rainforth. ceddiction-oriente Bayesian active learning. In International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, pp. 7331-7348, PMLR, 2023. 3, 6, 18, 30
11

---

Published as a conference paper at ICLR 2026
Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, et al. An introduction to vision-language modeling. arXiv preprint arXiv:2405.17247. 2024.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 101 mining discriminative compo- nents with random forests. European Conference on Computer Vision, pp. 446-461. Springer, 2014.7 7
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3558-3568, 2021. 32
Youngjae Cho, HeeSun Bae, Seungjae Shin, Yeo Dong Youn, Weonyoung Joo, and Il-Chul Moon. Make prompts adaptable: Bayesian modeling for vision-language prompt learning with data- dependent prior. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.
11552-11560, 2024. 18
Sanghyuk Chun. Improved probabilistic image-text representations. In International Conference on Learning Representations, 2024. 1, 7, 8, 17
Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8415-8424, 2021. 17
Sanghyuk Chun, Wonjae Kim, Song Park, and Sangdoo Yun. Probabilistic language-imaae pre- training. In International Conference on Learning Representations, 2025. 1, 3, 7, 8. 17, 35, 36
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortles Bayesian deep learning. Advances in Neural Infor- mation Processing Systems, volume 34, pp. 20089-20103, Curran Associates, Inc., 2021. 4, 19, 31
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 048-255,220 2009.
Karan Desai, Gaurav Kaul, Zubin Trivadi Aysola, and Justin Johnson. Redcaps: Web-curated image- text data created by the people, for the people. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. 32
Matteo Farina, Gianni Franchi, Giovanni lacca, Massimiliano Mancini, and Elisa Ricci. Frustratingly easy test-time adaptation of vision-language models. In Advances Neural Information Processing Systems, volume 37, pp. 129062- 129093. Curran Associates, Inc., 2024. 1, 7, 18, 35, 36, 41
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distributin detection. In Advances in neural information processing systems, volume 34, pp. 7068-7081. Curran Associates, Inc., 2021.
Hao Fu, Naman Patel, Prashanth Krishnamurthy, and Farshad Khorrami. Clipscope: Enhancing shot ood detection with bayesian scoring. In 2025 IEEF/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 5346-5355. IEEE, 2025. 3, 18
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36: 27092-27112. 2023.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In International Conference on Machine Learning. Proceedings of Machine Learning Research, pp. 1183-1192, PMLR, 2017. 2, 3, 6, 18
12

---

Published as a conference paper at ICLR 2026
Ido Galil, Mohammed Dabbah, and Ran Ylniv.... What learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers? In International Conference
on Learning Representations, 2023. 18
Indrayudh Ghosal, Yunzhe Zhou, and Giles Hooker. The infinitesimal jackikn and combinations of models. arXiv preprint vrXi::2209.00147, 2022.
Mark Gibbs. Bayesian Gaussian Processes for Regression and Classification. PhD thesis, University of Glasgow, 1998. 6, 20
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 2017. 3, 8, 18
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-221 22262226,, 2019. 2,9
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. generalization. In Proceed-
The many faces of robustness: A critical analysis of ings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8348 IEEE, 2021. 7, 32
Alex Holub, Pietro Perona, and Michael C Burl. Entropy-based active learning for object recogni- tion. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 1-8. IEEE, 2008. 3, 18
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: 2011. 3, 18, 30
Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause. Efficiently learning at test-time: Active fine-tuning of llms. In International Conference Learning Representations, 2025. 3, 6, 18
Jonas Hübotter, Bhavya Sukhija, Lenart Treven, Yarden As, and Andreas Krause. Transductive active learning: Theory and applications. arXiv preprint arXiv: 15898, 2024.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. 7, 32
Alexander Immer, Matthias Bauer, Vincent Fortuin, Gunnar Rätsch, and Khan Mohammad Emtiyaz. Scalable marginal likelihood estimation for model selection in deep learning. In International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 4563-4577 PMLR, 2021. 27, 30
Yatai Ji, Junjie Wang, Yuan Gong, Lin Zhang, Yanru Zhu, Hongfa Wang, Jiaxing Zhang, Tetsuya Sakai, and Yujiu Yang. Map: Multimodal uncertainty-aware vision-lant languue pre-training model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2023. 17
Li Ju, Max Andersson. Stina Fredriksson, Edward Glöckner, Andreas Hellander, Ekta Vats, and Prashant Singh. Exploiting the asymmetric uncertainty structure of pre-trained vlms on the unit hypersphere. In Advances in Neural Information Processing Systems. Curran Associates, Inc.,
2025. 3, 18
Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata, Hision-by-languee for training-fre free compositional image retrieval. In International Conference on Learning Repre- sentations, 2024.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 7
13

---

Published as a conference paper at ICLR 2026
Mare Lafon, Yannis Karmim, Julio Silva-Rodriguez, Paul Couairon, Clément Rambour, Raphaël Fournier-Snichotta, Ismail Ben Ayed, Jose Dolz, and Nicolas Thome Vilu: Learning vision- language uncertainties for failure prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. IEFE, 2025. 3, 17
Hao Li, Jingkuan Song, Lianli Gao, Pengpeng Zeng, Haonan Zhang, and Gongfu Li. differentiable semantic metric approximation in probabilistic embedding for cross-modie retrieval. In Advances in Neural Information Processing Systems, volume 35, pp. 11934-1 1946.. Curran Associates, Inc., 2022. 17
Rui Li, Marcus Klasson, Arno Solin, and Martin Trapp. Streamlining prediction in Bayesian deep learning. In International Conference on Learning Representations, 2025. 4, 19
Jihao Andreas Lin, Javier Antoran, and José Miguel Hermández-Lobato. Online laplace model selection revisited. In Fifth Symposium on Advances in Approximate Bayesian Inference, 2023. 6, 30
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings the the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 5206-5215, 2022. 18
David JC MacKay, Information- based objective functions for active data selection. Neural Computa- fion, 4(4):590-604, 1992. 2, 4, 18, 19
Katerina Margatina, Giorgos Vernikos, Loie Barrault, and Nikolaos Aletras. Active learning by
acquiring contrastive examples. In International Conference on Empirical Methods in Natural Language Processing, pp. 650--663, Association for Computational Linguistics, 2021. 6
James Martens and Roger Grosse. Optimizing neural networks with monnecker-factored approximate curvature. In International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 2408 2417. PMLR, 2015.
Lassi Meronen, Martin Trapp, Andrea Pilzer, Le Yang, and Amn Solin. Fixing overconfidence in dynamic neural networks. In IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2680-2690, 2024. 4, 19
Yibo Miao, Yu Lei, Feng Zhou, and Zhijie Deng. Bayesian exploration of pre-trained models for low-shot image classification. In Proceedingy of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 23849-23859, 2024. 18
Pablo Morales-Alvarezz Stergios Christodoulidis, Maria Vakalopoulou, Pablo Piantanida, and Jose Dolz. Bayesadapter: enhanced uncertainty estimation in clip few-shot adaptation. arXiv preprint arXiv:2412.09978, 2024. 17
Andrei Neculai, Yanbei Chen, and Zeynep Akata, Probabilistic compositional embeddings for multimodal image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 4547-4557, 2022. 17
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, pp. 722-729. IEEE, 2008.7
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Mcasuring calibration in deep learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, volume 2, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXhv:1807.03748 2018. 3, 18
Theodore Papamarkou, Maria Skoularidou, Konstantina Palla, Laurence Aitchison, Julyan Arbel, David Dunson, Maurizio Filippone, Vincent Fortuin, Philipp Hennig, José Miguel Hernández- Lobato, et al. Position: Bayesian deep learning is needed in the age of large-scale AI. In International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 39556-39588 PMLR, 2024.7
14

---

Published as a conference paper at ICLR 2026
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, 2024. 3
Joaquin Quinonero- Candell,,,, Carl Edward Rasmussen, Fabian Sinz, Olivier Bousquet, and Bern- hard Schölkopf. Evaluating predictive uncertainty challenge. In Machine Learning Challenges Workshop, pp. 1-27. Springer, 2005. 7, 8
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021. 1, 4, 7, 10, 18, 19, 32, 36, 41
Liva Ralaivola, Marie Szafranski, and Guillaume Stempfel. Chromatic pac-bayes bounds for non- iid data. In Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pp. 416-4236 PMLR, 16-18 Apr 2009. 21
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006. 25
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning, ACM Computing Surveys, 54(9):1-40,, 2021. 3, 6, 18
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable Laplace approximation for neural networks. In International Conference on Learning Representations, 2018. 4, 5, 19, 24,24,37 32
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10674-10685, 2022. 3
Subhankar Roy, Martin Trapp, Andrea Pilzer, Juho Kannala, Nicu Sebe, Elisa Ricci, and Amn Solin. Uncertainty-guided source-free domain adaptation. European Conference on Computer Vision, pp. 537-555. Springer, 2022. 4, 19
Bardia Safaci and Vishal M Patel. Active learning for vision-language models. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 4902-4912. IEEE, 2025. 18
Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen Yu Lee, Kate Saenko, and Tomas Pfister. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19305-19314, 2023. 3
Aidan Scannell, Riccardo Mereu, Paul Edmund Chang, Ella Tamir, Joni Pajarinen, and Arno Solin. Function-space parameterization of neural networks for sequential learning. In International Conference on Learning Representations, 2024. 4, 19
Nicol N Schraudolph. Fast curvature metrix-vector products for second-order gradient descent. Neural computation, 14(7):1223-1736 2002. 5, 23
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems, volume 35, pp. 3. 3,
Curran Associates, Inc., 2022. 1,
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representattons, 2018. 3, 18
Burr Settles. Active learning literature survey. Technical report, University of Wisconsin-Madiin Department of Computer Sciences, 2009. 3, 6, 18
15

---

Published as a conference paper at ICLR 2026
Divya Shanmugam, Davis Blalock, Guha Balakrishnan, and John Guttag. Better aggregation in test-time augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1214-1223, 2021. 18
the
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Annual Meeting of the Association for Computational Linguistics, pp. 2556-2565, 2018. 32
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5972-5981, 2019.
18
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprini arXiv: / 220022, 2012. 7
Weijie Tu, Weijian Deng, and Tom Gedeon. A closer look at the robustness of contrastive language- image pre-training (CLIP). In Thirty-seventh Conference on Neural Information Processing Systems, volume 36, pp.
Curran Associates, Inc., 2023. 1, 18
Weijie Tu, Weijian Deng, Dylan Camphell, Stephen Gould, and Tom Gedeon. An empirical study into what matters for calibrating vision-language models. International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 48791-48808. PMLR, 2024. 18
Uddeshya Upadhyay, Shyamgopal Karthik, Massimiliano Mancini, and Zeynep Akata. ProbVLM: Probabilistic adapter for frozen saionnlanguage models. Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 1899-1910, 2023. 7, 17
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5018-5027, 2017. 7, 8, 32
Dan Wang and Yi Shang. A new active labeling method for deep learning. In International Joint Conference on Neural Networks (IJCNN), pp.
IEEE, 2014. 18
Haohan Wang. Songwei Ge. Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, pp. 10506-10518. Curran Associates, Inc., 2019. 7.32
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3485-3492, 2010. 7
Yichen Xie, Han Lu, Junchi Yan, Xiaokang Yang, Masayoshi Tomizuka, and Wei Zhan. Active finetuning: Exploiting annotation budget in the the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23715-23724, 2023. 18
paradigm. In Proceedings of
Adam X Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. Bayesian low-rank adaptation for large language models. In International Conference on Learning Representations, 2024. 18
Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark A. Hasegawa-Johnson, Yingzhen Li, and Chang D. Yoo. C-TPT: Calibrated test-time prompt tuning for vision-language models via text feature dispersion. In International Conference on Learning Representations, 2024. 1, 18
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image traininn In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 11975 1986, 2023. 1, 3, 4, 7, 10, 18, 19, 32, 36
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5625-5644, 2024.
Lihua Zhou, Mao Ye, Shuaifeng Li. Nianxin Li, Xiatian Zhu, Lei Deng, Hongbin Liu, and Zhen Lei. Bayesian test-time adaptation for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 7, 18
16

---

Published as a conference paper at ICLR 2026
APPENDIX
The appendix is organized as follows: App. A summarizes the notation used in the paper. App. B reviews background on vision-language models and the Laplace approximation. App. C presents the derivation of the posterior estimation and the efficient computation of distributions over cosine similarity. App. D outlines the active learning setup. App. details the experimental setup, while additional results are provided in App.
Use of Large Language Models In this paper, L.MSs were used only for minor grammatical edits, word polishing, or rephrasing. They did not contribute to research ideation, experiments, core writing. All suggestions from LLMs were manually verified and edited by the authors prior final inclusion.
A NOTATION
We will briefly summarise the notation used throughout the paper. See Table for the modality- specific notation used and Table 5 for an overview of the notation of general operands and operators.
Table 4: Summary of codality-speccife notation.
Description
Image
Text
Input
x IMG
TXt
Encoder
((-)
(())
Projection matrix
P
Embedding
g
Normalised embedding
9
Stacked embeddings
G
H
Kronecker factors
AlMa, BIsO
Atxι, Bixt
Covariance matrix
Σ Eng
Σxxt
Jacobian matrix
JIMG
JIxT
Table 5: Summary of general notation.
Description
Notation
Number of data points
Number of test data points
Biest
Number of support set points
Kronecker product
Prior precision
Pseudo-dat count
B BACKGROUND
This section provides additional background information and an extended discussion of related work.
B. EXTENDED RELATED WORK
Uncertainty vision-language models Many efforts have aimed to learn probabilistic embeddings by making architectural changes the VLMs and pre-training with a probabilistic loss (Chun, 2024; Chun et al., 2025; 2021; Ji et al., 2023; Li et al., 2022; Neculai et al., 2022). To reduce training costs, several works have proposed enabling uncertainty estimation in pre-trained VLMs via additional training of adapters AMorales-Alvare al., 2024; Upadhyay al., 2023; Lafon
17

---

Published as a conference paper at ICLR 2026
et al., 2025), learning distributions of prompts (Cho al., 2024; Lu et al., 2022; Yang et al., 2024), model ensembles (Miao et al., 2024), or test-time adaptation (Zhou et al., 2025). These works use proxy data set different from the pre-training set to learn the predictive uncertainties. Test-time augmentation is a training-free method used for obtaining input-dependent predictive uncertainties by augmenting the test input (Ayhan & Berens, 2018; Farina et al., 2024; Shanmugam al., 2021), which trades off simplicity against higher inference costs. Other recent training- free approaches focus on zero-shot --iitiiiiiiiiitiinnnn detection in CLIP (Fu et al., 2025) or estimating the distribution on the hypersphere as a von-Mises Fisher distribution (Ju et al., 2025). Moreover, calibration of VLMs has been studied for mitigating overconfident predictions (Tu et al., 2023; 2024; Yoon et al., 2024) where temperature scaling is a common post-hoc method for calibrating pre-trained models using a held-out validation set (Galil et al., 2023; Guo et al., 2017). Here, we apply the Laplace approximation to estimate uncertainties directly from the pre-trained VLM without the need for additional training, architectural changes, or training from scratch. Our approach estimates a Bayesian posterior distribution with the pre-training data or proxy data set before test time and has a similar inference speed to the pre-trained VLM.
Active learning In active learning (Ren al., 2021; Settles, 2009), the model determines through an acquisition function which additional data points are needed to make reliable predictions on a given downstream task. The acquisition function quantifies the informativeness of samples using entropy (Holub et al., 2008; Safaei & Patel, 2025; Wang & Shang, 2014) or diversity-based scores ( (Ash al., 2020; Agarwal et al., 2020), coresets (Sener & Savarese, 2018), and parametric models (Sinha et al., 2019; Xie et al., 2023). Here, we focus on acquisition functions utilising model uncertainties from Bayesian active learning (Bickford Smith et al., 2023; Gal al., 2017: Houlsby et al., 201 I). A popular method is the BALD score (Gal et al., 2017; Houlsby et al., 201 I), which measures the reduction in epistemic uncertainties of the model. More recently, EPIG was proposed to measure the information gain in the space of predictions rather than parameters (Bickford Smith et al., 2023), building on MacKay's foundational work on informmtiionthhoreeee experimental design (MacKay, 1992). While such acquisition functions have gained traction in large language models (Hübotter ct al., 2025), they remain underexplored in VLMs, where ad-hoc strategies like prompt tuning (Bang et al., 2024) are more prevalent. This work bridges this gap by adapting Bayesian active learning methods to VLMs.
B.2 LANGUAGE- -MMAGE PRE-TRAINING
We consider VLMs trained by minimising the InfoNCE loss (Oord et al., 2018) (e.g.. CLIP (Radford
et al., 2021)) the SigLIP loss (Zhai et al., 2023). Specifically, the InfoNCE loss is defined as the sum of two cross-entropy terms, one for each relational direction-imas to text (LCE IMG X xIXT). ) and text image (Leu(X (XTT X (IM The total loss is defined as follows LIMINNCE X XXXT
exp(tt ha)
exp(th gi)
log
log
(11)
2n
L exp(to] h2)
2n
Σ)= exp(th 9j)
Cml
IMG-TXT, (20
IMG+-TXT, Len(XX xIMe))
where is learnable temperature parameter, denotes the number of image-text pairs, and ý and h are the unit-length normalised embeddings. This contrastive loss function encourages embeddings for matching image-text pairs to be similar while simultaneously pushing unrelated image-text pairs away from each other (Oord et al., 2018).
Recently, the LigLIP loss (Zhai et al., 2023) has been proposed alternative to the InfoNCE loss, aimed at improving numerical stability and training speed. In contrast to InfoNCE, the SigLIP loss uses binary classification loss over the cosine similarities, i..., LSigLIP XX X TXT)==
n log
(12)
1 + exp(2;) tg. h, + bb)
where 2ūz 1, 2i = if j and is a learnable bias term. For classification settings, the SigLIP loss does not provide normalised class conditional probabilities p(y x)) but provides binary classification probabilities. Henceforth, when fine-tuning a SigLIP pre-trained VLM for classification tasks, one typically uses the cross-entropy loss instead.
18

---

Published as a conference paper at ICLR 2026
B.3 LAPLACE APPROXIMATION
Given data set D = {o,, 4 and denote the model parameters as 0, in Bayesian deep learning, we aim estimate the posterior distribution
p(0)
1O p(y
2, 0)
p(0 D)
(13)
So ((0) IT P(y:
2c 0)d0
prior likelihood
marginal likelihood
Unfortunately, computing the denominator (marginal likelihood) is generally intractable (not feasible) as it requires integration over high-dimensional space w.tt a potentially non-linear function. classical approach to circumvent this challenge is to approximate the posterior using Laplace approximation MacKay (1992), which has recently gained traction the Bayesian deep learning community Ritter et al. (2018); Daxberger al. (2021); Li al. (2025); Meronen et al. (2024); Roy et al. (2022); Scannell et al. (2024).
The Laplace approximation hinges the idea that the posterior distribution is proportional to the joint, i.e.,
p(0 D) x p(0, = p(0) II ply: æc 0)
(14)
up to unknown normalisation constant (the marginal likelihood), Moreover, using second-order Taylor expansion the log joint around the maximum-a -aanuuum--psterroor (MAP) estimate θMAP (mode of the function) one obtains the unnormalised log density function of a Gaussian centred at θMAP, i.e. log p(0, D)= R
1 ) -0-- -MMAP) T -0MAP),
log p(MMAP, D)
(15)
2
where
Σ == (-V (-Vologp( log D) -1= -1 log 0)lo=0mAAP V8 log (())p(())omma))
(16)
the Hessian matrix of the log joint (prior likelihood) 0MAP- By matching the marginal likeli- hood in Eq. (2) with the normalisation constant of Gaussian, we obtain the Laplace approximation:
( (D) R (OMAP, Eー
(17)
with covariance given by the inverse of the Hessian matrix.
As LA fits Gaussian distribution to the posterior, centred at the MAP estimate of a pre-trained model, is 'post-hoc'. The prior is implicitly defined by the L2 regularisation (weight decay) commonly used during training Radford et al. (2021); Zhai et al. (2023), and corresponds to diagonal Gaussian prior on the parameters, i.e., p(0) AO A- I). The likelihood is defined by the training loss.
C DERIVATIONS
This section provides detailed derivations of the equations presented in the main text. App. C. discusses the setting where the i.i.d. assumption is not made and the challenges associated with App. C.2 discusses the i.i.d. assumption, the resulting probabilistic model, and the derivations for estimating the posterior. App. C.3 covers the derivations for efficient prediction, i.e., the distribution over cosine similarities.
C.I WHIA HAPPENS WITHOUT THE L.I.D ASSUMPTION
In this section, we derive the Laplace approximation when we don't make the i.idd. assumption. We will show this results in multiple computationally expensive infeasible terms in the posterior covariance, and the posterior obtained by our i.i.d. assumption keeps the computationally feasible term.
19

---

Published as a conference paper at ICLR 2026
Algorithm Turn VLM into BayesVLM
Algorithm 2 Compute Predictions
1: Input: VLM encoders (IMG, TXT], train-
1: Input: BayesVI.M, (FIMG: (IXT)
ing data
Compute embeddings using Eq. (7), i.e.,
2: for each encoder ENC E {IMG, TxT] do
Pg PMAP (
3: Compute AExe factor with Eq. (48)
Eg + (d(olMa))
A A 0f (alla))
B INC
4: Compute Bexe factor with Eq. (49)
5: end for
4:
Ph t QMAP (IExT)
6: Find À by maximising the marginal likeli-
5:
En + (ba()) T A AxT
Bixt
hood (Eq. (81))
7: (Optional) Find optimal T or set T =
Apply ProbCosine, i.e.. Compute ElSSos (g. h)] with Eq. (8)
8: for each encoder ENC E {IMG, TXT} do
7: Compute Var (Scos (g. h)) with Eq. (9)
9:
Update AENE + /T AExe + VXI
10: Update BENC < T/BBEN + VXI
Apply probit approximation (Gibbs, 1998). i.e..
11: end for
12:
Return: {(AIGG. BIMC), (ATXT: BIXT)}
8: Return: softmax
Seor
V1+ Var(S (g.h)l
We start by reformulating the InfoNCE loss. Given dataset with image-text pairs (a:!Mo xx. (XT). the InfoNCE loss is defined as LInfONCE( X LMG
X IXT
exp(1g, h.)
exp(th gi)
log
log
(18)
2m
exp(tg] hl)
2n.
exp(th 9,)
1=1
[=1
C (X ING (XT)
CE (X (XIXX X(1G)
where t learnable temperature parameter, 9 and h the unit-length normalised image and text embeddings. Evaluating this loss in practice is infeasible on billions of data points. Therefore, the common practice adopted in VLMs, such as CLIP, is to evaluate on a sufficiently large batch. IMO X [MG Then the InfoNCE loss over Specifically, denote a batch of image-text pairs as B = {X B the whole data set approximated by:
LIntoNCE( XIMG
X IXT
LINONNEE ING
x INT).
(19)
For each batch, we can view the InfoNCE loss as two separate classification losses, one over image inputs and the other text inputs. To avoid clutter, we drop the temperature parameter from now on. Looking at the loss for the image inputs LiM (X x xI we can reformulate it as follows:
IMG
B
B
CXp h log
IMG
X
IMG
X
TXT
(20)
2) 1
1=1 j exp(y. h.))
[88
log softmax (00, h1 he...., (oThii))
(21)
2 B
1=1
[8]
log softmax (Hg:)]
(22)
2
exp(a.) where (ooftmax(2)), Σ, exp(2,) is the i-th output of softmax function. We can see that the loss is equivalent to the cross-entropy loss on the following model, where label IMG is one-hot encoded vector with i-th element equal to one,
Image encoder ((-) and
use text embeddings Á
Po(a
image projection layer
9:
[Po(
to compute logit
Hg.
20

---

Published as a conference paper at ICLR 2026
Similarly, the text loss C (XX"x, X can be viewed as cross-entropy loss on the following model where label y! is one-hot encoded vector with i-th element equal to one,
Text encoder v-(+) and
use image embeddings &
xxxT
h;
Qu(a Tat)
Gh.-
text projection layer Q
Qu( [T)]
to compute logit
Under this view, VLMs trained with the InfoNCE loss can be viewed as using the following equivalent model and loss:
f(z,
3;
TXT
X IMG X .) Hgg Gh.].
is
(23)
J.MG.TXT_
log[softmax
[íg: 1:
log
(Gh: 1s
(24)
ClMO
CIMG
(X
IMG [8]
X
TXT) B
IxG.
(25)
Because data is only conditionally independent in this model, i.e.
(a: (xT) ~ p(æ: IMG
TXT
x TMO
X TXT 0),
(26)
the usual i.i.d. assumption made Bayesian models is violated. Note that performing Bayesian inference over non-i.i.d data general settings is active research field (Ralaivola al., 2009). Nevertheless, we can still consider applying the Laplace approximation in this case. Crucially, note that Laplace approximation is derived through second-order Taylor approximation of the negative log joint - log 0 p)p(() (), which only requires the negative log joint to be twice-iifferentiable function. Therefore, we can still consider the Laplace for local posterior approximation at MAP estimation. The interpretation of the underlying probabilistic model, however, may be more challenging those cases.
We will now derive the negative log likelihood Hessian for the image projection layer P. Define shorthand fP.Q(x;) = f(x! IMG x TXT
IMG X $
TxT 0), the GGN approximation for the Hessian
over image projection layer P is given as
0fp.((.))
0e, Qfp.(z.)
(27)
P2
0P
82 {p.((a))
OP
where
0fr @fp.q(x:)) 0P
0Hg: OP
oGh; OP
(28)
@2elMa. TxT 0 Hgg 8 g IMG
82 JMa.TXT aHg.
0 g JMG
(29)
82 fp.)((a))
8 JMG,TXT GC
aG
21

---

Published as a conference paper at ICLR 2026
When writing out the matrix multiplication, have:
82 G
@fp.
@fr.
2
(30)
OP
OP
(pQ(
OP
8 HHg.
(31)
0P
0 H ap
R4x18 aHg
R x18
T.T aHg.
aGhh
aHg:
+
(32)
OP
8 H OP
OP
aG aP
aHg.
C aGh,
(aGh;
g Txt aGh.
+
(33)
0P
ogg, OP
OP
8Gh,
Op
ae aHg
Here only the first term
H aP can be computed efficiently while terms red
aP
intractable computationally expensive. The approximated posterior for P obtained in Bayes corresponds to dropping the computationally expensive or infeasible terms in the exact model.
C.2 ESTIMATING THE POSTERIOR FOR BAYESVLM WITH LAPLACE APPROXIMATION
We now introduce the procedure for estimating the posterior of Bayes using the Laplace approximation in this section. We start by introducing the i.dd. assumption we made and the resulting probabilistic model for Bayes VLM in App. C.2.1. Then, we give the derivation for the posterior approximation for Bayes VLM in App. C.2.2.
C.2.1 I.I.D. ASSUMPTION AND THE RESULTING PROBABILISTIC MODEL
To efficiently estimate the approximated posterior using the Laplace approximation and obtain clear probabilistic model underlying it, we assume two independent probabilistic models, one for each modality. Specifically, for each modality, assume data are i..d. given the observations from the other modality:
x
JMG lid.
p(
IMG
X TXT
D
TXT lid.
p( TXT
X IMG 0).
(i.i.d. assumption)
Following this assumption, the image encoder o(-) and text encoder (i)) will become independent, and image projection layer P and text projection layer Q will become independent well:
<6-) IL wh)), Q.
(Consequence from i.i.d. assumption)
Under these assumptions, we can untangle the interaction between two modalities and approximate their respective likelihoods categorical distributions.
When the modalities become independent, for image input x:! IMG we can only look at the image loss defined as
1
B8
exp(g h4)
IMG X
IMG
X
TXT
log
(34)
exp(0. h,)
128
1
log softmax ( hi, ". ha..... . hhBi))
(35)
2 2|
[8]
(Hg:)]
log softmax
(36)
2 2
Σ, exp(2y) the i-th output of sofumax function. This corresponds to cross-entropy loss on the following model, where label y is a one-hot encoded vector with i-th element equal one,
where [so])max(2)), A exp(x;)
22

---

Published as a conference paper at ICLR 2026
Image encoder 6(-) and
given text embeddings H
IMO x
image projection layer
gi
[Po
Po(m
compute logit
Hg..
Therefore, for image input, the corresponding model is
f(o!a0
X TXT 0) Hg..
(37)
with the corresponding log likelihood
log p( X-MM
TxT 0) log
p x
IMG
X ,0)
(38)
log i=1
II
softmax
(Hg:)
(39)
Similarly, for text input, the corresponding model is
X IMG 0)
Ghi,
(40)
with the corresponding log likelihood
log 1 1xT
X 0) log
p
TXT
IMG .))
(41)
im
log I
softmax
Gh. ],
(42)
Why is this still a reasonable approximation? For VLMs, it is important to capture interactions between modalities, and assuming independence seems problematic at first. However, using local post-hoc posterior estimation through the Laplace approximation, we are effectively introducing an independence conditionally on the MAP estimate of the (joint) contrastive loss. Thus, crucially, even though we assume independence between modalities, we can still capture interactions between modalities. Note that this assumption also important for computational reasons, as it helps us derive computationally efficient approach.
C.2.2 POSTERIOR APPROXIMATION WITH LA
Now that we have a well-defined probabilistic model and likelihood, we apply the Laplace approxi- mation to it.
Why only treat P and Q probabilistically In the Laplace approximation, for the posterior covariance, need to compute the Hessian of the log likelihood. This is computationally infeasible for large models and large datasets, and a common approximation is Generalised Gauss-Newton (GGN) approximation (Schraudolph, 2002). Use shorthand fo( for the model and denote the log likelihood as E(y, fo(x)), the GGN approximates to the Hessian is given by
0fe(x) 00
Plly.fo(a) Ofe(z)
Velly.fo())) GG
(43)
(x)
00
Note that in GGN approximation, need to compute the Jacobian of the model output w.r.t. to 0fg(m) This computationally infeasible for image and text encoders due
the model parameters 00 to the large number of output dimensions. For image projection and text projection, this challenge can be bypassed as the Jacobian can be obtained analytically. Therefore, we treat the vision and image encoder fixed and apply the Laplace approximation only for the image projection and text projection P and Q.
KFAC GGN approximation to Hessian To estimate the Hessian of the log likelihood for P and approximate curvature (KFAC), which expresses the Hessian as a
Q. we use Kronecker-factoe Kronecker product of two smaller matrices. This significantly reduces computational and memory
23

---

Published as a conference paper at ICLR 2026
costs while preserving richer posterior structure than diagonal approximations. Following (Ritter et al., 2018), the KFAC GGN approximation for -Vp log p(X IMG
(-MM
TXT P) is
(a IMG (d(x;
Jim (x: IMG T AIMo Jma (a IMG)
(44)
Atso
Bisc
and the KFAC GGN approximation for
V22 log Q) is
n
(1)
IXT)
69
J (x ) AxxT JixT(an (x.
(45)
ATsT
B
where JIMG (2 (M)
aH and AIMG diag( πT) π with Te
exp(fe) E. exple2) 9 he =: fe-
Ög:
As estimating the Kronecker factors over billions data is computationally infeasible, following (Ritter al., 2018), we leverage subset of the data and include pseudo-data count compensate for the reduced sample size. Putting everything together, the posterior covariance over P and Q are approximated
ENG (T(AING & BIMG) + AI)-1 ~ (VF + vXT) &
(VT BING + vxI)
(46)
A
B
180
INc
Exxr (T(Aix @ BIx)) +11 ~ (VFAix + vXT)" 8 (VFBINT + vXT)= (47)
B
where the respective factors are given as:
A
oi ( (o:
vn
ATxT
(x TXT (2)
(48)
BING
J Jisa JMG AImG Jima (ar !MG)
Bixt
Jixr(x.XT)) ATxr (xxr(xx
(49)
Jacobian computation Here derive the Jacobians (x and (xx TXT) used in the KFAC GGN approximation.
Recall ğ. and h, denote the normalized image and text embedding, respectively. Let H denote the matrix of normalized text embeddings with h, its columns and G the matrix of normalized image embeddings with 9. as its columns. Then. for the InfoNCE likelihood, which depends on the dot product between the normalised embedding in the batch, compute the Jacobian for the image
24

---

Published as a conference paper at ICLR 2026
encoder as follows:
HH.:
(2:
(z!!ac)
(50)
IMG
0gi
8 gi
(51)
8gi |1
H
|9
9
Dg.
(52)
||
||2
|
||
9-9: gg.11
-H
(53)
|9:11
||
9:9,
H
(54)
||g.||
||9:13
Analogously, we obtain the Jacobian for the text encoder given as:
J InfoNCE
TXT)
G
1
h
(55)
h:
[h: ||
For SigLIP, we obtain the following Jacobians:
J SigLIP IMG
(a:
0g:
9:9,
(56)
0g.
[1g.1
11 ||
and
SigL.IP
(xt
TXT
ah,
1 hhal||
h.h!
(57)
|h
Hessian of likelihood w.r.f. model output computation Here we derive the loss Hessian w.r.t. model output AJM and Axx-- For InfoNCE loss used in CLIP, the zero-shot classifier induced computes unnormalised logits for each class c, represented by 9. he =: fc. By applying the softmax
exp(f.) The likelihood Hessian function, calculate the probabilities for each class me Σv exp(S2) of the cross-entropy loss for this classifier is represented by
A
InfoCCE = diag( (T) π IMG
(58)
Similarly, the likelihood Hessian for the text encoder follows analogous principles in the text-to- image direction. For more detailed derivation of the likelihood Hessian, refer (Rasmussen & Williams, 2006, Ch. 3.5). Rearranging terms in the analytical expression for J A IMG facilitates -fficceffi computation of the GGN approximation.
InfoNCE JIMG
The SigLIP loss is defined as follows
LSiLLIP (X IMG X TXT
(59)
72
n
1
log
(60)
n
1
h; + b))
log o (a,z).
(61)
n
h))
where (a) Itc-a denotes the sigmoid function, and aıj 2iy (tg. h, + b). with labels 2i (-1, 1), learnalle temperature sealing parameter i, and learnable bias b.
25

---

Published as a conference paper at ICLR 2026
In order to derive the loss Hessian A SigLIP
first derive the component wise loss gradient of f::
(g:. he))
k 0
(62)
0gk
0
i=k 0
(g..
h2z)
log (ais)
(63)
0ÿk
0gk
1 (a(5)) daii
(64)
a(a.)) 0asj ag:
(o(a;)) 1) 2i3 thj.
(65)
which we utilise to derive the womponent-wise loss Hessian
0²
[gg. h,) iæk
(66)
0g
0
i=k ① (olas))zsth; - 2,yths)
(gg. h₂))
(67)
0g
0gi
=2yth @a(a;;) dasj
(68)
daij 00
(a((a))) (1 a (ai))) h..
(69)
Finally, the likelihood Hessian for the SigLIP loss fSigLIP can be expressed as
82
A SigLIP IMG
C(g
him)
(70)
0g.0g,
0²
(g: h2)
(71)
0g.
(aii)) (1 o(ai))) h
(72)
for the image encoder and as
t2 x
A
Li Txt
a(aig) (1- o(ar))
(73)
n
for the text encoder.
Efficient Hessian Computation At first glance, computing the Hessian appears prohibitively expensive: the loss Hessian A has shape [B] x [B] with [B] 32k, while the embedding dimension is much smaller (d & 512). Forming A explicitly is therefore impractical. By exploiting its low-rank structure and contracting with the Jacobians, however, the computation can be carried out efficiently without ever materializing A, making the GGN approximation feasible even for large batches. For example, for the image encoder with the InfoNCE loss in CLIP, the GGN block simplifies to
JIMO (z) (Ma)
A InfoNCE IMG
(74)
T
h
G (diag(T) π
G
(75)
hhil
Thill
||
CR
B xa
MERix
MERoxd
M (GT diag(T))
GT कम G) M
(76)
M
GT T) M
(77)
ER4x
ERIGixe xd
ERI xa
where denotes row-wise scaling of & by the vector
26

---

Published as a conference paper at ICLR 2026
Marginal likelihood To learn the prior precision parameter X, we follow prior work (e.g., (Immer et al., 2021)) and optimise the log marginal likelihood within each probabilistic model. For the image projection layer P, denote the prior and posterior as below:
prior: (O ATMA
(78)
posterior (MMPA ΣIMG)
(79)
The marginal likelihood
log p IIM
X Ixt)
log pl pla
IMG
XIxI PMAP)
(80)
1
2 (PMAP log det (IMa) + log (eM(AA
(81)
We can learn the prior precision Alse using gradient-based optimisation.
Distribution over image and vision features For completeness, we will briefly derive the distribution over image and vision features. In particular, for the image encoder let P ~ MW BIMG AMa).. then:
Po(x
(82)
with (M)
MN (PMAAO(a IMG), Binlo ((xx T "A A ((rlNa))))) 9 ~N (PMAoo(( IMO), (#(ax A A A ((x))a)
B Binl))
(83)
C.3 DISTRIBUTION OVER COSINE SIMILARITIES
For the derivation of the distribution over cosine similarities, first recall the definition of the cosine h
similarity between two vectors, g and h, which is given Scos (g, h) || Now, let g and denote random vectors for the image and text embeddings, respectively. Further, let us assume that their distribution follows a Gaussian distribution with mean Hy = (Ig.1....: Hggd) and μh (1A,1,,.. d) and diagonal covariance structure, i.e., Eg = diag(o) g.1............ ,0.....,,,00, g.a) and Σn = d ...... R.a))
Then the expected value of the cosine similarity is:
Eig h
E [[Seos (g. hi]
(84)
EE [ |E ht
Σ Pgai
(85)
Elllg [|]||
Note that computing [][[=|[] is intractable, and we, therefore, bound the expected value by application of the triangle inequality, i.e.,
Σ 22 + 22..
(86)
where we use the fact that E[x²] 2 + 02. Consequently, we obtain an approximation to the expected value of the cosine similarity given by:
Σd Pggilini
ElScos (g, h))
(87)
VE, 12 + +0g VEiih, +o.
Next, we will derive the second moment (variance) of the cosine similarity of two random vectors. First, note that the variance be written the difference between two expectations, i.e.
Var Seos (g. h)) ElSeas (g. (g.h)² h) E[Scos h))
(88)
27

---

Published as a conference paper at ICLR 2026
Target Space
Figure 7: Illustration of targeted support set selection. We aim to select an informative support set that reduces the uncertainty over the predictions on the query set o. Only focusing on the epistemic uncertainties would not lead to a good selection, candidates with high epistemic uncertainty. Hence, we target the selection process.
would select uninformative support
where the second expectation corresponds to:
Ex Pgilthii)
E S[Sos (g. h))²
(89)
E, 1g + 0g Σ, 2h, + hii
Next we can obtain E[Scos (g, h)2² for which we will use the fact that E[+²] = 22 - of again, i.e.
[[[[ h)
E S[ooo (2, = Σ, 1g o0 E, M2 to
(90)
where
Blgg [h)? = ΣΣ Hi
(91)
Σ Gg + tgg og
(92)
Henceforth, we obtain the variance:
Σ, (o + +,h,)) + o u
Var[Scos h)]
(93)
E, 2g + +o E, 1h, + oh.
D ACTIVE LEARNING DETAILS
provide additional details on our active learning setup. Active learning provides a natural setting to evaluate the quality of uncertainty estimates, it relies selecting informative samples based on predictive uncertainty. We assess BayesVLM Bayesian active learning, combined with adaptive target region selection. Concretely, given query set Mest {xt ] Tist of unseen samples with unknown class labels, our goal is to select a support set Vesst reduced. To this end, we first target the selection process toward the predictive distribution of the query set, and then select support candidates based on their estimated influence on predictive or model uncertainty.
this setting using acquisition functions from
[==
{(z,, My J
labeled examples such that predictive uncertainty
We detail our method in three parts: App. D.I describes how we reduce the candidate pool by selecting samples that align with the target distribution; App. D.2 outlines the acquisition functions used for (targeted) active fine-tumie tuning; and App. D.3 explains how we update the Laplace approximation in an online fashion during the EPIG acquisition process.
D. TARGETED SELECTION
To target the active learning process towards relevant areas in the data space, perform a k-nearest neighbours (k-NN) search around the test data. The main idea behind our adaptive targeted region selection is illustrated in Fig. 7.
28

---

Published as a conference paper at ICLR 2026
Stage 1: 1-NN Selection
Stage 2: 2-NN Selection
Already occupied
feature dimension 2
feature dimension 2
feature dimension
feature dimension 1
Figure Illustration of the nearest neighbour baasleuu--baed support set selection for adaptive targeted selection. The circles show test data points with uncertainty scores depicted through their colours: high, medium, low, For each test datum we find the k = 1 nearest neighbour from the support set candidates If the k = 1 nearest neighbour is already selected, we increase l for those with occupied neighbours and choose the second nearest neighbour, i.e., A = 2. This recursion continues until every test datum has selected support set candidate. The selected candidates Note that in the case the blue test datum, the closest support set candidate has already been chosen by the yellow, and hence the second closest candidate is selected in the second stage.
shown in coloured circles.
Specifically, we greedily acquire an intermediate candidate set T S Durin using k-NN selection based on the test set Diest- For this, we need to compute a metric comparing the random feature projections. We assessed two different ways, first by computing the 2- 2-Wasserstein distance between the distributions of the embeddings and the second by computing the expected cosine similarity based on App. C.3. Recall that for multivariate Gaussian distributions, the 2- 2-ssssesteein distance exists in closed-form and is given as (1), Σ1), N/ (H E2))
|1 - 12l1 + tr Σ1 + Σ, 2(E - /31111111)
(94)
where lll denotes the Euclidean norm, tr(-) is the trace operator, and 1/2 is the matrix square root of Σ. As computing the Wasserstein distance exactly is computationally and memory intensive due to the matrix square root, we approximate it by assuming both distributions be isotropic. Hence, simplifying to W3 (N(H), : 11),N(M Σz))
Σ (mi, 22,)) + +i, 0² 1,
(95)
where Σ, diag(oj, .... a.a) and Eg is given respectively.
Based selected metric, select the training samples closest to the test set in the joint embedding space, resulting in:
T U Nu(g", Dunin), ET
(96)
with Nu (g", Drrain) denoting the set of k-nearest neighbours of g" in the training set Dinin- To ensure that we select distinct data points for each test sample, we perform an iterative search in which we discard already selected training samples and iteratively increase the search radius until & distinct samples are found for each test datum. This process is illustrated in Fig. 8.
D.2 ACQUISITION FUNCTIONS
Given labelled pool Durain and unlabelled target set Xiest {x (x, y) Dust), the goal is to select m maximally informative samples from Dirain to reduce predictive uncertainty on & In this section, we provide detailed explanation of the acquisition functions used for this purpose.
29

---

Published as a conference paper at ICLR 2026
Naive random For the naive random acquisition function, randomly sample data points from the train set Daiin to form the support set Sip.
Targeted random For the targeted random acquisition function, randomly sample data points from the unlabelled test set Rea to form an intermediate support set T According to App. D.I, we then select the nearest neighbours T from the training set Dirain based the cosine similarity of the normalized image embeddings to form the support set T..D-
Targeted maximum entropy For the entropy acquisition function, we compute the predictive entropy H(v |
x for each data point æi E Xee and select the m data points with the highest entropy. We use the predictive entropy on the MAP estimate of the model parameters to estimate the predictive entropy of the model:
H (y/ x, OMAP)
Σ ply = c/a, 0MAP) log = c/a,, MMAP)
(97)
According to App. D.1, we then select the most similar data points from Xrain to form the support set Twentropy-
BALD We compute the BALD score (Houlsby et al., 2011) for each data point in A and select the m data points with the highest score. The score approximated using nested Monte Carlo sampling, as in (Houlsby et al., 10.1).
(aLD(x):
(98)
= m)llm) [H (p(0)) - (p(e x. x.y))]
(99)
= E H (p(y x, 0)) - 7. (p(y x. D))]
(100)
Targeted BALD We compute the BALD score (Eq. (100)) for each data point an Yes and select the m data points with the highest score. According to App. D.I, we then select the most similar data points from Maain to form the support set T-BALD-
The Expected Predictive Information Gain (EPIG) score (Bickford Smith et al., 2023) calcu- lates the expected mutual information between the model parameters and the predictive distribution resulting from the acquisition of training data point. This method is specifically designed to target relevant information, eliminating the need for k-nearest neighbour search typically used in other acquisition functions. The EPIG score is given by
EPIG
EP)
Ep. (zo (p())))))) H (p(y" x=)) - H (p(v" ,,,,,,,) 2,,)))
(101)
(xr) [DLL (p(y.)) 8² xx x²) ll x)p(1" xx))]
(102)
Σ Σ ply. v" x xx,x") log ply. xx, x²)
(xr)
(103)
ply x x))(1" x+)
EY
where p=(x") denotes the target input distribution. The EPIG score is approximated using Monte Carlo sampling, as detailed in(Bickford Smith et al., 2023). For the EPIG selection, we perform online updates to the model weights using the online Laplace as described in App. D.3.
D.3 ONLINE LAPLACE APPROXIMATION
We use an online Laplace approximation efficiently update the posterior distribution over the image projection matrix P during active learning. Instead of recomputing the posterior from scratch after each support set update, we incrementally refine both the MAP estimate and the Kronecker-fe frottreer-factored Hessian approximation using the newly selected datapoint. Concretely, we perform gradient step to update PMAP, and adjust the Kronecker factors A and BIsG based on the contribution the new sample. This yields computationally efficient approximation to the posterior over P conditioned on the growing support set. Additionally, the prior precision can optionally be re-estimated after each update step, as commonly done in online Laplace methods (Immer et al., 2021; Lin et al., 2023). In the following, we outline the structure of the Laplace approximation and describe how is updated online during EPIG-based support set construction.
30

---

Published as a conference paper at ICLR 2026
Recall that obtain from post-hoe Laplace approximation the Kronecker factorized Hessian approximation HIM (VT AIGG + VXI) @ TVTBBIG vXT) with
A AIGG
1
Σ
( (aaa))@(z) (x( T and
(104)
vn
1=1
BIGG
( (x)
Alaa Juna (x (MC).
(105)
approximating a posterior distribution over the projection weights:
P (MMPPMPP B BING A
(106)
Q ~ MM B B ATxT)
(107)
with A, B denoting the Kronecker factors after applying and X.
Further, utilizing App. C.3 in combination with the generalized probit approximation (as deseribed, for instance, in (Daxherger et al., 2021)), we obtain an analytical form for the predictive posterior distribution ply æ. D) of our few-shot classifier.
Our goal with EPIG is to iteratively construct support set Tr, where t denotes the current number of selected training data points. We construct T, by greedily selecting the training datum that maximises the expected information gain on the predictive distribution in the target domain:
1t+11 = arg max EPIG(( (T))
(108)
D
arg max Bp. (p((ll)) H (p(1) |x" 7 (p x", To ()))
(109)
DEii
and obtain the corresponding label Ve+1, forming support set T++1 Te U (
The integration of {( 11414) into the few-shot training set changes the posterior distribution over the image projection. To obtain the updated posterior distribution
P T MM B 1 ING.++1).
(110)
we utilise the following online updates to the projection weights and the Laplace approximation:
PMARA1+1 PMΛP, p X
(111)
Vn +f A.MM + BA
AIMG,41
(112)
Vn+t+I BIMG +t BIMo,t + -BBm...
(113)
Vn+
where 7 2 0 and B 2 0 are hyperparameters and
A ( ((
IMG
(114)
B JIna IMQ]
(2/41) AIMaJING (
(115)
E EXPERIMENTAL DETAILS
This section details the experimental setup used our study. In App. E, we describe the pre-trained dision-language models and checkpoints used. App. E.2 explains how computed the Hessian matrices required for the Laplace approximation. In App. E.3, we outline how we selected the Laplace parameters, such the pseudo-data count and prior precision A. App. E.4 describes active learning setup, including dataset preparation, selection strategies, and training hyperparameters (see Table 6).
We run experiments on compute cluster with NVIDIA P100 16GB, V100 32 GB, and A100 80GB GPUs. We used V100 or A100 GPUs for the Huge model variants and the ImageNet experiments.
31

---

Published as a conference paper at ICLR 2026
E. PRE-TRAINED VISION-LANGUAGE MODELS
In this work, we used the OpenCLIP (Ilharco et al., 2021) implementations of CLIP (Radford et al., 2021), which was published under the MIT license. We present additional experimental results on the HuggingFace implementation of SigI.IP (Zhai et al., 2023), which was originally published under the Apache2 license.
For PCME+ we used the CLIP ViT-B/16 checkpoint provided by the authors with uncertainty adapter trained on CC-3M (Sharma al., 2018), CC- (Changpinyo et al., 2021), and Redcaps (Desai et al., 2021). For ProLIP, we used the VIT-B/16 model checkpoint released by the authors at ht https://gáttie com/ aaaver-ai/prolle which shares the same backbone architecture.
E.2 ESTIMATION OF THE HESSIAN MATRICES
We estimated the Hessians separately for the CLIP image and text encoders using the pre-training dataset LAION-4000 --00 (Schuhmann et al., 2022) published under MIT license. For this estimation, we randomly sampled a subset of 327.680 data points. The pre-training dataset was filtered to exclude NSFW content. For the Laplace approximation, we used the GGN approximation of the Hessian matrices as described in App. C.2 and estimated the covariance matrices A and B for the image and text encoders.
E.3 ESTIMATION OF THE HESSIAN PARAMETERS
To ensure that both zero-shot and active learning experiments rely on a well-calibrated posterior covariance, we estimated the pseudo-data count (Ritter al., 2018), r. by performing grid search over values in E [1, 5, 10, 15...., 200]. In the active learning experiments, the step size was reduced 2. The optimal value was selected by minimizing the negative log predictive density (NLPD) on a random subset of ImageNet consisting of 100 classes and 1097 test data points in total as a proxy. App. E.3 presents two plots illustrating the NLPD function of the pseudo-dar count for SigLIP-Basse and -LL-eBase, respectively. Once this optimal was identified, we further optimised the prior precision, X, using the marginal likelihood the MAION-400X (Schuhmann al., 2022) dataset.
1.6
NLPD +
..88
.656
1.08
NUPD
1.08
pscudo data-count
Figure 9: Grid search the pseudo-data count parameter for CLIP-Bass (-) and SigLIP-Base -). The optimal NLPD for CLIP-Bass is identified at T = 10, while the optimal NLPD for SigL.P-Base is identified at 100.
E.4 ACTIVE LEARNING EXPERIMENTS
We conducted active learning experiments the OfficeHome data set (Venkateswara al., 2017), which consists of the domains: art, clipart, product, and real-world, as well as on the ImageNet dataset with the domains ImageNet-R (Hendrycks al., 2021) and ImageNet-Sketch (Wang al., 2019). For
32

---

Published as a conference paper at ICLR 2026
these experiments, all training sets from the respective domains were combined into single, large training set, and the projection layer of either CLIP or SigLIP was fine-tuned for a specific domain. Data selection
performed based the acquisition functions described in App. D.2. Performance was evaluated at the checkpoint corresponding to the lowest NI.DD on domain-specific validation set. We also performed grid search for the online learning parameters of the EPIG acquisition rule, selecting the EPIG learning rate from the range [le-5, le-4, le-3, 1e-2] and the EPIG Hessian update scale from [1, 10, 100, 1000], based on the NLPD on the domain-specife validation set. Details on the training hyperparameter settings are given in Table 6.
Table Active fine-tuning hyperparameters.
config
value
optimiser
A
learning
1e-5
weight decay
5e-2
optimiser momentum
31. 82 0.9, 0.999
batch size
32
epochs
100
F ADDITIONAL RESULTS
In this Appendix, we provide additional results support the findings in the main paper. Specifically, we detail (i) the approximation quality of the Gaussian approximation to the distribution over cosine similarities in App. F.I, (ii) the active learning experiments in App. F.2, (iii) the ablation of the k-NN distance metric in App. F.3, (iv) the influence of the number of data points used for Hessian estimation in App. F.4, (v) the runtime overhead and inference costs of Bayes compared the baselines in App. F.5, and (vi) additional zero-shot learning results in App. F.7 to demonstrate the generality our approach.
F.1 APPROXIMATION QUALITY
To assess the approximation quality of the Gaussian approximation the distribution over cosine similarities, we generated 500 samples for the image and text feature distributions for a given input. For the resulting samples, then computed the respective cosine similarity for each pair and performed kernel density estimation with Gaussian kernel and length scale of 0.3 on the similarity scores. We added increasing shifts to the distribution mean evaluate the change in the approximation quality under varying cosine similarity values. The results are depicted in Fig. 10. We can observe that our approximation through ProbCosine results qualitatively in low approximation error.
0.3
Density
0.2
-10
5
10
15
20
25
30
35
Cosine Similarity
Figure 10: Approximation quality the Gaussian approximation (ProbCosine) (---) to the distribution over cosine similarities compared KDE over samples (-) for image-text pairs with increasing Euclidean distance between their feature projection means (Jag: pr).
33

---

Published as a conference paper at ICLR 2026
F.2 ACTIVE LEARNING EXPERIMENTS
We report the active learning results for CLIP-Huge in Fig. 13 and for SigLIP-Base in Fig. 14. For CLIP-Huge, we observe that active learning based on our post-hoc uncertainties consistently improves upon random and antropy-based selection. For SigLIP- -Bsee we observe significant improvements in terms of accuracy and NLPD on the ImageNet version with active learning based on our post-hoc uncertainties.
F.3 ABLATION OF THE K-NN DISTANCE METRIC
We performed ablation study on the k-NN distance metric for our proposed targeted selection in Sec. 3.3 while fixing the online LA learning rate = 1c-4 and the online LA pseudo-date count 10. We evaluate performance using distance metrics: Wasserstein and cosine similarity. Results are reported for EPIG and BALD, with Wasserstein (solid lines) and cosine (dashed lines) metrics. While Wasserstein- based k-NN selection demonstrates improved performance for datasets such as OfficeHome--rt and ImageNet-R, no clear trend is observed across the other datasets.
F.4 NUMBER OF DATA POINTS FOR HESSIAN ESTIMATION
We assessed the influence of the number data points used to estimate the Hessian by estimating the trace of the Hessian using varying number of data points for 10 random subsets of the Laion400m data set. Fluctuations of the estimated trace can be understood as an indicator that the Hessian estimates are unreliable. Fig. 11 shows that the trace of both the Hessian over the image projection and the text projection quickly converges to stable value with low fluctuations, as seen by the low variance and stable mean. Moreover, the results indicate that 10 mani-bachhes suffice obtain reliable estimate of the Hessians.
E
110
2.54
2.92
3.08
3.06
2.78
6 data points
14
10.
12
I. batch size
data points JEX butch size
Figure 11: Hessian (Trace) vs. number samples. Error bars indicate =1 standard deviation over 10 random subsets of Laion400m.
F.5 RUNTIME OVERHEAD
We compared our approach against the vanilla CLIP. We report runtimes on CIFAR- 100 with CLIP- Huge/-Base Tesla V100 with 3 warmup steps and averaged over 1000 runs (batch size in Table 7. Indicating minimal computational overhead. In addition, we present inference costs in GFLOPs, comparing the original VLM (deterministic) against TTA and Bayes in Table 8. We find that BayesVLM results in comparable inference costs, while TTA has an 80-fold increase in the inference cost.
Table 7: Average runtime measured in seconds on CIFAR-100.
Model
Vanilla
BayesVLM
rel. increase
CLIP-Huge
43.8178 9.4498
43,9712 9,8929
0.35% 4.69%
CLIP- Bas
34

---

Published as a conference paper at ICLR 2026
Table Inference computational cost per image on CIFAR-100 100 (in GFLOPs 10").
Model
Vanilla
TTA (FARINA ET AI.,, 2024)
BAYESVLM
CLIP- Base
8.83
687.78
8.84
CLIP-Lage
162.06
12638.00
162.07
CLip-Huge 334.71
26098.06
334.72
SigLIP-Bans
47.00
3652.05
47.03
F.6 INTERPRETING PROBABILISTIC COSINE SIMILARITIES
We qualitatively assessed the distribution obtained by ProbCosine on randomly selected test example from the OfficeHome clipart domain, evaluating the mean and variance of the cosine similarity under increasing corruption in both image and text domains. Text corruption was introduced by randomly replacing characters with 'x', and image corruption by randomly adding grey squares. Fig. 12 shows the mean and variance of cosine similarities corruption increases. We observe that the expected cosine similarity generally decreases and variance increases with more corruption, indicating that approximation effectively captures model uncertainties under distribution shift. Note that observe a slight increase in the cosine similarity after one character has been replaced, indicating that performing predictions solely the expected cosine similarity be problematic. In this case, the variance over cosine similarities can capture the change in the input, highlighting the importance of capturing and propagating the model uncertainties.
Expected Value
Variance
image of cup
Ax image of cup
cup
Ax imacesof a cup
Ax imasesef cup
As imasesd axcup
17:33
30,96
0.72
1.15
Illustration ProbCosine under increasing corruption. The mean similarity decreases and variance increases with higher levels of corruption, demonstrating effective uncertainty estimation under distribution shift.
Figure 12:
F.7 ADDITIONAL ZERO-SHOT RESULTS
Here, we report additional zero-shot results for CLiP-huge and SigLIP-base in Table 9 and Table 10, respectively. We also report the results for applying our ProbCosine the ProLIP (Chun al., 2025) model in Table 11.
CLIP-huge results In Table BayesVLM consistently matches outperforms baseline methods in terms of accuracy (ACC), negative log predictive density (NLPD), and expected calibration error (ECE). We note that TTA achieves worse ACC than CLIP on CIFAR- 10 and -100, which we believe due to test-time augmentation in (Farina et al., 2024) not being optimised for small images (32x32), where the classes could become unrecognisable with additional cropping in the augmentation.
Sisse results In Table 10, for SigLIP, Bayes still yields improvements several benchmarks (despite using proxy data for Hessian estimation, as the original training set is not publicly available), demonstrating robustness to such settings. Despite mismatched training and Hessian estimation datasets, Bayes VLM remains competitive, especially on CIFAR-I0, 10, UCF101, and SUN397, effectively improving calibration without sacrificing predictive performance. We note that TTA achieves better ACC on some benchmarks, which is sensible since the model will get better at predicting the correct class average with more chances (augmentations).
35

---

Published as a conference paper at ICLR 2026
ProbCosine combined with ProLIP results In Table 11, we observe that applying ProbCosine to ProLIP improves zero-shot performance across classification benchmarks and metrics.
Table 9: Zero-shot Results: Quantitative evaluation of uncertainty estimation across multiple data
sets in the zero-shot setting for the OpenCIIP 14 model. Our proposed Bayes consistently outperforms baseline methods across accuracy (ACC, in %6), negative log predictive density (NLPD), and expected calibration error (ECE, in %) metrics.
Metrics
Mottadk
FLowERs-
Foon-101
CIFAR-10
CIFAR-100
IMAGENIT-R
SUN897
CLIV (Raztoon al.. 2011
56. 06.00
05.57:1335
76677:114255
853128 496
71,881330
ACC1
CLIP (tmp. scaling)
TTA (herin 2024)
71 25es
87.1 11:e
60. 57 ase
W2.85+ soe
4915
87 (7.54 2600
85.01s9 sno
70 10
GM a0
71 598 sier
71 88.9 203.
Ro
20-0cen
sgn
71:124 d5.
(Kaallod X01)
1.2540007
0.30040007
0.3848
1:30
NI.D
TTA (Fainu al. 2024)
CLIP (taug walling)
1.740002
9044
1.15anooss
0.50
1.20-00300
1.84
com
BoveVVLM
L.2200015
0.15
0.87488055
0.57
0.58882054
(Itadiood N01)
9:47
0.37
5773
10.79
5,0
BCE
CLIP (trmp. scaling))
TEA
2024)
11.96
3.77
1,35
1:21
3-92
2,55
11.87
1.45
10.53
IoyesVLM
4.05
Table 10: Zero-shot results:
Quantitative evaluation of uncertainty estimation across multiple data sets in the zero-shot setting for the SigLIP-Bass model (Zhai et al., 2023). Our proposed Bayes VLM often performs competitively to baseline methods across accuracy (ACC, in %), negative log predictive density (NLPD), and expected calibration error (ECE, in %) metrics.
PooD-101
CIFAR-10
CIFAR-100
IMKOENET
SigLIP (2hai
80.71:0.5e
ACC1
TIA
SiaLIP (oep. scaling) 2024)
a
88.51a8.1994
97.06180254
20
89.7Lunu00
00:14:17992
50.00 20:61:07700
00.92.01342
BayesVLM
22:0
80.72um0807
50.09:5.590
SigLIP (2lmi al., 2023)
0.38annee.
0.41anatte
1.90033443
1:12
NLED
TIA
SiaLIP (eepp scaling)
2024)
0.
0.366
0.43umause
a.37
a.77
1.88030499
1.18
BavesVLM
0.20anume:
21:8006i
188834088
0.41am0114
1.12.
Sig
2023)
4.31
1.50
1.50
12.72
3.82
ECE
TTA cial, 2024)
Si,LIP (emp wallng)
6:38
4.56
0.72
5.54
1.50
4.85
18.47
2,83
BayesVLM
4,57
3.40
2.79
1.14
Table 11: Can ProbCosine improve the zero-shot performance of pre-trained probabilistic models? Yes. Applying ProbCosine to the ProLIP (Chun et al., 2025) model improves zero-shot performance across classification benchmarks and metrics.
Metric
Methods
PLowERi-102
Poom-101 101
CIFAR-10
CIFAR-100
IMACENET-R
UCP101
SUN 197
ACC
Moan (Chun et 2023)
ProbCosine (Eq. (10g
77 SULuee
77: 74
DO 20.39..0
80 35
96
82.41-1308
88.48-1304
$491:9c0
69 39
G5. B0
65. 60.8
NLPD
ProtCosine (Eq. (10g
Marn (Chuus et 30255
1 18
1.5000m
O 0.330ee))
0.54in0es
0.100
0.50
10.200004
1.24 1/24
1.25
BCE
Mean (Chum et 2025)
ProkCosine (Fa.
3.60
0.79
2.57
0.60
0.76
1.40
3.58
3.53
1.08
2
5.90
4.73
7.00
F.8 ROBUSTNESS WRT TO PSEUDO-DATA COUNT
To examine the influence of the pseudo-data count 7 on BayesVLM, we evaluated Bayes with the CLIP-Base configuration on four datasets (FLOWERS-IV 102, FOOD-101, CIFAR- CIFAR- 100) while varying 7 E {1. 5, 7, 7. For each setting, we report classification accuracy (ACC 1). negative log predictive density (NLPD J), and expected calibration error (ECE 4) in Table 12.
The results show that performance is stable across the tested range of pseudo data counts. Accu- racy and calibration metrics vary only slightly with t, indicating that the method is robust this hyperparameter. In the main manuscript, we used 4 as the default setting. We find that slight improvements can be observed when moving the pseudo-date data count away from 7 = 4, but the value found the proxy dataset (r 4) provides reasonable performance in general.
F.9 ROBUSTNESS WRT THE NUMBER OF NEGATIVE SAMPLES
To assess the impact of negative samples on the likelihood approximation, we vary the batch size K (327688 8192, 2048) and estimate the posterior from 1-5 random batches, reporting mean and standard deviation over five trials. Because the posterior depends on the number of negatives only
through the Hessian B- factor (cf. Eq. (5)), present the normalised trace (t((Bxxx)) (u((sxx)]
Ideally, this
36

---

Published as a conference paper at ICLR 2026
OH-Art
OHI-ClipArt
OH--Product
IN-R
IN-Sketch
0.98
0.88
Weighted ACC
0.91
0.07
0.57
0.87
0.9
0.90
0.96
0.06
60
100
150
200
196
60
100
150
700
150
200
195
100
150
200
18h
89
77
200
0.7
0.12
0.55
0.18
NL PD
0.35
0.6
0.1
0.5
0.16
0.08
0.14
20.5
0.06
0.45
100
150
2000
150
200
100
150
2000
100
150
200
100
150
200
Subset Size
Subset Size
Subset Size
Subset Size
Subset Size
Figure 13: Active learning results (CLIP Huge): We present results for EPIG (-), BALD (-),
Entropy (targeted) (-), Entropy (-), Random selection (targeted) (-), Random selection (---) on the OfficeHome dataset (OH) and ImageNet variants (IN). We observe that active learning based
on post-hoc uncertainties consistently improves upon random and entropy-based selection.
OH-Art
OH-CCipArt
OH-Product
IN-R
IN-Sketch
0.94
0.97
Weighted ACC
0,86
0.89
0.04
52
08
0.97
0.91
0.89
0.02
0.07
0.98
185
60
100
150
100
130
100
150
200
160
75
100
150
200
150
200
0.15
0.22
0.07
0.5
NI SLPD
0.17
0.3
0.06
0.05
0.45
0.16
75
100
150
200
61
76
100
160
200
150
200
60
100
150
200
100
200
Subsel Size
Subsel Size
Subset Size
Subset Size
Subeet Size
Figure 14: Active learning results (SigLIP Base): We present results for EPIG (-), ) BALD (-).
Entropy (targeted) (-), Entropy (-), Random selection (targeted) (-), Random selection (---)
on the OfficeHome dataset (OH) and ImageNet variants (IN).
ratio equals one with variance. Results for the image and text surrogate models shown in Fig. 16a and Fig. 16b, respectively.
The batch size 32768 yields mean values for all numbers of random batches and maintains low standard deviations, indicating reliable estimates for both modalities.
F. ADDITIONAL EXPERIMENTS ON CC12M
We find that Bayes VLM provided robust uncertainty estimates for CLIP even when estimated on the proxy dataset, cf., Table 13.
37

---

Published as a conference paper at ICLR 2026
OH-Art
OHI-ClpArt
OH--Product
IN-R
IN-Skelch
Weighted ACC
0.55
0.8
0.90
0.74
0.99
1.78
0.74
0.02
0.50
0.76
0
0.08
0.73
0.91
100
150
200
23
76
100
150
200
100
150
2000
75
100
200
75
100
150
0.4
0.25
1.12
0.56
0.9
0.35
SLPD
0.8
0.3
1.08
0.7
0.15
1.06
100
150
200
100
150
200
100
150
200
100
150
200
100
150
200
Subset
Subset Size
Subset Size
Subset Size
Subset
Figure 15: Ablation study the k-NN distance metric, fixing the online LA learning rate 7 and the online LA pseudo-data count 3 10. Results are shown for EPIG (Wasserstein)
1e-4
EPIG (cosine) -). BALD (Wasserstein) (-), and BALD (cosine) (-- As shown Fig. 15, Wasserstein-based k-NN selection demonstrates improved performance for datasets such as Ol1-Art and IN-R, while no clear trend is observed across the other datasets.
Table 12: Zero-shot CLIP-Base performance versus pseudo-count on four datasets.
CIPAR:
ACC1
NLPD)
ECEL
ACCT
NLPD]
ECE]
ACC1
NLPD]
ECE]
ACCC
NLPD]
ECEL
01:4-00559
1.75#004
0.20=0.01
0:950002
0.559
1.811:006
3.78
80,4520.25
0.6810.01
1.87
93.60:00244
0.201:001
1.10
73:80:10:4
0.95-0:02
4.31
1831000
3.97
80,37:00.25
0.88100001
2.31
93. 03.00:00.24
0.201:0.01
1.20
73.8210.44
0:96
4.82
0..59
188110.0
4.50
80,34.0.225
0,8810.00
2.83
93.00.10.24
0.211:0.01
1.25
73:81:10444
0,96
5.10
9
60.43-0.99
1.55+000
4.60
80.33:0.25
0.68+00001
2.81
93.61=00.24
0.21=00.0
1.27
73.79:00.4
0.96m0.02
5.29
F.11 ROBUSTNESS W.R.T. DISTRIBUTION SHIFT OF THE PROXY DATASET
Most OpenCLIP models are trained on the LAION-400M dataset, which allows us to directly estimate the Hessians on this distribution. However, some large CLIP variants are trained on closed- source data, requiring the use of proxy dataset. As shown in Table 13, BLy remains effective in simulated dlosed-source setting, where we estimate the Hessians using the CC12M dataset.
To further analyse robustness, we conduct additional experiments in which we progressively distort the images of the LAION-400M dataset using different augmentations before estimating the Laplace Hessians: GRAYSCALE (interpolating between the RGB variant and the Grayscaled variant with intensity coefficients in (0, 0.2, 0.4, 0.6, 0.8, 1.0)), JPEG COMPRESSION (using increasingly lower JPEG compression quality values in {100, 50, 25, 10, 5, 1}. and GAUSSIAN BLUR (with increasing radius in {0, 10, 20, 30, 40, 50)). We provide examples in Figure 17). This setup simulates con- trolled proxy-distribution shift. Importantly, all zero-shot evaluations are performed on the original (non-augmented) benchmarks.
The zero-shot results on FOOD-101, CIFAR- 10, and IMAGENET-R R in Tables 14 to show that augmentations that alter image style while preserving semantic content lead to gradual increase in ECE that closely tracks the augmentation intensity. In contrast, GAUSSIAN BL.R (which directly impairs object recognizability, especially at intensities 0.6 to 1.0) causes substantially stronger degradation in both ECE and NLPD. Overall, these findings indicate that Bayes is robust to mild stylistic shifts in the proxy data, but its performance deteriorates once the augmentations begin to compromise semantic information relevant to the model.
F.12 ADDITIONAL ZERO-SHIOT RESULTS ON ADVERSARIAL IMAGENET VARIANTS
To further assess the robustness of BayesVLM under substantial distribution shift, we evaluate the CLIP-Base model on two challenging adversarial ImageNet variants: IMAGENET-A and IMAGENET- SKETCH. These datasets introduce severe covariate shift. IMAGENET-A A contains naturally adversarial, small, or off-centre objects, whereas IMAGENET-SKETTE imposes strong stylistic changes that alter low-level statistics.
38

---

Published as a conference paper at ICLR 2026
1.00
Er
er
0.64
48k
18K
# Datapoints (1 K)
Dalapoints x K)
(a) Image modality
(b) Text modality
Figure 16: Normalized trace of the image Hessian B-factor for varying base batch sizes (2048 (---), 8192
(-.-.), ), 32768
(-)) and 1-5 random batches. Error bars show 1 std over five trials.
Table 13: Does BayesVLM work in closed-source data settings? Yes. With OpenCLIP ViT-B-32 trained on LAION-400M and Bayes VL.M estimated on the proxy dataset CC12M, we find that results are robust and show only slight degradation; statistically significant differences are bold (p = 0.05).
PLOWERS-
FOOD-
CIFAR-
IMAGENET-R
SUN397
ACC11
LAION-4IIM
CC12M
68.87 1840
68. 12:0
80.43.0.296
60.35=3.201
93. 57:0265
08.62212241
73.63 0.400
73. 73.78 4009
74.4510-451
74 32:0449
61 46444497
61-4511-486s
66 81 c00
G6.96
NLPD
LAION 100M
1.73a00m2:
1.77
0.68
0,96=0mm2
0.96
1.03
1.08
1.44
12
1.13
ECE
LATON-DUM 40MM
CC12M
3.84
422
1.00
0.99
0.70
0.22
1.43
1.39
1.78
3.77
3.83
3.89
Table 17 reports zero-shot accuracy, negative log predictive density (NLPD), and expected calibration eтror (ECE) for CLIP, temperature scaling, test-time augmentation (TTA), and BayesVLM. Across both datasets, BayesVLM does not show signs of epistemic underestimation. On IMAGENET- A, BayesVLM matches TTA in terms of calibration (ECE 0.23 vs. 0.23) while improving over CLIP. On BIAGGGNNE--SKETCEE Bayes VLM achieves the best calibration among all methods (ECE 0.08), outperforming temperature scaling and TTA. Although the NLPD on IMAGENET-A is slightly higher than with temperature scaling, Bayes VLM still provides an improvement over CLIP
BLy
Notably, TTA yields large accuracy gains, especially IMAGENET-A, A, which likely arise from improved viewpoint coverage rather than improved uncertainty modelling: using 64 crops increases the likelihood of capturing an informative region of the image. In contrast, BayesVLM, preserves single-pass inference efficiency while improving predictive calibration. Since Bayes is com- plementary TTA, both approaches can be combined to obtaín improvements in accuracy and uncertainty quality simultancously.
F.13 EMPIRICAL ASSESSMENT OF THE INDEPENDENCE ASSUMPTION
Our approach relies on an independence assumption between the parameters of the image and text projection layers when constructing the surrogate probabilistic models. This assumption follows from adopting modality-wise surrogate models satisfying the common i.i.d. assumption and is crucial for maintaining computational tractability. As detailed in App. C. relaxing this assumption would require estimating and storing the full joint Hessian of the projection parameters, which is computationally infeasible in practice.
To empirically assess the strength of cross modall dependence in the loss curvature, we estimate the relative magnitude of the off-diagonal Hessian block HPQ compared the modality blocks Hpp and HQq- Specifically, we compute the ratio of Frobenius norms
H HPQlIF
R
(116)
([ (Hooll))*
where R would indicate strong cross-modal dependence, and smaller values correspond to weaker coupling in the curvature.
Computing the full Hessian is intractable due to its size 5122 x Float32). We therefore approximate the curvature using a Monte Carlo estimate of randomly sampled gradient components per block. The Hessian of the contrastive loss with respect to the projection
approximately 1.6 TiB in
39

---

Published as a conference paper at ICLR 2026
Table 14: Zero-shot CLIP-Base performance on FOOD-101 versus augmentation intensity for three
augmentation types.
GRAYSCALE
GAUSSIAN
Intensity
ACC
NLPDJ
ECE
Quality
ACC1
NLPD1
ECE|
Radius
ACCT
NLPD)
ECE,
0.0
8014440 397
068200013
1.69
100
80.44-0.397
0.68200013
1.69
50,4440.300 397
068:20013
1.69
801,45+0.38
0688+0011
80.48-0.396
0.68=00012
2.21
S0.1++0.995
0.08+90012
2.38
80.43:00.3
0.8200013
1.73
25
80.54:0.396
0.68:0.012
2.26
$0.58:1.396
0.6820.012
3.25
0.5
40)4K-+0 396
068+00013
1.85
10
80 00.59900.336
0:68:000122
2.23
S0.4800 396
0.09:0.011
5.69
0.8
801.44=0 796
0.88+00012
2.06
5
80.58-0 396
0.68=00012
2.74
401
S0.43+0.20 397
0.72+0.011
8,33
0.75:0.011
Table 15: Zero-shot CLIP-Base performance on CIFAR-10 versus augmentation intensity for three augmentation types.
GRAYSCALE
JPEG COMPRESSION
GAUSSIAN BLUR
Intensity
ACC1
NLPD,
ECEL
Quality
ACC1
NLPD]
BCEL
Radios
ACC1
NLPDJ
ECEJ
0.0
93:214
0.20300.007
0.72
100
03.6110.245
0.203:0.007
0.72
93.61=0.245
0.20310.007
0.72
91.1:+0.245
03.60+0..255
0.301-0.0077
91.57-0.344
0.203+0.007
03.59+0.245
93.57-0.225
0.203+0.007
0.6
90.61
0.203=0.007
0.68
10
03,59420.245
0.200-00.007
0.92
30
93.58-0..445
0.20440.007
0.84
0.8
93.58:10.245
0:203-00007
0.69
5
93.57::0:245
0.202-00.007
0.89
40
91.58:00.245
0.2071100007
L88
93.58.10.245
0.203.0.007
0.80
93.60.10.215
0.20510.007
1.35
50
93.58..0.215
0.21211.007
2.73
parameters is approximated via the empirical Fisher, H P Elgg"] where g denotes the per-batch gradient. While the empirical Fisher is less accurate than the GGN approximation used in the main method, it provides a computationally feasible proxy for estimating relative curvature magnitudes.
For each Hessian block, estimate the Frobenius norm by averaging the squared magnitudes of the sampled entries. The ratio R is computed across three random seeds and multiple sample sizes. The results are reported in Table 18.
The results indicate only moderate cross-modal dependence in the Hessian (loss curvature). Hence, while secondary interactions between modalities exist, the dominant curvature structure is captured by the modality-wise blocks. This provides empirical support for the independence approximation adopted BLyesVLM. Importantly, the approximation does not eliminate the multimodal coupling learned during pre-training. The curvature is evaluated at the MAP solution obtained via multimodal contrastive learning, and the resulting posterior still reflects the cross-mode alignment encoded in the model. In practice, our method replaces the exact posterior with a tractable approximation that preserves model's principal geometric structure while discarding interaction terms of moderate magnitude. Finally, the entanglement of modalities in the resulting uncertainties can be qualitatively observed in App. F.6, where corruption one modality affects the predictive uncertainty of the joint model, indicating that cross modal dependencies remain reflected in the predictive distribution.
40

---

Published as a conference paper at ICLR 2026
Table 16: Zero-shot CLIP-Base performance on IMAGENET-R versus augmentation intensity for three augmentation types.
GRAYSCALE
Intensity
ACC
NLPD]
ECEL
Quality
ACCT
NLPD]
ECEL
Radins
ACCT
NLPDJ
ECEL
0.0
34.4
1.031=0.018
1.71
100
14.4820.436
103
1.31
0
14.491004436
L081=00013
L.71
0.2
74.5 51 436
1.31-+0.018
1.70
50
74.53+0-436
1031 -0017
2.05
10
74.67440.35
1020=0017
2.04
1051=00.018
74.56:b0.436
1081=00017
34.69+0.425
1081+00017
34.63200436
34.764404334
44.47-00436
1.03010.018
1.89
74.4720.436
1.000=0.018
1.89
70.71100:35
L077100006
9.28
L.0
71.36.10:436
1,02910.00
1.95
74.7210.435
1002200.017
3.23
50
74.75100.34
L109910.05
12.20
GRAYSCALE
0.0
0.2
0.4
0.6
0.8
1.0
COMPRESSION
100
50
25
10
5
BLUR
Figure 17: Visualisation of proxy-data augmentations across degradation levels. Rows correspond to augmentation types (GRAYSCALE, JPEG COMPRESSION, GAUSSIAN BLUR). Columns show increasing degradation severity, parameterised by intensity, JPEG quality, blur radius, respectively.
Table 17: Zero-shot performance on IMAGENET-A and SMAGENET-SKETCH.
IMAGENET-A
INAGENET-SKETCH
Method
Acc (%) f
NI.PD
ECF 4
Acc (%)
NLPD
ECE.
CLIP (Radford al., 2021)
25.87 4378744.4379
3.7720.0812
0.33
50.98
2.3720.0308
0.16
CLIP (temp. scaling)
25.8710.4577
3.22=0.1226
0.18
50.9820.100 1999
2,2940.028
0.13
TTA (Farina al., 2024)
38.67 4870
2.77=0.0279
0.23
52.40 22.400=..0 4004
2.36++0.000
0.15
BLyesVLM
26.67++04422
3.36=0.0257
0.23
50.59=0 -0.0000000000 5000
2.24+0.0250
0,08
Table 18: Estimated curvature ratio R for different numbers of sampled gradient components.
Number of random samples
50k
0.363
0.002
100k
0.374
0.035
41


Reviews:
[{"review_id": "ZpkiU3WKUV", "reviewer": "Reviewer_VDEV", "summary": "This paper proposes post-hoc uncertainty estimation in VLMs (CLIP) that does not require additional training. This method performs Laplace approximation in the last layer of encoder to induce probabilistic feature embeddings, which can quantify uncertainty. Then it transform the cosine similarity into a probablistic distribution, and use uncertainty score to support set selection in active learning. In the experiments, they show uncertainty estimates derived from this approximation improve the calibration of these models on several zero-shot classification benchmarks and are effective in active learning.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The mathmatical process is solid.\n2. This is a post-hoc method, which does not require any retraining, fine-tuning, or modifications to the VLM architecture. It is only a Laplace approximation to quantify uncertainty. It is easy to apply.", "weaknesses": "1. The motivation to measure uncertainty of VLM (clip) is not attractive. The significance of the topic need be emphasized. Maybe it is for effectively selecting training data in active learning. If BayesVLM has more application fields, it will be better.\n2. The method relies on two assumptions that may not always hold: (a) the image and text embeddings can be modeled with Gaussian distributions, and (b) the two modalities are independent. These are simplified situations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mXRVaCrLqS", "reviewer": "Reviewer_9dNF", "summary": "The paper proposes BayesVLM, a post-hoc, training-free method to endow pre-trained VLMs (CLIP, SigLIP) with epistemic+aleatoric uncertainty. The approach fits a Laplace approximation over the final projection layers of the image and text encoders, assumes modality-wise independence to get tractable posteriors, and then analytically propagates uncertainty to cosine similarities via a Gaussian (“ProbCosine”) moment approximation. The method improves calibration (ECE) in zero-shot classification.", "questions": "1. Please provide some empirical results on the independence of two modalities.\n\n2. For ImageNet setup, please include more ImageNet variants such as ImageNet-A.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is training-free and works with off-the-shelf CLIP-like models.\n\n2. The proposed method provides overall better calibration performance on the evaluated benchmarks with small computation overhead.\n\n3. Proxy-data robustness: Hessians from CC12M still work decently for CLIP trained on LAION-400M.", "weaknesses": "1. Treating image and textual modalities as independent is the core approximation. While the authors justify it via local post-hoc around MAP, it remains a potential mismatch for strongly coupled modalities; discussion is present but could use a stronger empirical illustration.\n\n2. The proposed method puts all epistemic uncertainty in the final projections. This may under-estimate uncertainty on heavy distribution shifts.\n\n3. For closed-source models, the approach needs proxy data; results are promising but slight degradations exist. A sensitivity analysis to proxy domain mismatch (e.g., caption style, image domain) would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S7RBQq4M4q", "reviewer": "Reviewer_88Gp", "summary": "This paper proposes BayesVLM, a post-hoc Bayesian uncertainty estimation method for large-scale vision-language models (VLMs) such as CLIP and SigLIP. The key insight is to apply a Laplace approximation over the last linear projection layers of pre-trained VLMs to obtain Gaussian posteriors over the parameters. This gives a local second-order uncertainty around the learned weights, thereby inducing probabilistic embeddings for both modalities. This also allows the model to analytically compute a distribution over cosine similarities, leading to uncertainty estimates without any retraining, architectural modifications, or additional data. The authors further propose ProbCosine, an analytical approximation for the mean and variance of the cosine similarity under this Bayesian formulation, avoiding costly Monte Carlo sampling. Empirically, the paper demonstrates that BayesVLM yields better-calibrated uncertainty estimates in zero-shot classification and improves sample efficiency in active learning, with minimal computational overhead.", "questions": "- **Direct baselines.** Can the authors comment on adding ProLIP (frozen or few-shot adapted) [1] and CLAP4CLIP (single-task setting) [2] as baselines on selected zero-shot datasets to validate BayesVLM’s calibration per FLOP gains?\n- **Data requirements:** When estimating the Hessian with proxy datasets, how do you ensure alignment between proxy and original data distributions? Could this introduce bias in uncertainty estimates?\n- **On scalability:** What is the memory or compute bottleneck when applying BayesVLM to very large projection layers (e.g. CLIP large)? Could low-rank approximations be used instead?\n\n**References:**\n\n[1] Chun, Sanghyuk et al. “Probabilistic Language-Image Pre-Training.” ICLR 2025.\n\n[2] Jha, Saurav et al. “CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models.” NeurIPS 2024.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- **Novelty and conceptual clarty:** The idea of using a post-hoc Laplace approximation on pre-trained VLMs to obtain Bayesian uncertainty estimates is novel and conceptually elegant. The formulation is well-grounded in Bayesian deep learning literature and bridges it with the practical need for scalable uncertainty estimation in large multimodal models.\n- **Practicality and scalability:** The method is traning-free and model-agnostic, requiring only access to the final projection weights and not the original training pipeline. The approach introduces negligible computational overhead (<5% runtime, <0.11% GFLOPs), making it attractive for large-scale or resource-constrained deployments.\n- **Analytical Contribution (ProbCosine)** The derivation of a closed-form Gaussian approximation for cosine similarity uncertainty is elegant, interpretable, and computationally efficient. This is a non-trivial mathematical contribution that generalizes beyond this specific application.\n- **Strong Empirical Evaluation:** Comprehensive experiments across multiple datasets (Flowers-102, CIFAR, Food-101, ImageNet-R, UCF101, SUN397) and tasks (zero-shot classification, active learning). The paper also shows comparison with strong baselines: temperature scaling, test-time augmentation, and probabilistic embedding models. Results show consistent improvement in calibration metrics (ECE) and competitive or improved accuracy.\n- **Robustness and Generality:** The method performs well even when the Hessian is estimated on proxy datasets, showing robustness to missing original training data. The paper provides ablations on hyperparameters (τ, λ) and data subset sizes for Hessian estimation.\n\nOverall, the main paper is well-written and complemented by detailed appendices with derivations, algorithms, and hyperparameters.\nThe authors mention plans to release code, aiding reproducibility.", "weaknesses": "- **Assumption of Independence (Modality Factorization)** The method assumes independence between image and text modalities (P ⊥⊥ Q) to enable tractable posterior estimation. Although the authors justify this as a local approximation around the MAP estimate, it weakens the theoretical rigor since VLMs are inherently cross-modal. Also, the impact of this assumption on downstream uncertainty fidelity is not fully explored. Can the authors comment on this?\n- **Limited Scope of Bayesian treatment:** The Laplace approximation is applied only to the final projection layers, while the encoders remain deterministic. This limits the expressiveness of the uncertainty model, potentially underestimating uncertainty arising from deep feature extraction. Although, I understand  that an approximation to the entire encoder layers would be a huge sacrifice for efficiency/practicality, could the authors comment on any other plausible reasons why/why not to consider this?\n- **Evaluation of Uncertainty Quality:** While calibration metrics (ECE, NLPD) are reported, it would strengthen the paper to show qualitative examples or diagnostics of uncertainty estimates (e.g., in failure or domain shift cases).\n- **Active Learning Evaluation Setup:** The active learning results are promising but limited to relatively small-scale benchmarks (OfficeHome, ImageNet variants). It is unclear how the approach would scale to real-world large-scale active learning pipelines or multimodal retrieval settings.\n- **Comparative baselines:** The paper primarily compares with test-time augmentation and temperature scaling, but omits comparison with recent Bayesian adapters or probabilistic prompt tuning methods (e.g., BayesAdapter [1]). A direct comparison could more clearly position BayesVLM in the landscape of uncertainty-aware VLMs.\n- **Interpretability of Uncertainty:** Although the authors claim interpretability under corruptions, the paper could better visualize what kind of uncertainties are captured (epistemic vs. aleatoric). Without this, it’s difficult to assess if BayesVLM truly models epistemic uncertainty or merely produces smoother predictions.\n\n**References**\n\n[1] Morales-Alvarez, Pablo et al. “BayesAdapter: enhanced uncertainty estimation in CLIP few-shot adaptation.” (2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)