Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere
Authors: Li Ju
Max Andersson
Stina Fredriksson
Edward Glöckner
Andreas Hellander
Ekta Vats
Prashant Singh
Venue: NeurIPS 2025 poster ()
Keywords: Uncertainty Quantification
Vision-Language Models
Probabilistic Modeling
Primary area: probabilistic_methods
TL;DR: The paper proposes AsymVLM - a probabilistic adaptation framework for vision–language models that models text uncertainty on the unit hypersphere, using spherical distributions to improve cross-modal retrieval and robust classification.

Abstract: Vision-language models (VLMs) as foundation models have significantly enhanced performance across a wide range of visual and textual tasks, without requiring large-scale training from scratch for downstream tasks. However, these deterministic VLMs fail to capture the inherent ambiguity and uncertainty in natural language and visual data. Recent probabilistic post-hoc adaptation methods address this by mapping deterministic embeddings onto probability distributions; however, existing approaches do not account for the asymmetric uncertainty between modalities, and the constraint that meaningful deterministic embeddings reside on a unit hypersphere, potentially leading to suboptimal performance. In this paper, we address the asymmetric uncertainty structure inherent in textual and visual data, and propose AsymVLM to build probabilistic embeddings from pre-trained VLMs on the unit hypersphere, enabling uncertainty quantification. We validate the effectiveness of the probabilistic embeddings on established benchmarks, and present comprehensive ablation studies demonstrating the inherent nature of asymmetry in the uncertainty structure of textual and visual data.

Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere
Ju Max Andersson
Stina Fredriksson
Edward Glöckner'
Andreas Hellander
Ekta Vats
1
Prashant Singh !t
Abstract
Vision-language models (VLMs) foundation models have significantly enhanced performance across wide range of visual and textual tasks, without requiring large-scale training from scratch for downstream tasks. However, these determinis- tic VLMs fail to capture the inherent ambiguity and uncertainty in natural language and visual data. Recent probabilistic post-hoc adaptation methods address this by mapping deterministic embeddings onto probability distributions; however, existing approaches do not account for the asymmetric uncertainty structure of the modalities, and the constraint that meaningful deterministic embeddings reside a unit hypersphere, potentially leading to suboptimal performance. address the asymmetric uncertainty structure inherent in textual and visual data, and propose Asym VLM to build probabilistic embeddings from pre-traines pre-trained VLMs the unit hypersphere, enabling uncertainty quantification. We validate the effec- tiveness of the probabilistic embeddings on established benchmarks, and present comprehensive ablation studies demonstrating the inherent nature of asymmetry the uncertainty structure textual and visual data.
this paper,
Introduction
Vision laiguuge models (VLMs) have demonstrated impressive capabilities in understanding visual and linguistic information, by constructing a joint embedding space that aligns image and text embed- dings [4, 29]. Models like CLIP [18] and BLIP [14], enable wide range of downstream applications, such as zero-shot classification [1]. image-to-texs retrieval [5], optical character recognition [27]. However, due to the inherent ambiguity within text and image data, aligned point estimates for text and image embeddings produced by deterministic VLMs may not fully capture the complex relationships between the visual and linguistic embedding spaces.
Alternatively, instead of learning deterministic point estimate embeddings, probabilistic embedding methods [8] map text and language data onto probability distributions, more effectively capturing the ambiguity and uncertainty of the data. However, they entail training probabilistic VLMs from scratch massive datasets
expensive process that forgoes the strengths of established deterministic VLMs. To explore post-hoc methods for constructing probabilistic embeddings from pretrained deter- ministic VLMs, recent frameworks such PLo [25] and Bayes VLM [3] have been proposed. However, the abmiguous nature text-to-image retrieval due to both textual abstraction and image variability, while image-to ambiguity arises from the multiplicity of textual descriptions. Existing methods overlook this asymmetric nature in texts and images, and the fact that the embeddings of most pre-trained VLMs reside unit hypersphere, instead of the Euclidean space.
Conresponding author (1i. www.j....uuu...
Department of Information Technology, Uppsala University, Uppsala, Sweden Science for Life Laboratory, Uppsala University, Uppsala, Sweden
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Pro6VLM
AsymVLM
eppessstaation
1-to-1 I-o-moyy
infinite dogs real
infinite dogs real life
the dog real life
text enceder
image encoder
object
dog
dog
dog
images of the dog
Figure 1: Left: Mapping relations between text, image, object, and representation spaces. Right: Comparison of methods - CLIP uses deterministic one-to-one mappings; ProbVLM introduces one- to-many mappings via probabilistic embeddings; AsymVLM further captures asymmetry structure between texts, images and objects.
In this work, propose post-hoc method, Asymmetric Probabilistic Vision-Language Model (AsymVLM), to model the uncertainty embeddings obtained from pretrained VLMs
We formally address the asymmetric uncertainty structure inherent in vision- language data, high aleatoric uncertainty in text versus lower aleatoric uncertainty in images.
We propose AsymVLM, an adapter that exploits the asymmetric uncertainty structure and per- forms post-hoc probabilistic adaptation on the unit hypersphere, rather than in Euclidean space. We also show that Asym VLM is a natural extension of CLIP's cosine-similarity loss with the added ability to quantify uncertainty.
Empirically, Asym VLM yields more accurate uncertainty estimates and higher cross-modal retrieval accuracy on multiple benchmarks. We further demonstrate its advantages in robust fine-tuning, zero-shot classification and tone-of-the-abboy rejection.
We formulate the problem, addressing the asymmetric structure of vision-language data in Section and derive our method, AsymVLM, in Section 4. We then present empirical evaluations, including uncertainty quantification, ablation studies, and downstream applications in Section 5. We conclude and discuss future work in Section 6.
2 Related work
Vision-language models Recent advancements in vision-language modeling have facilitated the development multimodal models capable of learning shared embedding space for images and text. These models typically achieve this by optimizing directional similarity metrics, such as cosine similarity. Notable works including CLIP [18], BLIP [14], SigLIP [28]. typically align paired image- text representations by maximizing the cosine similarity between corresponding embeddings. This strategy has proven to be effective for large-scale training, enabling these models to capture semantic relationships, and generalize well across diverse downstream tasks. However, these pretrained VLMs offer deterministic mappings that fail to capture the inherent ambiguity in the inputs.
Probabilistic VLMs Methods have been proposed to improve deterministic VLM embeddings by learning a unified probabilistic embedding space for both visual and textual data. Methods such as PCME [8], and PFE [24] learn map texts and images to aligned probability distributions, by maximizing the cross-modall likelihood between corresponding text and image data. However, these approaches require training from scratch, which computationally expensive as existing high-quality deterministic VLMs are not fully leveraged. A more resource- resource-efficient efficient alternative is the probabilistic adaptation of pretrained VLMs. ProbVLM [25] trains an adapter to construct probability distributions from deterministic embeddings, while BayesVIM [3] estimates the uncertainty of cosine similarity through post-hoe Bayesian posterior approximation. As mentioned in Section (and detailed in the following Section), although having demonstrated their efficacy for uncertainty quantification existing post-hoc approaches do not consider the asymmetry in uncertainty structures between textual and visual data, and the constraint that meaningful embeddings of most pretrained VLMs reside on the unit sphere, resulting in sub-optimal performance.
Embeddings on the unit hypersphere Cosine similarity is widely utilized in contemporary VL.Ms and foundation models pre-trained with contrastive loss [6, 18, 7]. This approach inherently constrains the learned embeddings to reside on the unit hypersphere. These embeddings have been shown to

---

AsymVLM
Image Encoder
A3
P(pa,ed) kg)
z
...
[Mg Ki))
11
[#2,
L21
b,a
Igg
122
Frozon Encoder
a photo dog
Adaptor
Kz))
Lgg1
1g.
Lg
Lg.8
Lx = loyls pa.na)
L...
L...
L...
(Jp Kn)
Lgg
Lgg
Lg
Lgg
Figure 2: Asymmetric probabilistic adaptation (AsymVLM): Texts are encoded with a frozen text encoder and adaptor to produce probabilistic hyperspherical embeddings (e.g.. vMF distribution), while image are deterministically encoded via frozen image encoder. The log likelihood matrix L, with element Lm,n representing the log likelihood of image vector o given text embedding 2m
mm Km). is optimized using InfoNCE to maximize diagonals and minimize off-diagonals.
have advantages such as improved uniformity and alignment compared to their Euclidean counterparts [26]. Beyond cosine similarity, embeddings the unit hypersphere also be explicitly modeled by assuming directional distributions, such as the von Mises-Fisher (vMF) distribution [10]. Scott et al. [21] introduced contrastive loss based the vMF distribution pretraining method, while [20] proposes method to distill pretrained VLMs, adapting student models with vMF assumptions for zero-shot classification. However, while these approaches focus improving empirical performance in downstream tasks, none explicitly model the uncertainty of embeddings from pretrained VLMs.
Asymmetric structure of texts and images
The relationship between vision and language exhibits inherent asymmetry in its uncertainty structure. textual concept, such as "dog." acts as an abstract type that can map to a vast number of unique real-world instances (tokens). Each of these instances can, in turn, be captured in countless images differing in viewpoint, lighting, and composition. This cascaded one-to-many mapping- from abstract type physical token, and from token to specific image-induces high aleatoric uncertainty in the text modality.
Conversely, image portrays single, specific token. While it depicts one object unambiguously. can be described by multiple valid captions (e.g.. "a dog," "a golden retriever," " "a pet"). This one-to-many mapping, from image text, arises from linguistic diversity. Therefore, the uncertainty in text-to-image retrieval (rooted in abstraction and visual variability) is fundamentally different from that in image-to-text retrieval (rooted in descriptive variance). Despite this structural asymmetry as shown in Figure 1, existing post-hoc probabilistic adapters typically treat both retrieval directions symmetrically.
3.1 Mathematical formulation
Let T,I, O and Ö denote the text, image, (real-world) object and (numerical) representation space respectively. The relationships between T, I,o and ō Ö can be presented as follows.
Building VLMs Existing datasets such as MS-COCO [15] and Flickr30K [17] collect data in the form of {t,,, ) such that there exist O which satisfies (t,(t,) and i, (o,).. Based such datasets, aim to find d-dimensional real-valued representation space ② c Rd. such that there exists a bijective function v O Ó. More importantly, the goal is approximate the composite function sp fr T - P(O) and s o +- + Ø with finite samples from datasets the text encoder fr and image encoder fr, following the Platonic representation hypothesis [1]].
3

---

MAPPING
DEFINITION DERIVATION
ONE-TO-MANY?
Text - Object
(r(()) = (o E O can be described by I)
Object + Image
21(0) = {i E Z: : is an image of o}
Text + Image
(0, 4r)
U0 $2(0)
Image Object
(1)
X X(assumption)
Object + Text
*7 = (tE o be described by {} (0," ,, (1) = [t T : 47' (i) E pT(())
Image Text
Object ++ Representation
(0() o, o (0)
Text encoder
R c
Image encoder
f1 = 07
3.2 Motivating the approach
Trained datasets in the form of {(bm, ) existing deterministic pre-trained models maximize the similarity measure between matching text-image pairs while minimizing it for non-matching
pairs [18, 28]. These approaches effectively learn real-valued embedding space Ô along with deterministic text and image encoders. Existing probabilistic adaptation methods, rather than mapping an image or caption to a single point in the embedding space, represent elements of T or I as random variables in O thereby capturing the inherent one-to-many nature of the relationships.
However, the relationships among texts, images, and objects are inherently asymmetric, leading to distinct structures in text-to-image and image-to-text retrieval tasks. While both tasks are one-to- many, the asymmetry arises from different sources: abstraction and visual variability in text-to-image versus diverse textual descriptions in image-to-text. This structural difference motivates the use of asymmetric probabilistic structure for encoders within the representation space.
From an uncertainly-quant quantificaatio perspective, we adopt the standard taxonomy of aleatoric vs. epistemic uncertainty [12]. Aleatoric uncertainty captures irreducible data ambiguity, e.g. the caption "a dog" denotes an entire class of breeds, poses and contexts, so no amount of additional training can pinpoint a single underlying object. Epistemic uncertainty captures model ignorance that could be reduced with more or better-distributed data, e.g., images, while depicting a unique scene with low aleatoric spread, may still suffer from coverage gaps certain viewpoints lighting conditions were under-represented during training. This asymmetry yields inherently broad aleatoric uncertainty in text versus more data-dependent epistemic uncertainty images. In practice, aleatoric uncertainty is modeled via parameterized distributions, whereas epistemic uncertainty is estimated by Monte Carlo methods (e.g.. sampling from dropout layers or Bayesian neural networks).
Assuming an underlying asymmetric uncertainty structure, a direction for improvement emerges. For text embeddings, it is essential to capture the wide variance inherent in language via distributions to reflect the multitude of corresponding objects. Conversely, image embeddings can be be modeled deterministically and the uncertainty reduced through large coverage of visual data in the datasets.
Method
With a pretrained VLM such as CLIP or SigLIP, we denote the text encoder as fr : T Sd-1 and image encoder as fr : I gd-1, where d is the dimension of the representation space.
4.1 Asymmetric probabilistic adaptation
As discussed in Section 3, capturing the variance is crucial for text embeddings, while ensuring coverage in training data is key for image embeddings. To achieve this, we introduce text adapter gr for the text encoder fr. while retaining the deterministic embeddings from the image encoder fi+ expanding image data coverage lies beyond the scope of this work. Formally, the embedding of any text t E T is modeled by random variable z".
I ~ P(O(t)) where 0(t) = gr ft))),
(1)

---

where P(O(t)) denotes a distribution parametrized by 8(t) predicted by neural network given corresponding text. The deterministic image embedding 2 from the image encoder fy is retained.
Objective function For any batch of data {(t1, (11,,,),..... (fB, iB)}, we aim to maximize In P= (z/1) for all [B] while minimizing In P=z, (2/)) for all n, E [B]. # m, where Pa denotes the First we define the log likelihood matrix L BxB where the (m, ni-th element is computed as L(m, n) = P=z (z,), for n,m m [B].
value of the PDF of random variable
To maximize the diagonal elements and minimize the off-diagonal elements of L, we apply the InfoNCE loss to and obtain the objective function to be minimized,
B
exp (T L(n, n.)
exp (T L(n,
arg arg min BEE
In B Σm= exp (T [(n,
In Σm= exp (T Lm,, n))
(2)
2B
where 7 is a temperature parameter and 0 the parameter set of distribution P_ (0(1))-
4.2 Modelling on the unit hypersphere
The shared representation space built by most pre-trained VLMs is gd-1 the domain in which both z" and z reside. This observation necessitates the selection a directional distribution for P(O()), defined on gd-1, To properly construct a probabilistic embedding on gd-1, directional distributions for the embeddings should be employed - either projected distributions (e.g.. the angular normal distribution) or native directional distributions (e.g.. the vMF distribution). In this work, we use the vMF, and power spherical (PS) distributions to model the text embeddings distribution. The former generally considered as the simplest directional distribution, while the latter reported to be an alternative to the vMF distribution with closed form normalization constant.
Derivation w.r.t vMF distribution The vMF distribution vMF (p,, K) is a probability distribution on gd-1, parameterized by mean direction μ E Rd and isotropic concentration parameter R, where || ll2 and 2 0. The PDF for random unit vector æ is given by,
4/2-1
p( H4 x) = Ca(x) exp(x p TTx), where Ca(k)) = (2m) ( (1))
and I denotes the modified Bessel function of the first kind, order p.
Following Equation 1, assuming that TT ~ vMF(u(t), k(t)) for text t E T, for any image i E I, we have In phrr(22 = k jutt) 2f + In (t))(()). However, computing Ca(R) is intractable in the high dimensional space, yet [21] have shown that In Ca(k) can be approximated by Fa(1s) + const., where (r( is given by,
d
d
d
d - 1
Fa(k)
1
In
2
+
V
2
+ +
2\
2
+ k²
/
d-1
d
d
1
+
In
+
+ R2
2\
2
+ 1x?
Injecting the log likelihood of vMF into Equation results in the vMF kernel for the objective function L.ME as follows,
(r, x(tr) p(t.) Fa(s(t.)), Vr, E [B].
The detailed derivation for the approximation is deferred Appendix A.I.
Derivation w.r.. PS distribution The power spheric distribution PS(p, x)) is an alternative to vMF, also parametrized by a mean direction j E IR and a concentration parameter is I. with ||a|| = and k > O. The PDF of a PS distribution is given by,
K)) = (1 + pT x)* Cd(s), where Ca(k) = { 22 T(a)
I(a 3)
5

---

and a= =
k, K, (d- 1)/2
Similarly, following Equation 1, assuming that 2' ~ PS(u(t), k(t)) for text t E T. for any image t Z, injecting the log likelihood of PS distribution in Equation results the PS kernel Les.
(rs(r,s) =k(tr) In(1 p(tr)² (((=))==) 21) (d k(t
d d-1 + n(t>)) + Ind ,(tr)), Vr, E [B].
4.3 Uncertainty quantification
We quantify text-embedding uncertainty as u(!) 1/x(t), due to the fact that for both the vMF and PS distributions on gd-1, the (angular) variance o² is strictly decreasing function of Equivalently, larger concentrates mass more tightly around the mean direction (lower a²), so 1/k grows monotonically with the distribution's variance, providing a natural scalar value for uncertainty estimate for probabilistic embeddings.
4.4 Connections to CLIP
Equation can be rewritten in the following form,
B
In exp (Tō(1, Im=1& exp (Tō(n, m)) > exp (r0(m,
exp oxp(r)(n,,)) n))
arg min
Σ
In
8
2B
Denoting CosSim(r, 8) - m((r) T z for any r, E [B] we have,
for CLIP: ŠCLIP(r s) CosSim(r,s))
for AsymVLM (r, x((,) CosSim(r, 8) + Fa(m(())), for AsymVLMps: (r,(r, k(t,) In(1 + CosSim(r, s) + In Ca(m(t,)).
(3)
Given that k(t,)) 2 0, it clear that the latter two objectives are monotonically increasing w.r.t. PCLPP with an additional parameter k. These can be interpreted as extensions of the CLIP loss by introducing K, the concentration parameter, model the variance of the text embedding without drifting from the pre-trained models using cosine similarities.
Equation demonstrates AsymVLM's consistency with the underlying ideas of CLIP, while incorporating mechanisms to account for embedding variance. AsymVLM ultimately offers more refined and probabilistically grounded approach to modeling semantic relationships between text and images. Furthermore, AsymVLM can be extended to SigLIP variant. Inference using AsymVLM also extends beyond simply computing the cosine similarity, to utilizing the maximum likelihood principle, with clearer statistical explanation. Details about the inference process and its SigLIP-variaal are provided in Appendix A.2 and A.3.
5 Empirical results
We compare Asym VLM to the baselines on benchmark datasets, and subsequently present an ablation study, Further, extended analyses and demonstration on two downstream applications are presented.
Datasets, haselines and metrics The MS-COCO [15] and Flickr-30k [17] datasets are used to train the adapters. Additionally, a subset of the Conceptual Caption dataset [23] of 200k samples (CC-200K) is also used. To understand the nature of learned uncertainty, the HierarCap dataset [2] is used, containing four captions of different abstraction levels for each image. The baselines include VLM, ProbVLM, ProLIP, PFE and PCME++ adapted from pretraining methods (denoted by PFE* and PCME++* respectively). The pre- pre-trained VLM used in the experiments in the main text is CLIP (ViT-B/32 backbone). We assess the uncertainty quality in text embeddings by evaluating recall @1 performance in cross-modal retrieval tasks. A strong positive correlation between uncertainty and errors indicates that higher uncertainty leads to poorer embeddings and lower recall. For each task, we group results by uncertainty levels and compute Spearman's 's rank correlation (S) between uncertainty and recall [22]. We then fit regressor (R²) to determine if the

---

performance drop is linear, and compare average @ecall@ across uncertainty levels for different methods. All experiments were repeated five times with different random seeds, and we report the mean results with full standard deviations provided in the Appendix. The code for this paper is available at https::/// htt////gttttb/tt cwoommmiijuust- haaaaajjj6///mmmm/////m asymvlm.
5.1
Evaluation of UQ and cross-modal retrieval
Recall Uncertaint: COCO
Uncertainty: Flickr
Uncertainty CC 200k
Irage-te-Tee Teat
Image-io- Tox
Inageto-Too Tea
0.6
0.7
0.4
04
6.-
02-
Teat-e-irag
0.7
0.6
Rovat
0.4
C3
VeraViMaa
AomeViMap
Pruttrat
03
PPS
FER-
Pad.I
PCME
FCME
5.0
Uncerarey Levels
Uncoraiity Londs
Unceneary Lovele
Figure 3: Recall@) @ versus uncertainty levels for cross-modal retrieval tasks for various datasets. Each subplot shows the performance degradation as uncertainty increases for PFE*, PCME++*, ProbVLM, Asym MLy. and AsymVLMps. Ideally @ should decrease monotonically.
Table 1: Evaluation of model performance and uncertainty benchmarks (MS-COCO, Flickr-30k, CC-200k) for Image-to- Text (i2t) and Text-to-Image (12i) retrieval. We report Recall 1 f. regression fit Ri between uncertainty and retrieval error, and Spearman's 's rank S + correlation between uncertainty and retrieval error. Bold font denotes the best results and underline denotes the second best results. Each value is the mean of 5 runs with standard deviation in the Appendix, Table A.2.
T21
12T
DATASET
METHOD
Recall@1
n²
$ 4
Recall 1
nt²
$ 4
PEE
0.558
0.304
0.946
-0,988
PCME++*
0.500
0.931
-0.99
0.304
0.948
-0,990
Bayes BLyesVLM VLM
0.506
0.884
-0.976
0.323
0.932
-0.985
MS-COCO
ProbVLM
0.480
0.951
.0.981
0.293
0,979
-1.000
Pp
0.500
0.808
-0.908
0.304
0.876
-0.985
VLM AR
0.561
0.948
-0.988
0.392
0.984
-1.000
ALy
0.558
0.937
-0.984
0.390
0.989
-1,000
PFE
0.680
0,675
-0.832
0.451
0.955
40.998
*CME++
0.679
0.455
.0.178
0.451
0.900
.0.188
BayesVLM
0.637
0.916
-0,976
0.425
0.934
-0,973
FLICKR-30K
ProbVLM
0.646
0.826
-0.914
0.422
0.964
-0,985
ProLip
0.829
0,928
-0,978
AsymVLM AF
0.688
0,860
-0.976
0.504
0.960
-0,995
Asy
0,688
0.854
-0.947
0.498
0.946
-0.995
PFE
0.352
0.857
-0,521
0.336
0.988
-0,595
*CME++*
0.336
-0.887
Bayes VLM
0.347
0.905
.0.96%
0.304
0.874
-0,937
CC-200K
ProbVLM
0.316
0.772
-0.837
0.302
0.775
-0,965
ProLip
0.351
0.968
-0,990
0.335
0.992
-1.000
AsymVLMMas
0.395
0.990
-1.000
0.383
0.991
.0.998
ALy
0.393
0.992
-1.000
0.380
0.993
-1.000
Figure 3 and Table I summarize the main results comparing the proposed AsymVLM with baseline
methods. For uncertainty quantification, results indicate that while all methods capture text uncertainty to some degree, AsymVLM consistently outperforms its counterparts. As the uncertainty level
7

---

increases, Asym VLM is the only method for which Recall @ decreases strictly monotonically. Moreover, AsymVLM also meets or exceeds competing approaches on both R and scores.
In cross-modal retrieval, AsymVL.M,MF and AsymVIMMp achieve the highest and second-highest Recall@1 1 scores across all datasets. Note that the retrieval performances of PFE* and PCME++ identical, as they only model the variance of embeddings while while keeping embedding directions constant, resulting in performance identical the pre-trained models. In contrast, both Asym VLM and ProbVLM adapt the directions and learn the uncertainty of the embeddings during post-hoc training. Although ProbVLM achieves reasonable uncertainty estimates overall, it consistently under- performs pre-trae trained models in recall, whereas Asym VLM delivers richer uncertainty estimates and improves cross-modee retrieval performance.
Furthermore, the similar performance of AsymVI.M, <MF and AsymVLMM in both cross-modal retrieval and uncertainty quantification shows the robustness of ALy w.r.t. the choice of the spherical distribution, suggesting possibilities of exploring other spherical or directional distributions.
5.2 Ablation study
We conduct an ablation study on the two important components of AsymVLM: the asymmetric architecture of the adapter, and the use of a spherical distribution for modeling. To assess the impact of two components, we designed following adapters with experimental results presented in Figure
Asym (SmMLLM( This variant modifies Asym VLM by only mapping deterministic image embeddings to spherical distributions, leaving text embeddings deterministic.
VLMM This variant map both deterministic image and text embeddings to spherical distribu- tions, with InfoNCE applied to symmetrized log likelihood kernel.
ALy)) This variant has identical architecture with ALy but maps deterministic text embedding Gaussian distribution, followed by normalization. The kernel of the loss function is replaced by Gaussian log likelihood.
Asymmetric architecture Both Asym (iMmmage) and SymVLM perform worse than the original AsymVLM with respect to uncertainty quantification. Specifically, AsymVLM(iman struggles to model image uncertainty, showing decrease in Recall @ uncertainty increases. Although Sym VLM exhibits a slight decrease in Recall@1 I with rising uncertainty, uncertainty estimates remain sub-optimal to AsymVLM. AsymVLM(ing) AsymVI VLM while Sym VLM performs much worse than either AsymVLM or Asym VLM(image) The results underscore the importance of an asymmetric architecture for uncertainty quantification and modsl retrieval performance.
achieves average Recall @ I comparable to
Choice of spherical distributions For both Image-to-Tex and Text-to-Image retrieval tasks, AsymVLM(() yields reasonable uncertainty estimates. As uncertainty levels increase, the Recall of Assmm VLM(G) declines almost linearly, exhibiting behaviour similar Asym VLM. However, the overall @ is inferior compared to Asym VLM. This emphasizes not only the effectiveness of the asymmetric architecture for uncertainty quantification, but also the necessity of spherical distributions for the post-hoc adaptation of pre-trained VLMs.
Image-to to Text
Text-to Image
Averge Recall
0.5
0.5
Roaile1
Auyalytad
6.3
0.1
AnyoW1
AaraVLM AotnnVAliinnni SnaVIM
Uncemintty Levok
Unumitty Leadls
Adqrom
Figure 4: Ablation study results for ALym The left and middle panel shows Recall @ I per- formance for cross-modal retrieval across various uncertainty levels. The right panel summarizes average Recall @ I for both retrieval tasks.
8

---

5.3 Understanding the uncertainty estimates
To better understand the uncertainty estimates produced by ALymVLM, we analyze caption estimates on the HaerarCap dataset. In this dataset, each image is paired with hierarchical captions spanning four levels from general descriptions (Level 0) detailed image descriptions (Level 3). Figure illustrates the following observations.
AsymVLM captures hierarchical caption structure Level captions with general descriptions exhibit higher uncertainty estimates, indicating greater ambiguity, while Level 3 captions with the most detail generally show lower uncertainty. This pattern aligns with the inherent ambiguity of language and the corresponding uncertainty estimates provided by AsymVLM.
Detailed captions enhance image retrieval likelihood The likelihood of the image embedding increases more detailed descriptions are provided, as shown by the shift towards higher log likelihood values from Level 0 This suggests that detailed captions better capture the nuances of the image, facilitating more accurate image retrieval.
Longer captions exhibit reduced ambiguity The results indicate that higher token count associated with lower uncertainty in the text, as depicted by the downward trend in uncertainty values. This aligned with the expectation, as shorter texts generally tend to be more ambiguous, while longer captions provide more context, reducing ambiguity.
Uncestainty Different Levels
Log Likelihood Different Levels
Num Tokens Uncertainty
1090
Decimns
1000
Occamene
Uscetaacy
0.29
Uncemairyy
Likillod
Figure 5: Uncertainty and likelihood analysis with the HierarCap dataset. The left panel shows the distribution of uncertainty estimates across hierarchical caption levels, showing decreasing uncertainty from Level 0 to 3. The middle panel illustrates the log likelihood distribution of image embeddings, with increased likelihood as caption detail increases. The right panel shows the relationship between the number of tokens and uncertainty, indicating reduced uncertainty with longer captions.
5.4 Applications
Below, we present two example applications of the uncertainty estimates obtained from AsymVLM. demonstrating their potential use cases.
Handling none-of-the-above For zero-shot classification, strategies enable model to say "none-of-tae tet-ttheebbbeee rather than as- signing incorrect label include threshold- based rejection, margin-basse rejection and other techniques. Asym VLM provides an alternative approach, by introducing a dummy prompt such "a photo" to signal -none-of-the-abore
Table 2: None-of-the-above-are zero-shot classifi-
cation using dummy prompts. The table reports the accuracy positive samples and negative samples.
DUMMY CLass
METHOD
Pos. Acc.
NEG. Acc.
CLIP
0,888
0.009
photo"
AsymVLMps
0.845
0.547
AsymVLM Mem.LMM,
0.857
0.587
We evaluate these methods on a combined test consisting of
CLIP
0,888
0.009
"a photo of
CIFAR-10 10 and CIFAR- 100 samples, perform- ing zero-shot classification over CIFAR- 10 classes. Ideally, CIFAR-10 samples (positive samples) should be correctly classified into their respective categories, while CIFAR- 100 samples (negative samples) should be rejected as "none-of- www.ffteeaboeeeeeeeeeeeeeeeeeeee sistently outperforms other methods (Table 2).
object"
AsymVLMps
0.849
0.609
AsymVLM,
0.858
0.557
CLIP
0,888
0.009
"a photo of something"
Determ.
AsymVVMMs
0.843
0.595
AsymVLM,
0.857
0.585
Margin- Bascd
0.584
0.579
AsymVL.M con-
NONE
Threshold-Baan
achieving higher accuracy both positive and negative samples.

---

Table Zero-shot classification accuracy. "None" denotes the baseline without fine-tuning.
VALIDATED ON
FINE-TUNED
METHOD
CIFAR-10
CIFAR-100
STL-10
IMAGENET-IK
Detern:
0.684
0.300
0.829
0.395
MS-COCO
AsymVLMps
0.847
0.470
0.952
0.502
Asy
Determ.
0.688
0.342
0.874
0.369
FLICKR-30K
AsymVLMps
0.791
0.413
0.920
0.451
AsymVIM..ar
0.792
0.422
0.918
0.464
Determ. FT
0.779
0.411
0.922
0.450
CC-200X
AsymVLMps
0.861
0.542
0.968
0.520
AsymVLM..
0.866
0.542
0.967
0.527
NONE
0.888
0.974
Robust fine-tuning Pre-trained VL.Ms are renowned for their zero-shot classification performance. However, fine-tuning these models smaller datasets typically impairs this capability on other datasets. Table compares AsymVL.M with its deterministic counterpart, denoted by Determ.F1, and the foundation model without fine-tuning. Models fine-tuned with AsymVLM consistently outperform those fine-tuned deterministically across all settings. This suggests that AsymVLM preserves the zero-shot classification proficiency acquired during pre-training to greater extent, whereas the deterministic approach is more prone to catastrophic forgetting. Therefore, AsymVLM offers more robust fine-tuning strategy for VLMs.
6 Conclusion and limitation
We introduced VLM, a post-hoc probabilistic adaptation framework for pretrained vi- sion-language models that leverages asymmetric uncertainty in textual and visual embeddings. Text embeddings are modeled as random variables using directional distributions (e.g., von Mises-Fisher and power spherical) on the unit hypersphere, while image embeddings remain deterministic. Incor- porating these spherical likelihoods into an InfoNCE-based objective enables effective modeling of wide-variance aleatoric uncertainty language.
While Asym delivers impressive performance in cross- cross-modal retrieval with reliable uncertainty estimates, it only models text uncertainty. Future work could extend uncertainty modeling image embeddings via data augmentation or similar methods. Also, although example applications of the uncertainty estimates are provided, further exploration into additional downstream tasks is warranted.
Broader impact
Uncertainty estimates for text are vital in high-stakes applications involving VLMs, such as medical diagnostics where ambiguous language must be treated carefully. AsymVLM enhances trust in textual inputs, enabling safer and more robust applications of VLMs in umcertainty-sensitee domains.
Acknowledgement
The computations/data handling were enabled by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre, and by the Alvis resource provided by Chalmers e-Commons at Chalmers. LJ acknowledges funding from the Centre for Interdisciplinary Mathematics at Uppsala University and NAISS through Project 6024/22-1358 AH and PS acknowledge support from the Swedish Research Council through grant agreement nos. 2023-05167 and 2023-05593 respectively. EV's research partially supported by the Kjell and Märta Beijer Foundation, and the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. PS also acknowledges support from the the Knut and Alice Wallenberg foundation through the Program for Academic leaders in Life Science (PALS). The authors thank Aleksandr Karakulev, Csongor Horváth and Mayank Nautiyal for valuable discussions and feedback.
10

---

References
[1] Rabab Abdelfattah, Qing Guo, Xiaoguang Li, Xiaofeng Wang, and Song Wang. Cdul: Clip- driven unsupervised learning for multi-label image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 348-1357, 2023.
[2] Morris Alper and Hadar Averbuch-Elor. Emergent visual-semantic hierarchies in image-text representations. In European Conference on Computer Vision, pages 220-238. Springer, 2024.
[3] Anton Baumann, Rui Li, Marcus Klasson, Santeri Mentu, Shyamgopal Karthik, Zeynep Akata, Arno Solin, and Martin Trapp. Post-hoc probabilistic vision-language models. arXiv preprint arXiv:2412.06014 2024.
[4] Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, et al. An introduction to vision-language modeling. arXiv preprint arXiv:2405. 17247, 2024.
[5] Min Cao, Shiping Li. Juntao Li, Liqiang Nie, and Min Zhang. Image-text retrieval: A survey on recent research and development. arXiv preprint arXiv:2203.1471: 2022.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PmLR, 2020.
[7] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 15750-15758. 2021.
[8] Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8415 8424, 2021.
[9] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[10] Inderjit S Dhillon and Suvrit Sra. Modeling data using directional distributions. Technical report, Citeseer, 2003.
[11] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. CORR, 2024.
[12] Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning, 110(3):457-566, 2021.
[13] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[14] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision- language understanding and generation. In International conference on machine learning, pages 28888-12900 PMLR, 2022.
[15] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[16] Daniel W Lozier. Nist digital library of mathematical functions. Annals of Mathematics and Artificial Intelligence, 38: 105-119, 2003.
[17] Bryan Plummer, Liwei Wang, Chris Cervantes, Juan C Caicedo, Julia Hockenmaier. and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. Proceedings of the IEFE international conference on computer vision, pages 2641-2649, 2015.

---

[18] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748-8763. PMLR, 2021.
[19] Diego Ruiz-Antolín and Javier Segura. A new type of sharp bounds for ratios of modified bessel functions. Journal of Mathematical Analysis and Applications, 443(2): 1232-1246,, 1246, 2016.
[20] Mahtab Sandhu, Yann Batiste Pequignot, Samer B Nashed, Sabyasachi Sahoo, and Liam Paull. Clip-enhance: Improving clip zero-shot classification via von mises-fisher clustering. 2025.
[21] Tyler R Scott, Andrew C Gallagher, and Michael C Mozer. von mises-fisher loss; An exploration of embedding geometries for supervised learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16112-0622,, 2021.
[22] Philip Sedgwick. Spearman's 's rank correlation coefficient. Bmj, 349, 2014.
[23] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3556-2565, 2018.
[24] Yichun Shi and Anil K Jain. Probabilistic face embeddings. In Proceedings of the IEEE/CVF international conference computer vision, pages 6902-6911, 2019.
[25] Uddeshya Upadhyay, Shyamgopal Karthik, Massimiliano Mancini, and Zeynep Akata. Probvim: Probabilistic adapter for frozen vision- models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1899-1910, 2023.
[26] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Hal Daumé III and Aarti Singh, editors, Pro- ceedings of the 37th International Conference on Machine Learning, volume I 119 of Proceedings of Machine Learning Research, pages
PMLR, 13-18 Jul 2020.
[27] Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang. Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision- language model. In European Conference on Computer Vision, pages 408-424. Springer, 2025.
[28] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1995-11986, 2023.
[29] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Yes, our main claims made in the abstract and introduction are reflected by the paper's contributions through the experimental results Section 5.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The main limitations of the work, as well as future directions that may address some of these limitations, are discussed in Section 6.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Lemmas and theoretical results are well defined, and used motivate and ground the proposed approach, with explicit assumptions and proofs as applicable.
4. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The code for the paper is attached in the supplementary materials for the submission, and will open-sourced through GitHub. The data for the paper is publicly available.
Justification:
5. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All necessary details for experiments are provided in the Appendix.
6. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All experiments were repeated five times with different random seeds, and we report the mean results in the main text, with full standard deviations provided in the Appendix.
7. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: All computing resources utilized for the experiments in the paper are listed in the Appendix.
8. Code of ethics
13

---

Question:
Does the research conducted in the paper conform, in every respect, with the httpp//blcc/cccccccccieeeeeeeeee
NeurIPS Code of Ethics
Answer: [Yes]
Justification: This work does not entail any harmful consequences as laid the Code of Ethics.
9. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Potential broader impacts of the paper are discussed in the main text.
10. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: No such models or datasets are involved in the scope of this paper.
11. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All code, data and models used in the paper are properly cited, and the license and terms of use are respected as specified.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code for the paper is well documented with license, and terms of use being appended in the supplementary materials.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No such models datasets involved in the scope of this paper.
14. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: There
human participants involved in the scope of this paper.
15. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: LLMs are not used for any important, original or non-standard component of the core methods.
14

---

A Method Details
A.1 Approximation of the log normalizer of vMF distribution
The normalizer of vMF distribution (a(x) is given by,
kd/2-1
Ca(Rs) = (2m) La/2-/1)
Remark 1 (Derivatives modified Bessel functions [16]). Let (z(2) denote modified Bessel function v-th order. For k = 0, 1,2,...,
1 d
=""
dz
Lemma 1. The first order derivate of [o(2) is,
dz (=) = [o+1(2) - (.).
Proof.
Using Remark 1, let k = 1 and we have,
1 d 2 2 LI(2) -² 22-211-116 12-)
2 dz
dl = ( (2).
[o(z)
dz
Reorganizing the equation gives us,
dz [o(2) - (o-=)=) -0_ (2).
Further we use the Recurrence Relations of modified Bessel function [16].
2r In(2),
(=-1(=)-A I++1(2)== 2
and obtain,
dz = "I(()).
Note that,
dx Cn((())= = d
dx [[d/2 - 1) In d/2 In 1 (1/2211(11)]
d/2 - 1 1
12/2-1(1c)
Using Lemma 1, we have,
In (n((()) d/2-1 1 Ta/2(K<) + 4/2-1 I 12/2-1 (R)
dx
1d)2-1(K)
Ta/2(1s)
(16)
Laya(1s)
Ruiz-Antolin and Segura [19] give a tight lower and upper bound for
1a/2 1a/) given by following
Remark:
15

---

Remark 2 (Tight lower and upper bound for the ratio of modified Bessel functions [19]). Lei [x(2) be the modified Bessel function of the first kind at order v. We have following bound,
Lo(z)
1 z 1/2.
1/2+ + / (v 1/2)2 +
1=-1
-- U-1/22 V 1/2)2 + +
With the tight lower and upper bounds,
T.(2) (2)1(=) can be approximated by,
(2) 1 ((((=)) + h(=)),
1=)
where g=2) and (z) denote the left and right hand sides of the inequality.
To further approximate In Ca(r)), by plugging v d 2 obtain following approximates,
we further integrate the function w.r.t and
1
d
In Ca(k)
~
In
V
+ R²
+ k²
4
2
2
2
2
(4)
d- d-1
d-1 1
d +
2
d +
2
+ ln
+
V
+ n²
+ n: + const.
4
2
2
2
2
A.2 Inference with maximum log likelihood
The inference of deterministic pre-trained VLMS (i.e. cross-modal retrieval) relies on computing the cosine similarities of each possible pairs of given texts and images, while the inference for AsymVLM, has a clear statistical interpretation maximizing log likelihood w.r.t. different image- text pairs. For Image-to-Tess Text retrieval, given a set of text embeddings {2." |r [R]]. where z! ~ vMF (MFF((()) or z PS(u(t,), k(t,)), and an image embeddings z the predictor is given by,
(k(t,) (u(t,)) 2' + Fa(n(t,)) (k((t,) In(1 + (u(t,)))) + +InCa(x(tt)) for PS.
for vMF,
8 arg max In Psy (=*) rE[R]
Similarly, for Text-to-lmage retrieval, given set of image embeddings {=gls [S]} and text embedding 2 ~ vMF(H, or zt ~ PS(H, x),, the predictor given by,
K p T =! Fa(k))
for vMF,
arg max In per (=/) In + = + In Cx(k)
xE(S]
for PS.
A.3 Derivation for the SigLIP variant
The objective function of SigLIP is defined as,
1
ΣΣ In 1 exp(o(r, s)(- CosSim(r, s) + b))*
B
where (r, 8) = if and s)) -1 if if * 8. SigL.IP shares the identical "kernel", i.e.
cosine similarity, with CLIP, while our methods employs different kernels with variance modeling derived from vMF/PS assumptions for the text embeddings. By replacing the kernel of the objective function of SigLIP, we can also obtain the SigLIP variant of our method.
B Experimental Protocol
B.1 Adapter architecture
The text adapter is implemented as a four-layer perceptron that takes the output of a pre-trained text encoder its input. It consists of two hidden layers, each with 1024 dimensions and activated by the
16

---

ReLU function. followed by an output layer that has the same dimensionality as the input, and no activation function. The un-normalized output of the final layer, denoted as 2/, is decomposed into z ki/4, where the vector || || 1 and the scalar K > O, serving as the parameters for either vMF or PS distribution. The temperature parameter in the InfoNCE objective function is trainable.
Other adapters (ProbVLM, PFE" and PCME++") share identical architectures as three-layer percep- tron, with different number of outputs for the last layer due the inherent different parameterization in the methods.
B.2 Datasets
We use the datasets MS-COCO and Flickr-30k to train and validate the adapters since they provide well-annotated image-caption pairs. Additionally, we randomly sample 200k image-caption pairs from Conceptual Caption dataset [23], building a dataset (CC-200k) for the training and validation. For ablation studies, all methods are trained on the training set of CC-200k and evaluated on the validation set of CC-200k. For the study of understanding the learned uncertainty, the model is evaluated on HierarCap dataset [2], dataset adapted from Conceptual Caption to reflect the hierarchical structure of captions. For each image, there exists four captions of different abstraction levels from general to detailed ones. For example applications, CIFAR-10, CIFAR-100 [13] and STLIO [9] are used for zero-shot classification.
The evaluation of the quality of aleatoric uncertainty / ambiguity estimates for text embed- dings is based on the recall performance in cross modal retrieval tasks. A strong positive correlation between uncertainty and eиror indicates that when the model is more uncertain, it tends to produce lower-quality embeddings, which leads to poorer retrieval performance. For each task, we first group the recall results according to the uncertainty levels of the text embeddings. Then we compare Spear- man's rank correlation (S) between the uncertainty levels and recall for cross-modal retrieval [22]. We also compute the regression fit R² between the uncertainty levels and Recall @ performances measure if the drop performance follows linear trend. To evaluate the overall performance of cross-modal retrieval of the models trained with different methods, we also compare the average Recall @1 across all uncertainty levels.
Metries
B.3 Baseline methods
The uncertainty quantification baselines include ProbVLM, PFE and PCME++ adapted/trained in a post-hoe manner from pretrained models (denoted by PFE and PCME++", respectively). Note that the post-hoc adaptation for both PFE and PCME++ only predicts the variances of probabilistic embeddings, while keeping the means unchanged from the pretrained models. This is due to PFE and PCME++ achieving poor performance (worse than pre-trained models) for moosl retrieval, when used to to adapt the means of the embeddings. For downstream tasks, vanilla pre-trained vision-language models are also included. Bayes is not compared against with other methods in the main experiments, as it does not provide a single scalar value quantifying the uncertainty of its embeddings, rather scalar uncertainty for cosine similarities propagated by probabilistic embeddings.
B.4 Optimization protocol
All methods are implemented in PyTorch, and all pre-trained VLMs are loaded by transf ormnrs All computations are conducted on NVIDIA A 100//400 GPUs. We use stochastic gradient descent with momentum (SGD-momen)
for the optimization of AsymVLM, PCME and PFE, and use Adam W for PLoM following its official implementation. The learning rates for different methods optimized using grid search within {10 4, 5 x 10 10 3 5 x 10 3 10 2 5 10 10-2), -0- and reported as Table A.I. We apply cosine annealing for learning rate scheduling with a minimal learning rate 10-6
B.5 Computational costs
The overhead of post-hoc adaptation with AsymVLM is negligible compared to pre-training CLIP or SigL.P. This efficiency stems from two factors: (1) the adaptor is a lightweight three-layer MI.P operating in the 12-dimensional latent space (for CLIP); (2) all CLIP embeddings for adaptation
17

---

Table A.1: Optimizer configurations for different methods
Method
Optimizer
Learning rate
Batch size
AsymVLM
SGD
10 10-2
2048
ProbVLM
AdamW
10
2048
PFE
SGD
10
2048
PCME++
SGD
10
2048
data are cached after a single forward pass through the pretrained VLM. Adaptation training then proceeds solely on these cached embeddings, avoiding repeated runs through the large model and dramatically reducing compute cost. On an NVIDIA A 100, completing 200 epochs of adaptation on either MS-COCO, Flickr-30K or CC-200k requires under one GPU-hour.
C Supplementary Empirical Results
Complete experimental results for the evaluation of AsymVLM on CLIP embeddings are presented Table A.2 (mean std. dev. over runs).
Table A.2: Complete experimental results for the comparison of different methods for CLIP embed- dings.
Y21
DATASET
METHOD
Recall 1
S L
Recall@1 1
PPE*
0.500 +: 0.000
0.558
0.253
.0.222 + 0.193
0.304 +: 0.000
0.946
0.021
-0.988 :: 0.008
PCME&4*
0.500 + 0.000
01911
0.0066
-0.9% + 0.064
0.304 + 0.000
0.948 0.010
-0.99) = 0.004
Bam
MS-COCO
ProbVLM
0,490
0.004
0.951
0.011
00981
0.006
0.293
0.001
0.979
0.002
-1.000
0.000
ProLip
0.500 :: 0.000
0.908
1: 0.025
0,908
+ 0.031
0 1: 0.000
0.876 1: 0.007
-0.955
: 0.012
0,561 +: 0,002
0348
.0.88 + 0.013
0.392
0
0.984 0.044
-1,000
0,000
AsymVLMes
0.558 +: 0,004
0.937
0.013
0.984
0,012
0.390
0.002
0.989 0.001
-1,000
0,000
PPE
0.800 +: 0.000
0.675 + 0.186
.0832 + 0.094
0.451 +: 0.000
0.955 + 0.002
.0,998 :: 0.005
RCME44*
0.679 + 0.001
0.455 : 0.386
-0.178 + 01650
0:451 + 0.000
0,000 0.063
-0.918 : 0.067
VLm
FLICKR-30K
ProbVLM
0.666 +: 0.006
0.826
0.062
-0914
0.079
0.422 0.002
0.964 0.016
-0.985
0.012
ProLip
0.578 + 0.000
0.829 + 0.032
0.954 + 0.041
0.450 + 0.000
0.928 + 0.010
-0.978
:: 0.005
0,688 + 0.005
0,800
-0,9% 0.010
0.504
0.960 0.006
-0.995
A
::
0.021 0.54
-0.47 + 0.032
0.002 0.498
0.96 0.021
-0.995
2: 0.000
PFE
0.352 :: 0.000
0.857 0.075
-0.521 +: 0.757
0.336 1: 0.000
0.988 0.008
-0.595 :: 0.798
3CMEss
0.352 +: 0.000
0.198 + 0.375
-0.148 + 0.430
0.336 + 0.000
0.779 0.105
-0.857
: 0.060
RoyasVLM
0.016
-1.937
CC-200K
ProbVLM
0.316 :: 0.000
0.772 0.102
-0837
0.066
0.302 0.000
0.775 0.029
-0.965
0.021
ProLip
0.351 +: 0.000
0.968 1: 0.007
-0,950
+ 0.009
0.335 1 0.000
0.992 1: 0.002
-1,000 =: 0.000
0.395 1:
-1,000 1: 0.000
0.383
0.991 0.002
-0.998
AsymVLMgs
0.393 3: 0.001
0 0.004
-1,000 :: 0.000
0.380 1: 0.002
0.993 0.001
-1,000 0,000
Complete experimental results for the evaluation of AsymVLM on SigLIP embeddings are presented in Table A.3.
Complete experimental results for the evaluation of Asym VLM for robust out-of out-of-distributtin zero- shot classification are presented in Table A.4.
Complete experimental results for the evaluation of AsymVLM for zero-shot classification, with none-of-the-abbe handling are presented in Table A.5.
18

---

Table A.3: Full experimental results for the comparison of different methods for SigLIP embeddings.
12r
721
DATASET
METHOD
Recall
R
8
Recall@
n²
PEF.
0.654 + 0.000
0.940 + 0.010
-0.900
+ 0.005
0.472 + 0.000
0.99
0.001
-1,000
= 0.000
PC
+
-0.101
-0.907
Bayes
0.664 +: 0.001
0.541
0.022
-0780
+ 0.066
0.488 4: 0.002
0,923
0.006
-0.931
0.027
MS-COCO
ProbYLM
0.649 + 0.001
01564 + 0:063
-0.622 + 0,044
0.469 + 0:00
09355
0.007
-0.99 = 0.005
0,489
Asy
0,84 :: 0,002
0,948 0,018
00,95
0.006
0.502 1 0.000
0.001 0.986
-1,000
0.000
AsymVLMps
0.811 + 0.002
0.929
0.021
.0.93
+ 0.015
0.497 + 0.001
0,987 0.002
-1,000
0.000
PFE
0.815 + 0.000
0.840 + 0.023
-0.947 + 0,030
0.638 + 0.000
0,922 + 0.005
-0,99 : 0.005
PCME++*
BoyesVLM"
0.818 +: 0.001
0.4% +: 0.042
-0.776 +: 0.021
0.642 +: 0:001
0,515 0.019
-0.742 :: 0.024
FLICKR-30K
ProbYLM
0.807 + 0.002
01519 + 0:097
-0.714 + 0.119
0.631 + 0.000
0.525 0.038
-0.640 = 0.151
Am
0,823 +: 0,002
0.800 + 0.044
-0.67 + 0.010
0.647 + 0:001
0.934 0.100
-0,998
0.008
Asy
0.819 + 0.0006
0.891 IMI24
-0.959 + 0.012
I1649 1 0.002
0.008 0.949
+0,998
0.005
PFE
0,503 +: 0,000
0.981 + 0.003
-1,000 + 0,000
0:484 +: 0.000
0,992 0.004
-1,000 : 0.000
PCME&
+
0933 + 0.909
-0956
0.010
0.484
0889 0.016
-0.96
BLy
0.512 + 0.001
0.993 1: 0.006
-0,99%
+ 0.006
0.488 +: 0.000
0.981 1 0.010
-0.992 :: 0.008
CC-200K
ProbVIM
0,489 +: 0.002
0.674 + 0.081
-0.817
+ 0,071
0:477 + 0.000
0,744 0.022
-0.934
:: 0.029
0.795
Aa
0.516 d: 0.002
0.991 + 0.004
-1.000
+ 0.000
0.4%6 1: 0.002
0.993 0.002
-1.000 =: 0.000
As
0.514 +: 0.001
0.964 1 0.003
-1,000 + 0,000
0.489 + 0.002
0,985 0.002
-1,0000 0.000
Table A.4: Full experimental results for Out-of-distribution zero-shot classification accuracy on CIFAR-10 10, CIFAR- 100 and STL-10 for pretrained VLMs fine-tuned with different methods on MS-COCO, Flickr-30k and CC-200k datasets.
VALIDATED ON
FINE-TUNED ON METHOD
CIFA R-10
CIFAR-100
STL-10
IMAGENET-IR 1K
Determ. FT
0.684 1 0.055
0.300 + 0.017
0.829 + 0.030
0.3951 0.002
MS-COCO
AsymVLMg
0.847 + 0.007
0.470 + 0.012
0.952 = 0,005
0.5024 0.002
AsymVLM,
0.837 = 0.010
0.477 0.010
0.940 = 0.010
0.507 0.003
Determ. FT
0.688 + 0.030
0.342 + 0.019
0.874 = 0.012
0.369 + 0.007
FLICKR-30K
Asym VL.M
0.791 0.019
0.413 0.009
0.920
0.009
0.451
0.003
Asy VI.M .MF
0.792 + 0.011
0.422 0.014
0.918 0.006
0.464
0.002
Determ. FT
0.779 0.037
0.411 0.011
0.922
0.017
0,450
0.004
CC-200K
Asym VLMMV
0.861 + 0.008
0.542 1 0.005
0.968 = 0.001
0,520 - 0.002
AsymVLM.MM MF
0.866 + 0.013
0.542 + 0.013
0.967 = 0.004
0.527 = 0.001
NONE
CLIP
0.888
0.642
0.974
0.632
Table A.5: Complete results for the evaluation metrics for various zero-shot classification methods
using different dummy prompts. The table reports the accuracy on positive samples and negative samples when classifying all inputs into CIFAR-10 classes. Baseline methods using threshold- and margin- based rejection are included for comparison.
DUMMY PROMPT
METHOD
POSITIVE Acc.
NEGATIVE AcC.
CLIP
0,888
0.009
Determ FT
0.769 + 0.028
0.115:10.135
"a photo"
AsymVLMM
0.845 + 0.011
0.547-+0. 106
AsymVLM
0.857 + 0.010
0.587+0.110
CLIP
0.888
0.009
Determ. FT AsymVLM,
0.778 1 0.040
0.037-10022
"a photo of an object"
0.849 + 0.012
0.609--0.119
AsymVLM MF
0.858 + 0.011
0.557+0.088
CLIP
0.888
0.009
Determ. FT AsymVLM,
0.769 1 0.031
0.125.10.132
"a photo of something" NONE
0.843 + 0.019
0.595-00.149
AsymVL VL
0.857 + 0.009
0.585+10.107
Margin-Band
0.584
0.579
Threshold-Bane Bladd
0.646
0.560
19


Reviews:
[{"review_id": "VJpAZ4sxDI", "reviewer": "Reviewer_Hj1D", "summary": "The paper discusses that while VLMs perform well, they fail to capture ambiguity and uncertainty in the data. There are existing works handling this with post-hoc methods. However, these methods do not handle the asymmetric uncertainty. The paper claims that having the representations on a hypersphere results in suboptimal performance. The paper proposes AsymVLM, which builds probabilistic embeddings from pre-trained VLMs on the hypersphere, enabling uncertainty quantification.", "questions": "Please check weaknesses.", "limitations": "Weaknesses are addressed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength:\n\nThe paper dives into the asymmetric nature of text and image uncertainty. The motivation is valid and clear. The problem formulation is novel and intuitive. \n\nThe paper provides ablation studies investigating uncertainty levels. The paper also provides experiments comparing with several models, showing better performance of the model. In addition, there are various applications, including zero-shot and none-of-the-above. \n\nThe writing is without grammar or spelling errors. \n\n# Weaknesses:\n\n## Method explanation: \n\nWhile the writing in general is good, the overall method explanation, task definition and experiments need polishing to be better grasped.\n\nLines 30–33 discuss the asymmetric nature of text and images, but the explanation is difficult to follow. Referencing Figure 1 and including concrete examples could improve clarity. Since this section appears in the introduction and addresses the core motivation of the paper, the explanation must be clear and compelling.\n\nWhat’s the numerical representation space in line 84?\n\nIt would also be beneficial to give an overview of the building VLM section with examples to make it easy to grasp.\n\nWhat is the role of the text adaptor introduced in 4.1? \n\nThe introduction misses an overview of the contributions of the proposed method and a distinction from the existing works. \n\n## Uncertainty:\n\nThe paper makes statements or assumptions about the uncertainty; however, it still needs to provide support for the claims and assumptions. For example, I have a problem with the simplification and generalisation in lines 104-110. Can the paper provide support and references for the explanation of uncertainty? \n\nIn Figure 3, how are different uncertainty levels generated or calculated? What is the measure? What is the data? Line 186 references Figure 3; however, it is not clear what the setup is. The same stands for the experiment setups in 5.3.\n\nLine 49: Why do the models fail to capture the inherent ambiguity? Are there any supports for this? Any references investigating? \n\n\nSmall writing comments:\n\nIn line 79, is there a space in the middle of e.g.?\n\nFigure 1 is not referenced nor explained in the text.", "was_revised": true, "final_justification": "The rebuttal has answered my questions so I change my rating from 3 to 4.", "rebuttal": "Thank you for the helpful feedback. Please find our responses below.\n\n__Improving method explanation__\nIn the camera-ready manuscript, we will address the concerns by clarifying the problem formulation and method motivation in Section 3, and by adding an additional subsection about the calculation of the uncertainty from the resulting probabilistic embedding in Section 4, to eliminate any potential ambiguities.\n\n__On the asymmetric nature of text and images__ \nCrucially, text and images exhibit different kinds of one-to-many ambiguity (Fig. 1, left). The word \"dog\" can denote any of infinitely many dog instances (objects), each of which can in turn be photographed under many viewpoints, lighting conditions, etc. Thus text $\\to$ object is one-to-many via abstraction, and object $\\to$ image is one-to-many via visual variability. In contrast, a photo of a golden retriever unambiguously depicts one physical object but admits multiple valid captions (e.g., \"a dog,\" \"a golden retriever,\" \"a pet\", \"an animal\"). Thus image$\\to$text is one-to-many through diverse linguistic descriptions. Existing post-hoc probabilistic adapters treat these two retrieval directions symmetrically. \n\nAdditionally, existing methods overlook the fact that embeddings of VLMs pre-trained with cosine similarity lie on the unit hypersphere. We argue that modeling this asymmetry by capturing the uncertainty for text only all on the sphere, is the key to better cross-modal uncertainty estimation.\n\n__The numerical representation space__\nNumerical representation space: The numerical representation space (Line 84) is the joint embedding space into which both image and text inputs are projected by the encoders of a VLM. In this space, semantically matching images and texts yield embeddings that are close under a chosen similarity metric. The numerical representation space should be $\\mathbb{R}^d$ for Euclidean distance as similarity metric, and $\\mathbb{S}^{d-1}$ for cosine similarity, where $d$ is the dimension of the embedding.\n\n__Overview of building VLMs__ \nExisting datasets such as MS-COCO collect data in the form of $\\{(t_n, i_n)\\}_{n=1}^N$ where textual caption $t_n$ is able to describe image $i_n$. Introducing the (real-world) object space $\\mathcal{O}$, Text$\\to$Object mapping $\\Phi_T$ and Object$\\to$Image mapping $\\Phi_I$, the relationship between $t_n$ and $i_n$ can be be formally stated as: there exist an object $o_n\\in\\mathcal{O}$ such that $t_n$ can be used to describe $o_n$, i.e., $o_n \\in \\Phi_T(t_n)$, and $i_n$ is an image of $o_n$, i.e. $i_n \\in \\Phi_I(o_n)$. However, with no concrete way to represent real-world object space, we assume that there exist a numerical representation space $\\widetilde{\\mathcal{O}}$, such that for any object $o_n\\in\\mathcal{O}$, there exist a unique real-valued vector $\\tilde{o_n}\\in\\widetilde{\\mathcal{O}}$. The Object $\\leftrightarrow$ Representation mapping is denoted as $\\psi:\\mathcal{O}\\to\\widetilde{\\mathcal{O}}$ and $\\psi^{-1}:\\widetilde{\\mathcal{O}}\\to\\mathcal{O}$. \n\nFor CLIP-like VLMs, we aim to build two encoder to learn the mappings from images and texts to the joint numerical representation space, i.e. Image $\\to$ Representation and Text$\\to$Representation. The mappings can be decompose as Image $\\to$ Object $\\to$ Representation, i.e., $\\psi \\circ \\Phi_I^{-1}$ and Text $\\to$ Object $\\to$ Representation, i.e., $\\psi \\circ \\Phi_T$. Thus, with finite samples $\\{(t_n, i_n)\\}_{n=1}^N$, building VLMs can be interpreted as $f_I \\approx \\psi\\circ \\Phi_I^{-1}$ and $f_T \\approx \\psi \\circ \\Phi_T$, where $f_I$ and $f_T$ are image and text encoders respectively.\n\n__Role of the text adaptor__ \nWe convert deterministic text embeddings into probabilistic ones by feeding them through the AsymVLM adaptor (a small neural network) that predicts the parameters of our chosen spherical distribution (von Mises–Fisher or power spherical). \n\n__On clearly mentioning contributions__\nTo make both our contributions and our differences to prior work explicit, we will add the following paragraph at the end of Introduction. Thanks a lot for the suggestion.\n\n\"Contributions:  This paper makes three main contributions:\n- We formally identify and address the asymmetric uncertainty structure inherent in vision–language data, high aleatoric uncertainty in text (one-to‐many mapping) versus lower (aleatoric) uncertainty in images.\n- We propose AsymVLM, an adapter that exploits the asymmetric uncertainty structure and performs post-hoc probabilistic adaptation on the unit hypersphere, rather than in Euclidean space. We also show that AsymVLM is a natural extension of CLIP’s cosine-similarity loss with the added ability to quantify uncertainty.  \n- Empirically, AsymVLM yields more accurate uncertainty estimates and higher cross-modal retrieval accuracy on MS-COCO, Flickr30K and CC-200k benchmarks.  We further demonstrate its advantages in robust fine-tuning, zero-shot classification and \"none-of-the-above\" rejection.  \"\n\n__Clarifications about the uncertainty__\n\n1. __Clarification for Line 104-110:__ From an uncertainty-quantification perspective we adopt the standard taxonomy of aleatoric vs. epistemic uncertainty [1, 2]. Aleatoric uncertainty captures irreducible data ambiguity, e.g. the caption \"a dog\" denotes an entire class of breeds, poses and contexts, so no amount of additional training can pinpoint a single underlying object. Epistemic uncertainty captures model ignorance that could be reduced with more or better-distributed data, e.g., images, while depicting a unique scene with low aleatoric spread, may still suffer from coverage gaps if certain viewpoints or lighting conditions were under-represented during training. This asymmetry yields inherently broad aleatoric uncertainty in text versus more data‐dependent epistemic uncertainty in images. In practice, aleatoric uncertainty is modeled via parameterized distributions, whereas epistemic uncertainty is estimated by Monte Carlo methods (e.g., sampling from dropout layers or Bayesian neural networks). Accordingly, in this work, focusing on aleatoric uncertainty, we represent text embeddings as directional distributions on the unit hypersphere and retain deterministic image vectors.\n\n2. __How are different uncertainty levels generated or calculated?__\nThe uncertainty of each text embedding is quantified by the inverse concentration parameter of the embedding distribution, $1/\\kappa(t)$. Following the probVLM [3] convention, we sort all the uncertainty estimates in ascending order and split them into ten equally sized bins (levels 0–9), so that larger uncertainty values are assigned to higher levels. This quantile‐based scheme lets us evaluate and compare the quality of uncertainty estimates from different methods by observing that cross‐modal retrieval accuracy declines monotonically as the uncertainty level increases.\n\n3. __Why do the models fail to capture the inherent ambiguity?__\nAs noted in Section 3.1, because the mapping Text$\\to$Object is inherently one‐to‐many, the induced Text$\\to$Representation should likewise be one‐to‐many.  In practice, however, VLM text encoders approximate this mapping deterministically, i.e., the text encoder $f_t$ maps each text to a single point in the representation space, thereby failing to capture textual ambiguity. Prior works [3,4,5] have shown that probabilistic embeddings, which represent each text as a distribution rather than a point, more faithfully model semantic uncertainty and yield improvements on downstream tasks. Our empirical results further validate this conclusion.\n\n__Small writing issues__\nWe apologize for the identified writing issues and will fix them in the camera-ready version. In particular, we will properly reference Figure 1 in Section 3.1 and refine the surrounding discussion to clarify the paper’s main idea.\n\n## References\n[1] Hüllermeier, Eyke, and Willem Waegeman. \"Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.\" _Machine learning_ 110.3 (2021): 457-506.\n\n[2] Gawlikowski, Jakob, et al. \"A survey of uncertainty in deep neural networks.\" _Artificial Intelligence Review_ 56.Suppl 1 (2023): 1513-1589.\n\n[3] Upadhyay, Uddeshya, et al. \"Probvlm: Probabilistic adapter for frozen vison-language models.\" _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2023.\n\n[4] Chun, Sanghyuk. \"Improved Probabilistic Image-Text Representations.\" _The Twelfth International Conference on Learning Representations_. 2024.\n\n[5] Shi, Yichun, and Anil K. Jain. \"Probabilistic face embeddings.\" _Proceedings of the IEEE/CVF international conference on computer vision_. 2019."}, {"review_id": "kMgpB528ZR", "reviewer": "Reviewer_95zV", "summary": "This paper proposes a method for adapting existing vision-language models (VLMs) into probabilistic models. Instead of maximizing and minimizing the cosine similarity of positive and negative image-text pairs, the authors convert deterministic text embeddings into probability distributions parameterized by the network and then maximize the likelihood the probabilities at the corresponding image embeddings' position. Experimental results demonstrate that the proposed method enables effective fine-tuning with limited data and provides a way to estimate model uncertainty.", "questions": "How to understand the relationship between the uncertainty and the equations and setups in Sec. 4.1? For example, does uncertainty means the `variance` in the predicted distribution of text embeddings?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths  \n- The motivation is reasonable: a general concept or category described by text can correspond to a wide variety of objects and images in the real world.  \n- The introduced capabilities—robust fine-tuning with limited data and improved handling of “none-of-the-above” cases—are interesting and practically valuable. These are areas where original models like CLIP often struggle.\n\n## Weaknesses  \nI don't have major concern on this draft. But I think the asymmetric assumption requires further clarification. The authors claim that textual labels are high-level abstractions, but if the textual labels refer to the labels used in datasets, they can vary significantly in granularity. Some may describe fine-grained objects, while others may be abstract class names, as in ImageNet.  Hence, it would strengthen the paper if the authors discussed the inherent variability of free-form textual data and how their asymmetry assumption accounts for this. In particular, linking this discussion to the uncertainty experiments in Section 5.3 would help ground the assumption in practical evidence and clarify its implications.", "was_revised": true, "final_justification": "The reviewer has explained some of the minor questions regarding the paper. Hence, I keep my original rating.", "rebuttal": "Thanks a lot for the feedback and encouragement. Please find our response addressing your concerns as follows.\n\n__Clarification on the asymmetric assumption__\nWe thank the reviewer for pointing out the wide variability in label granularity, from single‐word class names to richly detailed captions. In AsymVLM this variability is captured by the learned concentration parameter $\\kappa$ of the text distribution: Class labels (Level 0 in HierarCap, Sec. 5.3) produce high aleatoric uncertainty with small-valued $\\kappa$, whereas more elaborate captions (Levels 1–3) yield progressively lower uncertainty with greater $\\kappa$, as shown in Fig. 5 (left panel). However, even the most verbose description cannot convey every pixel‐level detail of an image due to the inherently lossy, highly‐uncertain nature of text data. Thus, our asymmetric assumption both adapts class label to free‐form caption granularity via $\\kappa$ and acknowledges the irreducible uncertainty of text data. The monotonic decrease of uncertainty with caption detail in Sec. 5.3 provides concrete empirical grounding for this design choice. We agree that explicit discussion over various label granularity helps ground the assumption and will add it in the camera-ready manuscript upon acceptance.\n\n__Understanding the relationship between the uncertainty, equations and setups in Sec. 4.1__\nWe quantify text‐embedding uncertainty as  $u(t)=\\frac{1}{\\kappa(t)}$, due to the fact that for both the vMF and PS distributions on $\\mathbb{S}^{d-1}$, the (angular) variance $\\sigma^2$ is a strictly decreasing function of $\\kappa$ [1, 2]. Equivalently, larger $\\kappa$ concentrates mass more tightly around the mean direction (lower $\\sigma^2$), so $1/\\kappa$ grows monotonically with the distribution’s variance, providing a natural scalar value for uncertainty estimate for probabilistic embeddings.\n\n## References\n[1] De Cao, Nicola, and Wilker Aziz. \"The power spherical distribution.\" arXiv preprint arXiv:2006.04437 (2020).\n\n[2] Sra, Suvrit. \"A short note on parameter approximation for von Mises-Fisher distributions: and a fast implementation of I s (x).\" Computational Statistics 27.1 (2012): 177-190."}, {"review_id": "HeeLwmnABp", "reviewer": "Reviewer_qd63", "summary": "The paper proposes AsymVLM, a post-hoc probabilistic framework for pre-trained vision-language models (VLMs) that captures asymmetric uncertainty in textual and visual embeddings. AsymVLM models text embeddings as probability distributions on the unit hypersphere, while keeping image embeddings deterministic. This design reflects the asymmetric nature of image-text data: textual labels are abstract and ambiguous; images are more concrete but visually varied. Empirical results demonstrate that AsymVLM enhances uncertainty quantification and cross-modal retrieval performance across various datasets, including MS-COCO and Flickr30K.", "questions": "1.\tCould the authors clarify the semantic meaning and practical implications of the mappings in Table 1 (line 84), especially the one-to-many relationships (e.g., Text -> Object, Object -> Image)?\n2.\tWhy is zero-shot classification only evaluated on small datasets such as CIFAR-10, CIFAR-100, and STL-10? Would the method maintain its claimed robustness on larger-scale benchmarks like ImageNet?\n3.\tA central design in AsymVLM is to model uncertainty only on the text side while treating image embeddings as deterministic. However, the paper does not provide sufficient justification for this asymmetry. In real-world scenarios--such as medical imaging, satellite imagery, or abstract visual inputs--visual data can also be ambiguous or uncertain. Could the authors clarify why image-side uncertainty is entirely omitted, and whether incorporating it might improve performance or robustness in such settings?\n4.\tWhile the framework is evaluated mainly on retrieval and zero-shot classification, can the same asymmetry framework be applied to generative tasks (e.g., image captioning, VQA) where both modalities need rich bidirectional understanding? If so, how would the asymmetry affect generation quality?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\nThe paper presents a compelling and original insight by explicitly modeling the asymmetric nature of text-to-image and image-to-text relationships, capturing the fact that textual descriptions are inherently abstract and ambiguous, while images are concrete yet variable. This asymmetry is often overlooked in existing VLM literature, and its incorporation into the design of AsymVLM represents a significant conceptual advancement that deepens our understanding of multimodal alignment.\n\nWeaknesses:\n- While the paper presents an informative table outlining the mapping relationships between text, image, object, and representation spaces (line 84-85), it lacks sufficient explanation and justification for several one-to-many assumptions. Key notations such as $\\Phi_T$, $\\Phi_I$, and their inverses are introduced without thorough semantic grounding or illustrative examples. \n- While the paper demonstrates improvements over earlier probabilistic embedding methods such as PFE, PCME++, and ProbVLM, it omits direct comparisons with more recent and competitive approaches like BayesVLM [1] and ProLIP [2]. Although BayesVLM is briefly mentioned and excluded on the grounds that it does not produce scalar embedding uncertainties, this justification is insufficient given that BayesVLM targets the same core problem—uncertainty-aware retrieval—and has shown strong empirical performance. The absence of these baselines weakens the empirical validation and makes it harder to accurately assess the relative strengths of AsymVLM within the current landscape of vision-language uncertainty modeling.\n- Zero-shot classification is a critical benchmark for vision-language models, particularly those based on contrastive pretraining like CLIP. AsymVLM evaluates this capability only on three relatively small datasets (CIFAR-10, CIFAR-100, STL-10), which restricts the scope and generalizability of its conclusions. Furthermore, the paper claims that AsymVLM improves robustness under fine-tuning, but fails to convincingly explain why fine-tuning would impair zero-shot performance -- especially given that only a lightweight text-side adapter is trained while the CLIP backbone remains frozen. Since the core parameters of the VLM are untouched, any degradation in zero-shot ability appears inconsistent and unexplained. Without more comprehensive experiments (e.g., on ImageNet) and clearer justification, the claimed preservation of zero-shot capabilities remains insufficiently supported.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed most of my previous concerns. Thus, I decided to raise my score to support this work.", "rebuttal": "Thanks for the feedback and suggestions. Please find our responses addressing your concerns and questions below.\n\n__On explanation and justification for several one-to-many assumptions__\nWe briefly expand each mapping below, giving concrete examples, and explain why the one-to-many assumptions naturally arise, and motivate our probabilistic treatment of text versus image embeddings.\n\n- Text $\\to$ Object $\\Phi_T(t)$:\n\nSemantics: $t$ is a text caption (label) while $\\Phi_T(t)$ is the set of all real‐world object that fall under the label.\n\nOne-to-many: a single word “dog” applies to infinitely many individual dogs in real-world.\n\n- Object $\\to$ Image $\\Phi_I(o)$:\n\nSemantics: $o$ is a specific object instance while $\\Phi_I(o)$ is the set of all images of that same instance.\n\nOne-to-many: each specific dog can be imaged under countless viewpoints, lighting conditions, backgrounds, etc.\n\n- Text $\\to$ Image $(\\Phi_I \\circ \\Phi_T)(t)$:\n\nSemantics: the set of all images of any object that the label $t$ denotes.\n\nOne-to-many: combines the two one-to-many mappings above.\n\n---\n- Image $\\to$ Object $\\Phi_I^{-1}(i)$:\n\nSemantics: the (assumed unique) object instance depicted in image $i$.\n\nOne-to-one (by assumption): each image is taken of exactly one object instance.\n\n- Object $\\to$ Text $\\Phi_T^{-1}(o)$:\n\nSemantics: the set of all valid textual descriptions of object $o$.\n\nOne-to-many: one object can be described in many different ways (varying specificity, synonyms, context).\n\n- Image $\\to$ Text $(\\Phi_T^{-1} \\circ \\Phi_I^{-1})(i)$:\n\nSemantics: all captions that correctly describe the object in image $i$.\n\nOne-to-many: inherited from Object$\\to$Text, this is mainly because of the abstraction of text data.\n\n---\nThen with additional Object $\\leftrightarrow$ Representation $\\psi(o)$ and $\\psi^{-1}(\\tilde{o})$, which are both one-to-one mapping, it is clear that for CLIP-like VLMs, the text encoder $f_T \\approx \\psi(o) \\circ \\Phi_T(t)$ should be one-to-many while the image encoder $f_I \\approx \\psi(o) \\circ \\Phi^{-1}_I(i)$ should be one-to-one. Semantically, this corresponds to the irreducible aleatoric uncertainty of language: e.g., no amount of additional examples will resolve the inherent ambiguity of the word \"dog.\" Although images can be uncertain in practice, a well‐trained model with sufficiently diverse visual data can learn to disambiguate them. Any remaining uncertainty is primarily epistemic and thus reducible by collecting more data or improving the model.\n\n__On comparisons to BayesVLM and ProLIP__\nWe originally excluded BayesVLM from our baselines because it does not yield scalar uncertainty estimates for embeddings. However, we see that comparisons to ProLIP and BayesVLM add value to the paper. We will add ProLIP and BayesVLM to our baselines and report their performance alongside AsymVLM (see updated results below).\n\n|Dataset|Method|Recall(T2I)|$R^2$(T2I)|$S$(T2I)|Recall(I2T)|$R^2$(I2T)|$S$(I2T)|\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n|MS-COCO|$\\text{ProLIP}$|$0.500\\pm0.001$|$0.808\\pm0.025$|$-0.908\\pm0.031$|$0.304\\pm0.002$|$0.876\\pm0.007$|$-0.985\\pm0.012$|\n| |$\\text{AsymVLM}_{\\text{vMF}}$|$\\mathbf{0.561\\pm0.002}$|$\\mathbf{0.948\\pm0.016}$|$\\mathbf{-0.988\\pm0.013}$| $\\mathbf{0.392\\pm0.001}$|$0.984\\pm0.004$|$\\mathbf{-1.000\\pm0.000}$|\n| |$\\text{AsymVLM}_{\\text{PS}}$|$0.558\\pm0.004$|$0.937\\pm0.013$|$-0.984\\pm0.012$|$0.390\\pm0.002$|$\\mathbf{0.989\\pm  0.001}$|$\\mathbf{-1.000\\pm0.000}$|\n|FLICKR-30K|$\\text{ProLIP}$|$0.678\\pm0.000$|$0.829\\pm0.032$|$-0.914\\pm0.041$|$0.450\\pm0.001$|$0.928\\pm0.010$|$-0.978 \\pm0.005$|\n| |$\\text{AsymVLM}_{\\text{vMF}}$ | $\\mathbf{0.688\\pm0.005}$ | $\\mathbf{0.860 \\pm  0.029}$ | $\\mathbf{-0.976\\pm0.010}$ | $\\mathbf{0.504\\pm0.001}$ | $\\mathbf{0.960\\pm0.006}$ | $\\mathbf{-0.995\\pm0.006}$ |\n| |$\\text{AsymVLM}_{\\text{PS}}$|$\\mathbf{0.688\\pm0.005}$| $0.854\\pm0.021$|$-0.947\\pm0.032$| $0.498 \\pm  0.002$ | $0.946 \\pm  0.021$ | $-0.993 \\pm  0.006$ |\n| CC-200K | $\\text{ProLIP}$ |  $0.351 \\pm 0.001$ | $0.968 \\pm 0.007$  | $-0.990 \\pm 0.009$  |  $0.335 \\pm 0.001$ | $0.992 \\pm 0.002$  | $\\mathbf{-1.000 \\pm 0.000}$ |\n| | $\\text{AsymVLM}_{\\text{vMF}}$ | $\\mathbf{0.395 \\pm  0.002}$ | $0.990 \\pm  0.003$ | $\\mathbf{-1.000 \\pm 0.000}$ | $\\mathbf{0.383 \\pm  0.003}$ | $0.991 \\pm  0.002$ | $-0.998 \\pm 0.005$ |\n| | $\\text{AsymVLM}_{\\text{PS}}$  | $0.393 \\pm 0.001$ | $\\mathbf{0.992 \\pm 0.004}$ | $\\mathbf{-1.000 \\pm 0.000}$ | $0.380 \\pm 0.002$ | $\\mathbf{0.993 \\pm 0.001}$ | $\\mathbf{-1.000 \\pm 0.000}$ |\n\nDue to rebuttal–period time constraints, we were unable to retrain BayesVLM (or re-compute its Hessian) on the same datasets as the other models. Instead, we use the authors’ provided checkpoint. This checkpoint is based on OpenCLIP’s __ViT-B/32-laion2b_s34b_b79k__ foundation model, which was pretrained on a substantially larger dataset than the original CLIP __ViT-B/32__ and thus yields stronger cross-modal retrieval performance [1]. For fairness, we evaluate both BayesVLM and AsymVLM using this identical foundation model as reported below, with BayesVLM’s uncertainties derived from the probabilistic cosine similarity between image–text pairs.\n\n|Dataset|Method|Recall(T2I)|$R^2$(T2I)|$S$(T2I)|Recall(I2T)|$R^2$(I2T)|$S$(I2T)|\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| MS-COCO | $\\text{BayesVLM}$ | $0.541$ | $0.873$ | $-0.948$ | $0.421$ | $0.913$ | $-0.952$ |\n| | $\\text{AsymVLM}_{\\text{vMF}}$ | $0.601 \\pm 0.003$ | $\\mathbf{0.919 \\pm 0.012}$ | $-0.985 \\pm 0.012$ | $\\mathbf{0.421 \\pm 0.001}$ | $\\mathbf{0.977 \\pm 0.002}$ | $\\mathbf{-1.000 \\pm  0.000}$ |\n| | $\\text{AsymVLM}_{\\text{PS}}$ | $\\mathbf{0.602 \\pm 0.003}$ | $0.914 \\pm 0.023$ | $\\mathbf{-0.990 \\pm 0.009}$ | $0.419 \\pm 0.001$ | $0.976 \\pm 0.003$ | $\\mathbf{-1.000 \\pm  0.000}$ |\n| FLICKR-30K | $\\text{BayesVLM}$ | $0.698$ | $0.778$ | $-0.951$ | $0.517$ | $0.936$ | $-0.987$ |\n| | $\\text{AsymVLM}_{\\text{vMF}}$ | $\\mathbf{0.723 \\pm 0.002}$ | $\\mathbf{0.884 \\pm 0.022}$ | $-0.962 \\pm 0.019$ | $\\mathbf{0.534 \\pm 0.001}$ | $\\mathbf{0.979 \\pm 0.005}$ | $\\mathbf{-1.000 \\pm 0.000}$  |\n| | $\\text{AsymVLM}_{\\text{PS}}$ | $0.721 \\pm 0.003$ | $0.871 \\pm 0.011$ | $\\mathbf{-0.974 \\pm 0.015}$ | $0.528 \\pm 0.001$ | $0.977 \\pm 0.005$ | $-0.999 \\pm 0.001$ |\n| CC-200K | $\\text{BayesVLM}$ | $0.415$ | $0.914$ | $-0.952$ | $0.397$ | $0.935$ | $-0.987$ |\n| | $\\text{AsymVLM}_{\\text{vMF}}$ | $0.475 \\pm 0.003$ | $\\mathbf{0.991 \\pm 0.003}$ | $\\mathbf{-0.998 \\pm 0.005}$ | $\\mathbf{0.462 \\pm 0.002}$ | $0.984 \\pm 0.003$ | $\\mathbf{-1.000 \\pm 0.000}$ |\n| | $\\text{AsymVLM}_{\\text{PS}}$ | $\\mathbf{0.476 \\pm 0.002}$ | $0.987 \\pm 0.004$ | $-0.995 \\pm 0.006$ | $0.457 \\pm 0.002$ | $\\mathbf{0.988 \\pm 0.006}$ | $-0.998 \\pm 0.005$ |\n\nAsymVLM demonstrates higher recall@1 across all three benchmarks and reliable uncertainty estimates with higher $R^2$ and lower $S$ scores. In the camera-ready version, we will include ProLIP and a fully retrained BayesVLM (Hessian-evaluated under identical conditions).\n\n__On zero-shot performance of AsymVLM__ \nFine-tuning a pretrained VLM on a downstream task often comes at the expense of its original zero-shot generality, which is also known as catastrophic forgetting. Empirically, several studies of CLIP-style models have shown that fine-tuning on specialized tasks yields better in-domain accuracy but degrades zero-shot performance on unrelated benchmarks [2]. Here we claim that AsymVLM is a more robust approach to fine-tune VLMs compared to its deterministic counterparts, retaining more generalizability.\n\nFurther we report the performance of zero-shot classification of the fine-tuned CLIP model on ImageNet. The experiment results on ImageNet are consistent with the conclusion we had on smaller datasets CIFAR10, CIFAR100 and STL10 in the manuscript. The additional experiment results will be added in the camera-ready manuscript.\n\n| Fine-tuned on | Method | Accuracy |\n| :---: | :---: | :---: |\n| MS-COCO | $\\text{Det. AsymVLM}$ | $0.395\\pm 0.002$ |\n| | $\\text{AsymVLM}_{\\text{ps}}$  |   $0.502\\pm 0.002$ |\n| | $\\text{AsymVLM}_{\\text{vMF}}$ | $\\mathbf{0.507 \\pm  0.003}$ |\n| FLICKR-30K | $\\text{Det. AsymVLM}$ | $0.369 \\pm 0.007$ |\n| | $\\text{AsymVLM}_{\\text{ps}}$  | $0.451 \\pm  0.003$ |\n| | $\\text{AsymVLM}_{\\text{vMF}}$ | $\\mathbf{0.464 \\pm  0.002}$ |\n| CC-200K | $\\text{Det. AsymVLM}$     | $0.450 \\pm  0.004$ |\n| | $\\text{AsymVLM}_{\\text{ps}}$  | $0.520 \\pm  0.002$ |\n| | $\\text{AsymVLM}_{\\text{vMF}}$ | $\\mathbf{0.527 \\pm  0.001}$ |\n| NONE | $\\text{CLIP}$ | $0.632$ |\n\n__AsymVLM and generative tasks__\nAsymVLM, as a probabilistic generalization of CLIP and SigLIP, cannot be applied directly to other VLMs. For generative tasks, however, one can extend the same principle by having the text encoder predict not only a mean (or direction) on the hypersphere but also a concentration parameter. The inverse concentration then quantifies uncertainty similarly, and sampling from the resulting vMF or PS distribution during the forward pass encourages diversity in the outputs. Back-propagation through the sampling process can be handled by the reparameterization trick, as routinely done in VAEs [3], diffusion models [4], and flow-matching [5] models. We plan to explore this as future work, and will note this in the camera-ready version.\n\n## References\n[1] Cherti, M, et al. \"Reproducible scaling laws for contrastive language-image learning.\" _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 2023.\n\n[2] Zhai, Y, et al. \"Investigating the catastrophic forgetting in multimodal large language model fine-tuning.\" _Conference on Parsimony and Learning_. PMLR, 2024.\n\n[3] Kingma, DP., and Max W. \"Auto-encoding variational bayes.\" 20 Dec. 2013.\n\n[4] Ho, J, Ajay J, and Pieter A. \"Denoising diffusion probabilistic models.\" _Advances in neural information processing systems_ 33 (2020): 6840-6851.\n\n[5] Lipman, Y, et al. \"Flow Matching for Generative Modeling.\" _The 11th International Conference on Learning Representations_."}, {"review_id": "vz3Pak5DoR", "reviewer": "Reviewer_X6GY", "summary": "The paper studies incoprtaing uncertainty into VLMs (e.g. CLIP), in particular, the text encoder now outputs a distribution (over the sphere) rather than a deterministic vector, and the motivation is simple: A given text description could correspond to multiple images due to the underlying ambiguity of textual description. To turn the deterministic text encoder into a stochastic one, the authors attached a text adapter on the text encoder, which outputs the parameter (mean and variance) of a vMF distribution or a PS distribution, and the adapter is trained using standard contrastive loss. The authors then demonstrate a wide collection of interesting applications using the uncertainty provided by the model. \n\nOverall I find the paper very well written, and the proposed method simple and promising. I am not an expert in SSL / VLM, but I've worked on uncertainty quantification a lot, and I think this paper should be a solid accept.", "questions": "- Would love to see how such probabilistic vision language encoding model can be used in multimodality LLM such as  LLaVA, to enhance uncertainty quantification.\n\n- How is the adapter implemented, is it a linear projection head attached to the pre-trained encoder model?\n\n- Is the uncertainty captured by AsymVLM aleatoric or epistemic? I think it should be aleatoric right?\n\n- What could happen when the test image goes out of distribution with respect to the data used for training the adapter?", "limitations": "(Minor) The method potentially requires some significant overhead training the adapter due to the large training dataset size, and since the large batch size (2048) requirement may limit the accessibility of the method. It would be better if the author could discuss the total GPU hours required.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n- The paper is very well motivated.\n\n- The paper is very well organized and well written, and the figures are clear, illustrative, and informative.\n\n- The empirical results are very thorough and interesting! The experiment considers a wide range of settings related to uncertainty, from \"using uncertainty to predict performance\" (Fig. 3), to \"uncertainty's relationship with caption granularity\" (Fig. 5), robust finet-tuning under data sparse regime (Table. 2) and \"using uncertainty for outlier detection\" (Table 3.). All these different settings provide solid evidence on why we should care about a VLM with uncertainty quantification ability.\n\n- I really like the summarization in Eq. 3 and the text box between Line 165 ! It explains \"how AsymVLM works and differs from existing work\" in a very intuitive and straightforward way! It's always good to see such analysis explaining the probabilistic objective function from a non-probabilitic perspective.\n\n# Weaknesses\n\n- In section 3.1, the concept of \"object\" is introduced but I did not quite understand how it is used in the method, it seems to me that the dataset and the method only uses text and image if I understand correctly.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments, and encouragement. Please find our responses below. \n\n__Weakness: The concept of an \"object\"__\nHere \"objects\" are the unobserved, real‐world entities that both captions and images refer to, e.g., the particular dog (object) that gives rise to one caption (\"a dog\") and many images (different photos of that same dog).  We introduce $\\mathcal{O}$ in Section 3.1 purely as a conceptual bridge to formalize the one‐to‐many (text $\\to$ objects $\\to$ images) and many‐to‐one (images $\\to$ object $\\to$ texts) relationships.  In practice, neither our datasets, nor our losses ever see or require explicit object labels: we only train on paired $(t,i)$ tuples.  The \"object\" layer explains why text is inherently aleatoric-uncertainty-dominant (it abstracts over infinitely many objects) while images, given enough data, admit low aleatoric uncertainty (they pinpoint one object).  The method itself encodes text as a directional distribution and images as points on the representation space $\\widetilde{\\mathcal{O}}$, without ever instantiating or inferring $o\\in \\mathcal{O}$.\n\n__Q1: On probabilistic VL encodings and multimodal models__\nMultimodal models like LLaVA typically combine a pre-trained visual encoder (e.g., CLIP’s visual backbone) with a large language model, while leaving the pre-trained textual encoder unused. Thus, AsymVLM is not directly applicable for such multimodal models. However, it is possible to fine-tune LLMs in a similar probabilistic approach as AsymVLM does by adding an additional adaptation layer (adaptor) in the bottleneck layers of LLMs. The adaptor predicts both the mean and concentration parameter of LLM’s text‐embedding distribution. The inverse concentration naturally serves as an uncertainty measure, and and sampling from this distribution during the forward pass promotes diverse generations. Back-propagation is enabled via the reparameterization trick, as is common in VAEs [1], diffusion [2] and flow matching models [3]. Although we have not yet empirically validated this method due to time constraints, it offers a promising avenue for principled uncertainty quantification in foundation models beyond CLIP-style encoders.\n\n__Q2: The adaptor's architecture__ \nThe architecture of the adaptor is described in the Appendix B.1. The adaptor is implemented with a multi-layer perceptron which consists of two hidden layers, each with 512 dimensions and activated by ReLU function, followed by an output layer that has the same dimensionality as the input with no activation function. Batch normalization is applied to all layers to stabilize training. The un-normalized output of the final layer, denoted as $z^\\prime(t)$, is decomposed into $z^\\prime(t)=\\kappa(t)u(t)$, where the vector $\\lVert u(t)\\rVert_2=1$ and the scalar $\\kappa >0$, serving as the parameters for either vMF or PS distribution.\n\n__Q3: Epistemic v/s aleatoric uncertainties__ \nThe uncertainty of text embeddings captured by AsymVLM is aleatoric. To estimate the epistemic uncertainty of embeddings, one in general needs to adopt dropout layers or Bayesian neural network in the adaptor, and sample the output with multiple forward passes, which is beyond the scope of our work.\n\n__Q4: Out-of-distribution test images__\nWhen a test image comes from a domain very different from captions used to train the adapter, our asymmetric uncertainty adapter preserves much more of the pre-trained generalization. As shown in Sec 5.4, Table 2, after training on MS-COCO, FLICKR-30K or CC-200K, the probabilistic CLIP model still achieves strong zero-shot accuracy on CIFAR-10, CIFAR-100 and STL-10 datasets to a large extent. This robustness arises because fine-tuning updates not only the mean $u(t)$ but also the concentration $\\kappa(t)$ of each text embedding, by allowing $\\kappa(t)$ to decrease (i.e., broaden the distribution) for more abstract captions, AsymVLM avoids over-specialization and remains effective on out-of-distribution images.\n\n__Limitation: Computational costs__\nThe overhead of post-hoc adaptation with AsymVLM is negligible compared to pre-training CLIP or SigLIP. This efficiency stems from two factors: (1) the adaptor is a lightweight three-layer MLP operating in the 512-dimensional latent space (for CLIP); (2) all CLIP embeddings for adaptation data are cached after a single forward pass through the pretrained VLM. Adaptation training then proceeds solely on these cached embeddings, avoiding repeated runs through the large model and dramatically reducing compute cost. On an NVIDIA A100, completing 200 epochs of adaptation on either MS-COCO, FLICKR-30K or CC-200k requires under one GPU-hour. Thanks a lot for the suggestion, a discussion on these lines will definitely improve the paper and we will add these details to the camera ready version.\n\n\n## References\n[1] Kingma DP, Welling M. Auto-encoding variational bayes, 2013 Dec 20\n\n[2] Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models. Advances in neural information processing systems. 2020;33:6840-51.\n\n[3] Lipman Y, et al. \"Flow Matching for Generative Modeling.\" _The 11th International Conference on Learning Representations_."}]

Author rebuttal:

Meta-review: The paper introduces AsymVLM, a lightweight, post-hoc framework to equip pre-trained Vision-Language Models (VLMs) with uncertainty quantification capabilities. The central scientific claim is that vision-language data possesses an intrinsic asymmetric uncertainty structure: text is subject to high, irreducible (aleatoric) ambiguity, whereas visual data has primarily reducible (epistemic) uncertainty. AsymVLM operationalizes this insight by modeling text embeddings as probabilistic distributions on the unit hypersphere, while keeping image embeddings deterministic. The findings demonstrate that this asymmetric approach improves cross-modal retrieval, provides better-calibrated uncertainty estimates, and enhances the robustness of fine-tuning on downstream zero-shot classification tasks.

The reviewers find the paper's core idea novel and well-motivated ("The paper presents a compelling and original insight by explicitly modeling the asymmetric nature of text-to-image and image-to-text relationships... This asymmetry is often overlooked in existing VLM literature, and its incorporation into the design of AsymVLM represents a significant conceptual advancement." by qd63), the empirical evaluation strong ("The empirical results are very thorough and interesting! The experiment considers a wide range of settings related to uncertainty... All these different settings provide solid evidence on why we should care about a VLM with uncertainty quantification ability" by X6GY), and the paper well written ("The paper is very well organized and well written, and the figures are clear, illustrative, and informative." by X6GY). 

In the meantime, reviewers also mentioned that the core asymmetry assumption and the methodology needs more justification ("it would strengthen the paper if the authors discussed the inherent variability of free-form textual data and how their asymmetry assumption accounts for this." by 95zV, and "While the writing in general is good, the overall method explanation, task definition and experiments need polishing to be better grasped." by Hj1D), and the experimental validation can be further improved ("omits direct comparisons with more recent and competitive approaches like BayesVLM and ProLIP" and that the zero-shot evaluation "restricts the scope and generalizability of its conclusions" by qd63).

During rebuttal period, the paper's initial weaknesses, primarily the omission of key recent baselines and the lack of large-scale zero-shot experiments, were decisively addressed during the rebuttal period. The authors provided significant new results comparing their method against state-of-the-art models and expanded their evaluation to ImageNet, substantially strengthening the paper's claims and leading reviewers to raise their scores. Therefore, the recommendation for a acceptance is based on the paper's elegant and original contribution, now supported by a rigorous and more complete empirical validation that resolves all major initial concerns, which passes NeurIPS' high quality bar.
Decision: Accept (poster)