Title: Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere
Venue: NeurIPS 2025
Accepted: yes
Keywords: Uncertainty Quantification

Vision-Language Models

Probabilistic Modeling
Abstract: Vision-language models (VLMs) as foundation models have significantly enhanced performance across a wide range of visual and textual tasks, without requiring large-scale training from scratch for downstream tasks. However, these deterministic VLMs fail to capture the inherent ambiguity and uncertainty in natural language and visual data. Recent probabilistic post-hoc adaptation methods address this by mapping deterministic embeddings onto probability distributions; however, existing approaches do not account for the asymmetric uncertainty between modalities, and the constraint that meaningful deterministic embeddings reside on a unit hypersphere, potentially leading to suboptimal performance. In this paper, we address the asymmetric uncertainty structure inherent in textual and visual data, and propose AsymVLM to build probabilistic embeddings from pre-trained VLMs on the unit hypersphere, enabling uncertainty quantification. We validate the effectiveness of the probabilistic embeddings on established benchmarks, and present comprehensive ablation studies demonstrating the inherent nature of asymmetry in the uncertainty structure of textual and visual data.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies incoprtaing uncertainty into VLMs (e.g. CLIP), in particular, the text encoder now outputs a distribution (over the sphere) rather than a deterministic vector, and the motivation is simple: A given text description could correspond to multiple images due to the underlying ambiguity of textual description. To turn the deterministic text encoder into a stochastic one, the authors attached a text adapter on the text encoder, which outputs the parameter (mean and variance) of a vMF distribution or a PS distribution, and the adapter is trained using standard contrastive loss. The authors then demonstrate a wide collection of interesting applications using the uncertainty provided by the model. \n\nOverall I find the paper very well written, and the proposed method simple and promising. I am not an expert in SSL / VLM, but I've worked on uncertainty quantification a lot, and I think this paper should be a solid accept.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n- The paper is very well motivated.\n\n- The paper is very well organized and well written, and the figures are clear, illustrative, and informative.\n\n- The empirical results are very thorough and interesting! The experiment considers a wide range of settings related to uncertainty, from \"using uncertainty to predict performance\" (Fig. 3), to \"uncertainty's relationship with caption granularity\" (Fig. 5), robust finet-tuning under data sparse regime (Table. 2) and \"using uncertainty for outlier detection\" (Table 3.). All these different settings provide solid evidence on why we should care about a VLM with uncertainty quantification ability.\n\n- I really like the summarization in Eq. 3 and the text box between Line 165 ! It explains \"how AsymVLM works and differs from existing work\" in a very intuitive and straightforward way! It's always good to see such analysis explaining the probabilistic objective function from a non-probabilitic perspective.\n\n# Weaknesses\n\n- In section 3.1, the concept of \"object\" is introduced but I did not quite understand how it is used in the method, it seems to me that the dataset and the method only uses text and image if I understand correctly.\n\nQUESTIONS:\n- Would love to see how such probabilistic vision language encoding model can be used in multimodality LLM such as  LLaVA, to enhance uncertainty quantification.\n\n- How is the adapter implemented, is it a linear projection head attached to the pre-trained encoder model?\n\n- Is the uncertainty captured by AsymVLM aleatoric or epistemic? I think it should be aleatoric right?\n\n- What could happen when the test image goes out of distribution with respect to the data used for training the adapter?\n\nLIMITATIONS:\n(Minor) The method potentially requires some significant overhead training the adapter due to the large training dataset size, and since the large batch size (2048) requirement may limit the accessibility of the method. It would be better if the author could discuss the total GPU hours required.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes AsymVLM, a post-hoc probabilistic framework for pre-trained vision-language models (VLMs) that captures asymmetric uncertainty in textual and visual embeddings. AsymVLM models text embeddings as probability distributions on the unit hypersphere, while keeping image embeddings deterministic. This design reflects the asymmetric nature of image-text data: textual labels are abstract and ambiguous; images are more concrete but visually varied. Empirical results demonstrate that AsymVLM enhances uncertainty quantification and cross-modal retrieval performance across various datasets, including MS-COCO and Flickr30K.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\nThe paper presents a compelling and original insight by explicitly modeling the asymmetric nature of text-to-image and image-to-text relationships, capturing the fact that textual descriptions are inherently abstract and ambiguous, while images are concrete yet variable. This asymmetry is often overlooked in existing VLM literature, and its incorporation into the design of AsymVLM represents a significant conceptual advancement that deepens our understanding of multimodal alignment.\n\nWeaknesses:\n- While the paper presents an informative table outlining the mapping relationships between text, image, object, and representation spaces (line 84-85), it lacks sufficient explanation and justification for several one-to-many assumptions. Key notations such as $\\Phi_T$, $\\Phi_I$, and their inverses are introduced without thorough semantic grounding or illustrative examples. \n- While the paper demonstrates improvements over earlier probabilistic embedding methods such as PFE, PCME++, and ProbVLM, it omits direct comparisons with more recent and competitive approaches like BayesVLM [1] and ProLIP [2]. Although BayesVLM is briefly mentioned and excluded on the grounds that it does not produce scalar embedding uncertainties, this justification is insufficient given that BayesVLM targets the same core problem—uncertainty-aware retrieval—and has shown strong empirical performance. The absence of these baselines weakens the empirical validation and makes it harder to accurately assess the relative strengths of AsymVLM within the current landscape of vision-language uncertainty modeling.\n- Zero-shot classification is a critical benchmark for vision-language models, particularly those based on contrastive pretraining like CLIP. AsymVLM evaluates this capability only on three relatively small datasets (CIFAR-10, CIFAR-100, STL-10), which restricts the scope and generalizability of its conclusions. Furthermore, the paper claims that AsymVLM improves robustness under fine-tuning, but fails to convincingly explain why fine-tuning would impair zero-shot performance -- especially given that only a lightweight text-side adapter is trained while the CLIP backbone remains frozen. Since the core parameters of the VLM are untouched, any degradation in zero-shot ability appears inconsistent and unexplained. Without more comprehensive experiments (e.g., on ImageNet) and clearer justification, the claimed preservation of zero-shot capabilities remains insufficiently supported.\n\nQUESTIONS:\n1.\tCould the authors clarify the semantic meaning and practical implications of the mappings in Table 1 (line 84), especially the one-to-many relationships (e.g., Text -> Object, Object -> Image)?\n2.\tWhy is zero-shot classification only evaluated on small datasets such as CIFAR-10, CIFAR-100, and STL-10? Would the method maintain its claimed robustness on larger-scale benchmarks like ImageNet?\n3.\tA central design in AsymVLM is to model uncertainty only on the text side while treating image embeddings as deterministic. However, the paper does not provide sufficient justification for this asymmetry. In real-world scenarios--such as medical imaging, satellite imagery, or abstract visual inputs--visual data can also be ambiguous or uncertain. Could the authors clarify why image-side uncertainty is entirely omitted, and whether incorporating it might improve performance or robustness in such settings?\n4.\tWhile the framework is evaluated mainly on retrieval and zero-shot classification, can the same asymmetry framework be applied to generative tasks (e.g., image captioning, VQA) where both modalities need rich bidirectional understanding? If so, how would the asymmetry affect generation quality?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for adapting existing vision-language models (VLMs) into probabilistic models. Instead of maximizing and minimizing the cosine similarity of positive and negative image-text pairs, the authors convert deterministic text embeddings into probability distributions parameterized by the network and then maximize the likelihood the probabilities at the corresponding image embeddings' position. Experimental results demonstrate that the proposed method enables effective fine-tuning with limited data and provides a way to estimate model uncertainty.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths  \n- The motivation is reasonable: a general concept or category described by text can correspond to a wide variety of objects and images in the real world.  \n- The introduced capabilities—robust fine-tuning with limited data and improved handling of “none-of-the-above” cases—are interesting and practically valuable. These are areas where original models like CLIP often struggle.\n\n## Weaknesses  \nI don't have major concern on this draft. But I think the asymmetric assumption requires further clarification. The authors claim that textual labels are high-level abstractions, but if the textual labels refer to the labels used in datasets, they can vary significantly in granularity. Some may describe fine-grained objects, while others may be abstract class names, as in ImageNet.  Hence, it would strengthen the paper if the authors discussed the inherent variability of free-form textual data and how their asymmetry assumption accounts for this. In particular, linking this discussion to the uncertainty experiments in Section 5.3 would help ground the assumption in practical evidence and clarify its implications.\n\nQUESTIONS:\nHow to understand the relationship between the uncertainty and the equations and setups in Sec. 4.1? For example, does uncertainty means the `variance` in the predicted distribution of text embeddings?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses that while VLMs perform well, they fail to capture ambiguity and uncertainty in the data. There are existing works handling this with post-hoc methods. However, these methods do not handle the asymmetric uncertainty. The paper claims that having the representations on a hypersphere results in suboptimal performance. The paper proposes AsymVLM, which builds probabilistic embeddings from pre-trained VLMs on the hypersphere, enabling uncertainty quantification.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength:\n\nThe paper dives into the asymmetric nature of text and image uncertainty. The motivation is valid and clear. The problem formulation is novel and intuitive. \n\nThe paper provides ablation studies investigating uncertainty levels. The paper also provides experiments comparing with several models, showing better performance of the model. In addition, there are various applications, including zero-shot and none-of-the-above. \n\nThe writing is without grammar or spelling errors. \n\n# Weaknesses:\n\n## Method explanation: \n\nWhile the writing in general is good, the overall method explanation, task definition and experiments need polishing to be better grasped.\n\nLines 30–33 discuss the asymmetric nature of text and images, but the explanation is difficult to follow. Referencing Figure 1 and including concrete examples could improve clarity. Since this section appears in the introduction and addresses the core motivation of the paper, the explanation must be clear and compelling.\n\nWhat’s the numerical representation space in line 84?\n\nIt would also be beneficial to give an overview of the building VLM section with examples to make it easy to grasp.\n\nWhat is the role of the text adaptor introduced in 4.1? \n\nThe introduction misses an overview of the contributions of the proposed method and a distinction from the existing works. \n\n## Uncertainty:\n\nThe paper makes statements or assumptions about the uncertainty; however, it still needs to provide support for the claims and assumptions. For example, I have a problem with the simplification and generalisation in lines 104-110. Can the paper provide support and references for the explanation of uncertainty? \n\nIn Figure 3, how are different uncertainty levels generated or calculated? What is the measure? What is the data? Line 186 references Figure 3; however, it is not clear what the setup is. The same stands for the experiment setups in 5.3.\n\nLine 49: Why do the models fail to capture the inherent ambiguity? Are there any supports for this? Any references investigating? \n\n\nSmall writing comments:\n\nIn line 79, is there a space in the middle of e.g.?\n\nFigure 1 is not referenced nor explained in the text.\n\nQUESTIONS:\nPlease check weaknesses.\n\nLIMITATIONS:\nWeaknesses are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}