Probabilistic Language-Image Pre-Training

Authors: Sanghyuk Chun, Wonjae Kim, Song Park, Sangdoo Yun
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: vision-langauge-pretraining, vision-language-model, probabilistic-embeddings

Abstract:
Vision-language models (VLMs) embed aligned image-text pairs into a joint space but often rely on deterministic embeddings, assuming a one-to-one correspondence between images and texts. This oversimplifies real-world relationships, which are inherently many-to-many, with multiple captions describing a single image and vice versa. We introduce Probabilistic Language-Image Pre-training (ProLIP), the first probabilistic VLM pre-trained on a billion-scale image-text dataset using only probabilistic objectives, achieving a strong zero-shot capability (e.g., 74.6% ImageNet zero-shot accuracy with ViT-B/16). ProLIP efficiently estimates uncertainty by an ``uncertainty token'' without extra parameters. We also introduce a novel inclusion loss that enforces distributional inclusion relationships between image-text pairs and between original and masked inputs. Experiments demonstrate that, by leveraging uncertainty estimates, ProLIP benefits downstream tasks and aligns with intuitive notions of uncertainty, e.g., shorter texts being more uncertain and more general inputs including specific ones. Utilizing text uncertainties, we further improve ImageNet accuracy from 74.6% to 75.8% (under a few-shot setting), supporting the practical advantages of our probabilistic approach. The code is available at https://github.com/naver-ai/prolip

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730260049625

thread_type:
official_review

<thread_object>
creator: reviewer_8wie

<content>
summary:
The paper introduces **Probabilistic Language-Image Pre-training** (ProLIP), an approach to vision-language models that integrates probabilistic modelling to capture the inherent uncertainty in image-text relationships. ProLIP departs from conventional deterministic embeddings by mapping inputs as random variables, estimating uncertainty using an "uncertainty token," and enforcing an inclusion loss to maintain distributional relationships between image-text pairs and between original and masked inputs.

strengths:
-  This paper is well-written.
-  The research problem and motivation are well-defined, highlighting that CLIP-like models tend to oversimplify image-text alignment by assuming a one-to-one relationship.
-  ProLIP is claimed to be the first approach to pre-training probabilistic VLMs on a billion-scale image-text dataset, a feat that requires extensive computational resources.

weaknesses:
-   The improvement over deterministic VLMs like CLIP is limited, with only a marginal gain of around 0.9% in ImageNet accuracy (Table 1) when models are exposed to the same number of samples.
-   While the paper introduces the uncertainty token for efficient uncertainty estimation, it lacks empirical evidence demonstrating the claimed efficiency.
-   Compared to PCME++, ProLIP incorporates an uncertainty token and an inclusion loss, scaling the training dataset to CLIP’s level, yet the observed improvement remains modest, limiting the paper’s overall contribution.
-  ProLIP requires the use of CSD to perform zero-shot prediction, which increases its complexity and makes it less straightforward than CLIP.

questions:
- Although the authors claim ProLIP efficiently estimates uncertainty by incorporating an "uncertainty token" without adding parameters, this approach effectively increases the input length. Given that transformer complexity is quadratic relative to the input length, this addition likely increases the computation required for multi-head attention. How does the "uncertainty token" quantitatively compare to previous probabilistic VLMs in terms of training efficiency and inference speed? For example, the authors can report training time per epoch, inference latency on standard hardware, or FLOPs compared to baseline models.

- Is there any limitation discussed in the paper? (e.g. computational requirements)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730519933329

thread_type:
official_review

<thread_object>
creator: reviewer_rzmv

<content>
summary:
The paper proposes a probabilistic VLM model that accounts for the many-to-many relationships between images and text. This model introduces an uncertainty token to estimate uncertainty without significantly increasing the parameter count. Additionally, the inclusion loss enhances interpretability by enforcing consistency between image-text pairs and between original and masked inputs.

strengths:
The paper creatively applies probabilistic modeling to VLMs to better capture the many-to-many relationships between image and text pairs. It introduces a straightforward setup by adding an uncertainty token, effectively minimizing additional parameters. The paper provides a detailed analysis of the sources and levels of uncertainty in both image and text modalities, and explores various applications of this approach.

weaknesses:
The paper primarily evaluates the model's performance on single-object scenarios, as seen with ImageNet in the main results (Table 1). While Table C.1 provides metrics for datasets with longer, more complex queries, such as Flickr, ProLIP shows a significant drop compared to CLIP. Including performance metrics and a detailed analysis of datasets with complex queries involving multiple objects and interactions in the main results would offer a more comprehensive evaluation of the model's capabilities.

questions:
1. In Table C.1, ProLIP underperforms compared to CLIP on several datasets, such as Flickr. Could the authors provide a detailed analysis to explore the reasons behind this discrepancy?
2. The paper suggests that images with a single object are more uncertain than those with complex scenes. Intuitively, one might expect complex scenes to have higher uncertainty due to the variety of possible descriptions. Could the observed uncertainty levels be influenced by the choice of dataset used in the analysis?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700271076

thread_type:
official_review

<thread_object>
creator: reviewer_t6gq

<content>
summary:
The paper works on a probabilistic CLIP that can model the uncertainty of the data. The model measures the uncertainty with a multi-dimensional Gaussian distribution, and it is trained with proposed novel inclusion loss. The paper also simplified the sampled distance loss from previous paper. Given that, the paper first shows some analysis that the estimated uncertainty aligns with several intuitions. Lastly, it shows two applications of image traversals (find a more concrete caption for the original caption iteratively) and prompt enhancement for the zero-shot image classification task.

strengths:
1. The paper constructs a probabilistic VLM that can capture of uncertainty from the multimodal dataset. 

2. The paper proposes the inclusion loss and also provide intuitive understanding of it. 

3. The paper contains interesting analysis of the uncertainty. 

4. The paper proposes to utilize the uncertainty to conduct prompt rewriting.

weaknesses:
1. The intuition behinds the paper is that multimodal paired data can be actually many-to-many mapping. I think that this intuition would lead to a multi-modes representation intuitively, as different text can be assigned to a single image, e.g., "A dog is walking" and "A man is looking at the building". However, the modeling in this paper is a Gaussian distribution with a single mean that can not conduct good mode coverage. 

1. (Minor) The basic behinds contrastive learning is that the positive pair is sampled from the joint distribution $P(x, y)$ while the negative pairs are sampled independently $P(x)P(y)$. Given that, contrastive learning does not force a 1-1 mapping from its maths intuition. Practically, the behavior might be different from the above as model learning might force 1-1. The motivation in Abs/Intro and Sec 2.1 might need a revision. The current claim is a little bit stronger to me and this might be fixed by writing or explanations. Also, in Sec 2.2, it claims that "VL tasks suffer from uncertainty as shown in Sec 2.1". This is a little bit beyond the context of Sec 2.1, as Sec 2.1 does not have strong VL task experimental evidence. I treated it as over-claim now but I think that it can be solved by writing revision. 

1. The two applications (image caption traversals and prompt enhancement) of the estimated uncertainty are not representative, and it would be better if more applications (more concretely, proof of concepts) are presented. For example, the data filtering, error estimation, etc.

questions:
1. Please specify the context why $\mu_1$ and $\mu_2$ are norm-1 vectors (i.e., because CLIP normalizes them during training?)? I think that the original CLIP only normalizes when calculating the contrastive loss, maybe there is a re-parameterization here. Also please include them in text which would be friendly to reader. 

2. I wonder how the estimated variance from the model is used in the classification task? 

2. I appreciate the comparisons to KL divergence in Sec 3.3. The paper only measures KL(p1||p2) which has a different form to the proposed inclusion metric. How about adding a comparison to metric KL(p1 || p2) - KL(p2 || p1)? 

3. The inclusion loss is defined as $L(Z1, Z2)$ = $Z1 \in Z2$. Thus, in Line 259 and Line 264, should that be $L(t, v)$ and $L(partial, all)$ instead the current $L(v, t)$ and $L(all, partial)$? 

4. In Fig 5, it shows that the estimated variance of image is smaller than text. The loss defined in Eqn (6) is unsymmetrical for image and text (because of the hyperparemeters and the text VIB loss). Would that be possible that the smaller image variance is from this loss design? 

4. Please specify the exact formulation (i.e., implementation) of VIB in main text or in appendix.

4. Please specify how the uncertainty information is utilized when conducting the image traversals task. 

5. (An open question; no concrete answer required) What would be the golden uncertainty of a particular data in the PrVLM? I.e., the best distribution that a model should converge to?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730727183726

thread_type:
official_review

<thread_object>
creator: reviewer_n1uc

<content>
summary:
This paper introduces ProLIP, a fully probabilistic approach to language-image pre-training. The authors begin by noting that CLIP's loss function oversimplifies the complex relationships between real-world images and texts. In response, ProLIP proposes a novel loss function that better captures the distributional relationships between these modalities. Extensive experiments were conducted to demonstrate the effectiveness of the proposed method.

strengths:
- This paper is well-motivated, starting from the many-to-many matching relationships within a batch of images and texts. It is also well-structured and easy to follow.
- The authors provided strong mathematical support for the proposed learning objective.
- Extensive experiments were conducted to demonstrate the effectiveness of the proposed method.
- The proposed method has been proven effective on datasets containing billions of image-text pairs.

weaknesses:
- The proposed method was trained only on ViT-B and ViT-L vision encoders. Scaling up to ViT-H  and comparing with other methods are important to further demonstrate the scalability of ProLIP.
- As shown in Table 1, ProLIP introduces a more complex loss design and training process compared to CLIP, yet it achieves only a 0.9% improvement in ImageNet zero-shot classification and an average gain of 0.7%. Also in Table C.1, ProLIP with ViT-L also just exhibits slight improvement over CLIP. This modest improvement may indicate scalability challenges for the proposed method.

questions:
Please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730967592528

thread_type:
official_review

<thread_object>
creator: reviewer_1uqo

<content>
summary:
The paper proposes a novel approach for vision-language models (VLMs) that leverages probabilistic embeddings to capture better the inherent many-to-many relationships between images and text descriptions. 

In general, the contribution includes
* Novel Probabilistic Embeddings: The paper introduces a fully probabilistic VLM that assigns each image-text pair a probability distribution rather than a deterministic embedding, addressing the inherent ambiguity in real-world image-text matching. This approach shows promise in capturing a richer semantic alignment by allowing multiple captions to represent an image and vice versa.
* Efficient Uncertainty Estimation: Unlike prior probabilistic VLMs, ProLIP efficiently estimates uncertainty by introducing an “uncertainty token” ([UNC]). This no-parameter simplification is computationally efficient and effectively captures the variance within embeddings, enhancing interpretability.
* Inclusion Loss: This paper introduces a novel inclusion loss, which enforces distributional inclusion relationships, and strengthens ProLIP’s interpretability by aligning learned uncertainties with human expectations (e.g., general captions should cover more specific ones).
* Performance on Zero-Shot Classification: ProLIP demonstrates strong results in zero-shot classification tasks, such as outperforming a CLIP baseline with similar architecture. The improvement in ImageNet classification accuracy (from 74.6% to 75.8%) with the use of uncertainty highlights the approach’s practical advantages.

strengths:
1. Originality
* The paper introduces a novel Probabilistic Vision-Language Model (PrVLM), named ProLIP, which represents a shift from deterministic vision-language models to probabilistic embeddings. This approach innovatively captures the natural ambiguity in image-text relationships, where multiple captions may describe a single image, and a single caption may match multiple images.
* The concept of an uncertainty token ([UNC]) is a creative contribution, allowing the model to quantify uncertainty in a computationally efficient way, without additional parameters. This method differentiates ProLIP from previous probabilistic VLMs, which typically rely on additional modules to estimate uncertainty.
* The inclusion loss is another novel component of the paper. By enforcing a distributional inclusion relationship, the loss function allows the model to create more interpretable representations that respect hierarchical relationships within image-text pairs. This focus on hierarchical embedding structures is an innovative step for VLMs.

2. Quality
* The authors have conducted a thorough and robust set of experiments to validate ProLIP's effectiveness. These include zero-shot classification tasks across multiple datasets, evaluations of ProLIP’s uncertainty estimation, and comparisons with both deterministic (e.g., CLIP) and probabilistic baselines.
* The paper is methodologically sound and clearly specifies the assumptions, architectures, and loss functions. The experiments are carefully controlled, with baseline comparisons and ablation studies that effectively illustrate ProLIP’s strengths in various contexts.

3. Clarity
* The paper is written in a clear and accessible manner. Technical terms and novel concepts (e.g., inclusion loss, uncertainty token) are well-defined, with explanations that make the content readable.
* Figures, such as the visualizations, are well-crafted and effectively illustrate key points. 

4. Significance
* The introduction of probabilistic embeddings and inclusion loss contributes to the broader AI field’s understanding of how uncertainty modeling can enhance interpretability and generalization in multi-modal tasks. This may inspire further work in this direction.

weaknesses:
1. Interpretability and Visualization: 
*  Although the inclusion loss aligns with human intuitions, the visualization methods could be improved. For example, it would be interesting to see more direct comparisons with non-probabilistic models in terms of how well ProLIP aligns visual and textual hierarchies in practice.

2. Limited Exploration of Task-Specific Uncertainties: 
*  The paper briefly touches on using uncertainty for image-to-text retrieval but could expand by exploring how uncertainty contributes to other downstream tasks. For instance, analyzing ProLIP’s impact on retrieval diversity or specificity could further highlight its advantages.

3. Human evaluation
*  While the experiments indicate that ProLIP’s learned uncertainties align with human intuitions, a concrete user study or human evaluation would reinforce these claims.

questions:
1. Could you clarify the design choice of the probabilistic pairwise contrastive loss (PPCL) over simpler alternatives, like KL divergence-based objectives? Have you tested how PPCL compares with other probabilistic loss functions in terms of training stability and final performance?

2. The inclusion loss is an interesting addition. Could you explain in more detail the intuition behind using this specific formulation, and how it compares with other metrics like Wasserstein distance?

3. The paper shows that ProLIP’s uncertainty estimates align well with general human expectations, such as more uncertainty for shorter text descriptions. Did you conduct any quantitative or qualitative user studies to validate this alignment? If not, how would you suggest evaluating the alignment between model uncertainty and human perception in future work?

4. You demonstrated ProLIP’s zero-shot classification ability, but could it also support a few-shot learning approach? What further modifications would be needed to explore few-shot learning capabilities and potentially improve its applicability in low-resource settings?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731936052858

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision summary

comment:
We truly appreciate all the reviewers and the chairs for their commitment to the community and thoughtful reviews. We are highly encouraged that the reviewers found that the proposed ProLIP is novel (Reviewer 1UQo, t6Gq, RZmV, 8Wie), the [UNC] architecture is creative (Reviewer 1UQo, RZmV), the inclusion loss and PPCL are novel and well supported (Reviewer 1UQo, N1uc, t6Gq), our manuscript is clear, well-written and well-motivated (Reviewer 1UQo, N1uc, 8Wie) and the conducted experiments are extensive and interesting (Reviewer 1UQo, N1uc, t6Gq, RZmV). We luckily have a chance to make our work stronger, with constructive and valuable comments from the reviewers. We have addressed all the raised concerns by the reviewers in the revised paper. The revised contents are highlighted in blue (if we modify the existing text) or red (if we add new text).

Here, we would like to clarify the changes since the last submission, and how they resolve the concerns raised by the reviewers.

First, we would like to clarify that while we addressed the concern raised by Reviewer RZmV, we found that our original experiments on 1.28B seen samples had a fair comparison issue; we made a mistake on data augmentation setting (we specified that we use 0.8-1.0 crop size, but we found that CLIP and ProLIP with 1.28B seen samples were trained with different crop sizes, e.g., 0.4-1.0 not 0.8-1.0). For a fair comparison, we re-evaluated ProLIP, CLIP and SigLIP on 1.28B seen samples and here is the result (also see revised paper Table 1):

| Method | # seen samples | IN | IN dist. | VTAB | Retrieval | Average |
|--------|-------|------|------|------|------|------|
| CLIP   | 1.28B | 67.2 | 55.1 | 56.9 | 53.4 | 57.1 |
| SigLIP | 1.28B | 67.4 | 55.4 | 55.7 | 53.4 | 56.7 |
| ProLIP | 1.28B | 67.8 | 55.3 | 58.5 | 53.0 | 57.9 |

Second, we found that we used the incorrect notation for “inclusion”. It should be $\subset$, rather than $\in$. We fixed all the notations in the revision.

Finally, we named the proposed prompt re-weighting method “Bayesian Prompt Re-Weighting (BPRW)” to make our contribution clearer.

To address the reviewers’ concerns, our revision also includes:

- We slightly revise the introduction and the related work to emphasize our contribution and avoid misunderstanding (Reviewer t6Gq [W2] [Q8])
- We add additional discussion of the desired probabilistic embedding space in Appendix A.1. This section addressed concerns by Reviewer N1uc [W2], Reviewer t6Gq [W2] [Q8], Reviewer RZmV [Q2] and Reviewer 8Wie [W1].
- We clarify the questions by Reviewer t6Gq [Q1] in Section A.3 (re-parameterization and l2 normalization) and Reviewer t6Gq [Q2] in Section B.2 (classification with variance)
- Section A.3 also clarifies the novelty of ProLIP compared to PCME++ to resolve the concern by Reviewer 8Wie [W2]
- We add more discussions and formulations for our loss function in Appendix A.6 and A.7 (Reviewer 1UQo [Q2], Reviewer t6Gq [Q3] [Q6])
- We add more details of image traversal and more visualization in Appendix B.3 (Reviewer 1UQo [W1] [W3] [Q3], Reviewer t6Gq [Q7])
- We add an architecture efficiency comparison experiment in Section C.3 and Table C.6 to address Reviewer 8Wie [W2] [Q1].
- Section C.7 contains more applications of uncertainty (Reviewer 1UQo [W2] and Reviewer t6Gq [W3])
- We add more discussion and limitation related to ProLIP in Appendix D (Reviewer t6Gq [W1], Reviewer 8Wie [Q2])
</content>
</thread_object>

</thread>
<thread>
creation date:
1733066643433

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Reminder

comment:
Dear reviewers,

Thank you for your constructive and positive feedback! We are encouraged by the positive consensus reached among the reviewers, with all scores indicating "6: marginally above the acceptance threshold".

As the discussion phase will be closed shortly (within a day), we would like to kindly remind you to acknowledge our latest responses:
- We are still awaiting Reviewer 1UQo's response. We have addressed all the concerns raised by the reviewer and updated the paper accordingly. Following the reviewer's suggestion, we add a new visualization (Figure B.1), a proxy of human study (Section C.6), and more downstream tasks of uncertainty (Section C.8)
- We clarified to Reviewer N1uc that (1) our work is not directly comparable to openclip or Cherti, Mehdi, et al (2) We have already provided results demonstrating performance improvements as parameter size increases (ViT-B/16 $\rightarrow$ ViT-L/16 $\rightarrow$ ViT-SO400M/14) (3) ViT-H experiments have not been conducted by prior vision-language representation works. In contrast, our approach operates at an even larger scale compared to other VLM works published in top-tier ML conferences, and (4) we are now running ViT-H fine-tuning, which is currently 60% complete and shows promising results (77.4% IN top-1, while SO400M at the same iteration achieved 77.3%). Note that we chose SO400M rather than ViT-H to fit the original author-reviewer discussion deadline which was planned to be closed last week.
- We are also waiting for the responses from Reviewer t6Gq, RZmV, and 8Wie. We kindly requested insights into why your scores are 6 (marginally above acceptance) rather than 8 (accept). If there are specific areas for improvement, we are eager to address them to strengthen our submission further. We would like to ask the reviewer to be willing to update their score in a more positive direction from marginally above the acceptance threshold (6) to accept (8).

Thank you once again for your valuable contributions to the community. We look forward to your positive feedback and responses.

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces Probabilistic Language-Image Pre-training (ProLIP), an approach to vision-language models that integrates probabilistic modelling to capture the inherent uncertainty in image-text relationships. ProLIP departs from conventional deterministic embeddings by mapping inputs as random variables, estimating uncertainty using an "uncertainty token," and enforcing an inclusion loss to maintain distributional relationships between image-text pairs and between original and masked inputs. Lastly, it shows two applications of image traversals (find a more concrete caption for the original caption iteratively) and prompt enhancement for the zero-shot image classification task. Extensive experiments were conducted to demonstrate the effectiveness of the proposed method.

Strengths:
+ This paper is well-motivated, starting from the many-to-many matching relationships within a batch of images and texts. It is also well-structured and easy to follow.
+ The paper contains a detailed analysis of the sources and levels of uncertainty. Meanwhile, the authors provided strong mathematical support for the proposed learning objective.
+ Extensive experiments were conducted to demonstrate the effectiveness of the proposed method.
+ The proposed method has been proven effective on datasets containing billions of image-text pairs.

Weaknesses:
+ More experimental results on large model sizes are expected.
+ The improvements over the baselines are limited.
+ ProLIP requires the use of CSD to perform zero-shot prediction, which increases its complexity and makes it less straightforward than CLIP.
+ Discussions about the limitations are needed.