Probabilistic Language-Image Pre-Training
Authors: Sanghyuk Chun
Wonjae Kim
Song Park
Sangdoo Yun
Venue: ICLR 2025 Poster (2025)
Keywords: vision-langauge-pretraining
vision-language-model
probabilistic-embeddings
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We introduce Probabilistic Language-Image Pre-training (ProLIP), the first probabilistic VLM pre-trained on a billion-scale image-text dataset using only probabilistic objectives, achieving a strong zero-shot capability

Abstract: Vision-language models (VLMs) embed aligned image-text pairs into a joint space but often rely on deterministic embeddings, assuming a one-to-one correspondence between images and texts. This oversimplifies real-world relationships, which are inherently many-to-many, with multiple captions describing a single image and vice versa. We introduce Probabilistic Language-Image Pre-training (ProLIP), the first probabilistic VLM pre-trained on a billion-scale image-text dataset using only probabilistic objectives, achieving a strong zero-shot capability (e.g., 74.6% ImageNet zero-shot accuracy with ViT-B/16). ProLIP efficiently estimates uncertainty by an ``uncertainty token'' without extra parameters. We also introduce a novel inclusion loss that enforces distributional inclusion relationships between image-text pairs and between original and masked inputs. Experiments demonstrate that, by leveraging uncertainty estimates, ProLIP benefits downstream tasks and aligns with intuitive notions of uncertainty, e.g., shorter texts being more uncertain and more general inputs including specific ones. Utilizing text uncertainties, we further improve ImageNet accuracy from 74.6% to 75.8% (under a few-shot setting), supporting the practical advantages of our probabilistic approach. The code is available at https://github.com/naver-ai/prolip

Published as a conference paper at ICLR 2025
PROBABILISTIC LANGUAGE-IMACE PRE-TNGNNING
Sanghyuk Chun NAVER AT Lab
Wonjae Kim
Song Park
Sangdoo Yun
ABSTRACT
Vision-language models (VLMs) embed aligned image-text pairs into a joint space but often rely on deterministic embeddings, assuming a one-to-one correspon- dence between images and texts. This oversimplifies real-world relationships, which are inherently many-to-many, with multiple captions describing single im- age and vice versa. We introduce Probabilistic Language-Image Pre-training (Pro- LIP), the first probabilistic VLM pre-trained on a billion-scale image-text dataset using only probabilistic objectives, achieving a strong zero-shot capability (e.g., 74.6% ImageNet zero-shot accuracy with ViT-B/16), ProLIP efficiently estimates uncertainty by an "uncertainty token" without extra parameters. We also introduce novel inclusion loss that enforces distributional inclusion relationships between image-text pairs and between original and masked inputs. Experiments demon- strate that, by leveraging uncertainty estimates, ProLIP benefits downstream tasks and aligns with intuitive notions of uncertainty, e.g. shorter texts being more un- certain and more general inputs including specific ones. Utilizing text uncertain- ties, we further improve ImageNet accuracy from 74.6% to 75.8% (under a few- shot setting), supporting the practical advantages of our probabilistic approach. The code is available at https: /tll:://giieee //gpt/bb..///////////////t/hhhtt com/ naver- wwwavverrrrrpproo ai/prolip.
INTRODUCTION
Vision-language models (VLMs) aim for a joint vision-language embedding space, and have become a comerstone in the recent advance of machine learning (Radford et al., 2021; Jia et al.. 2021; Li et al., 2022; Zhai et al., 2023). For training. VLMs map an aligned image-text pair (e.g., an image and its corresponding caption) into the same space using contrastive learning. Their rich joint representations learned from large-scale image-text aligned datasets have achieved significant success in various downstream tasks, such as zero-shot classification (by treating class labels as templated text, e.g., a photo of {.}) or image-text cross- retrieval.
Despite their great success, most VLMs encode representations into a deterministic Euclidean space. This assumes a one-to-one correspondence between images and texts, which oversimplifies the com- plex nature of real- relationships. In practice, image-text matching is inherently many-tor to-many. Multiple captions can accurately describe an image, each highlighting different aspects of the visual content. For example, a train image can be described by multiple captions, e.g,, "a train", "train station" or "train parked next to a station". Conversely, a caption may correspond to several images describing similar scenes or objects, e.g., "a train" can be matched to all the train images. However, as shown in Figure I (b), a deterministic model (e.g., CLIP (Radford et al., 2021)) fails to capture the multiplicity, e.g., "Train Station" embedding is located to an irrelevant point to the other train images and captions. This is because the CLIP loss forces the positive pairs close and random negative pairs far away, which has no stable solution when we map them onto a point Euclidean space.
Instead of representing an input to deterministic point vector, we aim to map an input to ran- dom variable. As shown in Figure (a), our probabilistic VLM (PrVLM) approach can handle the multiplicity, e.g., the distribution of "Train Station" covers all the train image distributions. This paper introduces Probabilistic Language-Image Pre-training (ProLIP), the first PrVLM pre-trained on billion-scale image-text pairs only using probabilistic objectives. Compared to previous PrVLM works (Chun et al., 2021; Ji et al., 2023; Upadhyay et al., 2023; Chun, 2024), ProLIP has several advantages. First, while the previous methods need dedicated module to predict the uncertainty, ProLIP estimates uncertainty very efficiently simply by adding an "uncertainty token" ([UNC)) to input without other additional parameters. Second, introduce novel inclusion loss, which

---

Published as a conference paper at ICLR 2025
"A commuter train traveling down the tracks the next station" "A train is passing by while an unlooker is standing next to track" "A yellow and blue train next to an overhang"
Prob, image embedkings embeddings
Det. image embeddings
Det. text embedddings
"Train Station
Returne backgrondd
THC2 0.109
"Trin Station
(a) ProLIP embedding space
(b) Deterministic embedding space
Figure 1: Comparison of ProLIP and deterministic embedding spaces. We visualize images and captions from MS-COCO Caption (Chen et al., 2015) using models trained on DataComp 1B (Gadre et al., 2024) with 1.28B seen samples (See Appendix A.2 for more details of visualization method). ProLIP can capture multiplicity of image-text matching (e.g., the text embedding of "Train station" covers all three train images), while deterministic embeddings fäil to capture the ambiguity. Furthermore, when synthetically remove the background, ProLIP maps the new embedding near the original embedding but with larger uncertainty value (0.109- + 0.117, while the deterministic model maps the new embedding very far from the original embedding.
forces the distributional inclusion relationship between an image-text pair and between the original input data and the masked one. Our new objective helps embeddings be more interpretable by hu- mans. Third, PIoLIP can be trained from scratch without needing any pre-trained models and achieve state-of-the-ar zero-shot capability without fine-tuning. Furthermore, ProLIP achieves strong zero- shot capability, e.g., 74.6% ImageNet zero-shot accuracy with the ViT-B/16 backbone, where the CLIP model with the same number of seen samples achieves 73.5% (Iharcoo al., 2021).
In the experiments, PIP slightly outperforms the deterministic CLIP model in zero-shot classi- fication (ZSC) tasks (e.g., CLIP shows 67.2 ImageNet ZSC accuracy, while ProLIP shows 67.6).
We also show the benefits of using uncertainty estimates for image-text tasks. First, we observe that our intuition and the learned uncertainty are aligned well. For example, (I) texts generally "include" images (i.e., texts are more uncertain than images), (2) shorter texts tend to be more uncertain, (3) more general texts/images tend to be more uncertain and include more specific ones (e.g., masked image and "Train Station" in Figure 1). Furthermore, we show two applications when a proper un- certainty estimate is helpful; Bayesian Prompt Re- Weighting (BPRW), fully Bayesian approach to seek better ImageNet zero-shot prompts which improves accuracy from 74.6% - 75.8%), and the uncertainty-basee based dataset traversal, which provides a better understanding of datasct hierarchy.
2
PRELIMINARY
2.1
INHERENT AMBIGUITY INDUCED BY THE MULTIPLICITY OF IMAGE-TER PAIRS
The nature of image-text matching is many-to-many Unfortunately, in practice, this multiplicity is not fully annotated in VL datasets; we only treat one corresponding caption as "positive" caption, while the others are considered as "negative". For example, in COCO Caption (Chen et al., 2015), more than 80% of positive correspondences are labeled as "negative" (Chun et al., 2022). As ob- served by Chun (2024), this hidden multiplicity inherently causes ambiguity for VL datasets. For example, assume we have the caption "a train is next to a train station" and three semantically simi- lar images showing a train next train station. Here, there will be only one positive image for the caption due to the construction protocol of VL datasets. If we approximate three image embeddings as the same image embedding, the correspondence between this approximated embedding, and the caption will be uncertain (i..,, either positive or negative). Suppose we use deterministic matching loss, such as contrastive loss used by CLIP (Radford et al., 2021). In this case, the best deterministic mapping will map the caption embedding not very close to the image embeddings but "properly" far away from them. CLIP does not have enough capability to capture multiplicity and ambiguity. We aim to achieve embedding space that represent the inherent uncertainty of the input (also known as "aleatoric uncertainty" ") for a more interpretable and understandable embedding space. We include more detailed discussion in Appendix A.
2

---

Published as a conference paper at ICLR 2025
Probahilistic Contrastive (Fq. 2) Inclusion Less () ((Fq..)
Inclusion Loss C T
cat wears red hat
log %
log o
Lo-norm
La-norm
[CLS]][UNC.]
[UNC]][CLS]
[M] [M] cat [M] a [M] hat
Inclusion
Vinoks)
Visual Encoder
Textual Encoder
grey
wears
red
hat
Figure 2: Overview of ProLIP. |CLS| and [UNC] tokens used for and logo respectively.
2.2 PROBABILISTIC IMAGE-TEXT REPRESENTATIONS
Probabilistic embeddings map each data point as random variable (e.g.. Gaussian distribution) rather than fixed vector, capturing the inherent uncertainty and diversity. This approach offers a bet- ter understanding of the semantic space by providing extra axis of uncertainty, e.g., can quan- tify the uncertainty of the input by using the estimated uncertainty (e.g., covariance of Gaussian). Recently, Kirchhof et al. (2023) theoretically have shown that probabilistic representation learning with a proper probabilistic matching loss can recover the correct aleatoric uncertainty, Namely, a probabilistic mapping can capture the ambiguity of the inputs. Probabilistic embeddings have been actively studied for applications with inherent ambiguity, such as word embeddings (Nguyen et al., 2017), image embeddings (Oh et al., 2019), face understanding (Shi & Jain, 2019; Chang et al., 2020), 2D-to-3D pose estimation (Sun et al., 2020), speaker diarization (Silnova et al., 2020), video understanding (Park al., 2022), and composed image retrieval (Neculai al., 2022).
As we discussed in Section 1 and A. VL tasks also suffer from aleatoric uncertainty caused by inherent multiplicity of image-text matching and sparse annotations. Recently, there have been attempts to tackle the inherent ambiguity of VL. tasks with probabilistic embeddings (Chun et al., 2021; et al., 2023; Upadhyay et al., 2023; Chun, 2024). However, these methods have a very lim- ited scale to be used as generic purpose VLM, such CLIP. For example. ProbVLM (Upadhyay et al., 2023) is an ad-hoc module top on the frozen pre-trained CLIP, limiting the full exploration of the probabilistic space. Furthermore, ProbVL.M is only trained small image caption datasets, such as CUB (Wah et al., 201 1) or COCO caption (Chen et al., 2015), which makes it not applicable more practical zero-shot classification applications. MAP (Ji al., 2023) proposes pre-training method using a cross-attention Transformer. However, it has a limited zero-shot capability, resulting in the need to fine-tune the model for each downstream task. Furthermore, its structure highly inef- ficient for retrieval systems; it needs both image and text pair to compute a similarity between them, i.e., we have compute all possible image-text pairs to get the full similarity. Lastly, PCME+ (Chun, 2024) showed a possibility of pre-trained PrVLM,, but their scalability is still limited (e.g.. achieving 34% ImageNet zero-shot accuracy). We empirically observe that the objective function of +CME++ shows slow unstable training under large-scale image-text pairs. Furthermore, all these PrVLMs need heavy additional parameters to estimate uncertainty from data. ProLIP does not need dedicated module for uncertainty estimate but employs very efficient strategy using [UNC].
3 PROBABILISTIC LANGUAGE-IMAGE
3.1 ARCHITECTURE
We model an input as a Gaussian random variable with a diagonal covariance by estimating mean and variance o² vectors from the imput. Similar to CLIP (Radford et al., 2021), ProLIP has separate visual and textual encoders. We use Tranfoerrn (ViT) (Dosovitskiy et al., 2021) 1) for the visual encoder and Transformer (Vaswani al., 2017) for the textual encoder.
3

---

Published as a conference paper at ICLR 2025
Previous Probabilistic VLMs (PrVLMs) introduce additional parameters for estimating uncertainty. For example, PCME+ (Chun. 2024) uses one multi-head self-attention block for this. However, this approach will require additional parameters and computational costs, limiting usability (See Table C.6). Instead, we introduce a new uncertainty token [UNC]. along with the class token [CLS] (See Figure 2). Compared to the previous PrVLMs, [UNC] requires almost negligible additional parameters. The visual encoder takes [CLS] ] and [UNC] at the beginning of the input sequences, while the textual encoder takes [UNC] and [CLS] the end of the input. This is because the textual encoder of the original CLIP uses the end-of-sentence token rather than [CLS] at the beginning.
Note that we assume diagonal covariance for simplicity, namely, [UNC] is the same dimension with [CLS]. , We use the L2-normalized [CLS] output as f and [UNC] output as log Similar to [CLS], [UNC] is projected to the final embedding space using a linear layer. We initialize the bias value of this layer to a small value (e.g.. 10) to initialize the initial o² scale small (e.g., exp(-10) x 5 10-5 for each dimension). This simple trick helps stable training.
3.2 PROBABILISTIC PAIRWISE CONTRASTIVE LOSS
In this subsection, we introduce the probabilistic pairwise contrastive loss (PPCL), the main objec- tive function of ProLIP. PPCL is similar to the probabilistic matching loss (PML) of PCME+ but modify PML for stable training based on the log sigmoid loss by SigLIP (Zhai et al., 2023).
Following PCME +. we use the closed-form sampled distance (CSD) as our probabilistic distance: + E2) = || 122 12 + ||o + 02111 (1)
d2 2z) = E2, 2o Z1 2212 = | - 1121- +r where 21 and Zg are Gaussian random variables with diagonal covariances. The probabilistic match- ing loss by +CME+ uses pairwise binary cross entropy (BCE) by taking -a desp (1, E2) + b as logits, where and b are learnable scalars. However, empirically observe that PML fastly converges to a small value, and its gradient is dramatically small, which makes the overall learning procedure slow or unstable (see Appendix A.3 for details). To solve the problem, we employ log sigmoid loss (Zhai et al., 2023). By replacing the squared 1.2 distance ||/|| pall to Equation (1) (details are in Appendix A.4), we have a new probabilistic pairwise contrastive loss (PPCL):
LPpCI (Z.,, Ze) log
1 + exp(Uea (-a(p) pr frr + Ex) b))
where a and b are learnable scalar values and you is I if v and t are matched otherwise -1.
3.3 INCLUSION LOSS
Although PPCL enables to learn probabilistic representations, we empirically observe that the learned uncertainty from data is often counterintuitive to humans. For example, we may expect that text captions with a general meaning (e.g., "photo") has a very large covariance that can cover all the photographic image embeddings, but sometimes it does not. Similarly, we may expect that if a text or an image loses some information (e.g., some tokens are randomly masked), its probability distribution will entail the distribution of the original sample. However, it is not always guaranteed that a model will learn desired uncertainty, especially under noisy image-text correspondences.
To tackle the issue, introduce novel objective function enforcing random variable Z₁ to be included in another random variable Zz. Let P1 and P2 be their corresponding probability density function (pdf). If Z, is included in Zz, then we can presume that the area with high pl will be overlapped to the area with high p2. From this observation, we propose a novel inclusion measure by emphasizing the area with high p1 and compute expectation of the emphasized P1 under the distribution p2- Specifically, we take the square to Pl, and compute J [1 (x))pe())dd This measure is related to Bhattacharyya distance (f /PiPzdæ) or the inner product (ʃ PiP2dz), but our measure designed for measuring "inclusion" (it becomes high if Z1 included in Zz and otherwise low) while the others are designed for measuring "distance" or "dissimilarity" between distributions.
The log inclusion measure (omitting constants) can be derived as follows:
1
B²
inc Zz) = log
(2())2(z))d = -2 of - -log of
log(A) (A)+ +
C,
-00
2
4A
(3)
1
2p1
n22 of
n
where A = of 202
B
C
o
20
4

---

Published as a conference paper at ICLR 2025
pdr - 0.0.
-
2d
squared pdt 2t
squared pdt of Z,
pdt 2, -1 -0.9 0.2)
plz 3,0.
2ffof2,=M11..0
M2, =0000
HIZ, cZ2) 0.917
MZZcZz)= -22200
Inaz,, 24) 25.489
InciZ;.zal 4.773
ind,, 23)= KL) 32.763
KLIZ5. 101,
KLIZ, 2,)= 0..225
20
15
15
15
10
1.0
1.0
65
35
0.5
2-5
15
10
05
00
55
1/
155
3.0
2-2
1.5
10
6.5
0. 05 1.5
2.0
2.0
3.5
16
6.
0.
05
1.0
155
Figure 3: Visual understanding of the proposed inclusion loss. We plot probabilistic density functions (pdfs) of three pairs of Gaussian distributions and their inclusion hypothesis, H(Z, C Z2) (Equation (4)), log inclusion (Equation (3)) and KL divergence. The dashed line denotes the squared pdf, ie., p (r)) H(Z1 C Zz) becomes (a) positive Z1 is included in Z2 and (b) otherwise negative, (c) and are the same level. then H will become zero. While log inclusion represents how Z, is included in Zz, KL measures the "dissimilarity" between distributions (e.g.. e.. (c) has the largest KL, but the smallest "inc"). Figure A.2 shows more examples.
The full derivation is in Appendix A.5.
Now, using Equation (3), introduce hypothesis test whether Z1 is included in Z2 as follows:
H(1 C Z2) log (i(=)p=(z)dz - log [ (1(x))2((x)d
(4)
H. positive if the hypothesis is true and otherwise negative (See Figure 3). It has two distinct properties compared to other probabilistic measures. First, H is asymmetric. The most probabilistic measures aim to measure the "distance", ""overlapping" or "dissimilarity" between two distributions, therefore symmetric (e.g., Wasserstein distance and Bhattacharyya distance). Meanwhile, we mea- sure the level of "inclusion" of two random variables, therefore asymmetric: if Z₁ included in Zg, then Z2 will not be included in Z1. i.e., H(Z1, Z2) * H. (Z2, Z1). Compared to the asymmetric measure, such as KL divergence, we aim to measure how Z1 is "included" in Zg. In contrast, KL measures the dissimilarity between them based on relative entropy. As shown in Figure even if Zı and Zz have the same variance, we have very high KL, while our measure becomes very small.
use the log sigmoid loss for stable convergence. We use log, J (f()) (1))p(())d -ae)dd-llog log l pi (x)) as the logit value, where the logit becomes positive Z₁ is included in Zz. Now, we introduce our novel inclusion loss as follows:
Similarly PPCL,
Linclusion (Z1 C Z2) -log 1+ exp (-CH(ZI C (Z2)''
where is positive scalar. For stability, fix large value, such as 1000. Like KL divergence, inclusion loss can be volatile if variance values become extremely small. To prevent a loss explosion, during training, we multiply a small _/z for computing A, B, C in Equation (3), mimicking each Gaussian has sufficiently large variances multiplied by 1/E. See Table C.5 for more details.
Using the inclusion loss, we enforce two properties to the model. First, we let text distribution include image distribution, i.e., Linelusion Zz C Zz). This intuition is from observations from the previous studies showing that "text entails image" (Chun et al., 2021; Desai al., 2023; Chun, 2024; Kim et al., 2024). Conceptually, when we describe an image, we select the product of the relevant concepts by an arbitrary choice. Therefore, text usually has more general information than images. As another property, we let an embedding of partial information include the embedding of its full information, iee., Linclusion (Z C Z For example, we generate a text containing partial information of the original caption by masking we generate a partial image by masking out the image tokens (He et al., 2022). In practice, we mask out 75% of the input tokens generate partial information. For text, we replace the input tokens with [MASK], and for image, we drop the input patch tokens for efficient computation (Li et al., 2023).
random tokens (Devlin et al., 2018). Similarly,
Finally, we use the VIB loss as a regularization of each Gaussian embedding (i.e., preventing too small Φ following Chun al. (2021) and Chun (2024) See Appendix A.7 for the details. Putting
5

---

Published as a conference paper at ICLR 2025
Equation (2) and Equation (5) all together, we have the following learning objective:
L= Σ
Σ
CPeeL (Zu, Zi) + Σ
Lincluscl (Zu C Zyjmstes) + 3L (Z.)
ZIEV
Z.ET
Z.EV
(6)
+ Σ
[022 7x C (mmrrs) + (2)) +
Σ
2n Linclussion (Zo C Ze);
Z.ET
(v,0))(2,T)
where 01, 012, 3 are control hyperparameters for each loss function. For computational efficiency, we generate masked samples only for 12.5% samples in the mini-batch and compute inclusion loss using them. VIB loss is computed for all samples. We report the loss ablation study in Appendix C.3.
3.4 PROMPT TUNING WITH UNCERTAINTY ESTIMATES
As observed by Chun (2024). the estimated uncertainty is not only beneficial to understanding the in- put data uncertainty but also effective to zero-shot classification (ZSC). In practice, we use multiple templated text prompts to estimate the textual embedding of class names. For example, the original CLIP paper uses 80 prompts, including "a photo of (.)" Although this prompt engineering with mixture of templates significantly improves ZSC performances, it is still unclear which tem- plate benefits ZSC. Furthermore, if we carefully explore images of each class, we can conjecture that each class might need different templates. For example, in ImageNet, "ferret" images often co-occur with the ferret's 's owner. In this case, prompts like "a photo of my ferret" can be helpful to estimate a text feature corresponding to the images, compared to using "a origami ferret".
How can we select the most informative text prompts? One possible solution will be filtering out highly uncertain text prompts. For example, for "black-footed ferret" class, "the embroidered {-}r; "the origami {-}" or "the plastic {-}" have high uncertainty values, while "a low resolution photo of {-)" or "a cropped photo of {.}" have small uncertainty values. Our experiment shows this strategy is moderately effective: it improves ImageNet ZSC accuracy 1p. We presume that it is because the variety of text prompt uncertainties for each class is not significantly large. Also, we presume that the suitability of text prompts is not solely dependent texts describe the corresponding images well. We propose Bayesian Prompt Re- Weighting (BPRW), simple probabilistic approach to find the optimal weight of prompts for each class.
the text itself; may need to consider how
Let Te E RN be the weight of each prompt, where N is the number of prompts (ie., 80 for ImageNet) and e is the class index. Our goal is to find the best n; that the new text embedding 2u E, Zi describe the given M image embeddings Zg. To achieve this goal, we optimize Te to have the best posterior for Z, and Zz. First, we sample K point vectors for each Z and assume they are observations (total K x M point vectors). Next, we optimize simple Expectation- Maximization (EM) algorithm to find the best æ achieving the best log-likelihood. Here, we set a Dirichlet prior for Tee using the uncertainty values of each Z,, i.e., prompt with higher uncertainty has a smaller prior. Due to the page limit, we describe the detailed algorithm in Appendix A.8.
Although our algorithm is theoretically well-founded and flexible by a Bayesian approach (e.g., set- ting prior assumption using of), it needs the image embeddings corresponding to the target class, which violates the ZSC assumption. We tackle this issue by collecting corresponding image embed- dings using KNN for each text class embedding. If we can use some true pairs under few-shot setting (e.g., 5 true images for each class), we observe significant performance improvement.
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS AND EXPERIMENTAL PROTOCOL
Model. We use VIT-B/16 (Dosovitskiy et al., 2021) our image encoder and a 12-layer 768-wide Transformer (Vaswani et al., 2017) as our text encoder. We set the embedding dimension to 768 and the context length 64 tokens, following SigLIP ViT-B/16 (Zhai al., 2023).
Optimization. We implement ProLIP based on openc ip (Ilharco et al., 2021) and the DataComp- 1B dataset (Gadre al., 2024). We list the optimization hyperparameters in Appendix B. We train ProLIP models using 32 NVIDIA H100 GPUs with Bíloat precision, taking about one day to train VIT-B/16 model with 1. 1.28B seen samples. We initialize the bias value of the linear projection top

---

Published as a conference paper at ICLR 2025
on [UNC] to -10 to initialize log o² with small value. We set the initial scale and bias parameters (a and in Equation (2)) to 10 and -10, following Zhai et al. (2023). We randomly select 12.5% image-text pairs from the mini-batch and masking out their 75% information using [MASK] for texts (Devlin et al., 2018) and token drop for images (He et al., 2022). Fine-tuning details are in Appendix B.2.
Evaluation. We evaluate the models on 38 tasks of Datacomp evaluation suite (Gadre et al., 2024) the full evaluation datasets are listed in Appendix B.3. We report five categories: ImageNet, Ima- geNet variants with distribution shifts, 13 VTAB tasks, 3 retrieval tasks, and the average of 38 tasks. In addition, we employ the HierarCaps dataset (Alper & Averbuch- Elor, 2024). which provides cap- tions with four different hierarchies (e.g., "water sports" + "kite surfing" = "kite surfer on top of the board" = "kite surfer in the air on top red board"). Similarly, we construct new Hierarlmgs dataset, which provides images with four different hierarchies (See Figure B.2 for examples). We will describe the details of HierarCaps, Hierarlmgs, and their evaluation in Section 4.4.
4.2
MAIN RESULTS
Table I shows the main result. We use multiple prompts for each task following the DataComp eval- uation suite. Similar to CLIP zero-shot classification (ZSC), ProI.I uses the ensemble multiple prompts by Z!mised ( E, u4. NV E; of). where 14 and of denote the mean and variance of i-th prompt and is the number of prompts (e.g., 80 for ImageNet). Note that if we treat this operation the "average" of N random variables, then the variance should be 1 E, of but empiri- cally observe that the division decreases the final ZSC performance, e.g., 74.51 where our approach shows 74.58 ImageNet. We did not use the uncertainty based ZSC described in Section 3.4 for evaluating 38 tasks. Instead, we use CSD to find the nearest class text embedding.
N2
Table shows that ProLIP outperforms CLIP in all metrics with 1.28B samples. Furthermore, when we train ProLIP with 12.8B seen samples, we achieve a high-performing PrVLM. We show more ablation studies in Appendix C.3, including loss design, hyperparameter, and architecture.
Table 1: Zero-shot classification results. The full results for each task are in Table C.I. VIT-L/16 and S0400M/14 results are the fine-tuned results from the pre-trained SigL.P models. More results in Table C.2.
Samples Soen
ImageNet IN dist. shifts
VTAB
Retrieval
Average
CLIP
1.28B
67.2
56.9
53.4
57.1
SigLIP
1,28B
67.4
55.4
55.7
53.4
56.7
VIT-B/16
ProLIP
1,28B
67.8
55.3
58.5
53.0
57.9
ProLIP
12,8B
74.6
63.0
63.7
59.6
63.3
ViT-L/16
ProLIP
1.28B*
79.4
68.6
64.0
61.3
65.9
VIT-50400M/14
ProLIP
1.28B*
79.3
69.0
65.1
62.5
66.6
4.3 UNDERSTANDING THE LEARNED UNCERTAINTY
Uncertain samples visualization. From Equation (1), can define the uncertainty of the given input by measuring tr(E), namely, Σ; o? (we simply denote of for image uncertainty and of simi- larly). Figure 4 shows the samples with low and high uncertainty values using this value. We extract samples from the 3.5M subset of 12.8B DataComp CommonCrawl small (Gadre et al., 2024) filtered by CLIP similarity and English filtering! !. We use ProLIP with 12.8B seen samples for the analyses.
Figure 4 shows that the texts with more general meanings have high uncertainty, e.g. "Screenshot" or "graphic". This is because a shorter text with more general meaning has more opportunity to be matched to various images. In contrast, certain captions describe a longer and distinct context, such as the exact address or proper nouns, which unlikely matched to multiple images. We will show that the context length of the text is highly correlated to the uncertainty value (Figure 6). Interestingly, there are captions with high uncertainty despite long context lengths (e.g., more than the specified context length). We empirically observe that such captions have almost no information, showing that ProLIP captures the text uncertainty well. The examples are shown in Appendix C.I.
hhttps://hugiing
ats.r/ddddaaooooo small 11 fi 1tered
7

---

Published as a conference paper at ICLR 2025
-0.00554 - 0,0072 P(a.) < 0.1%
0.0123 - 0.0126
9. 0.0154 0.0158
-..1144- 0.0199
0.0253 - 0.0200
0.0464 - 0.0661
P(a.) 21%
39% P
59% P(
P(a.)
P(aa)
More cersain samples
More uncettain sumples-
de peppre
handlry ii
Chocs chough Tive cnpon
Cipcitos cn arreort lecur
Page
Peloorn Malo 11, CIN Scom
Deinng Laca Cousi Iope of
Sal lice biue
Naus Stals
(411)) duly
was Pillk
Loaa puot Hocral Wriingg -CTicaal
baalch 20 head Cos Nagstable 200
but Ail Colagen
-Callutial Tary
-Koing Torters ledurtio
Soioo
a-- 0.0264 0,0303
17- 0.0369 - 0.0371
C -0.0391 0.0393 39% <. P(o) 4.1%%
a.- 0.0415 0.0418 59% < Pia.) x 61%
a. 0.0-60 - 0.0468
0. 0.0625 0,0673
0.11%
19% < Ps
79% < P(o.) 81%
99.9%% < P(o.)
Figure 4: Uncertain & certain samples. Visualization from the 3.5M filtered DataComp Small pool.
Imoge and Text uncortaities (BacsComp smolt)
Tec: uncersaintt va. test token langth (CCIM))
Text uncertainty vs. Test he hraranchy d'terarCassl
Locel
encertanry
Lavel
150005
Lewel
125000
Ruquno 40
Leqpl
106600
50160
25060
8.00
6045 no5e "est uncersiitt
Uncetiirty
11pert contect length
Figure 5: of vs. of. Generally, Figure 6: of context length. Figure 7: of by Text hierarchy. More general captions (i.e., lower level) are more uncertain.
texts are more uncertain than im- Shorter texts are generally more un- ages, shown in Figure 4.
certain than complex ones.
We can see certain and uncertain images in the upper row of Figure On the uncertain image side, we can find images solely with an object (e.g., a clock, book, or clipart) on a white background. Generally, such images can be matched to multiple possible descriptions, e-g., the name of the prod- uct, the detailed explanation of the product, or the written text in the image (e.g., the book title). On the other hand, certain images have more complex visual cues that can be described with more and specific captions. More samples with high and low uncertainty can be found in Figure C.I.
Statisties of of and o7. In Figure 4, we also observe that of is generally smaller than of, would be originated from the inclusion loss Linclusion (Ze, Z,) in Equation (6). Figure 5 shows that the image embeddings and text embeddings have distinct uncertainty values. In Appendix C.6, we provide a detailed discussion of the relationship between learned uncertainty and human preference.
What is the source of the uncertainty? We answer this question by analyzing text context length and data hierarchy. First, we plot the text uncertainty by the context length on the ConceptualCaption 3M (CC3M) captions (Sharma et al., 2018). As shown in Figure 6, a short caption tends to have a large uncertainty value. For example, we observe that the caption "film series" " has the largest uncertainty value in CC3M, while the caption "gangsta rap artist told by person @ person almost died you have to see this!" is the most certain caption. Namely, more uncertain captions tend to be more logically "general" captions, such as "dress sewing pattern" or "person before & after", while more certain captions specify particular situation. From this observation, we explore the relationship between the uncertainty and varying levels of description. We employ the HierarCaps dataset, whose images have four levels of descriptions from the full caption of COCO Caption and its logical entailment hierarchy with three different levels (e.g., "bird" + "blue bird" ...). Examples are shown in Figure 9. Figure 7 shows the relationship between the text uncertainty and text hierarchy levels. Here, Level 0 denotes the most general captions, e.g., "chair" or "bird". and Level 4 represents the original COCO Caption. As shown in the figure, more general captions (lower levels) tend to be more uncertain, while more specific captions (higher levels) tend to be less uncertain.
8

---

Published as a conference paper at ICLR 2025
Segmentation misk
L
Level
Level
Orignal image
70.5% camples soiidf indusiion hypornesi for evel
75.2% samples codsty nclusion hypotres for lovel
76.7% camp is satisty Inclusion hopothoes for lovel
Hlorig O1
Hoong 11
160
Hlong e H22
orgi
400
Ail
300
Fraquancy
400
100
200
100
200
50
100
50
Hypothess valU
Hypothess valu
Hypothols valus
Figure 8: of by image hierarchy using HierarImgs. We tested the inclusion hypothesis the original image and masked images from level i, H(orig C 6), and its inverse hypothesis, H(1 C. orig). In all tests, more than 70% images are included in their lower-level images (purple histogram bars with positive hypothesis values). The dataset construction of Hierarlmgs and related discussions can be found in Appendix B.S and C.S.
We further investigate whether similar phenomenon happens for the visual modality by construct- ing new Hierarlmgs dataset to represent the logical visual hierarchy. As shown in the upper of Figure 8, our Hierarlmgs dataset consists of four levels: Level 4 is the original image and Level 0 is the largest visual segment. The details of the dataset construction can be found in Appendix B.5.
Using the images, we analyze the relationship between the visual uncertainty and varying levels of visual information. We test whether each image becomes more uncertain than the original image by applying the inclusion hypothesis. Namely, we test if lower-level image includes its original image by Equation (4). As shown in Figure 8, most of the images satisfy the inclusion hypothesis (e.g., more than 70%), implying that ProLIP also captures image hierarchy. In Appendix C.5, we explain more details the image hierarchy experiments, including the absolute of value by different levels and possible pitfalls of Hierarlmgs (e.g., we need more careful filtering for a reliable evaluation).
4.4 APPLICATION USING UNCERTAINTY
Image traversals. Following Alper & Averbuch-Elor (2024), we first set the [ROOT] embedding. Then, we retrieve the nearest caption of the given image, and interpolate [ROOT] and the text embedding with 50 equally spaced steps. The null text embedding CLIP embedding space. In ProLIP case, we can utilize the additional uncertainty information and the inclusion hypothesis. Hence, we search the root embedding of the given embedding by searching the text embedding that includes the given image embedding most. The other procedure equals to Alper & Averbuch- (2024). We perform traversals on HierarCaps. Details are in Appendix B.4.
is used as the [ROOT] ] of the
We show image traversal results in Figure 9. Interestingly, the most inclusive caption (ie., [RCOT]) for each image is not always same to the ground truth level 0 caption of HierarCaps. For example, our estimated [ROOT] for the vase picture is "vase". while the GT level caption is "object". From the observation, we can presume that although our retrieval results are plausible, it could lead to inferior HierarCaps retrieval results. To ensure more diversity, we take an average of " and the most inclusive text embedding and use it as the root embedding. Using the newly proposed root embedding, we quantitatively measure the performance of our traversal in Table 2. First, our new [RCOT] embedding is more specialized to the inputs, rather than only using ""; we can achieve higher @@ [30OT] using our approach. The table shows that the proposed probabilistic image traver- sal achieves higher recall and precision than deterministic traversal using as [ROOT]. Namely, the probabilistic approach gives more opportunities to get more precise captions during the traversal.
Uncertainty-basee ImageNet prompt enhancement. As described in Section 3.4, we propose BPRW, a new prompt re- weighting method to find a weight मं for each class. A text embedding weighted by Te, ie., Z.ew Σ, Z will be used as class embedding for ZSC. Table 3 shows the ImageNet classification results with different strategies. First, solely using text uncertainty and filtering uncertain texts not sufficiently effective. They only improve about +0.05pp top-1

---

Published as a conference paper at ICLR 2025
Query Imagés
Retreved captions
Kite surfer in the
blue bird sitting on the top of branch with autumn leaves
outdoor table containing assorted bowls of food and beer
lange red chair has horse statue on it
Leaves and purple flowers come out brown vase on desk
on top of a red board
surfer the
blue bird sitting on the top of branch with autumn leaves
outdoor table containing assorted bowls of food and beer
large red chair horse statue on it
Leaves and purple flowers come out of desk
on top of a red board
brown
Kite surfer in the air
Blue bird sitting top of branch
Outdoor picnic
Red chair has a horse statue
Vase with flowers in it
Most inclusive
captions
kite surfing
bird
plenie
red chair
vase
Level's
Kite surfer in the aiz lop of rod beard
the top of branch with
A blue bird sitting
containing assorted bowls of food and heer
An outdoor table
A large red chair has home statue on it
There is small glass that has purple flowers
leaves
HierarCape Caps
Level
surfer top of the board
Blue sitting on top of branch
Outdoar
food and beer
Red borse statue
in violet
Level
kite surfing
blue bird bird
outdoor table outdoor
red chair chair
sports
object
Figure 9: Image traversals with ProLIP. For each image, we estimate (ROOT) caption which include the image most using Equation (4). Then, we interpolate the [ROOT] caption and the retrieved caption. We compare our interpolation and HicrarCaps GTs. Red denotes when the estimated and GT roots are different.
Table HierarCaps retrieval. measure the pre- cision and recall of 50 traversal results. "Prob?" de- notes inclusion-hased traversal, using the average of "" and "I "ROOOT]" 1 in Figure 9 as new [ROOT]. @@ [ROOT] is recall [ROCT) embeddings.
Table ImageNet prompt tuning by ProLIP. K denotes the number of few-shot samples for cach class (if required). "OpenAl 80 prompts" is the same as "ImageNet" of ProLIP 12.8B in Table 1.
Prompt strategy
Accuracy
Prob?
Prec
R@1
R@ [RO00]
"a photo OpenAl 80 prompts
{-}"
73.7
74.6
CLIP (1.28B)
x
25.0
63.0
0.1
Filtering by stats
74.6 (+0.03)
ProLIP (1.28B)
x
28.4
62.6
0.1
Top-K K prompts
74.7 (+0.07)
ProLIP (1.28B)
35.9
62.9
15.3
BPRW (proposed)
74.7 (+0.12)
ProLIP (12.8B)
x
31.7
67.8
0.1
BPRW (proposed)
75.6 (+0.99)
ProLIP (12.8B)
41.1
68.0
23.3
BPRW (proposed)
75.8 (+1.21)
curacy. On the other hand. BPRW achieves better accuracy by using additional visual information; we have +0. 12pp for ImageNet ZSC. we can use a few labeled images per class (e.g., e., five for each class), can get over 1.2% accuracy improvement by adjusting their weight. In Appendix C.7, we describe more details of BPRW, including hyperparameters and more visualization results of the learned weights by BPRW. Interestingly, the learned weights follow the actual image distributions; if the images are mostly low resolution, "a low resolution" prompt becomes the most significant.
More examples and discussion. We include more applications in Appendix C.8, including filtering. dataset understanding, and LongProLIP. Appendix D discusses the diagonal covariance assumption.
5 CONCLUSION
In this work, we introduced Probabilistic Language- Image Pre-training (ProLIP), a fully probabilis- tic vision- -siiun-aanguage model that addresses the limitations of deterministic embeddings by capturing the inherent multiplicity in image-text relationships. By mapping inputs to random variables and efficiently estimating uncertainty through an "uncertainty token" ([UNC)), ProLIP models distribu- tional relationships without additional parameters. The inclusion loss further enhances interpretabil- ity by enforcing distributional inclusion between image-text pairs and between original and masked inputs. Our experiments demonstrate that ProLIP is not only beneficial in zero-shot classification tasks but also provides an additional axis of understanding input data by capturing their uncertainty. Our approach highlights the potential of uncertainty modeling in vision-language applications.
10

---

Published as a conference paper at ICLR 2025
AUTHOR CONTRIBUTIONS
This project is the extension of Chun et al. (2021; 2022); Chun (2024). S Chun led the project; the other authors actively and significantly contributed the project with advice and feedback. W Kim and S Yun contributed to the baseline openclip implementation and evaluation toolkits from Kim et al. (2024); S Chun developed the main module upon the baseline implementation. The main ideas, such as [UNC], probabilistic pairwise contrastive loss, and inclusion loss, are designed and implemented by Chun. S Park contributed to constructing the Hierarlmgs dataset. S Chun wrote the initial version of the manuscript. S Park contributed to a better visualization. All authors contributed to the final manuscript.
REPRODUCIBILITY STATEMENT
For reproducible research, we the open-source training dataset (DataComp 1B (Gadre et al., 2024)) and codebase (openclip (Ilharco et al., 2021)). As we clarified in Appendix B.3, we had 1,121,356,,67 number of valid URLs among 1.39 billion URLs and 1,118,443,,92 number of unique images after de-duplicating URLs (there were 147,676,246 number of duplicated URLs in the DataComp 1B URLs). All the detailed hyperparameters are elarified in Appendix B. Our results can be reproducible by our open-source implementation httptt::://giittb..ooo/oo prolip) and released pre-trained weights in HuggingFace (ittps://huggiiggfeee co/ collections/ wwweeeeiionnnngggunneeeeeeeeeeee
the released weights include ProLIP ViT-B/16 (from-scratch), ViT-L/16, ViT-S0400M/14, ViT-H/14 (fine- tuned), and LongProLIP ViT-B/16 fine-tuned with different datasets.
ACKNOWLEDGEMENT
We thank the internal infrastructure team who helped to download the DataComp 1B dataset.
REFERENCES
Morris Alper and Hadar Averbuch-Elor Emergent visual-semantic hicrarchies in image-text representations. In European Conference on Computer Vision (ECCV), 2024. 7, 9, 22, 23
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias controondd dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems (NeurlPS), volume 32, 2019. 21
Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar. The iwildcam 2021 competition dataset.
arXiv preprint 105.:v221..000 033494 2021.
Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with ImageNet? arXiv preprint 200.022000607199 2020. 30
Yonatan Bitton, Nitzan Bitton Guetta, Yosef, Yuval Elovici, Mohit Bansal, Gabriel Stanovsky, Roy
Schwartz. Winogavil: Gamified association benchmark to challenge vision-and- lnd-lagu models. Advances in Neural Information Processing Systems (NewrIPS), 35:26549-26564, 2022. 21
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food- 100-1001-minn mminig discriminative components with random forests. In European Conference on Computer Vision (ECCV), pp. 446-461. Springer, 2014. 22
Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5710-5719, 2020. 3
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image- text ne-training to recognize long-tail tail visual concepts. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3558-3568, 2021. 21, 23, 26
Xinlei Chen, Hao Fang, Tsung Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint 154.0000 2015. 2, 21
11

---

Published as a conference paper at ICLR 2025
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):186511888 1883, 2017. 21
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco. Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive sunguage-image learning. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 2818-2829, 2023. 21
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6172-6180, 2018. 22
Sanghyuk Chun. Improved probabilisticc image-text representations. In International Conference on Learning Representations (ICLR), 2024. 1, 2, 3, 4. 5, 6, 11, 16, 17, 19, 21,
Sanghyuk Chun and Sangdoo Yun. LongProLIP: A probabilistic Haionnlanguage model with long context text. arXiv preprint arXiv:2503.0m 08048, 2025. 32
Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Proba- bilistic embeddings for cross-modal retrieval. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). 2021. 1, 3, 5, 11, 19.32
Sanghyuk Chun, Wonjae Kim, Song Park, Minsuk Chang Chang, and Scong Joon Oh. ECCV Caption: Correct- ann-ime-nndd-umaa-----
ing false negatives by collecting European Conference on Computer Vision (ECCV), 2022. 2, 11
image-caption associations for MS-COCO. In
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2014. 21
Adam Coates, Andrew Ng, and Honglak Lec. An analysis of single-layer networks in unsupervised feature learning. Proceedings of the fourteenth international conference on artificial intelligence and statistics,
pp. 215-223. JMLR Workshop and Conference Proceedings, 2011. 22
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. RedCaps: Web-curated image-text data created by the people, for the people. In NeurlPS Dataset and Benchmark (NeurlPS D&B), 2021. 21, 23, 26
Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakrishna Vedan- tam. Hyperbolic image-text representations. In International Conference on Machine Learning (ICML), pp. 7694-7731. PMLR, 2023. 5
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert Pre- Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/8/0.04800, 2018. 5,
Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations (ICLR), 2021. URL https: /wtp///eeeeee/eeeee/eeeeeeeeeeee net/ forum? id=YicbF CNTTy. 3,6
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. International Journal of Computer Vision (IJCV), 88:303-338. 2010. 22
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. In International Conference on Learning Representations (ICLR), 2024, 21, 25
Li Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178.178. IEEF, 2004. 21
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smymis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, al. Datacomp: In search of the next generation of multimodal datascts. Advances Neural Information Processing Systems (NeurIPS), 36, 2024. 2, 6, 7, 11, 21, 31
Andreas Geiger, Philip Lenz, and Raguel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In /EEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3354- 3361. IEFF, 2012. 21
12

---

Published as a conference paper at ICLR 2025
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16000-16009, 2022. 7,7
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):211722226 2019. 21
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, The many faces of robustness: A critical analysis of out-of- distribution generalization. In International Conference on Computer Vision (ICCV), pp. 8340-88-00 8049,,33,,0000 2021a. 21
Dan Hendrycks. Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In JEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15262-15271, 2021b. 21
Byeongho Hco, Sanghyuk Chun, Scong Joon Oh, Dongyoon Han, Sangdoo Yun, Gyuwan Kim, Youngjung Uh, and Jung- Woo Ha. AdamP: Slowing down the slowdown for momentum optimizers on scale-invariant weights, In International Conference on Learning Representations (ICLR), 2021. 21
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongscok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip, July 2021. URL https: //tps::////.io/ org/ 10. 5281/ /wwodzzeooo..ooooooooo 5143773.
11, 21
Yatai Ji, Junjie Wang. Yuan Gong, Lin Zhang, Yanru Zhu, Hongfa Wang, Jiaxing Zhang, Tetsuya Sakai, and usion-language pre-training model. In IEEE/CVF Con-
Yujiu Yang. MAP: Multimodal uncertainty ference on Computer Vision and Pattern Recognition (CVPR), pp. 23262 23271, 2023. 1,3
Chao Jia, Yinfei Yang, Ye Xia. Yi- Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le. Yun- Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision- vision-language representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), pp. 4904-4916, PMLR, 2021.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Gir- shick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In /EEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp.
2017. 21
Wonjae Kim, Sanghyuk Chun, Taekyung Kim, Dongyoon Han, and Sangdoo Yun. HYPE: Hyperbolic entail- ment filtering for underspecified images and texts. In European Conference Computer Vision (ECCV), 2024. 5, 11
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. 21
Michael Kirchhof, Enkelejda Kasneci, and Seong Joon Oh. Probabilistic contrastive learning recovers the cor- rect aleatoric uncertainty of ambiguous inputs. In International Conference on Machine Learning (ICML), 2023. 3
Alex Krizhevsky, Learning multiple layers of features from tiny images. 2009. 21
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied document recognition. Proceedings of the IEFE, 86(11):227--2324 1998. 22
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language- image pre-training for
unified vision-language understanding and generation, 2022.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In /EEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 23390-23400, 2023.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zimiick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV), 2014. 22
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018. URL ht httptpps///peeneviiee...o n forum? dww.z
13

---

Published as a conference paper at ICLR 2025
S. Maji, J. Kamnala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013. 22
Andrei Neculai, Yanbei Chen, and Zeynep Akata. Probabilistic compositional embeddings for multimodal image retrieval. In IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4547- 4557, 2022. 3
Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. and Alessandro Bissacco. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011. pp. 4. Granada, 2011. 21
Dat Quoe Nguyen, Ashutosh Modi, Stefan Thatee, and Manfred Pinkal. A mixture model for learning multi- sense word embedkings. In Proc. of the 6th Joint Conference on Lexical and Computational Semantics (* SEM 2017). pp. 121-127, 2017.
Maria-Elese Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEEE,
2008. 21
Scong Joon Oh, Kevin Murphy, Jiyan Pan, Joseph Roth, Florian Schroff, and Andrew Gallagher. Modeling uncertainty with hedged instance embedding. In International Conference on Learning Representations (ICLR), 2019. 3, 19
Jungin Park, Jiyoung Lee, Ig- Jae Kim, and Kwanghoon Sohn. Probabilistic representations for video contrastive learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14711-14721, 2022. 3
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In /EEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 3498-3505. IEEE, 2012. 21
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin. Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pp. 8748 8763. PMLR, 2021. 3, 16, 22
Vikram V Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron Adeock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. Geode: a geographically diverse evaluation dataset for object recognition. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024. 22
Benjamin Recht, Rebecca Roelofs. Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? International Conference on Machine Learning (ICML). 2019. 21
William A Gaviria Rojas, Sudnya Diamos, Keertan Ranjan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. The dollar street dataset: Images representing the geographic and sociocconomic diversity of the world. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 22
Olga Russakovsky, Jia Deng, Hao Su. Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision (ICCV). 115(3):111-22, 2015. 21
Christoph Schuhmann, Romain Beaumont, Richard Veneu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5h: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems (NeurIPS), 35:25278-2522222 255244, 2022. 21
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hyper- nymed, image alt-text dataset for automatic image captioning. In Association for Computational Linguistics (ACT.), pp. 2556-2565, 2018. 8, 21, 23, 26
Yichun Shi and Anil K Jain. Probabilistic face embeddings. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6902-6911, 2019. 3
Anna Silnova, Niko Brummer. Johan Rohdin, Themos Stafylakis, and Lukas Burget. Probabilistic embeddings for speaker diarization. In Proc. Odyssey 2020 The Speaker and Language Recognition Workshop, pp. 24-31, 2020. 3
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. Man computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks, 32:323-332, 2012. 22
14

---

Published as a conference paper at ICLR 2025
Jennifer J Sun, Jiaping Zhao, Liang-Chiee Chen, Florian Schroff, Hartwig Adam, and Ting Liu. View-invariant probabilistic embedding for human pose. European Conference on Computer Vision (ECCV), 2020.
Uddeshya Upadhyay. Shyamgopal Karthik, Massimiliano Mancini, and Zeynep Akata. Probvlm: Probabilistic adapter for frozen vison-languee models. In International Conference on Computer Vision (ICCV), pp. 1899-1910, 2023. 1,3
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Information Processing Systems (NewrIPS), pp. 5998 6008, 2017. 3,
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In Medical Image Computing and Computer Assisted Ciarreettoo-MMICCA 2018: s/ International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, pp. 210-218. Springer, 2018. 21
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The Caltech-UCSI Birds- 200-2011 Dataset. Technical Report CNS-TR-201 1-001, California Institute of Technology, 2.
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances Neural Information Processing Systems (NeurIPS), pp. 10506-10518, 2019. 21
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale seene recognition from abbey to zoo, In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3485-3492. IEEE, 2010. 21
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denota- tions: New similarity metries for semantic inference over event descriptions. Association for Computational Linguistics (ACL), 2:67-78, 2014. 21
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features, In International Conference on Computer Vision (ICCV), 2019. 17
Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. Re-labeling imagenet: from single multi-labels,, from global localized labels. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021. 30
Xiaohua Zhai, Joan Puigeerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. large-scale study rep- resentation learning with the visual task adaptation benchmark. arXiv preprint 1wXiv:1910.04867, 2019. 21
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In International Conference on Computer Vision (ICCV), 11975-11988 2023. 1, 4, 6, 7, 16, 25
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In European Conference Computer Vision (ECCV), pp. 310-323 Springer, 2024. 32
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations (ICLR), 2018. 17
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A MORE DETAILS OF PROLIP
A.1 WHY DO WE NEED PROBABILISTIC REPRESENTATIONS:
Deterministic embeddings (e.g.. CLIP (Radford et al., 2021), SigLIP (Zhai et al., 2023)) suffer from the difficulty of representing input data uncertainty (i.e., aleatoric uncertainty). For example, "A per- son is walking" can be matched to either "A person is walking in the rain" or "A person is walking on sunshine", but in deterministic embedding space (Figure A. it is challenging represent the ambiguity of "A person is walking". . On the other hand, a probabilistic embedding space (Fig- ure Ab) can represent this ambiguity by expanding the "area" of the ambiguous embedding. If we assume more ambiguous input, such as "person", probabilistic embedding space can represent the ambiguous input by assigning a larger uncertainty value to "person". However, a deterministic embedding space will map an imput to specific vector coordinate; one possible choice is map the uncertain input into the "average" of the possible matched inputs, e.g., let the "person" em- bedding located to the midpoint of all the person-related text embeddings. However, this approach more complex than the
a),
still cannot capture the complex actual semantic meaning of "person"; average embedding. This argument is empirically supported by the image traversal experiment in Section 4.4. Using a more proper text embedding as the root embedding [ROOT] performs better than using native null text embedding. Furthermore, this paper targets the vision-language repre- sentation learning scenario, where the ambiguity of inputs is multi-modal; each modality (text and image) can have inherent ambiguity as shown in Figure A. and the correspondence between image and text can have ambiguity due to the inherent many-to vo-many correspondence and abundant false negatives as shown by Chun (2024). Overall, we need to use probabilistic representations rather than deterministic representations to represent the inherent ambiguity of vision-language datasets.
"Person"
"A person is walking in the rain"
person walking on sunshine"
"A person is walking"
person bugging a cat"
Deterministic embedding
(b) Probubilistic embedding
Conceptual comparison between deterministic and probabilistic embedding spaces. While probahilistic representation space naturally represent the inherent ambiguity of input data (i.e,, aleatoric uncertainty) by estimating the uncertainty of each input, deterministic emhedding space suffer from mapping complex semantics ambiguous inputs.
Figure A.I:
What property should be learned by an ideal PrVLM? An ideal PrVLM should capture three po- tential input uncertainties: (1) uncertainty from the text modality, (2) uncertainty from the image modality, and (3) uncertainty from the text-image cross- mros-mtodality.. Uni-modal uncertainty is straight- forward: if an input has more detailed information (e.g., describing more detailed information in text, capturing very detailed and complex scene by photography), then it will have smaller un- certainty, otherwise, will have larger uncertainty (e.g., providing very high-level caption, such as "person", only part object with white background is taken by picture).
The cross-modal uncertainty should capture "how many possible instances can be matched this input?". We can think this in two different viewpoints: text-to-image and image-to-text. The text- to-image relationship is straightforward. all photographs. and if have caption with very detailed description (e.g., the full description of the hotel room), then it will be only matched to specific images. Image-to-tex relationships often determined by the dataset. For example, if have caption dataset exactly describing
we have a caption "photo", then it will be matched to
16

---

Published as a conference paper at ICLR 2025
which objects are in the image, then can think that an image with less objects will have larger uncertainty. However, if we consider a caption dataset where an image has multiple captions, each caption focuses on completely different objects. then an image with more objects will have larger uncertainty. In other words, unlike text uncertainty originating from a text-to-image relationship. image uncertainty originating from image-to-text relationship is highly affected by the property of the training dataset.
In practice, because our datasets have a mixed property and their captions are somewhat noisy, our image uncertainty will have a mixed property, namely, unlike text uncertainty, there could be no strong relationship between the absolute uncertainty value and the number of objects (or complexity of images) Empirically, we have more captions that exactly describing the scene, therefore, a more simple image tends to have larger uncertainty. Also, unlike texts, images always have the same number of pixels; the definition of "information" in images is not as straightforward as that of texts. Since there is no "uni-modal" uncertainty supervision, PrVLM trained only with image-text relationships will have no guarantee to represent proper image uni-modal uncertainty. To enforce a proper uni-modal image uncertainty, we first propose the uni-modal uncertainty supervision by proposing the inclusion loss, namely, the original image embedding should be included by an image embedding from the masked image.
A.2 2D VISUALIZATION FOR FIGURE
We use linear projection, such Principal Component Analysis (PCA), for the visualization. If we use a non-linear projection from a high-dimensional space to a 2-dimensional space, there is no guarantee that the projected Gaussian distribution still follows a Gaussian distribution. Therefore, we project the high-dimensional embeddings to 2-dimensional space using PCA. All models are trained on the DataComp 1B dataset with 1.28B seen samples (i.e., the 1,28B models in Table 1).
For both models, we select ten similar images and their corresponding captions from the Hierar- Caps dataset. Then, for ProLIP, we randomly sample ten embeddings for each embedding using the learned Gaussian distributions. We then apply PCA to the sampled embeddings for visualization.
A.3 PROBABILISTIC MATCHING LOSS VS. PROBABILISTIC PAIRWISE CONTRASTIVE LOSS
Probabilistic matching loss (PML) by PCME++ (Chun, 2024) and the proposed probabilistic pair- wise contrastive loss (PPCL) use almost the same logit based on CSD (Equation (1)). However, they have differences in (I) PML is based on the original CSD, which should compute L2-distance between jss and (2) PML uses binary cross entropy loss (BCE) while PPCL uses log sigmoid loss.
The first di diffrnence can make numerical errors when the difference between jas is extremely small. because L,2 distance should compute a "square" operation, i.e., Σ (a? b?)², where a; and b; scalar value of i-th dimension of vector a and b. On the other hand, we use logit based on matrix multiplication |4] μ22 (derived in Appendix A.4), showing a more stable and accurate computation in terms of float precision.
The second difference can cause fast gradient vanishing as already discussed by Chun (2024). Chun (2024) thus employed additional techniques to mitigate the issue, e.g., pseudo positives and mixed data sample augmentation method, such as Mixup (Zhang et al., 2018) and CutMix (Yun al., 2019). However, in this paper, we omit the techniques because simplicity is important when we train with large-scale training samples. To tackle the issue, we employ log sigmoid loss and a multiplication-be based logit, resulting a fast and stable convergence.
In our ablation study (Appendix C.2), we empirically show that the PML loss used by PCME++ fails to converged when it is used by a stand-alone way without any deterministic loss (e.g., CLIP loss). On the other hand, PPCL loss converges well even without any additional loss function.
Note that +CME+ and ProLIP use normalized mean for both training and inference to compute cosine similarity as same as CLIP. Also, +CME+ and ProLIP can estimate Gaussian random variable by the parameterization trick, i.e., = (μ, Σ) = μ + Σ O (0, 1) (where, O denotes the element-wise multiplication). In practice, because
the closed-form solution for calculating distance (CSD- Equation (1)) and loss functions (inclusion loss, PPCL), no sampling based on a ee-parameterization trick is required.
17

---

Published as a conference paper at ICLR 2025
A.4 DERIVATION OF EQUATION (2)
We first start from the fact that d_2 = || M2 2 2n1 #2 = - 2 d22c0s when Yr and f2 are L2- u2-normalized (i.e., | = 1). Namely, a cosine similarity dL2cos
]d By replacing
d12 to desD (Fquation (1)), we can conclude the derivation.
1
1
1 1r + Ee).
dCSDens = 1 desn - 1 d1
3 tr(o + +o²) - d1.200s
(A.1)
2
2
2
2
A.5 DERIVATION OF EQUATION (3)
Let us assume two Gaussian distributions:
p(z) = 1
[xxx - 121)
(a 122
exp
(()) =
exp
2
v 2
20
First, we take the square of p(z):
(x - H1)2 (-1=-N1)²)
))
(- (x (x- 11)
--(--11²²)
p(x)² = 2mog exp
exp
20²
2Tо7
of
Now, we compute the integral:
/ p(a))
1o o exp
(- (x (x-m1)²
1
(x u2)²
exp
dx
of
/2T03
20
1
exp (- (z (x-p1) 21)
(x
(22)
dr
2T / 2To²
o9
20
We expand the terms in the exponent as follows:
(x
m1)²
(x u2)²
1
2
112
x²
of
20h
o
20
0f
02
201
Let:
2p1
12
A
0
20
B =
0
2
2
C =
0
2 +
20
The term in the exponent simplifies
B
2
B 4A
A
+
C
2A
Now, the integral becomes:
B²
/
C
B 2A
exp
exp
-A
a
o
2
4A
Using the Gaussian integral formula, we have:
B 2
π A
exp
-A
r
2A
da
=
V
Thus, the integral becomes:
pla))(()) dx
B2
c)
2mo
22m0
exp
4A
V
A
By taking logarithm and omitting constant terms, have Equation (3).
18

---

Published as a conference paper at ICLR 2025
pdr 1,0.05)
pdf otZ 1,005)
of 0.33
pdt 2t (00,,N
inciZ, 10.530
2,1= 33.974
KLIZ,.Z,) - 40.889
KL 21) 1639.584
KLIZ..2))
KL KLG2, 2:) 1598 1198.695 695
20
14
1.5
10
1.-
05
2.0
1.5
1.0
3.5
0.0
0.5
2.0
200
1.5
055
0.3
0.5
3.0
3.5
Figure A.2: More visual examples of the inclusion loss. The difference of KL and reverse KL (DKL) cannot correctly capture the inclusion relationship.
A.6 MORE DISCUSSIONS RELATED TO INCLUSION LOSS
comparisons with the proposed inclusion loss and KL. divergence. Specif- ically, we compare inclusion loss with the difference of KI. and reverse KI. (DKL.), namely, DKL(Z,, Z2) K (Z Z2) (L2, Z1). As shown in the figure, the most significant problem with DKL is that it cannot correctly represent the "inclusion" relationship. For example, consider Iwo random variables Z1 and Ze where they do not include each other as shown in Figure A.2. In this case, although Z1 and Ze do not include each other, because DK PKL(ZZ Z2) one of D Zz) or DKL(Zz) (Z,, Z1) will become positive, while the other will become negative. Namely, DKL cannot correctly capture the inclusion relationship. On the other hand, in the same scenario, our inclusion test always returns negative values as shown the figure, which correctly represents the inclusion relationship.
Figure A.2 shows
DKL((Z1 Zz),
A.7
VIB LOSS
VIB loss is simple regularization term to prevent the collapse of the estimated variance and is widely used by previous probabilistic representation learning methods Oh et (2019); Chun al. (2021); Chun (2024). The VIB loss formulation is as follows:
(viB(Z) KL(ZINN 1)) - log 0² -2 -
(A.2)
Please refer Oh al. (2019) for the full derivation.
A.8 ALGORITHM OF UNCERTAINTY-AWARR
PERO-SHOT PROMPT SELECTION
Remark: Let E RV be the weight of each prompt, where N is the number of prompts (i.e., 80 for ImageNet) and e is the class index. Our goal is to find the best Tü that the new text embedding Znew Σ; ZiZi describe the given M image embeddings Z To achieve this goal, optimize Te to have the best posterior for Z, and Ze For each class c, sample K points from each Z Namely, we have total M' = M x point embeddings as observations for class c.
We assume Dirichlet prior for n. with a, where controls the "uniformity" of the posterior. If choose large a, such as 10, then π will be more uniform and if we choose small a, such as 0.1, then almost becomes Direc-delta distribution. We also set the initial π with the reversed uncertainty score, i.e., the normalized 1 1/trE,,) for each prompt. We also employ small trick for stability. After we got the prior value, we add small E, to make the overall operation stable. Now, we explain the Axpectation-Maxiiizaa mixture of N Gaussian components, incorporating Dirichlet prior over the mixing proportions. For simplicity, we omit the class index for the remaining section.
(EM) algorithm for estimating the mixing proportions of
We place a Dirichlet prior on the mixing proportions for the given a::
p(T) Dirichlet( ()),
(A.3)
We have a dataset of M' observations {x, } and assume that the data is generated from a mixture of N Gaussian distributions (i.e., the number of prompts).
19

---

Published as a conference paper at ICLR 2025
E-Step: Compute the responsibilities Yin, which represent the probability that observation Iy was generated by component n:
Tn (n(2))
7jn
Σ Te fx(z))
where Ju(2z) is the Gaussian probability density function of component evaluated at zyj
1
(-2 mn)) En (2) Hmn)
fa(z))
(2 (22) D/2 E. 1/2
exp
Since En is diagonal, the computation simplifies
D (xz) = II
(- (1jd und))
exp
2TTO
20 nd
d=1
M-Step: Compute the effective number of observations assigned to each component:
M
N₂
Yjn-
Then, update the mixing proportions for each prompt n.
N
+ On -1
No + - M' N(a-)
Tn
M (a 1)
Ensure that 2 0 and En Tn 1. We can simplify the algorithm as follows:
Algorithm Bayesian Prompt Re- Re-Weighting (BPRW)
1: Initialization (o: the Dirichlet prior hyperparameter, e: the stability hyperparameter)
2: for each class c = 1 to C' do
3: Collect M' observations by choosing M number of Zu and sample K points from each Z..
We can choose M "ground truth" observations where Z, is the embedding, including the c class. Otherwise,
select M nearest samples from the base text prompt embedding of 4: for each prompt index = to K do
Tr(En)) ) (EN Tm(2a) Modify covariance matrix for stability: EΦ f- Σn tel.
5:
Set initial mixing proportion: nn +
6:
7:
end for
8:
repeat
9:
// E-Step:
for each observation j to M' do
11:
for each prompt index n = 1 to do
Compute the responsibility 7jn = ( mn fn (x;))/ /EN ti fi(z;)).
13:
end for end for
14:
15:
// M-Step:
16:
for each prompt index n = to N do
17:
Compute N,, Σ/= 7jn
N₂ + a M N(a - 1)
18:
Update mixing proportion end for
19:
20:
until convergence
21:
end for
20

---

Published as a conference paper at ICLR 2025
B MORE EXPERIMENTAL DETAILS
B.1 HYPERPARAMETERS
We use the AdamW (Kingma & Ba, 2015) optimizer following the official openclip implemen- tation. We also tried AdamP (Heo et al., 2021), which was the main optimizer of PCME++ (Chun, 2024). We empirically observe that AdamP can improve the overall performances (e.g., average performance from 57.3 to 57.8 under the same setting), but for a fair comparison, we follow the protocol of openclip ip (Ilharco et al., 2021). We use learning rate of 0.0005, betal of 0.9, beta2 of 0.95, weight decay of 0.2, and batch size of 512 for each GPU (i.e. the full batch size is 512 x 32= 16384). We apply 10000 warmup steps and then the learning rate is decayed by cosine learning rate scheduling. We use image augmentations of scaling 0.8 to 1.0, color jittering and grayscale. Among the mini-batch, we select 12.5% image-text pairs and drop 75% tokens of them to compute CimetaCamatatt In all experiments, we fix on and (2 in Equation (6) as 10-7 and 0.001, respectively. We tried various combinations of a, but we found that the final result is relatively robust to the choice of a unless we choose very large Similarly, we tested various and for the inclusion loss (Equation (5)), but as shown in Table C.5, the final result is relatively robust to the choice of the pa- rameters. If not specified, we chose 20 and c = =1000, but for future research, recommend using E = - 100 and 100, which empirically performs well and stable large-scale training.
B.2 FINE-TUNING DETAILS
During the project, we indeed aspire to train large models, such ViT-H, VIT-S0400M, VIT-G or ViT-g. However, as reported by Cherti et al. (2023), ViT-H/14 CLIP model with 34B seen samples (achieving 78.0% ImageNet zero-shot accuracy) task 279 hours with 824 1000s. Using our resource (32 H100s), training the same backbone from scratch takes almost 300 days. For this reason, we tried to fine-tune the pre-trained strong backbones. Note that our goal is ministic VLMs, but we report four fine-tuning results, ViT-B/16 (76.0% IN-ZS), ViT-L/16 (80.5% IN-ZS) and V (82.0% IN-ZS) pre-trained by SigLIP (Zhai et al., 2023) and ViT-H/14 (83.4% IN-ZS) pre-trained by DEN (Fang et al., 2024), for achieving a stronger PrVLM.
fine-tune the existing deter-
For the architectural consistency between all ProLIP models, we remove the attention pooling of pre-trained SigLIP and fine-tune the models using [CLS] token-based pooling and fix the image resolution to 224x224, This will lead to slightly worse performance of the fine-tuned models than the original models. Note that we can implement both attention pooling and [UNC] token architec- ture simultancously, but we did not implement the architecture for simplicity. We again emphasize that we do not aim to achieve the sate-of-the-at art performance, instead, our goal is to verify whether ProLIP performs better by scaling up the architecture beyond ViT-B/16. For the fine-tuning, we use the learning rate 5.0e-5, the weight decay as 0.0, and the number of seen samples 1.28B.
B.3 TRAINING AND EVALUUTTION DATASETS
We mainly use the DataComp IB dataset (Gadre et al., 2024), a filtered version of the LAION-5B dataset (Schuhmann et al., 2022), as our training dataset. We had 1,121,356,767 number of valid URLs among 1.39 billion URLs and 1, 8,443,,492 number of unique images after de-duplicating URLs (there were 147,676,246 number of duplicated URLs in the DataComp IB URLs). For the ab- lation study, we use ConceptualCaption 3M (Sharma et al., 2018), ConceptualCaption 12M (Chang- pinyo et al., 2021) and RedCaps (Desai et al., 2021) datasets.
We use 38 tasks from the DataComp evaluation suite: ImageNet (Russakovsky et al., 2015), 6 benchmarks for evaluating robustness under ImageNet distribution shifts, including ImageNet-A,, ImageNet-O (Hendrycks et al., 2021b). ImageNet-R R (Hendrycks et al., 2021a), ImageNet v2 (Recht et al., 2019), ImageNet-Sketch (Wang et al., 2019) and ObjectNet (Barbu et al., 2019), and 13 VTAB task (Zhai et al., 2019), including Caltech-101 (Fei-Fei et al., 2004), CIFAR-100 (Krizhevsky, 2009), CLEVR Counts, CLEVR Distance (Johnson et al., 2017), Describable Textures (Cimpoi et al., 2014), Au (Helber et al., 2019), KITTI Vehicle Distance (Geiger et al., 2012), Oxford Flowers- 102 (Nilsback & Zisserman, 2008), Oxford-III Pet (Parkhi et al., 2012), PatchCamelyon (Veeling et al.. 2018), RESISC45 (Cheng et al., 2017), SVHN (Netzer al., 2011) and SUN397 (Xiao et al., 2010), and retrieval tasks, including Flickr (Young et al., 2014), MS-COCO Cap- tion (Chen al., 2015) and WinoGAViL. (Bitton et al., 2022). There are also 13 additional tasks,
21

---

Published as a conference paper at ICLR 2025
such CIFAR-10, Country211 (Radford et al., 2021), FGVC Aircraft (Maji al., 2013), Food-101 (Bossard et al., 2014), GTSRB (Stallkamp et al., 2012), MNIST (LeCun et al., 1998), Pascal VOC (Everingham et al., 2010), Rendered SST2 (Radford et al., 2021), STL-10 (Coates et al., 201 I), iWildCam (Beery et al., 2021), FMoW (Christie et al., 2018), Dollar Street (Rojas et al., 2022), and GeoDE (Ramaswamy al., 2024).
For performing zero-shot classification, we use CSD as the distance between the given image and the pre-extracted text template features representing each class. As CSD uses uncertainty (Equation (I), this protocol uses uncertainty value for the zero-shot classification.
B.4 HIERARCAPS AND IMAGE TRAVERSAL DETAILS
The HierarCaps dataset (Alper& & Averbuch-Elow 2024) is human-validated caption dataset where each image has four levels of captions. For example, "water sports" + "kite surfing" + "kite surfer on top of the board" + "kite surfer in the air on top of red board". These captions are human- validated, namely, the level of the HaesarCaps dataset is aligned to the "hierarchical perception" of humans. As shown in Figure 7, the human-validated level of HierarCaps (i.e., human intuitions) is aligned well to the learned uncertainty by ProLIP.
We also showed that the learned uncertainty can improve the image traversal task. The traversal task needs two information: the closed text embedding and [RCOT] embedding for the given im- age. Once the closed text embedding and [ROOT] embedding are chosen, we interpolate them with 50 equally spaced steps and find the closest text from the database for each interpolated caption. Uncertainty information is used for the traversal task in two perspectives. First, we use CSD to re- trieve captions, where CSD uses the uncertainty (Equation (1)). More importantly, we estimate the [RCOT] embedding using uncertainty. Previous approaches use the average text embedding or null text embedding as [RCOT] embedding. Instead of using average or null text embedding, we pro- pose to use uncertainty-based [ROOT] embedding. As clarified in Section 4.4, we first retrieve the most similar caption of the given image (following the common protocol (Alper Averbuch- 2024)). Then, we search for the most inclusive caption of the retrieved caption in the database using the inclusion measure that we proposed. Then, we use the most inclusive caption as the [ROOT] embedding, which is a more plausible "root" compared to the average or null embedding.
As shown in Figure B.1, the [ROOT] embedding of CLIP is always closest to the "umbrella". which makes the image traversal by CLIP inaccurate. On the other hand, the [ROOT] embedding of ProLIP correctly estimates the true hierarchy of the given image query.
B.5 HIERARCHICAL IMAGES DATASET CONSTRUCTION
We construct the hierarchical image dataset by using the validation set of the COCO dataset (Lin et al., 2014), which includes both images and their corresponding segmentation maps. First, the images were filtered to retain only those where the smallest segment was larger than 5000 pixels. Then, the three largest segments were extracted from each image and pasted onto a blank canvas in descending order of size. We illustrate the generated samples in Figure B.2.
C
MORE RESULTS
C.I UNCERTAIN AND LONG CAPTIONS
The following single caption shows very high uncertainty value despite its length (e.g., top 20% uncertain samples in Figure 4). Note that Figure 6 shows that a caption with longer length might have higher uncertainty, but it is not always true if the caption only has almost useless information. Below, "UNC" denotes the uncertainty value of the corresponding caption, namely tr tr(e)).
UNCS 0.0415 "Seorching for Meaning: Menliam Bright Minds, Divillnioement, and Hope (Third Series See Winfin) Books Cover Image"
USC: 0.0415 "Grphic derigo profession workdek monite printer books lamp yo computerr illsmtation"
us0: 0.041 "Phona ID. 2299333 Viewn: 14221 UK Ait Force Earofillter E6-2000 Tiphion EGR (ZK306) shot at Faieford (FFD (FFD/ EGVA) UK- England July
21, 2013 Ry Mirhar! Rrapie--tiatiinn Imager"
UNC: 0.0416 "Vigo iuch Rasmhouse Aprin Single Bool Gouge Sminiess Steee Kitchon Sink with Zarich Chouse Faucet, Grid, Smotner Soop Diquorser"
UNC: 0-0418 "XIMOMI iNeathh FT-1018 Medical Baby High Souslivityy
Electric Thermometer UndvornnVial Soft Head Thernometer Adult Bobry Tihes-
mometer Sensor"
USC: 0.0462 "Mondofoowwrtonnss Scenie Sumple Insoices Created With Our Online Invoicing Solooue With Licions Sumple Insoice Template With Comelly Fire Eacel Imonice Also Imoice Tomplate Employed Additioo Used Vehiele Imsoice And Invoice Poyment Reminder Well Print Template Addition ally Services besice Template From Inviceeeeryron With Maddefaamortsaane Liriowe Sample Insoices Created With Our Online Imnicing Suftwure With Comely
22

---

Published as a conference paper at ICLR 2025
Query images
ProLIP
CLIP
ProLIP
CLIPP
ProLIP
CLIP
Prol.I
CLIP
Top-1 Retrieved captions
A large red chair borse statue on
red chair has
Small train running down the tracks middlee (ield
Small train running the tracks
An elderly
opens pizza box stares
A man opens pizza and stares at the camera
on a beach
elderly sits on bench with a crowd behind her
statue
it
crowd beliind ber
of field
the camera
red chair has borse statue
train on the tracks middlc of field
train tracks the middlc of
elder woman sitting with crowd elderly woman
red chair has horso statue
man opcnss pizza box
msen opens pizza box
oiter sila
train on tracks
train
pizza
red chair
umbrella
train
umbrella
umbrella
elderfy
umbrella
[ROOT]
chair
elderly elderty
Captions
train tracks
pizza opened
chair
Figure B.1: Image traversal comparison between ProLIP and CLIP The image traversal examples on HicrarCaps (Alper & Averbuch-Eloy 2024). The highlighted captions denote the wrong captions.
Sample Iavice Tomplate Semic Imsice Also fnviic Template Employed Addition Vhhcce bnvoce Inviicruutrycom
USC: 0.0662 "Carterssuus Ravisking Invice Fneewndheuplaeesn With Inqpring Proforma Imvice With Easy de The Eye Chicken Receipts Also Reccipt No- der Addiiio Cash Receipts Templatr And Read Reveijo For Gmuit As Well Anmul Gros Receigtr Additionally Plateposs Rereipt From Feevssndrmpiureee
Impiring Fiewwrdrrmulatzne"
Proforma fnvoice And Rovisling Chickan Reveips Also
Addition
Receigts Template
Ipec= 0. 0&.6 "Aldiohloous Ravishing Addiiian Workshests Dimamically Create Addition Workshests With Estrancing Addition Worksheet With Appraling Phon- Warksheers Free Also Functions Worksbest Alyebng ln Addition Cellular Respiration Worksbert Middle School And Beginning Addilion Worksheetts As Well As
Worksheer Mathaidhrom Aldiablosus Entrancing Addition Worksheett Dynamicolly World Workshwets Additionally Worksherts Rovizhing Phonics Harkshests Additno Workshess Appesting
Workshest Algefuo Addition Cellaker Respiration
Warksheet Middle School From Muthnidsoonn
C.2 MORE HERO-SHOT CLASSIFICATION EXPERIMENTS
We show the full results of tasks in Table C.I. We can observe that in most benchmarks. ProLIP outperforms CLIP and SigLIP. We also fine-tune the pre-trained SigLIP or CLIP models with our probabilistic training strategy. As we clarified in Appendix B.2, these models were slightly modified due to the architectural difference between the original pre-trained model and our main ProLIP model (e.g., attention pooling vs. [CLS] token pooling, and image resolution). Table C.2 shows that the overall performance is improved by increasing the parameter size (from 62.1 to 66.9).
C.3 ABLATION STUDY
In this subsection, we conduct the ablation study of our design choices. We train the models on Conceptual Caption 3M (Sharma al., 2018), Conceptual Caption 12M (Changpinyo et al., 2021) and RedCaps (Desai et al., 2021) datasets with 96M seen samples. This efficient setting enables to train a model in 7 hours with A100 NVIDIA GPUs. We measure the effectiveness our design choice in three categories. First, measure ImageNet zero-shot top-1 accuracy (IN-Topl). Second, we measure the average o² values of visual and textual modalities. a model captures the inherent uncertainty well, we assume that it will have higher uncertainty for captions, rather than images. Finally, we measure the HierarCaps recall, to measure whether the learned uncertainty captures the hierarchy of captions well.
23

---

Published as a conference paper at ICLR 2025
tho
LEN
tho
utc
uho
uho
Figure B.2: Hicrarchical COCO samples.
Table C.3 shows that (1) PCME+ loss is not trainable when we use more limited architecture than multi-heat architecture for estimating uncertainty. It supports our assumption discussed in Ap- pendix A.3. (2) PCME+ loss becomes learnable if we use deterministic loss together. (3) Line(uC() enforces images to belong into texts, namely image uncertainty tends to be smaller than text uncer- tainty with the inclusion loss. (4) Lin((r makes the embedding space can capture the hierarchy of data, namely, improves HierarCaps recall.
Although Table C.3 shows a good intuition for each loss function, we remark that the learned uncer- tainty might need more training time be more accurate. From this observation, ProLIP models by ablating the main losses on DataComp 1B with 1 1.28B seen samples. Table C.4 shows that using all proposed objectives achieves the best ImageNet and VTAB results. Although it shows slightly worse performance in retrieval, we found that its performance is reasonably good compared other variants, especially for the baseline (first row) in all measurements.
also compare
Table C.5 shows the impact of the choice of the stability parameters for inclusion loss. As we men- tioned Appendix I. the final performance is relatively robust the choice of hyperparameters. In practice, we recommend e = 100 and = 10 for both stable training and performance.
We also compare the efficiency of the proposed [UNC] token architecture compared the deter- ministic baseline and multi-head uncertainty estimate module by PCME+- We compare them with five different architectures (ViT-B/32, ViT-B/16, ViT-B/16 with 768-widte ViT-L/16, and ViT-L/14), Here, we only compare image encoders because image encoders always take the same image token length, which makes it easier to compare the impact of different design choices. We report (1) the input image token sizes for each architecture (" ["NC] token architecture" uses more), (2) the base parameters of the deterministic one, (3) inference time for 50k ImageNet validation images (lower is faster), and (4) additional parameters compared to the deterministic baseline.
Table C.6 shows three findings. First, even we increase the input token length, the actual inference speed is not quadratically slow ViT-B/32 and ViT-B/16 have the same Transformer capability but only input lengths are different (49 vs. 196), but their inference times are 75.18s and 76.68s, which almost neglectable change. If choose larger model (e.g., ViT-L,, the difference
24

---

Published as a conference paper at ICLR 2025
Table C.I: Zero-shot classification full results. "FT" denotes the fine-tuned results the pre-trained models with deterministic objectives. Details of fine-tuning can be found in Appendix B.2.
KITTI Velicle Distnnee
Deveiiable Texture
CLEVR Courts
CLEVR Disaance
FOVC Aitonft
Imagehhec Stesch
Caltedt- 101
CIFAR
CIFAR 100
Counry211
EuroSAT
Food 101
CTSRB
ImagetNer
ImageNNo
Imags-NosA
ImageNer-D
ImageNee-R
MNIST
ObjectNet
1,288 ween samples
CLIP
91.75
94.34
77.85
18.01
15.79
15.63
60.43
45.09
20.77 86.
50.27
67.24
55
58.95
30.73
53.70
76.35
30,80
72.96
55
SigLIP
92.77
94.17
78.25
20.11
15.87
15.18
60.90
39
21.01
85.96
40.71
67.37
55.93
59,67
30.80
54.25
76.11
18.00
73.17
55.60
77.48
47.17
76.24
Prol (VTT-B FT)
93.69
96.39
83
21.03
15.1
18.35
65.
57
40.31
90.55
74
60
68
06
46.81
43.55
86
27.29
Prol IP (VIT-LFF)
94.58
98.
87 79
23.31
16.27
26.56
70.53
71
49.91
94.
52.76
79
40
70.28
72
67.67
33.45
92 26
19.97
K5.05
75.37
ProLIP (50400M PT)
94.29
98.17
87
23.02
16.09
29.75
NL.27
64.96
46.11
94.27
57
22
79.26
69.74
72.55
70.05
33.15
92.32
92
74
76.00
ProLIP (WT-H PT)
95.05
98.29
87.54
23.11
22.90
30.64
70.64
59
9.80
49.85
94.64
79.41
69.61
72.83
67.97
33.00
92.07
16.03
87.35
74.43
ProLIP (12.88)
93.61 96.42 83.25
29.78
15.13
21,29
66.91
60.89
38.02
91.03
$2.79
74.56
63.65
66.66
50.25
45.40
86,00
32.21
84.47
65.80
Dxford Flowere- 102
Oxford IIT
Poscal VOC 2007
PurchCamelyon
Renkeed SSY2
RESISCAS
Stunfard Cans
STL-10
SUN397
SWHN
iWildCam
Canelyon17
Fhow
Flickr
MSCOCO
WHnGGAVIL
Dellar Street
GeaDE
Average
CLIP
69.67 98.57
81.38
56.67
$0.08
58.57
83.09
96.78
6641
59.68
11.16
56,03
10.57
71.09
45.49
43,46
56.43
86.97
57.12
56.72
Prol.PP
71.16
89:05
80.03
63.76
49. 75
57.75
85
96.41
63.67
62.60
10.13
64.
8.99
71.13
48.73
42.12
56.78
86.97
57.91
ProLIP (NTE-B FT)
79.76
93.49
79.37
55.87
55. 68
65.30
21
26
70.26
68.
15
15.28
63.47
11,15
79.00
51.71
44.32
61.80
88.81
62.13
ProLIP (NTT-L
45.
65.93
Prol.P (S0400M PT)
84.18
95.55
82.58
58.54
66 78 74.78
93.20
98
74.25
68.00
16.66
67.29
23.18
83.46
56.61
47
66.96
91.02
6663
PhoLIP (VIT-
82.78
95 64
82.82
64.25
39.
80
72,94
94.62
99.12
75.08
71.47
13:40
79.91
20,56
55
55.82
47. 38
66.24
91.73
66.90
ProLIP (12.88)
78.37 93.45 81,74 61,41
54.31
68.27
91.33
97.91
71.35
72.77
12.64
57.85
15.12
79.97
53.18
45.56
62.27
90.31 63.31
Table C.2: Comparison of munet ProLIP with 1.28B seen samples. VIT-H/14 based on CLIP by DFN (Fang et al., 2024), while the other models are based on SigLIP (Zhai et al., 2023). FLOps are for the basc pre-trained models, not modified architecture by ProLIP.
Backbone
FLOps
Samples Seen
ImageNet
IN dist. shifts
VTAB
Retrieval
Average
ViT-B/16
44.44G
1.28B*
74.6
62.2
61.2
58.3
62.1
VIT-L/16
136.41G
1
79.4
64.0
61.3
65.9
VÝT-S0400M/14
233.54G
1.28B*
79.3
69.0
65.1
62.5
66.6
ViT-H/14
381.68G
1.28B*
79.4
68.3
64.4
61.6
66.9
becomes larger, e.g., 137.77s vs. 176.95s, but it is still not a quadratic order. Second, [UNC] token adds almost neglectable parameters (0.3M for B and 0.6M for L) and inference time compared the deterministic one. Finally, the multi-head architecture needs large number of additional parameters (e.g., 20M for B and 40M for L) and shows slower inference time, especially for a larger network (e.g., 176.95s 191.68s for ViT-L/14). Furthermore, in practice, multi-head architecture requires more memory than [UNC] token, which makes difficult to use large batch size and scale up larger backbone. On the other hand, [UNC] token only needs almost neglectable additional parameters, inference speed, and memory size, which makes it easier to scale up.
C.4 MORE VISUAL EXAMPLES
We visualize more samples with combinations of various image and text uncertainties. Figure C.1 shows the example image-text pairs with their uncertainty values and similarity score measured by CSD (Equation (1)). The results are similar to Figure
C.5 MORE RESULTS FOR HIERARIMGS
Unlike text uncertainty (Figure 7), observe that the visual uncertainty values are not discrimina- tive by the levels in contrast to text uncertainty See Figure C.2a. Lower-level images tend to have larger average uncertainty (0.015) than the original images (0.014), but their differences not
25

---

Published as a conference paper at ICLR 2025
Table C.3: Ablation study, All models are trained on Conceptual Caption 3M (Sharma et al., 2018), 12M
(Changpinyo et al., 2021) and RedCaps (Desai et al., 2021) where the number of seen samples 96M.
Loss
Une Arch
Cin(ece)
IN-Topl
avg og
avg of H.Cap Recall
CLIP
35.5
48.4
PCME+ + CLIP multi-head
x
33.6
0.0160
0.0077
55.2
PCME++
[UNC]
x
x
0.1
0.0354
0.0347
PCME+ CLIP
[UNC]
x
X
36.1
0.2552
0.2118
53.8
ProLIP
[UNC]
X
37.4
0.3276
0.0745
44.8
ProLIP
[UNC]
36.8
0.0076
0.2324
46.7
ProLIP
[UNC]
X
37.5
0.3319
0.0610
47.9
ProLIP
[DNC]
V
V
37.0
0.0086
0.2254
54.8
Table C.4: Large-scale ablation. models are ViT-B/16 trained on DataComp IB with 1.28B seen samples.
Lim(vCC)
ImageNet
IN dist. shifts
VTAB Retrieval
Average
x
x
67.0
54.6
56.2
53.6
56.6
x
67.3
54.6
56.4
53.2
$7.0
x
67.4
54.4
56.4
53.2
56.7
67.6
55.0
57.1
53.4
57.3
Table C.5: Impact of and for the inclusion loss. Details are the same Table C.4.
8
ImageNet
IN dist. shifts
VTAB
Retrieval
Average
-20
1000
67.6
57.1
53.4
57.3
-10
1000
67.4
55.1
57.3
53.1
57.1
-5
1000
67.7
55.2
55.9
52.9
56.6
-5
100
67.8
55.3
56.7
53.0
57.5
-5
68.0
55.5
56.8
53.6
57.4
-10
10
67.8
55.3
58.5
53.0
57.9
100
10
67.7
55.5
56.9
53.7
57.5
significant between levels as texts. Instead of plotting every image in the same histogram, we plot the difference of uncertainty between the original image and its maksed versions in Figure C.2 (b-d).
To understand why some images have reversed image uncertainty by their hierarchy, we visualize the images whose original image is not included by level 0 images. Interestingly, as shown in Figure C.3, we can observe that many images with improper uncertainty estimate by hierarchy actually have wrong visual semantic hierarchy with severely occluded main objects. For example, in the upper row image, the dog the image appears the second level, but only reveals its part rather than the whole body. These results show that proper filtering on the Hierarlmgs dataset should be required for a more reliable evaluation.
C.6 MORE DISCUSSIONS FOR HUMAN PREFERENCE AND LEARNED UNCERTAINTY
[soulc(source) Include leval- (targett)
In Figure 7, show that the learned uncertainty by ProLIP is well separable by the hierarchy of HierarCaps which is validated by humans. Namely, HierarCaps cap- tion quadruple from level 0 (the most abstract one) to level (the most detailed one) has an inclusion relation- ship verified by humans. From this observation, we can conduct virtual human study determine whether the learned uncertainty correctly captures human preference.
1.0
0.9
Trurr expiten Parl
0.397
1.000
0.8
0.923
0.7
First, we measure how consistently the numbers in each HierarCaps caption quadruple are ordered decreasingly. Namely, we compute the following metric:
(C.1)
Figure C.4: The uncertainty value com- parison between different levels.
ITI
EET
26

---

Published as a conference paper at ICLR 2025
Image Uncertainty
MORIO Augui sodyy
All trrmrt toder
Git 1-1 3.064 10th GE 20
Seading
Octeore
DIOT & sisa suble
Woat
61 BITTS
Qutthet NOT
Oppa orkac Ring
Oeer Ore at cohams
aechnd ai 00410
ecoDes
DAKU FUT EVTN
TRACE PILLO
Chrprtof
Alles DE Frod
Beber Curreeo Hndee
El
QSEAN
DFL
Warth
lann senvens vetpccecidd cdc
Bery 2000
Near Tuan dastedes
18le 20012 EIm
SPALE
Problems
ofter deigh notter infeo
Sou Sie 2005 2054 Co LS 300G MASA
mISH hous Foor Hoase Flers
o2 M02 OLPO car 22 oube cond
contone
tatre Sobmiy zoess
Text Uncertainty
Figure C.I: Samples with high/low image/text uncertainty. Samples are drawn from the DataComp small subset. We also report CSD (similarity, lower closer) between the pair, o and of (lower is certain).
where I denotes the indicator function and Σ; denotes the uncertainty value of the level i of the caption t. Using ViT-B/16 ProLIP model with 12.8B seen samples, 90.0% of adjacent uncertainties satisfy decreasing order.
Second, as similar to Figure C.2, we show the uncertainty value comparison between different levels in Figure C.4. In the figure, have smaller uncertainty values than their corresponding level 0 and level captions. Also, large number of level 3 captions (0.923) have smaller uncertainty values than their level 2 captions. We
observe that most of the level 3 (all) and 2 (2 0.997) captions
27

---

Published as a conference paper at ICLR 2025
70.6% samples become more uncertain
level
175
level
250
of (0) a. (orig)
level
150
level (orig)
200
125
Frequency
150
100
75
50
25
0
0.015
8.020
-0.015
-0.010
-0.005
0.000
0.005
0.010
Hypothesis value
Image uncertainty
(a) All image uncertainties by hicrarchy
(b) o difference of original and level images
75.3% samples become more uncertain
76.6% samples become more uncertain
300
250
(3) (orig)
(2) (orig)
250
200
Requency
Fraquency
150
150
100
50
50
0-
-0.015
-0.010
-0.005
0.000
0.005
0.010
-0.010
-0.005
0.000
0.005
0.010
A Image unceralinty
Image uncertainty
o difference original and level images
(d) o difference of original level images
Figure C.2: Hierarlmgs of statistics.
Figure C.3: Examples of HierarImgs when ProLIP fails to capture of by hierarchy.
28

---

Published as a conference paper at ICLR 2025
Table C.6: Uncertainty architecture comparisons. The total inference speeds (in seconds) each architec- ture for 50k ImageNet validation images are shown (lower is better). The numbers in parentheses denote the number of additional parameters compared to the deterministic baseline model.
ViT-B/32
VIT-B/16
ViT-B/16-768
ViT-L/16
ViT-L/14
img tokens
49
196
196
196
256
Base param
151.7M
150.0M
197.3M
428.5M
428.4M
Multi-head
75.75s (+20.7M)
78.18s (+20.7M)
80.08s (+28.9M)
146.12s (+40.0M)
191.68s (+40.0M)
[UNC] (proposed)
75.24s (+0.3M)
76.84s (+0.3M)
78.61s (+0.6M)
137.99s (+0.6M)
177.82s (+0.6M)
Deterministic 75.18s
76.68s
78.53s
137.77s
176.95s
found the level 0 and level captions have relatively similar uncertainty values (but still 78.2% level captions have larger uncertainty than their level caption, but as we observed in Figure and B.1, the difference between level 0 and captions are often vague (e.g., water sports kite surfing). Overall, we argue that ProLIP captures human uncertainty preference well as supported by Figure C.4.
C.7 MORE DISCUSSIONS FOR BAYESIAN PROMPT RE- WEIGHTING (BPRW)
Hyperparameters. When ground-truth labels are not accessible (i.e., K 0 in Table 3), we set the a to 5. Then, we select 5 nearest images from each class embedding made by 80 prompts. We then sample 10 samples from the selected image embeddings and get 100 point image embeddings. Now, the sampled embeddings are used as observation of the algorithm (See Appendix A.8). We use =0.02 for stable convergence. For few-shot settings with K
set a to 2 and select 100 samples (e.g.. if K = 9, then we sample 1l point vectors from the image embeddings).
K
Visualization the learned weight. We show examples of ग and its corresponding images. Fig- ure C.5 shows the examples. We select three classes as examples and show their images and the learned æ. Interestingly, for "black-footed ferret" images, we found that the context "my" has more than 0.5 weight. The actual images of black-footed ferrets mostly composed of pet images, which makes sense that "my" prompt matches the images the most. Similarly, we observe that "front cur- tain" images are mostly low resolution due to the insufficient light in theaters, resulting in "a low resolution" or "a dark photo" becoming the most important prompts. Lastly, we see the missile im- ages are mostly in the museums, resulting in "a close-up" prompt becoming the most important, but not significantly (0.1 162) as much as the most contributing prompt of black-footed ferret (0.5130) and curtain (0.3921) images.
Using the π with a few-shot setting (K = 5), we visualize the learned prompt weight. Figure C.6a shows the histogram of the maximum ne for each c an uniform distribution will have 0.01, while larger max т. denotes that specific prompts specifically selected for the class. In the figure, we can observe that the æ generally larger than uniform. In addition, we plot the entropy of π in Figure C.6b, where it shows similar result.
C.8 MORE APPLICATIONS OF THE LEARNED UNCERTAINTY
Dataset filtering. Below, we show the DataComp CLIP filtering small track (filtering 12.8B noisy web-crawled image-text pairs) by using our method and baselines provided by DataComp:
ProLIP uncertainty-awar features help better filtering compared to the other baselines. However, we note that ProLIP is not specifically designed for dataset filtering: proposing new dataset filtering method using ProLIP will be an interesting future work, but not the scope of the current paper.
Understanding image dataset. ProLIP's image uncertainty is not the same as the "image uncer- tainty" of classification tasks. In classification tasks, an image has high uncertainty if it can be matched to "multiple classes", while ProLIP assigns a high uncertainty if an image can be described in "multiple different and various captions" (Appendix A.1). For example, assume image with white background and a clear overall object shape. For classification, it has low uncertainty because there is confounder the classification. However, ProL.P will assign a high uncertainty for this.
29

---

Published as a conference paper at ICLR 2025
From this, can think two different scenarios: (1) When all images have homogeneous back- ground and only the quality of the image determines the classification performance (e.g. MNIST), (2) When images natural images and task is inherently multi-object classification, but the labels are single-labeled (e.g,, ImageNet as discussed by (Beyer et al., 2020; Yun et al., 2021))
Class name: Black-footed ferret
[0.5130] photo of my black-footed ferret.
[0.0250] itup of my Black-footed ferret.
[0.0056] origami black-footed ferret
[0.0056] sculpture of black-footed ferret
Class name: Front curtain
0306 Tawihan
[0.3921] low resolution photo of the front curtain.
[0.1102] dark photo of front curtain.
[0.0056] drawing of front curtain
[0.0056] a cartoon front curtain
Class name: Missile
[0.1162] a close-up photo of a missile.
[0.0842] black and white photo of the missile.
[0.0056] the plushie missile
[0.0056] doodle of the missile
Figure C.5: Visualization of learned by BPRW for each class.
30

---

Published as a conference paper at ICLR 2025
80
60
Freeuency
90
Prequencyy
30
20
0.1
0.3
0.5
2.8
3.0
3.2
3.4
3.5
3.8
4.0
max R
Entropy of
(a) Histogram of max The uniform distribution
(b) Histogram of entropy of The uniform distribu- tion will have 6.9078 entropy value.
will have 0.001 max
Figure C.6: Statistics of the learned a by BPRW. We use obtained from the few-shot setting with K 5.
Table C.7: Dataset filtering, Results on DataComp small track (Gadre et al., 2024).
Size
ImageNet
IN dist.
VTAB
Retrieval
Average
No filtering
12.8M
0.025
0.033
0.145
0.114
0.132
Random subset (25%)
3.2M
0.022
0.032
0,130
0.099
0.126
LAION-2B filtering
1.3M
0.031
0.040
0.136
0.092
0.133
English (fasttext), cap length, and img size
0.038
0.043
0.150
0.118
0.142
Image-based & CLIP score (I/14 30%)
1.4M
0.039
0.045
0.162
0.094
0.144
CLIP L14 (20%)
2.6M
0.042
0.051
0.165
0.100
0.151
ProLIP distance (20%)
2.3M
0.042
0.047
0.167
0.117
0.154
As the first example, we choose MNIST. We observe that the learned image uncertainty and the MNIST accuracy show a strong negative correlation, (-0.98), namely, if an image is more uncertain then ProLIP tends to estimate wrong label. As the second example, we choose ImageNet- which shows strong positive correlation (+0.98), a certain image tends be wrongly classified by ProLIP. This could be counterintuitive in "classification" but it is a correctly estimated value. For example, ImageNet contains various image distributions. Some images are thumbnail images with white background (high uncertainty) and some images are in-the-wild images with complex background and objects (low uncertainty). In this case, a certain image (more complex images) will be more "difficult" images to be classified, which supports the positive correlation.
Overall, ProLIP's image uncertainty tendency can be used to understand an image dataset. Convert- ing ProLIP's image uncertainty to image classification uncertainty would be interesting topic.
Uncertainty by image manipulation. We additionally show the relationship between image ma- nipulation and uncertainty. We evaluate ImageNet zero-shot accuracy by applying center oc- clusion. We applied 0% 10% occlusion ratio (Table C.8). We also tested optimized noise by the PGD attack (Madry et al., 2018) with sampled ImageNet (Table C.9):
Table C.8: Occlusion vs. image uncertainty. Numbers are measured in the ImageNet validation set.
Occlusion ratio
0%
2.5%
5%
7.5%
10%
ImageNet-Ik zero-shot avg(o.))
74.6
74.1
73.8
73.5
73.2
0.0148
0.0149
0.0152
0.0153
0.0153
Here, observe that the image uncertainty is increased by more severe manipulation. Note that as we discussed in "Understanding image dataset", converting ProLIP's image uncertainty to image qualification would be interesting topic, but remain this for future work.
31

---

Published as a conference paper at ICLR 2025
Table C.9; PGD vs, image uncertainty. PGD, denotes the PGD attack with iterations.
Clean
PGD:
PGDs
PGD1o
PGD40
ImageNet-Ik Ik zero-shot (1000 images)
72.9
20.0
3.8
2.6
2.5
av)(
0.0147
0.0149
0.0167
0.0175
0.0190
ProLIP with long context text. Although ProLIP can capture inherent ambiguity in vision- language tasks, ProLIP is limited to capturing long context text exceeding 64 tokens. To tackle the issue, we fine-tune the pre-trained ProLIP with long context texts, following LongCLIP (Zhang et al., 2024). In our primitive study, we observe that fine-tuning solely with the long text dataset can lead to a significant performance drop for general zero-shot tasks. More detail of our extension, LongProLIP (expanding the text context length from 64 to 256), is out of the scope of this paper, refer to a technical report for LongProL.IP for more interested readers (Chun & Yun, 2025).
D DISCUSSION AND LIMITATION
As ProLIP is based on a normal distribution with diagonal covariance, ProLIP also shares two con- cerns discussed by Chun (2024): (1) using diagonal normal distribution would be insufficient to represent compared to the full covariance, (2) if we use different probability distributions (e.g., von Mises-Fisher distribution or Laplacian distribution), the closed-form for PPCL and inclusion loss will not work anymore.
For the first concern, already discussed by Chun (2024), the diagonal covariance would be insuf- ficient if the dimensionality is too small (e.g.. less than ten). In this case, using the full covariance or mixture of Gaussian (MoG) will improve the representation power of the uncertainty. However, in practice, we use very high dimensionality, e.g., 768 for ViT-B/16, that can sufficiently capture complex semantics. One also can argue that using MoG is more sensible to capture many-to-many correspondences. However, if we have sufficiently large dimensionality, MoG will not be effective. Consider probabilistic embedding with 2-MoG, namely
[M( Σ1) with probability 0.5 and Z ~ (12,, Ez) with probability 0.5. We can compute the expected CSD between two Zs (where 21 is parameterized by pis p1z E1 ΣY and Ze is parameterized by م 12 E Σi by computing
4 {d(ut p4. 21 2n) + d(pt, 2 21, 23) + d(pb, uL. 212
23 dl (22 23. 2. 22)}
(D.1)
where d(-) is CSD. i.e., d(u), 12, Σ1, E2) || μu 12 + tr(E + Σ2). To simplicity, we omit for the remaining derivation. Now, consider virtual unimodal Gaussian embeddings Wi N Wpleen +1112))))) and W2- ~ NVur ep3 o 222)), where
@ denotes the "concatenation" operation, i.e., W and W2 have four times larger dimensionality than Zi and Zz. Interestingly, we can easily show that Equation (D.1) equals CSD between W1 and W2. Note that this derivation is invariant to the diagonal covariance, but also holds for the full covariance. In other words, using MoG is mathematically equivalent to using a larger dimensionality (as much as the square of the number of modes); therefore, if we have sufficiently large dimensionality that can capture the ambiguity of the dataset, MoG is not a mandatory option.
The second concern can be raised when Chun (2024), all objective functions and computations are distribution-free, but the derived closed- form solutions (e.g., CSD, inclusion loss) will not work anymore if we different distributions. One exception is MoG with equal mixing coefficients, but it equals simply using larger dimension- ality. In practice, if we really need different distributions, we can use Monte-Carlo approximation as Chun et al. (2021), which is known to be inefficient and inaccurate (Chun, 2024).
different probability distribution. As discussed by
32


Reviews:
[{"review_id": "6JfhzmfkXr", "reviewer": "Reviewer_1UQo", "summary": "The paper proposes a novel approach for vision-language models (VLMs) that leverages probabilistic embeddings to capture better the inherent many-to-many relationships between images and text descriptions. \n\nIn general, the contribution includes\n* Novel Probabilistic Embeddings: The paper introduces a fully probabilistic VLM that assigns each image-text pair a probability distribution rather than a deterministic embedding, addressing the inherent ambiguity in real-world image-text matching. This approach shows promise in capturing a richer semantic alignment by allowing multiple captions to represent an image and vice versa.\n* Efficient Uncertainty Estimation: Unlike prior probabilistic VLMs, ProLIP efficiently estimates uncertainty by introducing an “uncertainty token” ([UNC]). This no-parameter simplification is computationally efficient and effectively captures the variance within embeddings, enhancing interpretability.\n* Inclusion Loss: This paper introduces a novel inclusion loss, which enforces distributional inclusion relationships, and strengthens ProLIP’s interpretability by aligning learned uncertainties with human expectations (e.g., general captions should cover more specific ones).\n* Performance on Zero-Shot Classification: ProLIP demonstrates strong results in zero-shot classification tasks, such as outperforming a CLIP baseline with similar architecture. The improvement in ImageNet classification accuracy (from 74.6% to 75.8%) with the use of uncertainty highlights the approach’s practical advantages.", "questions": "1. Could you clarify the design choice of the probabilistic pairwise contrastive loss (PPCL) over simpler alternatives, like KL divergence-based objectives? Have you tested how PPCL compares with other probabilistic loss functions in terms of training stability and final performance?\n\n2. The inclusion loss is an interesting addition. Could you explain in more detail the intuition behind using this specific formulation, and how it compares with other metrics like Wasserstein distance?\n\n3. The paper shows that ProLIP’s uncertainty estimates align well with general human expectations, such as more uncertainty for shorter text descriptions. Did you conduct any quantitative or qualitative user studies to validate this alignment? If not, how would you suggest evaluating the alignment between model uncertainty and human perception in future work?\n\n4. You demonstrated ProLIP’s zero-shot classification ability, but could it also support a few-shot learning approach? What further modifications would be needed to explore few-shot learning capabilities and potentially improve its applicability in low-resource settings?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Originality\n* The paper introduces a novel Probabilistic Vision-Language Model (PrVLM), named ProLIP, which represents a shift from deterministic vision-language models to probabilistic embeddings. This approach innovatively captures the natural ambiguity in image-text relationships, where multiple captions may describe a single image, and a single caption may match multiple images.\n* The concept of an uncertainty token ([UNC]) is a creative contribution, allowing the model to quantify uncertainty in a computationally efficient way, without additional parameters. This method differentiates ProLIP from previous probabilistic VLMs, which typically rely on additional modules to estimate uncertainty.\n* The inclusion loss is another novel component of the paper. By enforcing a distributional inclusion relationship, the loss function allows the model to create more interpretable representations that respect hierarchical relationships within image-text pairs. This focus on hierarchical embedding structures is an innovative step for VLMs.\n\n2. Quality\n* The authors have conducted a thorough and robust set of experiments to validate ProLIP's effectiveness. These include zero-shot classification tasks across multiple datasets, evaluations of ProLIP’s uncertainty estimation, and comparisons with both deterministic (e.g., CLIP) and probabilistic baselines.\n* The paper is methodologically sound and clearly specifies the assumptions, architectures, and loss functions. The experiments are carefully controlled, with baseline comparisons and ablation studies that effectively illustrate ProLIP’s strengths in various contexts.\n\n3. Clarity\n* The paper is written in a clear and accessible manner. Technical terms and novel concepts (e.g., inclusion loss, uncertainty token) are well-defined, with explanations that make the content readable.\n* Figures, such as the visualizations, are well-crafted and effectively illustrate key points. \n\n4. Significance\n* The introduction of probabilistic embeddings and inclusion loss contributes to the broader AI field’s understanding of how uncertainty modeling can enhance interpretability and generalization in multi-modal tasks. This may inspire further work in this direction.", "weaknesses": "1. Interpretability and Visualization: \n*  Although the inclusion loss aligns with human intuitions, the visualization methods could be improved. For example, it would be interesting to see more direct comparisons with non-probabilistic models in terms of how well ProLIP aligns visual and textual hierarchies in practice.\n\n2. Limited Exploration of Task-Specific Uncertainties: \n*  The paper briefly touches on using uncertainty for image-to-text retrieval but could expand by exploring how uncertainty contributes to other downstream tasks. For instance, analyzing ProLIP’s impact on retrieval diversity or specificity could further highlight its advantages.\n\n3. Human evaluation\n*  While the experiments indicate that ProLIP’s learned uncertainties align with human intuitions, a concrete user study or human evaluation would reinforce these claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bHwqX3p4Gl", "reviewer": "Reviewer_N1uc", "summary": "This paper introduces ProLIP, a fully probabilistic approach to language-image pre-training. The authors begin by noting that CLIP's loss function oversimplifies the complex relationships between real-world images and texts. In response, ProLIP proposes a novel loss function that better captures the distributional relationships between these modalities. Extensive experiments were conducted to demonstrate the effectiveness of the proposed method.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper is well-motivated, starting from the many-to-many matching relationships within a batch of images and texts. It is also well-structured and easy to follow.\n- The authors provided strong mathematical support for the proposed learning objective.\n- Extensive experiments were conducted to demonstrate the effectiveness of the proposed method.\n- The proposed method has been proven effective on datasets containing billions of image-text pairs.", "weaknesses": "- The proposed method was trained only on ViT-B and ViT-L vision encoders. Scaling up to ViT-H  and comparing with other methods are important to further demonstrate the scalability of ProLIP.\n- As shown in Table 1, ProLIP introduces a more complex loss design and training process compared to CLIP, yet it achieves only a 0.9% improvement in ImageNet zero-shot classification and an average gain of 0.7%. Also in Table C.1, ProLIP with ViT-L also just exhibits slight improvement over CLIP. This modest improvement may indicate scalability challenges for the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o7QnS9rrw2", "reviewer": "Reviewer_t6Gq", "summary": "The paper works on a probabilistic CLIP that can model the uncertainty of the data. The model measures the uncertainty with a multi-dimensional Gaussian distribution, and it is trained with proposed novel inclusion loss. The paper also simplified the sampled distance loss from previous paper. Given that, the paper first shows some analysis that the estimated uncertainty aligns with several intuitions. Lastly, it shows two applications of image traversals (find a more concrete caption for the original caption iteratively) and prompt enhancement for the zero-shot image classification task.", "questions": "1. Please specify the context why $\\mu_1$ and $\\mu_2$ are norm-1 vectors (i.e., because CLIP normalizes them during training?)? I think that the original CLIP only normalizes when calculating the contrastive loss, maybe there is a re-parameterization here. Also please include them in text which would be friendly to reader. \n\n2. I wonder how the estimated variance from the model is used in the classification task? \n\n2. I appreciate the comparisons to KL divergence in Sec 3.3. The paper only measures KL(p1||p2) which has a different form to the proposed inclusion metric. How about adding a comparison to metric KL(p1 || p2) - KL(p2 || p1)? \n\n3. The inclusion loss is defined as $L(Z1, Z2)$ = $Z1 \\in Z2$. Thus, in Line 259 and Line 264, should that be $L(t, v)$ and $L(partial, all)$ instead the current $L(v, t)$ and $L(all, partial)$? \n\n4. In Fig 5, it shows that the estimated variance of image is smaller than text. The loss defined in Eqn (6) is unsymmetrical for image and text (because of the hyperparemeters and the text VIB loss). Would that be possible that the smaller image variance is from this loss design? \n\n4. Please specify the exact formulation (i.e., implementation) of VIB in main text or in appendix.\n\n4. Please specify how the uncertainty information is utilized when conducting the image traversals task. \n\n5. (An open question; no concrete answer required) What would be the golden uncertainty of a particular data in the PrVLM? I.e., the best distribution that a model should converge to?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper constructs a probabilistic VLM that can capture of uncertainty from the multimodal dataset. \n\n2. The paper proposes the inclusion loss and also provide intuitive understanding of it. \n\n3. The paper contains interesting analysis of the uncertainty. \n\n4. The paper proposes to utilize the uncertainty to conduct prompt rewriting.", "weaknesses": "1. The intuition behinds the paper is that multimodal paired data can be actually many-to-many mapping. I think that this intuition would lead to a multi-modes representation intuitively, as different text can be assigned to a single image, e.g., \"A dog is walking\" and \"A man is looking at the building\". However, the modeling in this paper is a Gaussian distribution with a single mean that can not conduct good mode coverage. \n\n1. (Minor) The basic behinds contrastive learning is that the positive pair is sampled from the joint distribution $P(x, y)$ while the negative pairs are sampled independently $P(x)P(y)$. Given that, contrastive learning does not force a 1-1 mapping from its maths intuition. Practically, the behavior might be different from the above as model learning might force 1-1. The motivation in Abs/Intro and Sec 2.1 might need a revision. The current claim is a little bit stronger to me and this might be fixed by writing or explanations. Also, in Sec 2.2, it claims that \"VL tasks suffer from uncertainty as shown in Sec 2.1\". This is a little bit beyond the context of Sec 2.1, as Sec 2.1 does not have strong VL task experimental evidence. I treated it as over-claim now but I think that it can be solved by writing revision. \n\n1. The two applications (image caption traversals and prompt enhancement) of the estimated uncertainty are not representative, and it would be better if more applications (more concretely, proof of concepts) are presented. For example, the data filtering, error estimation, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BbFCwohhgm", "reviewer": "Reviewer_RZmV", "summary": "The paper proposes a probabilistic VLM model that accounts for the many-to-many relationships between images and text. This model introduces an uncertainty token to estimate uncertainty without significantly increasing the parameter count. Additionally, the inclusion loss enhances interpretability by enforcing consistency between image-text pairs and between original and masked inputs.", "questions": "1. In Table C.1, ProLIP underperforms compared to CLIP on several datasets, such as Flickr. Could the authors provide a detailed analysis to explore the reasons behind this discrepancy?\n2. The paper suggests that images with a single object are more uncertain than those with complex scenes. Intuitively, one might expect complex scenes to have higher uncertainty due to the variety of possible descriptions. Could the observed uncertainty levels be influenced by the choice of dataset used in the analysis?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper creatively applies probabilistic modeling to VLMs to better capture the many-to-many relationships between image and text pairs. It introduces a straightforward setup by adding an uncertainty token, effectively minimizing additional parameters. The paper provides a detailed analysis of the sources and levels of uncertainty in both image and text modalities, and explores various applications of this approach.", "weaknesses": "The paper primarily evaluates the model's performance on single-object scenarios, as seen with ImageNet in the main results (Table 1). While Table C.1 provides metrics for datasets with longer, more complex queries, such as Flickr, ProLIP shows a significant drop compared to CLIP. Including performance metrics and a detailed analysis of datasets with complex queries involving multiple objects and interactions in the main results would offer a more comprehensive evaluation of the model's capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9JCGzbjayc", "reviewer": "Reviewer_8Wie", "summary": "The paper introduces **Probabilistic Language-Image Pre-training** (ProLIP), an approach to vision-language models that integrates probabilistic modelling to capture the inherent uncertainty in image-text relationships. ProLIP departs from conventional deterministic embeddings by mapping inputs as random variables, estimating uncertainty using an \"uncertainty token,\" and enforcing an inclusion loss to maintain distributional relationships between image-text pairs and between original and masked inputs.", "questions": "- Although the authors claim ProLIP efficiently estimates uncertainty by incorporating an \"uncertainty token\" without adding parameters, this approach effectively increases the input length. Given that transformer complexity is quadratic relative to the input length, this addition likely increases the computation required for multi-head attention. How does the \"uncertainty token\" quantitatively compare to previous probabilistic VLMs in terms of training efficiency and inference speed? For example, the authors can report training time per epoch, inference latency on standard hardware, or FLOPs compared to baseline models.\n\n- Is there any limitation discussed in the paper? (e.g. computational requirements)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-  This paper is well-written.\n-  The research problem and motivation are well-defined, highlighting that CLIP-like models tend to oversimplify image-text alignment by assuming a one-to-one relationship.\n-  ProLIP is claimed to be the first approach to pre-training probabilistic VLMs on a billion-scale image-text dataset, a feat that requires extensive computational resources.", "weaknesses": "-   The improvement over deterministic VLMs like CLIP is limited, with only a marginal gain of around 0.9% in ImageNet accuracy (Table 1) when models are exposed to the same number of samples.\n-   While the paper introduces the uncertainty token for efficient uncertainty estimation, it lacks empirical evidence demonstrating the claimed efficiency.\n-   Compared to PCME++, ProLIP incorporates an uncertainty token and an inclusion loss, scaling the training dataset to CLIP’s level, yet the observed improvement remains modest, limiting the paper’s overall contribution.\n-  ProLIP requires the use of CSD to perform zero-shot prediction, which increases its complexity and makes it less straightforward than CLIP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)