Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models
Authors: Simon Schrodi
David T. Hoffmann
Max Argus
Volker Fischer
Thomas Brox
Venue: ICLR 2025 Oral (2025)
Keywords: CLIP
modality gap
object bias
contrastive loss
data-centric
vision language models
VLM
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We find that an information imbalance between images and texts leads to the modality gap and object bias of contrastive VLMs. We study both phenomena  in depth, eliminate common misconceptions, and improve the understanding of both of them.

Abstract: Contrastive vision-language models (VLMs), like CLIP, have gained popularity for their versatile applicability to various downstream tasks. Despite their successes in some tasks, like zero-shot object recognition, they perform surprisingly poor on other tasks, like attribute recognition. Previous work has attributed these challenges to the modality gap, a separation of image and text in the shared representation space, and to a bias towards objects over other factors, such as attributes. In this analysis paper, we investigate both phenomena thoroughly. We evaluated off-the-shelf VLMs and while the gap's influence on performance is typically overshadowed by other factors, we find indications that closing the gap indeed leads to improvements. Moreover, we find that, contrary to intuition, only few embedding dimensions drive the gap and that the embedding spaces are differently organized. To allow for a clean study of object bias, we introduce a definition and a corresponding measure of it. Equipped with this tool, we find that object bias does not lead to worse performance on other concepts, such as attributes per se. However, why do both phenomena, modality gap and object bias, emerge in the first place? To answer this fundamental question and uncover some of the inner workings of contrastive VLMs, we conducted experiments that allowed us to control the amount of shared information between the modalities. These experiments revealed that the driving factor behind both the modality gap and the object bias, is an information imbalance between images and captions, and unveiled an intriguing connection between the modality gap and entropy of the logits.

Published as a conference paper at ICLR 2025
TWo EFFECTS, ONE TRIGGER: ON THE MODALITY GAP, OBJECT BIAS, AND INFORMATION IMBALANCE IN CONTRASTIVE
MODELS
Simon Schrodi David T. Hoffmann-.12 Max Argus' Volker Fischer Thomas Brox
University of Freiburg, Boosc Center for Artificial Intelligence
ABSTRACT
Contrastive vision-language models (VLMs), like CLIP, have gained popularity for their versatile applicability various downstream tasks. Despite their suc- cesses in some tasks, like zero-shot object recognition, they perform surprisingly poor other tasks, like attribute recognition. Previous work has attributed these challenges to the modality gap, a separation of image and text in the shared rep- resentation space, and to a bias towards objects over other factors, such at- tributes. this analysis paper, we investigate both phenomena thoroughly. We evaluated off-the-shelf VLMs and while the gap's influence performance is typically overshadowed by other factors, we find indications that closing the gap indeed leads to improvements. Moreover, few embedding dimensions drive the gap and that the embedding spaces are differ- ently organized. To allow for clean study of object bias, we introduce a definition and corresponding measure of it. Equipped with this tool, we find that object bias does not lead to worse performance on other concepts, such as attributes per However, why do both phenomena, modality gap and object bias, emerge in the first place? To answer this fundamental question and uncover some of the in- ner workings of contrastive VLMs, we conducted experiments that allowed us control the amount of shared information between the modalities. These experi- ments revealed that the driving factor behind both the modality gap and the object bias, is an information imbalance between images and captions, and unveiled an intriguing connection between the modality gap and entropy of the logits.
find that, contrary intuition, only
INTRODUCTION
Contrastive Vision- LanguuLe Models (VLMs) are successfully applied to numerous tasks. They benefit from their ability to exploit weak supervision in contrastive pre-ttrining (Radford et al., 2021; Jia et al., 2021), which can be acquired by scraping image-text pairs from the internet. In spite of this, they exhibit intriguing properties: strong zero-shot image recognition performance (Radford et al., 2021; Menon Vondrick, 2023), cross-modal understanding (Chen et al., 2023a), retrieval (Ma et al., 2022), robustness (Nguyen et al.. 2022). Despite such remarkable advancements, understanding of the representations learned by VLMs is still limited. Liang et al. (2022) identified modality gap in the shared embedding space and Bravo al. (2023) conjectured about bias towards objects. However, how do these phenomena affect downstream performance, why do they emerge, and we mitigate them? To answer these questions and enhance our understanding of the learned representations, we thoroughly study both, modality gap and object bias
The modality gap is geometric phenomenon characterized by the modalities lying in com- pletely separate regions of the shared emhedding space of contrastive VLMs. Liang et al. (2022) attributed its emergence the cone effect during model initialization with the contrastive loss pre- serving the gap. Subsequent work studied the influence of the temperature parameter (Udandarao, 2022: Shi et al., 2023). Intuitively, one would expect the gap to limit performance, but despite pre- vious efforts, the impact of the modality gap performance has remained unclear so far: Is the gap even a relevant problem worth fighting? Similarly, the interaction between closing the gap post-hoc
"qqul contribution. Correspondence to: {schrodi, hoffmann @cs.umi-freibur..de. Code is available https: / http:/gtttttt thub. wtw.bcoowwoo...ccccccccccottoaao cwm/ 1mb- fre wbbug/twooweeeeeeeeeeeereooo wwweeecttooeeeerreeeeeeee

---

Published as a conference paper at ICLR 2025
and its effect on performance also remained elusive. Beyond these performance-relatte questions, we aim to more thoroughly understand the modality gap: Are all dimensions contributing equally? Are the modalities structured similarly, in sense that neighborhood relations are similar? Do the representations of both modalities have similar meaning? Lastly, we investigate whether the modal- ity gap is rather bug feature. Despite the importance of these questions, they have not been thoroughly addressed so far. Our analysis offers answers to these questions and equips practitioners with actionable insights to address various issues related to the modality gap.
Beyond the modality gap, recent work (Bravo et al., 2023; Trager et al., 2023) found significantly worse performance of contrastive VLMs for attribute tasks compared object tasks, which led them to the hypothesis that they are biased towards objects (Bravo et al.. 2023). However, "bias towards objects" was previously not formally defined and was only assessed by the poorer performance. For thorough study of object bias and its emergence, we go beyond the previous notion of bias towards objects and introduce a novel metric to measure object bias: Matching Object Attribute Distance (MOAD). It assesses the bias towards objects compared to other factors, such attributes. Beyond assessing bias towards objects or attributes, MOAD's ' generic formulation allows to also study other types of biases in learned representations. Equipped with this new metric, can now answer questions about object bias: Does object bias even affect performance on non-object (attribute) tasks? Are objects just more often mentioned in captions, leading to the object bias? Answering these questions, can guide us to effective strategies to overcome such biases of VLMs.
Finally, investigate the fundamental ques- tion of what actually leads to the emergence of the modality gap and object bias. We identify a common cause: information imbalance. In- formation imbalance refers the availability of
photo of a red cat sitting on a tree with yellow leaves
oracle image encader
perfect alignment impossible
captioned by human
more information in one modality than the other. For example, captions are often sparse (lossy), focusing on the most salient object(s), while im- ages hold far more information not captured by their captions; see Figure 1. Since the caption is unknown to the image encoder, it cannot know what information of the image it needs to en- code to align the image encoding with the text encoding. The best that the image encoder can do is to focus on the most salient parts of the image, i.e., the parts that are typically present in the captions. As a result, the image encoder de- velops bias towards these parts. For natural language captions, these are typically object names. Besides that, the modality gap also emerges as a by-product of contrastive training under an in- formation imbalanced regime. Here, the model trades off alignment-which is limited due to the information imbalance- wwith uniformity by making all images and all texts more dissimilar. We thoroughly validate our hypothesis by manipulating information imbalance in experiments on syn- thetic well real data. Our insights equip researchers with the understanding needed to craft, for example, data filtering or caption enrichment strategies to reduce the modality gap or object bias.
a photo of a red cat
photo of red
oracle text encoder
Figure I: Illustration of information imbalance between images (top left) and captions (bottom left). This imbalance makes it even for an oracle image encoder virtually impossible to predict the content of caption, leading to undesirable ef- fects in contrastive training, such as the modality gap and object bias (see Section 6).
The findings of our analysis paper are: 1) For off shelf contrastive VLMs, larger modality gap correlates with better performance due to common confounders. However, controlling for these confounders suggests that a lower modality gap correlates with better performance. 2) Only few embedding dimensions drive the modality gap. 3) Image and text embeddings have distinct char- acteristics, like different neighborhood orderings. 4) Object bias does not negatively correlate with performance on object tasks. 5) An information imbalance between the modalities leads to both the modality gap and the object bias. 6) The modality gap is a by-product of the model's 's efforts to deal with the uncertainty (entropy) caused by information imbalance. 7) Object bias is caused by higher per-sample caption presence bias, which is also consequence of the information imbalance.
2 RELATED WORK
Contrastive VLMs have emerged as an effective approach to learn representations through weak supervision that work for wide range of tasks and have intriguing properties, such strong zero-
2

---

Published as a conference paper at ICLR 2025
shot abilities. However, the learning framework comes with potential issues and our understanding of it is in its infancy.
For example, recent work showed the existence of modality gap. They attributed it to the cone effect at model initialization and the contrastive loss that preserves it (Liang et al., 2022). The cone effect refers to the observation that the embedding space is restricted to narrow cone. The modality gap emerges at initialization, as it is unlikely that both encoders use the same cone with random iniiiiaiization. Subsequent work studied the influence of the temperature parameter (Udandarao, 2022; Shi et al., 2023). Other work found that the modality gap is orthogonal to the span of image and text embeddings (Zhang et al., 2023). In this work, we find that few dimensions drive the modalities apart and show that information imbalance is the main factor for the emergence of the modality gap.
Geirhos et al. (2021); Li et al. (2023b) found that large contrastive VLMs close the gap to human per- ception, but others found several failure modes of them (Yuksekgonul et al., 2022; Brody, 2023). The importance of data was studied by Nguyen et al. (2022); Xu et al. (2024), generalization/robusttess by Mayilvahanan et al. (2024); Crabbé et al. (2023), the learned features were analyzed by Goh et al. (2021); Materzyńska et al. (2022); Rashtchian et al. (2023), compositionality was studied by Jia et al. (2021); Couairon et al. (2022); Trager et al. (2023), and learned abilities and (social) biases by Agarwal et al. (2021); Yamada et al. (2023); Zhang et al. (2022); Wu & Maji (2022); Shtedritski et al. (2023); Hamidieh et al. (2023). Lastly, Bravo et al. (2023) hypothesized that VLMs may be biased towards objects. Here, we validate that VLMs are biased towards objects but find that VLMs with smaller object bias may not necessarily perform better on attribute tasks.
3 EXPERIMENTAL SETUP
In this work, we performed a series of experiments using both real as well fully- cullyceontrollable, synthetic data. Below, we briefly outline the common experimental details. We describe the specific experimental settings in the respective sections.
Pre-trained contrastive VLMs. We used a total of 98 contrastive VLMs such as CLIP (Radford et al., 2021) SigLIP (Zhai et al., 2023) for our analysis. The VLMs were pre-trained various datasets. Refer to Appendix A for the full list. We distinguished between VLMs pre-trained on medium- (i.e., dataset size of $140 M) and large-scale datasets. For in-depth analysis, we used the pre-trained CLIP ViT-B/16 (Radford et al., 2021) and SigLIP ViT-B/16 (Zhai et al., 2023).
Evaluation protocols. We evaluated the pre-trained contrastive VLMs on MS COCO (Lin al., 2014: Chen et al.. 2015), ImageNet (Russakovsky et al., 2015), MIT-States (Isola et al., 2015), and UT-Zappos (Yu & Grauman, 2014) with the standard evaluation protocols from the literature. We prepended the prompt "a photo of g to the description, object class, or attribute, following Radford et al. (2021). For evaluation, we used R@ for zero-shot image-to-text retrieval or top- accuracy for zero-shot object and attribute recognition. Further details on the datasets and evaluation protocols are provided Appendix A.
Fully-controlled, synthetic data. To study the effect of information imbal- ance on the modality gap and object bias, we built a fully-controllable dataset based on Morpho-MNIST (Castro et al., 2019), called Multi-modal Attributes and Digits (MAD). We used the following morphing or warping operations as latent factors (i.e., attributes) from Castro et al. (2019): thickness, swelling, fractures. We further added scaling, colors, and captions. Figure 2 and Ap- pendix B provide examples. To generate captions, we mapped the digit class and the other factors to words and chained them together in random order, e.g..
1
2
4
5
Figure 2: Exam-
ples from MAD.
a rrgeblle, We embedded each factor to single token. To study the effect of information imbalance, we varied the number of attributes within each caption while, importantly, keeping the images unchanged. For example, if we restrict each caption to one attribute (in addition to the always present object), above (full) caption becomes, e.g., 1-blue or 1 1-large. Model and training details are provided in Appendix B.
1-thickens i ng- swell 1. i ing-fraaturee
Experiments real data. To validate our hypothesis in realistic setting, we trained CLIP RN50 models on the image-text dataset CC12M (Changpinyo et al., 2021), and manipulated the captions. We provide further details in Section 6 and Appendix F.I.
3

---

Published as a conference paper at ICLR 2025
70
75
COCO
90
Dontact sizo
InoagoNor
accuracy
70
50
MS COCO
R&1
.00
InaggeNor
accurasyy
75
22
Large
0.5
0.5
1.0
0.6
0.5
0.5
0.0
0.5
0.6
RMG
RMG
Figure 3: Relation between modality gap (L2M & RMG, larger value- + larger gap) and down- stream performance for total of 98 contrastive VLMs pre-trained on medium- and large-scale datasets (each scatter point is a VLM). The plots indicate no to weak positive correlations between performance and modality gap (see the numbers in Table 1).
Table 1: Kendall's т rank correlation between downstream performance and various factors for models trained on medium and large datasets. denotes statistical significance (p < 0.05). Model, embedding, and dataset size correlate stronger with performance than the modality gap.
Dowastream task
Modality gap (L2M)
Modality gap (RMG)
Model
Embedding size
Dataset
size
size
MS COCO
10.16 (X) / 10148 (X) 0.083 (X) / -0.007 (X) -0.354 (2) 0 (V)
0.264 (X) /.662 (V) -0.129 (X) / 0.252 (V)
ImageNet
-0.088 (1) 0.28 -0.109 (X) 0. (
-0.5 (0.60 ( 0.318 (3) /0.66 -0.034 (X) /0.206 (V)
4 PARTING WITH FALSE INTUITIONS ABOUT THE MODALITY GAP
Liang et al. (2022) showed the existence of a modality gap; a geometric phenomenon of the shared embedding space of multi-modal models, such as contrastive VLMs. Specifically, they showed that the embeddings of each modality lie in completely separate regions. They proposed that the gap ex- ists at model initialization (cone effect) and that the contrastive loss preserves it. They proposed the L2-distance between the Means (L2M) to measure the gap: L2M == | Σi X; N EE vill, where x; is the i-th normmazll image embedding and y, the i-th text embedding.
Despite Liang et al.'s (2022) and subsequent work (Udandarao, 2022: Shi et al.. 2023; Zhang al., 2023), many questions about the modality gap remain unanswered so far: What is its impact on downstream performance? How does it manifest itself in the embeddings? We will answer these questions in the following: We discuss the relationship between the modality gap and downstream performance (Section 4.1) and find that only few embedding dimensions drive it (Section 4.2). In contrast to the cone hypothesis (Liang et al., 2022), the gap emerges even when we initialize with no gap and find evidence that the contrastive loss can narrow the gap (see Appendices F.4 and F.5). Thus, the cone effect can't 't be the main factor. What else leads to the emergence of the gap? We show in Section that an information imbalance is the main factor that leads to its emergence.
4.1 DOES A SMALLER MODALITY GAP LEAD TO BETTER PERFORMANCE?
The effect of the modality gap on downstream performance has been discussed controversially (So et al., 2023; Zhou et al., 2023; Liang et al., 2022; Zhang et al., 2023). We intuitively expect that smaller modality gap leads to better performance. However, do we observe this also in practice?
To answer this question, we compared the downstream performance and the modality gap across 98 contrastive VLMs, e.g., CLIP or SigLIP, pre-trained on various datasets such as LAION or WebLI. We evaluated the downstream performance retrieval yielded similar results) as well as on ImageNet zero-shot image classification. Using L2M to measure the gap and particulary to compare the gap across models comes with some limitations. For instance, 1.2M neglects whether an image-text pair matches, and it does not take the effectively used space into account; refer to Appendix D.1 for detailed discussion. To address these short- comings of L2M, we propose the Relative Modality Gap (RMG) measure:
MS COCO for image-to-text retrieval (text-to-image
1
Σ
d(x..
N
RMG+ ==
(1)
N
d(x;, x3) +
Σ d(y..,)) + N Σ d(x,,y))
2N(N-1)
4

---

Published as a conference paper at ICLR 2025
0.5
CLI WEE
CLIP VIT-B/16
MS COCO Ro
CLIP VTT-B/16
StyLIP WI3-003
Din 442
62.6
0.00
0.4
0.5
0.50
Image
50.0
RMG
0.25
0.50
diftcrescce of
0.3
0.0
0.5
0.2
0.2
SigLIP VIT-B/16
MS COCO Re1
SigLIP VIT-B/16
0.6
0.1
0.1
Din 78
8
RMG
100
3000
Emhhsdding dimensions (sonted)
Dim 122
Ablated dimensious
(a) Some dimensions have vastly different means.
(b) Separating dims.
(c) Removing dims.
Figure 4: Few embedding dimensions separate the modalities. Results MS-COCO. (a) We plot the absolute difference in the means of each embedding dimension between the modalities. Most dimensions have similar means for both modalities, but for some the differences are huge. (b) Pairs of these high difference dimensions can perfectly separate the modalities (we show the ones with largest mean for each modality). (c) Successive removal of embedding dimensions based on the sorting of embedding dimensions from (a) leads to sharp drop, followed by partial recovery of downstream performance, while the modality gap gradually closes (similar results for L2M). See Appendix D.3 for results on ImageNet and the plots in (b) with the largest two dimensions of (a).
where x,,y; are the i-th 12-normalized image or text embeddings, respectively, and d is some distance function (we used cosine dissimilarity scaled to [0,1]). Intuitively, the numerator measures the gap where it matters, i.e., for matching image-text pairs, and the denominator accounts for the effectively used space through an approximation using the intra-modality distances. We added the distances of matching image-text pairs to the denominator to scale RMG to [0, 1].
Figure and Table show that a larger modality gap counter-intuitively correlates with better down- stream performance. However, does this imply that larger modality gap leads to better perfor- mance? No, as Table 1 reveals, other factors such as model or embedding size have a stronger effect on performance. It appears that the negative impact of the modality gap is overshadowed by these other factors. Indeed, when we control for these factors, we find weak indications for the expected negative correlation for some of the pre-training datasets, i.e., smaller gap seems correlate with better performance (see Appendix D.2). Thus, the modality gap is a problem worth fighting.
A larger modality gap has mild positive correlation with downstream perfor- mance. However. there is no indication that larger modality gap leads to better performance: rather, it suggests the presence of common confounders (e.g., model size).
Takeaway 1:
4.2 FEW EMBEDDING DIMENSIONS DRIVE THE MODALITY GAP
To obtain better insights into the nature of the modality gap, we studied two questions: 1) Is the modality gap present in all dimensions only a subset thereof, and 2) does post-hoc closing of the modality gap improve downstream performance?
Do all embedding dimension contribute to the gap? To study the influence of each embedding dimension, we compared their means. Figure 4a shows that while most embedding dimensions have very similar means, interestingly, some have stark differences. When we plot the dimensions with the largest mean in each modality, we find that they suffice to perfectly separate the modalities (Figure 4b). Further, they have large variance within one modality but only negligible variance in the other. In summary, Figures 4a and 4c show that these dimensions are the main contributors.
Few embedding dimensions drive the modality gap. We find that two dimen- sions suffice to perfectly separate the modalities.
Takeaway 2:
Can we cluse the gap post-hoe? One could suspect that removing above embedding dimensions will close the modality gap and as result yield better performance. To test this, we successively
5

---

Published as a conference paper at ICLR 2025
Table 2: Dissimilarity of neighborhood orderings in the embedding space using normalized Kendall-T distance [0, 1]. Higher normalized Kendall-7 distances indicate that the ranking of neighbors differs more. We used three different ImageNet-Ile splits.
CIFAR-10
CIFAR-100
ImgNet-100-1
ImgNet-100-2
ImgNet-100-3
CLIP ViT-B/16
0,3399
0.4965
0.4975
0.5046 0.4965
0.5081
SigLIP ViT-B/16
0.5044
0.4981
0,5003
0.4987
MIT-Smies
attr accuaney
Dataset
MIT--Sutes
attr accuracy
Kendall'ss 74.5
Kendall's 78/1
20
mediun
20
Dalaset
10
10
10
0.02
0.04
0.06
0.08
0.02
0.03
20
40
40
60
60
UT-Zaappss arr aceuracy
Kehdaallss
Kendall's 39,0
UT-ZAppos attr accuracy
20
20
20
00
40
00
0.006
0,010
0.015
MS COCO
ImageNet accuracy
MOAD text
MOAD image
ReI
(a) Object bias vs. downstream performance.
(b) Object vs. attribute performance.
Figure 5: Object bias and performance on attribute tasks. (a) We find a bias towards objects (positive MOAD values) but no correlation with attribute performance. We attribute this to the (b) positive correlation between performance improvements on object tasks and attribute tasks.
ablated embedding dimensions (i.e., set their values to zero) based on the sorting from Figure 4a, re-normalized, and evaluated their downstream performance. Figure 4c shows that the modality gap and the downstream performance sharply decreases, followed by a partial recovery of the perfor- mance. The behavior on downstream performance is explained by the substantial change in cross- modal neighborhoods caused by the ablation and re-nommalization of the largest and thereby most influential embedding dimensions on the geometry of the embedding space. Refer to Appendix D.4 for an illustrative example. Thus, while removing the largest contributors closes the gap, it leads to degraded performance.
Similar to the above, other post-hoc approaches tested in previous work also did not lead to consis- tent improvements: Liang et al. (2022) show that computing the "modality gap vector" and closing the gap by adding this vector generally hurts performance. Similarly, for the ideal words approach of Trager et al. (2023) we find that simple shifting closes the gap, increases average similarities, but does not improve performance (refer to Appendix D.5 for details). But why can not just close the gap post-hoe with such translation approaches to improve the performance?
A key precondition of these translation approaches similar neighborhood relations across modal- ities. To test their similarity, we computed the mean embedding of each class in CIFAR-10, CIFAR- 100 (Krizhevsky al., 2009), and three ImageNet-100 splits (Hoffmann et al., 2022). We computed the Kendall-r distance normalized by the number of classes. Intuitively, it counts the percentage of bubble-sort swaps w.r.t. all possible swaps needed transform modality A's nearest neighbor list to match modality B's. Table 2 shows that the neighborhood orderings are dissimilar between the modalities. Hence, these simple transformations cannot improve performance.
Takeaway 3: Simple post-hoc approaches can close the modality gap but do not improve performance. One reason for this is that the modalities have different local neighborhoods.
Looking at the bigger picture, why does the modality gap emerge in the first place? In Section 6, we will show that an information imbalance between modalities is the main factor for its emergence.

---

Published as a conference paper at ICLR 2025
5 OBJECT BIAS IS A CAPTION PRESENCE BIAS
The hypothesis that contrastive VLMs are biased towards objects was based on their poorer perfor- mance non-object (e.g., attribute) compared to object tasks (Bravo al., 2023). However, this can be misleading, as an attribute-based task could just be more difficult than an object-based task.
Thus, study the bias towards objects, we propose a measure for object attribute bias, denoted as Matching Object Attribute Distance (MOAD). MOAD quantifies how well a model can distinguish matching from non-matching images (or texts) of objects o compared to attributes a. Matching im- ages (texts) show the same object or attribute, whereas non non-matching images (texts) show different objects or attributes. MOAD can be computed for each modality. For images, we define it as:
MOADang ang = 2,0
1
Σ objEO
(iim ahj - sim
Σ auEA
(sim an
sim
2
where O denotes the object classes, sim denotes the mean similarity between the L2- normalized image embeddings that entail the same object obj, sim the mean similarity be- tween different objects (i.e., the image does not entail the same object) and sim & siman are similarly defined for attributes A. Positive val- ues of MOAD indicate a bias towards objects, negative values bias towards attributes, and zero no bias. Refer to Appendix E.1 for the detailed definition. Intuitively, MOAD mea- sures whether the visible/present objects or at- tributes have a larger influence on the similarity of two images/captions. Note that MOAD is not specific to objects and attributes, we pro- vide a generic formulation, called Bias mea- sure via Relative Angle Change of Embeddings (BRACE). in Appendix E.2.
10"
Objects Attribustes
Text
10'
Always in caption
bias (MOAD)
0.6
Image
Counts
10a
10"
10a
10a
10
0.0
10 ou 60 00
Digit
Swellng
Praczure
Soaling
Thelesess
Calara
Object/Attribute ID
Always in caption
(a) Object and attribute (b) We can bias the model counts in LAION-2B. to any factor of choice.
Figure 6: Object bias caused by per-sample caption presence bias. (a) Object bias is not caused by the word frequency as attributes appear more frequently than objects. (b) We trained CLIP models on MAD and changed which factor is al- ways in the caption (i.e., caption presence bias). The models are biased towards whichever factor (e.g., color, thickness, etc.) that we made preva- lent in the captions.
Similar to the analysis on the modality gap in Section 4, first investigated the relation be- tween object bias and attribute performance. Figure 5a shows that the majority of contrastive VLMs exhibit bias towards objects (positive MOAD values), as expected. Notably, models trained on large-scale data are less biased towards objects (smaller positive values) compared to models trained on medium-scale data. However, we find no clear correlation between the bias towards objects and performance; especially for models trained on large-scale data.
Thus, reducing the object bias does not appear to affect attribute performance- but what does? Figure 5b shows medium-to-strong correlations between performance on object and attribute tasks. Thus, it seems that simply improving VLMs (without specific focus on attributes) also improves their performance attributes by-pruduct.
Takeaway 4: Contrastive VLMs trained on large-scale data tend to have a lower object bias than models trained on medium-scale. However, there no clear relation between object bias and attribute performance. This can be attributed to the observation that performance improvements on object tasks correlate with improvements on attribute tasks.
Is object bias explained by the global word frequencies of the dataset? One may suspect that the word frequency of the training dataset (i.e., prior distribution over words p( (word)) causes the bias towards objects. However, Figure 6a disproves this, as attributes are actually mentioned more often than objects in the LAION-2B's captions (Schuhmann et al., 2022) (we used object and attribute classes from Bravo et al. (2023)). Thus, we hypothesize that the bias towards objects arises from the per-sample prevalence of objects in natural language, i.e., humans tend to describe the most salient
7

---

Published as a conference paper at ICLR 2025
object(s) and typically only few of their attributes in caption. In other words, we propose that the conditional probability of words given an image pl n/word (image) is what leads to the bias.
To verify this, used MAD (Section 3) and redefined the prevalence of words in the caption. Specifically, we trained five contrastive VLMs for which we varied the prevalent factor by making always present in the caption and sampling one of the other factors randomly by chance. Figure 6b validates that each model is biased towards whichever factor was prevalent during its training (larger positive MOAD values). We also find that the bias is larger for the image encoder. This is expected, as it needs to match to the most likely caption, while the text encoder can simply encode the entire information (that information is likely always present in the image), as sketched in Figure 1 and outlined in detail in Section 6. We provide further experimental evidence in Appendix E.3.
Bias towards concepts, c.g., objects, is caused by their high probability of appearing in captions (given that said concept is present in image), rather than by their overall frequency in the dataset.
Takeaway 5:
Naturally, the question that arises is how we can reduce the object bias. We will address this in the next section.
6 INFORMATION IMBALANCE TRIGGERS MODALITY GAP AND OBJECT BIAS
The previous sections analyzed both the modality gap and bias towards objects. However, what causes their emergence in contrastive VLMs? In this section, we will show that an information imbalance in the data is the main cause of both. Information imbalance refers to unequal amount of information transmitted by the modalities, i.e., while an image contains a lot of information, its caption is typically incomplete; as illustrated in Figure 1. Specifically, it mostly only entails one or two of the most salient objects and a handful of other factors such as attributes these objects.
6.1 HOw DOES INFORMATION IMBALANCE CAUSE THE MODALITY GAP AND OBJECT BIAS?
The origin of the object bias. A consequence of the information imbalance in the data is that the encoders can hardly align their embeddings, as they have no way of knowing what information is available in the other modality. To achieve sufficient alignment (numerators in the CLIP loss (Equation 4)) despite the uncertainty from the information imbalance, the best both encoders can do (especially the image encoder) is to focus on the factors that are most likely present in the caption (or image)- typically abjects-while while using smaller scale or fewer dimensions for factors that are less likely to be present (e.g., attributes). By that, the average cosine similarity will be less affected when the unlikely factors not present. This leads to the caption presence bias discussed in Section 5, ie., a bias towards the most likely present words given the images.
The origin of the modality gap. We also hypothesize that the main factor for the emergence of the modality gap is an information imbalance between modalities in the data. This imbalance makes it difficult for the encoders to align their embeddings, as discussed above. As result, max- imizing alignment for matching image-text pairs (numerators of the CLIP loss (Equation 4)) be- comes challenging. In other words, the alignment term is bounded. With alignment being bounded (and optimized), the only way to further reduce the total loss is by maximizing uniformity (the denominators). Consequently, contrastive VLMs tend to focus more on increasing the distance of non-matching pairs, i.e., maximizing the uniformity. After all, the contrastive loss maximizes the similarity of matching image-text pairs relative to non-matching pairs. other words, VLMs com- promise on alignment, which is inherently limited by the information imbalance, to achieve higher uniformity. Since uniformity operates only across modalities, as in the CLIP loss, VLMs push the modalities apart, ultimately leading to the gap. We conjecture it does so by using few dimensions (e.g., the ones in Figures 4a and 4b) which substantially increase uniformity but have a small effect on alignment.
Experimental validation a fully-controllee synthetic setting. To validate that information imbalance triggers both the modality gap and the object bias, we manipulated information imbalance
For explanation, use the alignment and uniformity terms of the contrastive loss (e.g., CL.I loss Equation 4) defined by Wang & Isola (2020). Refer to Appendix C. for background.
8

---

Published as a conference paper at ICLR 2025
Bel. modally gop (KME)
0.05
0.30
Embeoding dimensinn
12
(MDAD)
0.0
Text: Image
0:30
UMAP dim
8:18
8.30
0.00
0.00
(11)
(111))
Attributes in caption
Attriburas in caption
Artributts caption
(a) Larger information imbalance (fewer attributes)
larger gap & smaller bias.
UNNAD AP din
Pull cagtion
Half caption
Quaater caption
1000
Spech
Spech
Epech
UMAP dim 0
(c) Larger information imbalance (shorter captions) - larger gap (real data).
(b) UMAP embeddings.
Figure 7: Increasing shared information between modalities (smaller information imbalance) improves the representations. (a) To study the influence of information imbalance between the modalities, we manipulated the number of attributes in the captions in MAD (the images are always affected by all attributes). As the amount of shared information between the modalities increases, the modality gap (I) and bias towards objects reduce (II), while downstream accuracy improves (III). We show additional results in Figure 13. (b) The contrastive loss is able to close the modality gap given full shared information (no information imbalance) between modalities, as illustrated by the UMAP embeddings (McInnes et al., 2018) at model initialization (top) and after training (bottom). (c) To verify our explanation also real data, we trained CLIP models on CC12M and created an information imbalance by dropping 1/2 or 3/4 of the captions. Similarly, we find that lower information imbalance leads to a smaller modality gap. Additional results are provided in Appendix F.I.
Entropy
3.0
Temp-
75
RMG
0.6
L2N
1.0
Frozcn toopp
50
2.5
Laurnabll
0.0
2.5
10.
Epoch
Epoch
Epoch
Epoch
Figure 8: A model trained with frozen temperature increases the modality gap more than model with trainable temperature to achieve a similar logit entropy.
in fully-controlled synthetic experimental setting. Specifically, we varied the number of attributes included the captions of MAD, while the object (digit) was always included. Importantly, we only manipulated the captions and left the images unchanged. The information imbalance is larger with fewer attributes and smaller with more attributes. Section 3 provides further experimental details.
Figure 7a shows that both the modality gap and object bias decrease balance. This validates our hypothesis. Notably, even when the modality gap is large at model initialization, contrastive VLMs can close in the full information setting (Figure 7a(I) and quali- tatively in Figure 7b). This is aligned with our hypothesis. In contrast, the cone effect hypothesis not consistent with our observation. Further, we find that the image encoders are more biased towards objects than the text encoders (Figure 5a, 6b and I3(II-VV)). This is expected, as the text encoders know what encode, whereas the image encoders must model the uncertainty over the captions" content. Finally, we find that performance improves as information imbalance decreases (Figure 7a(l))), and that our findings are consistent across different embedding dimensions.
reduce information im-
Experimental validation on real data. The main advantage of the synthetic setting above is that we have full control over the interventions and avoid confounding factors. To verify that our findings also apply to real data, we trained three CLIP models on CC12M, either with the "full" captions or omissions of 1/2 ("half") or 3/4 ("quarter") contiguous parts of them; see Appendix F. for details. The omissions increase the information imbalance. It is important to note that this also be viewed in reverse: the "full" captions represent enriched versions (i.e., less information imbalance) of the "half" or "quarter" captions, similar to work caption enrichment (Fan al., 2023), but

---

Published as a conference paper at ICLR 2025
without introducing hallucinations. Figure 7c confirms that the modality gap is smaller with lower information imbalance. Thus, our hypothesis (Section 6.1) also holds on real data.
Takeaway 6: An information imbalance between modalities leads to both modality gap and object bias. Reducing this imbalance decreases both the modality gap and object bias.
6.2 Is THE MODALITY GAP A BUG OR FEATURE?
Figure 7c shows that the validation losses for "full" and "half" captions are very similar at the end of training, but retrieval performance (R @ 1) is clearly higher for "full" captions. The lower per- formance for "half" captions suggests that the model does not align positives (w.r.. their negatives) as well and makes false retrievals more likely. However, why is this not reflected in the validation loss? A possible explanation is a different logit entropy, which directly influences the loss values. The entropy over the logits models the entropy (uncertainty) of the data.
To change the (global) entropy, contrastive VLMs have a learnable temperature parameter (r in the CLIP loss (Equation 4)). However, changing the modality gap may also change the entropy. For example, reducing the similarity between a pair of positives while keeping the negatives identical (increasing the gap), increases the entropy. Thus, changes the embeddings that increase the modal- ity gap also increase the entropy. Consequently, the training can control the entropy in the output logits (i.e., the uncertainty) in two ways: via the temperature or the embeddings (e.g., by changing the modality gap). One way to achieve this is by changing, few embeddings, as in Section 4.2. The ability to also change the entropy via the modality gap" equips the model with the flcxibility to change the entropy independently for each sample. In contrast, the temperature parameter only affects the entropy globally.
Experimental validation. To investigate whether CLIP changes the modality gap to change the entropy, we first trained CLIP on CC12M with the "full" captions (low information imbalance). We subsequently fine-tuned this model on the "quarter" captions (increased information imbalance, resulting in higher entropy) in two conditions: with learnable temperature (learn. temp.) or frozen (frozen temp.). Note that "learn. temp." "frozen temp." can only adjust the gap. Appendix F.2 provides further experimental details.
adjust both the temperature and modality gap, but
Following our explanation in Section 6.1 and the hypothesis that CLIP "increases the gap adjust the entropy", we expect low entropy and modality gap at the start of fine-tuning followed by a sharp increase. Importantly, we would expect that entropy is similar for both, "frozen temp" and "learnable temp." As consequence, the modality gap should be larger when the temperature is kept frozen. Indeed, we observe that "frozen temp." " increases the modality gap significantly more match the same entropy (Figure 8). Thus, the modality gap can be interpreted as a feature, not bug, as it adds flexibility to the model to change the entropy.
Takeaway 7: During training, models learn to estimate the entropy (uncertainty) of the data. For contrastive VLMs, changes to the embeddings that lead to a higher/smaller modality gap lead to a higher/lower entropy of the logits. This increases the model's flexibility in controlling logit entropy.
7 CONCLUSION
In this work, we showed that both, the modality gap and object bias of contrastive VLMs, are triggered by information imbalance between modalities. This information imbalance limits the achievable alignment in the embedding space. We showed that the effect of the modality gap on performance is typically overshadowed by other factors, and it is driven by only few embedding dimensions. We find that the modality gap is a symptom of changes to the embedding that lead to higher logit entropy. Further, confirmed that contrastive VLMs have bias towards objects, and could relate it to per-sample caption presence bias due to information imbalance. These insights will serve researchers and practitioners in understanding and modifying the learned representations of contrastive VLMs.
For simplicity, we refer to this as "incressing/decrrasingg the gap to increase/decrease the entropy". How- ever, do not claim causal relationship; the changes also just he measurable by the modality gap.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was funded by the German Federal Ministry for the Environment, Nature Con- servation, Nuclear Safety and Consumer Protection based on resolution of the German Bun- destag (67K12029A). the German Research Foundation (DFG) under grant numbers 417962828 and 499552394 (SFB 1597), and the Bosch Center for Artificial Intelligence.
We would like to thank Philipp Schröppel and Sudhanshu Mittal for helpful comments on the draft, Maria A. Bravo and Simon Ging for insightful discussions, and Simon Ging and Elias Kempf for preparing the CC12M and CC3M datasets.
REFERENCES
Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications. arXiv, 2021.
Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting VYT in Shape: Scaling Laws for Compute-Optimat Model Design. In NeurlPS, 2023.
Maria A. Bravo, Sudhanshu Mittal, Simon Ging, and Thomas Brox. Open-vocabulary Attribute Detection. In CVPR, 2023.
Justin Brody. On the Potential of CLIP for Compositional Logical Reasoning. In ICLP, 2023.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. COYO-700M: Image-Text Pair Dataset, 2022. URL https: //tp:::::://tiee httg//ttthhu/te// wow.a/b/.cmmaaaabbbaa..n. coyo- dataeet.....
Daniel C. Castro, Jeremy Tan, Bernhard Kainz, Ender Konukoglu, and Ben Glocker. Morpho- MNIST: Quantitative Assessment and Diagnostics for Representation Learning, JMLR, 2019.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing Web- Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts. In CVPR, 2021.
Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Y. Qiao, and Wenping Wang. CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP. In CVPR, 2023a.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Car- los Riquelme Ruiz, Andreas Peter Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. PaLI: Jointly-Scaled Multilingual Language-Image Model. In ICLR, 2023b.
Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO Captions: Data Collection and Evaluation Server. arXiv, 2015.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. CVPR, 2023.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a Similarity Metric Discriminatively, with Application to Face Verification. CVPR, 2005.
Guillaume Couairon, Matthijs Douze, Mathieu Cord, and Holger Schwenk. Embedding Arithmetic of Multimodal Queries for Image Retrieval. In CVPR, 2022.
Jonathan Crabbé, Pau Rodríguez, Vaishaal Shankar, Luca Zappella, and Amno Blaas. Robust multi- modal models have outlier features and encode more concepts. arXiv, 2023.

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR. 2020.
Abrar Fahim, Alex Murphy, and Alona Fyshe. Its Not Modality Gap: Characterizing and Address- ing the Contrastive Gap. arXiv, 2024.
Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving CLIP Training with Language Rewrites. In NeurlPS, 2023.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. DataComp: In search of the next generation of multimodal datasets. Datasets and Benchmarks Track @ NeurIPS, 2023.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. NeurlPS, 2021.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal Neurons in Artificial Neural Networks. Distill, 2021.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: new estimation principle for unnormalized statistical models. In AISTATS, 2010.
Kimia Hamidieh, Haoran Zhang, Thomas Hartvigsen, and Marzyeh Ghassemi. Identifying Implicit Social Biases in Vision-Language Models. Workshop@ @CLLR 2023.
Jun-Yi Hang and Min-Ling Zhang. Partial multi-label learning with probabilistic graphical disam- biguation. Advances in Neural Information Processing Systems, 36: 36:1339-1351, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
David T. Hoffmann, Nadine Behrmann, Juergen Gall, Thomas Brox, and Mehdi Noroozi. Ranking Info Noise Contrastive Estimation: Boosting Contrastive Learning via Ranked Positives. AAAI,, 2022.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP, 2021.
Phillip Isola, Joseph J. Lim, and Edward H. Adelson. Discovering States and Transformations in Image Collections. CVPR, 2015.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-Language Representation Learn- ing With Noisy Text Supervision. In ICML, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning Multiple Layers of Features from Tiny Images, 2009.
Yann LeCun. The MNIST database of handwritten digits. http://yann. lecun. com/exdb/mnist// 1998.
Xianhang Li, Zeyu Wang, and Cihang Xie. An Inverse Scaling Law for CLIP Training. In NeurIPS, 2023a.
Xianhang Li, Zeyu Wang, and Cihang Xie. Grounding Visual Illusions in Language: Do Vision- Language Models Perceive Illusions Like Humans? In EMLNP, 2023b.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. In NeurlPS, 2022.
12

---

Published as a conference paper at ICLR 2025
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In ECCV, 2014.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In CVPR, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In ICLR, 2019.
Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-CLIP: End-to-
End Multi-grained Contrastive Learning for Video-Text Retrieval. In International Conference Multimedia, 2022.
Joanna Materzyúska, Antonio Torralba, and David Bau. Disentangling visual and written concepts in CLIP. In CVPR. 2022.
Prasanna Mayilva Thaddäus Wiedemer, Evgenia Rusak, Matthias Bethge, and Wieland Bren- del. Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity? In ICLR, 2024.
Leland McInnes, John Healy, and James Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv,, 2018
Sachit Menon and Carl Vondrick. Visual Classification via Description from Large Language Mod- els. In ICLR, 2023.
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP. In NeurIPS, 2022.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Pre- dictive Coding. arXiv, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. In ICML, 2021.
Cyrus Rashtchian, Charles Herrmann, Chun-Sung Ferng, Ayan Chakrabarti, Dilip Krishnan, De- qing Sun, Da-Cheng Juan, and Andrew Tomkins. Substance Style: What Does Your Image Embedding Know? arXiv, 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet Large Scale Vi- sual Recognition Challenge. IJCV, 2015.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-SB: An open large-scale dataset for training next generation image-text models. In Datasets and Benchmarks Track@ NeurlPS, 20222.
Peiyang Shi, Michael Welle, Marten Björkman, and Danica Kragic. Towards understanding the modality gap in CLIP. In Workshop @ ICLR, 2023.
Alcksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does CLIP know about a red circle? Visual prompt engineering for VLMs. In ICCV, 2023.
Junhyuk So, Changdae Oh, Yongtack Lim, Hoyoon Byun, Minchul Shin, and Kyungwoo Song. Geodesic Multi-Modal Mixup for Robust Fine-Tuning In NewrIPS, 2023.
Kihyuk Sohn. Improved Deep Metric Learning with Multi-class N- N-pair Loss Objective. NeurlPS, 2016.
13

---

Published as a conference paper at ICLR 2025
Lukasz Struski, Adam Pardyl, Jacek Tabor, and Bartosz Zieliński. Propml: probability partial multi- label learning. 2023 IEEE 10th International Conference on Data Science and Advanced Analyties (DSAA), pp. 1-8, IEEE, 2023.
Quan Sun, Yuxin Fang. Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: Improved Training Techniques for CLIP at Scale. CVPR, 2023.
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The New Data in Multimedia Research. Communica- lions of the ACM, 2016.
Matthew Trager, Pramuditha Perera, Luca Zancato, Alessandro Achille, Parminder Bhatia, and Ste- fano Soatto. Linear Spaces of Meanings: Compositional Structures in Vision-Language Models. In ICCV, 2023.
Vishaal Udandarao. Understanding and Fixing the Modality Gap in Vision-Language Models, 2022. Master's 's thesis.
Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero- Soriano. A picture worth more than 77 text tokens: Evaluating clip-style models on dense captions. In CVPR, 2024.
Alexander Visheratin. MLLB-CLIP-train performant multilingual image retrieval model on a bud- get. arXiv, 2023.
Tongzhou Wang and Phillip Isola. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. ICML, 2020.
Chenyun Wu and Subhransu Maji. How well does CLIP understand texture? Workshop@ECCV, 2022.
Ming-Kun Xie and Sueng-Jun Huang. Partial Multi-Label Learning. In AAAI, 2018.
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li. Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP Data. In ICLR, 2024.
Yutaro Yamada, Yingtian Tang, and IIker Yildirim. When are Lemons Purple? The Concept Asso- ciation Bias of CLIP. In EMNLP, 2023.
Aron Yu and Kristen Grauman. Grained Visual Comparisons with Local Learning. In CVPR, 2014.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive Captioners are Image-Text Foundation Models. TMLR, 2022.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and Why Vision-Language Models Behave like Bags-Of-Worde and What to Do About It? In ICLR, 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre-Training. In ICCV, 2023.
Renrui Zhang, Ziyao Zeng, Ziyu Guo, and Yafeng Li. Can Language Understand Depth? In International Conference on Multimedia, 2022.
Yuhui Zhang, Jeff Z. HaoChen, Shih-Cheng Huang, Kuan-Chieh Wang, James Zou, and Serena Yeung. Diagnosing and Rectifying Vision Models using Language. In ICLR, 2023.
Chenliang Zhou, Fangcheng Zhong, and Cengiz Öztireli. CLIP-PAE: Projection-Augmentatiee Augmentation Embedding Extract Relevant Features for Disentangled, Interpretable and Controllable Text- Guided Face Manipulation. SIGGRAPH, 2023.
14

---

Published as a conference paper at ICLR 2025
A FURTHER DETAILS FOR THE ANALYSIS EXPERIMENTS OF PRE-TRAINED VISION-LANGUAGE MODELS
List of contrastive VLMs. For our large-scale analyses, considered a total of 112 contrastive VLMs trained across various datasets provided by OpenCLIP (Ilharco et al., 2021; Cherti et al., 2023)² before filtering (see below). We considered the following contrastive VLMs with the follow- ing characteristics:
CLIP variants: OpenAl's CLIP (Radford et al., 2021), OpenCLIP (Cherti et al., 2023), MetaCLIP (Xu et al., 2024), CLIP-A (Li et al.. 2023a), EVA-CLIP, EVA-02-CLIP (Sun et al., 2023), CoCa (Yu et al., 2022), NLLB-CLIP (Visheratin, 2023), SigLIP (Zhai et al., 2023; Alabdulmohsin et al., 2023).
Vision backbones: ResNet (He et al., 2016), ConvNeXt (Liu et al., 2022), or ViT (Doso-
vitskiy et al., 2020).
Pretraining datasets: OpenAl's proprietary (400 WeblmageTexx dataset (Radford et al., 2021), LAION-400 M, LAION-2 B, LAItN-Aesthetic (900 M) (Schuhmann et al., 2022), Merged-2 B (merge of 1.6 B samples from LAION-2 B and 0.4 B samples from COYO-700 M (Byeon et al., 2022)) (Sun et al., 2023), WebL. (Chen et al., 2023b), So- 400 M (Alabdulmohsin al., 2023), MetaCLIP (400 M) (Xu et al., 2024), Conceptual 12 M (Changpinyo et al., 2021). YFCC (15 M) (Thomee et al., 2016), CommonPool-s (max. 12.8 M; refer to Table 3 of Gadre et al. (2023) for the details of filtering), CommonPool (max. 128 M). CommonPool-I (max. 1.28 B), CommonPool-xt (max. 12.8 B) (Gadre et al., 2023), and DataComp-s (1.4 DataComp-n (14 DataComp-I (140 M), DataComp- xl/DataComp-II (1 B) (Gadre et al., 2023).
OpenCLIP contains a large variety of models, some of which, e.g., have poor performance or are redundant. Thus, we filtered models using the following criteria:
We removed models that were specifically fine-tuned, e.g., on MS COCO.
When there were models that were trained the same dataset but saved at different check- points, we selected the latest checkpoint.
We excluded models that achieve < 5% R@1 1 on MS COCO, under the assumption that such low scores indicate some sort of failure that is likely to distort the analyses.
After filtering, we used 98 (out of 112) contrastive VLMs for our analyses.
Dataset details. We ran evaluations MS COCO (Lin et al., 2014; Chen al., 2015), ImageNet (Russakovsky et al., 2015), MIT-States (Isola et al., 2015), and UT-Zappos (Yu & Grau- man, 2014). The datasets comprise 25000 (5000 images with 5 captions each), 50000, 12995, or 2914 test samples, respectively. MS COCO and ImageNet are standard datasets for evaluation of retrieval object recognition performance, respectively. We used the standard evaluation protocols to image-to-text retrieval performance (text-to-image retrieval yielded similar results) or computed top-1 accuracy. MIT-States consists of 245 objects and 115 adjectives (attributes), while UT-Zappos consists of 12 shoe types with 16 fine fine-graieed states (~ attributes). For both datasets, we assume that we do not know the object of a respective image and only want to find the adjective or fine- grained state. We considered this classification problem, following previous work (Trager et al., 2023). Note that these datasets implicitly assume that the adjectives are mutually exclusive per im- age. However, this may not be necessarily true, as multiple adjectives or fine-grained states may be present in the image.
Evaluation prompts. For MS COCO, we prepended the prompt "a photo of to the descrip- tion of each image following Radford et al. (2021). For ImageNet, we used the prompts "a photo of {obi)" (Radford et al., 2021). For both MIT-States and UT-Zappos, used the prompts "an image of a {attz} object".
hhttps://g wwi.ithub..oo www.tttubbcooooo.b /m1 foundat cns/open_ct
15

---

Published as a conference paper at ICLR 2025
Thickness
Swelinn
Base
Digit cizas
Fradtures
Scaling
Color
Figure 9: Causal graph of MAD.
B FURTHER DETAILS FOR THE EXPERIMENTS ON THE MULTI-MODAL ATTRIBUTES AND DIGITS DATASET
Multi-modal Attributes and Digits (MAD). Our dataset Multi-modal Attributes and Digits (MAD) based on the MNIST (LeCun, 1998) variation Morpho-MNIST (Castro al., 2019). The causal graph of the bata-generating process of MAD is depicted in Figure 9, We used the following words for digits (0, 9). altering image thickness (thickening, thinning, no thickthinni ng),,) swelling (swel ing, no swell swelling), fractures (fracture, no fracture), scaling arge, smal ) and color (gray, red, green, blue, cyan, magenta, yellow). Thus, we have 10 digits and 16 attributes. Figure 10 provides examples of image-text pairs of MAD.
In our experiments, we investigated information imbalance in the captions by restricting the number of attributes present within each caption. We provide examples below, where we sequentially remove the amount of information within the captions, i.c., fewer latent factors (attributes) present in the caption:
Full information setting (i.c., digit & all five attributes)
5-large-yelloowoo - htg.o//oeeooologooeaerrrrreeetar
Partial information I (i.... & four attributes)
swellinggthicceennnggeeeeeeaaal
5-large-yellow-no swelling-fracture
Partial information setting (i.e., digit & three attributes)
yeallowwseellllngthhhcheennnsss
-swellinnechiiccennnnn---errr
5-large-yellow-no swelling
Partial information setting III (i.c., digit & attributes)
yelloowswelling:
swellingchhiceeinng--
5-large-yellow
Partial information setting IV (i.e., digit & attribute)
16

---

Published as a conference paper at ICLR 2025
0 thickening no swelling fracture large magenta
8 thickthinning swelling fracture small magenta
8 thickening swelling no fracture large magenta
thickthinning no swelling fracture small gray
thickening no swelling fracture small gray
thickthinning no swelling fracture large magenta
9
0 thickening swelling fracture small green
thickening swelling fracture small gray
thinning swelling fracture small red
thickening swelling fracture small green
no thickthinning swelling no fracture small red
thickening swelling no fracture small yellow
2 no thickthinning no swelling fracture large blue
no thickthinning swelling fracture large magenta
thickening swelling no fracture small green
thinning swelling fracture small green
no thickthinning swelling no fracture small red
thickening swelling fracture large red
5
a
2
9 thickening swelling fracture small red
6 thickthinning no swelling fracture large magenta
2 thickening swelling fracture large blue
0 thickthinning no swelling no fracture small magenta
thickening swelling fracture small red
thickening swelling fracture small red
6
Figure 10: Example images with corresponding caption of our MAD dataset. For example, the image in the first and first column shows the digit 4 without altering the thickness, swelling applied, with fracture augmentation, scaled down and the color gray. Note that the words of the captions shuffled (not in this figure for convenience).
yellow-9
swelling-6
5-large
Note that while all the latent factors, i.e., digit and all five attributes, still affect the generated image, the caption may only provide partial information, i.e., attributes are missing from the caption but not the image.
Model details for MAD experiments. We used small CLIP models. Specifically, the ViT-based vision backbone comprises 6 layers, each with dimensionality d of 256 and [d/64] heads.
17

---

Published as a conference paper at ICLR 2025
The transformer-base language backbone also comprises 6 layers, each with dimensionality of 256 and heads. We set the patch size to 7 and context length to 8. The vocabulary consists of 28 words, i..,, all the words for digits (10) and attributes (16), as well as start and end symbol (2).
Training details. We trained all models with a batch size of 128 for 200 epochs with a learning rate warm-up period of 5 epochs. We used A (Loshchilov & Hutter, 2019) optimizer with cosine annealing learning rate schedule (Loshchilov & Hutter, 2019). We always selected the best performing learning rate across learning rates {5 10-4, 10-5, 10-5) each trained with random seeds. The best learning rate was selected by comparing average of the average ideal word accuracy and average zero-shot accuracy on all attributes and the class label. For all of our results, we report the average over 3 random seeds.
C BACKGROUND ON THE CLIP LOSS
Contrastive representation learning (Chopra et al., 2005; Gutmann & Hyvärinen, 2010; Sohn, 2016; Oord et al., 2018) leverages paired inputs as weak supervision signal. The basic idea is to learn rep- resentations in shared representation space that are as similar possible for "positive/matching pairs, while as dissimilar as possible for "negativennon-matching pairs. A popular choice for con- trastive learning approaches is the InfoNCE objective (Oord et al., 2018):
1 log
fy.)))
C(fafa))
N exp(
where far fy are two encoders for the inputs {x]1N,, [y]ıNN and t is the scalar temperature. Wang & Isola (Wang & Isola, 2020) define two components of the loss:
Alignment (numerator): matching pairs should be close, i.c., aligned.
Uniformity (denominator): representations should be roughly uniformly distributed on the unit hypersphere.
For multi-modal contrastive representation learning, it is popular to use symmetric version of above objective (Radford et al., 2021):
N
exp (x)(Tfx(x)) ff(x)))
N
(x)
CCUp
log
log
2N
2N
1=1
Σ
(xp(rf
(x)
We can define the alignment and uniformity term similarly for the CLIP loss. It is important to observe that the repulsive forces of the CLIP loss only act across the modalities but not within them.
D EXTENDED DETAILS FOR SECTION 4
D. FURTHER DETAILS ON RELATIVE MODALITY GAP MEASURE (RMG)
Limitations of L2M. L2M has been initially proposed modality gap distance by Liang et al. (2022). However, it has several limitations that we will discuss below:
(a) L2M does not account for difference the effectively used embedding space.
(b) L2M takes a distributional instead of a per-sample view.
(c) The L.2 norm can sensitive to outlier embedding dimensions.
(d) The L2 norm is an unintuitive distance measure when all points lie on the surface of sphere.
Regarding (a): note that different models can use different amounts the unit hypersphere, as the contrastive loss accounts for relative cosine similarities. Here, the similarity is always relative
18

---

Published as a conference paper at ICLR 2025
Table When controlling for the dataset choice, the modality gap seems correlate negatively with performance (i.e., the smaller the gap the better the performance) in most cases. We report the Kendall's rank correlations between RMG (top) or L2M (bottom) and the respective downstream performance. Additionally, we report the p-values of the correlations. Note, that the p-values are sometimes large due to the small number of models we are left with for each dataset (see parenthesis) and therefore results should be taken with a grain of salt.
RMG
Evaluation
DataComp- (10)
LAION-400M (7)
LAION-2B (20)
OpenAl (12)
WebLI (9)
datasct
MS COCO
0.244 (p=0.375)
-0.619 (p=0.069)
-0.263 (p=0. 114)
-0.143 (p=0.544) 0.302 (p=0.195)
-0.611 (p=0.02)
ImageNet
0.644 (p=0.011)
-0.714 (p=0.03)
-0.168 (p=0.319)
-0.5 (p=0.073)
L.MM
MS COCO
0,333 (p=0.221)
-0.619 (p=0.069)
-0.189 (p=0.259)
0.238 (p=0.304)
-0.5 (p=0.073)
ImageNet
0.733 (p=0.003)
-0.714 (p=0.03) -0.063 (p=0.738)
0.397 (p=0.077)
-0.333 (p=0.257)
the similarity to the negative samples. Consequently, models can use a varying degree of the unit hypersphere and, thus, L.2 distances can have a different meaning. For example, consider two models that have the same L2M but the first model uses the entire unit hypersphere, while the second model only uses a small fraction of it. While L2M suggests that the modality gap distance is the same, the actual gap the first model is significantly smaller, since the average distances between the samples are larger.
Regarding (b): intuition suggests that matching image-text pairs should be close but non mat-mat pairs can (and should) be large. L2M considers the distance of the means of all pairs. This can lead to misleading results. An illustrative (but unlikely) example is the case of two distributions that occupy exactly the same region of the hypersphere, but rotated by n-degree (i.e., they are very misaligned). Clearly, there exists a gap between the modalities but L2M does not indicate it.
Lastly for (c) (d), the L2 norm can be sensitive to embedding dimensions that exhibit vast differ- ences and are unintuitive since CL.PP's embeddings are on the unit hypersphere. For instance, our discovered most modality-separrate embedding dimensions qualify for this.
Relative Modality Gap (RMG). As a remedy to above outlined limitations, we proposed a Relative Modality Gap (RMG) measure in the main text (see Equation 1). RMG computes the distances between matching image-text pairs instead of the means to address (b). Since density estimation in high-dimensional spaces is difficult, we used the mean distances between all samples per modality as rough approximation to address (a). Finally, we used cosine similarities instead of the L2 norm address (c) and (d).
D.2 FURTHER ANALYSIS ON THE REI-ATIONSHIP OF THE MODALITY GAP AND DOWNSTREAM MANFORMANCE
Controlling for the dataset choice. To mitigate the impact of the dataset, we studied the rela- tionship between the modality gap and downstream performance for each dataset individually. For this, only datasets with least seven models were considered: DataComp- (10 models), LAION- 400M (7), LAION-2B (20), OpenAl's CLIP dataset (12), and WebLI (9).
Table shows that when controlling for dataset choice, we observe the expected negative rank correlation (i.e., smaller modality gap correlates with better downstream performance) in seven and six out of ten cases for RMG or 1.2M, respectively. However, we do not find statistical significance for these correlations in most cases. Note that this is most likely due to the small number of models used.
Through further inspection, we made interesting observation for models that were pre-trained on DataComp-13 1B and evaluated on MS COCO: the positive correlation we found can be attributed to the differences in training of CLIP and CLIP-A models (Li et al., 2023a). CLIP-A models utilize an inverse scaling law, where larger image/text encoders are trained on fewer image/text tokens, i.c., image resizing masking and shorter context length for the text tokens (e.g., through truncation or
19

---

Published as a conference paper at ICLR 2025
Table 4: Data filtering makes the modality gap more narrow. We report RMG (Ist, 2nd table) and L2M (3rd, 4th) on MS COCO (Ist, 3rd) and ImageNet (2nd, 4th). For all filtering strategies, the modality gap narrows after filtering the unfiltered dataset.
MS COCO, RMG
CommonPool scale
filtering strategy
none
basic
text-based
basee
LAION-2B-based
CLIP-based
small
0.533
0.462
0,461
0.445
0.447
0.461
medium
0.537
0.499
0,507
0.494
0.446
0.487
large 0.603
0.577
0.593
0.587
0.511
0.544
xlarge
0,586
n/a
n/a
n/a
0.531
0.543
ImageNet, RMG
small
0.598
0.502
0.492
0,461
0.468
0.485
medium
0.595
0.547
0.553
0.525
0.478
0.523
large
0.651
0.628
0.628
0.625
0.557
0.584
xlarge
0.647
n/a
n/a
n/a
0.593
0.603
MS COCO, L2M
small
0.733
0.508
0.515
0.356
0.346
0.519
medium
0.773
0.678
0.702
0.677
0.482
0.639
large
0.898
0.86
0.892
0.877
0.742
0.8
xlarge
0.875
n/a
n/a
n/a
0.849
0.862
ImageNet, L2M
small
0.805
0.525
0.516
0.3
0.3
0.491
medium
0.837
0.735
0.745
0.701
0.514
0,681
large
0.947
0.915
0.92
0.911
0.792
0.843
xlarge
0.961
n/a
n/a
n/a
0.935
0.95
masking). According to our explanations from Section 6, the use of fewer (text) tokens leads to larger modality gap due to the greater information imbalance. Interestingly, despite this, downstream performance still improves on common benchmarks, as shown by Li et al. (2023a). When consider- do find negative rank correlations: -0.667 (p=33.3%)
ing CLIP and CLIP-A models separately, for CLIP models and -0.6 (p=0.136) for CLIP-A models for RMG. When using L2M instead, we find 0.0 (p=1.0) for CLIP models and -0.6 (p=0.136) for CLIP-A models.
Impact of data filtering. There are two ways to reduce information imbalance from a data perspec- tive: (1) caption enrichment and (2) filtering out high information imbalance image-text pairs, i.e., pairs with high uncertainty between them. We have shown extensive results for the former in Sec- tion 6 and Appendix F. To study the latter, we used CLIP models trained on CommonPool (Gadre et al., 2023), a benchmark to assess data filtering approaches.
Table 4 shows that data filtering also effective in reducing the modality gap.
D.3 IMAGENET RESULTS FOR FIGURE 4
Figure 11 shows similar results on ImageNet to the ones shown the main text in Figure 4 on MS COCO.
Figure shows that the modalities are also well separable when we select embedding dimensions based the largest mean differences; similar Figures 4b and lc.
20

---

Published as a conference paper at ICLR 2025
CLIP VIT-B/16
CLIP VIT-B/16
0.60
Din 440
0.50
Image
LangeNet
sccuray
68.
RMG
0.4
0:23
62.0
difieencc
0.3
0.05
0.50
0.73
60.0
0,5
SigLIP VIT-B/16
0.2
SigLIP VIT-B/16
0.1
Din 78
0.5
Imugeto
Accunsoy
RMG
0.3
0.0
100
3000
5001
Emhhidiing dimensions (sonte)
Dim 122
Ablated dimeusions
(a) Some dimensions have vastly different means.
(b) Separating dims.
(c) Removing dims.
Figure 11: Few embedding dimensions separate the modalities. Results ImageNet. (a) We plot the absolute difference in the means of each embedding dimension between the modalities. Most dimensions have similar means for both modalities, but for some the differences huge. (b) Pairs of these high difference dimensions can perfectly separate the modalities. (c) Successive removal of embedding dimensions based on the sorting of embedding dimensions from (a) leads to sharp drop, followed by a partial recovery of downstream performance, while the modality gap gradually closes (similar results for L2M).
MS COCO
ImageNet
Din 462
Tel
0.00
0.25
0.00
0.25
0.50
0.75
0.2
0.4
0.5
0.5
0.50
750
Dim 635
12.23
0.00
0.0
11.2
0.4
-0.5
-0.4
-1.
0.0
Figure 12: Pairs based the largest mean differences. In the main paper, show the di- mensions with largest means. Note that using the largest mean difference also leads to separable dimensions. Top: OpenAl's CLIP ViT-B/16, bottom: SigI.P ViT-B/16. Left: MS COCO, right: Im- ageNet. In Figures 4b and 1lc, we selected the embedding dimension of each modality, as each of the embedding dimensions most dominate the direction of the embedding vector of each modality,
D.4 EXAMPLE FOR THE EFFECT OF THE EMBEDDING DIMENSION REMOVAL
Consider the following example: let x 8, 0.6, 0.7, 0.3)" be an image embedding, and y [5, 0.1,, 0.035, 0.02 and y' [5, 1.5, 0.7, 0.45]" be the matching non-matching text embed- ding, respectively. We have cosine similarities d of d(x.y y) = 0.995 and d(x,y') y') = 0.975. After ablating the first dimension, have cosine similarities of 0.822 and 0.917 and hence the image-text alignment flipped. Finally, it flips back after ablating also the second dimension.
D.5 CLOSING THE GAP FOR IDEAL WORDS
We showed that we can close the modality gap post-hoc by successive removal of embedding di- mensions in Section 4.2 but performance did not improve. Similarly, Liang et al. (2022) shifted the embeddings by the modality gap vector
N
N Y;
A gup
X;
Σ
(5)
N
N
1=0
1=0
21

---

Published as a conference paper at ICLR 2025
where X,, Ys are the i-th L2-normalized image text embeddings, respectively. Simple shifting closed the modality but also did not improve performance." 4 But what's 's the effect on ideal words (Trager et al., 2023)?
Ideal words. Ideal words (Trager et al., 2023) can be computed as follows:
Ys,
Σ
B.y.' - Yo
(6)
with
where is component of the factored set of concepts 2 E 2 21 x X Zx each with possible
concept values, y is the L2-normalized text embedding, azs, = mi B= = =II and yo is defined as
1
Yo = 32 EyEY
In words, ideal words are computed by marginalizing out all other factors j * i. Intuitively, we can understand the ideal word yı, for component 2; as the direction for that concept (e.g., "green") in the embedding space. Note that yo is simply the average text, which is used to remove the "textness" from the embedding vector.
Extension to ideal images. For our analysis, propose to extend the notion of ideal words to ideal images by simply replacing the text embeddings, y, by the image embeddings, x, and utilize datasets with object-atrribute information (MIT-States, UT-Zappos) to control z.
Results. Ideally, the ideal words and ideal images should align, i.c., the directions the embedding space have similar meaning. The cosine similarities between ideal words and ideal images in Table 5 show that ideal words and ideal images are roughly aligned. Once we correct for the modality gap, they well aligned. However, Table 5 shows that the shifting has negligible performance. This indicates that the removal of the main elements of the modality gap improves the cosine similarities (by removing the parts that lead low cosine similarity) but it does not fix the underlying problem, i.e., the modality gap does not limit performance of pre-trained contrastive VLMs.
effect
E EXTENDED DETAILS FOR SECTION
E. FURTHER DETAILS ON OUR OBJECT ATTRIBUTE BIAS METRIC MOAD
For readability and to provide a better intuition about the metric we substituted parts with more informative variables in the main text. The definition of MOAD without substitution is given by:
MOAD img
Σ
xTx
Σ
XTx,
2 dEO
N1
N₂ XXXXxxxXXX
X.,x, EX.
(#)
(8)
Σ
x
Σ
xfx;
2| aEA
N X,X, EX. (
N
XXX
where F; denotes the normalized image embeddings. N₁ Ni..... N4 are normalization factors given
by N₂ = xx,2 - [Xol. N₂ = |X, x X .ol. N₂ = |X, 2 IXa and N₂ IXa x X .al X., X-o, Xa, are all image embeddings x that (not) entail the object or attribute E A. respectively.
*Only when they optimized the shift length, they could improve performance.
22

---

Published as a conference paper at ICLR 2025
Table 5: Shifting ideal words makes them more similar ideal images but does not improve performance. We report the cosine similarities between ideal words and ideal images, and top-1 test accuracy on attributes and objects for CLIP ViT-B/I6 (Ist, 2nd table) and SigLIP ViT-B/16 (3rd, 4th) on MIT-States (Ist, 3rd) and UT-Zappos (2nd, 4th). Note that we subtracted yo for "ideal words" and for "ideal words with A gup we did not. The reason for this (approximate) equivalence between subtracting yo and adding Xo to the modality gap vector A gap
CLIP VIT B/16 on MIT-States
cosine similarity between
top-1 test accuracy
Variant
ideal words & ideal images
attribute
object
real words
n/a
18.65
44.69
ideal words
0.388
19.74
46.87
ideal words with A
0.858
19.70
46.84
CLIP ViT B/16 on UT-Zappos
real words ideal words
n/a 0.381
15.92
60.54
21.83
62.46
ideal words with A
0.898
21.76
62.42
SigLIP VIT B/16 MIT-States
real words ideal words
n/a 0.456
21.82
50.23
25.81
52.54
ideal words with A
0.858
25.73
52.66
SigLIP ViT B/16 on UT-Zappos
real words ideal words
n/a
27.14
73.27
0.439
35.38
72.07
ideal words with A
0.870
35.14
72.00
E.2 GENERAL FORMULATION FOR BIAS MEASURE VIA RELATIVE ANGLE CHANGE OF EMBEDDINGS (BRACE)
MOAD is instantiation of our novel metric Bias measure via Relative Angle Change of Embed- dings (BRACE), BRACE is defined follows:
BRACEing
Σ
x
Σ
Xxx,
B
N
N₂ EXXxxxx
BEB
X..N, EX, in-
(9)
Σ Xx (f
Xxx
1
Σ
xf'x,
C|
N₂
N
CEC
XXX
where z, denotes the normalized image embeddings, Ni.... N₂ are normalization factors given by N₂ N₂ Xx, x X N₂ x and N₂ X--el X X X X-c are all image embeddings x that (not) entail some group of concepts B (c.g.. objects O) or some other group of concepts C' (e.g., attributes A), respectively.
23

---

Published as a conference paper at ICLR 2025
Table 6: We define four conditions with various prior distributions over words p(word). Throughout all conditions, digits are treated as the objects. In condition A, we ensure that each word appears the same number of times the dataset, meaning p(word) uniform. Note that p(wordaigii (image)
1 and p(word-digii image) < 1. In condition D, all factors appear in the caption, meaning p( word image) = Conditions and interpolate between conditions and D. Note that the appearance probability (appearance counts in the dataset) of attributes p( word) is higher than for objects for conditions B, C, and D.
resulting p(word) given p(word|image)
object
attribute
Condition
digit
thickthinning
swelling
fracture
scaling
color
A
0.1
0.1
0.1
0.1
0.1
0.1
B
0.1
0.177
0.233
0.233
0.233
0.114
C
0.1
0.255
0.366
0.366
0.366
0.129
0.1
0.33
0.5
0.5
0.5
0.143
Table 7: Even though attributes are globally more frequent (see Table 6), the object bias per- sists. We observe object bias, even though p(word) is uniform (for A) attributes fre- quently present in the captions (for B-D).
MOAD img
Condition
emb size 12
emb size 15
emb size 18
A
0.41
0.40
0.34
B
0.34
0.25
0.31
C
0.29
0.30
0.29
0.14
0.17
0.22
E.3 ADDITIONAL EXPERIMENT ON WHERRE THE OBJECT BIAS STEMS FROM
Experimental setup. Complementary to the experiment Figure 5, we conducted experiments for which controlled ((wor)) and p( p(word image) on MAD. We considered four conditions A-D (see Table 6) that varied p( word) for objects and attributes.
Results. Table 7 shows that the models still exhibit bias towards digits, though attributes occur more frequently globally (in the dateset). This further confirms that the prior distribution over words p( worr) cannot be the main factor for the emergence the bias towards objects.
F EXTENDED DETAILS FOR SECTION
F.1 CC12M TRAINING INFORMATION IMBALANCE CAUSES MODALITY GAP EXPERIMENTS
Experimental setup. We largely followed the training setup of Ilharco et (2021). common for CC12M, we used ResNet-50 image encoder. We trained for 30 epochs, with each epoch having 9263104 104 image-text samples. We used a global batch size of 1660 and a learning rate of le-3 with 1000 steps of warm-up. For all other settings, we used the default values of Ilharco et al. (2021).
We trained CLIP models in 3 conditions:
full caption: using the complete captions.
half' caption: randomly dropped the first second half of a caption.
quarter caption: we only kept randomly selected quarter of the caption.
Note, that we do not sample words randomly, but select a contiguous string of words from the caption. All hyperparameters are identical between the conditions.
24

---

Published as a conference paper at ICLR 2025
modality gep (RMG)
Embedding dimension
Obj bios text (MOAD)
0.25
Accuracy digits
95
0.25
90
0.20
0.15
0.10
75
0.05
0.05
70
0.00
(I)
Attributes in caption
(III)
Attributes in caption
(V)
Attributes in caption
0.6
Obj. bias img. (MOAD)
100
Modality gap (L2M)
Accuracy attributes
0.5
90
0.4
0.3
0.3
0.2
0.2
0.1
0.0
0.1
(I1)
(IV)
(VI)*
3
Attributes in caption
Attributes in caption
Attributes caption
Figure 13: Complete version of Figure 7a. To study the influence information imbalance be- tween the modalities, control the number of attributes present in the captions (the image is always affected by all attributes) in MAD. As the amount of information shared between the modal- ities increases, the modality gap (I-II) and bias towards objects reduces (III-IV), while downstream accuracy improves (V-VI).
Validation loos
Temperaruue
Fall catiinn
RMG
1.30
30
coptiio
1.03
0.40
28
Epoch
Epoch
Epach
Figure 14: Complete version of Figure 7c, Training with "full" captions leads to smaller modality gap (RMG, L2I), lower validation loss (though, "full" and "half" are very similar the end of training), and higher temperature (i.e., more peaky distribution/less uncertainty).
Additional results. In addition to Figure 7c in the main paper, Figure 14 shows the I. 1.2-instance (I.21) modality gap metric and temperature. 1.21 could be alternative RMG, which compro- mise between RMG and L2M. In contrast to L2M, it computes the distance only where it matters, i... between the matching pairs. However, in comparison RMG, does not take into account the fraction of the embedding space used by the model. L2I is given by
N
121 =
||x: yill
(10)
N
where x, is the i-th 12-normalized image embedding and y; the i-th text embedding. Note that in contrast to RMG, L2I misses the drop of the relative gap (approx. epoch 25 in Figure 14), as the distance between the positives keeps increasing.
Figure 15 shows results for different vision encoder (ViT-S in (a)), an alternative information- dropping scheme ceentence-dropingg in (b)), and another dataset (CC3M in (c)). For all these vari- ations, we find that the modality gap is smaller with less information imbalance ("full" caption) similar to our findings in Figures 7c and 14.
F.2 FURTHER EXPERIMENTAL DETAILS FOR SECTION 6.2
We used the pre-trained "full" caption model from Appendix F.I. Next, we fine-tuned this model using the same hyperparameters, but with "quarter" captions, i.e., keeping contiguous quarter of
25

---

Published as a conference paper at ICLR 2025
Validaaion loss
Full caption
20
Quarter caption
Epoch
Epoch
(a) ViT-S vision encoder.
0.7
Valichtion loss
12N
0.6
40
Full caption
Quarter caption
0.5
0.40
Epoch
Epoch
Epoch
Epoch
(b) Dropping sentences instcad of words.
Validation loees
11.0
R&1
M
Full caption
Quarter caption
0.45
Epoch
Epach
Epoch
Epoch
(c) Other dataset: CC3M.
Further evidence for the effect of information imbalance the modality gap on real data. This figure supports the findings of Figures 7c and 14 with experiments using (a) different vision encoder backbone, (b) sentence-wise information-droppin training on different dataset (note that CC3M is not subset of CC12M) In all settings, "full" captions (i.e., less information imbalance) also leads to a smaller modality gap (RMG, L2M).
Figure 15:
scheme, and (c)
Perfect pairs
Imperfect pairs
0.5
dim
0.0
+
y2
-0.5
11.0
0.5
0.0
0.5
1.0
1.0
0.55
0.0
0.5
1.0
dim
dim
Figure 16: Visualization of the global minima of the contrastive loss under perfect or imperfect pairs in our toy 2D example. For perfect pairs (left), matching pairs are aligned and uniformly distributed. For imperfect pairs (right), the global minimum exhibits modality gap. Here, x; denotes i-th sample from modality 1, e.g., images, and y, i-th sample from modality 2, e.g., text.
the words in the captions. For the "frozen temperature" model we freeze the temperature to the value leamee in pre-training (i.e., 800)
F.3 CAN THE MODALITY GAP EVEN APPEAR IN THE GLOBAL OPTIMUM?
We show below that the modality gap can even be present in the global optimum under certain data properties. Under perfect image-text pairs, is straightforward to see that the global minimum the CLIP loss does not exhibit modality gap. Le.,, all matching image- mmage-text pairs perfectly aligned and the pairs are uniformly distributed on the unit hypersphere. But what happens when alignment bounded, e.g., due to miscaptioning, focus other factors, missing information?
26

---

Published as a conference paper at ICLR 2025
Table Effeet of information imbalance in the "idealized" experimental setting of Fahim et al. (2024). It is apparent that information imbalance has a large impact on the modality gap.
information imbalance
training time
L2M
RMG
I21
after initialization
0.0
0.007
1.0
X
0.058
0.006
1.0
after 1200 epochs
0.265 (+0.207)
0.149 (+0.143)
1.0 (:+0.0)
To illustrate the effect of such bounded alignment on the final image and text embeddings, we designed 2D toy example. We generated two sets of points on the unit circle and directly opti- mized their positions. The points represent the embeddings of both modalities: {x1,x2} } = X and {y1, v2) Y, where Y1; X2, ified which point of set X matches to which point in set Y. For the perfect matching setting, we considered the following pairs:
{aa, b/² a² + 82 1; a b [J, Further, we spec-
(1), Y2)3
(11)
Trivially, the global minimum (up to rotations) is: X1 yı [1,0]", x2 Y2 [-1, 0,"; see Figure 16 for a visualization.
However, what happens when the alignment is bounded, e.g., due to mismatches? For image-text pairs, this can happen if a human annotator miscaptions an image. could also stem from differing focuses among human annotators on distinct aspects of the image. We considered the following pairs:
{x),, (x), (x2,32), V2), (2), (x2,
(12)
mismatches
where we need the additional matching pairs (1) and (x2. to avoid the degenerated global minimum X1 = Y1 = X2 = Y2.
To search for the globally optimal embeddings for the points in Equation 12, we ran grid search with an angular resolution of 6 We found the following global minimum (again up to rotations): X1 [1, 0] y1 /cos 276", sin 276"]T X2 = [cos 42", sin 42- y2 [cos 126", sin 126"]T. see Figure 16 for a visualization. It is apparent that the global minimum exhibits a modality gap.
We recognize that identification and removal of the mismatches in Equation 12 may be straight- forward in practice. However, our goal is to illustrate the impact of such in a simplistic setting to provide intuition on the behavior of the contrastive multi-modal loss.
F.4 IS THE GAP PRIMARILY CAUSED BY THE CONTRASTIVE LOSS INSTEAD OF THE INFORMATION IMBALANCE?
Recently, Fahim et al. (2024) proposed that the modality gap is inherent to the two-encoder strastive loss. However, how does this relate to our explanation that the modality gap stems from an information imbalance the data (Section 6)? To answer this, we replicated their "idealized experi- ment" (with minor differences that are highlighted below) and show that information imbalance has a substantially larger effect than the "contrastive gap".
Experimental setup. Following Fahim et al. (2024), we trained the CLIP model on image-image pairs. Note that this is in contrast to standard CLIP models that are trained on image-text pairs. However, it removes the differences that may stem from modality-specifie properties. Further, since we can use the identical image, also eliminates mismatches and, more importantly, information imbalance. To train the CLIP model, we replaced the text encoder with copy of the image encoder. Fahim et al. (2024) initially closed the modality gap using a translation vector (i.c., similar to the post-hoc approach tested by Liang al. (2022)). While this closes the gap in terms of L2, it does not close the gap in terms of RMG, as the cone size might still different and representations might be rotated (or have completely different neighborhood relations). To address these issues, closed the gap at initialization by simply initializing both image encoders with the same weights.
We considered two settings:
27

---

Published as a conference paper at ICLR 2025
Table 9: The contrastive loss almost closes the gap. We directly optimize the embeddings.
Step
Loss
RMG
7.3789
0.4987
1000
0.0060
0,0061
2000
0.0014
0.0060
Without information imbalance: we ran experiments as described above. This is equiva- lent (except for the different initialization) to the setting of Fahim et al. (2024).
With information imbalance: we randomly resized and cropped (using the default param- eters) the second image half of the time and the other half leave the image untouched (i.e., no information imbalance). Otherwise, we ran experiments as described above.
Note that evaluation was always conducted without information imbalance.
Training details. We followed the experimental setting of Fahim et al. (2024). We used 2048 randomly selected images from the MS COCO validation set (they used the training set). We also trained the CLIP models with batch size of 64 for 1200 epochs. Both image encoders use the ViT-B/32 architecture with a 512-dimensional embedding dimensionality. We used learning rate of le-4 with AdamW (with weight decay of 0.1, 31 0.9, 82 0.98, and 1e-6) and cosine annealing learning rate scheduling. The maximum temperature was set to 100.0. We repeated trainings for five random seeds.
Results. Table 8 shows that we can replicate the result of Fahim et al. (2024): even though the model initialized without modality gap and achieves perfect performance, there is a small gap after training. However, we also find that an information imbalance leads to substantially larger modality gaps measured with RMG (same results for L2M). This validates that information imbalance the main driving factor for the modality gap.
F.5 CAN THE CONTRASTIVE LOSS CLOSE THE MODALITY GAP?
To further understand whether the contrastive loss is capable to close the modality gap, we created simplified example and stripped away all unnecessary factors, i.c., the encoder networks, similar to Appendix F.3. We randomly drew 1000 L2-normalized image embeddings and their matching text embeddings with 8 dimensions each. We directly optimized the embeddings using the CLIP loss. We trained for 2000 steps with learning rate of 0.01 using Adam.
Table 9 shows that the contrastive loss can indeed narrow the gap and does not preserve it.
F.6 CAN THE MODALITY GAP REDUCED IN PRE-TRAINED MODELS?
To show that the modality gap also be reduced for pre-trained contrastive VLMs, fine-tuned OpenAl's CLIP ViT-B/16 (Radford et al., 2021) and SigLIP ViT-B/16 (Zhai et al.. 2023) on the Densely Captioned Images (DCI) dataset (Urbanek al., 2024). DCI consists of 7805 dense and aighle-aligned image-text pairs. Thus, DCI has reduced information imbalance and, consequently, the modality gap should reduce as we fine-tune.
Training details. We fine-tuned both pre-trained VLMs for ten epochs with global batch size of 256 (distributed across four NVIDIA RTX 2080 GPUs) with warm-up of 100 steps, and learning rate of le-5 with Adam W (with weight decay of 0. 1, 31 = 0.9, 32 = 0.09, and e = le-6) and cosine annealing learning rate scheduling. The maximum temperature was kept at 100.0.
Results. Table 10 shows that training on the image-text pairs of DCI indeed reduces the modality gap, as expected by our arguments Section 6. Note that fine-tuning led slightly worse zero- shot transfer accuracy on ImageNet. This is not in conflict with our claims, other factors such as the distributional shift between the fine-tuning and test datasets play a larger role for performance. For example, we do not expect better performance when fine-tuning on image-ter text flower dataset and then evaluate on an image-text animal dataset. We suspect that DCI similarly may not be best suited for such object-centric benchmarks (ImageNet & MS COCO).
28

---

Published as a conference paper at ICLR 2025
MS COCO
CLIP ViT-B/16
SigLIP ViT-B/16
before fine-tuning
after fine-tuning
before fine-tuning
after fine-tuning
L2M
0.816
0.700
1.046
0.958
RMG
0.572
0.499
0.623
0.562
R@1
52.84
59.24
67.68
67.58
ImageNet
L2M
0.864
0.747
1,116
1.04
RMG
0.606
0.531
0.683
0.63
Top-1 accuracy
66.75
65.13
75.61
73.87
Table 10: Fine-tuning on the image-text pairs of DCI (Urbanek al,, 2024) reduces the modality gap (lower L2M RMG).
G LIMITATIONS
We validated our main hypotheses on small- (MAD, CC3M) and pediumm-sized (CC12M) datasets. While additional experiments on large-scale datasets like LAION would valuable, they are be- yond our computational resources, unfortunately. Nevertheless, our analysis of pre-trained large- scale CL.IP models aligns with our findings on smaller datasets, suggesting consistency across scales.
Additionally, our method of reducing the information imbalance on real data (keeping only a con- tiguous sequence of words) may not be optimal. For instance, dropping 50% words should, on average, remove 50% of information, but the variance may be large. A more refined method could produce even clearer results.
Lastly, the results in Tables and are correlation-buse based.. Due to the small number of comparable models, the findings in Table 3 lack statistical significance. Expanding our current analysis would require prohibitively large compute resources, which we unfortunately have not.
H CONNECTION TO PARTIAL MULTI-LABEL LEARNING
Information imbalance highlights the learning difficulties that arise from incomplete partial in- formation. In our case, a modality gap and object bias. This issue is also present in more classical multi-label learning. Specifically, the field of partial multi-label learning (Xie & Huang, 2018: Struski et al., 2023; Hang & Zhang, 2023) handles scenarios where an unknown number of ground truth labels missing while irrelevant adulterated ground truth labels are provided.
In this work, we showed that CLIP-like models face analogous challenges that stem from infor- mation imbalance. For example, information missing from the captions, captions entail mis- matches. We believe that future work could study the similarities and potentially transfer ideas from partial lulti-label learning to CLIP-like models, or vice versa.
29


Reviews:
[{"review_id": "MyCbWVhJg0", "reviewer": "Reviewer_MYFm", "summary": "This paper presents a comprehensive study of contrastive VLMs, particularly examining why they excel at object recognition but struggle with attribute detection. The paper investigates how the modality gap and object bias affect downstream performance, and why these phenomena emerge.\nThe study supports these conclusions through extensive experiments using various models (particularly CLIP ViT-B/16 and SigLIP ViT-B/16) and datasets.", "questions": "- For Takeaway 3, can you elaborate what \"the modalities have different local neighborhoods\" mean?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I've reviewed this paper and noticed many improvements from the previous version:\n- Takeaway 1 is now much clearer. The section effectively demonstrates how common confounding factors affect results, and shows that when they control for these factors, a smaller modality gap indeed leads to better performance in downstream tasks.\n- The authors have introduced a new metric called RMG, which successfully addresses the limitations of L2M.\n- Takeaway 2 is interesting, and it leads nicely into Takeaway 3, which shows that we can actually reduce the modality gap using post-hoc methods.\n- Overall, this is a well-written study that thoroughly examines both the modality gap and object bias in contrastive vision-language models.", "weaknesses": "- I think most of weaknesses that have been seen in the previous submission was addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1FReUEcDb7", "reviewer": "Reviewer_qe7i", "summary": "This paper extensively investigates the modality gap.\nDifferent from the previous works, which hypothesize the modality gap is caused by the cone effect at model initialization or object biases, the authors analyze the modality gap from a modality information imbalance perspective.\nThey hypothesize the information imbalance between the two modality data (i.e., image and text) causes the modality gap and object bias, and demonstrate the hypothesis through extensive empirical studies.\nIn addition, they provide an analysis that how models lead to the modality gap with imbalanced image-text data.\nSummarized 7 takeaways and corresponding analysis provide a novel insight into the modality gap research.", "questions": "**[Q1] L261**\n- I think \"with the largest mean\" in L261 should be \"with the largest mean differences.\"\n\n**[Q2] Caption setup in Section F.1**\n- The authors made the half and quarter captions by randomly dropping the part of a caption.\nThis approach may cause incompleteness in the caption itself rather than information imbalance.\nIs there a strategy to verify the quality of the generated captions?\nI think the paper would be better if there is an analysis using metrics that could verify that half/quarter captions are grammatically perfect but only contain different amounts of information.\n\n**[Q3] Reverse process in Section 6.1**\n- The authors provide the experimental validation by training the model with \"full\" captions and fine-tuning with \"half\" and \"quarter\" captions. It is reasonable and makes sense. But can we expect the same effect on the reverse process?\nIn other words, can the performance of the pretrained VLMs (e.g. CLIP, SigLIP in this paper), which have a large modality gap, be improved, while reducing the modality gap by fine-tuning on an enriched dataset?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "**[S1]** This paper introduced a new perspective on the modality gap, providing rich insight to the researchers and practitioners.\n\n**[S2]** The paper is well-written and extensive experimental analysis is sufficient to validate their hypothesis and argument.\n\n**[S3]** Proposed metrics to measure the modality gap (i.e., RMG) and object/attribute biases (i.e., MOAD) are well designed to intuitively compare the corresponding components.\n\n**[S4]** The provided appendix is ​​sufficient to follow the details and experimental setting of the paper, enabling a better understanding.", "weaknesses": "**[W1] Mismatch in the level of information imbalance between synthetic data and real data settings**\n- The authors provide experimental validations on fully-controlled synthetic data and real data.\nWith synthetic data, information imbalance is defined based on the number of attributes, making it reasonable and sufficient to validate their hypothesis.\nHowever, half/quarter captions of real data are derived by randomly dropping the part of captions.\nI wonder if this strategy can hold the same hypothesis as in synthetic data. Please see Q2 below.\n\n**[W2] The lack of evidence on actual pretrained VLMs**\n- This paper suggests that the modality gap or the object bias can be reduced by data filtering or caption enrichment, and the performance can also be improved. However, there is a lack of analysis on real pretrained VLMs (e.g. pretrained CLIP and SigLIP) with large modality gaps. See Q3 below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x2SAr8fPgK", "reviewer": "Reviewer_ZSLu", "summary": "- This is an analysis paper, investigating the modality gap and object bias in VLMs. The authors argue that information imbalance is the root cause of both phenomena.\n- A larger modality gap correlates superficially with better performance, but this is due to confounding factors like model size. Controlling for these shows that a smaller gap is beneficial.\n- The authors introduce MOAD, a metric to measure object bias, and conduct experiments on synthetic and real-world datasets.\n- They find that the modality gap is primarily driven by a few embedding dimensions and linked to entropy.\n- Authors suggest that the model leverages the modality gap as a feature to manage prediction uncertainty.", "questions": "Can you provide examples of caption enrichment strategies that could effectively reduce the information imbalance?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Clear and concise introduction of the problem, motivation, and takeaways.\n- MOAD metric provides quantifiable measure of object bias.\n- Well-designed experiments show compelling evidence for information imbalance hypothesis. Good coverage of 98 VLMs (e.g., CLIP and SigLIP).\n- The proposed connection between the modality gap and entropy is intriguing", "weaknesses": "- There could be more analysis on real-world datasets to help solidify the claims.\n- [Minor] There could be more in-depth discussion on the limitations of the findings.\n- [Minor] There could be more explanation of mitigating the information imbalance.\n- [Nitpick] Typo: \"Extend details for Section 5\" -> \"Extended details for Section 5\")", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O12mb1Ydu1", "reviewer": "Reviewer_6t26", "summary": "The paper investigates two potential causes of the inferior performance of contrastive VLMs on downstream attribute recognition (compared to the often good downstream object recognition performance), namely, modality gap and object bias. The paper suggests that both modality gap and object bias are not directly related to the inferior performance problem, though they claim that with better controls modality gap seems to somewhat related.\nFinally, the paper does controlled experiments on small-scale dataset to confirm that information imbalance (i.e., the concepts that are present in image but not in the corresponding captions) in a dataset is strongly correlated with modality gap and object bias. Decreasing information imbalance seems to give better downstream attribute recognition performance.", "questions": "1. I think detailed contents on modality gap and object bias can be put into supplementary materials, since these details might diverge into information that doesn't help readers grasp the core of the main claim.\n2. The author might consider adding more experiments (more model architectures, more datasets, and presumably a better way to control the number of attributes) to show that RMG also decreases with less information imbalance.\n3. The authors should consider removing the claim “when we control for these factors, we observe the expected negative correlation, i.e., a smaller gap seems to correlate with better performance” or providing more substantial evidence.\n4. The author might consider converting most of the question-answering patterns in the paper to statements.\n5. The field of \"Partial Multi-Label Learning\" seems to be relevant to this paper (as another way to solve the information imbalance). I wonder if the authors are aware of this and if this will give a better solution than trainable temperature.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper provides in-depth investigation of the relationship between the inferior performance of contrastive VLMs on downstream attribute recognition and two potential causes (modality gap and object bias). They show that the relation is not substantial (though they claim that with better control the relation on modality gap exists, but I am not personally convinced given their evidence). This is informative for people studying these two factors. Importantly, they demonstrate that information imbalance might be a more important (causal) factor to investigate.\n2. They give evidence for reducing the information imbalance leads to better attribute recognition performance, as a solution to this problem on contrastive VLMs. They also provide a way to do this by using trainable temperature.\n3. The experiments presented in the paper are in general well-controlled.", "weaknesses": "1. To me, the major claim in the paper is that information imbalance causes both modality gap and object bias, and that even though these two are not substantially correlated with attribute recognition performance, decreasing information imbalance mitigates them and seemingly increase the overall attribute recognition performance. However, the paper spends most of the content to demonstrate details on modality gap and object bias, many of which are overly detailed. For example: in section 4.2, the authors talk about many attempts to naively close the modality gap, all of which fails to keep the model performance. I think contents like this can be briefly summarised and the details can be put into supplementary materials, since these details might diverge into information that doesn't help readers grasp the core of the main claim.\n2. The major claim is interesting and informative. However, the evidence of it is mainly done on the simple synthesised dataset MAD. The authors give evidence on real-world datasets in Figure7 (c), but the evidence on RMG is much weaker (if significant at all) compared to that with MAD. Since this is evidence for the major claim, the author might consider adding more experiments (more model architectures, more datasets, and presumably a better way to control the number of attributes) for this.\n3. The authors claim that “when we control for these factors, we observe the expected negative correlation, i.e., a smaller gap seems to correlate with better performance” (L245). However, the data they provided cannot support this claim, even when treating the significance values loosely. Some training datasets (e.g., DataComp-1B, OpenAI) clearly give positive correlation. The authors should consider removing this claim or providing more substantial evidence.\n4. The overall presentation of the paper is not ideal, in that many “questions” (e.g., “Is object bias explained by the global word frequencies of the dataset?”) can be turned into statements. The question-answering patterns in the writing highlight the motivations. However, in this paper there are so many of them, and the text devoted for completing this pattern probably does not worth it. The author might consider using more statements instead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)