Closing the Modality Gap Aligns Group-Wise Semantics
Authors: Eleonora Grassucci
Giordano Cicchetti
Emanuele Frasca
Aurelio Uncini
Danilo Comminiello
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Learning
Representation Alignment
Modality Gap
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We prove that closing the modality gap, while irrelevant for instance-wise tasks, significantly enhances performance in group-wise tasks and we propose a combination of novel losses to do so.

Abstract: In multimodal learning, CLIP has been recognized as the \textit{de facto} method for learning a shared latent space across multiple modalities, placing similar representations close to each other and moving them away from dissimilar ones. Although CLIP-based losses effectively align modalities at the semantic level, the resulting latent spaces often remain only partially shared, revealing a structural mismatch known as the modality gap. While the necessity of addressing this phenomenon remains debated, particularly given its limited impact on instance-wise tasks (e.g., retrieval), we prove that its influence is more pronounced in group-level tasks (e.g., clustering). To support this claim, we introduce a novel method designed to consistently reduce this discrepancy in two-modal settings, with a straightforward extension to the general $n$-modal case. Through our extensive evaluation, we prove our novel insight: while reducing the gap provides only marginal or inconsistent improvements in traditional instance-wise tasks, it significantly enhances group-wise tasks. These findings may reshape our understanding of the modality gap, highlighting its key role in improving performance on tasks requiring semantic grouping.

Published as a conference paper at ICLR 2026
CLOSING THE MODALITY GAP ALIGNS GROUP-WISE SEMANTICS
Eleonora Grassucci, Giordano Cicchetti, Emanuele Frasca, Aurelio Uncini, Danilo Comminiello Department of Information Engineering, Electronics, and Telecommunications
Sapienza University of Rome
(name) ) (surname) @enirooaa it
ABSTRACT
In multimodal learning, CLIP has been recogmized as the de facto method for learn- ing shared latent space across multiple modalities, placing similar representations close to each other and moving them away from dissimilar ones. Although CLIP- based losses effectively align modalities at the semantic level, the resulting latent spaces often remain only partially shared, revealing structural mismatch known as the modality gap. While the necessity of addressing this phenomenon remains debated, particularly given its limited impact instance-wise tasks (e.g., retrieval), we prove that its influence instead strongly pronounced in group-level tasks (e.g., clustering). To support this claim, we introduce a novel method designed consistently reduce this discrepancy in two-modal settings, with straightforward extension to the general n-modal case. Through our extensive evaluation, we demonstrate our novel insight: while reducing the gap provides only marginal or in- consistent improvements in traditional instance-wise tasks, significantly enhances group-wise tasks. These findings may reshape our understanding of the modality gap, highlighting its key role in improving performance on tasks requiring semantic grouping. Code available at: https: //ttthub.../ttt//// com. ispanm///a/mmm. aapann/MMdGap ModGap
INTRODUCTION
The canonical goal of contrastive representation Iearn- ing to learn information from the original data by embedding semantically similar data points nearby and dissimilar ones far apart. Multimodal research has inherited this goal, assuming that a video, its caption, and its soundtrack should share the same neighborhood in joint latent space. Unfortunately, the multimodal latent space shows different behav- ior, and representations tend instead to preserve their modality cluster, hindering the semantic alignment. This phenomenon, known the modality gap Liang et al. (2022), is prevalent in all multimodal models grounded the conventional and widespread con- trastive InfoNCE loss function van den Oord et al. (2018) like CLIP Radford al. (20IT). Before train- ing, samples from the same modality initially cluster together due to different random model weights ini- tialization, forming distinct modality-speciffe groups. Unfortunately, these clusters persist even after train- ing, resulting in sparse and fragmented latent space.
37.5
V-Measure (Clustering)
R@1 1 (Retrieval)
35.0
CLIP Gap Value
30.0
27.5
25.0
22.5
0.2
0.4
0.6
0.8
1.0
Figure 1: Reducing the gap consistently im- proves clustering metries, while leaving un- affected retrieval ones. On the contrary, in- creasing the gap downgrades the V-Measure, bringing no improvements in R@ In CLIP, the gap results in very poor clustering perfor- mance due the latent space fragmentation.
In recent works, the modality gap has been studied and partially addressed for image and text pairs Es lami de Melo (2025); Yaras et al. (2025); ) /Fhim
al. (2024); Schrodi al. 2025); Mistretta al.
(2025 Concurrently, other works are instead

---

Published as a conference paper at ICLR 2026
accepting the gap, letting the space it is while focusing on hyperbolic similarities Ramasinghe et al. (2024). However, the reasons for mitigating or untouching such a gap are not well-grounde and the advantages in common downstream tasks such retrieval are inconsistent throughout the literature. 2025) argues that closing the gap may improve the retrieval performance in downstream tasks, while Schrodi et al. (2025) found that larger modality gap has mild positive correlation with downstream performance. Moreover, all these approaches draw conclusions focusing only on the Iwo-modal case (mainly image and text), without advancing cues in the case of multiple modalities.
Yaras et al.
Rather than following conventional views, we reconsider the modality gap in the context of group- wise tasks. We provide evidence that the modality gap is irrelevant for instance- wise tasks, such as retrieval, which depend on relative rankings, but strongly impacts multimodal group-wise tasks, such as clustering, which rely on absolute distances among representations of multimodal data in the latent space. Specifically, we show that closing the modality gap reduces the within-group scatter, leading to more coherent semantic groupings and better clustering performance, while leaving retrieval rankings mostly unaffected, it is clear from Fig. [ objective function that explicitly encourages the alignment of matching pairs while maintaining a uniform and expressive latent structure. We show that method effectively reduces the modality gap in both bimodal and trimodal benchmarks, improves clustering performance by several points in V-Measure, and maintains or slightly increases retrieval accuracy. These results confirm that the modality gap is not a harmless artifact, but a central factor shaping the geometry of the multimodal latent space. Crucially, we show that can be effectively minimized through simple objective, enabling better semantic organization without compromising instance-leve precision.
To address this, we propose novel and effective
Our main contributions can be summarized as follows.
We demonstrate that the modality gap, while irrelevant to instance-wise tasks such as retrieval, directly impacts group-wise semantics reflecting in tasks, like clustering, that require explicitly closing the gap.
We introduce novel objective that combines true-pair alignment and centroid-based unifor- mity to effectively close the modality gap. Our method scales to multiple modalities and requires architectural changes post-hoc corrections.
Across both bimodal and trimodal benchmarks, our approach reduces the gap while im- proving clustering and preserving retrieval performance, confirming its effectiveness for multimodal representation learning.
2 RELATED WORK
Multimodal Learning. Starting from CLIP Radford et al. 202 I) several multimodal models have
been developed for two modalities like CLAP Elizalde et al. (2023) CLIP4Clip Luo et al. (2021). Lately, the same InfoNCE loss van den Oord et al. (2018) has been extended to multiple modalities in ImageBind Girdhar et al. (223) VAST Chen et al. (203b). More recently, novel approaches have been proposed for multimodal learning to avoid the cosine similarity loss, namely GRAM Cicchetti al. 2025) and Symile Saporta et al. (2024).
Modality Gap. The modality gap has been observed for the first time by Liang et al. 2022). and then studied mainly for the CLIP model Wu et al. (2023); Fahim et (2024); Shi et al. (2023) for
generic image and text pairs Mistretta et al. (2025): Yaras et al. (2025) ). Schrodi et al. (025)) Wu et al. (2023); Zhang et al. (2023). These works provide theoretical justification for the gap and propose to mitigate the gap by fixing the temperature Yaras et al. (025)) by applying post-hoc translations in the latent space Liang al. (2022): Schrodi al. (2025 J or by sharing the transformer encoder and the projection layer in the vision and language encoders Eslami & de Melo (025). In any case, each of these methods studied the modality gap the case of two modalities, without advancing clues on the case of three more modalities.
Effect of Temperature in Contrastive Learning. The temperature T in the InfoNCE loss van den Oord et al. 2013 8) is recognized among the primary knobs for steering what features contrastive learner captures. SimCLR Chen et al. (2020) noted that smaller sharpens the softmax and boosts instance-level retrieval accuracy. Wang & Isola (2000) proved that 7 trades off alignment against uniformity, while higher 7 instead favors cluster or class structure. Gradient-based analyses later showed that effectively controls the penalty on hard negatives, with low values concentrating
2

---

Published as a conference paper at ICLR 2026
gradients the most confusable samples Wang & Liu (2020) Building this view, Kukleva et al. (2023) introduced a schedule between high-r (group- wise) and low-7 finstancewwise) phases. Most recently, Dinu al. (2025) showed that increasing compresses embeddings, lowering intrinsic dimensionality while preserving task performance, suggesting a second-order role for t in model size and deployment efficiency.
3 WHY CLOSING THE GAP?
3.1 PRELIMINARIES
We consider training batch B = {(x], x(.,,x
where each sample is observed across M modalities, indexed by m {1, 2, ..... M }. For each modality m. the encoder fm x - R maps the imput - into a d-dimensional embedding vector "." = fx((x"") ), that further normalized to obtain unitary norm vector. We use sim(z,"" zz) to denote the cosine similarity between embeddings from modality and n, i... sim(2 23} (z." 27))
Given pair of modalities (m, n) with o we define the InfoNCE loss for modality pair as:
exp (simm(2" (isimm"",,)). 22)/T)
((m-t)) = N
log
1
exp (sim(z (aim(z,22))/1 .2))/))
where 7 > 0 the temperature parameter controlling the softness of the distribution over negatives. The total contrastive loss defined by averaging the two directions of the InfoNCE loss, L(m-+n) and C(n-tm), This is the standard loss used in CLIP Radford et al. (2021) and its variants for different modalities. The temperature т modulates the contribution of hard negatives Kukleva et al. (202)). A low temperature penalizes harder negatives (i.e., those with high similarity to the anchor). favoring instance- discrimination. In contrast, a higher temperature distributes gradients more evenly across all negatives, which typically promotes the emergence of semantic clusters and benefits group- wise reasoning tasks Dinu et al. (2025). In multimodal learning, different approaches have been proposed to handle the temperature effect including learning it during training fixing at predefined values. Throughout the paper, we refer to methods with the standard InfoNCE loss with learnable temperature as CLIP (LTT) and with fixed temperature as CLIP (FT).
3.2
UNDERSTANDING THE MODALITY GAP
A gap among modalities exists the initialization phase, where different encoders initialized with random weights represent data in different narrow cones the shared latent space) Liang et al. (2022). Nevertheless, the gap persists contrastive learning dynamics have been recognized to be the root cause of the gap, regardless of the initialization. Therefore, even though the final learned space is somewhat semantically aligned, positive pairs are decoupled and very distant, as Fig. X in the Appendix shows. As demonstrated by Shi al. (2023): Cicchetti al. (2025), the traditional CLIP loss function easily gets stuck in local minima, in which positive pairs are somewhat matched but far from each other, fostering the modality gap.
during the entire contrastive training phase. What is more, such
In detail, previous works show that the conventional CLIP loss function is composed of two terms, each with specific objectives Wang Isola (2020); Shi al. (2023): first term tries align positive pairs, while the second one tries to spread away non-matching pairs. In practice. these two terms provide opposite contributions, resulting in balanced and opposite forces. Therefore, models easily end up in local minima, avoiding the gap closure while allowing the representations of the two modalities to align with each other in "semantic stripes". These semantic stripes allow however good performance in retrieval tasks since positive pairs have higher cosine similarity with respect to non- non-matching pairs, even though such similarity far from the ideal 1.0, as we show in Section 5 Therefore, representations of matching pairs do not lie in the same portion of the latent space and are instead quite far from each other, severely limiting the expressiveness and the group of the latent space.
group-wise wise alignment
3

---

Published as a conference paper at ICLR 2026
Following Liang et al. (2022) measure such gap between two generic modalities and measure the effective Euclidean distance between the centroids of each modality:
GaPm,n m. = ||c" c²Il>
where eMx = Σ effectively aligned in the latent space. Therefore, we further adopt the mean cosine similarity true pairs metric, defined
N i= "," Even though the gap is zero, this does not imply that the embeddings
N
Cos TP M,,
2 m
2
N
This metric measures how much the normalized matching pairs are near each other in the latent space. Intuitively, the closer to 1.0, the smaller the angle is between them, and the closer the matching pairs lie in the latent space, being more aligned.
3.3 ALIGNING GROUP-WISH SEMANTICS
While the modality gap does necessarily disrupt instance-wise semantics, which depend on relative similarity rankings, it may prevent effective structuring of the latent space for group-wise objectives such clustering. Let us formalize the reason starting from the InfoNCE loss in equation Such a loss directly optimizes the relative ordering between the similarity of the true pair sim (z." z and those of negatives sim(2," 2z), * satisfied whenever:
The condition for retrieval success (e.g., Recall @ K) is
sim 2, 22.) > max (sim(2"",2.) Vj f
Thus, as long as InfoNCE ensures this inequality, the relative ranking is preserved, regardless of whether absolute similarity values are close to 1.0 far from it. Since instance-wise tasks like retrieval depend only on relative ordering and not on the absolute placement of embeddings, they are insensitive to the modality gap.
For group-wise tasks, let us build the centroid of semantic class comprising two modalities m and in the following way:
#1 2 ((z. 8) (z. + +5)
(5)
where the modalities are shifted according a constant vector 8. representing the gap impact between the two modalities. Note that. in the case of 0 gap, the formula trivially resolves in 1ta 2 (z," + +z)). The expected within-class scatter for a generic modality (m) decomposes as
E, [||z"" - 1
-
E. [|z."' E, 2 1|8112
(6)
with the gap term 8 summed the expectation since it does depend the semanties of class since it is orthogonal to the span of semantic vectors and therefore constant for each of them, as proven by Zhang et al. (2023). Therefore, enlarging the gap uniformly inflates every semantic cluster, degrading homogeneity and completeness, while shrinking the gap tightens such clusters.
Intuitive consequence. The modality gap leaves rank-based, instance- decisions mostly un- touched, as they are the primary objective the InfoNCE loss, but widens absolute distances and thereby harms any objective that optimizes within/between-cluste geometry. Retrieval is therefore almost flat across the gap value, whereas clustering (and other group-wise tasks) benefit from driving the gap to zero, shown in Fig. [ For instance, suppose retrieve cat image caption, the sufficient condition is that the caption remains the single most similar text, irrespective of its absolute cosine value. Instead, if the goal is forming a "cat" cluster, the necessary condition is that every cat image-caption pair lies close to the "cat" centroid, and a gap between the two modalities inflates that radius, cluster purity drops even though each individual pair is still top-ranked for retrieval.
4

---

Published as a conference paper at ICLR 2026
Initialization
CLIP-based Learning
Our Method
A Text Vision A Audio
Figure 2: AV-MNIST multimodal latent space. The CLIP-based learning creates fragmented latent space with embeddings clearly clustered by modality and not by multimodal semantics. Our method closes the gap and enhances group-wise semantics, placing embeddings of the same class in the same portion of the space, effectively learning a semantically meaningful multimodal latent space.
4 CLOSING THE MODALITY GAP
We aim to close the modality gap while ensuring consistent alignment among the positive pair distribution. To achieve such scope, we propose two novel losses. The first one, the Align True Pairs loss LATP guarantees the alignment between true pairs. Considering M modalities, among which a is the anchor one (i.e., the modality to which all the other modalities are aligned to Girdhar et al. (003): Zhu et al. (2024) ) the loss is:
N Σ N
CXT = M - MEM,mpfe
1
Σ
2
z (2)
where is taken from the of available modalities, is the anchor modality and N the batch size. The second one, the Centroid Uniformity loss Lcu, ensures uniformity of the semantic classes among the modalities in the latent space by:
Lcu log
Σ
Σ
cxp (-211m4 11)11)
(8)
N
j=1.j
which uas with j. j. are the centroids defined as:
1 M
Σ 2%
MEM
and Cą the centroid of the k-th element of the batch built by averaging all the modalities embeddings. The effect of the two losses is complementary. The LAIP promotes closeness between positive pairs, effectively enhancing the mean cosine similarity between them. However, involving solely such a loss may produce side effect: the entire latent space collapses into small portions, placing representations of dissimilar data in the same portion of the latent space, as we show in the Appendix. Therefore, the contribution of the Lcu loss becomes crucial, ensuring the sparsification of the latent space by enforcing uniformity to the centroids while preserving the learned alignment. Indeed, moving the centroids of matching pairs in the space implies moving the modalities representations accordingly. effectively preserving alignment while leveraging the whole space. Without centroids, uniformity should have been applied independently to each modality as Wang Isola (2020), disrupting the learned alignment among similar semantic pairs, we show in the Appendix. Additionally, the radial basis function (RBF) kernel in equation glis is well related to the uniform distribution on the unit hypersphere where multimodal representations lie Wang & Isola (2020), therefore enforcing the coverage the whole surface of the hypersphere. The overall proposed loss function that aims at aligning the true pairs and closing the modality gap sum of the two terms:
Lynn LXTP Leu-
(10)
5

---

Published as a conference paper at ICLR 2026
Such loss should be then combined with the contrastive loss to obtain:
La gup + (((--n) + +L ((n-tm)
(11)
for the two-modality m and case to lighten the notation, but it can be expanded to more modalities.
EXPERIMENTAL VALIDATION
We perform extensive evaluations in controlled and real-world scenarios with four different datasets, four tasks, and a diverse number of modalities.
5.1 SETTING
To evaluate the proposed method in multimodal scenarios, we design a series of experiments that pro- gressively increase in complexity and scale. Following the literature, we begin with two foundational experiments using the CIFAR- 10 and AV-MNIST datasets, which involve and three modalities, respectively. Subsequently, we scale up our experiments using the MSCOCO and MSR- -TT datasets, which offer more complex multimodal data.
CIFAR1O (Image-Text). The CIFAR10 dataset consists of 60k 32 x32 color images evenly distributed across 10 classes, with 50k images for training and 10k for testing Krizhevsky et al. (2009). For our experiments, we pair each image with its corresponding class label as text, creating a bimodal dataset. For CIFARIO, we use two separate ResNet50 encoders, one for text and the other for images.
AV-NNIST (Audio- -ixual-Text). AV-MNIST is synthetic dataset combining visual, auditory, and textual modalities. It is the union of two well-known datasets: the MNIST dataset composed 60k 28 x2 digit images and the Audio-MNIST Becker et al. (2023) containing 30k audio samples of spoken digits (0-9) from diverse speakers. Each sample is also associated with a textual label that represents the digit. Encoder details in the Appendix.
MSCOCO (Image-Caption). The MSCOCO dataset contains over 330k images, each annotated with five human-generated captions, totaling more than 1.5M captions Lin al. (2014), and it is well-known benchmark for analyzing the modality gap. To scale up the model capabilities and test approach in this challenging dataset,
EVA-CLIP ViT-G al. (2023) the visual encoder and BERT-B Devlin et al. (2019) to process the textual captions.
The MSR-VTT dataset comprises 10k video clips spanning 20 categories, with each clip annotated with approximately 20 natural language sentences, resulting total of 200k captions Xu et al. (2016). Each video clip includes visual frames, audio tracks, and corresponding textual descriptions, making it trimodal dataset. Each sample in the dataset is associated to one of the 20 categories. This makes the dataset useful to be evaluated with clustering metrics. We maintain the same architectures used for MSCOCO plus BEAT's Chen et al. (2023a), transformer-base model, used to extract features from the audio tracks.
MSR-VTT (Video-Audio-Teex)
5.2 TASKS
We conduct a suite of experiments covering both instance wise and group-wise tasks.
Cross-Modal (CM) Retrieval. We evaluate the model ability to associate related data across modalities (e.g., text to audio or text to image) with tross-modal retrieval. For a given query in one modality, the task is to retrieve matching items from another modality using cosine similarity in the shared latent space. This task quantifies the instance-level alignment and cross-modal discrimination capacity of the learned embeddings.
Clustering. We assess the structural organization of the latent space using supervised and unsu- pervised clustering. We apply standard clustering algorithms (k-means and k-NN) and evaluate the resulting clusters using V-Measure and k-NN accuracy. High performance in clustering indicates that the latent space preserves semantic coherence across modalities and aligns samples belonging to the same class coarse- coarse-grained semantic group closely, regardless of the input domain.
Multimodal (MM) Retrieval with Cross-Conne Conditioiin.. We evaluate the performance of down- stream models that use the extracted embeddings from the modality encoders following Li al.

---

Published as a conference paper at ICLR 2026
(2021). For this task, we involve additional multimodal encoder that takes as input all the modali- ties and fuses them. Therefore, a proper alignment is crucial in this task. Such a multimodal encoder outputs the probability score of the semantic match among them. To this purpose, following Chen et al. (202bb), as the multimodal downstream encoder we reuse the text encoder attaching an MLP on top of it to extract the probability value. Along with brand-new loss functions, we add the data-anchor matching (dam) loss function to train these components, as in Li et al. (2021) Chen et al.
(2023b) Cicchetti et al. 2025):
dam E (a.ma."" (A,,))a .My) [y log Pdam + (1 - y) log(1 Pdam)) )]
(12)
This task stresses the ability of the encoders to build coherent, modality-agnostic representations and provides insight into the expressiveness and compositionality of the learned space. Higher performance in this task highlights an increased facility by the multimodal encoder to detect interde- pendencies among modalities.
Captioning. A downstream task to evaluate the benefit of an aligned and well-structured latent space data captioning. Following Yan et al. (2022). we involve a decoder serving as the language generator model and add specific loss term that, along with our proposed loss functions, trains the encoder- decoodr structure for this specific task. Intuitively, the more aligned and semantically coherent the latent space, the better the model will generate the captions. The captioning loss is:
Ceup
log Polyelyee,
(13)
where y is the exact tokenized text the model aims to learn by maximizing the conditional likelihood denotes the probability assigned to the token yt given as input the past history y<t and the input features x. Captioning requires both accurate semantic grounding and compositional generalization, and therefore serves as complementary proxy for evaluating the usefulness of the learned features in downstream generative tasks.
under the forward autoregressive factorization. Po (Y,,,)) P
5.3 RESULTS
2.0
Aligning group-wise semanties
10.0
CLIP (FT)
(LT)
Gredient Magnituee
Figure [ shows that increasing the modality gap leads drop in clus- tering metrics, despite having a negli- gible effect on retrieval scores, while decreasing it considerably improves the clustering metric (+17.5 points). This suggests that instance-level align- ment tolerant to cross-modal off- sets, while group-wise metrics benefit from a more globally structured latent space. Crucially, the optimal clus- tering performance coincides with the zero-gap configuration, indicat- ing that full semantic overlap facili- tates more coherent clusters. The plot is obtained by starting from the gap value of standard CLIP on the MSR- VTT dataset and then posthoc shifting the matching pairs up to 0.0 and 1.0 gap. Second, the simulation in Figure 3Ja) a)) shows that, in presence of data mismatching, our proposed loss induces a landscape in which the global minimum is achieved when the modality gap is zero. In contrast, standard contrastive losses such as CLIP achieve the minimum around 60 degrees, thus when matching pairs are still quite far in the latent space. Proof in the Appendix. Therefore, we can train the model with the proposed losses to reach the zero modality gap while aligning group-wise semantics and improving the clustering performance without affecting retrieval performance. The underlying mechanism is clarified in Figure Bbb). which shows that as the modality gap narrows, the gradient magnitude associated with non-matching
7.5
NatAGap
0.5
Pair-1 (Non-matchh
Ours
Pai:3-4 (Match)
5.0
Zero Gap
1.0
5 (Match)
Zero Gap
0.5
0.0
0.0
-80
20 Gap (Dagrees)
Gap (Dngrees)
Figure 3: (a) Following Liang et al. (2022). we place six sim- ulated image-text embedding pairs on a 3D sphere, with two mismatched pairs. We artificially move these pairs toward closing or enlarging the gap among them and we track the loss landscape during the simulation. (b) During the same simulation we keep track also of the gradient magnitude re- ceived by the six embeddings pairs through our design loss function LcI,.. When the gap is closer to zero, the contribu- tion to the loss is just matter of the non matching pairs.
7

---

Published as a conference paper at ICLR 2026
QUERY: "An of cet sitingg the safa'
QUERY:
sure mocal ooky serrenace
shape nold semaniis
CLIP (with gap!)
Ours (no gapl)
CLIP (with gap!)
Ours (no gap!)
Sim:
cat layingg on top of
Sim: 0.905
Sim: C. C.697
Sim:
lush green forest filled
0.524
chair
0.883
Sim:
cal sitting the back loaking window
0.484
laying on couch 0.891
0.614
0.831
Sim:
sitting top bed
black
0.422
0.868
leying on a couch
0.587
0.721
Sim:
laying top couch next to
Some leafy trees are 0.887 hiding black bear
0.387
0.823
0.577
computer
Figure 4: Most similar vectors from the MSCOCO vector databases with both modalities embedded. CLIP learned space has gap of 0.47 and, for a text query, all the closest embeddings come from the text modality, meaning that embeddings are clustered according to the modality and not according to the overall multimodal semantics. On the contrary, the proposed method has nearly zero gap, and most similar samples come from both the image and text modalities, proving that we can effectively build a multimodal latent space that is semantically coherent among the modalities. The same behavior holds for an image query.
pairs increases, while that of matching pairs decreases. With a large gap, non-matching samples provide the same gradient contribution as matching ones. As the gap closes, these negatives become harder and more semantically informative, thus dominating the optimization process and encouraging the model better shape the space by adjusting the missing matches. This naturally improves the structuring of semantic clusters without altering the rank of positive pairs (which have nearly zero gradients with the closed gap), hence preserving (or improving) retrieval performance. Altogether, these observations highlight that minimizing the modality gap leads to more informative gradient signals by emphasizing semantically meaningful negatives, thereby improving the alignment and group-wise structure of the latent space. This reflects in improved clustering performance without affecting retrieval. By explicitly encouraging the zero-gen gap configuration, our proposed loss LCLye provides a straightforward method to close the gap and align group- wise semantics.
Figure shows the latent space plot the case of embedding dimension equal to in the AV-MNIST dataset. Notably, the plotted spaces
stochastically generated but the real latent space in R.3 As it is clear, the proposed method closes the gap better than conventional CLIP-based learning. Crucially, this consistently improves group-wise clustering, placing embeddings from the same class closer to each other in semantically meaningful latent space.
Table quantitatively proves our claims the relation between the gap and the groep-wise semantics. Indeed, the gap progressively closed with the proposed method, the clustering performance crucially improves, while retrieval accuracy between text and video (TV) and text and audio (TA) is barely affected by the gap. Overall, this intuition is congruous across di- verse datasets with different numbers (two and three) and types of modalities (image, video, text, audio), even though the simpler the dataset, the easier is for the model to align group-wise semantics.
0.8
0.8
MECOCO
0.0
(T)
0.4
NaAAup
83
Ours
0.1
0.
Cosino Pairs
Closing the gap
Figure 5: CosTP and gap during training.
Figure g shows the most similar samples by querying the learned latent spaces of CLIP and of method without forcing the query to be
8

---

Published as a conference paper at ICLR 2026
Table 1:
Instance-wiss (cross-modal (CM) retrieval) group-wise (clustering) tasks correlated the gap value. The harder the dataset, the more evident the results.
CM Retrieval
Clustering
Method
Dataset
Gap
TV R@I
TA R@I
V-Measure
kNN
CLIP (LT) (Radford et
2021 Yaras et al.
0.86
82.0
67.0
81,2
CIFARIO (2 modal)
CLIP (FT)
2025
0.14
82.1
67.6
81.9
Ours
0.09
82.4
67.9
82.4
CLIP (LT)
Radford
2021
0.47
74.6
12.98
26.3
MSCOCO (2 modal)
CL.IP (FT)
Yaras el al.
2025
0.12
73.2
12.99
31.0
Ours
0.03
70.3
23.63
36.4
CLIP (LTT) (Radford al.
2021
0.20
87.1
84.2
77.6
87.0
AV-MNIST (3 modal)
CLIP (FT)
Yaras et al. 2025
0.24
84.1
80.4
73.8
85.0
Ours
0.09
88.7
89.1
82.7
89.2
CLIP (LT)
2021 Radfford et al.
0.29
34.2
10.3
23.3
52.9
MSR-VTT (3 modal)
CLIP (FT)
Yaras et al. 2025
34.8
31.3
55.7
Ours
0.07
32.8
11.8
32.1
58.0
Table 2: Multimodal (MM) retrieval and captioning results.
Space Measures
MM Retrieval
Captioning
Method
Dataset
Gap
Cos TP
R@D
BLEU@1
BLEU@3
CIDEr
CLIP (LT)
0.47
0.34
72.5
45.8
22.5
153.2
CLIP (FT)
0.12
0.63
73.8
45.9
23.0
155.0
MSCOCO (2 modal)
NotAGap
0.17
0.11
75.6
45.4
22.2
153.3
Ours
0.03
0.77
77.3
46.1
23.2
160.9
CLIP (LT)
0.24
0.27
30.6
26.7
9.5
63.6
CLIP (FT)
0.17
0.37
30.3
26.2
9.4
63.2
MSR- VTT (3 modal)
NotAGap
0.09
0.07
29.9
24.3
8.7
53.8
Ours
0.06
0.53
33.3
26.8
9.6
64.4
cross-modas by construction. This means that the similarity cares about the semantics only, regardless of the modality. Interestingly, when querying the CLIP space for the nearest vectors, while returning semantically similar samples, the emheddings always belong to the same modality as the query. In contrast, the gap-free latent space learned with our method, the closest vectors not only share the same semantics but also come from different modalities. This proves that closing the modality gap effectively helps build semantically aligned multimodal latent space.
Table 2 compares the proposed method and the current literature in closing the gap and downstream tasks. As it is clear from Tab. [ our novel loss closes the multimodal gap by achieving nearly 0.0 distance between the modality clusters centroids. Additionally, we better align true pairs by crucially increasing their cosine similarity from 0.34 of the standard CL.P (LT) 0.77 in MS COCO. The capabilities of modeling the multimodal latent space are evident in Fig. too, where the proposed method outperforms the conventional CLIP, other solutions with fixed temperature (FT) Yaras et al.) (2025), and NotAGap Fahim et al. 22024). Importantly and counter-intuitively during training, the cosine similarity true pairs slightly but progressively decreases in comparison configurations. This means that the true pairs progressively worse aligned. On the contrary, the proposed method considerably increases the cosine similarity between true pairs, truly aligning them, while also definitely closing the gap among modalities, regardless of whether there
three modalities.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
In this work, we revisited the modality gap in multimodal representation learning, providing insights into the impact of this phenomenon on downstream tasks. Through theoretical analysis and extensive experiments across bimodal and trimodal datasets, we showed that while the modality gap has limited influence on instance-level tasks such as retrieval, it significantly affects group-wiss -iip tasks like clustering. To address this, we proposed novel objective that explicitly aligns true pairs while promoting latent space uniformity. Our method consistently reduces the modality gap and improves clustering performance without compromising retrieval accuracy, offering simple yet effective solution for better semantic organization multimodal spaces.
ACKNOWLEDGEMENTS
This work was partially supported by the European Union under the NRRP of NextGenera- tionEU, partnership "Future Artificial Intelligence Research" (PE00000013 SPOKE 5- CUP B53C22003980006 FAIR: High Quality Al) and partnership on "Telecommunication of the Future" (PE00000001 program "RESTART"), and partially by the Progetti di Ateneo of Sapienza University of Rome under grant RM123 88F75F8072 and Ministry of University and Research (MUR) within the PRIN 2022 Program for the project "EX- EGETE: Explainable Generative Deep Learning Methods for Medical Signal and Image Processing", under grant number 2022ENK9LS, CUP 353023013030006.
and partially by the Italian
REFERENCES
Sören Becker, Johanna Vielhaben, Marcel Ackermann, Klaus-Rohee Müller, Sebastian Lapuschkin, and Wojciech Samek. AudioMNIST: Exploring explainable artificial intelligence for audio analysis on a simple benchmark. Journal of the Franklin Institute, 2023.
Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. In International Conference on Machine Learning, pp. 5178-5193, 2023a.
Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Ming-Ting Sun, Xinxin Zhu, and J. Liu. VAST: A iitoo--udio-subtillee Processing Systems (NeurIPS), 2023b.
foundation model and dataset. In Neural Information
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning. pp. 1597-1607. PMLR, 2020.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-28299 2023.
Giordano Cicchetti, Elconora Grassucci, Luigi Sigillo, and Danilo Comminiello. Gramian multimodal representation learning and alignment. International Conference on Learning Representations (ICLR), 2025.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Association for Computational Linguistics, 2019.
Georgiana Dinu, Corey D Barrett, Yi Xiang, Miguel Romero Calvo, Anna Currey, and Xing Niu. Effective post-training embedding compression via temperature control in contrastive training. In International Conference Learning Representations (ICIR), 2025.
Benoit Dufumier, Javiera Castillo Navarro, Devis Tuia, and Jean- Phillipi Thiran. What align in multimodal contrastive learning? In International Conference on Learning Representations (ICLR), 2025.
10

---

Published as a conference paper at ICLR 2026
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. CLAP: learning audio concepts from natural language supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE. 2023.
Sedigheh Eslami and Gerard de Melo. Mitigate the gap: Investigating approaches for improving cross-modal alignment in clip. International Conference Learning Representations (ICLR), 2025.
Abrar Fahim, Alex Murphy, and Alona Fyshe. It's not modality gap: Characterizing and addressing the contrastive gap. ArXiv preprint: arXiv:2405.1855 2024.
Rohit Girdhar, Alaacldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind one embedding space to bind them all. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 15180-15190, 2023.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 10 (canadian institute for advanced research). 2009.
Anna Kukleva, Moritz Böhle, Bernt Schiele, Hilde Kuehne, and Christian Rupprecht. Temperature schedules for self- superviid contrastive methods long-tail data. In International Conference on Learning Representations (ICLR), 2023.
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Neural Information Processing Systems, 2021.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision- ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings. part 13, pp. 740-755. Springer, 2014.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. CLIP4Clip: An empirical study of clip for end to end video clip retrieval. Neurocomputingg 508:293-300 304, 2021.
Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Andrew D. Bagdanov. Cross the gap: Exposing the intra-modal misalignment in clip via modality inversion. In Interna- fional Conference on Learning Representations (ICLR), 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
Sameera Ramasinghe, Violetta Shevchenko, Gil Avraham, and Ajanthan Thalaiyasingam. Accept the modality gap: An exploration in the hyperbolic space. In /EEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 27253-27262, 2024.
Adriel Saporta, Aahlad Manas Puli, Mark Goldstein, and Rajesh Ranganath. Contrasting with symile: Simple angoss-g representation learning for unlimited modalities. In Neural Information Processing Systems, 2024.
Simon Schrodi, David T. Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision- language models. In International Conference on Learning Representations (ICIR), 2025.
Peiyang Shi, Michael C. Welle, Mårten Björkman, and Danica Kragic. Towards understanding the modality gap in CLIP. In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023.
11

---

Published as a conference paper at ICLR 2026
Quan Sun, Yuxin Fang, Ledell Yu Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: Improved training techniques for clip at scale. ArXiv preprint: arXiv:2303.15389 2023.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv preprint: arXiv:1807.037 2018.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2495-2504, 2020.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929- 9939. PMLR, 2020.
Junkang Wu, Jiawei Chen, Jiancan Wu, Wentao Shi, Xiang Wang, and Xiangnan He. Understanding contrastive learning via distributionally robust optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5288-5298 2016. doi: 10.1 00.110//CVPR..01..777
Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. VideoCoCa: Video-text modeling with zero-shot transfer from contrastive captioners, 2022.
Can Yaras, Siyi Chen, Peng Wang, and Qing Qu. Explaining and mitigating the modality gap in contrastive multimodal learning. In Conference on Parsimony and Learning (CPAL), 2025.
Yuhui Zhang, Jeff Z. HaoChen, Shin-Cheng Huang, Kuan-Chieh Wang, James Zou, and Serena Yeung. Diagnosing and rectifying vision models using language. International Conference on Learning Representations (ICLR), 2023.
Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, Hongfa Wang, Yatian Pang. Wenhao Jiang, Junwu Zhang. Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, and Liejie Yuan. LanguageBind: Extending video-language pretraining n-modality by language- based semantic alignment. In International Conference on Learning Representations (ICLR), 2024.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A. PROOFS AND MOTIVATIONS
The gap is nearly the same for all the pairs. We empirically prove our theoretical claims and we compute both the Euclidean distance and the cosine similarities among true matching pairs, which could both be interpreted as gap measures. As it is clear from Tab. 3 the variance in the gap is extremely low and nearly O, so all the points have the same gap. Thus, it is reasonable and empirically ggguuddd to assume the same 6 measure for the gap in Sec. 4 to use the gap measured as centroid distance in equation 2 being it a single measure for the overall gap. The same findings be applied equation aas well. Concurrently, (Zhang al. 2023) demonstrated that the gap vector is orthogonal to the text and image embeddings, and that the gap can be approximated with constant vector, confirming our findings.
Table 3: Mean and variance of the distance between the pairs in MSCOCO and MSR-VTT datasets.
Dataset
Distance Mean
Distance Var
Cos TP Mean
Cos TP Var
MSCOCO (T-I)
1.153
0.004
0.334
0.005
MSR-VTT (T-V)
1.163
0.009
0.319
0.012
MSR-VTT (T-A)
1.252
0.010
0.211
0.016
Proof for global minimum in Fig. The 60 minimum in Fig. 3 is direct outcome of the controlled synthetic setup and the dynamics of the InfoNCE loss ((m-n)) In the experiment, image (v) and text (t) embeddings lie on two fixed rings with an initial angular offset of A 120. During the optimization, the true image-text pairs are brought closer (current gap = 0), while the swapped pairs (i.e., the negatives) remain their original positions. Therefore, from the image anchor perspective, these negatives are now at an angular distance of A - 0. This geometric property is exact and does not rely on any symmetry or regular placement.
The InfoNCE loss balances attraction to the true pair and repulsion from negatives. Its derivative becomes zero when the similarity to the true caption matches the similarity to the closest negative (this occurs when 0 = 2/2). In test case, that means CLIP converges to 60, observed in Fig. 3 Importantly, this also implies that InfoNCE never drives 0 to 0, regardless the initial gap A.
In contrast, method includes the explicit LATP term that always decreases as 0 0, and does not depend on the position of the negatives. Formally, our loss can be written as LCL,. ((m-n) + LATP + Lcu. The LATP term is LXTP ||v t 2(1 cos O), whose gradient is 0LATP = siin 0, always pulling the pair toward perfect overlap (0 0) and has no counteracting
repulsion. The Lcu term depends only on the class centroids, not on the intra-pate angle, so θCcu 0. Therefore, the total derivative becomes
dL ae
0L
+ sin A,
(14)
00
with the extra term siin O shifting the sole stationary point to 0 0, which is now global minimum. This holds for any initial configuration.
Proof for equation 6. For a semantic class and two modalities m, the class centroid under uniform shift 8 equation 5):
1 (z + +6 + (z) + +5) = 122 -,
5 2
(15)
where
112 (z 2.))
(16)
We expand the squared deviation:
13

---

Published as a conference paper at ICLR 2026
12:112 1|2" (14 + 8)
= =1|(2"" (4) 812
(17)
2 22z pba 8). 1z 22112 + 11812
Taking the expectation over classes yields:
Eg || - 1311 = E. ||2
-02 +|181| 2 [=,' -1g 871
(18)
Since 8 is approximately constant across classes and orthogonal to the span of semantic vectors Zhang et al. (2023), the cross-term vanishes:
Bs [2 - M% 8)] ~ O.
(19)
Thus, we obtain equation
Bg
2
101112]
E.
2 p0112 + +11812
(20)
A.2 EXPERIMENTS DETAILS
To process the AudioMNIST dataset. compute the Mel spectrograms with 128 nmels, fimax at 8000, hop length equal to 512, and 2048 nfft. As image encoder, we employ convolutional neural network (CNN) comprising two-layer convolutional architecture with 32 and 64 filters, respectively, ReLU activation functions, Max Pooling, and final MLP layer to map the features into the latent space. For the audio modality, design a three-layer convolutional encoder with 16, 32, and 64 filters, ReLU activations, and an MLP layer to similarly project audio features into the latent space.
Clustering on MSCOCO. MSCOCO contains the information about objects that are present inside an image, and each image could contain more than one object, to an undefined upper bound (also 10 different objects in an image), making standard clustering hard to apply. However, we devise pipeline to perform cross-modal clustering on this dataset. We take into consideration images from the test dataset that contain only a single object, so that we have univocal correspondence between the image and single textual caption representing such an object. We report clustering results on MSCOCO in Tab
Computing resources. The AV-MNIST experiments are conducted an RTX4080 with 16GB. The CIFARIO experiments on an A6000 with 48GB. The experiments on MS COCO and MSR-VTT experiments node with 4x A100.
A.3 ADDITIONAL EXPERIMENTS
We visualize the PCA latent spaces for two (MSCOCO) and three modalities IMSRVVTI at initial- ization, after the conventional based training. and after the training with the proposed method. Figure 66 shows the results. From Fig. 6 the gap closure is evident. Contrary to CLIP- based learning, our method can close the gap and spread modality embeddings leveraging the whole hypersphere space. Interestingly, Fig. shohwws similar behavior of the gap for the two- and three-modal case. At initialization, one modality tends to occupy more space, being more sparse, while the other/others are much more grouped. With CLIP-based learning, the modalities still tend to similarly lie in the space. However, regardless of the initialization and of the number of modalities, the proposed method completely closes the gap, building more compact and well-aligned space.
Additional gap measures. Together with the gap formulation in equation we further measure the gap suggested in Schrodi et al.] (2025), where the authors proposed the Relative Modality Gap (RMG) measure. We report the results in Tab. 4
Closing the gap zero-shot. We consider the models trained MS COCO and then evaluate them in a zero-shot fashion on the validation set OpenImage-V? V7. The entire validation set contains 41620 images, each of them has associated one class among more than 20k image classes. We evaluate the
14

---

Published as a conference paper at ICLR 2026
Modalitie
Text
Aucio
Modalliis
Intiaization
CLIP-besed Leerning
Oura
Figure 6: Left, latent spaces initialization with the modality clusters clearly defined. Center, learned after training from CLIP-based learning that preserves the modality gaps. Right, space learned with our method, covering the whole space with overlapping clusters. PCA visualizations.
Table 4: Relative Modality Gap (RMG) measure for the methods on MSCOCO and MSR-VTT datasets.
MSCOCO
MSR- ATR-VTT (T-V)
MSR-VTV (T-A)
CLIP (LT)
0.43
0.44
0.48
CLIP (FT)
0.27
0.40
0.46
Ours
0.17
0.33
0.44
ability of the proposed loss combinations to zero-shot closing the gap on the whole validation set, measuring the gap as equation 2] the Cosine True Pairs (Cos TP) as in equation 3 and also Relative Modality Gap (RMG) suggested in Schrodi et al. (2025). The results in Tab. shows that the proposed method well-scale to large-scale datasets, achieving consistent results in the metrics and outperforming the comparisons by considerably reducing the modality gap (absolute and relative). Furthermore, we also evaluate the clustering performance the methods on this dataset. For this purpose, select from the validation set the images containing one category among the 10 more common classes: 'Plant', 'Car', 'Person', 'Clothing'. Foodd, 'Flower', 'Tree', 'Mammal', 'Wheel', 'Dog'. Moreover, we rebalance the dataset that, the final version, each class contains an equal number of samples. In this experiment too, the V-Measure achieved by the proposed method is the highest with respect to other methods, proving the effectiveness the proposed method once again.
Table 5: Zero-shot gap and clustering results OpenImage- V7 validation set.
Method
V-Measure f
Gap +
RMG |
Cos TP 1
CLIP (LT)
16.7
0.459
0.462
0.210
CLIP (FT)
13.5
0.374
0.516
0.224
Ours
17.2
0.311
0.429
0.273
Relation between gap and clustering performance. To further highlight the relation between clustering performance and the gap. plot the gap measure and the V-Measure during the training of the ResNet50 encoders on the CIFARIO dataset. Fig 7 the results. As the gap decreases, the V-Measure evaluations increase reaching the best score with the proposed method, which better closes the gap.
Impact of the initialization. According to Liang et al. 2022) that first discovered the modality gap phenomenon, the gap exists at initialization and it then preserved by the conventional contrastive loss. Nevertheless, the key reason for the modality gap is the contrastive behavior of the InfoNCE
15

---

Published as a conference paper at ICLR 2026
0.7
0.6
1.25
VAMeusure
0.4
Ours
0.3
temp 0.01
temp 0.05
0.50
0.2
temp 0.07
0.1
temp 0.1
0.25
0.0
100
150
200
0
50
100
150
200
Epochs
Epochs
Figure 7: Clustering performance (V-Mcasure) and gap measure during training on the CIFARIO dataset.
loss and not the initialization. To prove it, compare two configurations with the same setup, model (ResNet50), and loss (InfoNCE). We enforce an initial sparsification of the learned space for the first epoch that the encoders first learn to sparsify the embeddings, regardless of the modality. We do so by using as loss the uniformity loss over the hypersphere by Wang & Isola (2020). However, once the encoders learn the sparsification and we activate instead the conventional contrastive learning loss, the encoders start to separate the modalities again and the gap begins to be recreated, as we show Fig. g We conduct the same experiment with the proposed losses to close the gap, and no effect is revealed with the initial sparsification as well. Therefore, although the gap is created initialization, this does not impact the learning procedure and therefore the conventional contrastive loss function that tends to create such a gap.
taticationn
500 Herotions sparsiication
CJP based
Figure 8: Latent space visualization during training of the conventional InfoNCE loss. The gap created at initialization, as already known in Liang et al. 2022). However, even we initially train the models to learn to sparsify the latent space (second plot) and reduce the gap, as soon reintroduce the conventional InfoNCE loss function, the gap is recreated (third plot). Experiment with ResNet50 from scratch on MS COCO dataset.
Experiments the MultiBench classification task. We perform additional experiments with comparison methods in a different task. We rerun the experiments using CoMM Dufumier et al. (2025) repository three datasets (MOSI, UR-FUNNY, MuSTARD) for the elassification task from MultiBench. Then, we apply on CoMM the method proposed in Yaras et al. (2025) for a further comparison (CoMM Yaras al. (2025).. Finally, we add our proposed losses to the CoMM framework (CoMM + Ours line in the table below). We report gap and accuracy measures. As it is clear from Tab. the proposed loss combination improves results in terms of accuracy with respect to CoMM and reduces the gap in all the datasets.
Visualizing the semantic stripes. In the presence of the modality gap, embeddings tend to form "semantic stripes". Such stripes allow good performance in retrieval tasks since positive pairs have higher cosine similarity with respect to non-matching pairs, even though such similarity is far from the ideal 1.0, but low performance in group-wise tasks embeddings coming from the same semantic concept lie separately in the space. We visualize these stripes in the AV-MNIST dataset with dimensional embeddings in Fig. [ Audio (triangles) and vision (squares) embeddings lie in separate regions of the latent space. Nevertheless, they form "semantic stripes" with each other, in which classes (colors) are matched in stripes with the corresponding class of different modality.
16

---

Published as a conference paper at ICLR 2026
Table 6: Classification results in MultiBench datasets compared with and with CoMM + Yaras et al. (2025).
et al.
2025
MOSI
UR-FUNNY
Method
Gap
Acc
Gap
Acc
CoMM Dufumier et al. 22025
0.33
65.91
0.81
62.83
CoMM Yaras (2025)
0.28
66.42
0.63
61.10
CoMM Ours
0.24
67.65
0.77
63.25
Figure 9: Semantic stripes in the AV-MNIST dataset for CLIP (LT).
Intra-modal expressiveness. Modality specific features are essential for some instance level or downstream tasks. Reducing the gap between modalities does not imply discarding or collapsing the unique characteristics of each modality. The proposed losses act only on the relative positioning of positive pairs. They encourage samples that share the same semantics to move closer without limiting the expressiveness of the individual encoders, as confirmed by our experiments. To support such claims, we conduct an additional experiment to verify whether intra- intra-modal discriminative features are preserved. We use pretrained encoders on the MSR-VTT dataset, specifically Eva-CLIP for visual features, BEATs for audio features, and BERT for textual features. After extracting the embeddings, we apply both the k-NN algorithm and simple linear classifier (In MSR-VTI VTT each sample belongs to one of the 20 categories). Table 7 reports the accuracy obtained by these two methods when applied independently to the features each modality. Each experiment conducted times. The results show that the expressive power of each individual encoder remains unaffected by the introduction of our additional losses and by the consequent reduction of the modality gap. In fact, both the kNN and linear probing performance are essentially unchanged in the scenario where the gap is present (CLIP (LT)) and in the scenario where the gap is reduced (Ours). These results mean that the discriminative intra-modality features are preserved even when the gap is mitigated, thus the embeddings are likely to preserve the intra- intra-modality features.
Table 7: Accuracy of KNN and linear classification on modality-specific features (visual, textual, and audio) using pretrained encoders. The results demonstrate that the proposed losses preserve discriminative intra-modality features, as accuracy remains consistent across both the original and reduced modality gap settings.
Only Visual Features kNN
Only Textual Features kNN
Only Audio Features kNN
Method
Ace
Acc
Acc
CLIP (LT)
60.39 + 2.9
63.84 1.2
45.79 + 2.4
48.49 + 2.5
38.28 + 2.5
42.36 3,5
CLIP (FT)
59.62 +:2 2.3
64.40 0.8
44.61 3.3
47.90 0.5
37.71 2.7
41.80 +: 3.4
Ours
60.39 + 2.1
64.20 1.0
44.34 + 2.5
49.98 1.8
37.50 + 2.8
42.26 2.6
17

---

Published as a conference paper at ICLR 2026
Cosine similarity
Euclidean similarity
Move closer
Move apart
X Centroid
Shape Modality
Color Semantics
clm ea) contribution
CTP contribution
Lau contribution
Figure 10: Schematic representation of losses contributions.
A.4 ABLATION STUDIES
Why using ((m-n)) comhination with the proposed losses? LATP and Lcu regularise the global geometry (closing the modality gap and preventing centroid collapse), but they do not provide the instance-level repulsion that keeps different samples apart. Instead, the InfoNCE supplies exactly that term, so all three losses are somewhat complementary required to obtain representations that simultaneously well aligned (no gap), uniformly spread, and discriminative. To prove our claims, we perform an ablation study to highlight the critical role of the InfoNCE loss (C(m-n)) in achieving optimal performance. While overall semantic alignment
still be attained using combination of LATP and Lcu, evidenced by comparable results in R@5 5 and R @ 10, the absence of (m--n) leads to a noticeable drop in @@ performance. This indicates that the framework becomes less accurate in retrieving the top-1 correct caption when the contrastive loss is removed, underscoring its importance for ane-grained alignment between modalities.
Table 8: Ablation the contribution of ((m-m) in combination with the proposed losses.
C [(m-a]
Cos TP
Gap +
T2V R@I
T2V
R@5
T2V R@ 10
V2T 1
V2T R@5
V2T R 10
0.71
0.05
30.0
68.3
83.0
31.2 36.4
69.3
82.0
0.63
0.04
35.2
71.5
84.2
72.1
83.8
Furthermore, visualize the contributions of each of the three losses L(m-+n), LATP, and Lcu in Fig. 10 The first one, the InfoNCE loss, aligns multimodal matching pairs while spreading all non-matching ones apart. This dual behavior encourages the formation of gap between modalities, as demonstrated in numerous previous works [iang et al. (2022); Shi et al. (2023). The InfoNCE loss function often becomes trapped in local minima, where the loss is minimized (semantic alignment is maximized) but a non-zero gap remains Shi et al. 2023). The LAI loss forces the match among true pairs, consistently reducing the modality gap. However, the LXT does not consider matchingg pairs, potentially making the latent space collapse in small regions, thus the contribution of Lcu becomes crucial. Indeed, the Lcu loss spreads the semantic centroids (i.e., the centroids of matching pairs) while preserving the alignment.
A Ablation on LXTP and Lev. We conduct an ablation study for the proposed loss functions. We train from scratch two ResNet50 encoders with OpenCLIP Cherti et al. (2023) the CIFARI0 dataset for 100 epochs. We set the temperature parameter of the CLIP objective 0.07 and we perform sensitivity experiments, weighting the two proposed losses. To this purpose, introduce two new hyperparameters for equation 10. X₁ and A2, combining the losses as
Lerpe = (mm-m)) + LA + XzLcu-
(21)
Tab. 9 reports the results for retrieval (R @1) and clustering (V) across all the investigated configura- tions.
It is interesting to note that the configuration with \ A2 0 (the standard InfoNCE loss alone) achieves the lowest performance in both retrieval and clustering tasks. Furthermore, the results show clear trend: increasing X2 consistently leads to improved performance, indicating the importance of
18

---

Published as a conference paper at ICLR 2026
Table 9: Sensitivity study on X (row) and Ag (column) showing retrieval (R@) 1), clustering (V- Measure), and their average (Avg) on the CIFARIO dataset. Bold indicates the best value; underlines highlight competitive second-bess settings.
A1 \2
0.00
0.25
0.50
0.75
1.00
R @@1 I: 79.39 V: 61.65
@ I: 78.64
R@ @ 1: 83,61
@@ 1: 84.05
R@ 1: 86.20
V: 63.67
V: 67.70
V: 66.87
V: 66.64
0.00
Avg: 70.52
Avg: 71.16
Avg: 75.66
Avg: 75.46
Avg: 76.42
R R@l: 80.59
@@1: 79.39
R@ 1: 80.15
R@ 1: 84.06
R@ 1: 83,82
V: 66.05
V: 63.26
V: 64.47
V: 69.93
V: 69.77
0.25
Avg: 73.32
Avg: 71.33
Avg: 72.31
Avg: 76.99
Avg: 76.80
R@ 82.37
R@1: 79.34
R@I: 81.69
R@1: 82.43
R@ 1: 84.14
V: 68.96
V: 63.16
V: 66.90
65.1.
V: 70.42
0.50
Avg: 75.67
Avg: 71.25
Avg: 74.30
Avg: 73.80
Avg: 77.28
R R@1 1:111 79.23
R@1: 78.40
R@I: 1: 82.20
R@ 1: 81.82
R@ I: 84.03
V: 65.27
V: 62.89
V: 67.60
V: 65.74
V: 70.17
0.75
Avg: 72.25
Avg: 70.65
Avg: 74.90
Avg: 73.78
Avg: 77.10
@@ 80.80
R@1 78.13
@@ 78.89
R@ 1: 84.21
R@ I: 84.64
V: 66.84
V: 61.55
V: 60.67
V: 70.88
71.47
1.00
Avg: 73.82
Avg: 69.84
Avg: 69.78
Avg: 77.55
Avg: 78.06
Table 10: Quantification of collapse in the latent space with (w/) and without (w/o) Lcu-
Gap
Cos TP
AV (t)
AV (v)
T2V R@I
V2T R@!
w/o Leu
0.08
0.76
0.122
0.091
30.86
31.64
w/ Lcu
0.09
0.58
0.001
0.005
37.5
38.67
the LCU component. In contrast, performance appears less sensitive to variations in A1, suggesting smaller but still complementary contribution of the LATP term. Additionally, without Leu, the LATP only makes representations latent space collapsing in a small region of the space, as show in Fig. II thus limiting the expressiveness of the space in downstream tasks. Involving both the proposed losses, instead, builds a more compact and well-aligned latent space, as the third plot in Fig. shows.
quantitative measure of this collapse is also given in Tab. 6 where we report the gap, the cosine true pairs, and the additional Angular Value (AV) per modality. This metric measures the intra-modal average cosine similarity. across the hypersphere. A value of this metric higher than 0 indicates that all the embeddings are very close to each other, while a value of 0 means that the intra-cosine similarity ranges from -1 to 1, indicating good sparsification of the latent space. We conduct these experiments on the MSCOCO dataset with two ResNet50 encoder backbones. Table/6|show shows that without Lcu the space built by solely the InfoNCE and LXTP losses tends to have pairs closer (Cos TP higher) the cost of lower sparsification (AV higher). This directly impacts retrieval results as the space much more condensed and it is harder for the model discriminate between matching and non-matching pairs.
indicates how much the embeddings of a single modality are spread
Promoting centroids uniformity vs overall uniformity. Wang & Isola (2000) proved and empirically demonstrated that the conventional InfoNCE loss can be decomposed into two objectives, one that pulls together embeddings from similar samples, and another that spreads all embeddings evenly on the hypersphere avoid collapse and improve generalization. The latter (i.e., the embeddings spread) applies to all embeddings regardless of the class. However, enforcing uniformity at the sample level tends to scatter representations arbitrarily, potentially disrupting the tight alignment that the alignment term aims to enforce. For this reason, we adapt the uniformity principle to the multimodal setting by applying it not to individual embeddings, but to the centroids of aligned samples, that is, to the average representations of semantically matching pairs across modalities. This design retains the benefits of uniform coverage of the latent space while avoiding crucial interference with alignment. The Leu loss encourages the centroids of aligned multimodal samples to be well- separtted on the hypersphere, effectively ensuring that different semantic concepts remain distinguishable. At the same time, the LXTP loss guarantees that all modalities representing the same semantic concept tightly
19

---

Published as a conference paper at ICLR 2026
Loc
Exro Cru
Latent space visualization of different learned spaces in ablation studies, with the precise effect of each of the proposed losses. Experiments with ResNet50 from scratch on MS COCO dataset.
Figure 11:
grouped around their centroid. To prove the limitations of the uniform spread of all embeddings, perform an ablation study on the CIFAR1O 10 dataset, in which we apply the uniformity loss proposed in Wang & Isola (2020) and our Lcu loss. As it is clear from Tab. [I] without any other changes, the proposed Lcu improves the performance of both retrieval (R @ 1) and clustering (V-Measure) over the uniformity loss of Wang & Isola (2020).
Table 11: Comparison between uniformity and centroid-based uniformity loss.
Method
R 1
V-Measure f
Luniform Wang & Isola (2020)
82.47
64.72
Lcu (ours)
84.64
71.47
A.5 LIMITATIONS
While our method effectively reduces the modality gap and enhances group-wise alignment across multiple modalities, it does not directly address potential limitations arising from highly imbalanced datasets or scenarios where semantic alignment ambiguous or weakly defined. Indeed, we observe in our experiments, which contain datasets with diverse modalities imbalances, different impact of the proposed method. While MSR-VTV and MSCOCO have one diverse matching sample for each modality (i.e., each image corresponds to single textual description in MSCOCO and similar in MSR-VTT), so there is no imbalance between modalities, CIFARIO and AV-MNIST have a different structure. AV-MNIST contains three modalities, where images and audio have the same size, while only 10 diverse textual captions are present in the dataset, making this modality slightly imbalanced. What is more, the CIFARIO datasets contains only 60k images but only 10 testual captions corresponding to AA photo of class". The CIFAR10 is the most imbalanced dataset in our set of experiments. Coming outperforms previous ones in all the experiments, the increase in the CIFAR10 is the most tight (+0.9 in V-Measure), followed by the AV-MNIST (+4.4 in V-Measure), likely due to the imbalanced modalities structure. Indeed, we find the highest improvements in the balanced datasets, in which we got +10.65 V-Measure for MSCOCO and +8.8 in MSR-VTT.
our method tests, we can observe that, even though our method
Moreover, although validate approach on both bimodal and trimodal benchmarks, extending the evaluation to more complex or underexplored modality combinations remains future work.
20


Reviews:
[{"review_id": "Uh9GkCjUFx", "reviewer": "Reviewer_4NhU", "summary": "The authors observe that modality gap does not necessarily affect fine-grained tasks like retrieval in some cases but could affect group-level tasks like clustering in multimodal learning. Based on this, they design a loss function which encourages groups of positive samples to be compactly embedded, while uniformly dispersing negative samples on the hypersphere. They evaluated their approach on several instance-wise and group-wise tasks using standard datasets.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The observation that modality gap does not necessarily affect fine-grained tasks like retrieval in some cases but could affect group-level tasks like clustering is a novel contribution.\n\n2. The idea of applying cluster-based contrastive losses to close the modality gap in multimodal models is an interesting.\n\n3. The motivation of the paper is clear and the authors do a good job conveying intuitions for some of their claims.\n\n4. Indeed as the authors claim, their empirical results confirm that reducing the modality gap improves group-level tasks like clustering, while leaving tasks like retrieval relatively unaffected in the cases considered.", "weaknesses": "1. In Section 3.2, the authors mention that one of the reasons contributing to modality gap is the use of networks with different random initialization for different modalities? How about the scenario where weights are shared across modalities (all modalities are encoded via the same network), since that is the way several widely used multimodal models are trained? Furthermore, although the authors claim this to be one of the sources of modality gap, their solution to resolving it only revolves around modified loss functions, and no attempt to align initialization weights across encoders.\n\n2. The way Section 3.2 is currently organized, it seems as if different random initializations for different modalities is the only source of modality gap. The point about the gap arising from the limitations of the CLIP loss does seems to be made suddenly in the second paragraph and does not follow naturally from the first. I would suggest reorganizing it so as to first enumerate the sources of modality gap, followed by descriptions of how they might be related / affect each other etc., if such relationships exist.\n\n3. In the second paragraph of Section 3.2, authors claim the existence of what they call \"semantic stripes\" in the representation space of CLIP. Although I can intuitively understand what they are, I would still like to know if the authors have any concrete evidence, theoretical or empirical, to establish their existence. Without any such evidence, such claims seem rather arbitrary.\n\n4. The idea of performing contrastive learning by contrasting cluster centroids is not particularly novel, as research along similar lines is being conducted for several years in the self-supervised learning literature [d, e].\n\n5. It is necessary to investigate further the purpose and effect of ATP. If I understand correctly, ATP is meant to encourage the formation of clusters and prevent positive pairs from being far apart. However, this is not stated explicitly, and there is no term in ATP, unlike the CU loss, involving cluster centroids.\n\n6. It is not entirely true that eliminating modality specific features is guaranteed to leave instance-level tasks like retrieval unaffected. There are several works in the literature [f, g, h] that point to the contrary based on the fact that in many retrieval tasks, there are modality-specific features which could be informative about the downstream task and the instances being compared. The assumption may hold for the experimental settings considered here, but is certainly not generally true.\n\nReferences:\\\n[a] Ngiam et al., \"Multimodal Deep Learning\", ICML 2011.\\\n[b] Hu et al., \"Towards Unsupervised Sketch-based Image Retrieval\", BMVC 2022.\\\n[c] Rastegar et al., \"MDL-CW: A Multimodal Deep Learning Framework with Cross Weights\", CVPR 2016.\\\n[d] Caron et al., \"Unsupervised Learning of Visual Features by Contrasting Cluster Assignments\", NeurIPS 2020.\\\n[e] Caron et al., \"Deep Clustering for Unsupervised Learning of Visual Features\", ECCV 2018.\\\n[f] Jing et al., \"Cross-Modal Center Loss for 3D Cross-Modal Retrieval\", CVPR 2021.\\\n[g] Chaudhuri et al., \"Cross-Modal Fusion Distillation for Fine-Grained Sketch-Based Image Retrieval\", BMVC 2022.\\\n[h] Ren et al. \"Cross-modal retrieval based on multi-dimensional feature fusion hashing\", Frontiers in Physics 2024.\n\nMinors:\\\nLine 142: \"in narrow different cones\" -> \"in different narrow cones\"\\\nLine 207: \"suppose to retrieve a cat image caption. The sufficient condition...\" -> \"suppose to retrieve a cat image caption, the sufficient condition...\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zyumtfntqy", "reviewer": "Reviewer_h2Ko", "summary": "This paper reviews the modality gap in multimodal representation learning, providing insights into the impact of this phenomenon on downstream tasks. By a series of experiments, this paper observes that the modality gap has a limited influence on instance-level tasks such as retrieval. However, it would significantly affect group-wise tasks like clustering. To this end, this paper proposes a new objective function to explicitly align true pairs while promoting latent space uniformity.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper clearly revisited an important distinction between instance-wise and group-wise tasks in multimodal learning, providing a new perspective on the modality gap phenomenon.\n2. The theoretical analysis offers a reasonable explanation for why the modality gap affects clustering performance but not retrieval, with the derivation showing how the gap inflates within-class scatter.\n3. The paper provides thorough experimental validation across diverse datasets and modalities, with clear visualizations of the latent space before and after closing the gap.", "weaknesses": "1. The core idea of combining alignment and uniformity losses is not fundamentally new. Some methods previously established that contrastive learning involves alignment and uniformity, and explored closing the modality gap with similar objectives. The specific combination of LATP and LCU doesn't represent a significant methodological advance.\n2. The paper states that \"closing the modality gap\" is necessary for group-wise tasks, but the marginal improvements on some datasets (like CIFAR-10) suggest the gap may not be the primary limiting factor in many cases. \n3. While the paper claims to achieve \"nearly zero gap,\" it doesn't address whether completely closing the gap might lead to other issues like reduced discriminative power between different concepts or increased vulnerability to modality-specific noise.\n4. This paper mentions that \"the entire latent space collapses into small portions\" without the LCU loss, but the ablation studies don't clearly quantify this collapse. The authors should provide visualization or metrics showing the extent of collapse when using only LATP versus the full approach.\n5. The paper claims that \"closing the modality gap\" is the key to improving group-wise tasks. However, could the observed improvements be due to other factors, such as the increased expressiveness of the latent space from your loss formulation rather than specifically from closing the gap?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uIsvP4IIJH", "reviewer": "Reviewer_d752", "summary": "This paper studies the *modality gap* in multimodal contrastive learning — the separation that naturally arises between different modalities (e.g., image and text) in shared embedding spaces. The authors argue that this gap, while having little effect on instance-wise retrieval, negatively impacts *group-wise* semantic alignment such as clustering or class-level organization. To address this, they propose two additional objectives: **Align-True-Pairs** ($L_{\\text{ATP}}$) to directly align paired samples, and **Centroid-Uniformity** ($L_{\\text{CU}}$) to promote balanced structure across modalities. When combined with InfoNCE, these losses explicitly reduce the modality gap. Experiments on several bi- and tri-modal datasets demonstrate improved clustering performance and maintained retrieval accuracy.", "questions": "1. I would like to clarify a conceptual point: the paper states that the canonical goal of representation learning is to organize the representation space according to semantic structure. I would instead define the canonical goal as *learning representations that capture the task-relevant structure of the data to enable effective downstream learning*. While this may often result in semantic organization, it is not necessarily the goal itself. I’m curious to hear the authors’ perspective on this distinction.  \n2. Have you analyzed the sensitivity of the results to the weighting of $L_{\\text{ATP}}$ and $L_{\\text{CU}}$?  \n3. Could the proposed objectives interact with the temperature $\\tau$ in the contrastive loss, or could temperature scaling alone reduce the modality gap?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a timely and underexplored problem in multimodal contrastive learning — the **modality gap** — and offers an insightful analysis of its effects.  \n2. The distinction between *instance-wise* and *group-wise* consequences of the modality gap is both novel and intuitively well-motivated.  \n3. The proposed loss functions are simple, differentiable, and easily integrated into existing CLIP-style frameworks.  \n4. The experimental validation is extensive, spanning multiple bi- and tri-modal datasets and evaluating with diverse metrics such as $V$-Measure, cosine similarity, and retrieval accuracy.  \n5. The paper is **clearly presented** and well organized, with intuitive figures and a coherent narrative connecting the modality gap, angular separation, and semantic alignment.", "weaknesses": "## Major  \n1. **Limited theoretical grounding.**  \n   - The main theoretical contribution — that the modality gap influences group-wise semantics — is persuasive but remains largely qualitative. A more formal treatment (e.g., linking it to the alignment–uniformity trade-offs in contrastive theory) would strengthen the argument.  \n   - The principal practical novelty appears to be Equation (8), as Equation (7) corresponds to the standard “alignment” term introduced by Wang & Isola (2020). Although the proposed term performs well empirically, the paper would benefit from a clearer intuition or justification for it — ideally from an information-theoretic or probabilistic perspective, as is common in recent contrastive learning research.  \n\n2. **Incremental methodological contribution.**  \n   Previous works (e.g., Liang et al., 2022; Schrodi et al., 2025) have already analyzed or attempted to reduce the modality gap. The proposed losses can thus be seen as practical refinements rather than fundamentally new formulations. Nonetheless, emphasizing *group-wise* effects remains an interesting and original perspective.  \n\n---\n\n## Minor  \n1. In Equation (3), the summation index should start at 1, not 0.  \n2. The meaning of the “semantic class” $c$ in Equation (3) is unclear — the notation suggests a single sample, which makes the formulation ambiguous. It would also be better to avoid reusing the symbol $c$, given that it is later used to denote centroids.  \n3. Equation (6) should be accompanied by a proof or derivation, ideally included in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A3zrtSwoxB", "reviewer": "Reviewer_pS85", "summary": "This paper proposes an approach to address the modality gap by introducing two complementary loss functions. The first, $\\mathcal{L}\\_{\\text{ATP}}$ directly aligns modalities by minimizing the Euclidean distance between true pairs. The second,$\\mathcal{L}\\_{\\text{CU}}$ makes semantic concepts (represented by their centroids) sparse and uniformly spread throughout the latent space.\n\nThese two losses are combined as $\\mathcal{L}\\_{\\text{gap}}$ and added to the standard contrastive loss. This method functions by pulling embeddings of the same concept from different modalities together ($\\mathcal{L}\\_{\\text{ATP}}$) while simultaneously pushing the centroids of different concepts apart ($\\mathcal{L}\\_{\\text{CU}}$).\n\nThen, this leads to reducing modality gap, not only with 2 modality like CLIP, but additional modalities within 3 or 4 modalities, which extend into general settings, improves performance in downstream tasks.", "questions": "1. (Related to the major weakness) What are the benefits of reducing the modality gap beyond merely improving model performance? While prior works have suggested that narrowing this gap leads to better VLMs or multimodal representations based on empirical results, a deeper discussion is needed. The authors should clarify why reducing the gap is inherently advantageous, rather than preserving some degree of modality-specific uniqueness.\n2. The relative strength of the proposed $\\mathcal{L}_{\\text{gap}}$ compared to existing methods, such as OTI, OVI, or CoMM, remains unclear. Therefore, additional experiments are likely required to demonstrate whether this approach provides a novel or complementary benefit when applied alongside other methods. Comparisons do not need to be limited to OTI, OVI, or CoMM; any reasonable alternative methods may be used.\n3. As mentioned in the Limitations (Appendix A.3), the method may struggle when datasets are highly imbalanced across modalities. As a minor concern, it would be valuable to include additional experiments or analyses evaluating performance under varying degrees of modality imbalance (e.g., via controlled imbalance scenarios or dataset subsampling). The choice of datasets for such experiments can be flexible.\n\n=======================================================\n\n**Note**: I acknowledge that I may have partially misunderstood certain aspects of the paper. Therefore, I am willing to raise my rating score if these questions and concerns are adequately addressed.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Recent studies have sought to interpret why CLIP-based models inherently exhibit a gap between modalities (e.g., intra-modal misalignment [1]). This paper clearly defines this problem and proposes an approach to mitigate the modality gap by grouping clusters with similar concepts to enhance intra-modal alignment and disentangle their representations.\n\n- In addition, the paper provides both qualitative and quantitative evidence supporting the effectiveness of the proposed method (e.g., qualitative results in Fig. 2 and Fig. 6; quantitative results in Fig. 3 and Fig. 4). The demonstrated reduction of the modality gap also leads to improved performance on downstream tasks, consistent with the paper’s main objectives.\n\n&nbsp;\n\n[1] Mistretta, Marco, et al. \"Cross the gap: Exposing the intra-modal misalignment in clip via modality inversion.\" ICLR 2025\\", "weaknesses": "**Major** \n\nIn my understanding, there remains a fundamental concern about whether reducing the modality gap is always beneficial [1][2]. For instance, in an image–text pair where the image contains additional contextual elements (e.g., trees) not mentioned in the text, the discrepancy reflects complementary rather than misaligned information. Specifically, let consider an image–text pair where the image depicts a dog playing with a frisbee under trees (Image Modality), while the text annotation reads *“The dog is playing with a frisbee.”* (Text Modality). In this case, the image contains additional, unique information (e.g., the presence of trees) that the text does not. This suggests that a certain degree of modality discrepancy can be natural or even desirable, as each modality may capture complementary aspects of the scene [3]. However, the paper assumes that reducing this gap is inherently optimal for multimodal learning but does not provide some justifications for this claim, just relying mainly on partial empirical observations (e.g., Fig. 6; Tables 6 and 7).\n\n[1] Yaras, Can, et al. \"Explaining and mitigating the modality gap in contrastive multimodal learning.\" CPAL 2025\\\n[2] Schrodi, Simon, et al. \"Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language representation learning.\" ICLR 2025\\\n[3] Sammani, Fawaz, and Nikos Deligiannis. \"Interpreting and analysing CLIP's zero-shot image classification via mutual knowledge.\" NeurIPS 2024\n\n**Minor** \n\n- The paper appears to address a slightly trivial problem — primarily focusing on reducing the modality gap — without presenting a sufficiently distinctive or theoretically grounded contribution. It seems to be incremental contribution.\n- Limited comparison: The paper lacks comprehensive comparisons with other relevant approaches, such as  OTI/OVI [1] and CoMM [2].\n\n[1] Mistretta, Marco, et al. \"Cross the gap: Exposing the intra-modal misalignment in clip via modality inversion.\" ICLR 2025\\\n[2] Dufumier, Benoit, et al. \"What to align in multimodal contrastive learning?.\" ICLR 2025\n\n\n(**Note**: *The authors do not need to respond to these weaknesses, but to the questions listed below.*)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)