Generalized Contrastive Learning for Universal Multimodal Retrieval
Authors: Jungsoo Lee
Janghoon Cho
Hyojin Park
Munawar Hayat
Kyuwoong Hwang
Fatih Porikli
Sungha Choi
Venue: NeurIPS 2025 poster ()
Keywords: multimodal Retrieval
unified representation space
generalized contrastive learning
Primary area: applications
TL;DR: The paper introduces Generalized Contrastive Learning (GCL), a novel loss function that enhances multimodal retrieval performance by leveraging existing image-caption datasets.

Abstract: Despite their consistent performance improvements, cross-modal retrieval models (e.g., CLIP) show degraded performances with retrieving keys composed of fused image-text modality (e.g., Wikipedia pages with both images and text). To address this critical challenge, multimodal retrieval has been recently explored to develop a unified single retrieval model capable of retrieving keys across diverse modality combinations. A common approach involves constructing new composed sets of image-text triplets (e.g., retrieving a pair of image and text given a query image). However, such an approach requires careful curation to ensure the dataset quality and fails to generalize to unseen modality combinations. To overcome these limitations, this paper proposes Generalized Contrastive Learning (GCL), a novel loss formulation that improves multimodal retrieval performance without the burdensome need for new dataset curation. Specifically, GCL operates by enforcing contrastive learning across all modalities within a mini-batch, utilizing existing image-caption paired datasets to learn a unified representation space. We demonstrate the effectiveness of GCL by showing consistent performance improvements on off-the-shelf multimodal retrieval models (e.g., VISTA, CLIP, and TinyCLIP) using the M-BEIR, MMEB, and CoVR benchmarks.

Generalized Contrastive Learning for Universal Multimodal Retrieval
Jungsoo Lee Janghoon Cho Hyojin Park Munawar Hayat
Kyuwoong Hwang Fatih Porikli
Sungha Choi
Qualcomm Al Research"
(jungsool, janghoon, hyojinp, hayat, kyuwoong, fporikli, sunghac) cou.aaccc..t....clcco.moommmmm
Abstract
Despite their consistent performance improvements, cross-modal retrieval models (e.g., CLIP) show degraded performances with retrieving keys composed of fused image-text modality (e.g,, Wikipedia pages with both images and text). To address this critical challenge, multimodal retrieval has been recently explored develop a unified single retrieval model capable of retrieving keys across diverse modality combinations. common approach involves constructing new composed sets of image-text triplets (e.g.. retrieving pair of image and text given query image). However, such an approach requires careful curation to ensure the dataset quality and fails to generalize to unseen modality combinations. To overcome these limitations, this paper proposes Generalized Contrastive Learning (GCL), a novel loss formulation that improves multimodal retrieval performance without the burdensome need for new dataset curation. Specifically, GCL operates by enforcing contrastive learning across all modalities within mini-batch, utilizing existing image-caption paired datasets to learn a unified representation space. We demonstrate the effectiveness of GCL by showing consistent performance improvements on off-the-shell multimodal retrieval models (e.g. VISTA, CLIP, and TinyCLIP) using the M-BEIR, MMEB, and CoVR benchmarks.
Introduction
With the growing availability multimodal data, the ability to retrieve relevant keys across different modalities has become increasingly important. While cross-modal retrieval, retrieving images with given text or vice versa, has garnered significant attention and progress [ performing retrieval with fused image-text modality still remains challenge (9 10] Consider, for instance, the task of finding a Wikipedia page composed of both images and text (e.g.. the Eiffel Tower paired with its history) in response to query (e.g.. "What is the history of the Eiffel Tower?"). For such real-world multimodal retrieval scenarios, existing approaches deliver limited performance.
3 5
6 8],
This performance drop stems primarily from the pervasive modality gap 11 13 14,
15|
16
17],
critical barrier in retrieval systems. The modality gap arises when semantically similar samples across different modalities (e.g., an image and its caption) exhibit low similarity in the embedding space, while semantically dissimilar samples within the same modality appear misleadingly close [18]. For example, although an image of teddy bears is paired with the annotated caption 'a photo of teddy bears', may be embedded closer to images of other animals in the representation space, rather than to its corresponding caption. This misalignment becomes especially acute when retrieval keys- spanning text-only, image-only, or fused text-image formats are stored in a unified database [9] For example, search query might ideally match Wikipedia page with both images and text. However, without a shared representation space to bridge the three different modalities, models struggle to pinpoint the most relevant candidate, which undermines their effectiveness.
"Qualcomm AI Research is initiative of Qualcomm Technologies, Inc. indicates corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Cross-modal Alignment
Multimodal Alignment
CLIP pair and triplets datasets (e-g, synthesized]
VISTA
GCL (Ours) pair dataset
pair dataset
Image
Text
T2IT triplet dataset
Embedding Model
Embedding Model
Embedding Model
Embedding
Model
Image
image
Loss applied
Multitzsk
Similarity matrix
Similarity matrix
Similarity matrix
Similarity matrix
Overview of GCL. Given embedding model pretrained for cross- moosl alignment, previous studies (e.g.. VISTA (19) constructed new triplet datasets to simulate specific multimodal retrieval scenarios. However, this approach limits generalization unseen retrieval scenarios (white squares). In contrast, GCL improves retrieval performance across diverse scenarios (black squares). Specifically, by utilizing off-the-shelf image-caption datasets, GCL enables the learning of retrieval tasks involving nine different modality combinations.
Figure 1:
To tackle this issue, recent research has extensively studied multimodal retrieval, which extends cross-modal retrieval by allowing searches across various modality combinations, including data
samples that contain both images and text 19 20 9 21 22 23
24 25 10 26). As demonstrated in
Fig. previous studies have attempted to improve multimodal retrieval performance by generating specialized datasets tailored to specific retrieval scenarios [19 20). For example, VISTA [19] finetuned pretrained cross-modal retrieval model with newly generated datasets composed of triplets: 1) IT2I dataset which includes queries of image-text pairs with candidate images and 2) T2IT dataset which includes query text with candidates of image-text pairs. While this approach can be effective for the targeted retrieval scenarios, it requires meticulous curation to ensure the quality of the generated samples (e.g., verifying that the generated images accurately represent the intended content). Moreover, models trained on these composed datasets may fail to generalize to unseen modality combinations beyond those encountered in the training sets (e.g., retrieving fused text-image samples when provided with corresponding text-image queries). Fig. shows that the model only learns the retrieval scenarios included in the datasets (black squares), leaving the other combinations (white squares) unlearned during the finetuning phase. While such studies resort to generating new datasets for learning specific retrieval scenarios, little has been explored fully utilize off- fff-the-shelf image-caption paired datasets for improving multimodal retrieval performances.
To this end, we propose Generalized Contrastive Learning (GCL), simple yet effective loss function that enhances multimodal retrieval performance by leveraging existing image-caption paired datasets, sidestepping the need for costly dataset construction. Specifically, GCL integrates three types of embeddings text embeddings, image embeddings, and fused text-image embeddings and applies contrastive loss across all modalities within a mini-batch learn unified representation space. As shown in Fig. [ GCL encourages positive pairs from different modalities to be pulled closer together while pushing apart all negative pairs, regardless of modalities. This training process enables the retrieval model to learn retrieval between all combinations of modalities, which was limited to certain pairs the previous methods that generated specific triplets. Despite its simplicity. GCL consistently improves multimodal retrieval performances across diverse tasks and datasets and outperforms model trained with newly composed triplet datasets. The key advantage of GCL is that it does not require expensive dataset curation and can generalize well to various multimodal retrieval scenarios.
The major contributions of this paper are:
We propose Generalized Contrastive Learning (GCL), a novel contrastive learning approach that improves multimodal retrieval by integrating different modalities (text, image, and fused text-image) within mini-batch for building unified representation space.
Unlike previous methods that rely expensive and manually curated composed datasets, GCL effectively leverages existing image-caption paired datasets, making it a cost-efficient and scalable solution for multimodal retrieval.
We show that GCL significantly enhances multimodal retrieval performance on diverse benchmarks (e.g., M-BEIR [9) MMEB [27], and CoVR (28) and retrieval models (e.g., VISTA 19]. CLIP II and CLnY 4 ) showing its broad applicability and effectiveness.

---

13
3 17
Cosine similarity between
averaged vectors of each modality
Modality Pair
t>>
Est
Eiz
VISTA
0.8799
0.7393
0.6772
VISTA GCL (Ours)
0.8682
0.9688
0.8740
VISTA
GCL (Ours)
Figure 2: PCA visualization of representation spaces using e,, et, and egt We use MSCOCO for e; (red) and e, (blue) and WebQA for e (green). We sampled 2K samples from each modality, using 6K samples in total. e indicates the average embedding vector each modality.
2 Related Work
2.1 Cross-modal Retrieval and Contrastive Learning
Cross-modal retrieval has been widely studied to enable searches across different modalities, such as retrieving images based on text queries or vice versa [T]
3
8. By leveraging
contrastive learning with large-scale image-text pairs, both image and text embeddings are mapped into a shared representation space [T. However, retrieval models trained in this manner still suffer from the modality gap, discrepancy between image and text embeddings even with the same
semantics 12
16 17 18].
To address this issue, recent studies have proposed diverse techniques to reduce the modality gap [12 [. For instance, AlignCLIP introduces an intra-modality separation loss, which pushes apart samples within the same modality to improve cross-modal alignment [[]] While this approach helps mitigate the modality gap, it is mainly designed for cross-modal retrieval and does not explicitly handle data samples that contain both images and text (e.g., social media pages with both images and text descriptions). As result, its effectiveness multimodal retrieval scenarios remains limited.
2.2 Multimodal Retrieval
Multimodal retrieval builds upon cross-modal retrieval by supporting searches across different combinations of modalities, including samples that incorporate both images and text [19 20 9 29
30 22] fusing representations for CLIP-based models, termed foorinfusion (SF) 99. The fused embeddings enable the retrieval of data samples that contain both images and text.
UnilR demonstrates that adding image and text embeddings is effective for
As mentioned previously, one common approach for multimodal retrieval is generating composed datasets containing paired image and text samples tailored for specific retrieval scenarios. For example, by using image generation models, VISTA generated the IT2T dataset, which consists of image and text queries with text-based keys, and T2IT dataset, which includes text queries with both image and text keys [19]. Similarly, MegaPairs constructed triplets consisting of two images and a descriptive text capturing the relationship between them, which are generated with multimodal large language models (20). While dataset generation through large generative models can be effective under certain scenarios, it requires careful curation to ensure data quality, making the process labor-intensive and computationally expensive. Additionally, the retrieval model trained with such generated datasets may fail to generalize to scenarios unseen during the training phase. Therefore, a more efficient and scalable approach is needed to enhance multimodal retrieval performance without relying on expensive dataset generation.
3 Method
3.1
Problem Setup
In this paper, we define (x,, æ;)) pair of image and text used train retrieval model 0, which consists of an image encoder 0 and a text encoder 9, The extracted embeddings of images and text are denoted as e; = (2:)) and e, (2z),, respectively. During inference, the model retrieves candidate samples e when given a query q.
In cross-modal retrieval, the model retrieves text candidates e for image query & (g=+t;)) and vice versa (q-- ())) The multimodal retrieval task generalizes this setting by incorporating samples that contain both images and text, denoted as zu. This results in more complex retrieval scenarios,
3

---

lu
17,
ITz
17v
Ti
T
Tu
Positive Pair
4
4.7.
4.7
4.7
Image-Text Paired Dataset
Embedding Model
Negative Pair
4
Ti
15 1.5
Masked
Figure 3: Training process of GCL. Given a dataset composed of image-caption pairs, we extract eg et, and Cg. For Cat, we follow the extraction method used by the retrieval model (e.g., VISTA and CLIP-SF). Then, we integrate samples of the three different modalities into a single mini-batch for contrastive learning. We mask out the supervision on the positive samples with identical modalities.
where candidates of arbitrary modalities are retrieved based on queries of arbitrary modalities, such as C 9it +i and 9ie +4g. Following UnilR 9 we consider two different retrieval settings: (1) global setting, retrieving candidates from shared database regardless of modalities and tasks, and (2) local setting, retrieving candidates from task -ask-specific database with the same modality. We conduct experiments on both settings in this paper.
Figure 2 clearly illustrates the challenge addressed in this work. The left side of Figure presents a PCA [4 visualization of the embedding spaces learned by VISTA and VISTA fine-tuned with our proposed loss function, Generalized Contrastive Learning (GCL). For this visualization, we use c and Ct from MSCOCO and est from WebQA. As shown, VISTA fails to construct unified embedding space for the three different modalities. The main reason that retrieval models trained on image-caption paired datasets fail to learn a shared embedding space that incorporates ext-
The table on the right further supports this observation. By using the same samples from the visualization, we compute the average embeddings of images, text, and image-text pairs, denoted as &, êt and Bix, respectively. We then calculate the cosine similarity between the average embedding vectors of each modality. As demonstrated in the table, VISTA exhibits low cosine similarity across modalities, indicating significant modality gap. Our goal is to mitigate this gap, as evidenced by the more intermixed scatter plots in the PCA visualization and the increased cosine similarities between modalities after finetuning with GCL.
3.2 Generalized Contrastive Learning
Using N samples per mini-batch, the standard contrastive learning loss is formulated
1
expl(e) c6))/]] Dk=1 expl(ed e
LeL
log
2N
j=1 (a.b)ES
where j and & denote the sample indices, is the temperature scaling parameter, and S denotes the set of modality pairs {(i, f), 1))- While this loss function has been effective in constructing a unified cross-modal representation space, multimodal retrieval performance remains limited within this embedding space. The main limitation arises because models are not explicitly trained with embeddings that represent data samples containing both images and text (e.g., news articles with both pictures and textual explanations). As mentioned earlier, previous studies created new datasets consisting of triplets simulate specific retrieval scenarios for addressing such limitation [19] 20 In contrast, our proposed method does not rely on these new triplet- based datasets.
Fig. 3 describes method well. Generally, multimodal retrieval models obtained by finetuning cross-modal retrieval models, which are pretrained with image-caption paired datasets. Instead of finetuning the cross-moda retrieval models with newly constructed triplet-based datasets, finetune them using off-the-shelf image-caption paired datasets with our proposed Generalized Contrastive Learning (GCL) loss function. In GCL loss, negative samples are constructed from all possible combinations of embeddings (pink squares), including 1) image embeddings es, 2) text embeddings e,, and 3) fused embeddings of images and text Cjt. obtain the fused embeddings of samples with both images and text, we follow the extraction method used in the retrieval model. For example, we either (I) use a specialized architecture pretrained for extracting fused embeddings for VISTA
For the experiment, the checkpoint of VISTA prior training with newly generated composed sets.

---

(e.g., appending visual tokens alongside text tokens as input to text encoder [19),, or (2) sum the individual image and text embeddings, Eit = e + e, for CLiP-based models following UnilR (.. The positive pairs are then defined as samples with different modalities from the same pair (green squares). Positive pairs with the same modality correspond to the sample itself, so they are masked out during training (gray squares). Using the image emhedding query as an example, the positive pair can be either the corresponding text embedding or the fused image-texi embedding. GCL loss can be formulated as:
1 6N
explic' c///]] j=1 (a.b)EP EMEM Ex expl(ea -/T]
N
Lea.
log
where M represents the set of modalities {i,l, it) and P denotes the set of positive modality pairs ((i,t), (i, it), i), (1, it), (it, ().. The factor 6N normalizes the total loss across all six modality combinations in P, with each combination contributing losses from N training samples.
Although AlignCLIP [1]] recently introduced an intra-modality separation loss to push negative image samples away from a given image query in order to reduce the modality gap, it does not incorporate contrastive learning across all possible combinations, particularly those involving fused embeddings eşe- In contrast, our GCL loss provides a more generalized contrastive learning framework by seamlessly integrating different modalities into a unified representation space within single mini-batch, leading to improved multimodal retrieval performances across diverse scenarios and tasks (results shown in Table 5).
4 Experiments
4.1 Experimental Settings
Benchmarks We evaluate the effectiveness of our proposed GCL using standard multimodal retrieval
benchmarks: M- M-BEIR[(], MMEB (27. and CoVR [28] While images used as input for M-BEIR and MMEB, CoVR handles video input. For M-BEIR, we conduct evaluations on 10 datasets under both local and global evaluation settings. For MMEB, perform experiments on 12 sub-datasets included in the retrieval benchmark. Regarding CoVR, following the original evaluation setting of CoVR, we sample 15 frames for each target video and average the embeddings of each frame to obtain a single visual embedding for a given video. For the image-caption paired dataset used during finetuning, we use the LLa Visual Instruct Pretrain LCS-558K dataset (35), in which personal information has been blurred for data sanitization. Note that our experiments are conducted in a zero-shot setting, meaning the model is not fine-tuned the training set of the evaluation benchmark.
Models We apply the GCL loss to recent multimodal retrieval models, VISTA [9). CLIP []] and TinyCLIP [4] For VISTA, use the checkpoint before the second stage, the one trained without the generated dataset to demonstrate that our method can enhance multimodal retrieval performance even without newly composed triplet datasets. Throughout the paper, we refer to this checkpoint as VISTA. Following UnilR [. we adopt score-level fusion for fused embeddings when using CLIP-based models, referred as CLIP-SF and TinyCLIP SF for CLIP and TinyCLiP, respectively.
Baselines We compare the effectiveness GCL with that of pretrained model and standard contrastive learning. For VISTA, we also compare ours with VISTA finetuned using the generated datasets composed of triplets (i.e., IT2I and T2IT) used in the original paper, which we denoted as Cl.+Triple.
Metries Following prior work p 27 we adopt the standard retrieval evaluation metric, Recall @ K. Following UniIR, we set K=5 for the local setting, except for Fashion200K and FashionIQ, where we use K=10, and set K=50 for the global setting. We set K=1 for MMEB and K=1, 5, 10, and 50 for Further details are provided in the Supplementary.
CoVR, following the previous work 27 28
4.2 Quantitative Evaluation
Table I demonstrates that applying GCL. consistently improves multimodal retrieval performance in the global setting of M-BEIR across various tasks and models. Notably, the performance improvement is particularly significant for tasks involving qit or cat. Even without generating new data samples composed of it, GCL loss improves tasks related to it with off-the-shell image-caption paired datasets. While VISTA trained with generated datasets (i.e., IT2I and T2IT) shows the best performance for
P BEIR and CoVR datasets are under the MIT license, and MMEB is under the Apache-2.0 license. VVISTA CLIP, and TinyCLIP are all under the MIT license.
5

---

Table 1: Comparisons global setting of M-BEIR using Recall CL and GCL indicates standard contrastive learning and our generalized contrastive learning, respectively. Triplet and Pairwise refers to training with newly composed triplet dataset and original image-text text paired dataset, respectively.
VISTA
CLIP-SF [
Task
GCL (Ours) +Pairwise
GCL (Ours)
Pretrained
Triplet
+Pairwisc
Pretrnined
+Pairwisc
+Pairwise
VisualNews MSCOCO
5.36
1.64
16.64
0.08
0.00
6.70
1. g + c3
3r
2.72
5.60
14:42
38.85
0.00
0.90
3.25
Fashion2000K
RK
0.00
0.00
0.00
4.25
0.00
0,00
0.00
2. g: -2.0
WebQA
96.25
3.
(e)
WehQA t
EDIS
25.15
14.22
44.37
80.88
36.90
31.74
64.00
49.06
23.39
19.87
68.42
34.19
54.43
40.62
News M6
1.35
0.08
1.18
4.71
0.00
0.00
2.48
4.
MSCOCO 37
12:90
0.50
26.82
60.32
0,00
0.00
24.84
Fashion200K X8
0.72
5. dg
NIGHTS 611
76,60
83.07
79.39
82.50
81.65
88.07
85.09
(gs, -
OVEN 22
8.72
0.00
InfoSeek 45
2.94
4,80
1.70
9.07
0.80
0.00
1.86
7. (9r ui) -
FashionlQ 44
6.66
16.41
6.10
10.88
11.61
0.00
4.25
CIRR
23.62
43.81
31.13
8. (Wrr)) + (c).c))
InfoSeek
OVEN
43
34.31
30.95
14.94
9.67
29.82
32.83
34.97
32.92
11.04
12.73
0.00
0.58
21.89
19:47
Avg.
21.18
25.28
24.65
34.06
14.92
17.52
21.89
Table Comparisons on MMEB dataset using Recall @ following VLM2Vec 27 as in Table !
Abbreviations
VISTA
1
CLIP-SF
Task
Dataset
CL +Pairwise
GCL (Ours) +Pairwise
CL +Pairwise
GCL (Ours)
Pretrained
+Triplet
Pretrained
+Pairwise
VisDial 46
10.1
17.3
17.2
16.6
22.5
27.2
31.1
VisualNews
38.4
50.5
70.5
1. g: + cz
MSCOCO
37
32,8
44.8
46.8
48.7
54.9
60.7
61.5
Wiki-SS-NQ
12.4
14.7
16.7
46.5
WebQA
30
65.9
83.9
73.3
79.5
61.1
73.7
62.8
9e
EDIS 21
64.6
78.2
78.5
85.4
VisnalNews MSCOCO
10
54.6
25.7
52.7
54.2
1.5
10.9
9:
-
44.0
32.9
55.3
52.8
2.0
0.1
23.1
g. +
NIGHTS
64.1
65.4
66.4
CIRR 45
8.1
14.1
9.0
11.2
10.9
46
11.6
FashionI
9.0
7.7
9.9
16.5
6.2
6. 9u Cu
OVEN
54.3
45.4
53.6
57.3
46.1
53.8
Avg.
40.3
37.7
43.4
44.9
39.3
29.9
44.2
tasks qa - and c it shows limited performance gain or performance drop with other tasks. That is, finetuning retrieval models with generated samples under certain scenarios may show promising performance for the targeted scenarios, but they may fail to generalize to the tasks unseen during the finetuning phase. It also shows degraded performance and di (c;) compared to the pretrained VISTA. We conjecture that further finetuning VISTA with composed sets targeting specific retrieval scenarios may degrade its original performance cross- modal tasks due to forgetting its initial cross- modal alignment. We want to emphasize that our goal is to perform well across wide range of tasks and datasets, not just to excel at specific task dataset.
cross-modal tasks (gi+c
Tables 2 and compare the multimodal retrieval performances under the local setting. Again, finetuning retrieval models with GCL. brings further performance improvements even under the local setting. Along with the global setting, we believe that performing well in the local setting is also important since may need databases separately divided for each task depending we pursue in the real-world applications. Although there may exist a slight performance drop in scenario-specife retrieval tasks (e.g. CIRR and FashionIQ), this can largely be attributed to the nature of the fine-tuning dataset used. The LCS-558K dataset is designed for general- which may not fully capture the nuances of domain-specite specifcc tasks. To achieve optimal performance in these specialized applications, we believe GCL serves as an effective initial training stage, and performance can be further improved through additional fine-tuning with task-specific data.
the cases
fine-tuning,
Table demonstrates that applying GCL also improves the video retrieval performance. When deploying retrieval models in real-worr real-world scenarios, visual content may be stored in video formats

---

Table 3: Comparisons local setting of M-BEIR. We report the results using Recal for the local setting except using Recall 10 for Fashion200K and FashionlQ, following UnilR Abbreviations in Table I
VISTA 19
CLIP-SF
CL.
GCL (Ours) +Pairwise
CL +Puirwsse
GCL (Ours)
Pretrained
Triplet
Painwise
Pretrained
Pairrise
NisualNows
16.04
10:01
20.97
36.71
1.
MSCOCO 37
50.65
58.40
61 34
61.09
61.09
71.94
67 67.69
Foshion200K
38
9.51
8.03
9.83
9.54
6.57
8.84
7.04
2. 9: -2.0%
WehQA 19
91.20
91.20
89.37
20.35
3. (e).
EDIS
36.69
40.98
35.76
45.88
43.29
34,56
48.97
WebQA
74.51
62.49
44.01
VesualNews 36
14.03
4.42
13:35
13.70
41.78
20.18
30.53
4.
MSCOCO Bxi
61.66
60.44
71,98
72.56
79.00
85.78
79.04
Fashion200K
6.71
5. o
NIGHTS 41
26.32
26.32
28.21
28.35
26.13
30.94
30.99
6. (g: Gt
OVEN 42
25.93
31.82
InfoSeek 3
29.87
23.16
28.47
34.26
0.29
0.00
6.78
7. (gg. 9:) CL
FashionIQ
2.43
9,03
2.25
5.00
6.95
11.48
5.28
CIRR
10.60
21.82
11.34
14.27
13.19
37.84
15.85
OVEN +X
37.45
31.11
35
40.60
19.94
0.37
31
InfoSeck
43
Avg.
30.18
32.53
31.49
38.56
28.51
29.51
30.42
Table 4: Comparisons on CoVR Benchmark. Following CoVR, we use the frame of middle index for the query video, while averaging uniformly sampled frames for the target video. Abbreviations as in Table [
VISTA 19
CLIP-SF 9
Rank
CL Puirwise
GCL (Ours) +Pairwise
CL Pairwise
GCL (Ours) +Pairwise
Pretrained
Pretrained
R@I
31.22
33.76
37.52
37.32
19.68
37.60
R@S
58.37
59.74
63.46
62.60
40.30
65.69
R G0
15
69.52
72.81
71.99
50.67
75.78
R@50
88.50
88.50
91.12
88.18
74.92
92.92
(e.g., detecting unexpected actions in CCTV), making video retrieval an important task due to its practicality. Consistent performance improvements in multimodal retrieval tasks even including video retrieval demonstrates that GCL robust and versatile approach for enhancing retrieval models across diverse scenarios.
5
Further Analysis
5.1 Ablation Studies
5 compares GCL. with the modra- separation loss proposed in AlignCLIP [T] while dissecting the contributions of the individual loss components of GCL. La25 indicates the loss function of GCL using a as the query modality and as the target modality from given positive pair. Regarding intra-modality separation loss, added the loss term in addition to standard contrastive learning during training. For the ablation study of GCL,, we excluded each of the following key loss functions: 1) cross-modal alignment terms (Lez and Lizi). 2) it-candidate learning terms (Lrzit and L12st). and 3) it-query learning terms (Cuzs and Luzs). For the comparisons, we use the global setting of M- M-BEIR. Results the local setting of M-BEIR and performance variance of multiple runs are included in our Supplementary.
As shown, adding the intra-modality separation loss indeed improves the multimodal retrieval performance compared to training with standard contrastive learning. However, we observe that the performance gain is limited for tasks involving retrieval with identical modalities (e.g., 4- -c and 4: (c2) or queries with it modality (e.g., qit and (;)));) compared to our GCL loss. This indicates that intratmodality separation loss mitigates the modality gap but it fails to consider diverse multimodal retrieval scenarios, which are effectively addressed by GCL.
Regarding the ablation study, we observe performance drop on the task that each module of GCL loss responsible for. To be more specific, by excluding Liz1 and L125, the performance on cross-modal tasks are degraded significantly. Also, the performances on tasks of ge ca are degraded after excluding Li2st and Lraa- We want to emphasize that we did not perform an extensive hyperparameter search for finding the optimal weighing values for each loss function in GCL. While
7

---

Table 5: Ablation studies on loss functions and comparisons with intra-modality separation loss using global setting of M-BEIR.
Intra-modality Separation
GCL w/o Coge. Lrze
GCL w/o Lyga. Lrza
GCL w/o Lazis Carz
Task
Dataset
CL
M
GCL
VisualNews 36
9.29
14.36
2.91
18.26
17.07
16.64
1. -
MSCOCO 37
14.42
36.67
9.77
39.43
38,99
38.85
Fashion200K
Q: -7
WebQA
96.17
97.68
96.21
96.25
EDIS 40
36,90
49.74
45.23
37.15
49.18
49.06
3. 9: (c)
WehQA 39
31.74
47.59
69.53
52.93
61.65
64.00
VisualNews 36
1.88
2.78
0.59
5.50
4.63
4.71
9: -
MSCOCO
26.82
48.64
13.4
63.06
58.60
60.32
Fashion200K
88
0.00
0.61
0.08
0.76
0.63
0.72
5. gi CL
NIGHTS 41
79.39
78.02
79.15
83.21
82.83
82,50
6. ((:)0)) +
OVEN 42
3.10
5.23
6.92
9.77
7.95
8.72
InfoScek 13
1.70
3.72
9.10
10.96
7.79
FashionlQ
6.10
6.33
9.48
11.76
10.61
10.88
CIRR 45
24.27
23.57
32:52
31.82
30,50
31.13
8. (9.))) (et,
OVEN 42
32.83
35.74
36.75
29.20
31.22
32.92
InfoSeek
29.82
34.26
40.26
33.31
32.27
34.97
Avg.
24.65
30.45
28.36
32.95
33.42
34.06
Table 6: Performance improvements M-BEIR under global setting with TinyCLIP.
Metric
VISTA
CL.PP-F
TinyCI.P-SF
TinyCI.IP-SF GCL.
Model Params.
196M
427M
120M
120M
Avg. Inference (ms)
26.06
21.58
14.67
14.67
M-BEIR 21.18
14.92
17.36
22.71
this may improve performance certain tasks, it may accompany performance drops other tasks. Since the main goal of this paper is to design a loss function that generally works well for diverse retrieval scenarios, we simply added the loss functions with identical weights. Depending on the use cases, each loss function in GCL may be weighed differently.
5.2 Empowering Lightweight Models
Deploying lightweight models for retrieval is essential for enabling fast and efficient inference in real-time or sesource-constraing environments, such as mobile or edge devices (e.g.. retrieving personal data on mobile phones). Table 6 illustrates that applying GCL improves the retrieval performance of TinyCLIP compared to pretrained retrieval models, including VISTA and CLIP-SE, despite having fewer parameters and lower average inference speed (ms). This result indicates that fine-tuning small and lightweight retrieval model with GCL is viable solution for improving retrieval performance.
5.3 Ranks of Ground Truth Candidates
Figure compares the ranks of ground truth candidates between VISTA and VISTA trained with GCL. The x x-axis denotes the ranks, and the y-axis indicates its frequency. For this analysis, we use 9: from MSCOCO and a candidate pool composed of c and e; from MSCOCO, where the task to retrieve the ground truth e; given (/- The numbers of queries and candidates are 2.5K and 29K, respectively. We visualize only candidates ranked within the top 10K.
Our findings show that when VISTA trained with GCL, most ground truth candidates achieve high ranks, with the majority ranked within the top 500. In contrast, VISTA without GCL exhibits non-trivial number of ground truth candidates that are ranked significantly lower. This highlights the challenge of retrieving ground truth & from a shared database of mixed modalities when the modality gap is not effectively reduced. GCL successfully mitigated the modality gap. demonstrated by the high ranks of ground truth candidates.
5.4 Cosine Similarity with Candidates
Ground truth candidates Figure (a) visualizes the cosine similarity between the embeddings of queries and their corresponding ground truth candidates. For this analysis, we selected one dataset from each task in M-BEIR. As shown, applying GCL to VISTA consistently increases the cosine similarities across diverse tasks and datasets, indicating improved alignment between query and ground truth representations. By improving the representation space, GCL ensures that relevant

---

18K
VISTA
VISTA
Top 500
10K
Frequency
0.1K
0.3K
0.8K
5K-
After top 500
2K
BK
10K
Ranks
Figure 4: Rankings of ground truth candidates. The x-axis and y-axis indicate the ranks and the frequency of ranks, respectively. We use the task of qe=+c; on the MSCOCO dataset, with a candidate pool composed of and from MSCOCO.
0.9
0.7
Cosine Similarity
0.8
0.7
0.6
0.5
0.45
0.4
Voue Wetan 2018 Vosdy, MGHTS oven GAR over p-0
p-d (r P-1 (1-1)) (r-i)
==5
K K=
100
(a) Cosine similarity between
(b) Cosine similarity between query and top-K candidates
query and ground truth candidates
Figure (a) Cosine similarity between query and ground truth candidates. X-axis and y-axis indicates the dataset and cosine similarity, respectively. VisualN. refers to VisualNews. (b) Cosine similarity between queries and top-ranked candidates. We use MSCOCO for the task of -->
multimodal pairs whether text, image, or fused are positioned closer in the embedding space, leading to more accurate retrieval.
Top-ranked candidates Figure (b) further examines the cosine similarity between queries and their top-ranked retrieved candidates to evaluate the retrieval consistency. Using MSCOCO (g)+ we analyze how cosine similarity trends change across different ranks. As shown, VISTA exhibits significant drop similarity with lower ranks, suggesting that lower-ranked candidates less semantically relevant. In contrast, applying GCL helps maintain high cosine similarity even with lower ranks, demonstrating that relevant candidates still retrieved even with lower ranks.
This stability across ranks highlights the ability of GCL build a unified representation space more effectively, ensuring that even when the top candidate is not perfect match, subsequent retrieved items remain semantically meaningful, Such improvements are crucial for real- real-world multimodal retrieval applications where retrieving set of relevant candidates would be viable solution, rather than retrieving the single best match.
Conclusion
In this paper, we introduced Generalized Contrastive Learning (GCL), simple yet effective loss function designed enhance multimodal retrieval performance without the need for generating triplet datasets simulating certain retrieval scenarios. By integrating text, image, and fused text-image embeddings into the contrastive learning framework, GCL mitigates the modality gap and improves multimodal retrieval performance across diverse tasks and datasets. Although not discussed in this work, one promising future work direction integrating GCL with multimodal large language models (MLLMs) to further enhance retrieval capabilities in generative and reasoning -aaed tasks (23 21 22
50]. As MILMs continue to advance, utilizing retrieved information generate responses By eliminating the need for labor-intensive dataset curation while improving retrieval across arbitrary modality combinations, GCL presents scalable and effective solution for multimodal retrieval. We hope this work paves the way for future research in leveraging contrastive learning for more generalizable and robust multimodal retrieval.
48 49
would be promising, especially with databases containing

---

Acknowledgement We would like to thank Qualcomm Al Research team for their valuable discus- sions.
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[2] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le. Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation lcarning with noisy text supervision. In ICML, 2021.
[3] Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. Mobileclip: Fast image-text models through multi-modal reinforced training. In CVPR, 2024.
[4] Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengehen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi (Stephen) Chen, Xinggang Wang, Hongyang Chao, and Han Hu. Tinyelip: Clip distillation via affinity mimicking and weight inheritance. In ICCV, 2023.
[5] Young Kyun Jang, Junmo Kang, Yong Jae Lee, and Donghyun Kim. MATE: Meet the embedding connecting images with long texts. EMNLP, 2024.
[6] Chuanguang Yang, Zhulin An, Libo Huang, Junyu Bi, Xinqiang Yu, Han Yang. Boyu Diao, and Yongjun Xu. Clip-kd: An empirical study of clip model distillation. In CVPR, 2024.
[7] Haoyu Lu, Nanyi Fei, Yuqi Huo, Yizhao Gao, Zhiwu Lu, and Ji- Ri-Rong Wen. Cots: Collaborative two-stream vision- language pre- ve-training model for cross- modal retrieval. In CVPR, 2022.
[8] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Scyedhosseini, and Yonghui Wu. Coca: Contrastive captioners image-text foundation models. Transactions Machine Learning Research, 2022.
[9] Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and henchmarking universal multimodal information retrievers. ECCV, 2024.
[10] Lang Huang, Qiyu Wu, Zhongtao Miao, and Toshihiko Yamasaki. Joint fusion and encoding: Advancing multimodal retrieval from the ground up, 2025.
[11] Sedigheh Eslami and Gerard de Melo. Mitigate the gap: Improving cross-modal alignment in CLIP. In ICLR, 2025.
[12] Weixin Liang, Yuhui Zhang, Yongehan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurlPS, 2022.
[13] Peiyang Shi, Michael C. Welle. Mårten Björkman, and Danica Kragic. Towards understanding the modality gap in CLIP. In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023.
[14] Simon Schrodi, David Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language representation learning. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024.
[15] Siddharth Srivastava and Gaurav Sharma. Omnivec: Learning robust representations with cross modal sharing. In WACV, 2024.
[16] Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P. Calmon, and Himabindu Lakkaraju. Interpreting clip with sparse linear concept embeddings (splice). In NeurlPS, 2024.
[17] Yan Xia, Hai Huang. Jieming Zhu, and Zhou Zhao, Achieving cross modal generalization with multimodal unified representation. In NewrlPS, 2023.
[18] François Role, Sébastien Meyer, and Victor Amblard. Fill the gap: Quantifying and reducing the modality gap in image-text representation learning, 2025.
[19] Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, and Yongping Xiong. VISTA: Visualized text embedding for universal
retrieval. In ACL, August 2024.
10

---

[20] Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong. Megapairs: Massive data synthesis for universal multimodal retrieval, 2024.
[21] Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Gme: Improving universal multimodal retrieval by multimodal Ilms, In CVPR, 2025.
[22] Sheng-Chiel Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS.. In ICLR, 2025.
[23] Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng. Feng Sun. Qi Zhang, Deqing Wang, and Fuzhen Zhuang. ES-v: Universal embeddings with multimodal large language models, 2024.
[24] Woongyeong Yeo, Kangsan Kim, Soyeong Jeong, Jinheon Back, and Sung Hwang. Universalrag: netrieval-augmeenne generation over multiple corpora with diverse modalities and granularities. arXiv preprint arXi::254420034, 2025.
[25] Siwei Wu, Yizhi Li, Kang Zhu, Ge Zhang, Yiming Liang, Kaijing Ma, Chenghao Xiao, Haoran Zhang, Bohao Yang, Wenhu Chen, Wenhao Huang, Noura Al Moubayed, Jie Fu, and Chenghua Lin. Scim- mir: Benchmarking scientific multi- modal information retrieval. In Findings of the Association for Computational Linguistics: ACL 2024, 2024.
[26] Adriel Saporta, Aahlad Manas Puli, Mark Goldstein, and Rajesh Ranganath. Contrasting with symile: Simple model-agnostic representation learning for unlimited modalities. In NeurIPS, 2024.
[27] Ziyan Jiang. Rui Meng. Xinyi Yang. Semih Yavuz. Yingbo Zhou, and Wenhu Chen. VLM2vec: Training vision-language models for massive multimodal embedding tasks, In ICLR, 2025.
[28] Lucas Ventura, Antoine Yang, Cordelia Schmid, and Güil Varol. CoVR: Learning composed video retrieval from web video captions. AAAI, 2024.
[29] Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischinski. Data roaming and quality assessment for composed image retrieval. AAAI, 2024.
[30] Yiyang Chen, Zhedong Zheng. Wei Ji, Leigang Qu, and Tat-Seng Chua. Composed image retrieval with text feedback via multi-grained uncertainty regularization. In ICLR, 2024.
[31] Wei Li, Hehe Fan, Yongkang Wong, Yi Yang, and Mohan Kankanhalli. Improving context understanding in multimodal large language models via multimodal composition learning. ICML, 2024.
[32] Yongqi Li, Wenjie Wang, Leigang Qu, Liqiang Nie, Wenjie Li, and Tat-Seng Chua. Generative cross-modal retrieval: Memorizing images in multimodal language models for retrieval and beyond. In ACI., 2024.
[33] Zhengyuan Zhu, Daniel Lee, Hong Zhang, Sai Sree Harsha, Loie Feujio, Akash Maharaj, and Yunyao Li. MuRAR: simple and effective multimodal retrieval and answer refinement framework multimodal question answering. In Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations, 2025.
[34] L.T Jolliffe. Principal Component Analysis. Springer Verlag, 1986.
[35] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024.
[36] Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. Visual news: Benchmark and challenges in news image captioning. In EMNLP. Association for Computational Linguistics, 2021.
[37] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmick. Microsoft cocn: Common objects in context. CORR, 2014.
[38] Xintong Han, Zuxuan Wu, Phoenix X. Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S. Davis. Automatic spatially-aware fashion concept discovery. In /CCV, 2017.
[391 Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. In CVPR, pages 16495-16504, June 2022.
[40] Siqi Liu, Weixi Feng. Juu-Jui Fu, Wenhu Chen, and William Wang. EDIS: Entity-driven image search over multimodal web content. In EMNLP. Association for Computational Linguistics, 2023.
11

---

[41] Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. In NeurlPS, 2023.
[42] Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. CVPR, 2023.
[43] Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Was Wei Chang- Can pre-trained vision and language models answer visual information -eeiionnn-eeigggg questions? In EMNLP, 2023.
[44] Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad AI-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. The fashion iq dataset: Retrieving images by combining side information and relative natural language feedback. CVPR, 2021.
[45] Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Conditioned and composed image retrieval combining and partially fine-tuning clip- based features. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, pages 4959-4968, June 2022.
[46] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, and Dhruv Batra. Visual dialog. CVPR, 2017.
[47] Xueguang Ma, Cheng Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multimodal retrieval document screenshot embedding. In EMNLP, 2024.
[48] Davide Caftagni. Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Wiki-llava: Hierarchical ettrievll-uggmenn generation multimodal llms. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024.
[49] Sahel Sharifymoghaddam, Shivani Upadhyay, Wenhu Chen, and Jimmy Lin. Unirag: Universal re- trieval augmentation for large vision language models. In Findings of the Association for Computational Linguistics: NAACL 2025, 2025.
[50] Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, and Maosong Sun. VisRAG: Vision-based retrieval-augmentt generation multi-modality documents. In ICLR, 2025.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The contributions and the scope of the paper are well explained in the abstract and the introduction. For example, we summarized the contributions with bulletin points in the introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discussed the limitation of this work in the conclusion section by proposing possible future work direction. Specifically, while not included in our work, we believe that extending our work with multimodal large language models would be an interesting research directin.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
13

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not include theoretical assumptions and proofs in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We included the implementation details regarding our experiments in Section 4. I and our Supplementary. We also include the pseudocodes in our Supplementary in order help readers reproduce our algorithm.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
14

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We currently refactoring and cleaning the codes. We are planning release the codes after internal review of the codes is finalized.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.cm
htaiii//uuuides//eeeebmmiii wwtplimmsssoooooooooiooc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
httppp.cc/pubcccguuuceeee Codeeubmii ssi crrionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we specified the training and test details in our implementation details of Section 4.1 of the main paper and Supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Since we conducted extensive experiments, was challenging report results of multiple runs for all experiments. However, we reported error bars for the ablation study on each loss term of GCL in our Supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We described the computing resources including the type of GPU and memory consumed for the experiments in our Supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/neurips.. cc.pppscccp/ulttt / htt.a//pp......bbltttttttccccc.a
Answer: [Yes]
Justification:
We confirmed and conformed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discussed both potential positive societal impacts and negative societal impacts in our Supplementary.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Section 4.1 of the main paper, we mentioned that we blurred the facial images included in the train set for the sanitization of data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In the main paper, we denoted the licenses of each dataset in the footnotes.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
19


Reviews:
[{"review_id": "43ICbF8bNg", "reviewer": "Reviewer_g4sT", "summary": "The paper considers producing representations all combinations of image and text to improve retrieval from multimodal signals without splitting the modes. The idea is to use a standard contrastive objective for the individual mode embeddings and a joint embedding of the modalities from a paired image and caption dataset; GCL (generalized contrastive learning) also minimizes the alignment between any pairs of mismatched individual embeddings or joint embeddings or across both.", "questions": "In table 5, the biggest loss in average performance comes from removing single-mode pairs. This seems to stem mainly from single modal loss in performance. This presents an interesting trade-off between what is stored for retrieval and performance. To elaborate, compare the two cases\n\n1. For each doc, we store a different embedding based on what kind of query comes (retrieve  given one or two modes). Then we can get the best in each column in table 5. This still can be achieved without requiring triplet datasets by dropping appropriate terms in GCL. \n2. For each doc, we store the embedding given by the full GCL. You only get the performance of the last column which is not always the best. \n\nI think this is a fundamental trade-off because you cannot align the a bi-modal signal to each separately when there is information in each mode beyond what is explained by the other. Can the authors comment on when and where I would prefer 2 over 1?\n\nA related question is how GCL would/should be adapted in the context of multi-linear contrastive objectives (e.g. [this paper](https://arxiv.org/abs/2411.01053)) that can handle missingness?", "limitations": "See Weaknesses.\n\nI want to see bootstrap confidence intervals for at least a few results. I do believe the improvements will still be significant but it will help with the ablation study.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- The paper proposes a simple way to build representations for individual modes and also jointly such that there is alignment between any combination of modalities (figure 2 provides empirical intuition)\n- The single method provides strong empirical improvement in multimodal retrieval across different datasets and combinations of modes compared to existing methods. The important fact is that the retrieval did not have to get multimodal pairs (image + text 1 associated with image + text 2) to produce these improvements.\n- The ablations in section 5 show the value of removing certain terms from the GCL loss which helps uncover a little about where the improvements come from.\n\n\nWeaknesses\n- The space of terms combinatorially explodes as the number of modes increase. There is a lot of image-text multimodal data, so GCL as it stands is still valuable, but I do wonder how training can be easy with even the two modes considered.\n- Contrastive learning has an interpretation of learning a classifier that looks at the density-ratio between the joint of two modes and the marginals. That interpretation seems lost with GCL. Can the authors explain?\n- The experiments use existing pretrained multimodal retrieval models. Would the method work similarly well if starting from scratch?", "was_revised": true, "final_justification": "Strong paper. No serious concerns original. Discussion clarified a few things.", "rebuttal": "We sincerely appreciate the reviewer’s thoughtful and encouraging comments. The reviewer found our proposed method to be simple yet effective and appreciated the empirical intuition presented in Figure 2. We also thank the reviewer for highlighting the strong performance of our method across diverse datasets without relying on fully multimodal pairs, as well as the usefulness of the ablation studies in uncovering the contributions of individual loss components. We would like to note that Reviewer h6VU similarly praised the novelty of our loss function and the thoroughness of our experimental analysis. Regarding the concerns raised, we have carefully considered each point and addressed them below. All relevant revisions will be incorporated into the final version.\n\n**[W1] Scalability of GCL with Multiple Modalities and Its Practical Advantages**\n\nWe sincerely appreciate the reviewer’s insightful observation regarding the potential combinatorial complexity that may arise in multimodal learning scenarios. This is indeed a valid concern, particularly in settings involving a large number of modalities. However, current approaches in the field require the creation of dedicated datasets for every pairwise or higher-order modality combination, which entails a significant annotation burden. In contrast, while extending GCL to incorporate additional modalities may increase computational cost for the extended similarity map, we believe that it remains comparatively more efficient and practical than constructing new annotated datasets for each modality combination.\n\nA key advantage of GCL is its ability to leverage readily available image-text paired datasets, eliminating the need to manually construct complex triplet-based samples or design specialized datasets for certain retrieval scenarios. Within each mini-batch, GCL automatically generates negative examples by considering all pairwise combinations of image, text, and fused image-text representations. This reduces reliance on labor-intensive data curation and simplifies the training process while preserving the ability to learn across diverse modalities. Moreover, GCL can be readily applied to existing pre-trained cross-modal models without necessitating significant changes to model architectures or training pipelines. This simplicity and flexibility make GCL particularly well-suited for real-world applications, where scalability and ease of integration are critical.\n\n**[W2] Density-Ratio Interpretation in GCL**\n\nWe thank the reviewer for this deep theoretical question. We believe that GCL does not discard the density-ratio perspective; rather, it extends it to suit multimodal retrieval with three modalities: image (I), text (T), and fused image-text (IT).\nTraditional contrastive learning estimates the density ratio\n$\\frac{p(x, y)}{p(x)p(y)}$\nby contrasting positive pairs against negatives sampled from the marginal distributions. This is implicitly achieved by training a classifier that distinguishes positive pairs from negatives, where the output logits can be interpreted as approximating this density ratio.\n\nGCL generalizes this idea by simultaneously modeling the distribution of the density ratio across six combinations. As shown in Equation 2 of our main paper, for each of these positive pairs $(e^j_a, e^{j}_{b})$, the loss function encourages their similarity to be high relative to all other possible combinations of query $e^j_a$ with any other sample $e^k_m$ from any modality $m \\in$ {I,T,IT} within the mini-batch. This inherently means that GCL implicitly estimates a corresponding density ratio for each positive modality pair. From this perspective, we believe that the density-ratio intuition remains preserved in GCL while being extended to a greater number of modality combinations.\n\n**[W3] Effectiveness of GCL from Scratch**\n\nOur experiments utilize pre-trained models to highlight the effectiveness of GCL as a fine-tuning method that enhances existing systems without requiring new data generation. This design choice was made to emphasize the practicality of GCL and its ease of integration. In theory, GCL is also applicable when training from scratch, as it continues to promote consistent alignment across image, text, and fused representations. However, in practice, training from scratch would be considerably more demanding. Pre-trained models such as CLIP already provide strong vision-language representations learned from large-scale datasets like LAION-2B. GCL builds on this solid foundation to further refine alignment across modalities by reducing the modality gap present in such representation spaces. When training from scratch, the model must simultaneously learn both the basic vision-language associations and the more nuanced fused-modal relationships. This would require significantly larger datasets and greater computational resources. Therefore, while training retrieval models with GCL from scratch may be feasible, it introduces practical challenges that are not encountered in our fine-tuning-based approach.\n\n**[Q1] Trade-off Between Modality-Specific vs Unified Embeddings**\n\nAs the reviewer correctly pointed out, there exists a trade-off between storing embeddings for each modality separately and using a single unified representation space.\n\n**Strategy 1 (modality-specific embeddings)** is useful when the query modality is known in advance (e.g., a system dedicated solely to image-to-text or text-to-image retrieval). In such cases, it is possible to drop certain GCL terms during training and optimize for specific tasks. This setup aligns with our \"local setting\" experiments, which show that this approach often yields the best performance for a given task.\n\n**Strategy 2 (embeddings in unified representation space)** is designed for \"global setting\" use cases where the system must support retrieval with arbitrary types of queries and candidates (image, text, or fused). This is common in real-world systems such as search engines or knowledge base retrieval tools, where the query modality is unknown and can vary at runtime. In this case, having a single, unified embedding for each document greatly simplifies storage and retrieval, since only one index and one retrieval pass are needed regardless of the query type.\n\nWhile Strategy 2 may show slight performance drops in highly specialized retrieval tasks, it brings major benefits in flexibility, efficiency, and generalization. Therefore, the choice depends on the application: Strategy 1 is best for fixed and known tasks, while Strategy 2 (enabled by GCL) is better for real-world systems requiring universal retrieval across mixed modalities. We will include this valuable insight regarding the use cases of multimodal retrieval in our camera-ready version.\n\n**[Q2] Relation to Symile and Handling Missing Modalities**\n\nWhen applying GCL to Symile [1], we can generate embeddings for modality combinations that the study does not yet address. Although Symile considers three modalities, image, text, and audio, it omits the pairwise combinations of $it$, $ia$, and $ta$. GCL enables the construction of these embeddings and supports additional retrieval tasks, such as $ita$ $\\rightarrow$ $ia$ or $ita$ $\\rightarrow$ $i$, which are not included in the evaluation of this paper. We will cite Symile in the related work section of the revised manuscript and highlight how adopting these ideas can broaden the applicability of GCL.\n\n[1] Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities (NeurIPS 2024)\n\n**[L1] Performance Variance Across Multiple Runs**\n\nWe agree that variance estimates strengthen the experimental results. We reran two additional experiments (three in total) for GCL and its ablation study with different random seeds. As shown, we observe a similar tendency to that in the original manuscript, indicating that each tuple of loss term contributes meaningfully to the modality pairs it is designed to supervise. In the supplementary of the revised manuscript, we will report both the mean and standard deviation of performance across these runs.\n\n| Method                                                    | Score         |\n|-----------------------------------------------------------|---------------|\n| GCL w/o ($\\mathcal{L}{t2i}, \\mathcal{L}{i2t}$)       | 28.09 ± 0.24  |\n| GCL w/o ($\\mathcal{L}{t2it}$, $\\mathcal{L}{i2it}$)      | 32.85 ± 0.09  |\n| GCL w/o ($\\mathcal{L}{it2i}$, $\\mathcal{L}{it2t}$)      | 33.61 ± 0.17  |\n| GCL                                                       | 34.03 ± 0.13  |"}, {"review_id": "SiaqvBYhOO", "reviewer": "Reviewer_PYSB", "summary": "This paper proposes Generalized Contrastive Learning (GCL), a loss function designed to improve universal multimodal retrieval. The method targets a key weakness in standard cross-modal models like CLIP: their poor performance when retrieving composite items containing both images and text (e.g., a Wikipedia page). Instead of the costly approach of curating new triplet-based datasets for specific retrieval scenarios (e.g., text query to image-text key), GCL works by simply extending standard contrastive learning to a three-modality setting. Using existing image-caption datasets, it generates embeddings for the image (e_i), the text (e_t), and a fused image-text representation (e_it). It then applies a contrastive loss across all three modalities within a mini-batch to learn a more unified representation space. The authors demonstrate consistent performance gains on several benchmarks (M-BEIR, MMEB, CoVR) when applying GCL to off-the-shelf models like VISTA and CLIP.", "questions": "Please check the above for details.", "limitations": "Need more discussion on the possible drawback of the method.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n1. The targeted problem is important.\n2. The proposed idea is interesting and straightforward.\n3. The paper is overall easy to follow and well presented.\n\nCons:\n1. The authors claimed that this reduces the modality gap in Figure 2, which is attractive. However, there is no following up study on how the minimized modality gap translates to the improvement in downstream performance. Specifically, the datasets and evaluations used in [1] to demonstrate the harm of modality gap are not revisited to truly demonstrate the benefit of smaller modality gap.\n\n2. The results are not very convincing.\n- a. In Table 1, the baseline CLIP+Pairwise under CLIP-SF has much worse performance than the pretrained model, which is quite inconsistent with the the trend in Table 2 and Table 3.\n- b. In Table 2 and 3, the proposed method actually harms the performance on datasets like CIRR and FashionIQ, which indicates that the proposed is not that general and it might not be in an ideal formulation.", "was_revised": true, "final_justification": "The reviewer appreciates the authors' rebuttal.\n\nThe reviewer meant to say [12] in the original review but the reviewer thinks it is reasonable to expect the the settings where one of the first papers identified modality gap of CLIP embedding would be carefully revisited in a following-up work that claims mitigating the gap. Specifically, Section 5 in [12], including more serious study on relationship between modality gap measure and downstream tasks, as well as the performance in fairness setup.\n\nGiven this as well as the incoherent performance on some of the tasks without further justification in the settings like fine-tuning which the author mentioned, the reviewer cannot recommend accept. But the reviewer believe the manuscript is potentially important for the community once these issues are fully resolved.", "rebuttal": "We deeply appreciate the reviewer’s recognition of the importance of the addressed problem, the simplicity and clarity of our method, and the overall presentation of the paper. We are especially encouraged by the acknowledgment that the proposed reduction in modality gap is an attractive property. We would like to note that Reviewer g4sT similarly praised the benefits of aligning different modality combinations, as demonstrated in our empirical design (e.g., Figure 2). Regarding the concerns raised, we have carefully reviewed and addressed each point below, and all necessary changes will be faithfully incorporated into the final camera-ready submission.\n\n**[W1] Relationship with Modality Gap and Downstream Task Performance**\n\nWe appreciate the reviewer’s thoughtful observation about the modality gap in Figure 2 and the request for stronger evidence connecting it to downstream performance. We want to emphasize that one of our main goals is to retrieve the ground truth candidate from a database composed of samples with different modalities, which is referred to as the \"global setting\" in our work. In this context, embeddings from different modalities must reside in a unified representation space, making it essential to minimize the modality gap.\n\nTo further demonstrate our point, we analyzed the modalities of the top-100 retrieved candidates in the global setting of M-BEIR using CLIP-SF and report the ratio of each modality. We observe that the modalities of the top-100 candidates are mostly identical to the modality of the query, rather than the candidate. For example, when the query modality is text, the majority of the top-100 retrieved candidates are also text, even when corresponding paired images are available. This suggests that the modality gap inherent in the representation space of CLIP biases retrieval toward the same modality as the query, regardless of semantic alignment, which results in degraded performance in downstream tasks. We will include this analysis in the camera-ready version.\n\n| Query → Cand.        | Dataset | Image (\\%) | Text (\\%) | Image + Text (\\%) |\n|--------------------------|----------------|-----------|----------|------------------|\n| $q_t \\rightarrow c_i$    | VisualNews     | 0.00      | 97.52    | 2.48             |\n|                          | MSCOCO         | 0.00      | 99.97    | 0.03             |\n|                          | Fashion200K    | 0.00      | 100.00   | 0.00             |\n| $q_t \\rightarrow c_t$    | WebQA          | 0.00      | 99.84    | 0.16             |\n| $q_t \\rightarrow c_{it}$ | EDIS           | 0.00      | 95.04    | 4.96             |\n|                          | WebQA          | 0.00      | 96.73    | 3.27             |\n| $q_i \\rightarrow c_t$    | VisualNews     | 94.89     | 0.00     | 5.11             |\n|                          | MSCOCO         | 95.91     | 0.00     | 4.09             |\n|                          | Fashion200K    | 100.00    | 0.00     | 0.00             |\n| $q_i \\rightarrow c_i$    | NIGHTS         | 99.03     | 0.00     | 0.97             |\n| $q_{it} \\rightarrow c_t$ | OVEN           | 0.14      | 0.88     | 98.98            |\n|                          | InfoSeek       | 0.37      | 0.20     | 99.43            |\n| $q_{it} \\rightarrow c_i$ | FashionIQ      | 63.54     | 0.33     | 36.12            |\n|                          | CIRR           | 32.09     | 0.09     | 67.82            |\n| $q_{it} \\rightarrow c_{it}$ | OVEN        | 0.06      | 0.64     | 99.31            |\n|                          | InfoSeek       | 0.25      | 0.14     | 99.61            |\n\nAdditionally, this motivation aligns with recent work such as AlignCLIP [11], which demonstrates that reducing the modality gap leads to improved cross-modal retrieval performance. While AlignCLIP focuses on the cross-modal retrieval setting and does not address retrieval across arbitrary combinations of modalities (e.g., involving the $it$ modality), our work addresses such multimodal retrieval settings. We further show that minimizing the modality gap using GCL leads to strong performance in these richer multimodal retrieval settings. As shown in Table 5 of the original manuscript, GCL outperforms the intra-modality separation loss proposed in AlignCLIP under diverse multimodal retrieval settings. This suggests that GCL offers a more effective strategy for unifying representations across multiple modalities.\n\nWe will clarify this relationship between reducing the modality gap and improving multimodal retrieval performance in our camera-ready version. Additionally, since the reviewer did not specify the paper [1] in the initial review comment, we could not identify the datasets and evaluations used in [1]. However, we want to emphasize that our evaluation includes three comprehensive benchmarks, M-BEIR (16 subtests), MMEB (12 subtests), and CoVR (1 subtest), covering a total of 29 subtests. We believe this provides strong evidence of GCL’s generalizability and broad applicability to multimodal retrieval.\n\n[11] Mitigate the Gap: Improving Cross-Modal Alignment in CLIP (ICLR 2025)\n\n**[W2] Clarification on Results**\n\nWe thank the reviewer for carefully examining the results and raising two important points.\n\n(a) Regarding the inconsistent performance of CLIP + Pairwise: \n\nThe difference in performance trends between Table 1 (global setting) and Tables 2 and 3 (local setting) reflects a fundamental difference in retrieval context. CLIP-SF fine-tuned with pairwise data generally performs well in local settings, where candidates are retrieved only from the same modality and task, as shown in Tables 2 and 3. However, Table 1 shows that the representation space of CLIP is insufficient for global retrieval, where queries and candidates can come from arbitrary modality combinations. As further shown in the table above, which reports the modality ratios of the top-100 candidates, CLIP-SF in the global setting tends to retrieve candidates that share the same modality as the query, making it difficult to retrieve the ground truth candidates. We initially discussed such performance degradation of CLIP-SF in Supplementary Section B.\n\n(b) Regarding performance drops on CIRR and FashionIQ:\n\nWe acknowledge that GCL shows small performance drops on scenario-specific datasets like CIRR and FashionIQ. These datasets demand fine-grained and domain-specific understanding (e.g., fashion knowledge), which may not be fully captured by models fine-tuned on general-purpose image-text datasets such as LCS-558K. We want to clarify that our goal with GCL is not to specialize for every dataset but to provide a broadly applicable loss function that supports general multimodal retrieval across diverse settings. For example, as discussed in lines 203–206 of our manuscript, approaches that train with newly generated datasets targeting specific tasks (e.g., datasets for IT2I or T2IT) may work effectively for the targeted tasks but often fail to generalize to other tasks.\n\n**[W2, L] Discussing Limitations**\n\nWe agree with the reviewer that including a clear discussion of limitations is important.\nWe will clarify that GCL may underperform on scenario-specific retrieval tasks (e.g., CIRR and FashionIQ), particularly in zero-shot settings, as the reviewer rightly noted. While this is one of the limitations of our work, we emphasize that it stems from the training data used, not from the GCL loss itself. GCL is designed to align multiple modalities within a unified representation space, which is beneficial for general-purpose retrieval tasks. However, for optimal performance in domain-specific applications, GCL can serve as an effective initial training stage, followed by additional fine-tuning using task-specific or domain-relevant data. This hybrid strategy preserves the general alignment achieved by GCL while enabling the learning of fine-grained, task-specific features. We will include this limitation and its potential mitigation strategy in the revised manuscript to provide a more balanced perspective on the capabilities of GCL."}, {"review_id": "hwGr9P9Y6h", "reviewer": "Reviewer_kEbb", "summary": "This paper proposes Generalized Contrastive Learning (GCL), a loss function for multimodal retrieval that integrates text, image, and fused image-text embeddings within a single mini-batch for contrastive learning. The method aims to address the modality gap problem in retrieving keys with mixed modalities (e.g., Wikipedia pages containing both images and text) without requiring expensive triplet dataset curation. The authors demonstrate improvements on M-BEIR, MMEB, and CoVR benchmarks across VISTA, CLIP, and TinyCLIP models.", "questions": "Three fundamental questions require clarification before this work can be properly evaluated:\n\nWhy does the method fail on certain tasks despite using the same underlying principle? The substantial performance drops on tasks like WebQA (qt→ct) in Table 1 suggest the approach may not be as robust as claimed. What are the failure modes, and can they be predicted a priori?\n\nHow does computational cost scale with the number of modalities and batch size? The GCL loss requires computing similarities across all modality combinations within each batch, potentially creating a quadratic scaling issue. The authors provide no analysis of training time or memory requirements compared to standard contrastive learning.\n\nWhat is the theoretical basis for equal weighting of all loss components? Real-world datasets have varying amounts of data across modalities and differing task difficulties. The assumption that all six modality pairs should contribute equally to the loss seems unjustified and warrants either theoretical support or empirical validation through extensive hyperparameter search.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\nThe central premise of the work is reasonable - current cross-modal retrieval models struggle with fused modality combinations, and existing approaches that generate specialized triplet datasets are computationally expensive and fail to generalize.\n\n**Weaknesses:**\nThe method is relatively simple, but whether it is effective needs more experimental validation to support the claims and it is hard to find potential insight behind the method. This makes the work appear to be incremental work based on VISTA and lacks insights and experimental depth equivalent to VISTA.\n\nMy main concern is whether the NeurIPS community truly needs such incremental image-text matching work, even if the experiments are very comprehensive. This type of work might be more suitable for communities like MM.\n\nThe reproducibility is insufficient - there are surprisingly 0 pages of appendix, which is unacceptable for the NeurIPS community that requires highly reproducible work. For future three top-tier AI conference submissions, it is recommended to place the appendix after the main text.\n\nFrom Table 1, we can see that this method does not always work. For example, in the WebQA qt→ct task, GCL's performance is worse than the baseline. However, the authors did not add bold or underline to highlight this, attempting to hide this point.\n\n**Minor revision suggestions:**\n1. The text shape overlap in Figure 1 can be optimized, such as for the `fused text embedding`.\n2. The related work analysis is not comprehensive enough. It fails to serve the purpose of discussing the development trajectory, with many works crammed into single sentences.\n3. Figure 4 could save space by using half the space with such a figure layout.", "was_revised": true, "final_justification": "I've read the rebuttal and decided to raise my score by 1.", "rebuttal": "We appreciate the reviewer’s acknowledgment of the core motivation behind our work. This aligns with Reviewer PYSB’s appreciation of our problem framing and Reviewer g4sT’s recognition of the method’s effectiveness without relying on dense multimodal pairings. We have clarified points of misunderstanding and concerns and will incorporate all necessary changes into the final camera-ready manuscript.\n\n**[W1] Clarification on Novelty and Experimental Depth**\n\nWe appreciate the reviewer’s concern and agree that GCL is conceptually simple. However, we believe this simplicity is a strength of GCL because it is lightweight, broadly applicable, and easy to integrate into existing frameworks. Its core novelty lies in generalizing contrastive learning to a unified multimodal retrieval setting without requiring complex data curation (e.g., triplet synthesis), which was necessary in previous multimodal retrieval studies, including VISTA, as the reviewer noted. Furthermore, our approach is grounded in the observation that the modality gap remains significant, especially when the fused $it$ modality is incorporated, as shown in Figure 2 of the main paper.\n\nBelow are the detailed uniqueness and experimental depth of our work compared to VISTA:\n- Training with off-the-shelf image-caption datasets: While VISTA requires constructing task-specific datasets, GCL operates directly on standard image-caption datasets. This eliminates the need for labor-intensive dataset curation and demonstrates broader applicability.\n- Diverse Model Evaluations: GCL is applied not only to VISTA, but also to CLIP and TinyCLIP, demonstrating its architecture-agnostic nature. This highlights that GCL is not merely a VISTA add-on, but a general-purpose, plug-and-play loss function.\n- Broad Benchmarks: We evaluate across M-BEIR (16 subtests), MMEB (12 subtests), and CoVR (1 subtest), including the video retrieval task, which was not addressed by VISTA. We demonstrate the effectiveness of GCL across 29 subtests in total, while VISTA evaluated only 5 subtests.\n- Global and Local Retrieval: GCL consistently performs well across both global and local retrieval settings, while VISTA conducted experiments only under the local retrieval setting.\n\nWhile we referred to VISTA multiple times, this was to provide fair and direct comparisons with a recent state-of-the-art method that relies heavily on curated datasets. We will further clarify these distinctions in the revised version to better highlight the contributions and uniqueness of GCL along with the exhaustive experiments conducted in our work.\n\n**[W2] Suitability of our work for NeurIPS**\n\nFirst, we want to clarify that GCL is not limited to traditional image-text matching. Our main goal is to improve the representation space by reducing the modality gap, thereby enhancing universal multimodal retrieval performance overall. To this end, we believe that our work fits within the \"General Machine Learning\" category at NeurIPS. \n\nSecond, GCL demonstrates that a single objective function trained only on existing image-caption pairs can generalize to multiple modalities and tasks. This design aligns with the NeurIPS community’s interest in efficient, scalable, and generalizable machine learning frameworks. We believe GCL is well-aligned with these priorities. \n\nLastly, while not addressing the identical problem setup, we note that research focused on improving retrieval or embedding spaces has been well represented in recent NeurIPS [1,2,3], ICLR [4], and ICML [5] proceedings. Building on this precedent, we believe our work continues this line of inquiry while making a distinct contribution focused on multimodal retrieval through its broad applicability and empirical thoroughness. We respectfully suggest that GCL is well suited for and valuable to the NeurIPS audience.\n\n[1] Self-Retrieval: End-to-End Information Retrieval with One Large Language Model (NeurIPS 2024)\n\n[2] Multi-Lingual Acquisition on Multimodal Pre-training for Cross-modal Retrieval (NeurIPS 2022)\n\n[3] Large-Scale Retrieval for Reinforcement Learning (NeurIPS 2022)\n\n[4] MM‑Embed: Universal Multimodal Retrieval with Multimodal LLMs (ICLR 2025)\n\n[5] Contradiction Retrieval via Contrastive Learning with Sparsity (ICML 2025)\n\n\n**[W3] Missing Appendix and Reproducibility**\n\nWe would like to clarify that we had already submitted an 8-page comprehensive appendix as a separate file, following the NeurIPS guidelines. Our supplementary material includes pseudocode for GCL, detailed implementation settings, additional results and analyses, and compute resources. In fact, we already indicated in our initial submission that certain content is included in the supplementary (see lines 196 and 231, as well as Checklist items 4, 6, 8, and 10 in the main paper). Additionally, in supplementary Section B, we noted that our code is currently under legal review and will be released upon approval. Since the NeurIPS guidelines do not penalize authors for submitting the appendix separately from the main text, we believe this review comment stems from a simple misunderstanding of our submitted materials.\n\n**[W4, Q1] Performance Drop in WebQA and Formatting**\n\nFirst, we acknowledge the slight performance drop observed in WebQA ($q_t \\rightarrow c_t$) in Table 1. Our primary objective was to improve performance across a broad range of multimodal retrieval tasks without relying on dataset construction tailored to specific retrieval objectives. In this context, we believe the minor drop in WebQA (from 97.07 to 96.25) is acceptable, especially considering that the task may already be saturated and less receptive to gains from learning other tasks.\n\nSecond, we would like to clarify that we did not intend to hide this performance drop. The reviewer noted that we did not bold or underline the degraded score, implying an attempt to hide it. However, the standard practice is to highlight only the best-performing results per task, not to indicate drops. We have transparently reported all sub-task results, including those with degraded performance, and will explicitly discuss such cases in the limitation section of the final camera-ready version.\n\n**[Minor] Presentation (Figures and Related Work)**\n\nWe thank the reviewer for the helpful suggestions. We will revise Figure 1 to improve readability by reducing label overlap, especially for fused text embeddings, and redesign Figure 4 for more efficient use of space. We also agree that the related work section could better reflect the development trajectory and will expand it with a more structured discussion, moving less critical references to the appendix if needed.\n\n**[Q2] Computational Costs**\n\nImportantly, GCL introduces no additional memory or latency overhead during inference, as it modifies only the training loss function and leaves the model architecture unchanged. This ensures that the deployed model remains as efficient as those trained with standard contrastive learning. We will include a detailed analysis and supporting data in the final camera-ready version.\n\nDuring training, GCL does incur some additional computational and memory costs due to the need to compute similarities across a broader set of modality combinations. While standard contrastive learning typically computes similarities between two modalities (e.g., image-text), GCL extends this to include four additional modality pairs, such as fused combinations. Furthermore, GCL introduces an additional forward pass for the fused image-text ($it$) modality, which contributes to additional time overhead.\n\nTo quantify this computational overhead, we compared peak VRAM usage and per-iteration training time between standard contrastive learning and our proposed GCL. This analysis was conducted using a batch size of 128 on a single NVIDIA H100 GPU, averaged over 100 iterations with the VISTA model. The results show a modest increase in both VRAM usage and training time per iteration. However, we believe this overhead is well justified given the performance improvements GCL achieves across a wide range of tasks and benchmarks.\n\n| Method | Peak VRAM | Time     |\n|---|---|---|\n| Standard  | 24.45 GB  | 0.3325 s/iter |\n| GCL    | 45.78 GB  | 0.6074 s/iter |\n\n**[Q3] Equal Loss Weighting Justification**\n\nAs described in the original manuscript (lines 241–246), we chose to apply equal weights to each loss term to avoid overfitting to specific modality pairs or retrieval tasks. Our objective was not to tune the model for the best performance on individual datasets but rather to demonstrate the general applicability and robustness of the proposed GCL loss function.\n\nTo further validate this choice, we conducted an ablation study on loss weighting, varying the loss coefficients for three distinct tuples of loss terms: {($\\mathcal{L}{\\scriptstyle{t2i}}$, $\\mathcal{L}{\\scriptstyle{i2t}}$), ($\\mathcal{L}{\\scriptstyle{t2it}}$, $\\mathcal{L}{\\scriptstyle{i2it}}$), ($\\mathcal{L}{\\scriptstyle{it2i}}$, $\\mathcal{L}{\\scriptstyle{it2t}}$)}. For each tuple, we varied the coefficient values over {0, 0.1, 1, 10}, while keeping all other loss terms fixed at 1. Due to the space limit, we could only report the average scores of each experiment. If requested, we will also show the results of each sub test during the discussion period. \n\n|       | {1,1,1} | {0,1,1} | {0.1,1,1} | {10,1,1} | {1,0,1} | {1,0.1,1} | {1,10,1} | {1,1,0} | {1,1,0.1} | {1,1,10} |\n|---|---|---|-----|-----|------|----|----|-----|----|---|\n| **Average** | 34.06   | 28.36   | 34.59     | 33.02    | 32.95   | 33.65      | 34.02     | 33.42    | 33.39      | 34.50     |\n\nWhile certain weighting configurations yielded slight improvements, we found that equal weighting provides consistently strong results across different tasks and datasets. We will include the full results of this hyperparameter search in the camera-ready version."}, {"review_id": "Eu8MlOObbx", "reviewer": "Reviewer_h6VU", "summary": "The paper introduces Generalized Contrastive Learning (GCL), a novel loss function designed to improve multimodal retrieval performance by leveraging existing image-caption paired datasets without requiring costly curated triplet datasets. GCL integrates text, image, and fused text-image embeddings into a unified representation space through contrastive learning, addressing the modality gap and enabling retrieval across diverse modality combinations. The method is evaluated on benchmarks like M-BEIR, MMEB, and CoVR, showing consistent improvements across models such as VISTA, CLIP, and TinyCLIP.", "questions": "1. Clarify the process of (q_i, t_t) -> c_i. Could you elaborate on how the mapping from (q_i, t_t) to c_i is performed? \n\n2. Distinguish between (q_i, q_t) -> (c_i, c_t) and q_it -> c_it. What are the key differences between these two tasks? \n\n3. Consider using wikipedia articles or other easy-obtained datasets for experiments. Since multimodal wikipedia articles are readily accessible, why were they not used in the experiments? Their inclusion could provide additional validation and broader applicability.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The proposed loss function is novel and demonstrates improved performance over existing methods.\n\n- The paper provides extensive experimental results, facilitating a deeper understanding and thorough analysis of the method.\n\nWeaknesses:\n\n- The core idea is validated only on existing datasets, which were not specifically designed for this method. A specially constructed dataset containing challenging samples (e.g., q_it -> c_it) would further strengthen the contribution.\n\n- The loss function defined in Eq. (2) and illustrated in Fig. 3 appears relatively easy to optimize, as the positive pairs are derived solely from different modality combinations of the SAME sample, potentially limiting its robustness.", "was_revised": true, "final_justification": "The concerns have been addressed. And I keep change my rating.", "rebuttal": "We are grateful for the reviewer’s recognition of the novelty of our proposed loss function and the comprehensiveness of our experimental results. This feedback aligns with Reviewer g4sT’s acknowledgments of our method’s effectiveness, simplicity, and empirical depth. Regarding the concerns and questions raised, we have systematically addressed each point below, and all corresponding modifications will be incorporated into the final manuscript.\n\n**[W1, Q2, Q3] Clarification on Evaluation**\n\nWe appreciate the reviewer’s suggestion regarding the use of specially constructed datasets that are both challenging and well aligned with our proposed method (e.g., including the $q_{it} \\rightarrow c_{it}$ task). However, we would like to clarify that our experiments already include evaluations on $q_{it} \\rightarrow c_{it}$ retrieval tasks using existing datasets. Specifically, both OVEN and InfoSeek include tasks of this nature, as shown in Table 1 - Task 8: ($q_{i}, q_{t}$) $\\rightarrow$ ($c_{i}, c_{t}$). We confirm that GCL shows strong improvements in these settings.\n\nAlso, although the tasks ($q_{i}, q_{t}$) $\\rightarrow$ ($c_{i}, c_{t}$) and $q_{it} \\rightarrow c_{it}$ represent the same retrieval task, they were denoted differently in Tables 1 and 2. Recognizing this potential ambiguity, we will unify the notation as $q_{it} \\rightarrow c_{it}$ for clarity in our camera-ready version.\n\nRegarding the inclusion of Wikipedia in our evaluation, we clarify that our experimental setup already includes several datasets derived from Wikipedia articles and images. Details of the test sets using Wikipedia-derived samples are as follows:\n- WebQA: Uses Wikipedia pages (text and images) as retrieval candidates for answering multimodal questions.\n\n- OVEN: Provides visual recognition questions with Wikipedia images and corresponding summaries as reference targets.\n\n- InfoSeek: Evaluates VQA and retrieval on web-based multimodal content, including Wikipedia-derived samples.\n\n- Wiki-SS-NQ: A retrieval-based VQA dataset using Wikipedia page screenshots, offering richer context than plain text.\n\nThese demonstrate that the textual and visual content of Wikipedia is already well represented in our test sets. Additionally, the three benchmarks used in our work, M-BEIR (16 subtests), MMEB (12 subtests), and CoVR (1 subtest), covers 29 subtests in total. By showing consistent performance across these tests, we believe that we demonstrated the generalizability and broad applicability of GCL.\n\n**[W2] Optimization Simplicity of GCL Loss**\n\nWhile positive pairs in GCL originate from different modalities of the same semantic instance, this does not make the optimization trivial. In fact, the optimization remains non-trivial due to the inherent modality gap. As illustrated in the PCA visualization of Figure 2 in our main paper, even semantically aligned content (e.g., an image and its corresponding caption) often lies in distant regions of the representation space. The image-text fused modality introduces an additional alignment challenge, as it requires learning a unified space that accommodates all three modalities. If the GCL loss was trivial to optimize, we would expect these semantically related samples across modalities to be closely clustered in the representation space, which is clearly not the case. These observations underscore that GCL performs meaningful optimization across heterogeneous modalities and is essential for bridging the modality gap in multimodal retrieval.\n\n**[Q1] Clarification on $q_{i,t}$ $\\rightarrow$ $c_{i}$ retrieval setting**\n\nThe notation $q_{i,t}$ $\\rightarrow$ $c_{i}$ refers to a compositional image retrieval task. In this setup, the query is composed of an image and a modifying textual phrase, and the goal is to retrieve an image that reflects the modified concept. For example, if the query consists of an image of a red dress and the text \"is white with dots and a black belt,\" the target image $c_{i}$ would be an image of a \"white dress with dots and a black belt.\" Thus, the task requires compositional reasoning to interpret the modification and apply it to the visual concept. We will describe the details of each task in our camera-ready version."}]

Author rebuttal:

Meta-review: The authors propose Generalized Contrastive Learning (GCL), which extends the standard contrastive learning formulation to the multi-modal setting by using the contrastive objective with negative samples being all possible combinations of embeddings (e.g., image embeddings, text embeddings, fused embeddings) and positive samples being samples with different modalities from the same pair (i.e., contrastive learning is performed across all possible combinations). The resulting embedding is designed to improve universal multimodal retrieval without requiring constructing complete datasets (i.e., containing examples aligned across modalities) and can use existing datasets. Using GCL to augment VISTA, CLIP, and TinyCLIP, the authors demonstrate consistent performance improvements on subsets of widely used multimodal retrieval benchmarks (M-BEIR, MMEB, CoVR) over baseline models and those finetuned with generated data triplets. 

Strengths identified by the reviewers includes:
- GCL is a simple, yet effective way to construct embeddings for individual modalities in a joint space such that there is alignment between any combination of modalities. GCL is conceptually intuitive and doesn't require generating specialized datasets. Both the backbone model and datasets are essentially off-the-shelf.
- There has been a recent increased interest in multimodal embeddings, particularly in post-training LLM scenarios.
- GCL demonstrates strong empirical performance on relevant datasets against relevant baselines.
- The secondary experiments (e.g., ablations) help understand the relative contribution of different innovations and the overall dynamics of GCL.
- The paper is well-written and has a sufficient number of experiments to validate the performance for text-image retrieval settings (and pooled image embeddings for video settings).

Weaknesses identified by the reviewers includes:
- GCL is presented as a general multimodal embedding, but computing all combinations of modalities may be challenging at high-cardinality. However, in most practical settings, there is likely to be <5 modalities, so this isn't a practical concern (although maybe a bit of an over-claim). As stated in rebuttal, existing methods would likely require more effort in constructing the training data.
- In this vein, experiments with additional modalities would (obviously) strengthen the demonstrated utility of GCL.
- While novel, this work can be interpreted as a incremental innovation over existing multi-modal embedding papers and may be of primary interest to a more narrow community (e.g., multimodal since single-mode embeddings may perform better for single-mode tasks). Specifically, the authors are not defining a new problem nor providing new datasets, etc. -- so it is incremental work methodologically. While the formulation is general, the instantiation in this paper is more narrow.
- There were several more theoretical/analytical questions, possibly the most interesting being the relationship between the modality gap and downstream performance (reviewer PYSB). However, these were well addressed in rebuttal -- even if it would be good to see a version of the paper with these rebuttal incorporated into the main text. 
- GCL doesn't outperform competing models is all experimental settings (e.g., FashionIQ), but this isn't surprising and just needs to be discussed in the paper. 

Overall, as the authors state, GCL is a simple but effective method for constructing multi-modal embeddings. The experiments are exhaustive and sufficiently convincing for the text-image/video retrieval setting. The paper would be stronger with more modalities (likely not possible in this revision) and some of the theoretical and clarification discussion raised by the reviewers (likely possible as these were sufficiently addressed in rebuttal) -- especially since the method is so 'simple', more rigorous analysis is possible. The reviewers that engaged with the authors during rebuttal raised their scores (including h6VU which intended to do so) and the reviewers that did not had their concerns adequately addressed in my opinion. Thus, I view the paper as borderline, but lean toward accepting contingent on competing papers.
Decision: Accept (poster)