Title: Generalized Contrastive Learning for Universal Multimodal Retrieval
Venue: NeurIPS 2025
Accepted: yes
Keywords: multimodal Retrieval

unified representation space

generalized contrastive learning
Abstract: Despite their consistent performance improvements, cross-modal retrieval models (e.g., CLIP) show degraded performances with retrieving keys composed of fused image-text modality (e.g., Wikipedia pages with both images and text). To address this critical challenge, multimodal retrieval has been recently explored to develop a unified single retrieval model capable of retrieving keys across diverse modality combinations. A common approach involves constructing new composed sets of image-text triplets (e.g., retrieving a pair of image and text given a query image). However, such an approach requires careful curation to ensure the dataset quality and fails to generalize to unseen modality combinations. To overcome these limitations, this paper proposes Generalized Contrastive Learning (GCL), a novel loss formulation that improves multimodal retrieval performance without the burdensome need for new dataset curation. Specifically, GCL operates by enforcing contrastive learning across all modalities within a mini-batch, utilizing existing image-caption paired datasets to learn a unified representation space. We demonstrate the effectiveness of GCL by showing consistent performance improvements on off-the-shelf multimodal retrieval models (e.g., VISTA, CLIP, and TinyCLIP) using the M-BEIR, MMEB, and CoVR benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Generalized Contrastive Learning (GCL), a novel loss function designed to improve multimodal retrieval performance by leveraging existing image-caption paired datasets without requiring costly curated triplet datasets. GCL integrates text, image, and fused text-image embeddings into a unified representation space through contrastive learning, addressing the modality gap and enabling retrieval across diverse modality combinations. The method is evaluated on benchmarks like M-BEIR, MMEB, and CoVR, showing consistent improvements across models such as VISTA, CLIP, and TinyCLIP.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The proposed loss function is novel and demonstrates improved performance over existing methods.\n\n- The paper provides extensive experimental results, facilitating a deeper understanding and thorough analysis of the method.\n\nWeaknesses:\n\n- The core idea is validated only on existing datasets, which were not specifically designed for this method. A specially constructed dataset containing challenging samples (e.g., q_it -> c_it) would further strengthen the contribution.\n\n- The loss function defined in Eq. (2) and illustrated in Fig. 3 appears relatively easy to optimize, as the positive pairs are derived solely from different modality combinations of the SAME sample, potentially limiting its robustness.\n\nQUESTIONS:\n1. Clarify the process of (q_i, t_t) -> c_i. Could you elaborate on how the mapping from (q_i, t_t) to c_i is performed? \n\n2. Distinguish between (q_i, q_t) -> (c_i, c_t) and q_it -> c_it. What are the key differences between these two tasks? \n\n3. Consider using wikipedia articles or other easy-obtained datasets for experiments. Since multimodal wikipedia articles are readily accessible, why were they not used in the experiments? Their inclusion could provide additional validation and broader applicability.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Generalized Contrastive Learning (GCL), a loss function for multimodal retrieval that integrates text, image, and fused image-text embeddings within a single mini-batch for contrastive learning. The method aims to address the modality gap problem in retrieving keys with mixed modalities (e.g., Wikipedia pages containing both images and text) without requiring expensive triplet dataset curation. The authors demonstrate improvements on M-BEIR, MMEB, and CoVR benchmarks across VISTA, CLIP, and TinyCLIP models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\nThe central premise of the work is reasonable - current cross-modal retrieval models struggle with fused modality combinations, and existing approaches that generate specialized triplet datasets are computationally expensive and fail to generalize.\n\n**Weaknesses:**\nThe method is relatively simple, but whether it is effective needs more experimental validation to support the claims and it is hard to find potential insight behind the method. This makes the work appear to be incremental work based on VISTA and lacks insights and experimental depth equivalent to VISTA.\n\nMy main concern is whether the NeurIPS community truly needs such incremental image-text matching work, even if the experiments are very comprehensive. This type of work might be more suitable for communities like MM.\n\nThe reproducibility is insufficient - there are surprisingly 0 pages of appendix, which is unacceptable for the NeurIPS community that requires highly reproducible work. For future three top-tier AI conference submissions, it is recommended to place the appendix after the main text.\n\nFrom Table 1, we can see that this method does not always work. For example, in the WebQA qt→ct task, GCL's performance is worse than the baseline. However, the authors did not add bold or underline to highlight this, attempting to hide this point.\n\n**Minor revision suggestions:**\n1. The text shape overlap in Figure 1 can be optimized, such as for the `fused text embedding`.\n2. The related work analysis is not comprehensive enough. It fails to serve the purpose of discussing the development trajectory, with many works crammed into single sentences.\n3. Figure 4 could save space by using half the space with such a figure layout.\n\nQUESTIONS:\nThree fundamental questions require clarification before this work can be properly evaluated:\n\nWhy does the method fail on certain tasks despite using the same underlying principle? The substantial performance drops on tasks like WebQA (qt→ct) in Table 1 suggest the approach may not be as robust as claimed. What are the failure modes, and can they be predicted a priori?\n\nHow does computational cost scale with the number of modalities and batch size? The GCL loss requires computing similarities across all modality combinations within each batch, potentially creating a quadratic scaling issue. The authors provide no analysis of training time or memory requirements compared to standard contrastive learning.\n\nWhat is the theoretical basis for equal weighting of all loss components? Real-world datasets have varying amounts of data across modalities and differing task difficulties. The assumption that all six modality pairs should contribute equally to the loss seems unjustified and warrants either theoretical support or empirical validation through extensive hyperparameter search.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Generalized Contrastive Learning (GCL), a loss function designed to improve universal multimodal retrieval. The method targets a key weakness in standard cross-modal models like CLIP: their poor performance when retrieving composite items containing both images and text (e.g., a Wikipedia page). Instead of the costly approach of curating new triplet-based datasets for specific retrieval scenarios (e.g., text query to image-text key), GCL works by simply extending standard contrastive learning to a three-modality setting. Using existing image-caption datasets, it generates embeddings for the image (e_i), the text (e_t), and a fused image-text representation (e_it). It then applies a contrastive loss across all three modalities within a mini-batch to learn a more unified representation space. The authors demonstrate consistent performance gains on several benchmarks (M-BEIR, MMEB, CoVR) when applying GCL to off-the-shelf models like VISTA and CLIP.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n1. The targeted problem is important.\n2. The proposed idea is interesting and straightforward.\n3. The paper is overall easy to follow and well presented.\n\nCons:\n1. The authors claimed that this reduces the modality gap in Figure 2, which is attractive. However, there is no following up study on how the minimized modality gap translates to the improvement in downstream performance. Specifically, the datasets and evaluations used in [1] to demonstrate the harm of modality gap are not revisited to truly demonstrate the benefit of smaller modality gap.\n\n2. The results are not very convincing.\n- a. In Table 1, the baseline CLIP+Pairwise under CLIP-SF has much worse performance than the pretrained model, which is quite inconsistent with the the trend in Table 2 and Table 3.\n- b. In Table 2 and 3, the proposed method actually harms the performance on datasets like CIRR and FashionIQ, which indicates that the proposed is not that general and it might not be in an ideal formulation.\n\nQUESTIONS:\nPlease check the above for details.\n\nLIMITATIONS:\nNeed more discussion on the possible drawback of the method.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers producing representations all combinations of image and text to improve retrieval from multimodal signals without splitting the modes. The idea is to use a standard contrastive objective for the individual mode embeddings and a joint embedding of the modalities from a paired image and caption dataset; GCL (generalized contrastive learning) also minimizes the alignment between any pairs of mismatched individual embeddings or joint embeddings or across both.\n\nSTRENGTHS AND WEAKNESSES:\n- The paper proposes a simple way to build representations for individual modes and also jointly such that there is alignment between any combination of modalities (figure 2 provides empirical intuition)\n- The single method provides strong empirical improvement in multimodal retrieval across different datasets and combinations of modes compared to existing methods. The important fact is that the retrieval did not have to get multimodal pairs (image + text 1 associated with image + text 2) to produce these improvements.\n- The ablations in section 5 show the value of removing certain terms from the GCL loss which helps uncover a little about where the improvements come from.\n\n\nWeaknesses\n- The space of terms combinatorially explodes as the number of modes increase. There is a lot of image-text multimodal data, so GCL as it stands is still valuable, but I do wonder how training can be easy with even the two modes considered.\n- Contrastive learning has an interpretation of learning a classifier that looks at the density-ratio between the joint of two modes and the marginals. That interpretation seems lost with GCL. Can the authors explain?\n- The experiments use existing pretrained multimodal retrieval models. Would the method work similarly well if starting from scratch?\n\nQUESTIONS:\nIn table 5, the biggest loss in average performance comes from removing single-mode pairs. This seems to stem mainly from single modal loss in performance. This presents an interesting trade-off between what is stored for retrieval and performance. To elaborate, compare the two cases\n\n1. For each doc, we store a different embedding based on what kind of query comes (retrieve  given one or two modes). Then we can get the best in each column in table 5. This still can be achieved without requiring triplet datasets by dropping appropriate terms in GCL. \n2. For each doc, we store the embedding given by the full GCL. You only get the performance of the last column which is not always the best. \n\nI think this is a fundamental trade-off because you cannot align the a bi-modal signal to each separately when there is information in each mode beyond what is explained by the other. Can the authors comment on when and where I would prefer 2 over 1?\n\nA related question is how GCL would/should be adapted in the context of multi-linear contrastive objectives (e.g. [this paper](https://arxiv.org/abs/2411.01053)) that can handle missingness?\n\nLIMITATIONS:\nSee Weaknesses.\n\nI want to see bootstrap confidence intervals for at least a few results. I do believe the improvements will still be significant but it will help with the ablation study.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}