Title: Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Large Language Models

Preference learning

Cross-modality retreival

fine-grained retrieval
Abstract: Despite Contrastive Language–Image Pre-training (CLIP)'s remarkable capability to retrieve content across modalities, a substantial modality gap persists in its feature space. Intriguingly, we discover that off-the-shelf MLLMs (Multimodal Large Language Models) demonstrate powerful inherent modality alignment properties. While recent MLLM-based retrievers with unified architectures partially mitigate this gap, their reliance on coarse modality alignment mechanisms fundamentally limits their potential. In this work, We introduce MAPLE (Modality-Aligned Preference Learning for Embeddings), a novel framework that leverages the fine-grained alignment priors inherent in MLLM to guide cross-modal representation learning. MAPLE formulates the learning process as reinforcement learning with two key components: (1) Automatic preference data construction using off-the-shelf MLLM, and (2) a new Relative Preference Alignment (RPA) loss, which adapts Direct Preference Optimization (DPO) to the embedding learning setting. Experimental results show that our preference-guided alignment achieves substantial gains in fine-grained cross-modal retrieval, underscoring its effectiveness in handling nuanced semantic distinctions.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the persistent \"modality gap\" in cross-modal (image-text) retrieval, a discrepancy between visual and textual features in their shared embedding space. The paper introduces training framework named MAPLE. The core idea of MAPLE is to leverage a powerful, off-the-shelf MLLM as a \"teacher\" or \"reward model\" to transfer its fine-grained alignment priors to a \"student\" model intended for retrieval. Experimental results demonstrate that MAPLE significantly outperforms existing CLIP-based and MLLM-based retrieval models on several general and, most notably, fine-grained retrieval benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed metric, based on Wasserstein distance, providing a effective tool for quantifying and comparing the alignment capabilities of different model architectures.\n2. This paper intelligently extends the principles of DPO from language model alignment to the domain of cross-modal representation learning.\n3. The experiments are thorough.\n\nWeaknesses:\n1. The proposed framework is highly complex and computationally expensive. The entire pipeline requires a powerful MLLM (e.g., Qwen2.5-VL-72B) for caption generation and continuously uses another large MLLM (Qwen2-VL-7B) as a reward model for dynamic scoring during training.\n2. The absence of an ablation study on the choice of the reward model. It is unclear how performance would degrade if a smaller or different family of MLLM were used. This is crucial for understanding the method's robustness and generalization.\n\nQUESTIONS:\n1. Since the performance appears fundamentally upper-bounded by the reward model, have you investigated the impact of using a less capable MLLM (e.g., a 1.5B or smaller model) for scoring?\n2. The RPA loss is weighted by the difference in alignment scores from the reward MLLM. This implies that distinguishing between two very similar, high-scoring samples (small difference) receives less weight than distinguishing between a high-scoring and a low-scoring sample (large difference). Is this design optimal in all cases? Could it potentially cause the model to neglect learning the most difficult, fine-grained distinctions?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to solve the \"modality gap\" problem that exists in models like CLIP. The work finds that off-the-shelf Multimodal Large Language Models (MLLMs) possess stronger inherent alignment capabilities. To this end, the paper proposes a framework called MAPLE, which utilizes a powerful MLLM to automatically construct preference data and introduces a novel Relative Preference Alignment (RPA) loss function, inspired by Direct Preference Optimization, to guide the learning of the retrieval model. Experiments show that this method significantly narrows the modality gap and achieves excellent performance on multiple cross-modal retrieval benchmarks, with particularly significant improvements on fine-grained tasks.\n\nSTRENGTHS AND WEAKNESSES:\nPros: \n- The introduction of a unified metric (Δgap) based on Wasserstein Distance. This provides the research community with a valuable tool to measure and compare the modality gap across different model architectures.\n- The experimental design is highly comprehensive. \n\nCons:\n- Figure 5 reveals a performance trade-off: optimizing for fine-grained tasks comes at the expense of retrieval capabilities, challenging the generalization of its learned representations.\n- MAPLE framework's high dependency on a single reward model raises questions about whether it is learning a general representation or simply \"overfitting\" to that specific model's biases and flaws.\n- The method's key paradigm is a complex form of knowledge distillation, using a large model to guide a smaller one, which weakens its claim to fundamental algorithmic novelty.\n\nQUESTIONS:\nNA.\n\nLIMITATIONS:\n- Only Qwen2 series is adopted. \n- The training of MAPLE hearily relyies the hardware, a cluster of 32 NVIDIA A100 GPUs. Theferfore, the producibility is very limited.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MAPLE (Modality-Aligned Preference Learning for Embeddings), a framework that leverages off-the-shelf Multimodal Large Language Models (MLLMs) to guide cross-modal representation learning through preference alignment. The key insight is that MLLMs inherently possess stronger modality alignment capabilities than traditional CLIP-based models with Wasserstein distance-based metric (lines 105-108). Based on this observation, the authors construct DPO training data and propose two RPA losses inspired by preference optimization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nI think the lightweight MLLM prior approach is what the current community needs, particularly using MLLM reasoning/preference [1] and other general capabilities to efficiently enhance cross-modal representation learning.\n\nFrom the data requirement perspective, adopting more efficient post-training methods to solve alignment problems also makes technical sense. The adaptation of DPO to embedding learning through RPA loss is technically sound and addresses a genuine limitation of coarse-grained contrastive learning.\n\nBesides the motivation [figure 3], I think this work is expressed quite clearly, such as the example in Figure 1. The presentation and experimental completeness are also professional and standardized, which is above baseline level. It also comes with detailed appendices, which seems to meet the requirements of the NeurIPS community.\n\nMinor revision suggestion:\n1. Add commas after formulas (if the following word is lowercase like \"where\") or periods (if the following word is capitalized)\n2. There are still some typos, for example line 122 mentions that Preference Data Construction is offline, but Figure 2 shows it as offline. I understand it should be that the data for subsequent DPO can be prepared offline.\n3. The text in Figure 2 could be optimized - the text size should match the caption and use Times New Roman font\n4. From a writing perspective, Figure 3 could be moved into the main text to strengthen the motivation.\n\nWeaknesses:\nThe computational overhead and scalability concerns are significant. The method requires dynamic MLLM inference during training to compute alignment scores (lines 146-150), significantly increasing computational cost. Particularly for Scoring and Structuring Preferences, can this be done offline as an efficient DPO dataset?\n\nThe method fundamentally relies on off-the-shelf MLLMs as preference oracles (e.g., Qwen2-VL in line 238-240), but **lacks discussion of how MLLM hallucinations affect preference quality**. Since the approach's effectiveness is directly tied to the reliability of MLLM alignment scores, how do the authors address cases where the reward model provides inconsistent or incorrect preferences? The paper would benefit from ablation studies examining performance degradation when using MLLMs with known hallucination issues, and analysis of the method's robustness bounds relative to MLLM reliability.\n\n[1] Zhao X et al. Omnialign-v: Towards enhanced alignment of mllms with human preference[J]. arXiv preprint arXiv:2502.18411, 2025.\n\nQUESTIONS:\nTraining efficiency and practical deployment: Given the computational overhead of dynamic MLLM scoring during training, how does the total training cost compare to strong baselines like EVA-CLIP? Have the authors considered offline preference data construction or distillation approaches that could reduce inference costs while maintaining performance gains?\n\nHow sensitive are the results to the choice of MLLM architecture? Have the authors tested whether preferences from different MLLMs (e.g., qwen2.5vl, internvl2 and llava ov) lead to consistent improvements, and what happens when reward and policy models have different architectural biases?\n\nHow to handle MLLM hallucinations, what is the hallucination rate in pairs, and whether there are relevant explorations?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes MAPLE (Modality-Aligned Preference Learning for Embeddings), a novel framework that transfers alignment priors from off-the-shelf Multimodal Large Language Models (MLLMs) to cross-modal embedding models for fine-grained retrieval. MAPLE leverages MLLMs to generate automatic preference data and introduces a Relative Preference Alignment (RPA) loss, adapted from Direct Preference Optimization, to optimize embedding-level representations. Experiments show that MAPLE significantly outperforms CLIP and MLLM-based baselines, especially in nuanced semantic retrieval tasks.

STRENGTHS AND WEAKNESSES:
Pros  
1. The idea of transferring MLLM alignment priors via preference optimization is well-motivated.  
2. This work is the first to adapt DPO-style preference optimization to cross-modal embedding learning, with both pairwise and listwise RPA losses.  
3. The ablation study is well-structured and provides strong empirical support for the method design.  

Cons  
1. The method is resource-intensive. MAPLE relies on large MLLMs (e.g., Qwen2-VL-72B) queried online and requires 32×A100 GPUs for training, limiting reproducibility.  
2. The training signal depends entirely on pre-trained MLLMs, which may encode social or domain-specific biases. However, the paper does not explore any debiasing techniques or propose mitigation strategies for these risks.

QUESTIONS:
1. How robust is MAPLE to reward model size or architecture? Can smaller MLLMs still generate effective preference data?  
2. Have the authors considered incorporating bias mitigation techniques to reduce the impact of biased MLLM outputs?

LIMITATIONS:
The paper acknowledges two limitations—MLLM bias and lack of evaluation on complex tasks such as compositional retrieval.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05