Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment
Authors: Pengfei Zhao
Rongbo Luan
Wei Zhang
Peng Wu
Sifeng He
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Models
Preference learning
Cross-modality retreival
fine-grained retrieval
Primary area: deep_learning

Abstract: Despite Contrastive Language–Image Pre-training (CLIP)'s remarkable capability to retrieve content across modalities, a substantial modality gap persists in its feature space. Intriguingly, we discover that off-the-shelf MLLMs (Multimodal Large Language Models) demonstrate powerful inherent modality alignment properties. While recent MLLM-based retrievers with unified architectures partially mitigate this gap, their reliance on coarse modality alignment mechanisms fundamentally limits their potential. In this work, We introduce MAPLE (Modality-Aligned Preference Learning for Embeddings), a novel framework that leverages the fine-grained alignment priors inherent in MLLM to guide cross-modal representation learning. MAPLE formulates the learning process as reinforcement learning with two key components: (1) Automatic preference data construction using off-the-shelf MLLM, and (2) a new Relative Preference Alignment (RPA) loss, which adapts Direct Preference Optimization (DPO) to the embedding learning setting. Experimental results show that our preference-guided alignment achieves substantial gains in fine-grained cross-modal retrieval, underscoring its effectiveness in handling nuanced semantic distinctions.

Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment
Pengfei Zhao, Rongho Luan, Wei Zhang, Peng Wu, Sifeng He*
Apple (pzhao23, he_sifeng)@apple. com
Abstract
Despite Contrastive Language- Pre-training (CLIP)'s remarkable capability to retrieve content across modalities, a substantial modality gap persists in its fea- space. Intriguingly, we discover that off- fffttf MILMs (Multimodal Large Language Models) demonstrate powerful inherent modality alignment properties. While recent MLLM-based retrievers with unified architectures partially mitigate this gap, their reliance on coarse modality alignment mechanisms fundamentally limits their potential. In this work, We introduce MAPLE MModalit-Allignee Preference Learning for Embeddings), novel framework that leverages the fine- grained alignment priors inherent MILM guide cross- moosl representation learning. MAPLE formulates the learning process as reinforcement learning with two key components: (1) Automatic preference data construction using off-the- shelf MLLM, and (2) new Relative Preference Alignment (RPA) loss, which adapts Direct Preference Optimization (DPO) to the embedding learning setting. Experimental results show that our preference-guide alignment achieves substan- tial gains in fine-graine fine-grained moosl retrieval, underscoring its effectiveness handling nuanced semantic distinctions.
Introduction
Cross-modal retrieval, which aims to retrieve relevant content across different modalities (e.g.. retrieving images with text queries), has long been core topic in the vision-language research community. It also serves as a fundamental building block for a wide range of downstream task, including visual question answering. retrieval-augnee auggented generation, and increasingly. LLM-based modti-modal agent systems. Despite the remarkable success of large-scale contrastive pretraining frameworks such as CLIP [ fundamental challenge still remains: the modality gap, i.c. the discrepancy in feature representation between visual and textual modalities. This modality gap often manifests as spatial separation between image and text embeddings the shared latent space, and also limits the effectiveness of learned representations in various retrieval tasks (24 Prior works have shown that this misalignment might attributed from architectural choices, training dynamics, and even input information imbalance.
While prior works mainly focused on measuring the modality gap using explicit embeddings the alignment properties of models that operate directly via logits, such as off-the-shelf Multimodal Large Language Models (MLLMs), remain less explored. To better understand the modality gap across different model architectures, we propose a unified metric based on the 1- I-Wasserstein Distance (WD) [5] that enables direct comparison between logit-based and embedding-
models. For MILMs,, we compute pairwise alignment scores using output logits (as detailed in Section 2). while for embedding models, we use cosine similarities. In both cases, we apply WD to measure the discrepancy between the similarities distributions. Intriguingly, through this quantitative comparison,
"Corresponding Author, Project Lead
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Query: Eagle with wings closed
TOP1
TOP4
Getting digrenntt
off-the-shelf MLLM
amaacaaaasi sigh with Taimiaa masqa
(a) Computing the alignment score
Paineise
(b) Constructing preference data
(c) Retrieval Comparison: W/ vs. W/O Preference Alignment
Prompting the off- off-the-shelf MLLM to output "yes" or "no" token for the paired text-image, and then calculating the alignment score based on "yes" and "no" token logits. (b) Constructing Preference Data: data based on the calculated alignment scores. (c) Retrieval Comparison: Through preference alignment, the retrieval model can capture fine-grained distinctions between retrieved images under same query.
Figure 1: (a) Computing the Alignment Score:
Constructing pairwise or listwise preference
discover that MLLMs like Qwen2-VL. exhibit strong inherent modality alignment capabilities even without relying on explicit embedding representations. Our analysis, illustrated in Appendix Figure reveals that MLLMs demonstrate emergent ability to align image and text inputs more effectively than CLIP.
Recent efforts have fine-tuned MLLMs for retrieval by enabling cross- moosl representation Icarn- ing [0] However, transitioning from generative architectures to MLLM-based retrievers often diminishes the inherent alignment capabilities of MLLMs, even after further fine-tuning. Motivated by the above observation, we aim to adapt MLLM for retrieval tasks while preserving their strong cross-modal alignment strength.
To this end, we propose MAPLE (Modality-Alignee Aligned Preference Learning for Embeddings). a novel framework that bridges the alignment capabilities of off-the-shelf MILM with MLLM- retrieval model. MAPLE transfers MLLM's alignment capabilities embedding spaces through two key dimensions: data-level preference construction and
preference alignment. At the data level, we retrieve top-K hard samples for each anchor and leverage MLLM to score their matching degree with the anchor's corresponding cross- cross-modal content. These alignment scores then used to construct both pairwise preferences and listwise preferences for training. At the training level, we derive the Relative Preference Alignment (RPA) loss from Direct Preference Optimization (DPO) 11 specifically adapted for embedding models to achieve fine-grained cross-modal alignment. Through optimization with the RPA loss these constructed preferences, illustrated in Figure [ our method effectively captures subtle distinctions between retrieved images under the same query.
The related work is in Appendix D The main contributions of our paper include:
We propose Wasserstein distance as a unified metric measure the modality gap for both logit-based and embedding based models, revealing that off-the-shelf MLLMs inherently exhibit strong cross- modality alignment capabilities.
We introduce the MAPLE framework for extracting powerful multimodal embeddings. Our approach features a novel strategy that automatically leverages the inherent modality align- ment capabilities of MILM to construct preferences after hard sample mining. Additionally, we explore the adaptation of Direct Preference Optimization (DPO) for cross-modality representation learning. which yields substantial improvements in fine-grained retrieval.
We validate our framework through comprehensive experiments on different benchmarks including general retrieval (e.g., COCO [12], Flickr30K [3] ) fine-grained retrieval (e.g.
Winoground [4] NaturalBench [15), MMVP [16), BiVLC [7]] ) The experimental results

---

demonstrate the superiority and effectiveness of our approach in both general and fine- grained retrieval tasks.
2 Preliminaries and Notation
Given batch of N image-text pairs (<"",","""""))) "e to extract embeddings for each modality, we leverage the MILM,, a unified architecture, to obtain embeddings for each modality. Following the prior works [8] [O]. construct prompts using predefined templates such as "<text> Describe this text in one word:" for text and "<image> Describe this image in word:" for image. These prompts are used to process the image-text pairs, then we extract the corresponding text embeddings {=!")]
unlike CLIP, which utilizes separate encoders
Ji=1 and image embeddings {=, 12 J.V Ji=1
Contrastive Loss. Once obtaining normalized embeddings { late the standard autoregressive training paradigm of LLMs into a discriminative framework via symmetric InfoNCE-style contrastive loss:
Ji=1 and { N. Ji=1 we reformu-
exp(2 img
/T)
exp 2M
2,
img
Coontrast
log
log
(I)
2N
Σ
1j=1 exp(2;
ing
/T/
2j=1 cxp(2
Jog
where r is a temperature hyperparameter. Prior works 18 19 established contrastive learning inher-
ently employs a coarse-grained alignment strategy that uniformly pushes away all negative samples in the embedding space, with limited consideration of fine-grained semantic similarity between these samples. This uniform treatment struggles to establish nuanced discriminative boundaries. particularly for semantically similar negatives.
Computing the Pairwise Alignment Score. To probe the implicit modality alignment within an off-the-shelf MILMM, we measure the alignment score between an image-text pair (a .rt))) based on the MLLM's output logits for "Yes" (Cxe) and "No" (1 tokens in response to a relevance query (details in Appendix A.I.. Then, we employ the softmax function for these two tokens to get the img alignment score ozi. which represents the MLLM's confidence that the image v. and text x. form a semantically matching pair.
Measuring the Modality Gap. The modality gap represents the misalignment between visual and textual feature distributions. Prior work (2) measured this using the average distance between mean embeddings:
Ming where Pmnod
N
Di=1
N mod
Expanding the squared distance, 1| | 2 2/rra Wimg + ||Himg 2 reveals that it measures the difference between the mean intra-modal similarity (N² ΣN E 2mod mod) and the mean img
cross-modal similarity V ΣM j=1
However, this mean-based comparison overlooks the full distributional characteristics of similarities. Furthermore, it requires explicit embeddings, making it less suitable to measure the modality gap for the logits-based model. capture distributional discrepancy more effectively, we employ the 1- Wassersteen Distance (WD) [5] to compare similarity distributions. For two distributions IP, and Fg. WD defined as:
W (PA,PP)) =
inf 7EU(PA,Pn)
B( sblll
where 8a, Sb are samples from the distributions, and II(;,)) .) is the of joint distributions with the given marginals.
We measure the modality gap using WD the Winoground-style [14]
dataset, where img and two captions (2: 22. forming matching
each test instance contains two images img _1
pairs img _00 and (1 jun (..). We denote the sets of all a o ag and x as To. Ti, To. and 1 respectively. We consider it from two perspectives: the distributional gap W (IT to 1 IT..)))
3

---

Constructing Preference Data (offline)
Gerarated Captiio
Iitiona
Coand
Original Dataset
Dediulication
Hard Exemple Mining
Highlight the difference
Candicaate
Constructing Preference (online)
frosen persrraterr
fino tuning
Algeemet Scors
MILMM
Prinet Donatrie
con procuct
off-the-sher MLLM (Foward)
Proforence
Proforence Alignment
AT,, MT
brronking
Topp worl
MLLM-based Retriever (Polay)
Regulerization Item
Rebative Preferance Alignment
Figure 2: The Training Schema of the Proposed MAPLE. We first prepare the candidate set for each anchor sample through a series of dataset processing operations. In the training stage, we leverage an off-the-shelf MLLM as reward model to dynamically calculate the alignment scores and subsequently construct the preference data. We extract the embeddings from the policy model (MLLM-baseed retriever) and align them with the preference data through the RPA loss. This schema primarily illustrates the pairwise training paradigm.
which quantifies the alignment between intra-modal PT, and cross-modal PT. similarity dis- tributions (where a lower value indicates better alignment but risks representation collapse (20) when approaching zero), and the discriminative gap W (ITT, (T,r)) which assesses the model's ability distinguish between matching and mon-matching pairs (where higher value indicates better discriminative ability). Figure 3 in the Appendix illustrates the modality gap comparison between CLIP and Qwen2-VL VL based on this unified metric.
Finally, distributional gap Wais-gm gap and discriminative gap Wdise-gan are computed as the mean values across all respective gap measurements in the dataset. We integrate both perspectives the modality gap into our proposed metric (Ag))): Agap = Wais.gap/ gap- lower Agup reflects a superior model, characterized by both small distributional gap and large discriminative gap.
3 Method
We propose MAPLE (Modality-Aligned Preference Learning for Embeddings), a novel framework that guides cross-modal representations with MILM priors via preference alignment. As illustrated in Figure 2 MAPLE consists of two key components: (1) Preference Data Construction: An offline stage retrieves hard negative samples, followed by online process where off-the-shelf MLLM dynamically computes text-image alignment scores during training to establish preference data; and (2) Preference Alignment: Derived from the DPO loss, introduce a novel Relative Preference Alignment (RPA) loss that explicitly enhances the model's nuanced discriminative capability by contrasting preferred and dispreferred data samples.
MLLM-based Retriever Architecture. We initialize our model from pretrained MLLM backbone to inherit its multi-modal alignment capability. Refer the work (21), to convert the autoregressive MLLM into a discriminative retrieval paradigm, we make two small modifications: replacing the causal attention mask with bidirectional attention and adding mean- pooling final hidden states to aggregate sufficient features for retrieval.
3.1 Preference Data Construction
The preference data construction involves two stages: an offline preparation stage and an online scoring and structuring stage.

---

Offline Stage: Candidate Generation. The pipeline begins with an offline stage to prepare candidate sets. We first extract DINOv2 [2 embeddings from the image dataset and apply Semantic Deduplication (SemDeup) [23 a clustering- method- to filter near-duplicate samples. For each deduplicated image n, in a batch, we retrieve its top-K nearest neighbors { J from the (
gallery based on cosine similarity, forming an image candidate set C. { } U {ag 13=1- To enrich caption diversity and create challenging negatives, we leverage the multi-image reasoning capability of an off-the-shelf MILM. We prompt the MILMM with the image set C. to generate discriminative captions that explicitly highlight inter-image differences. This process yiclds corresponding text candidate set C. {x.²1) U {4)) J==1. Candidate generation details are provided in Appendix B.I
img
K
Online Stage: Scoring and Structuring Preferences. During the online training phase, we use the off-the-shelf MLLM to dynamically compute fine-grained alignment scores between anchors and candidates. For each anchor image a.!" we compute alignment scores with all text candidates: imgg2xx {align (a: 2) a E C.a).. Symmetrically, for each anchor text zr! we compute scores with all image candidates: ax ixt2ing {align(z))) .>) | E c, Each score vector a; (either img2tei a;
! represents the MILM's preference for candidates in C, relative to the anchor zq. We sort the candidates in descending order based on these scores, obtaining ranked indices {rx k= such that the corresponding scores satisfy Cijra 2 Oiirr 2 ... 2. 0i,rx Based on this ranking, we structure the preference data in two ways for the subsequent alignment loss:
O;
Pairwise Preferences: We construct set of preference pairs P; = { Zi.r.) 10 a b S K). where Vi,r. is the ra-th ranked candidate and Hi,r, is the r6-th ranked candidate from the corresponding set C. Each pair (air,, Fi,r,) indicates that Vi,r, preferred over Ti,r, according to the MILM's alignment score with the anchor 24.
Listwise Preferences: Instead of breaking the ranking into independent pairs, we leverage the structure of the entire ranked list (Iirr Ta ......... ...., Ti.rx) This approach considers preferences within all possible suffixes of the list. Specifically, for each starting rank (from 0 to K- 1), the item Ti,ra is treated as the preferred item relative to the set of all subsequent items { j= in the suffix (Ia. Firg)) ). This captures the relative ordering across the whole list more directly than pairwise comparisons.
Through these offline and online stages, for batch of N anchor image-text pairs {(x;"" imng a. )
we construct the corresponding sets of pairwise preferences and listwise preferences.
3.2 Preference Alignment
In contrast to the contrastive loss's coarse-grained alignment approach, our goal is to establish nuanced discriminative boundaries leveraging fine-grained preference data. Drawing inspiration from Direct Preference Optimization (DPO) [I1. which effectively fine-tunes LLMs to align with human preferences, we similarly aim fine-tun tune our MLLM-based retrieval model to align with the sophisticated preference signals from off- off-the-shelf MLLMs.
DPO Loss. Traditional DPO Loss relies on pairwise comparisons between preferred (yw) and dispreferred (gr) outputs for a given input x to align policy models (Tе) with human preferences, often using a reference model (T.)). The DPO training objective is constructed as a maximum likelihood loss:
LorO (mo; Tw)) - (m....m..n log (Blog Tw (Nh
To (yue x:)
Tw (3) xx) }]
To (w xz)
3 log
(3)
x))
Here, a denotes the sigmoid function, and 3 a hyperparameter scaling the log- bog-probability difference. However, directly applying DPO to retrieval presents challenges: (1) The diverse captions generated in the offline stage (Section 3.1 create a combinatorial explosion of potential image-caption pairs, making it impractical to explicitly enumerate all preference pairs; (2) The standard DPO framework requires maintaining policy, reference, and potentially reward models, imposing substantial memory and computational burdens.
5

---

Eliminating the Need for the Reference Model. The
issue can be alleviated
by adopting a uniform prior for the reference model Tw. similar CPO [24]. In this case, the
reference model terms Ta (yw | x) and The (1) z) cancel out in the log-ratio difference (up to
constant), eliminating the need for costly computations and storage associated with Tw. The simplified objective becomes:
IpPOssimpiited (TO) (r.. (log (3 no (ym x)) 3 log wa (y1 xx)]]
Relative Preference Alignment (RPA) Loss. We adapt the simplified DPO objective for embedding models by replacing the log probabilities log To x) with scaled similarity scores B(2 "ncchor 2 canddddat)
between anchor and candidate embeddings produced by our MLLM- --MMeddadddd--- retrieval model. This approach, which we term Relative Preference Alignment (RPA), incorporates the MLLM-derive preference structures (pairwise listwise). We explore two primary strategies for implementing RPA: pairwise and listwise optimization.
The Pairwise RPA loss optimizes preferences using the pairwise data P,. For a given text anchor 2;" img
p.uliims (where 0), it aims to ensure the similarity score of and preference pair (fi,7, ".,m.
txt2img 3(x -i.re) denote the scaled the preferred image is higher than the dispreferred one. Let &tk similarity score. The pairwise RPA loss for text-to-image alignment (txt2img) is weighted by the difference between the MLLM's alignment scores for the pair, giving more importance to pairs with larger preference margins:
iing
N
ChatZimg Rai-Pair
a 1,r
1x12img
log ($
tx22ing
ext2ing
(5)
N
i=
OSKKISK
Symmetrically, we define the image-to-text loss cimg2uxt and pairs from pimgarur The total pairwise RPA loss is:
Par-Paiiwise using scores i;
img2txt
line
2irr )
)
LRPA Pairwise
ClM2iing RPA Pairwise
pimg2txt RRPA.Pairwise/
66)
Alternatively, the Listwise RPA approach directly optimizes the model's ability to align with the MLLM's ranking using the listwise preference data. Inspired by PRO 225. this loss encourages the model to assign the highest similarity score to the top-ranked item within each suffix of the MILM's ranked list. For a text anchor x! and its ranked image candidates (zirre (..
img ). the loss iterates
through each possible top element art (for i from 0 K 1) and maximizes the prgpobbbbbllty of this element being ranked highest among the suffix {> K j== using softmax over the model's
similarity scores *iy fx12ing To incorporate the fine-grained preference strength from the MLLM, each term in the sum (corresponding to specific suffix starting at k) is weighted by the average preference margin assigned by the MILLM to time over the subsequent items in that suffix:
N
K-1
exp(s 1X1Zimg
2ix2inng RPA-Listwise
1
Σ
Σ
txt2img wik
log K Ej=k exp(s (x22mng)
N
R=0
1222mg ex22ing is the average MLLM alignment score where the weight w = K- Ei=k+1 (a at.r,
rx22img
=
difference between candidate k and all less preferred candidates (ranks k+ 1 to K) in the list (defined as if k K). This weighting gives more importance to correctly ranking the top element in suffixes where the MILM preference is strong and clear. The corresponding image-to-text loss, 2img2txt jing2txt and weights w defined symmetrically using sik RPA loss is:
Listwiie
img2txt derived from a. img2txt The total listwise
1
CRPA- Listwie
2 RPA-Listwise +
pixiiing
pinglixt
RPA-Lissiise
(8)

---

Regularized Relative Preference Alignment. To prevent excessive alignment to the MLLM preferences, which might lead feature collapse, we introduce regularization term Coontuss. This typically a standard contrastive loss computed on the original anchor pairs (α: x within the batch. The final training objective combines the chosen RPA loss (either pairwise or listwise) with this regularization:
C = XCRPA + (1 C
(9)
where CRAA represents the chosen RPA loss component (either LRPA-Liswise or LRPA-Pairise)) ). and serves balancing hyperparameter controlling the strength of the preference alignment relative to the regularization term.
4 Experiments & Results
In this section, we evaluate MAPLE across multiple standard benchmarks to assess its effectiveness in cross-modal retrieval tasks. We compare MAPLE against strong baselines and analyze its performance through ablation studies.
4.1 Experimental Setup
We train MAPLE curated subset of the OpenImage dataset (26 (details in Appendix B.I. We
evaluate its performance on standard general (MS-COCO [12. Flickr30K [33 and fine-grained (Winoground 14), NaturalBench [13), MMVP (16), BiVLC 117 retrieval benchmarks using standard
metrics (c.g., Image/Tex: Recall @ 1, Image/Text scores). About the more fine- nne-grained evaluation
details, please refer to Appendix [B.3] MAPLE is compared against strong CLIP-based [1
and MLLM- based [O] retrieval models. We use LoRA for fine-tuning MAPLE MLLM-based
embedding models. Comprehensive implementation details are provided in Appendix B.2
4.2 Main Results
Performance Retrieval Benchmarks. As shown in Table 11 our MAPLE(Qwen2-VII-7) approach consistently outperforms both CLiP-based and MLLM-based models across multiple benchmarks. Under the similar-scale parameters settings, MAPLE(Qwen2-VV-22) also shows competitive advantage. The improvement is even more pronounced on fine-grained retrieval tasks, where MAPLE demonstrates substantial gains on Winoground and NaturalBench, significantly outperforming previous methods. These results validate the effectiveness of our preference siieerence-giieedd guided alignment approach in capturing nuanced cross-modal relationships.
4.3 Ablation Studies
To rigorously evaluate our proposed method MAPLE and dissect the contributions of key compo- nents, we conduct a series of ablation studies. Our final proposed loss function combines a standard contrastive loss Leomtru with our novel RPA loss LRPA- The standard contrastive loss Leontrust is computed using image-text anchor pairs gathered across all devices to maintain the model's general cross-modal aligmment capabilities, serving as a regularization term to prevent excessive alignment. Meanwhile, LRPA operates specifically preference data derived from retrieved examples, aiming to refine the model's 's understanding of fine-grained distinctions. We investigate the impact of using these components individually and in combination.
Relative Preference Alignment. We analyze the impact of different loss components, referencing Table 2 The baseline uses only the standard contrastive loss (Ccontrsst.
Compared with the baseline, using only standard contrastive loss preference examples (erentrst-pret)) generally degrades general retrieval performance and shows moderate improvements on the NaturalBench dataset, with only slight improvements the Winoground Image task. In contrast, applying our RPA losses (LRPA Pairwise: LRPA LRPA.Liawied Liswise) directly to preference data, despite reducing general performance when used alone, achieves substantially better fine- fine-grained results than
7

---

Table
Performance Comparison on General and Fine-grained Retrieval Tasks. Best results are in bold, second best are underlined. VladVA haven' 't released the model weights, so represents unavailable results.
General Retrieval (R@1)
Fine-grained Retrieval
Model
COCO
Flickr30k
Winoground
NaturalBench
Text
Image
Text
Image
Text
Image
Text
Image
CLIP--bsed Models
CLIP (Vi7-L)
58.1
37.0
87.2
67.3
27.5
12.3
41.8
45.0
OpenCIII (VTT-//14) 27
66.3
48.8
91.5
77.8
32.0
12.8
46.2
46.5
SigLIP (so/14) 28
70.2
52.0
93.5
80.5
37.5
16.3
62.7
63.9
Si2LIPv2 (g/I6-2B)
30
72.8
56.1
95.4
86.0
39.8
17.0
65.5
68.7
EVA-CLIP (8B) [29
70.1
52.0
94.5
80.3
36.5
14.8
58.5
EVA-CLIP (18B) 22
72.8
55.6
95.3
83.3
35.8
15.0
58.7
61.2
MLLM-based Models
AS-V(LLaVA-Next-BB)
62.0
52.0
88.2
79.5
32.3
14.8
60.3
67.6
VadVeA(Qee222 VL-28)
10
71.9
52.5
93.7
80.4
VladVA(LLaVA-I15-22)
[10
72.9
59.0
94.3
83.3
40.5
17.5
MAPLE<(Qwe2--V-22B
72.8
56.8
92.8
82.6
43.0
22.5
69.2
70.2
MAPLEE(Qen22-VL-725
75.5
60.3
94.3
86.1
56.0
32.7
76.1
76.8
Table 2: Ablation Study Loss Components. Performance General and Fine-grained Retrieval Tasks, compared to the baseline (Ccontrss). Differences are shown parentheses with arrows ( ( for improvement, |. for decline).
General Retrieval
Fine-grained Retrieval
Method
COCO
Winoground
NaturalBench
Text
Image
Text
Image
Text
Image
Baseline Contrastive Loss Only
Baseline (Ccontas)
74.0
54.4
42.5
20.5
61.4
62.5
Training with Preference Data Only
Coostrsssppoet
64.6
46.2 (L-82)
42.3 (2-3(
20.7 (1422)
66. (144))
67.8
Paps-Paiwise
51.9
52.4
48.8 (7+6.3)
34.7 (1412))
70.1 c1e87)
(1+14)
LRPA.Lisnsse
57.1 (1-169)
55.4("+1.))
48. (1+55
(1+160)
71.1 (1+97))
78.1 (1+15.6)
Combining Preference Loss with Contrastive Regularizer
Lecamna Leoutras-guee
70.9
53.9 (L-0.))
21.5 (1+10)
65.5
67.0
Lovatuas
71.2 (3:28)
57.7 (-+3.3)
49.8 49.8
26.8 (1+6.)
68.6
71.4 cr.8.9
Lecauras
LRPV-Liwwer Liswise
71.9 (U-2.)
58.6(-H4.)
(1.0
28.2 (1#7.))
69.2 (1+7.8)
71.2 (1+87)
feomtrast pref- This highlights the importance of introducing preference data to improve the model's ability to make nuanced distinctions.
We then combine preference-basee losses with the standard contrastive loss (Ccontrast) regularizer. Through extensive experimentation, we identify the optimal À parameter for each combination that maximizes average performance on both general and Ane-grained retrieval tasks. When combined with the Lvontrst regularizer, this approach effectively mitigates performance degradation on general retrieval while maintaining strong fine- mene-ged retrieval performance. Moreover, LRPA consistently outperforms Leomtrast pref- Additionally, we observe that LRPA Liswisse consistently achieves better results than CRPA Pairwise across nearly all scenarios. We attribute this the fact that listwise loss aligns preferences across the entire ranked list, making it more effective than pairwise comparisons.
Impact of Expanded Negative Pool for Contrastive Loss. Expanding batch size plays critical role contrastive learning. However, increasing batch size for MILMS incurs substantial computa- tional overhead. To address this challenge, propose efficient strategy that implicitly enlarges

---

Table Ablation Study on Using Expanded Negatives (Exp. Neg.) and LiPA-Liswise (CRpA). The baseline (x/X) uses neither component. / indicates the component is used, X indicates it is not. Best results are bold.
Components
General Retrieval
Fine-grained Retrieval
COCO
Winoground
NaturalBench
BiVLC
MMVP
Exp. Neg.
CRPA
T/I
T/I
T/I
T/I
T/I
x
X
74.0/ 54,4
42.5 20.5
62.5 61.4/ 62.5
86.1 60.
33.3/ 20.0
/
X
75.3 / 57.3
49.3
70.7 / 70.2
89.2/ 66.6
/9.3
X
V
71.9/
51.0 28.2
09.2/71.3
88.3/
46.7 /7.0
V
J
75.5 60.3
56.0 32.7
76.1 76.8
89.4/
45.9 43.7
Table 4: Robustness to Reward Model Scale and Architecture. The policy model is fixed to Qwen2-VL-7B. Best results within each model family are in bold. The indicates the baseline without reward model.
General Retrieval
Fine-grained Retrieval
Reward Model
COCO
Flickr30k
Wineground
NaturalBench
Text
Image
Text
Image
Text
Image
Text
Image
73.4
54.3
93.6
80.3
40.7
18.2
60.2
62.7
Qwen Family
Qwen2-VLL -BB2B
74.1
59.1
93.0
84.1
53.5
31.0
70.4
72.3
Qwen2-VL-7B
75.8
60.2
94.2
85.3
55.0
31.0
74.5
75.2
Intern VL.3 Family
Intern VL3-1B
73.9
58.9
92.8
83.7
48.5
26.0
69.3
72.6
Intern VL3-2B
75.9
59.2
94.2
84.8
53.8
28.5
72.8
74.3
InterV
75.6
59.7
93.8
84.8
54.0
31.5
74.3
74.8
Additional Architectures
SAIL-VL-116-8B
76.1
59.9
94.3
85.6
54.8
29.8
75.4
74.3
Inter BB
75.2
59.7
94.3
85.3
54.8
30.5
74.6
74.8
Intem VL2.5-8B SB-MPO
75.5
59.5
93.8
85.1
53.5
30.8
74.1
74.2
the effective batch size through the incorporation of readily available hard negatives, eliminating the need for additional computational resources. The detailed implementation of this strategy provided in Appendix C.I. Empirical results in Table demonsttate demoonnnns that our expanded negative pool strategy yields consistent performance improvements across both general and fine- ane-grained retrieval tasks.
Robustness to Reward Model Choice. To assess the generalizability and robustness of MAPLE, we conduct a comprehensive ablation study on the choice of the reward model. We fix the policy model Qwen2-VL-7B and evaluate its performance when guided by reward signals from diverse set of MLLMs, varying in both architectural family and scale. For quick comparison, all experiments in this ablation were trained for epochs. As shown in Table 4 our analysis reveals that MAPLE robust to the scale the reward model. While larger reward models generally yield stronger results, even small models like Owen2-VL-2B and Intern VL lead consistent and significant performance gains over the baseline. This result underscores the framework's resource efficiency, as demonstrates that substantial performance gains can be achieved by leveraging alignment signals from even small-scale reward models.
Agnostic Reward Model Architecture. Furthermore, Table shows that MAPLE agnostic to the reward model's architecture. Using reward models from different architectural families (e.g., Intern VL.3 (31, Intern VL2.5 [32]. Intern [33], and SAIL-VL (34 to guide the Qwen2- VL-7B policy model still leads to clear improvements over the baseline. To further demonstrate this, we conduct systematic cross-architecture evaluation in Table where both the policy and reward
models are varied. The results confirm that mismatched reward-policy pairs highly effective,

---

Table 5: Cross-Architecture Robustness Analysis. Performance of MAPLE with varying policy and reward model architectures. The indicates the baseline without a reward model. The best results are bold, the second best are underlined.
COCO
Flickr30k
Winoground
NaturalBench
Policy Model
Reward Model
T/1
T/I
T/I
T/1
73.4/ 54.3
93.6 / 80.3
40.7 / 18.2
60.2 / 62.7
Qwen2-VL-7B
Qwen2-VL-7B
75.8 / 60.2
94.2/ 85.3
35.0/3100
74.5 14.5/75
InternVL3--BB
75.6 59.7
93.8/ 84.8
54.0/31.5
74.3 / 74.8
Qwen2-VL-T7
76.7/ 61.1
95.4/ 86.8
53.5/315
78.4/77.7
Intern InternVL3-8B
InternVL.3-8B
76.9 61.6
95.9 / 87.5
53.5 30.5
78.4 79.4
Table 6: Comparison of Modality Gap Metrics Across Different Models. Values of Wiist gop and
W dip are scaled by 10?. indicates higher values are better, indicates lower values are better. About the gap metric, the best results in bold, the second best underlined.
MMVP
Winoground
Model
dist gap (4)
disc
A gap (L)
W dist gap (4)
W Wass-gap (f)
A Agap (L)
OpenCLIP (ViT-H/I4) 27
23.64
0.50
47.28
18.05
0.50
36.1
Qwen2-VL.-3 VL-7B (6]
6.89
6.34
1.09
7.63
2.13
3.58
MAPLE (w/ CRPA-Listwise!
6.32
0.33
19.15
4,40
0.44
10.0
MAPLE (w LRPA-Liatsse)
5.32
0.94
5.66
3.47
0.48
7.22
which underscores the framework's flexibility and high degree of interoperability. This suggests that MAPLE learns truly transferable, architecture-agnosii alignment signal, rather than simply overfitting to the intrinsic biases of a specific reward model architecture.
Measuring the Modality Gap. We measure the gap the MMVP and Winoground datasets, where for the Winoground dataset we randomly sample 100 instances (out of 400 total instances) due computational constraints in calculating pairwise alignment scores with MLLM. Table presents modality gap metrics across various models. The off-the-shelf MILM (Qwen2-VL-7B) demonstrates strong discriminative capability (having the largest W Wdise-gap). while maintaining low distributional gap. MAPLE with RPA loss substantially reduces the distributional gap while improving discriminative power. The evolution of this modality gap throughout the training process is visualized in Figure 6 Appendix
5 Conclusion
In this work, using a unified metric based on the Wasserstein distance, we reveal that Multimodal Large Language Models (MLLMs) possess strong inherent modality alignment capabilities. Motivated by this finding, we explore leveraging off-the-shelf MILM to mitigate the modality gap in cross coos- retrieval. We introduce MAPLE, a novel training framework designed to transfer the modality alignment priors MILMM into corss-modal representations. MAPLE first constructs preference data via an automatic curation pipeline and subsequently employs a novel Relative Preference Aligmment (RPA) loss tailored for MLLM-based retrieval model. Extensive experiments across various benchmarks demonstrate that MAPLE brings significant improvements fine- tiee-graned tasks. Ablation studies validate the effectiveness each component within MAPLE.
Impact and Limitations: Our work may contribute to exploring new directions for guiding cross- modal representations with alignment knowledge from MILMS and potentially offers insights for future research in multimodal representation learning and preference -etereeeeeguddddd optimization. However, we acknowledge several limitations. Known limitations of this work include: 1) The cross-modal representations may be affected by the inherent biases present in the MILM;; 2) our approach requires further validation on more complex tasks such as composed retrieval. We plan address these limitations in future work.
10

---

References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748 8763. PmLR, 2021.
[2] Victor Weixin Liang. Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 55:17612-17625, 2022.
[3] Qian Jiang, Changyou Chen, Han Zhao, Liqun Chen, Qing Ping, Son Dinh Tran, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Understanding and constructing latent modality structures in multi-modal representation learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 7661-7671 2023.
[4] Simon Schrodi, David Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: on the modality gap, object bias, and information imbalance in contrastive vision-language representation learning. arXiv preprint arXivv2404.07983. 2024.
[5] Victor M Panaretos and Yoav Zemel. Statistical aspects of wasserstein distances. Anmual review of statistics and its application. 6(1):405 431,, 2019.
[6] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint 1xXi::2409.12111, 2024.
[7] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392 13923, 2025.
[8] Ting Jiang, Minghui Song, Zihan Zhang. Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580 2024.
[9] Sheng-Chieh Chich Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal llms. arXiv preprint arXiv:2411.02571, 2024.
[10] Yassine Ouali, Adrian Bulat, Alexandros Xenos, Anestis Zaganidis, loannis Maniadis Metaxas, Brais Martinez, and Georgios Tzimiropoulos. Discriminative fine-tuning of Ivlms. arXiv preprint arXiv:2412.04338, 2024.
[11] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. 2023.
Advances in Neural Information Processing Systems,
[12] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755, Springer, 2014.
[13] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the association for computational linguistics, 2:67-78. 2014.
[14] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition,, pages 0238-5248, 2022.
[15] Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. Naturalbench: Evaluating vision-language models natural adversarial samples. arXiv preprint 4rXiv:2410.14600 2024.
11

---

[16] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shorteomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568 -5788, 2024.
[17] Imanol Miranda, Ander Salaberria, Eneko Agirre, and Gorka Azkune. Bivlc: Extending vision-language compositionality evaluation with text-to-image retrieval, 2024.
[18] Chaoya Jiang, Wei Ye, Haiyang Xu, Shikun Zhang, Jie Zhang, Fei Huang, et Vision language pre-training by contrastive learning with cross-modal similarity regulation. arXiv preprint MxKi:23005.04474 2023.
[19] Haonan Wang, Minbin Huang, Runhui Huang, Lanqing Hong, Hang Xu, Tianyang Hu, Xiaodan Liang. Zhenguo Li, Hong Cheng, and Kenji Kawaguchi. Boosting visual-language models by exploiting hard samples. arXiv preprint 230i::2300..02008, 2023.
[20] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09344 2021.
[21] Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training Ilms as generalist embedding models. arXiv preprint 1tX.i::2445.17422 2024.
[22] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[23] Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semd- edup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.0954) 2023.
[24] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Ken- ton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint 2xXx:2201.084/ 2024.
[25] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18990-18998, 2024.
[26] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scale. International journal of computer vision, 128(7): 1956-191,, 2020.
[27] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278 255994. 2022.
[28] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975-11986. 2023.
[29] Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. Eva-clip-18bs Scaling clip to 18 billion parameters. arXiv preprint 2.Xi:22402.04222 2024.
[30] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Nacem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip Multilingual vasion-language encoders with improved semantic understanding. localization, and dense features. arXiv preprint arXiv: /4786, 2025.
[31] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint 2XX4:2504.10477 2025.
12

---

[32] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open- source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.
[33] Weiyun Wang. Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arX1v:2411.1044.. 2024.
[34] Hongyuan Dong, Zijian Kang, Weijie Yin, Xiao Liang, Chao Feng, and Jiao Ran. Scalable vision language model training via high quality data curation. arXiv preprint arXiv:2501.0599 2025.
[35] Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Document ranking with a pretrained pereence-t-sequenee model. arXiv preprint arXiv:2003.06713 2020.
[36] Weiwei Sun, Lingyong Yan, Xinyu Ma. Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. chatgpt good at search? investigating large language models as re-ranking agents. arXiv preprint urXiv:2304.09555 2023.
[37] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, (2):3, 2022.
[38] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory--fficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344-16359, 2022.
[39] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128-3137, 2015.
[40] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- laiguale representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021.
[41] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances neural information processing systems, 34:9694-9705 2021.
[42] Sedigheh Eslami and Gerard de Melo. Mitigate the gap: Investigating approaches for improving cross-modal alignment in clip. arXiv preprint arXiv:2406. 22240.7679,, 2024.
[43] Xiang Ma. Xuemei Li, Lexin Fang, and Caiming Zhang. Bridging the modality gap: Dimension information alignment and sparse spatial constraint for image-text matching. In Proceedings of the 32nd ACM International Conference Multimedia, pages
2024.
[44] Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong. Megapairs: Massive data synthesis for universal multimodal retrieval. arXiv preprint arXiv:2412.14477 2024.
[45] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 27777730-27744, 2022.
[46] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXi::200..0586 2022.
13

---

A Preliminaries
A.1 Detailed Pairwise Alignment Score Computation.
prompt LLMSs to make binary relevance judgments and construct candidate rankings, Inspired by these works, we leverage MLLMs generate fine-grained alignment scores. For each image-text pair, we construct a specific relevance prompt "<image> Does the image align with the text <text>? Answer Yes or No". This prompt, containing both the image and the text, fed into the MLLM. The MLLM then processes the input and generates token logits, including those for the target tokens "Yes" and "No".
Recent works 35 36
We then calculate the pairwise alignment score, denoted as oú, by applying the softmax function specifically the logits of the "Yes" and "No" tokens. This normalization yields the probability of the "Yes" token:
exp(1x exp(2 + exp(
0g align(1 (2, img ..
(10)
Here, yess and oo represent the output logits produced by the MLLM for the "Yes" and "No" tokens, respectively, in response the relevance prompt for the specific pair (n:"e, . The resulting score og serves quantitative measure of the MLLM's semantic confidence that the given image and text constitute a matching multimodal pair.
B Detailed Experiment Settings
B.1 Detailed Training Data Curation
Our primary training data originates from the large-scale OpenImage dataset [26]. To curate a high-quality dataset suitable for fine-grained learning, employed the following procedure:
Clustering and Deduplication: We first applied clustering and deduplication to the Human- verified OpenImage data to reduce redundancy and group similar images. Following the SemDeDup method [23, we set the number clusters to 50,000 and used an epsilon value of 0.07 to filter out near-duplicate images.
2. Hard Negative Mining: For each anchor image in the deduplicated set, we identified hard negative images by retrieving the top-3 most similar images based on their DINOv2 22 embeddings.
3. Stratified Sampling: We performed stratified sampling based on the clustering results balance data diversity with training resource constraints. The dataset divided into 8 partitions, and we sampled only one partition (approximately 700K instances) for our training set.
4. Caption Generation:
For each anchor image and its selected top-3 hard negative images, we employed Qwen2.5-VL-72B 7 to generate corresponding captions for paired images (constructing 6 distinct image pairs from each anchor image and its top-3 hard negative images). To enhance descriptive diversity, we configured the generation temperature to 0.7 and applied the generation process three times, resulting in rich and varied textual representations. As demonstrated Figure 4 this method enabled us to generate detailed comparative descriptions that effectively capture the visual distinctions between images.
The final training instances thus consist of an anchor image, its associated positive caption(s), and the captions corresponding to its identified hard negative images. While alternative data curation strategies might further enhance model performance, our work primarily focuses on preference alignment optimization; thus, exploring advanced data strategies is left for future work.
B.2 Detailed Implementation Settings
MAPLE consists of two models: a reward model (Qwen2-VIL-7B) and policy model (initialized with Qw-22-VL-2B Qwen2-VL-7B). We fine-tune the policy model using Low-Rank Adaptation
14

---

Wase-ap
35.02
W
1.41
Witeepop
01.12
W
4.09
CLIP
Qwen2-VL
CLIP
Qwen2-WV VL
14
24h co0
Canberty Caee
(a) The distributional gap
(b) discriminative gap
Figure 3: The Modality Gap Comparison Between CLIP and Qwen2-VL. The gap is computed on the MMVP dataset. For the CLIP model, we use cosine similarity to construct the similarity distribution. For the Qwen2-VL VL model, distribution. Waistgap indicates lower values are better, Waise-gap indicates higher values are better.
the alignment score to construct the similarity
User Prompt
Observe these two images and generate two brief captions, highlighting the key visual differences between them. The captions should clearly indicate which description corresponds the first/second image. The visual differences should be reflected in the descriptive captions. Please output strictly in the following JSON format:
{
"Image 1": "ccrreepondiig caption",
"Image 2": "corresponding caption",
}
Please provide your response following the above specific format:
Generated Response
"Image 1": "A large military aircraft parked near water with smaller planes visible the background.", "Image 2": "An old-fashioned airplane suspended from the ceiling an indoor aviation exhibit." }
Figure 4: Example of Prompt and Response for Generating Comparative Descriptions. An example illustrating a prompt for generating comparative descriptions for pair of images and the corresponding JSON response generated by the Qwen2.5-VLL-22.
(LoRA) 37.. applying it only to the LLM component's attention and projection layers, while keeping the vision encoder and connector frozen. We employ differential learning rate strategy, with the base learning rate applied the LoRA parameters and amplifying the base learning rate by 100 times for the newly added parameters т and 3. Key model-related hyperparameters are summarized in Table
The contrastive loss Lcontrast temperature т and the CRPA coefficient 3 are treated as learnable parameters, initialized at 0.07 and 1/0.07 respectively.
Infrastructure and Efficiency. Training was conducted on cluster of 32 NVIDIA A100 GPUs (80GB memory each). To optimize computational efficiency and memory usage, employed
15

---

Table 7: Key Hyperparameters for MAPLE Training.
Parameter
Value
Policy Model Base
Qwen2-VL VL-L-VLLBBBBB/B Seee///eenneee VL-7B
LoRA Rank (r)
32
LoRA Alpha (a)
32
Optimizer
AdamW
Base Learning Rate (LoRA)
5e-4
LR Schedule
Linear Warmup Cosine Decay
Warmup Ratio
0.025 (of total steps)
Batch Size (per GPU)
96 (for 2B model)/ (for 7B model)
Total Epochs
Max Image Resolution
384x384
Initial (learnable)
0.07
Initial (leamnalee)
1/0.07 14.29
6 mixed-precision training, gradient checkpointing for the LLM component, and Flash Atten- tion (38. The maximum image resolution was constrained 384x384 during training to manage memory and allow for larger batch sizes. The whole training takes about 32 hours in this setting.
B.3 Detailed Fine-grained Evaluation Dataset and Metrics
To comprehensively evaluate fine-grained capabilities, we employ diverse set of benchmarks that assess compositional reasoning and subtle visual distinctions:
Winoground [14]:
Tests compositional reasoning abilities through 400 carefully designed instances that require understanding nuanced relationships between objects images and their textual descriptions.
NaturalBench [15] An expanded benchmark containing 1,200 instances that builds upon Winoground's 's principles with greater diversity.
MMVP [16]:
Comprises 135 instances distributed across 9 distinct visual reasoning cat- egories: Orientation and Direction, Presence of Specific Features, State and Condition, Quantity and Count, Positional and Relational Context, Color and Appearance, Structural and Physical Characteristics, Text, and Viewpoint and Perspective.
A large-scale benchmark with 2,933 instances, each categorized into one of three transformation types (Replace, Swap, or Add) that test different aspects of text-image alignment.
BiVLC 17
Evaluation protocol. Each test instance in the above datasets contains two images (I inng iing and two captions (r.it, ) forming two correct pairs (zi , .rBi)) and (a img ). The goal is to correctly associate images with their corresponding captions. The Image score measures whether the model
correctly identifies the image for both captions: [[s _ > s(x iim8) A s(xP", =x > s(xl", ),) where s(:,)) is the similarity function and 1[-] is the indicator function. Simi- larly, the Text score measures whether the model correctly identifies the caption for both images: img
img 1 [ss(z)me ,xx)) > s(x
iing 2 A s x > s(x) img
C Supplementary Experimental Results
C.1 Expanded Negative Pool for Contrastive Loss
Within single device, instead of computing similarities solely among the N anchor examples, we leverage the K retrieved hard negatives associated with each anchor. This expands the effective pool of examples for contrastive comparison to N(1 + K) per device. When extending this across multiple devices, we gather all anchor and hard negative examples globally. Since duplicates may arise in this aggregated set (c.g., e.. the same hard negative might be retrieved for different anchors), we perform
16

---

(a) General Retrieval
Fine-grained Retrieval
Mean Performance (9)
Mean Performance (%6
60-
52
15.0
12.5
50
56
48
Earsrurwse
Cars-ursse
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Epochs
Figure Impact Varying the Hyperparameter À on Retrieval Performance. The mean of the y-axis represents average performance across Text and Image retrieval tasks.
Figure 6: Evolution of the Modal- ity Gap Throughout the Training Process. The gap is computed on the MMVP dataset.
Method
baseline
MAPLE
80
Image Score
60
20.00
20
$3.33
13.35
+26.67
Perspective
Color
Orientation
Presence
Quantity
Spatial
State
Structural
Textt
Categories
Figure 7: Performance Breakdown Comparison on the MMVP dataset.
a deduplication step on the globally gathered examples to ensure uniqueness before computing the final contrastive loss.
C.2 Impact of Balancing CRPA with Lcontrast
Our full method combines the targeted CRPA with the general Ceomtras. We analyze the impact of the balancing weight ACRPA + (1 L Figure 5 illustratees performance trajectories as X changes from (LRPa only) abilities slowly get worse, while fine-grained discrimination shows clear improvement. This opposite relationship highlights a basic trade-off our approach. Finding the right value of À important-ite balance between general general retrieval and fine-grained retrieval. Also, our experiments show that LRPA-Liswise works much better than its pairwise version, with this advantage becoming more obvious at higher À values.
(Lcomtrast only). Interestingly, as increases, general retrieval
C.3 Impact of the strategy of sampling captions
For our caption generation approach, we generate comparative descriptions between image pairs. Specifically, for each anchor image and its top-3 hard negative images, construct six distinct image pairs by sampling without replacement from the four images. To ensure diversity the captions, we employ generation temperature of 0.7 and generate three captions per pair. This process yields
17

---

Table Sampling Strategy Comparison on Generated Captions. Best results are in bold. These
experiments are conducted Conntratt + LRPA-Listiie
General Retricval (R@1)
Fine-grained Retrieval
Strategy
COCO
Flickr30k
Winoground
NaturalBench
Text
Image
Text
Image
Text
Image
Text
Image
A
71,2
57.6
91.0
83.9
49.0
27.0
68.8
70.0
71.4
58.0
91.4
84.3
49.4
27.5
69.2
70.4
71.9
58.6
84.9
51.0
69.2
71.2
total of 18 comparative descriptions, with each image appearing in six different captions, resulting in a rich textual training corpus.
To understand the impact of different caption sampling strategies, we evaluate the following three approaches for each image. Strategy A uses only the first caption from each of the six generated caption sets, without any sampling. Strategy B randomly samples one caption from the first three generated captions per pair, without additional regeneration. Strategy C randomly samples from all six generated captions. As illustrated in Table 8 we find that increasing caption diversity gradually improves model performance. As for more sophisticated caption generation processes (e.g., leveraging multiple MLLMs generate comparative descriptions and mixing them together), we leave them for future work.
C.4 Detailed Performance Analysis on MMVP and BiVLC Datasets
While Table provides overall metrics across different fine-grained datasets, we further analyze improvements across specific visual patterns. The MMVP dataset cat- egorizes visual tasks into distinct patterns. As shown in Figure / the baseline demonstrates strong discrimina- tive capabilities for Color and State patterns, but performs considerably worse other patterns. Our MAPLE ap- proach delivers significant improvements on these chal- lenging patterns. Additionally, compare the baseline with MAPLE across 3 categories in the BiVLC dataset, where each category represents different dimension of variation between image-text pairs. Figure reveals that MAPLE consistently improves upon baseline performance, with particularly notable gains in the Swap category.
Metted
boctte
Image Store
Categories
Figure 8: Performance Breakdown Comparison on the BiVL.C dataset.
D Related Work
Cross-Modal Retrieval and the Modality Gap. Cross-modal retricval aims to establish correspon- dences between visual and textual information shared embedding space for effective search [39].
While contrastive learning approaches 1 40 have advanced the field by training separate en- coders paired datasets, they consistently face challenges from the modality gap 4 42 43
Our work introduces unified metric that quantifies this gap across different architectures, showing that MLLMs inherently possess strong cross-modal alignment capabilities. Recent MLLM-based retrieval models [8] T0
9 [1 44 have reduced this gap through unified architectures, but still rely on relatively simple alignment strategies that limit their effectiveness in tasks requiring nuanced cross-modal understanding.
Preference Learning and Direct Preference Optimization. Preference learning has become central technique in fine-tuning large language models, particularly in alignment with human feedback 45| (6). Direct Preference Optimization (DPO) [1] has emerged a principled alternative to reinforcement learning-based methods, offering a stable, reward-free approach for modeling preferences. While DPO has shown success natural language domains, its application to cross- modal representation learning remains underexplored. In this work, extend DPO to retrieval-basee
18

---

settings and propose reinforcement learning framework that transfers the fine grained modality aligmment priors of MLLMs into the cross-modal representations learned by an MLLM-base retriever.
E Visualization of Retrieval Results
Query: Uncut oranges
WD Prikees
Query: An airplane flying away from the camera
Santorr
Bisendar
Query: sign showing "22"
Siniirr
62
2
22
2,2
222
m
Query: room without the dog
Wi tretimon
Figure 9: Visualization Comparison. The retrieved images are sorted by similarity scores. The first row shows retrieval results without preference alignment, while the second row shows results with preference alignment. (Best viewed color and when zoomed in)
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Refer to Section eeperiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section FastionnIiisee discusses the limitations of this work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
20

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix B we provide detailed description of the experimental settings, including the model hyperparameter configurations, the complete training dataset curation process, and the evaluation protocol. These thorough specifications ensure that our reported results can be easily reproduced by other rescarchers.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
21

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should possible for other researchers have some path reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We plan release our training code and the complete data curation pipeline as open-source upon acceptance of the paper. However, at the current submission stage, we are unable to share any code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
Soblitt//uiiaee/dddeaaabisssssss iolci for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https http:///ccppaaal/ccaaoooo////oos
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section and Appendix C cover the main results and ablation studies, Ap- pendix B provides the detailed experimental settings.
Guidelines:
22

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The experiments in this work are conducted using Multimodal Large Language Models trained on large-scale datasets. The computational cost for training these models is substantial, making reporting error bars prohibitively expensive. To mitigate randomness, we fixed all random seeds across all experiments. Additionally, reporting error bars is not common practice in research involving MLLM due to the computational resources required.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-gim error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources along with training hyperparameters are provided in Appendix B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
23

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httprppss.//ppllic//cccccccieeie
Answer: [Yes]
Justification: This research adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss this in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

---

Answer: [NA]
Justification: Our work is based entirely on publicly available open-source datasets and models. We believe that this paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: For our experiments, we use publicly available open-source models and datasets, which we have properly cited throughout paper and used according to their original licensing requirements.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
25

---

Researchers should communicate the details of the dataset/codeemmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
26

---

Justification:
The paper does not involve LLMs as any important, original, or non- standdddd components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hutpss///eerrtiocccc///ooorreeee for what should or should not be described.
27


Reviews:
[{"review_id": "onXpyuE9nY", "reviewer": "Reviewer_yFbK", "summary": "This paper proposes MAPLE (Modality-Aligned Preference Learning for Embeddings), a novel framework that transfers alignment priors from off-the-shelf Multimodal Large Language Models (MLLMs) to cross-modal embedding models for fine-grained retrieval. MAPLE leverages MLLMs to generate automatic preference data and introduces a Relative Preference Alignment (RPA) loss, adapted from Direct Preference Optimization, to optimize embedding-level representations. Experiments show that MAPLE significantly outperforms CLIP and MLLM-based baselines, especially in nuanced semantic retrieval tasks.", "questions": "1. How robust is MAPLE to reward model size or architecture? Can smaller MLLMs still generate effective preference data?  \n2. Have the authors considered incorporating bias mitigation techniques to reduce the impact of biased MLLM outputs?", "limitations": "The paper acknowledges two limitations—MLLM bias and lack of evaluation on complex tasks such as compositional retrieval.", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros  \n1. The idea of transferring MLLM alignment priors via preference optimization is well-motivated.  \n2. This work is the first to adapt DPO-style preference optimization to cross-modal embedding learning, with both pairwise and listwise RPA losses.  \n3. The ablation study is well-structured and provides strong empirical support for the method design.  \n\nCons  \n1. The method is resource-intensive. MAPLE relies on large MLLMs (e.g., Qwen2-VL-72B) queried online and requires 32×A100 GPUs for training, limiting reproducibility.  \n2. The training signal depends entirely on pre-trained MLLMs, which may encode social or domain-specific biases. However, the paper does not explore any debiasing techniques or propose mitigation strategies for these risks.", "was_revised": true, "final_justification": "I have carefully reviewed the authors' rebuttal, particularly their supplementary experiments on empirical robustness analysis and reward model architectures. I appreciate the additional effort to address these points, and I understand the performance trade-offs discussed in their response.\n\nAfter considering the authors' clarifications and the perspectives of fellow reviewers, I have decided to maintain my original score.", "rebuttal": "Dear Reviewer yFbK,\n\nWe sincerely thank you for your constructive feedback and questions. We will address each point below.\n\n---\n\n> [W1] The method is resource-intensive. MAPLE relies on large MLLMs (e.g., Qwen2-VL-72B) queried online.\n\n[A1] We thank you for this important question regarding computational cost. First, we wish to clarify that the largest model (Qwen2-VL-72B) is used only for offline caption generation; our online reward model is the more efficient Qwen2-VL-7B.\n\nWhile our method uses 32 A100 GPUs, we believe this is a reasonable and even efficient cost when viewed in the context of current SOTA models. For instance, among the baselines we compare against, VladVA [1] uses an **identical 32 A100 GPU setup**. Furthermore, EVA-CLIP (8B) [2] requires **a significantly larger 384 A100 GPUs**. In this context, our resource requirement is not only competitive but also reflects an efficient path toward achieving strong fine-grained performance..\n\n> [W2, Q2] The training signal depends entirely on pre-trained MLLMs, which may encode social or domain-specific biases. How to mitigate it?\n\n[A2] Thank you for raising the issue of potential bias propagation from the reward MLLM. We address it from two complementary angles.\n\n1. **Empirical robustness analysis**: Inspired by your question, we benchmarked several reward models that differ in hallucination rate (HallusionBench [3] qAcc). The results below show that—even as qAcc varies—the downstream retrieval performance of MAPLE stays quite stable, demonstrating robustness to variations in hallucination‑ or bias‑related noise.\n\n*Ablation on Reward Model Architectures (Policy: Qwen2-VL-7B)*\n| Reward Model              | COCO (T/I)  | Flickr30k (T/I) | Winoground (T/I) | NaturalBench (T/I) | HallusionBench(qAcc↑) | Training Setup |\n| ------------------------- | ----------- | --------------- | ---------------- | ------------------ | ---- | ------ | \n| Baseline (No reward, 8e) | 74.0 / 54.4 | 93.6 / 80.3     | 42.5 / 20.5      | 61.4 / 62.5        | —    |  8 epochs / CL-only|\n| Baseline (No reward, 4e)  | 73.4 / 54.3 | 93.6 / 80.3     | 40.7 / 18.2      | 60.2 / 62.7        | —    | 4 epochs / CL-only| \n| SAIL-VL-1.6-8B            | 76.1 / 59.9 | 94.3 / 85.6     | 54.8 / 29.8      | 75.4 / 74.3        | 46.6 | 4 epochs / MAPLE      |\n| Qwen2-VL-7b               | 75.8 / 60.2 | 94.2 / 85.3     | 55.0 / 31.0      | 74.5 / 75.2        | 43.7 | 4 epochs / MAPLE      | \n| InternVL2.5-8B-MPO        | 75.5 / 59.5 | 93.8 / 85.1     | 53.5 / 30.8      | 74.1 / 74.2        | 42.6 | 4 epochs / MAPLE      | \n| InternVL2.5-8B            | 75.2 / 59.7 | 94.3 / 85.3     | 54.8 / 30.5      | 74.6 / 74.8        | 39.1 | 4 epochs / MAPLE      |\n\n**Note**:  \n*All MAPLE variants in this table are trained with 4 epochs to enable fast and fair comparison across reward models.  \nThe 8-epoch baseline corresponds to the standard CL-only setup reported in Table 2 of the main paper.  \nThe 4-epoch baseline was retrained for this ablation study.*\n\n2. **Built‑in mitigation mechanisms**:\n*   Relative‑ranking supervision: MAPLE optimizes pairwise/listwise consistency across multiple candidates, relying on relative scores instead of a single absolute judgment. This naturally attenuates occasional biased or noisy outputs.\n*   Contrastive regularizer: We mix in a standard contrastive loss to help keep the learned features stable and prevent the model from getting pulled too far by any quirks or noise in the reward model.\n\n> [Q1] the robustness of MAPLE to reward model size and architecture.\n\n[A3] Thank you for this thoughtful and important question — we agree that understanding MAPLE’s robustness to the reward model’s size and architecture is crucial for assessing its generality. \n\nTo this end, we conducted new experiments using reward models of varying scales (1B–8B) and architectures (Qwen2, InternVL). The results below show that MAPLE performs consistently well across this diverse set of reward models\n\n*Ablation on Reward Model Architectures (Policy: Qwen2-VL-7B)*\n| Reward Model | COCO (T/I) | Flickr30k (T/I) | Winoground (T/I) | NaturalBench (T/I) |\n| :--- | :--- | :--- | :--- | :--- |\n| Baseline (No reward) | 73.4 / 54.3 | 93.6 / 80.3     | 40.7 / 18.2      | 60.2 / 62.7 |\n| Qwen2-VL-2b | 74.1 / 59.1 | 93.0 / 84.1 | 53.5 / 31.0 | 70.4 / 72.3 |\n| Qwen2-VL-7b | 75.8 / 60.2 | 94.2 / 85.3 | 55.0 / 31.0 | 74.5 / 75.2 |\n| InternVL3-1B | 73.9 / 58.9 | 92.8 / 83.7 | 48.5 / 26.0 | 69.3 / 72.6 |\n| InternVL3-2B | 75.9 / 59.2 | 94.2 / 84.8 | 53.8 / 28.5 | 72.8 / 74.3 |\n| InternVL3-8B | 75.6 / 59.7 | 93.8 / 84.8 | 54.0 / 31.5 | 74.3 / 74.8 |\n\n**Our key findings** are:\n1.  Robustness to model size: While larger reward models generally yield stronger results, even small models like Qwen2-VL-2B and InternVL3-1B lead to consistent performance gains, demonstrating that MAPLE can effectively leverage small reward models.\n2.  Robustness to architecture: MAPLE is agnostic to reward model type. Notably, using InternVL models with a Qwen2 policy still leads to clear improvements, confirming that MAPLE generalizes across model families.\n\nWe believe these findings reinforce the flexibility and robustness of MAPLE’s preference-based learning. We will include this analysis in the final version of the paper.\n\n\nReferences:\n\n[1] Ouali Y, Bulat A, Xenos A, et al. VladVA: Discriminative Fine-tuning of LVLMs[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 4101-4111.  \n[2] Sun Q, Wang J, Yu Q, et al. Eva-clip-18b: Scaling clip to 18 billion parameters[J]. arXiv preprint arXiv:2402.04252, 2024.  \n[3] Guan T, Liu F, Wu X, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 14375-14385. \n\n---\n\n**Thank you again for your valuable comments. We’ve carefully addressed each concern and would be happy to discuss any further questions. We’ll stay responsive during the entire rebuttal phase and look forward to hearing more from you!**"}, {"review_id": "5JFMWEfbQg", "reviewer": "Reviewer_ztPf", "summary": "This paper introduces MAPLE (Modality-Aligned Preference Learning for Embeddings), a framework that leverages off-the-shelf Multimodal Large Language Models (MLLMs) to guide cross-modal representation learning through preference alignment. The key insight is that MLLMs inherently possess stronger modality alignment capabilities than traditional CLIP-based models with Wasserstein distance-based metric (lines 105-108). Based on this observation, the authors construct DPO training data and propose two RPA losses inspired by preference optimization.", "questions": "Training efficiency and practical deployment: Given the computational overhead of dynamic MLLM scoring during training, how does the total training cost compare to strong baselines like EVA-CLIP? Have the authors considered offline preference data construction or distillation approaches that could reduce inference costs while maintaining performance gains?\n\nHow sensitive are the results to the choice of MLLM architecture? Have the authors tested whether preferences from different MLLMs (e.g., qwen2.5vl, internvl2 and llava ov) lead to consistent improvements, and what happens when reward and policy models have different architectural biases?\n\nHow to handle MLLM hallucinations, what is the hallucination rate in pairs, and whether there are relevant explorations?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\nI think the lightweight MLLM prior approach is what the current community needs, particularly using MLLM reasoning/preference [1] and other general capabilities to efficiently enhance cross-modal representation learning.\n\nFrom the data requirement perspective, adopting more efficient post-training methods to solve alignment problems also makes technical sense. The adaptation of DPO to embedding learning through RPA loss is technically sound and addresses a genuine limitation of coarse-grained contrastive learning.\n\nBesides the motivation [figure 3], I think this work is expressed quite clearly, such as the example in Figure 1. The presentation and experimental completeness are also professional and standardized, which is above baseline level. It also comes with detailed appendices, which seems to meet the requirements of the NeurIPS community.\n\nMinor revision suggestion:\n1. Add commas after formulas (if the following word is lowercase like \"where\") or periods (if the following word is capitalized)\n2. There are still some typos, for example line 122 mentions that Preference Data Construction is offline, but Figure 2 shows it as offline. I understand it should be that the data for subsequent DPO can be prepared offline.\n3. The text in Figure 2 could be optimized - the text size should match the caption and use Times New Roman font\n4. From a writing perspective, Figure 3 could be moved into the main text to strengthen the motivation.\n\nWeaknesses:\nThe computational overhead and scalability concerns are significant. The method requires dynamic MLLM inference during training to compute alignment scores (lines 146-150), significantly increasing computational cost. Particularly for Scoring and Structuring Preferences, can this be done offline as an efficient DPO dataset?\n\nThe method fundamentally relies on off-the-shelf MLLMs as preference oracles (e.g., Qwen2-VL in line 238-240), but **lacks discussion of how MLLM hallucinations affect preference quality**. Since the approach's effectiveness is directly tied to the reliability of MLLM alignment scores, how do the authors address cases where the reward model provides inconsistent or incorrect preferences? The paper would benefit from ablation studies examining performance degradation when using MLLMs with known hallucination issues, and analysis of the method's robustness bounds relative to MLLM reliability.\n\n[1] Zhao X et al. Omnialign-v: Towards enhanced alignment of mllms with human preference[J]. arXiv preprint arXiv:2502.18411, 2025.", "was_revised": true, "final_justification": "I have updated my score accordingly, given the rebuttal provided clarify some of my concerns, I raise my score to weak accept accordingly.", "rebuttal": "Dear Reviewer ztPf,\n\nWe are sincerely grateful for your positive and insightful review. We will address each point below.\n\n---\n\n> [W1, Q1] Training Efficiency and Offline Data Construction.\n\n[A1] Thank you for raising this important point regarding computational efficiency and deployment practicality.\n\nFirst, while MAPLE involves online scoring with MLLMs during training, its overall training cost remains significantly lower than many large-scale baselines. For instance, EVA-CLIP [3] **requires over 360 A100 GPUs**, while VladVA [4] — **another strong baseline included in our comparison — uses 32 A100s**, the same as ours. In this context, MAPLE’s resource usage is both reasonable and efficient, achieving strong fine-grained performance without excessive compute demands.\n\nFurthermore, once training is complete, MAPLE requires only the lightweight policy model for deployment — similar to EVA-CLIP and VladVA. No reward model inference is needed at test time, making MAPLE comparably efficient in real-world deployment scenarios.\n\nSecond, we agree with you that offline preference construction is a promising direction to further reduce cost. To investigate this, we conducted an ablation simulating an offline scenario (Appendix, Table 6), where preferences were precomputed using only one caption per image. This setup removes the need for dynamic scoring. We found that the performance remained strong, with only a modest regression compared to the online version. This confirms that MAPLE remains effective even under a fully offline setting.\n\nIn this work, we present the online variant to demonstrate the upper bound of what MAPLE can achieve. For low-resource setups, the offline variant is a viable and efficient alternative. We will emphasize this trade-off more clearly in the camera-ready version.\n\n> [W2, Q3] Robustness to MLLM Hallucinations.\n\n[A2] We sincerely thank you for raising this important concern and for pointing us to the valuable OmniAlign-V paper [2], which inspired us to conduct further robustness analysis.\n\nSpecifically, we evaluated a range of reward models using both HallusionBench [1] and MM-AlignBench [2], as well as their downstream performance in MAPLE. The results are shown below:\n\n*Ablation on Reward Model Architectures (Policy: Qwen2-VL-7B)*\n| Reward Model | COCO (T/I) | Winoground (T/I) | NaturalBench (T/I) | HallusionBench(qAcc↑) | MM-AlignBench(Win Rate↑) | Training Setup |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| Baseline (No reward, 8e) | 74.0 / 54.4 | 42.5 / 20.5      | 61.4 / 62.5        | —    | —    | 8 epochs / CL-only|\n| Baseline (No reward, 4e)  | 73.4 / 54.3 | 40.7 / 18.2      | 60.2 / 62.7        | —    | —    |4 epochs / CL-only| \n| SAIL-VL-1.6-8B | 76.1 / 59.9 | 54.8 / 29.8 | 75.4 / 74.3 | 46.6 | - | 4 epochs / MAPLE|\n| Qwen2-VL-7b | 75.8 / 60.2 | 55.0 / 31.0 | 74.5 / 75.2 | 43.7 | 44.4 | 4 epochs / MAPLE|\n| InternVL2.5-8B-MPO | 75.5 / 59.5 | 53.5 / 30.8 | 74.1 / 74.2 | 42.6 | 40.1 | 4 epochs / MAPLE|\n| InternVL3-8B | 76.9 / 61.6 | 54.0 / 31.5 | 74.3 / 74.8 | 40.2 | - | 4 epochs / MAPLE|\n| InternVL2.5-8B | 75.2 / 59.7 | 54.8 / 30.5 | 74.6 / 74.8 | 39.1 | 31.3 | 4 epochs / MAPLE|\n\n**Note**:  \n*All MAPLE variants in this table are trained with 4 epochs to enable fast and fair comparison across reward models.  \nThe 8-epoch baseline corresponds to the standard CL-only setup reported in Table 2 of the main paper.  \nThe 4-epoch baseline was retrained for this ablation study.*\n\nThese results reveal a striking insight: reward model quality on standalone hallucination benchmarks does not directly predict its effectiveness within MAPLE. For instance, **InternVL2.5‑8B achieves the lowest qAcc and win-rate among all models, yet still enables strong fine-grained retrieval performance when used as a reward model**.\n\nThis empirical finding supports our hypothesis that MAPLE’s preference-based learning framework exhibits strong resilience to a certain degree of hallucination or bias noise. We attribute this to two core design choices:\n*   Robustness through relative rankings: Instead of relying on absolute correctness of a single caption or label, our method uses pairwise and listwise ranking — which inherently tolerates occasional noisy preferences, as long as the global ordering remains coherent.\n*   Contrastive regularization: Our training objective includes a standard contrastive loss, which anchors the learned representations and prevents the policy model from overfitting to erroneous reward signals.\n\n> [Q2] Robustness to MLLM Architecture.\n\n[A3] Thank you for this important question. We agree that architectural mismatch between the policy and reward model is a valid concern when transferring alignment priors.\n\nTo investigate this, we present a set of cross-architecture experiments, where we evaluate all combinations of two distinct MLLM families: Qwen2-VL and InternVL3. The results are summarized below:\n\n| Policy Model | Reward Model | COCO (T/I)  | Flickr30k (T/I) | Winoground (T/I) | NaturalBench (T/I) |\n| ------------ | ------------ | ----------- | --------------- | ---------------- | ------------------ |\n| Qwen2-VL-7B  | Baseline (No reward)| 73.4 / 54.3 | 93.6 / 80.3     | 40.7 / 18.2      | 60.2 / 62.7\n| Qwen2-VL-7B  | Qwen2-VL-7B  | 75.8 / 60.2 | 94.2 / 85.3     | 55.0 / 31.0      | 74.5 / 75.2        |\n| Qwen2-VL-7B  | InternVL3-8B | 75.6 / 59.7 | 93.8 / 84.8     | 54.0 / 31.5      | 74.3 / 74.8        |\n| InternVL3-8B | Qwen2-VL-7B  | 76.7 / 61.1 | 95.4 / 86.8     | 53.5 / 31.5      | 78.4 / 77.7        |\n| InternVL3-8B | InternVL3-8B | 76.9 / 61.6 | 95.9 / 87.5     | 53.5 / 30.5      | 78.4 / 79.1        |\n\nWe highlight two important takeaways:\n\n*   Cross-architecture learning remains highly effective: All combinations lead to strong performance, including mismatched reward-policy setups. This shows that* *MAPLE benefits from reward models of varying architectures, consistently outperforming CL-only baselines and demonstrating robustness to architectural mismatch**.\n*   Surprisingly, some cross-family setups even outperform our original results: For example, the fully InternVL3-8B setup achieves stronger overall numbers than those reported in our main paper using Qwen2-VL. This further supports the generalizability and extensibility of our framework.\n\nWe also note that the earlier table in our response to [W2, Q3] included similar cross-architecture observations (e.g., using SAIL-VL or InternVL2.5-8B as rewards for Qwen2-VL), reinforcing this conclusion. However, the new experiments here provide a more systematic and exhaustive evaluation across architectures, offering deeper insights.\n\nTogether, these results strengthen our claim that MAPLE is architecturally robust and can flexibly accommodate a wide variety of MLLM reward sources without requiring tight coupling between model families.\n\n> On Minor Revisions.\n\n[A4] Thank you for these thoughtful suggestions to improve the clarity and presentation of the paper. \n*   Punctuation & Figure 2 Style: We have revised the manuscript to ensure consistent punctuation after inline formulas and have updated Figure 2’s text style to match the rest of the paper, including font size and Times New Roman typography.\n*   Offline vs. Online Clarification: Preference data construction in our method consists of both offline and online components. Specifically, hard negative mining is performed offline, while dynamic scoring via MLLMs is carried out online during training. This hybrid design is reflected in Figure 2, where the corresponding stages are clearly labeled as “offline” and “online”.\n*   Figure 3 Placement: Thank you very much for this suggestion. We initially intended to place Figure 3 under the \"Measuring the Modality Gap\" subsection in Section 2: Preliminaries and Notation. However, due to page limits, we had to move it to the appendix. We agree that integrating it into the main text would help strengthen the motivation, and we plan to restore it in the camera-ready version.\n\nReferences:\n\n[1] Guan T, Liu F, Wu X, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 14375-14385.  \n[2] Zhao X, Ding S, Zhang Z, et al. Omnialign-v: Towards enhanced alignment of mllms with human preference[J]. arXiv preprint arXiv:2502.18411, 2025.  \n[3] Sun Q, Wang J, Yu Q, et al. Eva-clip-18b: Scaling clip to 18 billion parameters[J]. arXiv preprint arXiv:2402.04252, 2024.  \n[4] Ouali Y, Bulat A, Xenos A, et al. VladVA: Discriminative Fine-tuning of LVLMs[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 4101-4111.  \n\n---\n\n**Thank you once again for your valuable and constructive comments. We’ve responded to each concern with care and would love to clarify anything further. We’ll stay actively engaged throughout the rebuttal phase and truly look forward to continuing this conversation.**"}, {"review_id": "9WEea45OWa", "reviewer": "Reviewer_Bq5C", "summary": "This paper aims to solve the \"modality gap\" problem that exists in models like CLIP. The work finds that off-the-shelf Multimodal Large Language Models (MLLMs) possess stronger inherent alignment capabilities. To this end, the paper proposes a framework called MAPLE, which utilizes a powerful MLLM to automatically construct preference data and introduces a novel Relative Preference Alignment (RPA) loss function, inspired by Direct Preference Optimization, to guide the learning of the retrieval model. Experiments show that this method significantly narrows the modality gap and achieves excellent performance on multiple cross-modal retrieval benchmarks, with particularly significant improvements on fine-grained tasks.", "questions": "NA.", "limitations": "- Only Qwen2 series is adopted. \n- The training of MAPLE hearily relyies the hardware, a cluster of 32 NVIDIA A100 GPUs. Theferfore, the producibility is very limited.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros: \n- The introduction of a unified metric (Δgap) based on Wasserstein Distance. This provides the research community with a valuable tool to measure and compare the modality gap across different model architectures.\n- The experimental design is highly comprehensive. \n\nCons:\n- Figure 5 reveals a performance trade-off: optimizing for fine-grained tasks comes at the expense of retrieval capabilities, challenging the generalization of its learned representations.\n- MAPLE framework's high dependency on a single reward model raises questions about whether it is learning a general representation or simply \"overfitting\" to that specific model's biases and flaws.\n- The method's key paradigm is a complex form of knowledge distillation, using a large model to guide a smaller one, which weakens its claim to fundamental algorithmic novelty.", "was_revised": true, "final_justification": "The author's detailed responses including the performance trade-off, complexity, along with the ablation experiments conducted on multiple reward models, have successfully resolved my doubts. Consequently, I raised my score.", "rebuttal": "Dear Reviewer Bq5C,\n\nWe sincerely thank you for your detailed review and insightful comments. We will address your concerns below.\n\n---\n\n> [W1] the Retrieval Performance Trade-off.\n\n[A1] We appreciate your insightful observation regarding the performance trade-off. This phenomenon indeed reflects a fundamental tension in representation learning: optimizing for fine-grained discrimination often comes at the cost of global retrieval generalization.\n\nImportantly, **this trade-off is not unique to our method, but appears to be a common challenge across cross-modal learning frameworks**. For instance, Alhamoud et al. (CVPR 2025) [1] report a similar effect when jointly training with a standard CLIP loss and a fine-grained MCQ loss — performance on broad retrieval tasks slightly declines as the model focuses more on nuanced distinctions.\n\nIn our case, this behavior is expected given MAPLE's design goal: to push beyond coarse alignment and capture subtle semantic relationships. We believe the ability to make these fine-grained distinctions is valuable for many real-world applications, even if it introduces a slight drop in generic retrieval performance.\n\n> [W2, L1] Generalization and Dependency on a Single Reward Model.\n\n[A2] We appreciate your thoughtful question on generalization and potential over-reliance on a single reward model. To directly examine this, we conducted a thorough ablation study during the rebuttal period, where we tried the different reward model — guiding the same Qwen2-VL-7B policy using reward signals from diverse architectures, including InternVL3-8B and SAIL-VL-1.6-8B.\n\nThe results are summarized below:\n\n*Ablation on Reward Model Architectures (Policy: Qwen2-VL-7B)*\n| Reward Model              | COCO (T/I)  | Flickr30k (T/I) | Winoground (T/I) | NaturalBench (T/I) | Training Setup |\n| ------------------------- | ----------- | --------------- | ---------------- | ------------------ | ------ | \n| Baseline (No reward, 8e) | 74.0 / 54.4 | 93.6 / 80.3     | 42.5 / 20.5      | 61.4 / 62.5        | 8 epochs / CL-only|\n| Baseline (No reward, 4e)  | 73.4 / 54.3 | 93.6 / 80.3     | 40.7 / 18.2      | 60.2 / 62.7        | 4 epochs / CL-only| \n| SAIL-VL-1.6-8B            | 76.1 / 59.9 | 94.3 / 85.6     | 54.8 / 29.8      | 75.4 / 74.3        | 4 epochs / MAPLE      |\n| Qwen2-VL-7b               | 75.8 / 60.2 | 94.2 / 85.3     | 55.0 / 31.0      | 74.5 / 75.2        | 4 epochs / MAPLE      | \n| InternVL2.5-8B            | 75.2 / 59.7 | 94.3 / 85.3     | 54.8 / 30.5      | 74.6 / 74.8        | 4 epochs / MAPLE      |\n\n**Note**:  \n*All MAPLE variants in this table are trained with 4 epochs to enable fast and fair comparison across reward models.  \nThe 8-epoch baseline corresponds to the standard CL-only setup reported in Table 2 of the main paper.  \nThe 4-epoch baseline was retrained for this ablation study.*\n\nDespite architectural differences, the downstream performance remains consistently strong. This strongly suggests that **MAPLE is not merely fitting to the specific model's biases and flaws, but rather learning a transferable, general alignment signal**.\n\nWe will include these findings in our final version, as we agree this is a valuable aspect of MAPLE’s behavior.\n\n> [W3] On the Novelty of MAPLE beyond Knowledge Distillation.\n\n[A3] Thank you for raising this important point. While MAPLE does involve using a freezing model to guide another model, we would like to emphasize that our framework is fundamentally different from traditional knowledge distillation.\n\nClassic distillation methods aim to reproduce a teacher model’s output — whether that be logits, soft labels, or hidden representations — often minimizing a direct divergence between the student and teacher distributions.\n\nIn contrast, MAPLE introduces a distinct learning paradigm:\n*   The reward model is not the learning target.\n*   It serves only to **provide relative preference judgments** (e.g., “A is better than B”) among multiple retrieval candidates.\n*   Our policy model is trained using a Direct Preference Optimization–inspired (DPO) loss — applied on these rankings — rather than attempting to mimic any output of the reward model.\n*   In this sense, MAPLE is closer to preference-based alignment than to classical distillation.\n\nMoreover, we conducted experiments where the reward model is much smaller than the policy model (e.g., using InternVL3-1B or Qwen2-VL-2B to guide Qwen2-VL-7B), The results are shown below, yet we still observe meaningful performance gains.\n\n*Ablation on Reward Model Architectures (Policy: Qwen2-VL-7B)*\n| Reward Model | COCO (T/I) | Flickr30k (T/I) | Winoground (T/I) | NaturalBench (T/I) |\n| :--- | :--- | :--- | :--- | :--- |\n| Baseline (No reward)  | 73.4 / 54.3 | 93.6 / 80.3     | 40.7 / 18.2      | 60.2 / 62.7 |\n| Qwen2-VL-2b | 74.1 / 59.1 | 93.0 / 84.1 | 53.5 / 31.0 | 70.4 / 72.3 |\n| Qwen2-VL-7b | 75.8 / 60.2 | 94.2 / 85.3 | 55.0 / 31.0 | 74.5 / 75.2 |\n| InternVL3-1B | 73.9 / 58.9 | 92.8 / 83.7 | 48.5 / 26.0 | 69.3 / 72.6 |\n| InternVL3-2B | 75.9 / 59.2 | 94.2 / 84.8 | 53.8 / 28.5 | 72.8 / 74.3 |\n| InternVL3-8B | 75.6 / 59.7 | 93.8 / 84.8 | 54.0 / 31.5 | 74.3 / 74.8 |\n\nThis provides strong empirical evidence that MAPLE does not merely inherit large model capabilities — it extracts useful alignment signals from relative scoring, even when the reward model is weaker.\n\n> [L2] Training Resource Cost and Reproducibility.\n\n[A4] Thank you for raising this valuable concern. While our main experiments use 32 A100 GPUs, we note that this is still an order of magnitude lower than some comparable SOTA methods (e.g., EVA-CLIP-8B requires 384 A100s, and VladVA also uses 32 A100s). \n\nMore importantly, MAPLE is designed with efficiency in mind. To explore cost-effective alternatives, we simulated a purely offline variant of our framework (Appendix, Table 6), where preferences could be precomputed offline. This setup removes the need for online dynamic scoring, with only a minor drop in performance. We believe this variant makes MAPLE highly practical.\n\nTo further ensure reproducibility and community adoption, we plan to release all training code.\n\n> On Paper Formatting Concerns. \n\n[A5] Thank you for pointing this out. We agree that the main text should be more self-contained. We will revise the paper to integrate the most critical details from the appendix into the main body, improving the overall readability and flow.\n\nReferences:  \n[1] Alhamoud K, Alshammari S, Tian Y, et al. Vision-language models do not understand negation[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 29612-29622.\n\n---\n\n**We sincerely appreciate your insightful feedback. We've done our best to address each of your points, and we're happy to engage further if any questions remain. We're looking forward to hearing your thoughts during the remainder of the rebuttal period**."}, {"review_id": "4oo8xhUCdr", "reviewer": "Reviewer_34CD", "summary": "This paper addresses the persistent \"modality gap\" in cross-modal (image-text) retrieval, a discrepancy between visual and textual features in their shared embedding space. The paper introduces training framework named MAPLE. The core idea of MAPLE is to leverage a powerful, off-the-shelf MLLM as a \"teacher\" or \"reward model\" to transfer its fine-grained alignment priors to a \"student\" model intended for retrieval. Experimental results demonstrate that MAPLE significantly outperforms existing CLIP-based and MLLM-based retrieval models on several general and, most notably, fine-grained retrieval benchmarks.", "questions": "1. Since the performance appears fundamentally upper-bounded by the reward model, have you investigated the impact of using a less capable MLLM (e.g., a 1.5B or smaller model) for scoring?\n2. The RPA loss is weighted by the difference in alignment scores from the reward MLLM. This implies that distinguishing between two very similar, high-scoring samples (small difference) receives less weight than distinguishing between a high-scoring and a low-scoring sample (large difference). Is this design optimal in all cases? Could it potentially cause the model to neglect learning the most difficult, fine-grained distinctions?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed metric, based on Wasserstein distance, providing a effective tool for quantifying and comparing the alignment capabilities of different model architectures.\n2. This paper intelligently extends the principles of DPO from language model alignment to the domain of cross-modal representation learning.\n3. The experiments are thorough.\n\nWeaknesses:\n1. The proposed framework is highly complex and computationally expensive. The entire pipeline requires a powerful MLLM (e.g., Qwen2.5-VL-72B) for caption generation and continuously uses another large MLLM (Qwen2-VL-7B) as a reward model for dynamic scoring during training.\n2. The absence of an ablation study on the choice of the reward model. It is unclear how performance would degrade if a smaller or different family of MLLM were used. This is crucial for understanding the method's robustness and generalization.", "was_revised": true, "final_justification": "The author's explanation of complexity and the ablation experiment on the reward model have addressed my concerns. I believe that the contribution of the method outweighs the improvement in complexity. Therefore, I have decided to maintain my score and am inclined to accept this paper.", "rebuttal": "Dear Reviewer 34CD,\n\nWe sincerely thank you for your positive assessment and for your sharp and insightful questions, which help us clarify the finer details of our method.\n\n---\n> [W1] On the Computational Cost.\n\n[A1] We agree that computational efficiency is a key consideration. While our online training variant does involve dynamic reward scoring and uses 32 A100 GPUs, we emphasize that this is already significantly more efficient than many existing state-of-the-art models. For example, **EVA-CLIP (8B) requires over 384 A100 GPUs, and VladVA—a strong baseline in our comparison—also uses 32 A100s**, making our setup well-aligned with current standards in this domain. \n\nThat said, we fully understand the need for accessible alternatives. To this end, we simulated a low-resource “offline-style” training strategy (Appendix, Table 6), where preferences are precomputed once using only a single caption per image, removing the need for dynamic reward scoring during training. This dramatically reduces the overall cost to the level of standard supervised fine-tuning.\n\nEncouragingly, this offline variant achieves strong performance, with only a modest drop compared to the online version. This result confirms that MAPLE remains effective even without heavy online inference。\n\nIn this paper, we presented the online variant to demonstrate the performance upper bound of our framework, but we will clearly highlight the offline variant as a viable, efficient alternative in the final version.\n\n> [W2, Q1] On the Robustness to the Choice of Reward Model.\n\n[A2] Thank you for highlighting this important point. We fully agree that a method’s robustness to the choice of reward model is central to its generality and practical value.\n\nTo directly address your concern, we conducted a comprehensive ablation study during the rebuttal period. We evaluated MAPLE’s performance across a range of reward models that vary in both scale (1B–8B) and architecture family (Qwen2, InternVL), while keeping the policy model fixed (Qwen2-VL-7B).\n\n*Ablation on Reward Model Architectures (Policy: Qwen2-VL-7B)*\n| Reward Model              | COCO (T/I)  | Flickr30k (T/I) | Winoground (T/I) | NaturalBench (T/I) | Training Setup |\n| ------------------------- | ----------- | --------------- | ---------------- |  ---- | ------ | \n| Baseline (No reward, 8e) | 74.0 / 54.4 | 93.6 / 80.3     | 42.5 / 20.5      | 61.4 / 62.5 |  8 epochs / CL-only|\n| Baseline (No reward, 4e)  | 73.4 / 54.3 | 93.6 / 80.3     | 40.7 / 18.2      | 60.2 / 62.7 | 4 epochs / CL-only| \n| Qwen2-VL-2b | 74.1 / 59.1 | 93.0 / 84.1 | 53.5 / 31.0 | 70.4 / 72.3 | 4 epochs / MAPLE      |\n| Qwen2-VL-7b | 75.8 / 60.2 | 94.2 / 85.3 | 55.0 / 31.0 | 74.5 / 75.2 | 4 epochs / MAPLE      |\n| InternVL3-1B | 73.9 / 58.9 | 92.8 / 83.7 | 48.5 / 26.0 | 69.3 / 72.6 | 4 epochs / MAPLE      |\n| InternVL3-2B | 75.9 / 59.2 | 94.2 / 84.8 | 53.8 / 28.5 | 72.8 / 74.3 | 4 epochs / MAPLE      |\n| InternVL3-8B | 75.6 / 59.7 | 93.8 / 84.8 | 54.0 / 31.5 | 74.3 / 74.8 | 4 epochs / MAPLE      |\n\n**Note**:  \n*All MAPLE variants in this table are trained with 4 epochs to enable fast and fair comparison across reward models.  \nThe 8-epoch baseline corresponds to the standard CL-only setup reported in Table 2 of the main paper.  \nThe 4-epoch baseline was retrained for this ablation study.*\n\nKey takeaways:\n*   Robustness to Size: As expected, larger reward models tend to offer better supervision (e.g., 7B > 2B), but even **the smallest models (e.g., InternVL3-1B) still yield strong improvements, confirming that MAPLE performs well even with relatively limited reward capacity**.\n*   Robustness to Architecture: Reward models from different families (InternVL, Qwen2) are all effective, indicating that **MAPLE is not overfitting to a specific architecture**. Instead, it generalizes across diverse MLLMs, so long as they provide meaningful alignment signals.\n\nTogether, these results suggest that MAPLE is not overly dependent on a single powerful reward model, and can maintain effectiveness across different model configurations. We believe these findings reinforce the flexibility and robustness of MAPLE’s preference-based learning. We will include this analysis in the final version of the paper. \n\n> [Q2] On the Design of the RPA Loss Weighting.\n\n[A3] We are very grateful for this deep and insightful question about the loss weighting. Our preference pairs are constructed via KNN retrieval, meaning they are already hard negatives by design. A small reward margin in our setting may indicate a near-duplicate or extremely similar pair since all candidate samples are already selected via KNN-based hard negative mining. In other words, even the reward model struggles to differentiate the samples. Over-weighting such pairs could risk forcing the embedding space to learn noisy, non-semantic variations, which may lead to unstable gradients or even representational overfitting. \n\nFrom this perspective, our weighting mechanism can be interpreted as a form of confidence-aware learning — giving more emphasis to the samples the reward model is more certain about, while still preserving training signal from hard examples, since all candidates are retrieved through KNN-based hard negative mining.\n\nBased on this intuition, we performed a preliminary comparison between weighted and unweighted RPA losses under partial training settings. The weighted variant yielded clearer improvements on fine‑grained benchmarks, which informed our final design choice. These exploratory results were not included in the submission, given your concern, we will include these results in the appendix of the revised version.\n\n| Strategy | COCO (T/I) | Flickr30k (T/I) | Winoground (T/I) | NaturalBench (T/I) |\n| :--- | :--- | :--- | :--- | :--- |\n| Weighted | 71.7/57.2 | 92.9/83.8 | **52.0/24.0** | **67.2/70.6** |\n| Unweighted | 72.5/56.7 | 92.7/83.3 | 49.5/23.4 | 65.7/69.8 |\n\n---\n\n**Thank you again for your thoughtful feedback. We’ve carefully addressed all your concerns and remain open to any further discussion. We look forward to continuing the discussion throughout the rebuttal phase.**"}]

Author rebuttal:

Meta-review: This paper ended with 2 borderline accept scores, 1 accept score, and 1 borderline reject. Initially, reviewers liked the paper for its method being well motivated; the clarity of writing; and highly comprehensive experiments. However, they had some concerns regarding the resource intensive nature of the proposed method and lack of discussion regarding hallucinations/viability of different MLLMs which were both mentioned by multiple reviewers. The rebuttal attempted to address these key issues, with a large discussion over the online training stage only requiring a smaller model (compared to the offline captions) as well as new results comparing different MLLMs and showcasing that MAPLE was not reliant on a single MLLM. After the discussion, the majority of reviewers ended with a positive final rating with one reviewer recommending reject. The AC has looked at the arguments from both sides and agrees with the majority of reviewers recommending acceptance in this case: the carefully thought out experiments and discussion within the rebuttal provide evidence for the claims in the paper and answer the reviewers earlier questions. Accordingly the AC recommends acceptance for this paper.

The AC reminds the authors to make the changes they promised during the rebuttal and any changes specified as necessary by the reviewers for the camera ready version.
Decision: Accept (poster)