Supervised Fine-Tuning or Contrastive Learning? Towards Better Multimodal LLM Reranking
Authors: Xin Zhang
Ziqi Dai
Mingxin Li
Yanzhao Zhang
Dingkun Long
Pengjun Xie
Meishan Zhang
Wenjie Li
Min Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Retrieval
LLM Reranking
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: In information retrieval, training reranking models focus mainly on two types of objectives: metric learning (e.g., contrastive loss to increase predicted scores on relevant query-document pairs) and classification (binary label prediction of relevance vs. irrelevance). For BERT-style encoders, various studies have shown that contrastive learning (CL) can be more effective than discriminative (classification) learning. However, for large language models (LLMs), classification via supervised fine-tuning (SFT), which predicts "yes" (resp. "no") token for relevant (resp. irrelevant) pairs, appears more promising as it aligns well with the generative nature of LLMs. This divergence raises a central question: which objective is intrinsically better suited to LLM-based reranking, and what mechanism underlies the difference? In this work, we conduct a comprehensive comparison and analysis between CL and SFT for reranking, taking the universal multimodal retrieval (UMR) as the experimental playground. We first decompose the objectives into two components: weight, which controls the magnitude of those updates, and direction, which guides the model updates, then present a unified framework for understanding their interactions. Through probing experiments, we find that SFT provides a substantially stronger weighting scheme than CL, whereas the preferred scoring direction shows no clear winner. Taken together, these results point to a consistent advantage of SFT over CL for LLM reranking. To further validate our findings, we conduct large-scale training with SFT and present new state-of-the-art rerankers on the compiled MRB benchmark. We also provide ablations on SFT settings and expect our findings to benefit future research and applications.

Published as a conference paper at ICLR 2026
SUPERVISED FINE-TUNING OR CONTRASTIVE LEARN- ING? TOWARDS BETTER MULTIMODAL LLM RERANKING
Xin Zhang- 2,; Ziqi Dai'; Mingxin Li, Yanzhao Zhang, Dingkun Long Pengjun Xie, Meishan Zhang', Wenjie Li2, Min Zhang'
Harbin Institute of Technology, Shenzhen The Hong Kong Polytechnic University {ziqi ,aai, zhangxin2023)@sstu Model released at http:////////////o////////////aa
cn
ABSTRACT
In information retrieval, training reranking models focus mainly on two types of objectives: metric learning (e.g., contrastive loss to increase predicted scores on relevant query-document pairs) and classification (binary label prediction of rel- evance vs. irrelevance). For BERT-style encoders, various studies have shown that contrastive learning (CL) can be more effective than diseriminative (classifi- cation) learning. However, for large language models (LLMs), classification via supervised fine-tuning (SFT), which predicts "yes" (resp. "no") token for relevant (resp. irrelevant) pairs, appears more promising as it aligns well with the genera- tive nature of L.L.s. This divergence raises a central question: which objective is intrinsically better suited to LLM-based reranking, and what mechanism under- lies the difference? In this work, we conduct a comprehensive comparison and analysis between CL and SFT for reranking, taking the universal multimodal re- trieval (UMR) as the experimental playground. We first decompose the objectives into two components: weight, which controls the magnitude of those updates, and direction, which guides the model updates, then present a unified framework for understanding their interactions. Through probing experiments, we find that SFT provides substantially stronger weighting scheme than CL, whereas the preferred scoring direction shows no clear winner. Taken together, these results point to a consistent advantage of SFT over CL for LLM reranking. To further validate our findings, we conduct large-scale training with SFT and present new wate-offthe art rerankers the compiled MRB benchmark. We also provide ablations on SFT settings and expect our findings to benefit future research and applications.
INTRODUCTION
Reranking is crucial step in the retrieval pipeline (Lin et 2022), aiming to refine the initial re- sults obtained from the previous search stage by reordering them based on their relevance to a given query. In recent years, the integration Large Language Models (L.LMs) into reranking techniques has shown promising results in text retrieval (Ma et al. 2024b) and has gradually become the stan- dard approach (Sharifmogghdddan et al. 2025). When extending to the multimodal setting (Liu) eet al. 2023 Wei et al.) 2025), multimodal LLMs (MLLMs) also become the promising backbone (Lin et al. 2025 Zhang et al. 2025b) as their strong multimodal understanding capabilities.
choice
Current widely used rerankers are typically in the point-wise setting (Lin et al. 2022), which in- dependently scores each candidaae pair and ranks the candidates. The simple architecture of point-wise rerankers makes them easy and efficient applicate in real-world scenarios, and there emerges various open-source state-of-the-ar (SOTA) models (Chen et al. [2024] //hang et al. 2024), particularly LLM-bbsed ones (Sharifymoghaddam al.) 2025 ZZang al.] (222d)). To train such rerankers one straightforward approach follows the pre- LLM practice of contrastive learning (CL)
"Equal contribution, order determined by random dice toss. Thhouuhoutt this work, reranking primarily refers to point-wise reranking setting.

---

Published as a conference paper at ICLR 2026
Instruction
p Oi.... 0x -1
Query
MLLM
LM head
Contrastive Learning
Reranker
Document
pCyess ( q.
Assistant:
p("no" q, d -1
Positive Output
Negative Output
Supervised Fine-Tuning
Comparison of Supervised Fine-Tuning (SFT) and Contrastive Learning (CL) for the MLLM reranker. SFT treats reranking binary classification task ("yes"''no"), In contrast, CL trains the model to directly distinguish the positive document from a list of negatives.
Figure 1:
Nogueira al. 2019 Zhang al. 2024), computing InfoNCE loss (Oord et al. 2018) predicted relevance scores. Another approach is to directly perform supervised fine-tuning (SFT) NNguuirra al., 2020: Zhang al. 2025d), which optimizes the model predict the next token ("truc/yes" for relevant "false/no" for irrelevant) and takes the "true/yes" token probability as the relevance score. The illustration of them is shown Figure TT Before the emergence of LLMs, contrastive learning was the dominant approach for leveraging BERT-style encoders due to its strong perfor- mance (Nogueira et al. 2019: Zhang et al. 2024). However, SFT are now widely applied to LLMs (Nogueira al.) 2020; Zhang al. 2025d) and appears deliver competitive results. This raises natural research question: which objective intrinsically better for LLM reranking, and why?
Meanwhile, research multimodal reranking remains largely restricted to single datasets or rowly defined tasks (Xu et al.) 2025), limiting the generalizability cxisting approaches. Building recent advances in universal multimodal retrieval (Zhang al. 2025b), objective is to de- velop a universal multimodal reranking model that can consistently adapt across diverse modalities.
In this work, aim explore the question by providing analysis and empirical comparison of the two approaches on the universal multimodal retrieval task as testbed. We first design the General Multimodal Reranker (GMR, $3.1). and then analyze the training approaches and decompose their loss functions (3.33 into weight and direction. Based on this, we implement unified frame- work for CL and SFT losses and conduct experiments to compare and analysis them ($4). To make comprehensive evaluations of multimodal reranking, we compile new unified benchmark called MRB (multimodal reranking benchmark, $5).
Through analysis and comparison, we find that SFT consistently outperforms CL for LLM- rerankers, and: (1) The weight component, rather than the direction, accounts for the most perfor- mance gap; (2) A larger weight improves robustness to numerical errors in training, where SFT intrinsically assigns larger weights than CL.; (3) The function of weight a input-specific guidance: down-weight wreeady-wellleeameee input pairs and up-weight hard or under-fit pairs; (4) The native SFT direction almost optimal, whereas CL can further improved by tuning its direction ma- trix. To further validate the potential of SFT, train two reranking models (i.e., GMR-3B and GMR-7B), which set new state-of-the-as results on MRB. We will release code, data and models to facilitate future research in this area. Our contributions
We provide a unified analysis of SFT and CL for LLM-based reranking, showing that SFT intrinsically outperforms CL. By decomposing the loss into weight and direction components, we reveal that SFT's weight term delivers stronger optimization signals.
We introduce the MRB benchmark, comprising 40 datasets across single- cross-, and fused- modal retrieval, offering a comprehensive evaluation for universal multimodal reranking.
We develop GMR models, instruction-awer aware multimodal LLM rerankers trained on 1.5M di- verse pairs. GMR-3B and GMR-7B achieve state-of-the-art results on MRB, highlighting the effectiveness of SFT and providing strong backbones for future research.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
Reranking with Large Language Model. Reranking improves retrieval output quality by jointly modeling the query and retrieved candidates and reorder the candidates (Lin et al. 2022). In recent years, reranking is dominated by methods based on pretrained language models (Nogueira et al.) 2019 2020), with LLM-based approaches becoming particularly prominent in the latest advance-
ments Ma al. 2024b; /Zuang al. 2024 Sharifymoghaddant et al. 2025 Zhang et al. 2025c).
Compared to the widely studied list-wise reranking (Ren et al., 2025; Liu et al. [2025] in this work,
we focus on the more straightforward and widely used point-wise approach (Zhang et al. 2024 Guo
ct al.
2025), which scores each query-candidate pair independently and ranks the candidates.
Training point-wise rerankers has traditionally relied on contrastive learning (CL) (Nogueira al.
2019 Zhang et al., (024), which is also a verified choice for LLM-based models (Ma et al.| However, for such generative language models, a supervised fine-tuning (SFT) approach (Nogueira et al. 2020) seems to be more aligned with the model nature, as it directly optimizes the model to predict the next token ("true/yes" for relevant, "false/no" for irrelevant) based on the input query and candidate, rather than relying on contrastive loss that compares the relevant and irrelevant candi- dates. There is no clear consensus on which approach is better yet. To bridge this significant research gap, conduct analysis with empirical comparison of the two approaches, and demonstrate that SFT outperforms CL in terms of performance.
Multimodal Information Retrieval. Multimodal Retrieval aims to retrieve relevant candidates from and based on modalities beyond text (Wang et al., 2024), which involves various sub-tasks such as image-text retrieval retriveva (Cao et al. (0022) composed image retrieval (Song et al. 2025). and text- image interleaved retrieval (Zhang et al. al. 2002aap. Recent advancements this field have been shifted
to more generalized view, exploring the universal multimodal retrieval (UMR) (Liu et al.) [023]
Wei et al. 2025 Zhang et al. (025b) which compile a wide range of datasets and tasks into a unified benchmark. Retrievers (Lin et al. 2025 Zhang et al.] 2025b) driven by multimodal LLMs have shown significant improvements in understanding and processing multimodal data, enabling more effective retrieval across different modalities. While the reranking stage is crucial for enhancing the precision of retrieval system, it has been less studied in UMR (Lin et al. (2025). In this work, investigate how to build better LLM reranking models, presenting oatte-ofttheard MLLM-based rerankers for UMR.
METHOD
In this work we analyze the contrastive learning (CL) and supervised fine-tuning (SFT) approaches reranking, taking the multimodal retrieval the experimental playground. We first introduce our reranking model training by CL or SFT ($3.2. and then present our tools for analysis ($3.3)
3.1 RERANKER IMPLEMENTATION
Our general multimodal reranker (namely GMR) follows the conventional design of LLM-based point-wise reranking models. We employ a strong MLLM as the backbone, which could process diverse input modalities, encompassing images, text, and multimodal combinations.
Instruction-Aware Reranking. Given query q and document d, we set an instruction ins. to describe detailed task objectives, which has proven highly effective in MLLM-based multimodal retrieval (Lin et al. 2025 Zhang et al.) 2025b). For example, in the Visual Document Retrieval (Ma) ct al. 2024a| Faysse et al. 2025), we use instruction "Find a screenshot that relevant to the user's question." 5t to guide the model to better evaluate the relevance between query and visual document. We list all instructions of our model in Appendix D.3 The inputs are in the form of (ins., q, d) and formatted into the template shown in Figure before being fed into the MLLM backbone.
Relevance Score Computation. In the SFT setting, given the task instruction ins., query q. and document d, our reranker computes the LM-head logits of the tokens "yes" and "no". and uses the normalized probability of the "yes" logit as the similarity score This process could be formally
3

---

Published as a conference paper at ICLR 2026
expressed as:
exp(x*)
s(ins., q. exp(2*) + exp(2")
where 2d and 2" denote the LM-heat logits corresponding to the tokens "yes" and "no", respectively, given the instruction, query, and document context. With such relevance scores, we could rerank all retrieved candidates more precisely. This method is more aligned to the generative nature of MLLMM and thus allows us to leverage its powerful understanding ability while providing an effective scoring mechanism for reranking purposes.
In the CL setting, the relevance score the LM-head logit of the token "yes" only:
s(ins., 9, d) 2²
3.2 RERANKER TRAINING
In the training of reranking, each data example contains one query q, one relevant document (pos- itive) dj. and several irrelevant documents (negatives, the selection described in Appendix C.3
{dī. ..... ). As shown in Figure we explore both CL and SFT based training.
we compute the InfoNCE loss Contrastive Learning: With relevance score & from Equation 2
Oord et al. 2018) for each example:
exp(s(ins.,
C log exp(s(ins..g.dg) + Σ, exp(s(ins.. ))
(3)
Supervised Fine-Tuning, The objective is independently predicting correct label ("yes" or "no"" for each input pair. We reorganize each example into multiple triplets (ins., q, de), with each triplet corresponding to a distinct document d, For each triplet, the model predicts the LM-head logits of "yes" and "no", then compute the crosspyntropy loss using the ground-trute label l:
STP log(p(()=(( (("yes", "no"") Hins. q.d,))))
(4)
where 2( "ioo""""""""|"""""""""""""" Hins., q. d)) denotes the LM-head logits of "yyes" and "no". The relevance label I is "yes" for positive documents and "no" for negatives. This loss encourages the model to assign higher probabilities to the correct label, thereby improving the ranking performance.
3.3 LOSs FUNCTION DECOMPOSITION
We analyze two reranking loss functions by decomposing them into two key components: weight and direction. With this decomposition, we perform probing experiments in $
Basic Notation. We denote the SFT-style data instance with positive (resp. negative) doe as % {ins., q, d (resp. o₃ = {ins., d, }. 1.2...., N). as two components: a mapping function f(-10) (parameterized by 0) that converts o, to the feature representation h; [o)] and a transformation M that maps h, into the LM-head logit of "yes" token 20(h) = h; _M" while the LM-head logit of "no" token z" could be computed by M
The reranker is conceptualized
Unified View. From 3.22 the SFT loss is calculated separately for each positive or negative doc of an example, while the CL loss computed in an integrated manner across all positive and negative docs of the same example. To enable fair comparison,
adopt the total loss L((o, V / f) over an entire example (with one positive and N negatives) as the unit of analysis. So we have the gradient
ac
ac Ohe
OL Oh
+ Σ Ob 00
00
Oh
00
where hy is the feature of positive doc and h, is that of i-th negative.
To understand the influence of positive and negatives on the model, we calculate the partial derivative of the loss function with respect to the hidden state. For CL, we only use "yes" token, and by substituting the specific loss (Equation 3 into the gradient, we obtain the partial derivatives:
Σ3 exp((=
exp(2*(h,) + E, exp(2*(h,)) M"
(6)
Oh
4

---

Published as a conference paper at ICLR 2026
OC
exp(2" (h₂
M
(7)
ah
(xp(=*(h))) E. exp(">(h77)
In SFT, we first merge the Equation of multiple pairs in one example into the total loss
exp(=
exp(x" (hp(z((,,)
CSP log exp (b)) + (pp22)(())))
log exp(22(h,)) + exp(2"(h) (h7))
Then we have partial derivatives
SC
exp(="(hī))
(My - -Mu),
(8)
Thg
exp(2>(h,)) exp(2" (b
0C
exp(=*(h,))
Oh exp(2"(h, (h,)) )) exp(="(h; (M, - -Mn).
(9)
The complete derivation of the above process is provided in the Appendix A.2
Loss Decomposition. As above gradients looks similar, we can break them down into two parts: weight and direction. They reflect the differences between CL and SFT.
Algorithm Unified Reranking Loss
Require: inputs O <- {oo ..... o
Ensure: loss value L
1: M 1m_heamaeeee ci (""""e"""""""" "no")
Weight W is scalar that controls the magnitude of the updates. From Equation we obtain the weights shown below:
2: logits + M f(0
1/ weight branch
3: if weight==shh then
E, exp((2 (2" (xxp(2(hh)))
$ 8 Softmax(logits))0 detacc)))
W
(10)
(xp 1) E, exp(2" (h,))
W+ - sat s(0]
W + W
s[L:]
exp(2 (h,
W
(11)
7: else
> weight== "cl"
exp(> (h)) )) + Σ, exp(=> (h;
$ + Softmax { detach()
(xp(="(h)))
W + -wt + s|0]
W SFT
(12)
exp(>> (b)) exp(=" (h
10: W- < Wa + [1::]
11: end if
(h,
WSFT exp(2> exp( (h,7)
(13)
1/ direction branch
12: My + logits|; 0]: M, +- logits(; 1]
Compared with CL, WSFT only focus on the single document, without the interactions with all negatives of the same query like CL.
13: if directioons ion= "sft" then
14: D+ D M, 00 My [0]
15:
D D My 1:] Ma
16: else
- dict ineeciion=90 "cr"
Direction D is vector that controls the direction of model updates. From Equation 6 and g the direc- tion from the positive dt for CL is De and that of SFT DEFT direction from negatives are
17:
D+ D My
18:
D Dj My
My.
19: end if
My Mn. While
from Equation 7 and 9 Dē.== -My and DEPT = - (My- MM)). Appar- ently, for both CL and SFT, the update directions of positive and negatives are opposite.
20: L <- mean D+ + W,
21: return C
In summary, CL and SFT share similar direction components, and we believe that differing initializa- tions? are insufficient to account for performance differences. In contrast, CL computes the weight using all positive and negative documents within sample, while SFT assigns weights independently per document, making this the likely key factor in performance variation.
Unified Framework Building on the above decomposition, we propose unified reranking loss framework (URL),, with pseudo-code provided in Algorithm This framework allows us to indepen- dently analyze weight and direction, thereby facilitating a deeper understanding of the differences between the two training paradigms through controlled adjustments during computation. We then validate our analysis through probing experiments in the following
My compared My -Mn
5

---

Published as a conference paper at ICLR 2026
4 ANALYSIS
In this section, we continue and validate the analysis of $3. through probing experiments. We choose universal multimodal retrieval as the testbed, compiling new benchmark (MRB $5.1) in- cludes single-modal tasks (text-to-text, image-to-image), cross-modal tasks (e.g., text-to-image), as well as fused-modal tasks (either the query or the document could consist of text + image). We defer the description of experiment settings and evaluation
benchmark to $5.1
General Empirical Comparison. We first train both CL and SFT rerankers with the original implementation and our URL framework to (1) find the winner in practice, and (2) verify that URL faithfully reproduces the original implementation, supporting the subsequent analyses built on URL. As shown in Figure 2 under the identical setting, SFT consistently outperforms CL. Meanwhile, URL yields statistically indistinguishable performance to the original. thus could be trusted in the following analysis.
Perf.
57
SFT
CL
56
Original URL Original URL
Figure 2: Performance com- parison of the original imple- mentations and our URL.
Weight W Dominates Performance. To investigate why SFT outperforms CL, we first dissect the contribution of weight and direction. In Table we train the model with all combina- tions by URL. We observe that the improvements from weight (i.e., Aw) ) is more significant than that of direction (Ap). This suggests that the weight W is the dominant factor in the per- formance gap between SFT and CL, guiding us to foeus the weight in the following section. However, the direction also contributes to the gap, which investigated in $4.2
DSFT
DeL
Ap
WSFT 58.09
57.88
0.21
Wa. 56.99
56.40
v 0.59
A w 1.10
1.48
Table 1: MRB results of all loss components combinations, where the weight W delivers the domi- nant influence on performance.
4.1 FUNCTION OF WEIGHT
To figure out why WeL less effective that WSFT and what is the function of W. we start from the observation of (Chen et al. (021). In small-batch CL training with InfoNCE, gradients would shrink to very small scale, close to random precision errors, and thus cease to provide effective learning guidance. We suppose this is more salient in reranking where the small batch size is commor Then we validate their findings by training CL model with fully half-precision loss computation, which yields degraded performance compared to precision-safe training (refer to Appendix B.2
Back to our framework, W controls the steps of model updates, or say the gradient scale. According to Chen et al. 22 WeL should be small in the training process. And we expect WSET to be larger than CL to provide better optimization signal as SFT presents better performance. To verify this, plot the W of CL and SFT in training in Figure where WCL indeed show relatively small values. SFT provides larger (better) W than CL, thereby achieving stronger empirical performance. Equation [0quationT00oo] 12 also shows that WsFT is larger than WCL, since the denominator of WeL involves a sum of all negatives while the denominator of WSFT only adds up current instance.
Next, investigate the fine-grained function of W. To create cleaner analysis setting, fix the direction in URL as DSFT unchanged, as it performs better. We first set weights of both positive and negatives the fixed constant 1 bascline (Woase) following (Chen al. (02I):
W+ W,W, exp(=(h7))
Σw
W =1
(14)
E, exp(2(h-))
Although the earlier analysis suggests that the larger W is preferable, this value never appears in Figure expect this setting to perform poorly. The experiment in Table also align this.
CConsider a batch of instances, (O....., 0, is forward simultaneously during training with & negatives sample. While dense retrieval can achieve the negative size of j-k++11 1) per instances, reranking models' are limited to k + 1. Furthermore, the increased number of input tokens the reranking stage, compared to dense retrieval, imposes additional constraints memory usage, resulting in reduction in the value of negative size.

---

Published as a conference paper at ICLR 2026
1.0
SFT Pos Weight
No. Method
Avg.
A
Average Weights
0.8
SFT Nog Weight
CL Pos Weight
W Buse
49.47
0.6
CL Neg Weight
+ T mask
56.57
A 7.10
+ WcL.
56.23
6.76
0.4
WSFT
58.19
8.72
0.2
Table 2: Evaluation of different properties of the weight compo- nent, where A denotes perfor- mance gain relative W Base-
0.0
1000
1500
2000
2500
Training Step
Figure 3: Evolution of positives and negatives average weights during training for SFT and CL.
Hence, we suppose that W should be in a reasonable range. Meanwhile, the failure of constant W indicates that instance-speciffc adjustment is necessary: the model should update less already- mastered instances and more on those it has not yet grasped.
We adopt the predicted relevance scores high enough, f.e., s(ho) > 1 1-3 т, (or, conversely, negative score is low enough, s(hy) < T), we set W+ (resp. W; = 0) to halt further learning on that instance. addition, we further set WCL and WSFT the baseline and conduct training under the same conditions. The results are shown in Table we can see that the simple masking rule can provide strong performance, comparable to CL. This indicates both CL and SFT follow the above instance-specifie weight feature. More details of the experiment can be found in the Appendix B.2
guide and apply masking rule: if a positive score is
4.2 SEARCHING BETTER DIRECTION
Results in Table I indicate that the direction component also affects model performance, but it is not the dominant factor. Here we conduct additional experiments and try to find better direction.
Does adding more tokens improve performance? SFT-based training is actually binary classification on the token labels, where DSFT only involves "yes" and "no" tokens. One natural question whether adding more tokens (e.g., "true", "false", "maybe", etc. ) during training could improve the direction component and model performance? To investigate this, we randomly select 10,000 train- ing instances and identify the top 16 tokens with the highest logits from the model's 's output, including "yes" and "no". For compre- hensive list of these tokens and details, please refer the Appendix B.3] We then train the model using this expanded token set. Figure 4 presents the results, which indicate that increasing the number of tokens does not significantly impact model performance. This result suggests that using only "yes" and "no" tokens is sufficient for effective SFT.
Perf
57.07
57.9
58.02
58
57.88
56
2
4
8
16
# Tokens in SFT
Figure 4: Results with dif- ferent token numbers in SFT. The setting with 2 tokens is the standard SFT training.
Is it possible to learn better direction? The direction components, in essence, corresponds to the token embed- dings of the LLM, which are pre-trained and keeping frozen during training. Before LLM, CL-based rerankers often learn score-projeccton matrix from scratch. To see whether this still helps, we implement the random-initialize learn- able weight DRand. in URL. shbls that, for CL mod- els, it does improve performance, yet still trails behind SFT. For SFT models, however, the strategy hurts performance. This is in line with the intuition: SFT is trained to predict the "yes/no" tokens, so replacing the weight with a randomly-
Weight Direction Perf.
A
DSPT 58.09
WSPT
Dnand.
56.75
1.34
DeL 56.40
WeL
Duand.
57.72
1.32
Table 3: Performance comparison of SFT and CI. directions against dom initialization DRand.
initialized projection will loss the semantic signal from the pre-trained token embeddings.

---

Published as a conference paper at ICLR 2026
Model
Size
Single-Modal
Cross-Modal
Fused-Modal
Avg
T-)(())
I+
T-+h))
T-VDDs)
T-ITT
IT
IT
IT
Al.
GME-2B
2.21BB
49.59
30.75
48.46
66.39
52.62
77.02
39.88
36.70
66.89
52.54
Owen3
4.02B
60.49
Jina-mo
2.21B
55.36
27.50
59.46
73.13
55.43
74.95
27.82
37.65
51.54
54.36
MonoQwen
2.21B
48.89
12.59
58.73
71.29
19.62
76.46
14.35
31.75
35.83
44.20
GMR-3B
3.75B
59.22
29.76
58.85
72.38
63.06
81.96
48.81
43.97
79.08
61.40
GMR-7B
8.29B
61.08
32.83
61.18
72.94
66.61
84.55
53.29
47.39
82.19
63.85
Table 4: Performance of different models on MRB. Each column corresponds to a task category, with the number of test sets indicated in parentheses. Evaluation metrics are provided in Appendix [E.1 We adopt GME-2B as the retrieval backbone, while all other models rerank the top-100 retrieved candidates. indicates the best result in reranking models, and indicates the second-best.
EXPERIMENTS
5.1 SETTINGS
Training Dataset To develop a universal multimodal reranking model, we follow the settings of GME and curate training data from three categories: single-modal data (T-TT, I-+1), cross-modal data (I++T. V-VD),, and fused-modal data (IT++T, IT-xI II-+IT). In total, we compile approxi- mately 1.5 million training instances from diverse sources, including M-BEIR (Wei et al. 2025).
ViDoRe (Faysse et al.] 2025), ImageNet-1K K (Deng et al,, 2009), E- E-VQA (Mensink et al.
2023).
and MS MARCO (Nguyen et al.) 2016). To ensure fairness and efficiency in the comparative ex- periments reported in $4 we additionally construct a balanced and category-representativ subset consisting of about 270K samples drawn from the full training dataset. The models, GMR-3B and GMR-7B, are trained on the complete dataset to achieve optimal performance, whereas the models evaluated in $
are are trained on the constructed subset. Details could be found in Appendix C.I
MRB Benchmark To facilitate more rigorous evaluation of model performance, we construct the MRB benchmark, which comprises 40 test datasets sourced from BEIR (Kamalloo et al. 2024). UMRB (Zhang al.) 2025b), ViDoRe (Faysse al., 2025 Macé al.) 2025), and MIEB (Xiao et al. 2025). Collectively, these datasets span diverse modalities, domains, and task types, ensuring that the benchmark provides comprehensive and representative assessment of model generaliza- tion. To more clearly highlight performance differences among models, we exclude test datasets on which GME-2B exhibits exceptionally high performance. detailed description of the MRB benchmark composition is provided in Appendix C.2
Training Configuration We adopt the Owen2.5-VLLInnstuu (Team] 2025 model series as the backbone of our multimodal large language model (MLLM), and conduct training at both 3-billion (3B) and 7-billion (7B) parameter scales. For efficient adaptation, we employ Low-Rank Adaptation (LoRA) with a rank of 16 and a learning rate of le-4. As evidenced by the comparative results in $4) within the domain of multimodal LLM reranking, SFT consistently outperforms CL. Consequently, we adopt SFT as the training strategy for our GMR series models.
During training, set the maximum input length to 3,200 tokens. Fach training sample is paired with 16 negative instances for the GMR-3B and GMR-7B models, and with 4 negative instances for the models mentioned in $4 Regarding the selection of negatives, we employ two strategies: Ran- dom Selection and Hard Mining, maintaining balanced ratio of 1:1 between them. Further details on the negative sampling strategy are provided in Appendix C.3] To optimize GPU memory usage, we train the model using b9loat16 precision. All experiments were conducted on eight NVIDIA 100 GPUs, each equipped with 80 GB of memory.
Baselines We adopt GME-2B as the retrieval backbone to generate candidate results for each task. Specifically, the top-100 retrieved candidates are retained, and all reranking models are subsequently evaluated on this candidate pool. For the experiment described in $4 we reorder the top-25 candi- dates to balance faimess with efficiency. Our method is compared against three representative types
8

---

Published as a conference paper at ICLR 2026
of reranking systems: (1) A representative textual model: Owen3-Reranker (Zhagg et al. 2025d (Qwen3), exemplifying recent advancements in text-based reranking. (2) A versatile multimodal reranking model: Jina-rerank-min (ireraanmmm((innnn))
This model natively supports single-mddal tasks and cross-modal tasks. Leveraging the flexibility of its MLLM architecture, we extend its application to fused-modal tasks by adopting its input template. The specifies of these adaptations are detailed in Appendix D.4 (3) A cutting-edge visual document reranking model: MonoQwen2-VL-vO. (Chaf-) fin & Lac 2024) (MonoQwen). Similar our approach with Jina-rerank-mo, we evaluate this model across all task types. The input templates used is provided in Appendix D.5
This comprehensive evaluation benchmarks our method against leading models across diverse modalities and task types, enabling a thorough assessment of its effectiveness.
5.2 MAIN RESULTS
We first examine the effect of the number of negatives. In SFT, where query-candidate similarity is formulated as binary clas- sification task, the number of negatives directly affects model performance. To identify an appropriate setting under our com- putational budget, we experiment with varying numbers of neg- atives (Figure 5). Performance consistently improves with more negatives, peaking at 16. Moreover, SFT outperforms CL across all settings (Appendix E..2. Based on these results, we set the number of negatives to 16 in training. Given the impact of ran- dom initialization on performance (s4.2), we also conduct an ab- lation on freezing the LM head (Appendix E.3 and find that has effect on SFT performance.
59
58.68
Perf.
57.97
58.08
58
57.78
2
4
16
#Neg in SFT
Figure 5: Results with different
numbers of negative in SFT.
We next examine the evaluation results. Table presents a comprehensive overview of the baseline systems' performance. The reported scores are averaged across the respective sub-tasks and are organized according to the retrieval modality: Single-Modes Cross-Modal, and Fused-Modal. For completeness, the overall micro-average
across all sub-tasks is provided in the final column.
Achieve state-of-the-art performance in universal multimodal reranking. Analyzing the aver- age metrics, smaller model, GMR-3B, exhibits superior results compared to the fused-modal reranking model Jina-Rerank-mo). The larger GMR-7B further elevates this performance, under- scoring the efficacy addressing universal multimodal reranking challenges.
Rival and surpass leading textual reranker. We conduct a comparative analysis with the state-of- the-art textual reranking model, Qwen3-Rerankerr, which is specifically optimized for the T-+T task within the Single-Modal category and comprises approximately 4 billion parameters. Our smaller model exhibited similar performance metrics when evaluated against models of similar parameter scale. Notably. our larger model surpass the performance of Swen3--Rernkee providing strong empirical evidence for the efficacy of our proposed methodology.
Maintain strong robustness reranking depth increases. Recent studies (Jacob et al.) (2255) suggest that reranker per- formance may degrade as the number of documents increases. To examine this effect, we evaluate GMR and key baselines under the top-25 and top- 100 settings, with results shown in the Figure 6 In T-+T tasks, all models improve when mov- ing from t5p-25 t0p-100, and the gains correlate with model models benefit more. On the full MRB bench- mark, GMR models also exhibit consistent improvements (1.47% / 2.39%), whereas Jina-Rerank-min shows slight drop (- 0.08%). These results indicate that our models maintain stronger robustness across different top-N configurations.
Qwen3 Jina-mo GMR
8
-61
60
62
59
T2T
58
ALL
55
55
Top25
Top100
Figure Performance compar- ison under top-25 and top-100 reranking settings.
Adapt seamlessly to visual-document reranking. In Table 4 we compare with the visual document reranking model, MonoQwen2-VL--ol which is specifi- cally tailored for the VD task. Our proposed models demonstrate performance metries that are surpass those of this task-specific baseline, which suggests promising direction for developing

---

Published as a conference paper at ICLR 2026
Task
Model
Art
Med.
Sci.
Hum.
Avg.
Retrieval
GME-2B
73.86
46.99
62.69
56.16
59.93
MonoQwen
57.49
42.81
47.55
44.36
48.05
Rerank
jina-mo
77.91
55.21
70.05
66.35
67.38
Top-100
GMR-3B
82.38
59.47
72.14
68.79
70.69
GMR-7B
82.91
63.93
75.90
74.14
74.22
Table 5: MRMR Knowledge Performance across Models (reported using NDCG@ 10). the best result in reranking models, and indicates the second-best.
indicates
more efficient and adaptable information re-reanking systems that can seamlessly handle diverse modalities within single architecture.
5.3 ADDITIONAL EXPERIMENTS IN MULTI-IMAGE SETTING
To evaluate model performance beyond the setting where queries and documents each contain a single image, further assess models the MRMR et al. 2026). The results pre- both
sented in the 55 In the Knowledge subset of MRMR, queries are mu muultimodll-conaimmee text and images-wall while documents cover the full modality space, including pure-text documents as well as mixed text-image documents with either single or multiple images. This design enables comprehensive examination of model robustness and multimodal reasoning capabilities.
As shown in the Table the GMR series achieves the strongest performance among all reranking models and delivers substan- tial improvements over the retrieval model (GME-2B). These results demonstrate that our approach remains effective multi- image scenarios and is capable of achieving superior perfor- mance under more complex multimodal conditions.
MonoQwen Jina- m0 GMR
75
70
In addition, we evaluate the average performance of the models under the top-25 and top-100 settings, as shown in the Figure 7 Both ima-rerank-aa mo and MonoQwen2-VLL-.e VL-v0.1 exhibit perfor- mance drops when moving from top-25 to 10p-100, whereas our models (GMR-3B/ GMR-7B) achieve consistent gains (+ (+1.66% / /+1.90%). These results demonstrate that our approach remains effective in fully multimodal scenarios, including cases with multiple images, and continues to deliver superior performance under more challenging top-N configurations.
50
Top25
Top100
Figure 7: Average Performance comparison under top-25 and top- 100 reranking settings in the Knowledge subset of MRMR.
CONCLUSION
In this work, we show that supervised fine-tuning (SFT) consistently outperforms contrastive learn- ing (CL) for L.MM-based reranking. By decomposing the loss into weight and direction components, find that the weight term primarily drives performance gains by strengthening optimization sig- nals and providing input-specific guidance. While SFT's directional component is nearly optimal, CL requires learning score-projection matrix to achieve comparable results. Building on these insights. we develop the GMR-3B and GMR-7B models, which set new state-of-the-ard results on the MRB benchmark covering 40 datasets. By releasing MRB, models, and code, provide solid foundation for future research in large-scale multimodal retrieval and universal L.M rerank- ing, underscoring both methodological and practical significance.
ACKNOWLEDGMENTY
We thank the anonymous reviewers and chairs for their valuable feedback and suggestions. This work receives support from the Natural Science Foundation of China (under Grants 62336008, 624B2048), Research Grant Council of Hong Kong (PolyU/15209924)) and the Shenzhen Basic Research Program (Grant No. JCYJ20240881110511 11016).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Min Cao, Shiping Li, Juntao Li, Liqiang Nie, and Min Zhang. Image-text retrieval: A survey on re- cent research and development. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence. IJCAI-22, pp. 5410-5417, 7 2022. htt.h:tt///// //doi. org/10.
24963/1jcai 2022/759
Survey Track.
Antoine Chaflin and Aurélien Lac. Monogwen: Visual document reranking, 2024. URL https:
Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3-embedding:
Multi-linguality. multi-functionality, qulli-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguisties: ACL 2024. pp. 2318- 2335, Bangkok, Thailand, August 2024. E ahniings-acl.137.
URL httpps//aallaatbbooooo..oo/////.
Junya Chen, Zhe Gan, Xuan Li, Qing Guo, Liqun Chen, Shuyang Gao, Tagyoung Chung, Yi Xu, Belinda Zeng, Wenlian Lu, Fan Li, Lawrence Carin, and Chenyang Tao. Simpler, faster, stronger: Breaking the log-k curse on contrastive learners with flatnce, 2021. URL /tttss//aaxxx
org bogg/bbs/ee 21 01152
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, CELINE HUDELOT, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
Fang Guo, Wenyu Li, Honglei Zhuang, Yun Luo, Yafu Li, Le Yan, Qi Zhu, and Yue Zhang. Mcranker: Generating diverse criteria on-the-fly to improve pointwise Ilm rankers. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, pp. 944-953, otg. 1eteg/1..111537111110
Hannover, Germany, 2025. URL /ttppp/////.....c///000111////11
3703583
Mathew Jacob, Erik Lindgren, Matei Zaharia, Michael Carbin, Omar Khattab, and Andrew Drozdov. Drowning in documents; Consequences of scaling reranker inference, 2025. URL https://
arxiv. orw. abs/ 0tpps/211..1176
Ehsan Kamalloo, Nandan Thakur, Carlos Lassance, Xueguang Ma, theng-Hong Yang, and Jimmy Lin. Resources for brewing beir: Reproducible reference models and statistical analyses. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1431-1440, Washington DC, USA, 2024. URL httpps://do org/10. 100..111533666 00175/36667722 3657862
Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. Pretrained transformers for text ranking: Bert and beyond. Springer Nature, 2022.
Sheng-Chiee Cheng-Chhihhae Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. MM-EMBED: Universal multimodal retrieval with multimodal LLMS. In The Thirteenth Inter- national Conference on Learning Representations, 2025. URLAttpa://peenrriieee net/
forum?1d=145Ngb221.
Qi Liu, Bo Wang, Nan Wang, and Jiaxin Mao. Leveraging passage embeddings for efficient listwise reranking with large language models. In Proceedings of the ACM on Web Conference 2025, pp. 4274 4283, Sydney NSW, Australia, 2025. 3714554
Zhenghao Liu, Chenyan Xiong, Yuanhuiyi Lv, Zhiyuan Liu, and Ge Yu. Universal vision-language dense retrieval: Learning unified representation space for multi-modal retrieval. In The Eleventh International Conference on Learning Representations, 2023. URL htt.https //pppereepi...../

---

Published as a conference paper at ICLR 2026
Xueguang Ma, Sheng- Ching-ChihhLaa Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multimodal retrieval via document screenshot embedding. Proceedings of the 2024 Conference on Empir- ical Methods in Natural Language Processing, pp. 6492 6505, Miami, Florida, USA, November 2024a. URL htl.:tts: /cc anthology 2024/22244 emn p-ma 3733
Xueguang Ma, Liang Wang. Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi- stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2421-2425, 2425, Washington DC, USA, 2024b. URL
3657951
Quentin Macé, António Loison, and Manuel Faysse. Vidore benchmark v2: Raising the bar for visual retrieval, 2025. URL /tL/tttt////axxi.ooggg http:/v...rr/a///255..... 17166
Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, and Vittorio Ferrari. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 3113-3124, October 2023.
Tri Nguyen, Mir Rosenberg, Xia Song. Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: human generated machine reading comprehension dataset. CoRR, abs/1611.0922 2016. URL httt:://axxix wwtp:oggg///oooooooooo...osoo 1et.s/12111.111
Rodrigo Nogueira, Wei Yang. Kyunghyun Cho, and Jimmy Lin. Multi-stage document ranking with bert. arXiv preprint urXiv: 111119900...4,,.. 2019.
14424
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with pre- trained seneennceto-sequeeee model. In Findings of the Association for Computational Linguis- fics: EMNLP 2020, pp. 708-718, Online, November 2020. URL httl:tttss//aclaathooll org/2020. www.iigsnnniiiie emmlp..mmpp.......mm 637
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive pre- dictive coding. arXiv preprint arXiv: 1807.08807,0374, 2018. URL
0877.037481
Ruiyang Ren, Yuhao Wang, Kun Zhou, Wayne Xin Zhao, Wenjie Wang, Jing Liu, Ji-Rong Wen, and Tat-Seng Chua. Self-calibrated listwise reranking with large language models. In Proceedings of the ACM on Web Conference 2025, pp. 3692-3701, Sydney NSW, Australia, 2025. URL
https:: htt.:p:://do..org///// www.ool...... 1145 33614399964140 37..641117771111 37146771555 3614
Sahel Sharifymoghaddam, Ronak Pradeep. Andre Slavescu, Ryan Nguyen, Andrew Xu, Zijian Chen, Yilin Zhang, Yidi Chen, Jasper Xian, and Jimmy Lin. Rankllm: A python package for reranking with lims. In Proceedings of the 48th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, pp. 3681-3690, Padua, Italy, 2025. URL
/ttp://....0r//1111113333333222. 3730331
Xuemeng Song. Haoqiang Lin, Haokun Wen, Bohan Hou, Mingzhu Xu, and Ligiang Nie. A com- prehensive survey on composed image retrieval. arXiv preprint MXXi::2502.18495 2025. URL
https:://axivv org/ abs /2002 18495
Qwen Team. Qwen2.5-vl, January 2025.
URL https://gwenJm.iithuue btttchub.../bogg//
qwen2. 5-v17
Tianshi Wang. Fengling Li, Lei Zhu, Jingjing Li, Zheng Zhang, and Heng Tao Shen. Cross-modal retrieval: A systematic review of methods and future directions. Proceedings of the IEEE, 112 (11):1716-1754, 2024.
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Unir: Training and benchmarking universal multimodal information retrievers. In Proceedings of the 18th European Conference Computer Vision, pp. 387-400, Milan, Italy, 2025.
12

---

Published as a conference paper at ICLR 2026
Chenghao Xiao, Isaac Chung, Imene Kerboua, Jamie Stirling, Xin Zhang, Márton Kardos, Roman Solomatin, Noura Al Moubayed, Kenneth Enevoldsen, and Niklas Muennighoff. Mieb: Massive image embedding benchmark. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 22187-22198, October 2025.
Mingiun Xu, Jinhan Dong, Jue Hou, Zehui Wang, Sihang Li, Zhifeng Gao, Renxin Zhong, and Hengxing Cai. Mm-r5: Multimodal reasoning-enhane reranker via reinforcement learning for document retrieval. arXiv preprint arXiv:2506. 12364, 2025. URL stEEp::/aaxxv.org
abs/25055 12364
Siyue Zhang, Yuan Gao, Xiao Zhou, Yilun Zhao, Tingyu Song, Arman Cohan, Anh Tuan Luu, and Chen Zhao. MRMR: A realistic and expert-level multidissiplinary benchmark for reasoning- intensive multimodal retrieval. In The Fourteenth International Conference on Learning Repre- sentations, 2026. URL pictps://operree
net. to www.mmmmm=dxxxx........m rHG
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang. Meishan Zhang, Wenjie Li. and Min Zhang, mGTE: Gen- eralized long-context text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing: Industry Track, pp. 1412- Miami, Florida, US, November 2024. URL. https: 7/ #/aclanthology org//024/ wwwle- ing-iddustty... 103/|
Xin Zhang, Ziqi Dai, Yongqi Li, Yanzhao Zhang, Dingkum Long, Pengjun Xie, Meishan Zhang, Jun Yu, Wenjie Li, and Min Zhang. Towards text-image interleaved retrieval. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 4254 4269, Vienma, Austria, July 2025a Association for Computational Linguistics. URL www.aattthoooooooro/22222222
https://cclantioooggyyooo org. wtt.:///2..2.2 acl -ong. 2147
Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai. Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Bridging modalities: Improving universal multimodal re- trieval by multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp. 9274-9285, June 2025b.
Xin Zhang. Yanzhao Zhang, Wen Xie, Dingkun Long, Mingxin Li, Pengjun Xie, Mcishan Zhang, Wenjie Li, and Min Zhang. Phased training for LLM-ppoered text retrieval models beyond data scaling. In Second Conference on Language Modeling, 2025c. Utthtttss:////// httttss://opereevieee
Yanzhao Zhang, Mingxin Li, Dingkun Long. Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xic, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05116, 2025d. URL https:
05176
Shengyao Zhuang. Honglei Zhuang, Bevan Koopman, and Guido Zuccon. setwise approach for effective and highly efficient zero-shot ranking with large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 38 47, Washington DC, USA, 2024. URL https:
362 12, 2, 3657813
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A METHOD DETAILS
A.1 GMR INPUT TEMPLATE
Following chat-based template, The prompt formulates binary classification task by providing the model with a specific Instruction, Query, and Document for evaluation as shon in Figure
<jim_ starti>system:
Judge whether the Document meets the requirements based on Query and the Instruct provided. Note that the answer can only be "yes" or "no". < end|>
faim__sart>aaa user:
<Instructtoo>>: [Instruction]
<Query>: [Query]
<Document>; (Document) <(im_end) _nd
<im start/> assistant:
Figure 8: The structured input template for GMR series models.
A.2
LOss FUNCTION DECOMPOSITION
In this section, we elaborate on the derivation process of the equation $33
Equation o
OC
OC
02 02
an
(2)(h
ahT
0(-log log exp(s(ins.g.d5 1) exp(s(ins.g.dd)) )) Oz (hit)
exp(s(ine.e.555)
02
Oh
exp(="(hh] ))
Olog exp()* )) E, exp(>*(h 1)
O2 (h))
02 (h+))
Oh
exp(=" (h
exp(>*(h.)) )) + E; cxp(2"(h,7)
exp(2>(b,) à 1)+ E, exp(=><(h, 11
O2 (bx)
exp(2 (xp)2"(h,))
Oz5 (hg
Ohg
exp(="(h,t) )) + Σ; exp(:"(h, exp(2" (h )
Σ, exp(x (2" (h,
(exp(2 (b + E. exp (2, )))
(ex)(="(ht)) 02
O2
Oh
cxp(2>(h )) + Σ, exp(2"(h,))
Σ, (hp(2"(h,))
exp(2*(hg)
(exp(z* (h, + E, exp( (h; ))}²
exx(==((xx)) 0z (h²))
ah
Σ, exp
(hx))
exp(2*(hxt) + E, exp(2=(h)))))) (h,))
Oh
E, exp(2" (h,)) )) exp(>*(h;)) )) + E, exp(2"(hī)) My
(15)
14

---

Published as a conference paper at ICLR 2026
Equation 7
OCCL
OL
(2,
Th
02 (h,)
Th
0(-log
exp(s(ina.g.d))
exp(s(ine...dd 15 exp(e((mevedd 0. (h,
(z
oh
O(log (xp 1)+ exp(="(h, 02 (h,)
exp(= (b)))
Oz (h,
Th
(xp(="(h))) >; exp(=*(h,)
exp(2*(hxt)
exp(2 (2²(h²))
(- (exp(="(h,t) + E, exp(<"(h;
(h, )))}
exxp(=((h7)) 0z
exp( 2 (h
02
exp(=>(h))) )) + E, exp(2*(h,))
Oh,
exp(2>(h))) + E, (xp(2*(h,)) My
(16)
Equation g
SCSFT
OCSFT Oz (bb))
OCSFT (z
Obg
(zith ab
(zi) Oh
o log(p( yes" (=((((())))) ("yes","no" Himss, q, de ()))) (b
Oz (2,)
Oh
a loolo(""yeeeoooooooo (=(((= ("""""""" "no" " d:H)))) 0z" (b
(b
Ohj
Olog "0 (inn.
=0e ((m 024 (hz
02 (bp)
Ohg
alog |*( (ft ((maaaaa))
0z (h²)
0z
Ob.
alog exp(a (exxx(eeet)))) 0z²(h²))
exp(="(b.)) ))
(h
Oh
O(log exp t)+exp(==
exp(2" (h+)
0z
(hz)
02
Ohj
exp(2"
(2,)
(xp((>((b)) + exp(2*(h)))
Oh
exp(="(ho))
0za(h
exp(=>(h])) + exp(( (b
exp(="(h,)) - exp(2>(h))) + (h)
(2
Oz dz
Ohg
abg
exp(2"(hx))
(My - -Ma)
(17)
exp(2>(hx)) + (xx(2*(h)))
15

---

Published as a conference paper at ICLR 2026
Equation 9
SLSFT
OL (z
OC (z"(h;)
Th
0z (h))
Th
(z), Oh
a log( (p( "no"|=(( (=("=""""" (ins,, d: d:))))) (h,)
(2,,
oh
I=-oo(p("no" {1"""
(""es"""""""""""""" "no""""""""""""""""""""""""""" qi d:H)))) 0z
(2"
dh,
Olog 6 (eal Rinereddy 02
(2
Oh,
e ( Ölog eatye fett
02 02"(h, )
Oz exp(=" (h. O(log exp (h, +)+=
dh,
02 (h,)
dew (h,
Oh
Ollog oxp(>>(h, ((a)))))))))e) +exp(=
exp(="(h]
0=" (h,
02 (h,
Oh
0z (h
0z" 0zm(h,)
exp(>> )) + exp( (h
Oh
Uh.
exp()"(h,))
(M -Ma)
(18)
exp(>>(h,)) exp(="
B ANALYSIS EXPERIMENT
B. THE INFLUENCE OF PRECISION ON CL
We validate the findings of FlatNCE by performing full
half-precision training during loss function computation on the contrastive learning (CL) model. Specifically, we configure the model to use BF16 for accuracy, and in the loss computation process (refer to Algorithm []. we con- trol all other variables while varying the precision of the weight computations between FP16 and FP32 assess their impact on model performance. The results show that FP32 precision yields better performance than FP16 precision, confirming that computational preci- sion significantly affects the effectiveness of contrastive learning.
Method Precision
Avg
A
FP16
56.09
CL
FP32
56.40
0.31
Table 6: Impact of precision on Con- trastive Learning's performance.
B.2 FUNCTION OF WEIGHT
To investigate the role the weight, we first define s(h;) =
Method
Avg
+pxp Since s(h;) is bounded within [0. 1]. prior experience with embedding models suggests that ap- propriate scaling factor is necessary to accelerate model conver- gence. Therefore, introduce temperature parameter
(x.(="(h))
1e-2
55.07
w/ T mask
le-3
56.57
1e-4
55.89
exp(=(h, 3/00 In addition, for experiments involving the masking rule, we vary
x 10 into Equation 14 yielding W, = Σ, exp(=(h-)) 1/a)-
Table 7: The performance of the model under different values of r.
T E 10-2, 10 10-4 to identify the configuration that achieves optimal performance. For the ex- periment with WeL. follow Equation I0 and II consistent with the requirements of contrastive learning, where the positive and negative weights must satisfy the constraint W+ Σ W. Since directly setting W Wez
W Base WeL would violate this condition, we instead use W W-wez
16

---

Published as a conference paper at ICLR 2026
WeL for comparison with W Buse- For the experiment with WeFr, aim to demonstrate that WSFT can effectively enhance the performance of W WBase Following Equation 12
and [3
we set
W+WSer = W Base WFF and evaluate its impact model performance.
B.3 THE INFLUENCE OF TOKEN SELECTION
To examine whether introducing additional tokens during training can enhance the directionality component and improve model performance, we randomly sample 10,000 instances together with their corresponding positives and negatives. Based on the model outputs, we identify the top 16 tokens with the highest average logits, which include "yes" and "no." " The remaining tokens in this set are: ["No," "Yes," "NO," "YES," "The," "None," "In," "Answer," "This," "To," "Not," "not," "There," "-no")
C EXPERIMENT SETTING
C.I TRAINING DATASETS
Our training dataset is curated from diverse sources, including M-BEIR, ViDoRe, ImageNet-IK, K, E- wide
32.1%
VQA, and Ms Marco. These datasets array of domains, ensuring that the model is ex- posed varied and representative examples across different tasks. To ensure balanced representation across task domains, we sample 100k instances from ImageNet- IK and integrated them into our training corpus.
50.2%
4.1%
In total, our training dataset consists of approxi- mately million instances, which are distributed across various domains to ensure robust learning. The detailed distribution of the data across these do- mains is carefully visualized in Figure 9
6.5%
7.1%
MBEIR ViDoRe ImageNet:1K EVQA MS Marco
Figure 9: The proportion of the training data.
To ensure fair comparison between supervised fine-tuning and contrastive learning, we construct balanced, category-represenin representative subset of approximately 270K samples from our training dataset, and the details could be found in Table g
Class
Task
Datasets
Number
T T-T (2)
WebQAi Ms Marco
30000
Single-Modal(4)
I-I
Nightsi ImageNet-
IK IK
30000
T-11 (2)
Fashion200ki VisualNewst
29958
Cross-Modal(6)
T-VDD
ViDoRe
30000
13)
Fashion200ki MScocot VisualNewst
30882
T-IIT (2)
EDIS WebQAi
30000
(oxed-Modal(11)
IT IT-+T (3)
LLavat OVENi Remuqt
30382
IT-+1 (2)
CIRRI FashionlQt
29528
IT-ITT (3)
E-VQA OVEN
30000
Table 8: The details of sub trainset. f means that they belong the M-BEIR dataset.
17

---

Published as a conference paper at ICLR 2026
C.2 MRB BENCHMARK
Since overly simple tasks fail to effectively differentiate the performance of various rerank models, we exclude the dataset on which the GME-2B model achieves exceptionally high performance. Detailed descriptions of MRB Benchmark are provided in Tables and 10
Class
Task
Datasets
ArguAnaf Climate-FEVERi CQADupStacki DBPediaf
T-T (14)
FIQA2018f HotporQAt MSMARCO NFCorpust NQł
Single-Modal(15)
Quorat SCIDOCS SciFactt Touche2020f TRECCOVIDI
(1)
Nights"
T->1
VisualNews" Fashion200k Memotion" HatefulMemes*
TAT-DQA ArxivQAi DocVQA!
Cross-Modal(14)
T->VD (5)
MIT Tissue Interaction' World Economic Reports
VisualNews" Fashion200K
1-T (5)
Memotion" GL.vv2" HatefulMemes"
T-*IT (2)
WebQA" EDIS
IT-
OVEN" INFOSEEK OKVQA* VizWiz"
(ased-Modal(11)
IT-1 (2)
FashionIQ" CIRR
IT- (3)
OVEN E-VQA" INFOSEEK*
Table An overview of datasets in MRB. t means belong to BEIR. * means belong to UMRB. i means it belong to ViDoRe. means it belong to MIEB.
C.3 NEGATIVE SELECTION
The quality and diversity of negatives greatly affect the final performance of the reranker. Overly simple negatives can make the model lack the ability to distinguish hard negatives from positives, while overly difficult documents very likely be false negatives that give the model incorrect update signal. Therefore, we adopt two strategies to select negatives: (1) Random Selection. Ran- domly select irrelevant document negatives to enhance the generalization ability of the model. (2) Hard Mining. For each query in every dataset, we use GME-2B to search for the correspond- ing documents to obtain the top 100, and randomly select i irrelevant samples from them as hard negatives to improve the reranking performance. We employ this set of hard negatives for all the models trained in this paper. While training, we always maintain the ratio of random negatives to hard negatives at 1:1 to balance the diversity and quality of the data.
D MODEL SETTINGS
D.1 GME-2B
We employ the GME-2B model as the foundational retrieval model, generating the initial retrieval results that serve the input to diverse reranking approaches. Recognizing that the GME ries models leverage instruction fine-tuning, we incorporate task-specific instructions into the input query to enhance the retrieval model's performance.
Aligning with the UMRB benchmark, we curate the specific instructions for each task, as compre- hensively detailed in Table 14
D.2 QWEN3-RERANKER
Paralleling our approach, -eennk-e leverages Large Language Models for point- wise rerank- ing within singular contextual framework. To facilitate instruction-followee following capabilities, the model incorporates task-specific instructions directly into the input context. By utilizing the LLM's inherent chat template, the similarity assessment is reframed binary elassification paradigm.
18

---

Published as a conference paper at ICLR 2026
Name
Type
Categ.
Eval Samples
Candidates Nums
Eval Query chars
Eval Candidate
chars
ArguAnu
Single-Mon
T->T
192.98
Climate-FEVER
Single-Modul
T--T
1,535
5,416,593
20.13
84.76
CQADupStack
Single-Modal
T-=T 13,145
457,199
8.59
129.09
DBPedia
Single-Modal
T->T
400
4,635,922
5.39
49.68
FIQA2018
Single-Modal
T-+T
57,638
10.77
132.32
HotporQA
Single-Modal
T--T 7.405
5,233,329
17.61
46.30
MSMARCO
Single-Modal
T--T 6,980
8,841,823
5.96
$5.98
NFCorpus
Single-Moda
T-TT 323
3,633
3.30
232.26
NQ
Single-Modal
T-+T 3,452
2,681 468
9.16
78.88
Quora
Single-Modal
T->T 10,000
522,931
SCIDOCS
Single-Modul
T->T 1,000
25,657
9.38
176.19
SeiFact
Single-Modal
T-+T 300
5,183
12.37
213.63
Touche2020
Single-Modal
T->T 49
382,545
6.55
292.37
TRECCOVID
Single-Modal
T-+T
171,332
10.60
160.77
Nights
Single-Modal
I-I 2,120
40,038
VisualNews
Cross-Modal
T-+I 19,995
542,246
18.78
Fashion200k
Cross-Mocks
T-11
1,719
201,824
4.89
Cross-Modal
T-+1
Memotion
Cross-Modal
T->1 697
14.77
TAT-DQA
Cross-Modal
T-VD
1,646
277
12.44
ArxivQA
Cross-Modal
T-VD
500
500
17.12
DocVQA
Cross-Modal
T->VD
451
WER
Cross-Modal
T-VD
58
13.05
MITTI
Cross-Modal
T-VD
160
1016
13.91
VisualNews
Cross-Modal
I-T 20,000
537,568
18.53
Fashion200k
Cross Modal
1-T 4,889
61,707
4,95
GLDv2
1-+T
Memotion
Cross-Modal
T-11 697
6988
14.67
HatefulMemes
Cross-Modal
1-T
1000
10000
11.53
WebQA
Fusod-Modal
T--IT 2,511
403,196
16.43
12.83
EDIS
Fused-Modal
T-+IT
3,241
L047,067
15.53
OVEN
Fused-Modal
- 50,004
676,667
6.52
82.13
INFOSEEK
Fused-Modal
IT-T 11,323
611,651
8.76
91.49
OKVQA
Fused Modal
IT-T 5,046
114,516
8.09
102.55
VizWiz
Tused-Modal
IT-T
FashionIQ
Fused-Modal
IT-+1 6,003
74,381
11.70
CIRR
Fused-Modal
IT->1 4,170
21,551
11,01
OVEN
Fuscd-Modal
IT-IT 14,741
335,135
5.91
94.76
EVQA
Fused-Modal
IT-IT 3,743
68,313
211.12
INFOSEEK
Fused-Modal
IT-IT
17,593
481,782
96.00
Table Tasks in MRB. Following UMRB, We count the number of datasets under each task type, the number of evaluation instances, the size of the candidate set, and the average length of the text.
Specifically, for T T tasks, we task-specific instructions the same as GME, as comprehen- sively illustrated in Table 14
D.3 GMR
In our GMR series models, we incorporate the retrieval instructions into the input context, yielding two advantages. Primarily, this approach eliminates the need for task-specific instruction redesign at the reranking stage, enabling seamless instruction transfer from the retrieval phase.
Moreover, by strategically integrating instructions into the contextual input, we effectively guide the model's comprehension, facilitating enhanced task understanding and robust instruction-followee capabilitics. The comprehensive instruction sets for both training and testing phases are meticu- lously detailed in Tables and 14 respectively.
D.4 -INANKOO
mo0 demonstrates inherent capabilities for processing single-moda and cross-modal tasks. By leveraging the architectural flexibility of Multimodal Large Language Model framework, extend its operational scope to encompass fused-modal tasks through input template adaptation.
19

---

Published as a conference paper at ICLR 2026
Task
Dataset
Query Instruction
WebQA
Retrieve pussages from Wikipedia that provide answers to the following question.
T-T
question, retrieve relevant passages that answer question.
Nights
Find day-4o-diy image that looks similar to the provided image.
ImageNet-IK
Retrieve images of the same type as the one in the question.
Fashion200k
Based the following fashion description, retrieve the best matching image.
T
VisualNews
Identify the news-related image in line with the described event.
V-VD
ViDoRe
Find screenshot that relevant
user's 's question.
VisualNews
Find caption for the news in the given photo.
L
Fashion200k
Find product description for the fashion item in the image.
MSCOCO
Find an image caption describing the following everyday image.
WehQA
Find a Wikipedia image that answers this question.
T--IT
EDIS
Find news image that matches the provided caption.
OVEN
Retrieve a Wikipedia paragraph that provides
the given query about the imuge.
IT-T
LLava
Provide specific decription of the image along with the following question.
Remuq
Retrieve a fact-bused paragraph that provides an answer to the given query about the image.
FashionIQ
Find fashion image aligns with reference image and style
IT-I
CIRR
Retrieve a toy-to-day image that aligns with the modification instructions of the provided image.
OVEN
Retrieve a Wikipedia mange-description pair that provides evidence the question of image.
I--IT
E-VQA
Determine the Wikipedia image-snippe snippet puir that mutches my question about this image.
Table 11: The instructions for training dataset. We the instructions for the GMR series models on each task during training shown in the Table.
For text and image-modal inputs, Jina-rerank-mu organizes Query/Document configurations, as comprehensively illustrated in Table [2] Building upon this foundational template, we design input organization strategy for fused-modal scenarios, represented the Fused configuration.
Ultimately, the model's 's input is standardized to the canonical format:" (Document) format:" \n (nuuery))
Query
Document
.+ :a-uurra=:
Documeett +wil (doc)
Text
Image
+.Query**:
Document
cvtatin start>i padsicuisio ends
start>-cimage padsicuia end>
Fused
**Quezya::
<visiionstaree htt.lioontttrttt-mmm....o.. htw.aaaawc.ccc......ossssssee.o
Doccmentt..:
Table 12: The input template of Jina-rerank-mO. We refer to it's format settings for Text and Image set the imput format of mnsel-modal data, then format the input as " "[Document) (n ("uuer))"
D.5 MONOQWEN2-VL-VOL.
Analogous to our method approach with Jina-rerank-m0, we conduct a comprehensive evaluation of MonoQwen2-VL--ve across the full spectrum of task types. Given that MonoQwen2-VLL VL-v0. is exclusively trained and tested the T- task, its input configuration is specifically tailored to this particular scenario, as illustrated in Table [3]
does not incorporate additional instructions during training and lacks inherent instruction- mnsruution-follooinng capabilities, we leverage plate to uniformly configure the inputs for all other tasks, shown under the Others in Table [3]
Notably, since MonoQwen2-VLL-oo
the established T-VD input tem-
Input Format
T - VD [doc)(nAaser) the relevance of the previous image document to the following query, answer Tzue False. The query (query)
(doc)(nAssert the relevance the previous document to the following queey, answer True False. The query is: (query)
Table 13: The input template of MonoQwen2-VLL-one VL-v0.1. T + VD is the original input format of it, and we design the input formats for other tasks based on this format, as shown in Others.
20

---

Published as a conference paper at ICLR 2026
Task
Dataset
Query Instruction
ArguAna
Given claim, find documents that refute the claim.
Climate FEVER
Given claim about climate change, retrieve documents that support orrefute the claim.
CQADupSmck
Given question, retrieve detailed question descriptions from Stackexchange that are duplicates to the given question.
DePid
Given query, retrieve relevant entity descriptions from DBPedia.
T-+T
FiQA2018
financial question, retrieve user replies
answer question.
HotpotQA
Given multi-hop question. retrieve documents that can help answer the question.
MSMARCO
Given web scarch query, retrieve relevant passages that answer the query.
NFCorpus
Given question, retrieve relevant dncuments that best answer the question.
NQ
Given question, retrieve Wikipedia passages that answer the question.
Quora
question, retrieve questions that semantically equivalentto the given question.
SCIDOCS
Given scientific puper title, retrieve paper abstracts that are cited bythe given paper.
SciFact
Given scientific claim, retrieve documents that support or refute theclaim.
Touche2020
Given question, retrieve detailed and persuasive arguments that answer the question.
TRECCOVID
Given query COVID-19 retrieve documents that answer the query.
Nights
Find a day-to-day image that looks similar to the provided image.
VisualNews
Identify the news related image in line with the described event.
T-1
Fishion200k
Based on the following fashion description, retrieve the best matching image.
Memotion
Retrieve meme hased the given caption.
HatefulMemes
TAT-DQA
ArxivQA
T-+VD
DocVQA
Find sercenshot that relevant the user's question.
MITTI
VisnalNews
Find a caption for the news the given photo.
1-T
Fashion200k
Find a product description for the fashion item in the image.
GLDv2
Retrieve the name of the landmark based on the given image.
Memotion
HatefulMemes
Retrieve the caption based on the given meme..
WehQA
Find Wikipedia image that answers this question.
T-IIT
Find news image
provided caption.
OVEN
Retrieve Wikipedia paragraph that provides an answer to the given query about the image.
IT+T
INFOSEEK
Find a paragraph from Wikipedia that answers my question about this image.
OKVQA
Retrieve documents that provide an answer to the question alongside the image.
VizWiz
Retrieve
for question about image.
Fashion
Find fashion image aligns with the reference image and style note.
CIRR
Retrieve a day to day image that aligns with the modification instructions of the provided image.
OVEN
Retrieve Wikipedia image- description pair that provides evidence for the question of this image.
IT
INFOSEEK
Find image and subject description from Wikipedia that answers my question about this image.
B-VQA
Obtain illustrated documents correspond inquiry alongside provided image.
Table 14: The instructions for different tasks. We set the instructions for the GME-2B and GMR series models on each task as shown in the Table. WER means World Economic Reports, and MITTI means MIT Tissue Interaction
21

---

Published as a conference paper at ICLR 2026
E MAIN RESULT
E.1 DETAILED RESULTS
We evaluate models described on our benchmark. The evaluation metrics and the detailed
results for each dataset are reported in Table 15
E.2 THE INFLUENCE THE NUMBER OF NEGATIVE
In $.2 we examine the effect of incorporating negatives
59.0
in supervised fine-tuning (SFT) and observed that, within the limits of available computational resources, increas- ing the number of negative examples consistently im- proved model performance. The best performance was achieved when number negative examples reached further conduct experiments
58.5
58.0
57.5
57.0
16. For comparison, the role of negatives in contrastive learning. As shown in Figure TO] the results indicate that, similar to SFT, larger number of negative examples leads to better per- formance. Nevertheless, the overall performance of con- trastive learning remains lower than that of supervised fine-tuning.
56.5
SFT
56.0
CL
55.5
2
4
16
#Neg
Figure 10: Average performance of the number of negatives per sample.
E.3 THE INFLUENCE OF THE FROZEN OF LM HEAD
In $ we observe that SFT can exploit semantic signals from pre-trained token embeddings, whereas CL must learn the score-projection matrix from scratch. To rule out the potential influence of freezing the language modeling (LM) head pa- rameters, we conduct an ablation study LM head parameter freezing, with the results presented in Table 16 The findings show that freezing unfreezing the LM head has effect on SFT. In contrast, CL achieves better performance when the LM head parameters are frozen. These results suggest that SFT effectively leverages the semantic information embedded pre- pre-trained token of LLM, while CL requires relearning the score- projecproo matrix.
-F
-NF
A,
SFT
57.97
57.94
0.03
CL
55.95
57.20
1.25
Table 16: Impact of frozen of the LM head on performance. -F de- notes frozen, while -NF denotes not frozen.
E.4 THE INFLUENCE OF NEGATIVE SAMPLE SELECTION
In train models using 1:1 ratio of randomly sampled negatives and hard negatives. To validate the ef- fectiveness of this negative-sample selection strategy, well as to examine the sensitivity of SFT and CL to neg- atives, conduct additional analysis based setting. Specifically, we train the models on the MS MARCO using configurations largely consistent with those in $ and evaluate them on the T- test sets. The results are presented in the Figure
52
Perf.
51
pure text-
50
SFT
CL
49
Random
Balance
Hard
Figure 11: Average performance on the T-+T under different negative sampling strategies. Random denotes using ran- dom negatives, Balance denotes a 1:1 mix of random and hard negatives, and Hard denotes using hard negatives.
We observe that both SFT and CL achieve their best per- formance under the balanced condition, with SFT con- sistently outperforming CL, demonstrating its superior- ity. Moreover, SFT exhibits greater robustness to varia- tions in regative-sampl. selection compared to CL, fur- ther highlighting the advantages of the SFT framework.
These findings further verify the soundness and accuracy of negative-sample selection strategy.
22

---

Published as a conference paper at ICLR 2026
Model
Class
Dataset
GME-2B
Qwen3
MonoQwen
Jina-mo
GMR-3B
GMR--B
ArguAna"
47.11
86.00
50.93
56.07
80.42
84.49
SCIDOCS
22.65
26,42
18.31
22.12
25,49
28.77
TRECCOVID!
79.11
87.83
79.84
85.36
87.23
85.56
Quora'
87.35
88.16
82.71
87.98
89.51
89.91
SciFactt
66.53
79.83
74.94
79.18
77.52
79.70
NFCorpus'
36,90
41.88
38.29
40.99
40.51
40.81
"iimate-FEVER'
32.15
49.08
19.78
34.33
50.14
50.26
(14)
FiQA2018!
46.35
56.25
44.11
50.72
54.79
59.64
HotpotQAt
70.45
82.66
71.64
80.49
82.86
DBPedial
43.17
52.69
41.75
49.60
52.99
53.96
Touche2020*
33.18
43.00
36.71
38.40
32.17
37.26
NQ
51.22
63.33
49.08
62.06
62,49
66.48
MSMARCO
40.79
44.57
35.57
43.09
45,90
47,60
CQADupStack'
37.25
45.18
40.83
44.66
47.10
46.81
Avg.
49,59
60.49
48.89
55.36
59.22
61.08
I+)(1)
Nights* Avg.
30.75
12.59
27.50
29.76
32.83
Fashion200k*
25.77
29.14
29.38
25.01
27.57
HatefulMemes!
52.09
74.93
76.57
75.07
75.19
(-)
Memotion
77.41
93.47
93.40
93.17
93.52
VisualNews+
38.55
37.39
38.48
42.16
48.44
Avg.
48.46
59.46
58.73
58.85
61.18
TAT-DQAf
71.23
79.99
82.05
83.23
84,00
VQA
56.44
57.51
61.69
61.48
62.87
ArxivQAf
84,21
87.61
89.38
88
90.99
(S)
WERT
58.78
63.00
63.47
62.13
MITTI
61.29
68.32
69.07
66.06
65.82
Avg.
66.39
71.29
73.13
72.38
72.94
Pashion200k
27.67
7.55
17.14
26.22
29.80
HatefulMemes!
57.85
32.27
80.90
81.21
81.23
(5)(5)
Memotion!
80.01
44.74
94.84
96.08
GLDv2!
59.28
5.72
59.21
68 68.68
76.74
VisualNewsx
38.28
7.83
25.05
43.12
48.60
Avg.
52.62
19.62
55.43
63.06
66.61
WehQA*
83.03
87.30
87.14
86.98
87.46
(2)T(
EDIS*
71.00
65.63
62.76
76.95
81.64
Avg.
77.02
76.46
74.95
81.96
84.55
OKVQA"
29.71
20.13
30.34
37.71
40.09
VizWiz
29.56
5.11
20.36
35.96
41.29
IT)T(()
INFOSEEK*
39.77
23,97
36,84
59.17
63.01
OVEN*
60.46
8.18
23.74
62.41
68.78
Avg.
39.88
14.35
27.82
53.29
FashionIQ"
26.57
21.41
25.97
30.70
33.32
(2)1
CIRR*
46.83
42.09
49.33
57.24
61.46
Avg.
36.70
31.75
37.65
43.97
47.39
INFOSEEK*
44,61
35.39
53.28
73.89
76.31
E-VQA*
79.11
55.81
61.21
84.66
86.08
IT (3)
OVEN+
76.96
16.28
40.12
78.68
84.17
Avg.
66.89
35.83
51.54
79.08
82.19
AL1(40)
Avg.
52.54
44.20
54.36
61.40
63.85
Table 15: Detailed scores of each model on various datasets MRB. WER denotes World Economic Reports, and MITTI refers to MIT Tissue Interaction. For datasets marked with */* we report Recall@5/Recall @ 10, and NDCG@5 NDCG@ 10 is used for t-labeled / f-labeled datasets.
23

---

Published as a conference paper at ICLR 2026
F THE USE OF LARGE LANGUAGE MODELS (LLMS)
We primarily employ large language models (LLMs) for manuscript refinement, encompassing edit- ing and stylistic enhancement improve clarity, coherence, and consistency. Furthermore, our series as the foundational architecture,
GMR models are built upon the leveraging its multimodal understanding and cntrructionfollowwinn capabilities, while incorporating task-specific supervised fine-tuning to achieve pergh-erffrmance reranking across diverse scenarios.
G LIMITATION
In this work, we introduce MRB, a benchmark designed for training and evaluating multimodal reranking tasks. To address this challenge, we investigate strategies for adopting Multimodal Large Language Models (MLLMs) into general-purpose multimodal reranking models, and propose GMR, a reranking model capable of handling candidates across different modalities. Despite these contri- butions, our work has the following limitations:
Single-language constraint. Although backbone model, Qwen2.5-VLL -L--sstutnnnt supports multiple languages, we trained and evaluated GMR exclusively in English. Consequently, the per- formance of GMR in other languages remains unexplored.
Single-image constraint for queries and documents. For reasons of training efficiency and limited availability of relevant data, both queries and candidates in MRB are restricted to a single image for each query and document. As result, the benchmark cannot assess performance interleaved inputs that involve multiple images and texts.
24


Reviews:
[{"review_id": "ADME92qcd7", "reviewer": "Reviewer_51SZ", "summary": "This paper explores the tradeoffs between direct supervision (binary cross entropy) and contrastive loss (InfoNCE) for training reranking models. They decompose each loss into two parts: weight (gradient magnitude) and direction. Given that both losses rely on the same data, one can mix and match components to have a new loss that uses the weight of direct supervision and direction of contrastive (and vice versa). The authors find that direct supervision is more effective than contrastive in their experiments, and analysis reveals that it is in large part because of the \"weight\" component of direct supervision --- extrapolating from the analysis, my sense is the weight is potentially more robust against easy negatives, given that contrastive leads to relatively small gradients. They train a reranker with direct supervision and see strong results across multiple benchmarks.", "questions": "Did you investigate whether SFT or CL is more robust against hard negatives? Jacob et al investigate reranker failures and find that reranker can get progressively worse as they need to rerank more documents at test time, suggesting they are undertrained against noisy negatives. See: https://arxiv.org/abs/2411.11767", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The paper nicely walks through the differences of direct supervision and contrastive loss, fundamental concepts for embedding and reranker training.\n\nS2. There was extensive analysis of the two losses, including of their hybrid forms. This analysis reveals some insights for reranker training, although it was unclear how this influenced their main result experiments. Those experiments simply could have use direct supervision and contrastive as a hyperparameter to explore.\n\nS3. The authors train multiple versions of Qwen2.5-VL-Instruction for reranking with strong performance across multiple benchmarks. Details for training and data selection are provided. The authors also say they will release code, data and models.", "weaknesses": "W1. The paper is not self contained in the main text. Many of the main results are in the appendix. For this statement \"Rival and surpass leading textual reranker\", there is no easy way for the reader to verify. The relevant results are in the appendix, and not averaged across datasets.\n\nW2. There are only experiments when reranking the top-100 retrieved documents, but many real world settings will require reranking more or less than this amount. Given that the reranker here is pointwise, it would be relatively straightforward to report results across a large range of top-N rather than only N=100. See https://arxiv.org/abs/2411.11767v2 which incorporates a related evaluation protocol and shows how reranker quality can degrade as more documents are reranked.\n\nW3. The masking approach in sec 4.1 seems very similar to margin-based methods, yet no margin-based methods are explored. It would be interesting to see if contrastive loss with additive margin yields a similar effect. https://arxiv.org/abs/2203.02167\n\nW4. There are more losses that can be explored. The RD-Suite paper https://arxiv.org/abs/2306.04455 is focused on distillation, but it is a good starting point for exploring other losses. RankGPT and Pairwise Rank Prompting (PRP) are not well suited for multimodal reranking unless multiple documents can be represented in the same context.\n\nW5. Importantly, not many approaches for negative selection are explored. Given that one of the main weaknesses of CL is small gradients, likely caused by easy negatives, it seems important to explore the proposed framework under different negative selection strategies. Previous work shows more than a 10% difference in nDCG is possible when using different negative selection strategies for embedding models, and it's plausible this would apply to rerankers, see https://arxiv.org/abs/2407.15831\n\nW6. Finally, the authors should make it clear whether they plan to release the model or data. These would greatly strengthen the contribution of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EgGdkgXoSv", "reviewer": "Reviewer_XjCJ", "summary": "This paper systematically investigates two primary training paradigms for large language model (LLM)-based reranking—Supervised Fine-Tuning (SFT) and Contrastive Learning (CL)—within a universal multimodal retrieval setting. The authors introduce a unified theoretical framework that decomposes both objectives into weight and direction components, enabling a fair analytical comparison. Through extensive probing experiments, they show that the weighting mechanism in SFT provides stronger optimization signals than CL, accounting for most of the performance gap. The study culminates in the development of GMR (General Multimodal Reranker) models (3B and 7B parameters), trained via SFT, that achieve state-of-the-art results on a newly constructed MRB benchmark comprising 40 multimodal retrieval datasets. The work provides both theoretical insight and practical advances for multimodal reranking.", "questions": "1, What about list-wise reranking (where llm generates a sequence of doc ids)? e.g (https://aclanthology.org/2024.emnlp-main.491.pdf) In this case, what's the difference between sft and cl?\n\n2,  How sensitive is the performance gap between SFT and CL to negative sampling strategies or batch size?\n\n3, Does SFT’s superior performance persist when the LLM backbone is not generative? e.g. A randomly initialized LLM", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1, Extensive experiments, including ablations, controlled comparisons, and probing analyses, convincingly support the claims.\n\n2, MRB provides a comprehensive, standardized testbed likely to benefit the broader research community.\n\n3, GMR models consistently outperform existing multimodal rerankers across 40 datasets.\n\n4, The paper addresses a timely and underexplored question—the objective design for LLM-based reranking—bridging theoretical understanding with real-world performance.", "weaknesses": "1, The conclusion of this theoretical analysis – \"CL computes the weight using all positive and negative documents within a sample, while SFT assigns weights independently per document, making this the likely key factor in performance variation\" – is extremely obvious. It can be verified simply by checking which terms are involved in the calculation when computing the loss. What we are more curious about is:  Why would training with both positive and negative examples lead to better or worse performance? We need a theoretical explanation from perspectives such as generalization, embedding space, and the difficulty of training convergence. These are exactly the areas where mathematical analysis should play its role.\n\n2, In the ai search era, we mainly care about llm reranker for text. Experiments on mllm rerankers are somehow limited.\n\n3, While weight and direction are well formalized, their intuitive interpretation in semantic space could be further elaborated. e.g. what's the difference between \"generating\" the score and the traditional \"encoding then scoring with mlp\" pipeline?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "km3kZVa4D9", "reviewer": "Reviewer_W9sD", "summary": "This paper asks whether Supervised Fine-Tuning (SFT) or Contrastive Learning (CL) is inherently better for LLM-based (multimodal) reranking. Authors build a unified reranking loss (URL) that decomposes training signals into two orthogonal parts—weight (magnitude) and direction (update vector)—such that SFT and CL can be compared under the same lens. Empirically, SFT beats CL (on a newly assembled, broad Multimodal Reranking Benchmark). Ablating URL shows the weighting scheme explains most of the gap while update direction matters less. Finally, the authors train GMR-3B/7B rerankers with SFT and report SOTA on MRB.", "questions": "See other questions in weaknesses; see also\n\n* In Appendix E.3, unfreezing the LM head benefits CL but not SFT. Could you explore whether adding a small projection layer (instead of the LM head) narrows the gap? That would indicate whether SFT’s advantage comes from leveraging pretrained lexical priors?\n* Do you think your analysis could extend to pairwise or listwise rerankers where the loss itself defines relative weighting across candidates rather than independent pairs? Clarifying this could help generalize the decomposition beyond pointwise reranking.\n* Authors directly tackle multi-modal reranking; could you show the same findings in standard text scenarios (mono- and/or multi-lingual)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Multi-modal retrievers are very popular these days, and less work is dedicated to multi-modal rerankers. It is an interesting topic per se—given how rerankers usually boost performance in any given setting—we can see how GMR (but also the tested baselines) improve over the tested retriever (GME-2B). \n* Overall the paper is clear, straight forward and well written (a few things could be improved though, see later). Related works contain most of the relevant references I am aware of. I would maybe add [1] somewhere (the first paper doing a cross-encoder with BERT).\n* The weight/direction derivation gives a mechanistic account of why SFT wins: SFT’s weights are per-document (no across-negative normalization), while CL’s weights couple all negatives, which weakens positive signal magnitude-especially when many easy negatives are present. The derivation looks correct to me. The loss decomposition shows that supervised fine-tuning (SFT) and contrastive learning (CL) differ mainly in how they weigh each training example, not in the basic direction of the update.\n* This feels like an interesting (**although not ground breaking**) contribution imo. But the main takeaway is SFT > CL for training pointwise rerankers, which is a valuable finding by itself.\n* After ablations (SFT vs CL), authors train two multi-modal rerankers (GMR 2B and GMR 7B) on 1.5M instances. They also assemble a new benchmark MRB (40 **existing** datasets; multiple modalities and tasks), making results quite robust; Table 4 shows GMR outperforming strong baselines across categories. Note that the baselines are 2B (or 4B for Qwen). \n\n[1] Passage Re-ranking with BERT", "weaknesses": "* Novelty/contribution feels a bit limited, but I still could see this paper fit into ICLR (especially if authors release the code and models as stated in the paper). Especially, it focuses (by design) on pointwise approaches, and obfuscate listwise rerankers which might become more predominant in the future (due to the nature of LLMs).\n* “MRB excludes datasets where the retriever already scores very high”; in these cases rerankers have no effect? Scores could be very high for retrievers but still be improved by rerankers in practice no? Anyway, authors could also have included “weaker” retrievers; this would even show more the effect of reranking. Maybe varying topk could have been beneficial too, to test whether SFT’s advantage holds when the negative mix changes.\n* Although the paper is clear overall, some parts could be improved/clarified (not critical). There are many typos/weird formulations across the paper; for instance, in the Supervised Fine-Tuning paragraph (Section 3.2), “The objective is predicting correct next token” or “Then predict the likelihood of “yes” [...]”. And there are many more...\n* It seems that what makes ICL different from SFT is the fact that ICL couples all negatives through a softmax denominator (vs independently for SFT); it feels that the impact of negatives could have been studied further. Do findings still apply with 1 neg? 100 negs? What about temperature which is barely discussed (although is it common practice)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jYx9PNdB0r", "reviewer": "Reviewer_xxqB", "summary": "This paper compares two popular approaches for training reranking models—Supervised Fine-Tuning (SFT) and Contrastive Learning (CL)—in the context of large language model (LLM)-based reranking. The paper introduces a unified framework that decomposes the loss functions into two key components: weight and direction. The authors conduct a comprehensive theoretical analysis and empirical comparison between SFT and CL using the Universal Multimodal Retrieval (UMR) task. Their findings suggest that SFT offers better performance than CL due to its stronger optimization signals, specifically in the weight component. The authors also introduce a new multimodal reranking benchmark (MRB), which consists of 40 datasets across various modalities. Through this analysis, they develop new state-of-the-art rerankers, GMR-3B and GMR-7B, achieving superior results in the MRB benchmark.", "questions": "1. Since the MRB benchmark is restricted to single-image inputs, would the performance of the models differ when dealing with queries or documents containing multiple images or mixed modalities (e.g., a combination of images and text)?\n\n2. The paper mentions the use of random and hard negative sampling strategies. How do the negative sampling strategies influence the performance in specific modalities (e.g., images vs. text)? Is there a particular strategy that consistently outperforms the others across all tasks?\n\n3. While the paper primarily focuses on SFT and CL, would it be worth exploring hybrid architectures that combine the strengths of both methods? How could a mixed approach impact the performance of multimodal rerankers?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel framework to analyze two popular reranking training methods, SFT and CL, by decomposing their loss functions. This decomposition into weight and direction provides new insights into their effectiveness. Additionally, the creation of the MRB benchmark is a significant contribution to the multimodal reranking field, offering a diverse and comprehensive dataset for evaluating models.\n\n2. The authors provide a thorough theoretical analysis combined with extensive experiments. They rigorously evaluate both SFT and CL, demonstrating that SFT outperforms CL in terms of model performance. The experiments are well-designed, and the results are statistically significant.\n\n3. The paper is clearly written, with well-defined objectives, methodology, and results. The use of equations and illustrations, such as Figure 1, helps clarify the distinctions between the two training paradigms. The explanations of the framework, loss function decomposition, and the MRB benchmark are easy to follow.\n\n4. The findings that SFT offers superior performance to CL for LLM-based reranking are crucial for future research and practical applications in multimodal information retrieval. The introduction of the MRB benchmark and the state-of-the-art models developed (GMR-3B and GMR-7B) have the potential to impact a wide range of tasks in multimodal retrieval systems.", "weaknesses": "1. While the MRB benchmark is comprehensive, it might be seen as overly focused on tasks where image-text retrieval is central. Future work could explore additional modalities or larger-scale datasets to evaluate the generalizability of the reranking models to other forms of multimodal data, such as video or audio.\n\n2. Given the reliance on MRB for validation, there is a risk of overfitting to this specific benchmark, as many models may perform well only on the datasets included in MRB but struggle on entirely new or more complex multimodal benchmarks. The paper could benefit from additional cross-benchmark evaluations to test the robustness of the proposed models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)