Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrieval
Authors: Lanyun Zhu
Deyi Ji
Tianrun Chen
Haiyang Wu
Shiqi Wang
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal retrieval
Primary area: applications

Abstract: The success of DeepSeek-R1 demonstrates the immense potential of using reinforcement learning (RL) to enhance LLMs' reasoning capabilities. This paper introduces Retrv-R1, the first R1-style MLLM specifically designed for multimodal universal retrieval, achieving higher performance by employing step-by-step reasoning to produce more accurate retrieval results. We find that directly applying the methods of DeepSeek-R1 to retrieval tasks is not feasible, mainly due to (1) the high computational cost caused by the large token consumption required for multiple candidates with reasoning processes, and (2) the instability and suboptimal results when directly applying RL to train for retrieval tasks. To address these issues, Retrv-R1 introduces an information compression module with a details inspection mechanism, which enhances computational efficiency by reducing the number of tokens while ensuring that critical information for challenging candidates is preserved. Additionally, a new training paradigm is proposed, including an activation stage using a retrieval-tailored synthetic CoT dataset for more effective optimization, followed by RL with a novel curriculum reward to improve both performance and efficiency. Incorporating these novel designs, Retrv-R1 achieves SOTA performance, high efficiency, and strong generalization ability, as demonstrated by extensive experiments across multiple benchmarks and tasks.

Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrieval
Lanyun Zhu'
Deyi Ji
Tianrun Chen²
Haiyang Wu
Shiqi Wang'
City University Hong Kong
Tencent Zhejiang University
Abstract
The success of DeepSeek-RI demonstrates the immense potential of using rein- forcement learning (RL) to enhance LLMs' reasoning capabilities. This paper introduces Retrv-R1, the first RI-style MLLM specifically designed for multimodal universal retrieval, achieving higher performance by employing step-by-step rea- soning to produce more accurate retrieval results. We find that directly applying the methods of DeepSeek-RI R1 RIII to retrieval tasks is not feasible. mainly due to (1) the high computational cost caused by the large token consumption required for multiple candidates with reasoning processes, and (2) the instability and suboptimal results when directly applying RL to train for retrieval tasks. To address these issues, Retrv-R1 introduces an information compression module with details inspection mechanism, which enhances computational efficiency by reducing the number of tokens while ensuring that critical information for challenging candi- dates is preserved. Furthermore, a new training paradigm is proposed, including activation stage using retrieval-tailored synthetic CoT dataset for more ef- fective optimization, followed by RL with novel curriculum reward to improve both performance and efficiency. Incorporating these novel designs, Retrv-R1 achieves SOTA performance, high efficiency, and strong generalization ability, as demonstrated by experiments across multiple benchmarks and tasks. |Project page
Introduction
Information retrieval is a key direction in AI with broad applications, such search engines 9 Early work primarily focused on retrieval for single,
and -enrietal-augmented generation 28 63)
fixed-format data types, such text-to-text 76
40.
text-to-image [10 73), and image-to-image
retrieval [4] [11. To improve efficiency and facilitate real-world deployment, recent efforts [59 38 34]
have increasingly emphasized universal multimodal retrieval, where single model is required to handle diverse retrieval tasks and data modalities simultaneously. Earlier work in this setting typically
employs small foundation models such CLIP, while more recent methods [38 34 77 27 explore
leveraging stronger Multimodal Large Language Models (MLLMs) to achieve further improvements, taking advantage their rich pretrained knowledge and strong generalization capabilities.
Some retrieval methods 77 apply MLLMs by leveraging their embeddings for similarity com- parison, but their effectiveness and robustness often unsatisfactory due to the imprecision of embeddings and similarity computations. Other approaches [3 38) formulate retrieval as a question- answering (QA) task by constructing input instruction from the query and candidates, prompting the MILM to generate the retrieval result. These methods achieve improved performance by exploit- ing the strong QA capabilities of MLLMs. However, they typically rely on the MILM directly generate the result without any explicit reasoning process, which limits their ability to handle complex cases. Moreover, the supervised fine-tuning (SFT) strategies commonly used train retrieval MLLMs
*Coreesponiig Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

suffer from several shortcomings, such as poor generalization and frequent hallucinations, as widely reported in prior studies 668 24 These challenges motivate us to explore new paradigm for MI..M-bassd retrieval, one that incorporates stronger reasoning capabilittess and a more effective training mechanism, thereby achieving higher retrieval accuracy and greater reliability.
Recent studies [61 67 43 have shown that reinforcement learning (RL) can effectively optimize a model's reasoning abilities, enabling LLMs to solve complex tasks, such as mathematics and pro- gramming, more effectively through step-by-step chain-of- -fhoughhtt (CoT) reasoning. As a prominent example of such RL-based methods, DeepSeek-RI [4] has achieved particularly remarkable success, significantly improving LLM performance by utilizing novel Group Relative Policy Optimization (GRPO) algorithm. Moreover, DeepScek-RI computes rewards using predefined rules, rather than relying on additional models as in many previous RL approaches, thereby simplifying the pipeline and reducing optimization complexity. These advantages inspire us to explore the question: whether the paradigm of DeepSeek-RI can also be leveraged to enhance the capabilities of retrieval MILMS?
Based on the above motivations and answer the aforementioned question, in this work, we propose Retrv-R1, the first RI-like reasoning MLLM framework designed for universal multimodal retrieval tasks. As initial exploration, we first attempt directly train a standard MLLM using the naive GRPO on the retrieval datasets. However, the performance is unsatisfactory due to the following issues: (1) We observe that the model's convergence is difficult and unstable. Moreover, the trained model often generates incorrect reasoning processes, leading to suboptimal performance and inaccurate retrieval results. This aligns with observations from prior work [16), which indicates that directly applying RL struggles to effectively incentivize the reasoning capabilities of ML.MMs, especially when the training data limited in quantity or quality. (2) During inference, feeding tokens from all candidates into the MILM introduces significant computational and memory overhead. This issue becomes particularly problematic when CoT reasoning is involved, as the fine- fine-grained reasoning process consumes even more tokens, potentially exceeding the model's context length and available computational resources. To address these challenges, we introduce a novel information compression module alongside details inspection mechanism. In this approach, most candidates are compressed into a small number of tokens, while only a few challenging candidates, identified automatically by the MLLM during the CoT process, retain their full features. This strategy not only effectively reduces the token count, thus preserving sufficient context for CoT reasoning, but also mitigates the negative impact of compression-causel information loss on the judgment of difficult candidates. Additionally, synthetic CoT dataset we propose an activation method prior to RL training, where a difficulties arising from directly is generated and used for SFT, thereby avoiding the optimization applying RL. We also introduce novel curriculum-based mechanism for the RL reward, enabling the MLLM to develop strong reasoning abilities while maintaining high efficiency.
Incorporating these novel designs, Retrv-RI constitutes a powerful and efficient retrieval MLLM framework. We conduct extensive experiments across multiple benchmarks, such the universal retrieval benchmark M-BEIR and out-of-domaan benchmarks on multimodal recommendation tasks. The strong performance across diverse settings demonstrates the high effectiveness and generalization ability of Retrv-R1. 1. In summary, the main contributions of this work are: (1) We propose Retry-R1, the first RI l-style reasoning MLLM framework specifically designed for universal retrieval tasks. (2) We introduce several innovative and task-tailored designs within the Retry-R1 RI framework, including new model structure reduce token consumption, a details inspection mechanism to address challenging candidates, and a curriculum-basel reward system to improve both effectiveness and efficiency. (3) Extensive experiments across multiple benchmarks and task settings demonstrate the strong performance, efficiency, and generalization ability of Retrv-R1.
2 Related Work
Multimodal Retrieval. The development of deep learning
18 has driven rapid progress across a wide range of retrieval tasks, such as text- text-image cross modal retrieval multimodal document retrieval 72 D. In recent years, vision- language models
86
45
10
73
g
23 22 composed image retrieval 4 5T 13, B 3
6
15
39 and instruction-
image retrieval 62
such as CLIP 48 have achieved remarkable success this domain 4
44 21.. For example, enhances CLIP through prompt tuning to enable highly generalizable retrieval. More recently, MI Ms have been applied to further improve retrieval performance 38 20 777. Some methods
55
50
22
7 27 34 leverage embeddings extracted by MLLMs for similarity-based retrieval, while LamRA

---

38 employs MLLMs to rerank candidates and identify the best match. In this work, we propose the first Rs-slel MILM for multimodal retrieval, leveraging the strong reasoning capabilities acquired through reinforcement learning to significantly enhance retrieval effectiveness.
extend the capabilities of L.MSs to the multimodal domain and have achieved remarkable success across various tasks, such as visual question answering [25
MLLMM MILMS 37 29
57
79
52
46
and image segmentation [26 4 To improve their efficiency, recent studies [74
70
65
32
have explored reducing the number of input tokens to lower computational costs. For example,
774
introduces compression module that reduces visual tokens while performing early fusion of image and instruction information. [70] compresses video tokens to support long-contex, video inputs. Our work also incorporates a token compression mechanism to accommodate more candidate inputs, but it features the following key novelties compared to prior methods: (1) Our compression mechanism the first specifically tailored for universal retrieval, including a novel network architecture and training strategy to ensure that compressed tokens preserve retrieval-relevant information. (2) We introduce a novel details inspection mechanism that enables the MILMM to automatically identify and incorporate the full, uncompressed token sequences of challenging candidates during the CoT process, thereby effectively mitigating the negative impact of information loss caused by token compression.
LLM and MILM Reasoning, Enhancing the reasoning capabilities of LLMs and MLLMs is key direction for AI research. Early work 664
typically employed fixed-forna Chain- of of-Thought (CoT) prompting to enable LLM reasoning. Inspired by the success of DeepSeek- R1 many recent methods have adopted reinforcement learning optimize the reasoning abilities of 83 17 In this work, we propose the first RI -style reasoning MILMM framework for multimodal retrieval, introducing several task see--speifiic innovations, including reasoning activation training strategy using synthesized CoT retrieval dataset, and novel reward function that incorporates a curriculum-Be token efficiency constraint. These designs significantly improve both the effectiveness and efficiency of MLLM-based multimodal retrieval.
60
75
58
85
19
14.
LLMs 61 67 and MILMs [6 56 1 66
53
3 Method
3.1 Preliminaries and Overview
This paper proposes a novel framework, Retrv-RI, to tackle the task of universal retrieval. this task, given query q of any modality (text, image, interleaved formats), the goal is retrieve the most relevant sample from a set f2 = {Ca]] with N candidates. Following (38) Retry-R1 adopts two-stage approach improve retrieval efficiency and effectiveness through coarse-to-fine process. In the first stage, embeddings are generated for and each candidate e, using an MLLM ø. The top-K candidates with the highest embedding similarity q are then selected, forming a subset denoted as C = {ck] J k=1- In the second stage, another MLLM 0 is employed to further refine the retrieval by identifying the best- matching item from C, yielding the final retrieval result 0(g. C). We adopt the same method as (38) for constructing the embedding model 6 in the first stage. In the following sections, we primarily illustrate the proposed new method for building. training, and applying the second-stage selection model @ within our Retry-R1 framework.
LK
3.2 Baseline Method and Shortcomings
The recent success of reasoning LLMs and MLLMs motivates explore the potential of chain-of-thought (CoT) reasoning for enhancing MLLM-based retrieval. As an initial attempt, we first construct baseline method by directly fine-tuning Qwen2.5-VL on the retrieval dataset using Group Relative Policy Optimization (GRPO) [1]. In this approach, the model receives the following prompt input: Please select the candidate that best matches <query> and tell me its index. Candidate 1: <c,>, Candidate 2: <C2>,
Candidate K: <cg>. Please think it step-by-step. A set of generated CoT's (0r, 02,.0...00 .G] is randomly sampled, and the model is optimized by maximizing the following objective:
G
(o,)
(o) = P(Q).)))))))))_,_>,rp,,,,. (01v)
min
Tea (o:) As
1=1
(1)
(o())) clip
Tem (o,) 1 c, 1 1 41) - DKL (Tell Srat)
3

---

[T] Overall Pipeline
Overall Training Process
Qutry
Candidale
Candidate 2
Candidale K
Stage
Selt-Alinmeet Protraining ICM
felltiooship Prompt Template: According creb. descrite the cimlarity dffereere
CiT E.oox
potectial mistionniip betwoen the In
Find image with different animal the
ICM
Prompt Template: dctail
Information Compression Module (ICM)
Feom
T
Stage SFT Synthatic
Retrieval result result apcculation fisation regether samples exrification samples go
Large Language Model
<thinko...
end- <irsgettion
Indes: IndoossaarkkKiinpetttto-ddeeeeer </think canswnn2</answen
Armusasion
MLUM
Syntheik Dotaser
Stage 3: RL. with Curriculum Reward for Enhancerenti
Information Compression Module (ICM)
Reward Calculation
P
Format
Besult
Efficiency
GRPO
ATT
ATT
Rolous
Optimize
Roward
Figure 1: Illustration of (A) overall pipeline; (B) structure of information compression module (ICM); and (C) overall process model training with three stages.
where DKL refers to the KL divergence. A, meani (a....a))))))))) the average of the group Pa......
rewards {ri) where r; if the final retrieval result of o is correct otherwise r; 0. All other hyperparameters in follow the same settings as [16]. As discussed in the introduction, this naive RL-trained model exhibits significant shortcomings in effectiveness, due to the convergence difficulties and training instability associated with direct RL-based optimization, as well as high computational costs caused by the large number of tokens from multiple candidates and fine-grained CoT reasoning. To address these challenges, Retrv-RI extends the model architecture of the standard MILM improve retrieval efficiency, and introduces new training strategy to enhance retrieval effectiveness. In the following Sec 3.3 and 3.4
we introduce these two key innovations, respectively.
3.3
Information Compression Module
We first design simple but effective Information Compression Module (ICM) based intuitive idea: reducing the number of each cą 's tokens for the language model (LM)'s inputs, thereby leaving sufficient context space for the complex and token-intensive CoT reasoning. As shown [[) the ICM is placed before the LM of the MLLM. Inspired by 774, given the original token sequence T for each candidate e, ICM uses learnable embedding econ and two-layer attention module ATT, to compress the information into a single content token "con This process is expressed as:
Ck con
ATT1 (Qe... KT., Vr.,,)
(2)
where Q, K and V refer to the query, key and value in the attention mechanism, respectively. In retrieval tasks, beyond the standalone content of each candidate сą captured by t% its correlation with the query (e.g., their differences and similarities) is also critical for the accurate matching. To capture this crucial information, we further generate a relationship token to for each Cg as follows:
rel = ATT, (Q..... KR V V where Rg.. = ATT, (Qr., KT,, Vv,,))
(3)
where T, denotes the token sequence of query q and ATT, represents another 2-layer attention, By.eg denotes the value features derived from Rg where Ry.e, the relationship feature between the query and the k-th candidate Ck- In this way, ICM compresses each c; into only two tokens, each carrying different types of information, thereby preserving key retrieval-relevan features while significantly reducing context usage.
Self-Alignnen Alignment for Pretraining ICM. Despite its simple structure and intuitive functionality, training the ICM is non-trivial. Directly fine-tuning the ICM together with the LM in the MI.LM often leads to convergence difficulties and suboptimal results. To address this challenge, adopt pretraining

---

mechanism inspired by BLIP-2 (29), aligning the output space of the ICM with the input space of the LM before jointly training them for the retrieval task. To ensure that the compressed tokens produced by the ICM effectively captures key retrieval-related information, we propose self-alignment mechanism for pretraining with the following optimization funetion:
Lea = Ec,, [Lee (LM (Jeon (rom)) LMM (Jcon (T) + Lace (LM (Trel [+aa)) LM (Trel [Tey TgD)]
where Lee denotes the cross-entropy loss. Loon and Leel are 2 instruction templates that prompt the LM generate content and relationship descriptions, respectively. There details are follows:
Teon [<c,>]: Describe the content of <Cp> in detail.
Lret [<rel>]: According to <rel>, describe the similarity, difference and any potential relationship between the pair in detail.
We optimize ICM using Eq while keeping the LM frozen. This alignment constrains the tokens con and E, generated by the ICM to capture comprehensive information critical for the retrieval task, providing a preliminary pretraining of the ICM's functionality and thereby facilitating subsequent fine-tuning on the retrieval tasks. The overall training process will be detailed in 3.5
Details Inspection Mechanism. Through empirical experiments, we find that the condensed tokens toon and fee are sufficient for the MLLM to determine the matching outcome between most candidates and the query. However, for some challenging candidates, these tokens are insufficient, information loss caused by token compression can impair retrieval accuracy. To address this issue, we further propose a details inspection mechanism, where the MILM autonomously identifies challenging can- didates requiring "careful examination" during the CoT process and retrieves their full token sequence Te, as supplementary information. This mechanism is simple and effective, requiring only the addi- tion of two special tokens, <inspecti on co-oonndeeeeeeoe MLLM's vocabulary. During the CoT process, the generation of <inspectionn-idee--tttee gers the inspection operation. An index idr is generated between the <inspection-index and <inspection-inde innde--------e -odex--end>> tokens, after which the uncompressed token sequence Teuss for Cıdx is appended following <inspection- inmeectioonnneexx -ende--end>>eee as supplementary information. During training, we construct synthetic training samples optimize the MLLM's ability identify challenging candidates, and design novel reward function to prevent excessive use of this mechanism, which could reduce efficiency. The details of these designs will be introduced in the following sections.
and <inspection-index-eeee
to the trig-
3.4 Model Training with Activation and Enhancement
After constructing the framework, the next key question is how to train the MLLM's CoT-based retrieval capability. To enhance effectiveness, inspired by [56, 16). we adopt two-phase training strategy: SFT for reasoning initialization followed by RL for reasoning enhancement.
Phase I: SFT with Synthetic Data. We first perform SFT on CoT datasets to activate the model's s basic reasoning abilitics. To address the challenge that existing retricval datasets do not provide CoT annotations, sample 100K instances from M-BEIR and use 1wwn2..5-L-77. dataset structured into four reasoning stages: (1) Retrieval result speculation: Based on the query, the model speculates the ideal retrieval result and generates its description.
to synthesize CoT
(2) Quick verification for negative samples: We find that most candidates can be easily identified as negative samples without requiring complex reasoning. To improve efficiency, for the quick negative identification: "A are clearly the negative samples," Here, N refers the set of clearly negative candidates identified by Qwen2.5-VIL-221 (3) Refined verification for positive samples: For the remaining candidates, a fine-grained reasoning process-including analyzing the candidate-query relationship and comparing different candidates is conducted to identify the correct positive sample. At this stage, we further instruct Qwen2.5-VL-77 candidates that are difficult to judge and add the following text the CoT: "To facilitate better judgment, the full token sequences are provided for the following candidates: : cinspection-indes www.--indxx--tttttdddaee wewpeettooonnnde--ttteeee itt.eee.starrti..iiiiiiiiiiiiiin on <ndxdx.r <Ta., >" This trains the model to identify challenging candidates and perform the details inspection mechanism described in the previous section. (4) Retrieval result generation: Based on the above CoT reasoning, the final retrieval result is generated. An example of the synthesized CoT annotation is shown Appendix More details using Qwen2.5 for this data generation are provided in Appendix Sec.A.I
include the following text into the CoT label
to select
Fig 3
5

---

Phase II: RL with Curriculum Efficiency Constraint. Through SFT on the synthesized dataset described above, the model acquires preliminary CoT reasoning capabilitiess for retrieval. We then fine-tune the model using RL to further enhance its reasoning performance and improve effi- ciency. Specifically, GRPO [ is used as the optimization objective, and the reward model in this RL framework consists of two components: the formatting reward ry and the result- efficiency reward r.. The formatting reward ry equals the CoT follows the correct structure "<think>... </think><answer>> ted as "<inspection inniiioee---tarttte ... <inspectionnindee inddi-nnded""""" Otherwise, Ty = The result-efficiency reward r, is computed as follows:
</answer>" and the details inspection operation is format-
1 (1-) Nins
r, = 1 (ê ēgt)
(5)
where è and Egt denote the predicted and ground-truth retrieval results, respectively. Nins represents the number of <inspection- wewttiiionnnn---eeeeeeeee CoT, i.e., the number of candidates for which the full token sequence is used. K is the total number of candidates, and À is a weighting coefficient. This reward captures both the accuracy of the retrieval result and the frequency of using the token-consuming full sequences, thereby promoting improvements in both model effectiveness and retrieval efficiency.
pairs in the generated
We empirically find that directly setting a large value for À leads to suboptimal results. This may be because the MLLM relatively weak in the early stages of training and thus relies more heavily on access to the full, information-rich token sequences Te, to make accurate decisions. As result, imposing a strong token efficiency constraint too early can hinder performance optimization. In contrast, the MLLM becomes more effective in later stages, increasing the constraint to emphasize efficiency becomes more appropriate. Motivated by this observation, we propose a simple yet effective curriculum learning strategy, in which À is gradually increased throughout training. Specifically, define A = iter where X; denotes the value for À in the i-th training iteration and Nater is the total number of iterations. This creates progressively stronger efficiency constraint schedule that improves model performance, as demonstrated by the experimental results in Table 7
3.5 Overall Process of Training and Inference
After introducing the proposed new methods, we then present the overall process for model training and inference in this section. As shown in Fig[[(c), the training pipeline consists of three stages: First, the MLLM is frozen, and the Information Compression Module (ICM) is pretrained on M-BEIR using the methods described 3.3 and Eq Next, we perform SIT to jointly train the ICM and MLLM on the synthesized dataset using cross-entropy loss. Finally, both the ICM and MLLM are fine-tuned via the reinforcement learning framework introduced in S.4 using 10K challenging samples from M-BEIR on which the SFT model fails to produce accurate results.
During inference, we treat the combination of the MLLM and ICM as the second-stage -ttgge model described in S.c It takes as input the top- K candidates preselected by the first-stage model & (see 3.1 for details) and produces the final retrieval result.
4 Experiments
4.1
Implementation Details
All experiments in our work are conducted on 16 A100 GPUs, using Qwen2.5-VL-33 and Qwen2.5- VL--B pretrained through the methods in LamRA (8) as the MILLM 0. As described in Sec (.5) the overall training process consists of three stages. Both the first stage (pretraining) and the second stage (SFT) are trained for epoch with a batch size of 64 and a learning rate of 1e-5. The third stage (RL) is trained for I epoch with learning rate of lc-6, using 8 rollouts and 3 0.2 in Eq During training, the vision encoder remains frozen in all stages, the ICM is fully updated, and the language model is fine-tuned using LoRA. The number of candidates K input the MLLM is set 50.
4.2 Main Results
Comparison of Effectiveness. We first evaluate the effectiveness of Retrv-RI RI on the M-BEIR test set. The results on the Recall K metric are presented in Table covering 16 sub-tasks across 8

---

Table Comparison with other methods on M-BEIR test set. R@K refers the @ metric. q¹, q.. and denote the text query, image query, text candidates and image candidates, respectively.
v'
(cl,c))
4'-
-' (s'.0")
(v'.of) -+
(1 - (1,d)
Methodis
COCO
F200K
WebQA
EDIS WebQA
VN
COCO
F200K NIGHTS
OVEN
InfoS
FIQ
CIRR
OVEN
InfoS
Avg.
R.855
R65
R
R405
R
RG55
RG:55
R475
R 10
R.655
R@5
RGB5
R
R
RG5
R6:5
CLIP-L
45
43.3
61.1
6.5
36.2
43.3
45.1
41.3
79.0
7.7
26.1
24.2
20.5
7.0
13.2
38.8
26.4
32.5
Stp
75.7
36.5
39.8
27.01
41.5
30.8
88.2
28.9
29.7
25.1
14.4
22.7
41.7
277
37.2
BIJP 40
16.4
74.4
15.9
44.9
26.8
20.3
17.2
83.2
19.9
27.4
16.1
10.2
2.3
10.6
27 4
16.6
26.8
BLIP2 29
16.7
63.8
14.0
38.6
26.9
24.5
15.0
80.0
14.2
25.4
12.2
5.5
4.4
11.8
27.3
15.8
24.8
UniIR-B1.1Ph
K0L0
22.8
224
52.2
18.0
84.7
78.7
43.1
$2.0
27.9
44.6
Qwen2.5.VLL.3B
36.0
67.8
16.1
69.5
45.2
61.7
23.3
82.3
12.0
20,9
36.7
22.3
24.3
$3.5
56.4
49.8
42.4
Qwen2.5.V1
2
40.2
71.9
20,3
71.9
49.4
64.5
29.3
84.6
19.4
25.5
42.4
32.1
25,0
55.1
60X
$4.9
46.7
Vision-R1-7B
16
41.9
75.0
22.0
70.6
51.3
35.4
85.1
25.9
48.8
44,0
29.2
57.7
66.2
VLM-RI-7B
22.5
72.3
67.9
36.2
86.3
20.9
26,4
29.9
57.4
MM-Fnhhed-78
41.0
71.3
17.1
95.9
68.8
K5.0
41.1
90.1
18.4
32.4
42.1
42.1
25.7
500
64.1
57.7
$2.7
LamRx-7B
[S
48.0
85.2
32.9
96.7
75.8
87.7
48.5
92.3
36.1
33.5
59.2
64.1
37.8
63.3
79.2
78.3
63.7
Retrv-R--3
34,4
96.7
92.7
Retrv-Ri-78
50.5
%6.7
39,3
97.4
82.1
89.5
51.4
94.1
39.8
39.5
69.9
75.6
49.5
72.3
86.6
83.7
69.2
Table 2: Comparison of effectiveness and efficiency. ITR: The ratio of the inference time required by a method compared to the time required by Retrv-R1 (K 50). GMR: The ratio of GPU memory usage required by a method compared to that of Retrv- -arrv-R--BB (K = 50).
(0'-0") (CIRR)
Methods
K =
KC 20
X 30
K
J - 50
[881
ITR 1 GMR
R
ITR
GMR
RA55 1
ITR
GMR
R85 1
ITR
GMR 1
R@85
ITR
GMR 1
Qwen2 Qwe2255-VL-7 5-VL-7B 1BB M
30.5
0.53
0.64
52.6
1,10
1,02
53.1
2.52
1.58
54.4
3.63
1.97
55.1
4.79
2/44
Vision-R--78 16
1.30
1.75
4.58
2.25
VLM-RI-7B
1.96
1.39
53.1
3.28
1.80
2.27
56.4
2.94
57.4
7.20
LamRA- Maiitt-Rnn----77 7B
62.7
0.56
0.65
64.7
1.10
1.03
65.5
2.57
1.60
63.9
3,70
1.97
66.2
4.98
2,46
Retry-RI -B
67.9
0.17
0.29
69.9
0.26
0.35
70.8
0.46
0.69
71.8
0.76
0.85
72.3
1.00
1,00
different combinations of query and candidate formats. To assess the scalability and generalization of our approach, report results for both -ttrv-R1-3B and Retrv-R1-7B. The compared methods include: (1) zero-shot general-purge purpose models, such as BLIP-2
29 and Qwen- VL [57]; (2) prior R1-
style reasoning MILMs, including Vision-R1 [16] and VLM-RI (3 and (3) MLLMs specifically
38 As shown in T Retrv-R1 consistently achieves sfate-of-the-art (SOTA) performance across all scenarios. It is important to note that: (1) Even with only 3B parameters, Retry-R R-3B still outperforms larger models like MMM--mme--7B- and LamRA-7B in most cases. (2) On challenging tasks such as (q',q') o on FIQ. Retrv-R1 maintains strong performance and outperforms prior advanced methods by an even larger margin. These results demonstrate the strong retrieval capabilities of Retrv-RI across diverse data formats, highlighting its high effectiveness and universality.
designed and trained for retrieval tasks, including MM-Embed Md and LamRA
Comparison of Efficiency. presents efficiency comparisons by using the metries of ITR and GMR, which respectively calculate the ratio of average inference time and GPU memory usage of a method compared to our Retrv-R1-7B with K 50. The evaluation is conducted on the (g q²) cf setting from the M-BEIR test set. Two types of methods are included for comparison: (1) standard
MLLMS and other R R1-style MILMs not designed for retrieval, including Qwen2.5-VI. (2. Vision-R1 [6] and VI.M-R [53]: and (2) LamRA (38) with listwise reranking (LamR -Rank-L,,,, which operates similarly to Retrv-RI by retrieving from the top-K preselected candidates. As shown in Table 2 increasing the number of input candidates (K) generally improves retrieval performance, but it also leads higher token consumption and computational cost. Moreover, too large K may exceed the memory limits of a single GPU card, which restricts other methods to smaller K values in practical applications. In contrast, benefiting from the proposed ICM and the efficiency constraint in the RL reward, Retrv-R1 achieves significantly lower consumption of both inference time and GPU memory. Also note that: (1) Under the same K, Retry-RI achieves the best performance with the lowest computational cost. (2) Even with more input candidates (K 50), Retrv-R I's inference time still remains lower than that of Qwen2.5-VL. and LamR A- A-Rnk-LL with only = 20. These results demonstrate the substantial advantages of Retrv-RI in both effectiveness and efficiency.
7

---

Table 3: Experimental results on unseen datasets. q thnls and (q' @ q²) refer to the dialog queries and multi-interleaved image-text queries, respectively.
Table 4: Experimental results on held-out tasks. indicates that training is performed on the remaining tasks, w/o any exposure the three held-out tasks.
(w + of
o yune +
(ff (a4)) -
(g'-0)) of (g'-9") (ct,e)
Methods
CIRCO GeneCIS
VisD
VIST MT-FIQ
Methods
NIGHTS R
OVEN
InfoS OVEN R85
InfoS Avg. Ri5
R05
Ree
5AP@5
1
R@5
Supervissed
CLIP-L 48
4.0
13.3
23.7
00.6
17.7
UniRR-BLIPp:
33,0
11.0
22.4
55.8
33.0
37,0
UniiR-CIIPP
12.5
16.8
26.8
39.4
UniRR-CLIPs
39
32.0
45.5
27.9
67.6
48,9
44.4
ES-V 20
24,8
18.5
54.6
10.0
19.2
MagicLens-11. 1
72
29.5
16.3
28.0
3.3
22.6
Zere-shor
MM. Embed-7e-7
7B
35.5
22.9
64.7
25.7
Qwen2 5-VL-BB
20.3
38.5
40.4
53.6
44.9
39.5
LmaRA-78 3
42.8
24.8
70.9
28.6 6
63.9
Vision-R R1-BB 16
22.9
39.8
42.9
57.4
46.5
41.9
LamRA-78' 3%
29.2
46.9
54.2
65.1
59.1
50.9
Retry-R1-78
47.8
29.2
74.2
30.9
69.6
Retrv-R R1-28
57.3
65.0
Table 5: Experimental results on multimodal se- quential recommendation task. Three subsets from the Amazon Review henchmark are evaluated. HR @ K and N@K refer to the ratio and normalized dis- counted cumulative gain. # denotes fine-tuned model.
Table 6: Experimental results of ablation study. ITR denotes the ratio of the inference time required by method compared the time required by Retrv-Ri- 3B. DIM refers to the details inspection mechanism introduced in Sec 3.3
Methods
Sports
Beauly
Toys
Methods
(6',c' + d (w'.d')
+ (d)cc OVEN
@R@ N@
@R@ Nθ
HR @ 10 N@ 10
5
ITR.
RØ51
ITR.
S3-Rec 778
3.85
2.04
6.47
3.27
7,00
3.76
B4SRe:
13
2.37
7.58
7.98
4.79
83.3
P5 [1]
4.60
3.36
6.64
4.29
7,09
5.87
Retrv-RI Satev-R1-3 3B wio ICM
68.8
7.40
84.4
7.99
VIPS 12
3.65
6.77
7.49
Beirx-R1-33 wlo
60.7
0.97
77.6
0.95
POD
81
5.76
4.19
6.88
4.43
7.42
6.10
Retrv-R1-3B feet
64.7
0.97
80.7
0.96
ICSRcc
(7
3.35
9.60
10.55
6.57
Retrv-R1-38 wio sell-aligmment
64.3
1.05
80.7
1.02
Retry-R R1 RBB3B DIM
62.3
0.87
78.
0,81
Retrv-R1-38
7.13
4.72
8.90
5.65
9.69
6.11
Retrv-R 1-3B SFT stage
63.0
1.60
79.4
1.85
Retry-R R1
9.95
6.39
12.71
7.78
12.89
7.53
Retrv-R1-3B wio RL stage
61.1
1.39
78.4
1.75
4.3 Extended Experiments to Validate Generalization
We conduct the following three experiments to further evaluate the generalization ability of Retry-R1: (1) Evaluate Retrv-RI on unseen retrieval datasets that were not used during training. (2) Assess Retrv-R1 on unseen retrieval tasks by removing certain tasks from training set and evaluating the retrained model on these held-out tasks. (3) Test Retrv-R1's I's performance on multimodal recommen- dation tasks (see Appendix [A2 for method details). The results are shown in Tables 4 and 5 respectively. Retrv-RI consistently achieves strong performance across all settings, demonstrating its robust generalization to diverse data types and tasks. Notably, shown in Table Retrv-R1 performs effectively on multimodal recommendation even without any fine-tuning, and becomes even stronger with SOTA results after task-specific fine-tuning, outperforming other task-specific models. These results highlight the great potential of Retry-R1 as a general framework for broader domains.
4.4 Ablation Study
We further conduct an ablation study to evaluate the effectiveness of the following components and designs in our framework: (1) ICM for token compression enhance efficiency; (2) Two compressed tokens teon and tyet extracted by the ICM; (3) Self-alignment for pretraining the ICM; (4) The details inspection mechanism (DIM) for handling challenging candidates; (5) SFT for activating the model's basic reasoming abilities; and (6) RL for reasoning enhancement. The results shown in Table indicate that removing any of these components significantly degrades either performance or efficiency, validating their importance. Notably, while ICM may slightly affect performance due to feature compression, it substantially improves retrieval efficiency by reducing inference time. Similarly, the details inspection mechanism introduces only minor computational overhead while significantly boosting accuracy. Additionally, we examine (7) replacing the curriculum strategy with fixed À in 5q As shown in Table this modification leads to notable drop in retrieval accuracy, further demonstrating the effectiveness of our design in reward settings. Note that we report results on the (g',g') + c² task on CIRR and the (g', q²) + (c', e task on OVEN are reported, as these are highly challenging settings that can effectively evaluate the performance of different methods under difficult scenarios. Appendix Table 10 presents the
8

---

Table 7: Experimental results of ablation study for A in Eq5i ITR denotes the ratio of the inference time required by method compared to the time required when using the curriculum efficiency constraint.
Table 8: Comparison of RAG capabiliitees on the Knowledge-based Visual Question Answering (KVQA) tasks. PR@K k and ACC denote the preci- sion and accuracy metries, respectively.
( dd) -dd d W''a") + (e' (f) CIRR
Method
OKVQA
Infoweck
E-VQA
Methods
OVEN
Reirieval (PR@:5)
RØ5t
ITRR R@51
ITR.
PreFi Pr 36
70.9
62.1
73.7
LamRA-7B 38
89.0
73.4
75.0
Curriculum Edficiency Constraint
67.9
1.00
83.3
1,00
RI-7B
Fixed 0.5
1.23
82.6
VQ24 (ACC)
Fixed
Fixed À 0.75
65.2
1.01
81.1
81.6
0.87
0.96
LamRA-7B
RA- RA-2 w/ Pr
R5
61.9
30.7
54.5
Fixed A 1.5
64.1
0.87
80.0
0.80
Retr-R1-78
66.0
31.5
58.4
averaged ablation results across all tasks M- BEIR, where each component of Retrv-RI con- tinues to show significant contributions, further demonstrating the effectiveness and generality of our design choices in the framework.
4.5 Experiments on RAG
2.0
706
100
1.0
(b) Chonges of aliggement betwoeen the genented CoTs with the step synthesized dataset.
Retrieval-Augmenner Generation (RAG) is an important application area of retrieval, which enhances the capabílities of LLMs by retrieving relevant external information as supplementary context. We evaluate the proposed Retrv-R1
(4) Chenges for the mumber detiled inspection CoT during training.
format defined
Figure Analysis results of RL fine-tuning.
within RAG settings, with results presented in Table 8 Following 33 the experiments are conducted on three Knowledge-basee trained both the retrieval and KVQA. As shown in Table 8 Retrv-RI achieves the best performance in both retrieval accuracy and VQA accuracy, demonstrating its strong effectiveness when applied in the RAG framework, highlighting the broad applicability of our method.
Visual Question Answering (KVQA) tasks, where the MLLM is jointly
4.6 Analysis of RL Fine-tuning
To gain deeper insights from our framework, we further analyze the RL fine-tuning stage in our method and observe the following patterns:
(1) Effectiveness before efficiency: In the synthetic data used for SFT, we employ a details inspection mechanism to identify and use the full features for challenging candidates. During RL stage, as shown Fig a). we observe the number of detailed inspections each generated CoT initially increases while then declines throughout the training process. This could be because, in the early phase of RL fine-tuning, the model prioritizes optimizing accuracy through more fine-grained reasoning, resulting in more thorough inspections to enhance performance. Once a high level of accuracy is reached, the model shifts focus toward improving efficiency, adapting to the stronger efficiency constraint introduced in 5 These observations highlight the effectiveness of our curriculum- based RL reward design for promoting efficiency.
(2) Increasingly flexible reasoning processes: We further evaluate the formats of the generated CoTss using (.wen.55VVL-7255 scoring their alignment with the 4-step reasoning format defined in the synthesized dataset (on a scale from 0 to 10, with higher scores indicating better alignment). As shown in Fig CoTs become progressively more flexible and diverse structure as training progresses, no longer strictly following the fixed format of the synthetic dataset. In the later stages of RL fine-tuning, we observe the emergence of some new reasoning steps and strategies automatically learned by the model, such as: (i) reffeceflection for reevaluation MLLM occasionally performs self-reflection, identifying cases where a positive sample may have been mistakenly classified as negative during quick verification, and then re- re-examines it to reach the correct conclusion. (ïi) Indicating absence of correct results - as shown in Appendix Fig when none of the top-K candidates is correct, the model can explicitly signal this at the end of the COT and suggest including more candidates for retrieval. These behaviors demonstrate the model's effective ability to learn flexible, powerful and robust reasoning capabilities through our RL framework.
as shown in Appendix Fig 4 the

---

Conclusion
This paper introduces Retrv-R1, the first RI-style MLLM framework designed for multimodal universal retrieval tasks. The framework incorporates several novel, task-tailored designs for both model architecture and training strategies, leading to substantial improvements in model effectiveness, inference efficiency and task generalization, demonstrated by extensive experiments across multiple benchmarks. We regard Retrv-R1 as a key step toward leveraging the reasoning capabilities of MLLMs to enhance downstream multimodal tasks, laying solid foundation for the development of general-purpose purpose multimodal systems with advanced reasoning abilities.
Acknowledgment
The research was partially supported by the RGC General Research Fund 11200323, NSFC/RGC JRS Project N_ C9tyU198/24. We thank Mr. Liqun Liu and Mr. Peng Shu from Tencent for their collaborations, insightful discussions, and support with computational resources in this work.
References
[1] Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. Task-aware retrieval with instructions. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3650-3675, 2023.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint UrXiv:2502.13923. 2025.
[3] Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. Zero-shot composed image retrieval with textual inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15338-15347, 2023.
[4] Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Effective conditioned and composed image retrieval combining clip-based features. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 21466-21474, 2022.
[5] Stefan Buttcher, Charles LA Clarke, and Gordon Cormack. Information retrieval: Implement- ing and evaluating search engines. Mit Press, 2016.
[6] Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming- Wei Chang. Can pre- pre-trained vision and language models answer visual information-seeking questions? Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14948-14966 2023.
[7] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes. A comparative study of foundation model post- post-training. arXiv preprint arXiv:2501.17166 2025.
[8] Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross modal retrieval. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8415 8424, 2021.
[9] W Bruce Croft, Donald Metzler, and Trevor Strohman. Search engines: Information retrieval in practice volume 520. Addison- Addison-Wesley Reading, 2010.
[10] Zheren Fu, Lei Zhang, Hou Xia, and Zhendong Mao. Linguistic-aware patch slimming frame- work for fine-grained cross-modal alignment. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 26307-26316, 2024.
[11] Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (rlp): unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM conference recommender systems, pages 299-315, 2022.
10

---

[12] Shijie Geng, Juntao Tan, Shuchang Liu, Zuohui Fu, and Yongfeng Zhang. Vip5: Towards multimodal foundation models for recommendation. In The 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
[13] Geonmo Gu, Sanghyuk Chun, Wonjae Kim, Yoohoon Kang, and Sangdoo Yun. Language-on training of zero-shot composed image retrieval. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 13225-13234, 2024.
[14] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 2025.
[15] Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming- Wei Chang. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 12065-12075, 2023.
[16] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.066 50.099449, 2025.
[17] Deyi Ji, Yuckui Yang. Haiyang Wu, Shaoping Ma, Tianrun Chen, and Lanyun Zhu. Raven: Robust advertisement video violation temporal grounding via reinforcement reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 22-31, 2025.
[18] Deyi Ji, Feng Zhao, Lanyun Zhu, Wenwei Jin, Hongtao Lu, and Jieping Ye. Discrete latent perspective learning for segmentation and detection. arXiv preprint arXiv:2406. 10475, 2024.
[19] Deyi Ji, Lanyun Zhu, Siqi Gao, Peng Xu, Hongtao Lu, Jieping Ye, and Feng Zhao. Tree-of- Unleashing the power of Ilms for enhanced large-scale table understanding. arXiv preprint urXiv:2411.085/6 2024.
[20] Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407. 12580, 2024.
[21] Huilong Jin Yingxue Zhang, Lei Shi, Shuang Zhang, Feifei Kou, Jiapeng Yang, Chuangying Zhu, and Jia Luo. An end-to-end graph attention network hashing for cross-modal retrieval. Advances in Neural Information Processing Systems, 37:2106-2126, 2024.
[22] Dongwon Kim, Namyup Kim, and Suha Kwak. Improving cross-modal retrieval with set of diverse embeddings. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23422-23431, 2023.
[23] Jae Myung Kim. A Koepke, Cordelia Schmid, and Zeynep Akata. Exposing and mitigating spurious correlations for cross- cross-modal retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2585-2599 2023.
[24] Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on Ilm generalisation and diversity. In The Twelfth International Conference on Learning Representations, 2024.
[25] Jiayi Kuang, Ying Shen, Jingyou Xie, Haohao Luo, Zhe Xu, Ronghao Li, Yinghui Li, Xianfeng Cheng, Xika Lin, and Yu Han. Natural language understanding and inference with milm in visual question answering: A survey. ACM Computing Surveys, 57(8):1-36, 2025.
[26] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pages 9579 9589, 2024.
[27] Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. Llave: Large language and vision embedding models with hardness- weiigttd contrastive learning. arXiv preprint arXiv:2503.04822 2025.

---

[28] Patrick Lewis, Ethan Perez, Aleksandra Piktus. Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. aetrieval-augmentt generation for inowledge-intenstie nlp tasks. Advances in neural information processing systees, 33:9459 9974,, 2020.
[29] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 22ip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. International Conference on Machine Learning, 2023.
[30] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision- lainuela understanding and generation. In International conference machine learning, pages 12888-12900. PMLR, 2022.
[31] Lei Li, Yongfeng Zhang, and Li Chen, Prompt distillation for efficient Ilm-based recommenda- tion. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 1348-13577 2023.
[32] Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker. Efficient visual projector for multimodal Ilm. arXiv preprint arXiv:2407.02392, 2024.
[33] Xinhang Li, Chong Chen, Xiangyu Zhao, Yong Zhang, and Chunxiao Xing. E4srec: An elegant effective efficient extensible solution of large language models for sequential recommendation. arXiv preprint arXiv:23/2.02443, 2023.
[34] Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal llms. In International Conference on Learning Representations, 2025.
[35] Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Bymn. Fine-grained late- interaction multi-modal retrieval for retrieval augmented visual question answering. Advances in Neural Information Processing Systems, 01222220222442 2023.
[36] Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Bymee Preflmr: Scaling up fine- finengrained late-interaction multi-modal retrievers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages
2024.
[37] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[38] Yikun Liu, Pingan Chen, Jiayin Cai. Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. Lamra: Large multimodal model as your advanced retrieval assistant. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
[39] Zhenghao Liu, Chenyan Xiong, Yuanhuiyi Lv, Zhiyuan Liu, and Ge Yu. Universal vision- language dense retrieval: Learning unified representation space for multi-modal retrieval. In The Eleventh International Conference on Learning Representations, 2023.
[40] Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning Ilama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2421-2425, 2024.
[41] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
[42] Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, and Vittorio Ferrari. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3113-3124, 2023.
[43] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 95:27730-27744, 2022.
12

---

[44] Renjing Pei, Jianzhuang Liu, Weimian Li, Bin Shao, Songeen Xu, Peng Dai, Juwei Lu, and Youliang Yan. Clipping: Distilling clip-based models with student base for video-language retrieval. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 18983-18992, 2023.
[45] Khoi Pham, Chuong Huynh, Ser-Nam Lim, and Abhinav Shrivastava. Composing object relations and attributes for image-texi matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14354 14363, 2024.
[46] Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: multi-modal visual question answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4542-4550, 2024.
[47] Xiuyuan Qin, Huanhuan Yuan, Pengpeng Zhao, Guanfeng Liu, Fuzhen Zhuang, and Victor S Sheng. Intent contrastive learning with cross subsequences for sequential recommendation. In Proceedings of the 17th ACM international conference web search and data mining, pages 2024.
[48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 876488763. PmLR, 2021.
[49] Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26374 26383, 2024.
[50] Aneeshan Sain, Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song. Clip for all things zero-shot Aketch-based image retrieval, fine-grained or not. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2775, 2023.
[51] Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen- Yu Lee, Kate Saenko, and Tomas Pfister. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19305-19314, 2023.
[52] Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. Prompting large language models with answer heuristies for knowledge-based visual question answering. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 14974-14983, 2023.
[53] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable r r1-style large vision-language model. arXiv preprint ar arXiv:2504. 076615, 2025.
[54] Leqi Shen, Tianxiang Hao, Tao He, Sicheng Zhao, Yifeng Zhang, Pengzhang Liu, Yongjun Bao, and Guiguang Ding. Tempme: Video temporal token merging for efficient text-video retrieval. arXiv preprint arXiv:2409.01 156, 2024.
[55] Yucheng Suo, Fan Ma, Linchao Zhu, and Yi Yang. Knowledge-enhanced dual-stream zero-shot composed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26951-26966 2024.
[56] Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint 2032222503.20752. 2025.
[57] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing language model's perception 2024.
of the world at any resolution. arXiv preprint
13

---

[58] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consisteney improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
[59] Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision, pages 387-404. Springer, 2024.
[60]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022.
[61] Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv:2503. 2025.
[62] Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. Fashion iq: A new dataset towards retrieving images by natural language feedback. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11307-11317, 2021.
[63] Shangyu Wu, Ying Xiong, Yufei Cui, Haolun Wu, Can Chen, Ye Yuan, Lianming Huang. Xue Liu, Tei-Wei Kuo, Nan Guan, et al. Retrieval-augmented generation for natural language processing: A survey. arXiv preprint arXiv:2407.131 13193, 2024.
[64] Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-ol: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10444 2024.
[65] Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. Fit and prune: Fast and training-free visual token pruning for multi-modal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 22128 221122,66,66 2025.
[66] En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jian- jian Sun, Chunrui Han, Zheng Ge, et al. Perception-rl: Pioneering perception policy with reinforcement learning. arXiv preprint urXiv:2504.07954 2025.
[67] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system scale. arXiv preprint arXiv:2503. 21ee22503..447.,.. 2025.
[68] Tianyu Yu, Yuan Yao, Haoye Zhang. Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13807-13816, 2024.
[69] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975-11986. 2023.
[70] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng. Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501. 2025.
[71] Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. RI-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503, 0eee22533..9993 12937, 2025.
[72] Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming- Ming-Wei Chang. Magiclens: self-supervised image retrieval with open-ended instructions. In Proceedings of the 4/st International Conference on Machine Learning, pages
2024.
[73] Qi Zhang, Zhen Lei, Zhaoxiang Zhang, and Stan Li. Context-aware attention network for image-text retrieval. Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 3536-3545, 2020.
14

---

[74] Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token. arXiv preprint arXiv:2501.03895, 2025.
[75] Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. Chain of preference optimization: Improving chain-of-thougee reasoning in llms. Advances in Neural Information Processing Systems, 37:333-33 356,, 2024.
[76] Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Jing Wen. Dense text retrieval based on pretrained language models: A survey, ACM Transactions Information Systems, 42(4):1-60, 2024.
[77] Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong. Megapairs: Massive data synthesis for universal multimodal retrieval. arXiv preprint arXiv:2412.14445, 2024.
[78] Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang. Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In Proceedings of the 29th ACM international conference information & knowledge management, pages 1893-1902, 2020.
[79] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vasion-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[80] Lanyun Zhu, Tianrun Chen, Deyi Ji, Peng Xu, Jieping Ye, and Jun Liu. Llafs++: Few-shot image segmentation with large language models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[81] Lanyun Zhu, Tianrun Chen, Deyi Ji, Jieping Ye, and Jun Liu. Llafs: When large language models meet few-shot segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 3065-3075, 2024.
[82] Lanyun Zhu, Tianrun Chen, Deyi Ji, Jieping Ye, and Jun Liu. Not every patch is needed: Towards a more efficient and effective backbone for video-based person re-identification. IEEE Transactions on Image Processing, 2025.
[83] Lanyun Zhu, Tianrun Chen, Qianxiong Xu, Xuanyi Liu, Deyi Ji, Haiyang Wu, De Wen Soh, and Jun Liu. Popen: Preference-based optimization and ensemble for Ivim-based reasoning segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 30231-30240, 2025.
[84] Lanyun Zhu, Tianrun Chen, Jianxiong Yin, Simon See, De Wen Soh, and Jun Liu. Replay master: Automatic sample selection and effective memory utilization for continual semantic segmentation. IEEE Trunsuctions on Pattern Analysis and Machine Intelligence, 2025.
[85] Lanyun Zhu, Deyi Ji, Tianrun Chen, Haiyang Wu, De Wen Soh, and Jun Liu. Cpef: A cross- prompt contrastive framework for referring multimodal large language models. In Forty-second International Conference Machine Learning.
[86] Lanyun Zhu, Deyi Ji, Shiping Zhu, Weihao Gan, Wei Wu, and Junjie Yan. Learning statistical texture for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12537-12546, 2021.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see Abstract and Sec
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Appendix Sec D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
Our method does not involve theory assumptions proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please Sec
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer:
[No]
Justification: The codes are not available at the moment, but they could be released after approval from the relevant authorities.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/ppp wwwppbblccgguiecuceccccceooooope Detgguddsse/CeSbbiiiiinn htt.SSbbbiiiiooooiiiiiooooii for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please see Fig
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Sec 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aaar.....cooo..ccccccaaaa
Answer: [Yes]
Justification: Yes, confirm.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There are no such problems in our work.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Please Sec
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur/ie htt.://s//.///ooeeeeeee///////// for what should or should not be described.
22

---

A More Details of Method
A.1 Details of Synthesizing Retrieval CoT Dataset
As discussed in S.c of the main paper, before using RL for further optimization. we leverage Qwen2.5-VL-727 to synthesize retrieval CoT dataset to activate the reasoning capabilities of the MILM. Given a query-candidae set {q. C1 .2,,...0 Cx] ). the process of synthesizing CoT involves the following steps:
Step 1: Retrieval result speculation: We provide Qwen2.5-VL-722 with the following prompt, instructing it to generate a speculative description of what an ideal retrieval result should look like. The generated description denoted drrs-
Prompt for retrieval result speculation
You assistant tasked with retrieving the candidate that best matches the input query. The query is: <q> Please provide detailed description what the ideal retrieval result should look like.
Quick verification for negative samples: We provide Swen2.55VL-72BB with the following prompt, instructing it to identify which candidates can easily be classified negative samples without the need for extensive step-to-step reasoning. The outputs of Qwen are the indexes of these candidates, denoted as (ns1, ns2, ..., nsN
Step 2:
Prompt for quick verification of negative samples
You assistant tasked with retrieving the candidate that best matches the input query. The query is: <p>. Candidate 1: <eι>, Candidate 2: <C2> Candidate K: <Ck> Some candidates clearly not the correct retrieval results and can therefore be easily and directly identified as negative samples, without the need for complex step-by-step reasoning (e.g.. carefully examining the candidate's details, using logical relationships for reasoning, or comparing different candidates to find the best option). Please identify these candidates, provide the reasons for classifying them such, and output their indexes. Please ensure that these candidates can indeed be easily and directly classified as negative samples.
Step Identification for challenging candidates: For candidates not classified negative samples, we instruct Qwen2.5-VL-77 VL-72B to further identify challenging candidates for which the compressed tokens are insufficient and the full features are therefore required. The generated indexes for these candidates are denoted as {ccl, cc2, .... c. The prompt used for this step is:
Prompt for identification of challenging candidates
You are an assistant tasked with retrieving the candidate that best matches the input query. The query is: <g>. Candidate 1: <Cns1>> 12, Candidate 2: <Cns2>, .... Candidate K: <Cnsk>> Among these candidates, some are particularly challenging and require a detailed examination, as they cannot be accurately assessed based solely their rough summary compressed features. Please identify these challenging candidates, provide the reasons for classifying them as such, and output their indexes. If no such challenging candidates exist, you can return empty set. Please do not output an excessive number of challenging candidates; ensure that providing detailed information for these candidates is truly necessary, while the remaining candidates can be processed using only their general, compressed features.
Step Refined verification for positive samples For candidates not elassified negative samples, we instruct Swee2.55-V-722 to generate a fine-grained reasoning process, such as analyzing the candidate-query relationship and comparing different candidates, to help identify the correct positive sample. The generated reasoning process is denoted as Teps. The prompt used in this step is:
23

---

Prompt for refined verification of positive samples
You assistant tasked with retrieving the candidate that best matches the input query. The query is: <q> Candidate I: <Cna1>, Candidate 2: <Cns2>, Candidate K: <> Please generate step-by-step, fine-grained reasoning process to identify the correct retrieval result <gt> from these candidates. The reasoning process can follow any effective flow structure, such as, but limited to, describing key details of the candidates, analyzing the candidate-quee query relationship, and comparing different candidates. Please ensure that the generated reasoning is comprehensive, logically coherent, and leads to the correct result.
Step 5: Integrating into a Complete CoT: We combine the information obtained from the previous steps into a CoT with the following template:
Template of the synthetic dataset
To assist the user identifying the correct retrieval result, I first need infer the character- istics of an ideal result. Based on the user-provided query, an ideal retrieval result should be: dyrs- Next, will analyze and evaluate the candidates. First, {nsl, ns2. nsN,,) ) are clearly the negative samples and not the correct retrieval result. For the remaining candidates, to facilitate better judgment, the full token sequences are provided for the following can- didates: <inspection inppeectio-index---e index- start>een,,. <inspection-index-ndds oe... >" Next, I will perform detailed reasoning on these candidates: Tups- Based on the reasoning above, the correct retrieval result is <gt>,
index-end><<. Goct >, <inspection-ing -indexxioonnnex-- index-
A.2 Details of Applying Retry-R1 Multimodal Recommendation
As discussed in Sec and 5 of the main paper, evaluate Retrv-R1 on out-of- multimodal recommendation task, where its outstanding performance demonstrates the strong generalization ability of Retrv-R1. In this experiment, Retrv-R1 is applied to multimodal recommendation using some simple extensions but without any modifications to the model structure. Specifically,
the ICM (SecB.3) to generate content token from each item of the human behavior data. All these tokens are concatenated to form the query in our method. The LLM's input instruction is modified to: Please recommend an item the user based on his past behavior. The user's past behavior is: <query>. The candidate items include: Candidate 1 <cq>, .... Candidate <eg>. Please output the index of the recommended item. The remaining components, such as candidate processing and result generation, follow the same strategy as the original method.
B More Experimental Results
Experiments on Text-Only Retrieval. Our Retrv-RI is designed for universal retrieval, meaning it can handle not only multimodal retrieval tasks as discussed in the main paper, but also text-only retrieval tasks with a single modality. To demonstrate this, we conduct additional experiments on the BEIR dataset, a standard text-only retrieval suite. Following the PE-Rank protocol, we rerank the top-100 retrieved passages across eight BEIR subsets: COVID, NFCorpus, Touché, DBPedia, SciFact, Signal, News, and Robust. Table presents comparison of the average NDCG@ 10 scores across these subsets. As shown, Retrv-R 1 achieves the best performance, further demonstrating its strong generalization ability and versatility across both multimodal and text-only retrieval tasks.
C Qualitative Examples
To illustrate our approach and demonstrate the effectiveness of the proposed Retrv-R1 R1 more intuitively. we present two types qualitative examples: (1) examples of the synthesized retrieval CoT data for SFT (Fig 3). and (2) examples of retrieval results along with the reasoning processes generated by
Retry-RI (Fig Fig 5 Fig and Fig 77). These results showcase how Retrv-RI achieves accurate
retrieval through inc-grained reasoning, underscoring the high efffcctiveness of the proposed method.
24

---

Table 9: Comparison on BEIR.
Table 10: Average ablation results all tasks in M-BEIR.
Methods
Average NDCG@
Methods
Avg. Score M-BEIR
ITR
BM25
0.4380
Retrv Retrv-R1-38 R1-38
65.5
1.00
monoBERT
0.4716
Retrv-RI--3 wlo ICM
66.3
6.97
monoTS
0,5136
Ketrv-R1-38 foo
59.2
0,97
Retrv R1 wlo feet
63.2
0.97
RankMistral
0,4365
Ketrv-R1-38 self-alignment
63.5
1.0
PE-Rank
0.4843
Retrv R1-3B wlo DIM
60.9
0.89
Retrv-R
stage
Retry-RI 0.5267
Retrv-R1-3B w/o RL stage
61.5
1,26
D
Discussion of Limitations
This work presents the first RI RI-style MLLM specifically designed for multimodal universal retrieval, achieving SOTA performance, high efficiency. and strong generalization ability. However, the proposed Retrv-R1 still has one limitation: the Information Compression Module (ICM) introduces slight performance drop- -0.9% on (q',9") - ct and -1.1% on (g', g")) - (c', (see Table 6 - due to information loss caused by token compression. Nonetheless, given that ICM significantly reduces inference time by over 7x, we believe this minor trade-off in accuracy is entirely acceptable and well justified. Furthermore, our proposed details inspection mechanism helps alleviate this issue by identifying challenging samples and leveraging their full tokens when necessary. In future work, we plan to explore better paradigms to achieve both stronger performance and higher efficiency.
Find nighttime photo of the same location taken from the samne angle.
Query
Aiti In
Candidate 1
Candidate 2
Candidate 3
Candidate
Candidate 5
Synthetic CoT Data:
<think> To assist the user in identifying the correct retrieval result, first need to infer the characteristics an ideal image depicts skyline Lujiazui,
match. Based on the user-provided query, an ideal retrieval result should Shanghai, captured at night, and taken from the same viewpoint as query image-specificaly, from across the Huangpu River. Next, will analyze and evaluate the candidates. First, Candidate 2 and Candidate are clearly negative samples and not the correct retrieval result. For the remaining candidates, to facilitate better judgment, the full token sequences are provided for the following candidates: <inspection ieatttiooninnex-tteeee sendex-strtt-<<inspeeeee innettiioo-nndex-eeeeee index-end-77, Next, I will perform detailed reasoning on these candidates: Candidate 1 taken night, but the angle is slightly shifted to the left. It shows the Oriental Pearl Tower from more oblique perspective, with only partial view of the Shanghai Tower on the right. In contrast, the query image is taken from more direct angle, almost head-on to the Oriental Pearl, providing full view of the Shanghai Tower. Candidate building arrangement matches the query image almost exactly, with the Oriental Pearl Tower positioned around the left third of the frame, just as in the query image. Candidate 5 is taken from high-angle aerial viewpoint, likely from drone or tall building, featuring clear downward perspective. The query image, on the other hand, is captured from low, eye-level viewpoint across the rives, affering nearly head-on view. Additionally, Candidate 5 is taken from the northwest, with the curved riverside and greenery visible in the foreground, while the query image taken from the east, with the skyline arranged horizontally, Based on the reasoning above, the correct retrieval result is Candidate 3.</think> <answer>3</answer>
night photo with a dark sky and vibrant city lights. The
Figure 3: A qualitative example of the synthesized retrieval CoT data for SFT.
25

---

Find a text that can provide additional information for this image.
Query
The Earth and the Moon form the Earth-Moon satellite system with shared center of berycenter. This barycenter is 1,700 km (1,100 mi) (about quarter of Earth's radius) beneath the Earth's surface. The Moon's orbit alightly elliptical, with en orbital eccentricity of 0.055. The semi-major axts of the geocentric lunar orbit, called the lunar distance, is appreximately 400,000 km (250,000 miles 1.28 light- seconds), comparable to going around Earth 9.5 times. The Moon makes complete orbit around Earth with respect to the fixed stars, its sidereal period, about once every 27.3 days. However, because the Earth-Moon system
Candidate 1
the same same lunar phase, completing full cycte, as seen from Earth. This synodic period synodic month is commonly known as the lunar month and is equal to the length of the solar day on the Moon. Due to tidal locking, Moon
its orbit around
Sun, takes sighttly longer, 29.5 days, to
the
makes Moon's orbital periods around Earth equal to corresponding rotation periods. This is the reason for only side the Moon, so- co-called near side, being visible from Earth.
1:1 orin- resonance. This rotation-
The Milky Way Milky Way Galaxy is the galaxy that includes the Solar System, with the name describing the galaxy's appearance from Earth: hazy band of light seen the night sky formed from stars in other arms of the galaxy, which are so far away barred spiral galaxy with D25 isophotal diameter estimated at 26.8 + 1.1 kiloparsess (87,400 3,600 light-years), but only about 1,000 light-years thick at the spiral arms (more at the bulge). Recent simulations suggest that a dark matter area, also containing some visible stars, may extend up to diameter of almost 2 million light-years (613 kpc). The Milky the Virgo Supercluster which is itself a component of the Laniakea Supercluster. It is estimated to contain 100-400
they cannot individually distinguished by
eye. The Milky Way
has several satelitt galaxies and part the Local Group galaxies, forming part of
Candidate 2
billion stars and at least that number of planets. The Solar System is located at a radius of about 27, 77,000 light-yeang spiral-shaped concentration
kpc) the Galactic Center, on inner edge the Orion Arm, and dust. The stars the innermost 10,000 liart-years form bulge and more bars that radiate from the bulge.
A terrestrial planet, tellurian planet, telluric planet, or rocky planet, is planet that is composed primarily of silicate, rocks or metals. Within the Solar System, the terrestrial planets accepted by the IAU are the inner planets closest the Sun Mercury, Venus, Earth planet, two or three planetary-mass satellites- Earth's Moon, lo, and sometimes Europa may also be considered terrestrial planets. The large rocky asteroids Pallas, and Vesta are sometimes included as well, albeit rarely. The terms "terrestrial planet" "telluric planet" are derived from Latin words Earth planets are, in terms of structure, Earth-like, Terrestrial planets generally studied by geologists, astronomers, and geophysicists, Terrestrial planets have a solid planetary surface, making them substantially different from larger gaseous planets, which are composed mastly of some combination of hydragen, helium, and water existing various physical states.
Among astronomers who the geophysical definition of
Candidate 3
Tellus), as these
Nine-Planet Alignment, also known nine-planet syzvgy, is astronomical event which celestial bodies traditionally considered planets in our solar system appear roughly aligned in the same region of the sky when viewed from Earth. This alignment does not mean the planets form perfect straight line in space, as their orbits lie on different planes Earth's perspective. Such an event is extremely rare, occurring only a few times over the course of several centuries, especially when including Pluto, whose highly tilted and elliptical orbit makes perfect alignment even more unlikely. While the alignment has no measurable gravitational physical Impact on Earth, it holds cultural and symbolic ficancee and has aften been linked myths, astrology, and pseudoscientific predictions. Some of the aligned planets may be visible to the naked eye during such an event, typically around dawn dusk, while others like Uranus, Neptune, and Pluto require telescopes to observe. The last widely referenced nine-planet alignment occurred in May 2000, though planets were visible naked eye.
distances, but rather that they cluster in same general direction from
Candidate 4
Output from Retrv-R1:
<think The user's request is to find a text that can provide additional information about the given image. To do that, I first need to understand the content of the image. The image shows the structure of the solar system, including the eight major planets and Pluto, which was formerly considered the ninth planet. The Sun, star at the center of the solar system, provides light and heat, and all planets revolve around it. The curved paths around each planet represent their orbits, illustrating how they travel around the Sun. The user is likely laoking for a text that offers a more detailed introduction to the solar system, such as explanations of its formation or the characteristics of each planet. Let me now examine each candidate. think Candidate 1, Candidate 2, and Candidate 4 are clearly not correct matches. end=TT Candidate 3 mentions the solar system and includes Mercury, Venus, Earth, and Mars, which are all consistent with the image. However, the main focus of this text is on terrestrial planets, whereas the image also includes outer planets like Jupiter and Saturn. Therefore, Candidate 3 is not the right match either. It seems that none of the candidates fully meet the user's pequest-perhaps none of them Let re-examine both the query image and the candidates. Candidate discusses the phenomenon of nine-planet alignment, which also occurs within the solar system and thus bears some relation the image. In the query image, the nine celestial bodies arranged a straight line, which may represent visual depiction of such alignment. Therefore, Candidate 4 could potentially serve as relevant supplementary information for the query image. </think> <answer<4</answer>
suitable?
Figure 4: A qualitative example of the retrieval result generated from Retry-R1.
26

---

Find image In a moment that would forever be etched into football history, Argentina captain Diego Maradona hoisted the FIFA World Cup trophy high above his head at the Arteca Stadium, surrounded by ecstatic teammates and a sea of jubilant supporters. With their 3-2 victory over West Germany, Argentina secured their second World Cup title, capping off tournament defined by brilliance, drama, and the unparalleled magic their No. 10. The iconic image of Maradona, clad in the blub and white stripes of Argentina, lifting the golden trophy under the blazing Mexican sun, became defining symbol Maradona smiled triumphantly photographers and fans rushed capture the historic toumamentt Maradona had been nothing short of sensational, delivering one of the most dominant individual performances in World Cup history. From his infarnous "Hand of God" goal to the breathtaking solo run against England
illustration
news:
teammates, 1986 World Cup. Carried on the shoulders Throughout the
Query
unmatched flair and determination. In the final, Argentina took 2-0 lead before West Germany mounted comeback level the score at 2-2. With just minutes remaining, Jorge Burruchaga scored the decisive goal after sublime pass from Maradona, sealing declared after the match, emotionally overwhelmed. "We knew could do and did with heart and soul." The image of Maradona lifting the World Cup remains powerful emblem of footballing greatness snapshot of genlus at the peak of his powers, uniting a nation and inspiring generations to come.
dubbed
"Goal
Century" Maradona led Argentina
victory and sending the stadium into raptures. "This for
Argentina," Maradona
Candidate 1
Candidate 2
Candidate
Output from Retrv-R1:
The user asks to image suitable to be used as the illustration need to summarize the main content the article. The news focuses on Argentina's 3-2 victory over West Germany 1986 World Cup Final at Mexico's Azteca Stadium, where captain Diego Maradona delivered legendary performance. His leadership throughout the tournament - highlighted by the "Hand God" and the "Goal of the Century" helped secure Argentina's second World Cup title. article emphasizes Maradona's pivotal team's triumph and his emotional display after the win, particularly the iconic moment when he lifted trophy above his head. Therefore, the ideal illustration should clearly feature Maradona as the central figure, shown lifting the World Cup trophy at Azteca Stadium, with strong emotional tone victory and celebration. Images taken outside the stadium or those that fail to depict the trophy or the post-match celebration are not appropriate. Next, let me check all candidate images. Candidate shows Maradona holding a championship trophy and may seem relevant to the given news, Let me carefully check it:
given news article. To begin,
in
However, upon closer inspection of this image, he is wearing sky-blue jersey, which Is the color of Napoli, not Argentina. The "N.R." logo on the chest confirms is a kit from Ennerre, Napoli's kit sponsor at the time. The trophy in the image is tall and flared, with sculpted figures on the base characteristics of the UEFA Cup rather than the world cup. Therefore, this image is from Napoli's UEFA Cup victory and does not match the content of the news article. Candidate is a team photo of Argentina's starting lineup. While it is related to the 1986 World Cup, it does not prominently feature Maradona, nor does it capture the emotional post-victory moment described in the news. Candidate 3 shows the famous "Hand of God" moment. Although this is one of the most iconic scenes from the 1986 World Cup and is briefly mentioned in the article, it is not the main focus of the news. The image does not convey the celebration or culmination of the World Cup victory. In conclusion, while all three candidate images are related to Maradona, none of them captures the specific scene of him lifting the World Cup trophy in celebration, which is central to the given news. The most suitable Illustration would be photo of Maradona at the Azteca Stadium, wearing the Argentina jersey, lifting the FIFA World Cup trophy high above his head while being carried on the shoulders of his teammates, with backdrop of a cheering crowd of tens of thousands. I recommend the user submit additional candidate images for further evaluation. </think> <answerr\oon<//nsserr
Figure 5: A qualitative example of the retrieval result generated from Retry-R1.
27

---

Find a text that explains the formation mechanism of this type of cloud.
Query
Cumulanimbus clouds form through strong atmospheric convection. When warm, moist air near the Earth's surface rises due to solar heating or lifting mechanism (such as a cold front or mountain slope), it coals and condenses form clouds. that further fuels the updraft. As the cloud grows vertically, it can reach the of the troposphere, where the updraft spreads out, forming an anvil-shaped top. This mature cumulonimbus cloud can produce severe weather such thunderstorms, heavy rain, hail, strong winds, even tornadoes.
atmosphere is unstable, the rising air continues ascend, releasing latent
Candidate
Towering cumulonimbus clouds are typically accompanied by smaller cumulus clouds. cumulonimbus may extend several kilometres (miles] across be small several tens of metres (yards) across and accupy low to upper altitudes within the tropasphere- formed at altitude from approximately 200 to 4,000 m [700 to 10,000 ft). Normal peaks usually reach to as much as 12,000 m (39,000 ft), with unusually high ones typically topping out around 20,000 (66,000 and extreme instances claimed Well-developed cumu lonimbus clouds are characterized by flat, anvil shaped (anvil dome), caused wind shear or inversion at the equili level near the trogopause. The shelf of the anvil may precede the main cloud's vertical component for many kilometres (miles) parcels surpass the equilibrium level [due momentum) and form overshooting maximum parcel level. When vertically developed, this largest of all clouds usually extends through all three cloud regions. Even the smallest cumulonimbus cloud dwarrs its neighbors comparison.
21,000 (69,000 ft]
Candidate
accompanied by lightning. Occasionally, rising culminating the
Stratus clouds form when a large, moist air mass is gradually lifted over a wide area, typically due to gentle atmospheric processes
warm air gliding over cooler surface radiative coaling the ground. the air rises slowly and cools, the moisture condenses into uniform, flat cloud layer. Unlike cumulonimbus clouds, stratus clouds do not involve strong vertical convection and are usually associated with overcast skies, light drizzle, or mist rather than intense weather events.
Candidate 3
Altocumulus (from Latin altus 'high' and cumulus 'heaped') middle-altitude cloud genus that belongs mainly to the stratocumuliform physical category, characterized by globular masses or rolls in layers or patches the individual elements being larger and darker than those of cirrocumulus and smaller than those stratocumulus. However, layers become tufted appearance clouds become more purely cumuliform in structure. Like other cumuliform and stratocumulifomm clouds, altocumulus signifies convection. A sheet of partially conjoined altocumulus perlucidus is sometimes found preceding weakening warm front, where altostratus, Altocumulus also commonly found between the
to increased airmass instability,
the altocumulus
Candidate
is starting to fragment, resulting patches
altocumulus perlucidus between the and cold fronts in a depression, although this is aften hidden by lower clouds.
Output from Retrv-R1:
<think The user asks me to find a text that explains the formation mechanism of this type of cloud. To begin, need to identify what type cloud shown in image. From the image, we can observe that the cloud exhibits significant vertical development, stretching from the ground all the way to the top of the troposphere. Its top extends outward an anvil shape, resembling hammer or an iron anvil. The lower part the cloud appears dense and dark, while the top is illuminated by sunlight, giving it a golden hue. These visual features strongly indicate that the cloud the image is a cumulonimbus cloud. Therefore, my task is to locate text that can explain the formation mechanism of cumulonimbus clouds. is important that the text is specifically about cumulonimbus clouds-now any other type- and that focuses primarily on the mechanism of formation, rather than on other aspects such as appearance or consequences. Let me examine the candidates. Candidate 3 and Candidate are clearly not suitable, as their content pertains to other types of clouds. Now let me check Candidate and Candidate Candidate 1: <inspection-indes start>1<<nsse sinpeetiiiecctionddeeeeeeeee end>T This text provides a detailed explanation of the entire process-from warm, moist air rising to the development of cumulonimbus clouds-which aligns well with the user's request. Although the final sentence of the text describes potential weather effects caused by cumulonimbus clouds, which falls outside the user's query, that part is minimal and occupies only small portion of the text. Candidate 2: <inspection irrttziinseetioonindeeeee
This text mainly discusses the altitude and structural features of cumulonimbus clouds. While it briefly mentions their formation, that entirely focused on describing the formation mechanism cumulonimbus clouds. me revisit Candidate and Candidate they occupies the majority of Candidate 1, it is the most suitable retrieval result. </think> <answer>1</answer>
main focus of the passage. Candidate and Candidate
relevant
query. Overall, since the explanation of the formation mechanism
Figure 6: A qualitative example of the retrieval result generated from Retry-R1.
28

---

Find image with different animal but the style.
Query
Candidate
Candidate
Candidate
Candidate
Candidate 5
Output from Retrv-R1:
<think> The user's request is to find image with different animal but in the same style as the input query Image. accomplish this task, first need to identify the animal in the query image as well as the image's visual style. the query image, cat is sitting the image
patch of land where golden and light green grass intersect. Therefore, the animal The image rendered hand-drawn style with smooth color transitions, low contrast, and soft, low-saturation tones. The background resembles watercolor picture book rather than realistic setting, giving it the appearance of retro Japanese animation style. Given this analysis, need to find image that does not contain cat but features another kind of animal, and the style should also match the retro Japanese animation aesthetic. I will then analyze all the candidate images to identify the one that best meets these criteria. First, Candidate Candidate and Candidate clearly not the correct results. They either contain cats, do not include animals, have visual style that significantly differs from the query image. Next, turn to deeper analysis of Candidate 2 and Candidate 5. First, Candidate seems promising. shares the cartoon-like style and with different animal panda. Let check it carefully:
find that Candidate 5 includes urban elements such city streets, buildings, and traffic lights, evoking modern, countryside elements as open fields, grass, blue with white clouds, and ralling hills. Moreaver, Candidate features higher contrast balder colors, unlike the soft, low-saturation tones leans toward American Candidate the other hand, shares with match. </thinko <a/swer22//nswwer>
atmosphere. contrast, query image depicts
query image. Overall, Candidate style, whereas the query image clearly the Japanese anime tradition. setting, soft tones, and saturation Japanese animation style and contains different animal dog, making it
query image.
aesthetic
Figure 7: A qualitative example of the retrieval result generated from Retry-R1.
29


Reviews:
[{"review_id": "3GahV1YonG", "reviewer": "Reviewer_x1gL", "summary": "This paper studies the task of multi-modal retrieval I.e. identifying relevant documents for a given query from a given set of documents, where both document and query can contain multi-modal entities like text, image, etc. RL-based training has appeared as a powerful way to improve reasoning abilities of foundation models (e.g. Deepseek-R1), this paper sets out to incorporate such methods in improving multi-modal retrieval using Multi-modal Large Language Models (MLLM). It identifies two challenges in naively applying RL-based training for multi-modal retrieval - (1) instability in training, (2) extensive computation costs due to inherent long-context nature of putting multiple documents in the context of the model. To mitigate (1), the authors propose a two-stage approach where they first perform SFT on a synthetically curated reasoning-based SFT data using a bigger MLLM, and then train further using RL (GRPO) on challenging inputs. To mitigate (2), the authors propose an information compression model which represents each document with just two tokens while giving the retrieving MLLM ability to expand any document if required. The performance of the proposed approach (Retrv-R1) is evaluated on M-BEIR dataset showing significant quality and efficiency improvements over existing methods.", "questions": "1. In the proposed approach it seems there is no specific design component to make it specialized for multimodal other than the base model being a VLM, this is a good thing by its own but if this is the case, then why not evaluate the validity of the approach on text retrieval tasks (such as BEIR[1], MTEB [2], etc) which have more extensive baselines and comparison points against reasoning based retrieval? Is there anything which makes the proposed approach more applicable on multimodal datasets?\n2. In table 6, why are the results only reported for 2 datasets in M-BEIR instead of the average over all datasets as I expected reading through this section? it'll give more confidence in the ablation results when looking at the average results.\n3. The approach is mainly solving the \"second-stage\" retrieval or re-ranking task, so perhaps calling it a retrieval framework is a bit misleading, since the term \"retrieval\" usually corresponds to retrieving from a big corpus?\n\n---\n\n1. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models\n2. MTEB: Massive Text Embedding Benchmark", "limitations": "The limitation of slight drop in performance with ICM is mentioned but no societal impact is discussed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strengths\n- The proposed approach gets significant improvements over baselines on M-BEIR dataset\n- The experiments section is reasonably thorough given the setting of multi-modal retrieval, with experiments on M-BEIR, multi-modal recommendation, and RAG; baselines seem appropriate.\n- To the best of my knowledge the proposed approach presents some novel techniques like employing ICM with expansion tool for efficient context processing.\n\n### Weaknesses\n- Some evaluation and task choices need further justification (see questions section below)\n- The proposed approach stacks up multiple engineered techniques which make the overall approach requiring a careful setup, for example, it is not clear the 4 step synthetic dataset curation is something that transfers well to general retrieval process. Moreover, it also makes the proposed approach not entirely comparable to existing baselines as the training data is now a teacher augmented version of it. \n- Writing needs improvement: equations in section 3.3 are hard to parse just by reading them and the descriptions around it, the notations (like $K_{T_{C_k}}, K_{R_{q, {c_k}}$) are not clearly explained and until one makes some assumptions based on the context it's hard to disambiguate. Moreover, figure 1 is very dense, it can benefit from a clearer presentation.", "was_revised": true, "final_justification": "The proposed method is a significant contribution to reasoning based retrieval, the approach is sound and empirically well justified.", "rebuttal": "Thank you very much for your time in reviewing our paper and for your valuable comments! We are sincerely encouraged by your recognition that our approach gets significant improvements, the experiments are thorough, and the approach has novelty. For your questions and concerns, we are happy to provide the following responses:\n\n### **Q1: Transfer ability of the 4 step synthetic dataset curation**\n\nA1: Thank you for highlighting this important issue! \n\n* In Table 1 of the main paper, we report results on 8 different retrieval tasks, where our method consistently achieves the best performance on all tasks, demonstrating its strong ability to transfer to general retrieval processes in different scenarios. \n* Moreover, our experimental results show that removing the 4-step synthetic dataset leads to performance drops across all tasks, indicating that this strategy is beneficial for a wide range of different retrieval scenarios and highlighting its strong generalization capability. \n* Additionally, kindly note that the SFT stage based on synthetic data serves only as an initial training phase. The subsequent RL stage allows the MLLM to explore more flexible retrieval processes, which can vary and self-adapt to different cases, as discussed in Section 4.6(2) of the main paper.\n\n### **Q2: The comparability with baseline methods**\n\nA2: Thank you for this valuable comment! \n\n* Although the training data is now a teacher augmented version, **the augmented data is synthesized only from the M-BEIR training set, without incorporating any additional datasets or manual annotations**. The M-BEIR dataset is also widely used for training in other retrieval MLLMs, such as LamRA and MM-Embed. \n* **Moreover, even without using the augmented data for SFT, our Retrv-R1-7B can achieve an average score of 65.9 on the M-BEIR test set, still outperforming all compared methods in main paper's Table 1 (e.g., LamRA-7B with 63.7).** This demonstrates that the superiority of our approach is NOT solely attributed to the SFT stage based on augmented synthetic data, but also to our effective designs in many other key aspects, such as the ICM, the self-alignment pretraining strategy, and the reinforcement learning mechanism. \n\nWe will incorporate these results and discussions into the revised paper. Thank you very much!\n\n### **Q3: Writing and figures**\n\nA3: Thank you very much for your valuable suggestion, and we sincerely apologize for any confusion caused by our unclear explanation. \n\n* Regarding the writing, $K_{T_{c_k}}$ denotes the key features in the attention mechanism derived from the $k$-th candidate’s token sequence $T_{c_k}$. $K_{R_{q, c_k}}$ represents the key features in the attention mechanism derived from $R_{q, c_k}$, where $R_{q, c_k}$ is the relationship feature between the query $q$ and the $k$-th candidate $c_k$, extracted by the method described in the right part of Eq. 3. We will carefully revise Section 3.3 based on your comments to improve writing clarity, and update the text following Eq. 2 and Eq. 3 in the main paper to clearly define the meaning of each symbol used in the formulas.\n* Regarding the figures, to avoid excessive visual density and enhance clarity, we will revise and divide Figure 1 in the main paper into several sub-figures, organized by different components, and place them on the same pages as the corresponding text sections to enhance readability and avoid page-turning. As PDF files or external links are not permitted during the rebuttal stage, the revised figure will be included in the next version of our paper. Thank you very much! \n\n### **Q4: Evaluation on text retrieval tasks**\n\nA4: Our Retrv-R1 is designed for universal retrieval, which means that **it can handle different retrieval tasks, including text-only, image-only, and multimodal retrieval, within a unified framework**. This capability is empirically validated by the results in Table 1 of the main paper, which evaluate 8 diverse retrieval tasks with different types of queries and candidates. Retrv-R1 achieves SOTA performance across all task types, demonstrating its broad applicability and effectiveness. We fully agree with your suggestion that including more text retrieval tasks would enhance the comprehensiveness of our study. Accordingly, we have conducted additional experiments on BEIR, following PE-Rank by reranking the top-100 passages on eight subsets of the BEIR benchmark: Covid, NFCorpus, Touché, DBPedia, SciFact, Signal, News, and Robust. A comparison of the average NDCG@10 scores on these subsets is presented as follows:\n\n| Method       | Avg. NDCG@10 on BEIR |\n| ------------ | :------------------: |\n| BM25         |        0.4380        |\n| monoBERT     |        0.4716        |\n| monoT5       |        0.5136        |\n| RankMistral  |        0.4365        |\n| PE-Rank      |        0.4843        |\n| **Retrv-R1** |      **0.5267**      |\n\nThese results demonstrate that Retrv-R1 achieves strong performance on text retrieval tasks, further highlighting its high generalization and versatility. Due to the time constraints of the rebuttal period, we will evaluate more text retrieval benchmarks such as MTEB and include the results in the next version of our paper. We also plan to explore more retrieval techniques specifically designed for multimodal scenarios in our future work. Thank you very much!\n\n### **Q5: Average results on M-BEIR for ablation study**\n\nA5: The $(q^i, q^t) \\to c^i$ task on CIRR and the $(q^i, q^t) \\to (c^i, c^t)$ task on OVEN are highly challenging and are thus reported by us in Table 6 of the original paper to evaluate the performance of different methods under difficult scenarios. We fully agree with your comment that presenting average results across all datasets in M-BEIR can provide greater confidence in the ablation study. These results are presented below:\n\n| Methods                        | Avg. Score on M-BEIR ↑ | ITR↓ |\n| ------------------------------ | :--------------------: | :--: |\n| Retrv-R1-3B                    |          65.5          | 1.00 |\n| Retrv-R1-3B w/o ICM            |          66.3          | 6.97 |\n| Retrv-R1-3B w/o $t_{con}$      |          59.2          | 0.97 |\n| Retrv-R1-3B w/o $t_{rel}$      |          63.2          | 0.97 |\n| Retrv-R1-3B w/o self-alignment |          63.5          | 1.02 |\n| Retrv-R1-3B w/o DIM            |          60.9          | 0.89 |\n| Retrv-R1-3B w/o SFT stage      |          62.4          | 1.52 |\n| Retrv-R1-3B w/o RL stage       |          61.5          | 1.26 |\n\nWe will update Table 6 in the revised paper to present the averaged results as shown above.\n\n### **Q6: Framework name with the term “retrieval”**\n\nA6: Thank you for the comment! We notice that many recent LLM-based retrieval methods, such as LamRA, adopt a two-stage alignment–reranking strategy, where both stages, including the coarse filtering of a large corpus through alignment and the fine-grained decision from a smaller candidate set via reranking, can contribute significantly to retrieval performance. Therefore, our focus on the reranking stage is also crucial for enhancing retrieval effectiveness. We agree with your comment that “reranking” can more accurately describe the task we focus on, and we will clarify this more clearly in the revised paper.\n\nThank you again for your valuable comments and thoughtful questions! We hope that our responses can address your concerns, and we would be happy to engage in any further discussion if you have any questions. We sincerely wish you all the best in your future life and work! :)"}, {"review_id": "hT36dLHRGL", "reviewer": "Reviewer_arHR", "summary": "The authors propose Retrv-R1, a reasoning-driven multimodal LLM framework for universal retrieval. The key ideas are (i) an Information Compression Module (ICM) that condenses each candidate into two learned tokens while preserving query–candidate relations, and (ii) a details-inspection mechanism that lets the model pull the full token sequence of only the “hard” candidates during chain-of-thought (CoT) reasoning. Experiments on M-BEIR and other benchmarks demonstrate state-of-the-art performance with significant efficiency improvements.", "questions": "1. Larger models: Have you tried scaling Retrv-R1 to≥13 B parameters or open LLaMA-3-V? If the compression scheme still yields memory wins, this would strengthen the universality claim.\n\n2. Curriculum learning necessity: The curriculum-based efficiency constraint shows mixed results across datasets. Can you provide clearer guidelines for when and how to apply this constraint for different retrieval tasks?", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nQuality: The three-stage training pipeline is methodologically rigorous, and extensive experiments across multiple benchmarks validate effectiveness. The ICM architecture and curriculum learning approach show clear engineering merit.\n\nClarity: The paper is well-structured with clear motivation and comprehensive technical details.\n\nSignificance: Addresses an important problem of applying reasoning-enhanced LLMs to retrieval tasks. The 7× efficiency improvement while maintaining SOTA performance has practical implications. \n\nOriginality: First work to systematically adapt R1-style reasoning to multimodal retrieval. \n\nWeaknesses:\n\nQuality: Limited evaluation on larger foundation models (only 3B/7B tested). The ICM introduces a small but consistent performance drop (-0.9% to -1.1%) due to information compression. \n\nOriginality: While the application is novel, core components (token compression, synthetic data generation, curriculum learning) build incrementally on existing techniques. ICM resembles recent token-merging work; curriculum reward is an incremental twist on length penalties.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time in reviewing our paper and for your valuable comments! We are sincerely encouraged by your recognition of our work and the positive score. For your questions and concerns, we are happy to provide the following responses:\n\n### **Q1: Evaluation on larger foundation models**\n\nA1: Thank you for this insightful comment! Following your suggestion, we conduct experiments based on a larger model, LLaMA-3.2-11B-Vision. As shown in the table below, Retrv-R1-11B outperforms both Retrv-R1-7B and Retrv-R1-3B, demonstrating the scalability of our approach. Due to resource constraints, we are currently unable to experiment with even larger base models, but we plan to include them once sufficient resources become available.\n\n| Method       | Avg. on M-BEIR |\n| ------------ | :------------: |\n| Retrv-R1-3B  |      65.5      |\n| Retrv-R1-7B  |      69.2      |\n| Retrv-R1-11B |      70.7      |\n\nAdditionally, the table below presents a comparison between Retrv-R1-11B and the baseline LLaMA-3.2-11B in terms of both effectiveness and efficiency, based on the ITR and GMR metrics defined in Table 2 of the main paper. The comparison is conducted on the $(q^i, q^t) \\to c^i$ task of FIQ with 50 candidates. As shown in the table, Retrv-R1-11B not only outperforms LLaMA-3.2-11B but also requires fewer computational (shown by ITR) and memory resources (shown by GMR), further demonstrating the advantages of our approach.\n\n| Method        | R@5 ↑ | ITR ↓ | GMR ↓ |\n| ------------- | :---: | :---: | :---: |\n| LLaMA-3.2-11B | 59.9  | 3.83  | 1.95  |\n| Retrv-R1-11B  | 74.3  | 1.00  | 1.00  |\n\nWe will incorporate these results into the revised paper. Thank you very much!\n\n### **Q2: Performance drop from ICM**\n\nA2: Yes, our ICM introduces a small performance drop (-0.9% to -1.1%) due to information compression. However, given that it can substantially reduce inference time by 7×, we believe this minor sacrifice in accuracy is worthwhile in practical applications that typically require a good balance between performance and efficiency. We agree that developing better token compression methods with less accuracy loss is a promising and important research direction, which will be a key focus of our future work.\n\n### **Q3: Novelty of core components**\n\nA3: Thank you for this valuable comment! We would like to kindly provide the following clarifications: \n\n1. ICM is not a simple application of existing token-merging methods, but instead **incorporates several task-specific designs tailored for retrieval**. Specifically, for _**compression**_, ICM is designed to condense each candidate’s feature sequence into two tokens: one representing the candidate’s content and the other capturing its relevance to the query, which is based on the unique characteristics of retrieval tasks. For _**training**_, a novel self-alignment strategy is proposed to pretrain ICM, ensuring that the compressed tokens can capture comprehensive information critical for the retrieval tasks. For _**post-compression processing**_, a new details inspection mechanism is further introduced to address the information loss problem caused by compression when processing challenging samples. With these innovations, we believe ICM is a novel approach that is different from existing token-merging techniques.\n2. **The format of the synthetic data is also specifically designed for the retrieval task**, consisting of four stages: retrieval result speculation, quick verification of negative samples, refined verification of positive samples, and retrieval result generation. Please refer to Page 5, Lines 197-216 of the main paper for details. Training on the synthetic dataset with these new designs helps to improve both the effectiveness and efficiency of the model during the reasoning process. Therefore, the data synthesis in our work is not a simple and trivial step, but instead a carefully crafted strategy with high effectiveness.\n3. **Although the curriculum reward is a simple strategy, it is very effective and can achieve significant performance improvements without requiring complex operations.** In fact, we have also experimented with more complicated curriculum strategies, such as dynamically adjusting $\\lambda$ in Eq.5 of the main paper based on the loss at each training step, but found that it can not outperform our simpler approach. Therefore, the simpler yet more effective method is adopted in our final framework.\n\nIn conclusion, based on the above discussions, we believe that our Retrv-R1 can provide new insights and techniques for the fields of retrieval and reasoning MLLMs. These discussions will be incorporated into our revised paper. Thank you very much!\n\n### **Q4: Curriculum learning necessity**\n\nA4: Thank you for highlighting this important issue! We would like to provide the following discussions from the perspectives of both effectiveness and efficiency:\n\n1. **For effectiveness, our curriculum method can consistently improve performance across different retrieval tasks.** This is demonstrated by the results in Table 7 of the main paper, where our method outperforms all other strategies on both tasks in terms of the R@5 metric, highlighting its strong generalizability.\n2. For efficiency, we find that the computational cost of our curriculum method is slightly higher than that of directly setting a fixed and large $\\lambda$ in Eq. 5. For example, as shown in Table 7 of the main paper, our curriculum method increases computation time by about 20% compared to using a fixed $\\lambda = 1.5$. This is because, in our curriculum strategy, $\\lambda$ is not initialized with a large value but instead begins very small in the early stages of training while gradually increasing in the later stages. As a result, more computation is required at inference since efficiency is less strictly optimized during training. However, given that our method can achieve a substantial performance gain (over 3 points on R@5), we believe this modest increase in computation is acceptable and worthwhile.\n\nWe have further extended the evaluations in Table 7 of the main paper to all 8 retrieval tasks in M-BEIR and find that the same trend consistently holds. Specifically, **the curriculum strategy improves the average score across all tasks by 1.94 while incurring only a 17.6% increase in average computation time compared to a fixed $\\lambda = 1.5$**. These results demonstrate that **our curriculum strategy can be effectively and consistently applied to different tasks to achieve a better balance between performance and efficiency, without requiring task-specific adjustments**. We will incorporate these discussions and the detailed results on all 8 tasks into the revised paper.\n\nThank you once again for your valuable comments and thoughtful questions! We hope that our responses can address your concerns, and we would be happy to engage in any further discussion if you have any questions. Thank you very much! :)"}, {"review_id": "XrBi6gv0k5", "reviewer": "Reviewer_8vrD", "summary": "The authors present Retrv-R1, a multi-modal language model designed for multi-modal retrieval. They introduce Information Compression Module (ICM) that compresses input candidates (except the ones that need to be inspected) to increase the efficiency of their method. The combined MLLM and ICM then SFTed and they apply reasoning style RL. The authors show the effectiveness of their approach on M-BEIR and study the effect of different parameters and design choices that is being made.", "questions": "1) ITR and GMR metrics heavily depend on the implementation. Can authors clarify how they have made sure that the comparisons are done based on best implementation for each method?\n2) What maximum sequence lengths authors have used for RL training? Did they need to apply masking?\n3) There is a typo in line 162. ICN --> ICM", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n - The ICM module seems to be working well and hugely impacts the effectiveness of their approach\n - The authors provide detailed ablations to provide insight into how different choice affect the results\n - Their results on M-BEIR is good\n\nWeaknesses:\n - Although the paper has a good flow, some parts are not clearly explained. For example, I found section 3.3 was hard to follow and I needed to spend some time to go back and forth between this section and figure 1\n - Although the authors show the importance of RL in their ablation studies and provide some insight in Section 4.6, Figure 2(a) results need further investigation. The fact that number of inspections goes down might indicate that it might be better to feed the model with more challenging prompts as the training progresses.", "was_revised": true, "final_justification": "The paper studies an important problem, provides novel contributions, and presents good results. The authors addressed weaknesses and questions I raised during the review process. As a result, I rate it 5 and keep the paper as accept.", "rebuttal": "Thank you very much for your time in reviewing our paper and the valuable comments! We are sincerely encouraged by your recognition that our ICM module works well, the ablation studies are detailed, and the results are good. We would like to kindly provide the following responses regarding your concerns and questions:\n\n### **Q1: Clearer Explanation**\n\nA1:  We sincerely apologize for any confusion caused by our unclear explanation and will revise the writing carefully to improve clarity. Additionally, we will divide Figure 1 in the main paper into several sub-figures, organized by different components, and place them on the same pages as the corresponding text sections to enhance readability and avoid page-turning. As PDF files or external links are not permitted during the rebuttal stage, the revised figure will be included in the next version of our paper. Thank you very much! \n\n### **Q2: Further investigation of results in Figure 2(a)**\n\nA2: Thank you for this insightful comment! To investigate the possibility you raised, we conduct an experiment in which the model, during the later stages of training, is trained on increasingly difficult tasks from M-BEIR with more challenging prompts. Under this setting, we observe that the pattern in Figure 2(a) still holds: the number of detailed inspections in each generated CoT initially increases and then decreases as training progresses, although the decline in later stages is slightly less steep than in the original curve. However, when the proposed curriculum strategy is removed and $\\lambda$ in Eq. 5 is fixed to 1, the pattern disappears. This indicates that **the decreasing trend in inspection count is NOT affected by changes in prompt difficulty under different training schedules, but is instead more closely associated with the gradually stronger efficiency constraint introduced by our curriculum strategy**. Moreover, the performance of this alternative method, which uses progressively harder samples, is nearly identical to that of our original approach without yielding additional improvements, further demonstrating the soundness of our original training method. We will include these results and discussions in the revised experimental section and provide a deeper analysis of the pattern observed in Figure 2(a).\n\n### **Q3: Implementation for each method**\n\nA3: In our work, we ensure fair comparisons by using a consistent codebase, the same version of the Transformers library, and identical hardware for different experiments. Additionally, the evaluations for the ITR and GMR metrics are conducted using the open-source vLLM library, which can automatically optimize inference for each method based on the current hardware conditions, ensuring that each method’s inference implementation runs under the best possible settings. We will illustrate this more explicitly in the revised paper.\n\n### **Q4: Details of  RL training**\n\nA4: The maximum sequence length used for RL training is set to 3072. Masking is applied in the rare cases where this limit is exceeded. We will include these details in the implementation section of the revised paper.\n\n### **Q5: Typos**\n\nA5: Thank you for indicating this issue! We will carefully check our paper and fix the typos.\n\nThank you once again for your valuable comments. We hope that our responses can address your concerns, and we sincerely wish you all the best in your future life and work :)"}, {"review_id": "EdV1VXiZBG", "reviewer": "Reviewer_j4TE", "summary": "This paper proposes a cross-modal retrieval re-ranking method Retrv-R1 based on the Multimodal Large Language Model, and uses CoT and reinforcement learning for training. At the same time, Retrv-R1 introduces an information compression module (ICM)  to improve computational efficiency and optimize the training stability. The evaluation results on M-BEIR show that the average Recall of Retrv-R1 under various retrieval settings has been significantly improved compared with existing methods, and its re-ranking accuracy and efficiency are significantly better than Qwen2.5-VL-7B, Vision-R1-7B, VLM-R1-7B and LamRA-7B under the same settings.", "questions": "Overall, I have a positive attitude towards the methods and contributions of this paper, but I have some concerns, which have been presented in the disadvantages, mainly in three aspects: 1) the innovation of ICM; 2) the fairness of method comparison; 3) the ambiguity of the results. I hope the author can solve my confusion, and I will readjust my score based on this and the comprehensive opinions of other reviewers.", "limitations": "The training samples in the cross-modal retrieval datasets may have racial bias.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper is the first to use the powerful understanding ability of the Multimodal Large Language Model and the Chain-of-Thought derivation ability of reinforcement learning to re-rank multimodal retrieval results, which is meaningful for the fine sorting of difficult image-text samples.\n2. The effectiveness of Retrv-R1 is verified by the results of M-BEIR, a series of generalization ability test datasets, and ablation experiments.\n3. The writing of this paper is clear and easy to understand, and the presentation quality is high.\n\nWeaknesses:\n1. The meaning and calculation method of the content token and relationship token in ICM have a high degree of overlap with LLaMA-VID, which reduces the innovation of ICM module.\n2. The SFT stage is equivalent to perform knowledge distillation from Qwen2.5-VL-72B. Will this cause unfairness in comparison with other methods?\n3. I notice that the performance of BLIP-2 on the COCO dataset in Table 1 is weaker than BLIP, which seems to contradict the conclusion of the original paper. I hope the authors can explain this.", "was_revised": true, "final_justification": "After the rebuttal phase, my concerns were resolved. In addition, I also carefully read the comments of the other reviewers.\nOverall, I believe that the strengths of this paper outweigh its weaknesses, so I ultimately decide to maintain borderline accept.", "rebuttal": "Thank you very much for taking the time to review our paper and for your valuable comments! We are sincerely encouraged by your positive attitude towards the methods and contributions of our paper. In response to your concerns and questions, we would like to kindly provide the following clarifications:\n\n### **Q1: Innovation of ICM compared to LLaMA-VID**\n\nA1: Thank you for this insightful comment! We would like to kindly clarify that, although LLaMA-VID also condenses a sequence into a few tokens, our ICM differs from it significantly in the following four aspects:\n\n1. The **calculation methods** for compressed tokens are different. LLaMA-VID employs a simple averaging operation to produce content tokens, which, while straightforward, may lead to too much loss of fine-grained information. In contrast, our ICM utilizes a learnable query to attend to candidate tokens via the attention mechanism, allowing the model to better preserve critical features through the more fine-grained compression process. To validate the effectiveness of our approach, we also implement LLaMA-VID’s method for content token generation and observe a 2.11% drop in the average score on the M-BEIR test set, demonstrating the advantage of our design.\n2. The **meanings of the compressed tokens** are different. Although both LLaMA-VID and ICM extract relationship information, their underlying semantics are distinct. In LLaMA-VID, the relationship captures context aggregation between the *user instruction and each video frame*. In contrast, our ICM encodes the correlation between *each candidate and the query*, which is specifically tailored to retrieval tasks and thus differs from LLaMA-VID.\n3. The **training strategies** are different. Our method introduces a self-alignment approach (Pages 4–5, Lines 162–177) to pretrain the ICM, ensuring that the compressed tokens generated by ICM effectively capture comprehensive information critical for the retrieval task. This strategy is novelly introduced by us, not employed in LLaMA-VID, and its effectiveness is demonstrated by the results in Table 6 of the main paper.\n4. The **subsequent operations** after compression are different. LLaMA-VID only performs token compression, but does not address the resulting problem of information loss. In contrast, our method mitigates this issue through a newly proposed detail inspection mechanism (Page 5, Lines 178–192), which enables the full token sequences to be leveraged for the automatically identified challenging cases during the reasoning process. This distinguishes our approach from LLaMA-VID and allows for more effective handling of difficult examples.\n\nIn conclusion, due to the four differences outlined above, we believe our method is novel compared to LLaMA-VID. We will incorporate these discussions into the revised paper. Thank you very much!\n\n### **Q2: Fairness of method comparison**\n\nA2:  Thank you for this insightful comment! We would like to kindly clarify the following points:\n\n1. The SFT data is synthesized from only the M-BEIR training set, without incorporating any additional data. This dataset (M-BEIR) is also widely used for training in other retrieval MLLMs, such as LamRA and MM-Embed.\n2. **Even without the SFT stage, our Retrv-R1-7B can achieve an average score of 65.9 on the M-BEIR test set, still outperforming all compared methods in main paper's Table 1 (e.g., LamRA-7B with 63.7).** This demonstrates that the superiority of our approach is NOT solely attributed to the SFT stage based on synthetic data, but also to our effective designs in many other key aspects, such as the ICM, the self-alignment pretraining strategy, and the reinforcement learning mechanism. \n3. To enable a fairer comparison, we further conduct experiments by also using Qwen2.5-VL-72B to distill previous methods such as LamRA and MM-Embed. The distilled LamRA-7B achieves an average score of 64.8 on M-BEIR, which is still lower than our Retrv-R1-7B’s score of 69.2, demonstrating that **our method retains a clear advantage compared to other approaches under the same distillation setting**. The distillation experiment based on MM-Embed is still in progress and will be included in the revised experimental section.\n\nWe will incorporate all of the above results and discussions into the revised paper. Thank you very much!\n\n### **Q3: The ambiguity of the results**\n\nA3: Thank you for highlighting this important issue! The results for BLIP and BLIP2 in our paper are directly taken from previous papers [a, b], and upon careful re-evaluation, we have confirmed their correctness. We think **the ambiguity you mentioned may arise from the differences in evaluation protocols**. Specifically, the “BLIP” and “BLIP2” results reported in Table 1 of our main paper refer to the zero-shot performance **_without fine-tuning_**. In contrast, “UniIR-BLIP$_{\\rm FF}$” in Table 1 refers to BLIP's results trained using UniIR’s method with **_multi-task fine-tuning_** on M-BEIR, while the results in the original BLIP2 paper are based on **_single-task fine-tuning_** on COCO. For clarity, we provide a comparison of these three different evaluation settings (zero-shot, multi-task fine-tuning, and single-task fine-tuning) on the COCO $q^i →c^t$ task in the table below:\n\n| Training Method                   | Blip | Blip2 |\n| --------------------------------- | :--: | :---: |\n| w/o fine-tuning (zero-shot)          | 83.2 | 80.0  |\n| mult-task fine-tuning (on M-BEIR) | 89.9 | 91.6  |\n| single-task fine-tuning (on COCO) | 95.4 | 96.0  |\n\nAs shown in the table, without fine-tuning, BLIP achieves better zero-shot performance than BLIP2 on COCO. However, after fine-tuning, BLIP2 outperforms BLIP, consistent with the conclusions reported in the original BLIP2 paper. One possible explanation is that BLIP’s image encoder is trained on 129M image-text pairs including COCO, whereas BLIP2 uses a fixed CLIP encoder pretrained on a much larger dataset of 500M pairs. As a result, BLIP2 exhibits stronger foundation capabilities, while BLIP is more fitted to COCO. Consequently, in the zero-shot setting without fine-tuning, BLIP performs better on COCO due to prior exposure, while BLIP2 surpasses BLIP after fine-tuning thanks to its more powerful pretrained foundation ability. We think this is an interesting phenomenon worthy of further in-depth exploration and will make it a focus of our future work.\n\nThank you once again for your valuable comments and thoughtful questions! We hope that our responses can address your concerns, and we would be happy to engage in any further discussion if you have any questions. Thank you very much :)\n\n[a] LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant, CVPR2025\n\n[b] UniIR: Training and Benchmarking Universal Multimodal Information Retrievers, ECCV2024"}]

Author rebuttal:

Meta-review: This paper proposes Retrv-R1, a reasoning-driven multimodal LLM retrieval framework that combines an Information Compression Module (ICM) with a details-inspection mechanism and a two-stage training process (synthetic CoT supervision + curriculum RL). The approach achieves  state-of-the-art performance across multimodal retrieval benchmarks.

Reviewers agreed the paper addresses an important problem, with solid methodology, and thorough experiments. Concerns about clarity, fairness of synthetic data, and limited evaluations were largely resolved by the rebuttal. Some novelty is incremental, and clarity in technical sections needs improvement, but the overall integration is meaningful and impactful.

One notable limitation is that inference time is reported only as ITR—the ratio of a method’s inference time relative to Retrv-R1-7B (K = 50)—without providing actual wall-clock measurements. Although the proposed method substantially improves retrieval recall, it may run much slower and thus be less practical than purely embedding-based approaches, especially on commodity hardware. **The authors are highly encouraged to include real timing results in the camera-ready version to support broader adoption and future research.**
Decision: Accept (poster)