MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains
Authors: Xuying Ning
Dongqi Fu
Tianxin Wei
Mengting Ai
Jiaru Zou
Ting-Wei Li
Hanghang Tong
Yada Zhu
Hendrik Hamann
Jingrui He
Venue: ICLR 2026 Oral (2026)
Keywords: Multimodal
RAG
Vision-Language
Agent
Benchmark
Primary area: foundation or frontier models, including LLMs

Abstract: With the increasing demand for step-wise, cross-modal, and knowledge-grounded reasoning, multimodal large language models (MLLMs) are evolving beyond the traditional fixed retrieve-then-generate paradigm toward more sophisticated agentic multimodal retrieval-augmented generation (MM-RAG). Existing benchmarks, however, mainly focus on simplified QA with short retrieval chains, leaving adaptive planning and multimodal reasoning underexplored. We present MC-Search, the first benchmark for agentic MM-RAG with long, step-wise annotated reasoning chains spanning five representative reasoning structures. Each example specifies sub-questions, retrieval modalities, supporting facts, and intermediate answers, with fidelity ensured by HAVE (Hop-wise Attribution and Verification of Evidence), resulting in 3,333 high-quality examples averaging 3.7 hops. Beyond answer accuracy, MC-Search introduces new process-level metrics for reasoning quality, stepwise retrieval and planning accuracy. By developing a unified agentic MM-RAG pipeline, we benchmark six leading MLLMs and reveal systematic issues such as over- and under-retrieval and modality-misaligned planning. Finally, we introduce Search-Align, a process-supervised fine-tuning framework leveraging verified reasoning chains, showing that our data not only enables faithful evaluation but also improves planning and retrieval fidelity in open-source MLLMs.

Published as a conference paper at ICLR 2026
MC-SEARCH: EVALUATING AND ENHANCING MULTI- MODAL AGENTIC SEARCH WITH STRUCTURED LONG REASONING CHAINS
Xuying Ning"", Dongqi Fu Tianxin Wei"", Mengting Ai*, Jiaru Zou', Ting-Wet Wei Li', Hanghang Tong', Yada Zhu Hendrik Hamann', Jingrui He University of Illinois Urbana- Chhama-Champpign, 2Meta, 3BBM Research, 4Stony Brook University "Equal contribution. CCorresponding author.
[xuyingn2, jingrui @illinniiiiiiiii edu
projeppoosetrrrrppjjeeett github.
ABSTRACT
With the increasing demand for step-wise, cross-modal, and knowledge-grounded reasoning, multimodal large language models (MLLMs) are evolving beyond the traditional fixed retrieve-then siaeeeee-teengggerrrate generate paradigm toward more sophisticated agentic multimodal retrieval- -uggmened generation (MM-RAG) Existing benchmarks, however, mainly focus simplified QA with short retrieval chains, leaving adap- tive planning and multimodal reasoning underexplored. We present MC-SEAR RCHH, the first benchmark for agentic MM-RAG with long, step-wise annotated reasoning chains spanning five representative reasoning structures. Each example specifies sub-questions, retrieval modalities, supporting facts, and intermediate answers, with fidelity ensured by HAVE (Hop-wise Attribution and Verification of Ev- idence), resulting in 3,333 high-quality examples averaging 3.7 hops. Beyond answer accuracy, MC-SEARCH introduces new process-level metrics for reasoning quality, stepwise retrieval and planning accuracy. By developing unified agentic MM-RAG pipeline, we benchmark six leading MLLMS and reveal systematic issues such as over- and under-retrieval and modality-misalige we introduce SEARCH-ALIGN, a process-supervised fine-tuning framework lever- aging verified reasoning chains, showing that our data not only enables faithful evaluation but also improves planning and retrieval fidelity in open-source MLLMs.
planning. Finally,
INTRODUCTION
Multimodal large language models (MLLMs) have rapidly advanced in their reasoning abilities (Wang et al., 2025b; Huang et al., 2025b; Li et al., 2024), moving beyond text-only inputs to interleaved cross-modal contexts. To ensure factuality and robustness, Aarreetal-uummenteed Generation (RAG) has emerged as a key paradigm for grounding model outputs in external textual evidence (Gao al., 2023; Zhao et al., 2024; Fan al., 2024). Building on this, Multimodal RAG (MM-RAG) equips MLLMs with visual seeking and extends knowledge grounding from text-only evidence to multimodal contexts, thereby facilitating cross-modal reasoning for seowledge-nntensin queries.
In practice, queries for MILMS are often ambiguous and complex (Figure 1). They go far beyond the shallow cases solvable with fixed minimal retrieval and instead require multi-step, cross-modal, and knowledge-intensive reasoning. As MLLMs advance toward long reasoning models (Wang et al., 2024), they are increasingly posi- tioned backbones for supporting such problem-solving needs. Re- alizing this potential, however, demands richer agentic behaviors (Li et al., 2025abb), such as iterative task decomposition, adaptive cross- modal retrieval, and multimodal evidence integration. This shift calls for moving beyond the fixed retrieve-then-geneee classic RAG toward multimodal agentic search-enhanced reasoning (multimodal agentic RAG) (Wang et al., 2025a; Li et al., 2025a; Huang et al., 2025a; Jin et al., 2025; Zheng et al., 2025b; Chen al., 2025). Accordingly, new benchmarks and evaluation pipelines are needed faithfully capture and evaluate these capabilities.
spactal esaltied the len inceraal
quraker Appioodd
bengg Sencival Dleeage Vilerntt Pepiic
(Thoagat The compti's sere -aa-
paradigm of
Figure Example query requir- ing multimodal agentic search with long reasoning chain.

---

Published as a conference paper at ICLR 2026
Table Left: Comparison existing multimodal retrieval retrivall-augmente QA datasets. Right: Distribution of the five reasoning topologies in SEARCH, with outer rings illustrating hop diversity (2-5 hops).
Benchmark
Knowledge Modality
Typical Hops
Long Chain Hops)
Stepwise Annotation
Reasoning Topology
OK-VQA (Marino et 2019)
Text
Hop
VIQUAR (l.emnrr a1, 2022]
Hop
WebQA (Chang et al., 2022)
TexVCaption
52 Hops
Heap
austaciaal
InfeKook (Chen al. 2023)
Text
Hop
MMSaarch (liagg all, 2024)
Tex//maag
Hop
X
Dyn-VQA (1 al. 2025b)
Text/mmage
52 Hops
x
Tert-Baly
MRAG (Hu 2025)
Image
Hop
X Hap
Chals
M"RAG (1.in et al,, 2025c)
Tex/tmage
Hop
MC-SEARCH
Text/Image
24 Hops
Although number of recent benchmarks (Hu et al., 2025; Jiang al., 2024; Liu et al., 2025c; Li et al., 2025b) have provided valuable evaluations of MM-RAG, they primarily focus on straightforward visual evidence: seeking questions with only 1-2 retrieval steps, thus falling short of examining the adaptive search-enhanced reasoning behaviors required in practice. Specifically, current datasets remain limited in three key respects: (i) most adopt simple question-answer formats with a fixed retrieve-then -eeee-then-generate pipeline that collapses multimodal evidence into primary textual channel (Hu al., 2025); (ii) they restrict evaluation short 1-2 hop retrievals without long, adaptive reasoning trajectories (Li et al., 2025b); and (iii) they lack stepwise annotations and explicit reasoning topologies that clarify the roles of different modalities in the reasoning process (Hu et al., 2025; Liu al., 2025b; Li et al., 2025b). As summarized in Table I (Left), these limitations make it difficult to determine whether MLLMs can truly perform long, structured reasoning over multimodal evidence.
To fill this gap, we present MC-SEARCH, the first benchmark for Multimodal agentic RAG with long. reasoning. Each example paired with a golden step-wise trajectory specifying the sub-question sequence, retrieval modality, supporting fact, and intermediate answer, enabling fine-grained evaluation, laying the foundation for process reward modeling.
structured Chains of
MC-SEARCH is designed to be long, diverse, and non-redundant. It covers five representative multi-hop reasoning structures as visualized in Figure 2, including Text-Only Chain, Image-Initiane Initiated Chain, Initiated Chain, Parallel Image- Fork, and Multi-Image Fork, capturing both serial and parallel reasoning patterns across modalities. To guarantee that each hop is necessary and structurally meaningful, we introduce HAVE (Hop-wise Attribution and Verification of Evidence), which filters spurious or redundant steps, resulting in 3,333 high-quality annotated examples with an average chain length of 3.7 hops, surpassing existing benchmarks (Hu et al., 2025; Liu et al., 2025b; Li et al., 2025b). Table (Right) reports the chain-length distribution across reasoning topologies.
Beyond answer-level accuracy. MC-SEARCH, we propose three process-level evaluation metries: (i) LLM-as LLLM-as-a-Jaaee a--deee for open-ended reasoning quality, (ii) Structure-Aware per Step Hit Rate for per-step retrieval fidelity, and (iii) Rollout Deviation to quantify execution drift. To evaluate the abilities, we further develop a unified agentic MM-RAG pipeline for fair benchmarking and extensive experiments with six leading MILMs, including both proprietary and open-source source models. Our analysis reveals key weaknesses such as over-retrieval, under-retrieval, and modality-specife planning errors, underscoring the need for better adaptive planning and retrieval over multimodal evidence.
In addition to serving as evaluation resource, our data also provides training signals for improving model capabilities. To this end, we present SEARCH-ALIGN, a conversation-leww alignment framework built HAVE-verified reasoning chains. It applies supervised fine-tuning beyond final answers, using step-wise trajectories to provide process-level supervision that strengthens open-source models' ability to plan and retrieve across modalities.
Our main contributions are summarized as follows:
Benchmark:
We present MC-SEARCH, the first benchmark for agentic multimodal RAG with long, step-wise annotated reasoning chains spanning five representative structures, verified by the HAVE procedure for necessity and non-redundancy.
Metrics: We propose new process-level metries that move beyond answer accuracy to precisely attribute reasoning quality, per-step retrieval and planning fidelity.
Evaluation: We develop a unified agentic
pipeline and conduct extensive benchmarking modality-
of six leading MLLMs, revealing systematic issues such as over- and misaligned planning, and eight characteristic error types.
2

---

Published as a conference paper at ICLR 2026
MC Search Benchmork Construction
Multimodal Agentic BAG Pipeline
Iopat
quection:
tretuiing Arthonger
depirted in tmoge, where's this curmenty when and how Lucained thccollictionn and what instemaa
vagtiaane cnd the soge si reprosant fo the attol reparding Armo maaccmt
Neve
Multimodal Large Language Models
Thinking... --
Rasson Mocil GFT-4o)
Duta Five Reasoning Topologgic
Astkam
Drnage Eoarch with Trpet Gorrg Image imraee Searet
bisiict scene fecouringg depieter Imagee?
S-i-iine fet
Reaton in
Tre
Data Fitering by HAVE: Hop Wise Attribution and Verificaioon
Oxtpue
Heal Asswert I'ne painting depits the onsurciat soons, sch butthe Phiadeiphia
Contoct Lisiityy Exsmication
Musoum Art, was accurred on Aprils, 1809 for St, 7505 and was Torner's first work be purchated on
Nevigation
Redundonn lasp Shrinagge
Go Chll
Predited
"IX Soreples
Chale-liaal Hite Per Sceps (MPP))
3,66 Roll Ceuletion (RO)
Figure 2: Overview of MC-SEARCH benchmark and evaluation. Left: Benchmark covering five reasoning topologies, filtered via the hop-wise attribution and verification of evidence (HAVE) process. Right: Multimodal agentic RAG pipeline, where MLLMM iteratively generates sub-queries and actions, retrieves multimodal evidence, reasons over the retrieved information, and integrates it to produce the final answer. Our framework further aligns predicted reasoning chains with golden trajectories to assess chain-level retrieval and planning.
Method: We introduce SEARCH- -LIGN, which leverages verified reasoning chains for process- supervised fine-tuning of MILMs, demonstrating the effectiveness of our data for training and improving planning and retrieval fidelity in agentic MM-RAG.
2 MC-SEARCH BENCHMARK
We propose MC-SEARCH, a large-scale benchmark for evaluating agentic multimodal RAG. In contrast to prior benchmarks that focus on shallow VQA tasks, MC-SEARCH provides long, step-wise verified reasoning chains across diverse reasoning topologies, enabling both fine-graines process- level analysis and model alignment. In this section, we detail the benchmark construction, dataset composition, and process-level evaluation metrics, with the overall workflow illustrated in Figure 2.
2.1 BENCHMARK CONSTRUCTION AND COMPOSITION
Search-Enhanced Reasoning Topologies. To capture the diversity of real-world agentic MM- RAG workflows, we design five distinct reasoning topologies that characterize how multimodal knowledge interacts and depends on each other within long search-enhanced reasoning process. Each topology is represented as reasoning graph (or reasoning chain) G G(Q, A), associated with an overall question Q and its final answer A. The graph consists of sub-questions qt, retrieval modalities E (text, image), ). retrieved evidence rt, and intermediate answers a, derived from the evidence at hop (Figure 2). We denote by R the retrieval function that, given a sub-question 9: and modality my, returns the corresponding evidence Formally,
A) ((g, m,, m,, (,, r, = RR=R(q mi), A f(fat)
(1)
where f(-) denotes the reasoning procedure that aggregates intermediate answers into the final answer for the overall question Q.
Concretely, the five reasoning graphs are defined as: (i) Image-Initiated Chain, where reasoning is anchored in image retrieval at the first step and subsequently builds on textual retrieval, applies to both image-containing and text-only queries; (ii) Teit-anit Chain, where reasoning begins with text retrieval of factual knowledge and later incorporates image retrieval for visual validation; (iii) Parallel Image- maxt Fork, where image and text retrieval evolve concurrently without dependency across specific hops, so that interchanging the retrieval of such hop order does not alter the final answer, requiring mross-modal coordination; (iv) Images Fork, where multiple images must be
3

---

Published as a conference paper at ICLR 2026
retrieved for visual comparison before turning to textual evidence for factual support; (v) Text- Onlyy Chain, serving as a baseline where reasoning proceeds entirely through text. Data examples for each reasoning graph are provided in Appendix C.
Data Generation. We construct the MC-SEARCH dataset by collecting multimodal knowledge from Wikipedia and organizing it into coherent clusters of text and images, typically drawn from the same or closely related pages, following Chang et al. (2022). From these clusters, we curate salient text facts and associated images as the basis for multi-hop QA generation, supplemented with illustrative examples aligned with each reasoning topology. From the curated pools, we then generate questions scale while ensuring balanced coverage across the five reasoning structures. Specifically, we prompt Gemini-24 2.5mni-2.5-P 2.a-Flash to select one topology and synthesize structured multi-hop question together with answer and coherent reasoning chain grounded in supporting facts. This procedure yields approximately 2 examples aligned with the five search-enhanced reasoning topologies. Additional construction details and prompt templates are provided in Appendix N.
Data Filtering. A key challenge in constructing a high-quality dataset is that MILMS often generate long reasoning chains with hallucinated steps, that is, plausible but ungrounded hops not supported by evidence, and redundani sleps that do not contribute to the final answer. To address this, we propose HAVE (Hop-wise Attribution and Verification of Evidence), a data filtering mechanism that determines whether each hop in the reasoning chain is both necessary and non-redundant.
Let = f (2,, ........... ,TT} denote the set of retrieved evidence in each hop of a reasoning graph. We define the context utility of step r; the drop in answer accuracy (measured by the F1 score between golden and generated answers) when Fy is removed:
(ti = (I F1 r.)-
A step is identified necessary if Util(t) exceeds threshold. However, some steps may not directly affect answer accuracy but still serve to connect sub-questions and guide downstream reasoning. For example, in two-hop chain, the first step may identify the building in the input image "Christ Church Cathedral," while the second step queries factual knowledge about this entity. Although the first step has low context utility on its own, its extracted entity is essential for the second hop, making it indispensable for the reasoning process. To capture such cases, we further assess the navigational role of step by checking whether entities in its intermediate answer a appear in downstream sub-questions:
(1. if Ent(a,) n Ent(g(+1:7) + 0, Nav(!) 0. otherwise,
where Ent(-) denotes the set of entities extracted from a sub-question or an intermediate answer. If a step has Util(t) below the threshold and Nav(t) 0, it is marked redundant.
We apply HAVE at scale by first filtering the generated dataset to remove samples with more than two redundant hops in their reasoning chains using the open-source Qwen2.5- VL-77B, yielding 4.8k candi- date chains. To further mitigate model-specific bias, we then perform hop shrinkage and sample verification with Gemini-2.5-Pro. while simultancously extracting key knowledge entities from the golden answers to facilitate future answer evaluation. Finally, we conduct a redundancy check using HAVE with the data-generation model 2.mmini-255-Flasht and yielding 3,333 high quality samples with all necessary hops and coherent reasoning trajectories. The final dataset composition and statistics are reported in Table 2 and Appendix E.
Table 2: Benchmark statistics.
Statistic
Number
Parallel Image-Text Fork
680
Image Initiated Chain
1,306
Text Chain
945
Text-Initiad Initiated Chain
169
Multi Images Fork
233
Total Samples
3,333
KB Images
389,750
Documents
784.473
Avg. Hops
3.79
Moreover, to ensure the reliability of subsequent step retrieval accuracy evaluation, we enforce that the reasoning graph for each question is unique. To this end, we use Gemini-2.55Flash for automatic verification, detecting and filtering confounding knowledge pieces in the knowledge base so that no unused text fact image within the same topic cluster independently resolve any sub-question. This provides a solid foundation for step-wisc wise evaluation of agentic MM-RAG.
Quality Verification. To ensure dataset quality in terms of reasoning coherence, grounding, and QA accuracy, we employ Gemini-2.55 2.5-Pro to evaluate each chain on a 1-5 scale along four dimensions: factual correctness, step necessity, clarity, and multimodal alignment (Appendix F). The dataset
4

---

Published as a conference paper at ICLR 2026
achieves overall average score of 4.87, and low-scoring samples further refined through targeted answer editing, yielding a high-quality benchmark for search- eeranned multimodal reasoning.
2.2 EVALUATION PROTOCOL
To evaluate multimodal search-enhanced reasoning, we believe that measuring only final answer accuracy insufficient, especially for long reasoning trajectories where errors in retrieval or planning may not be reflected in the final output. To enable more fine- fine-grained assessment, we introduce metries that capture both answer correctness and the fidelity of step-wise retrieval and planning.
Answer Accuracy. We four complementary metrics: (i) F1, the standard token-level overlap score between the generated final answer and the golden answer; (ii) AFI, which measures the performance gain brought by agentic MM-RAG beyond the model's parametric knowledge, defined as the difference between performance with and without retrieval: AFI = F1 - -Fl Ri (iii) Golden F1, an upper bound obtained by supplying the model with gold reasoning chains and golden retrieval content; and (iv) LLLM-as-a-Judge (LJ), where a strong reasoning model (Gemini-2.5--P) predicted reasoning chains against the gold ones along four dimensions: answer accuracy, reasoning coherence, key knowledge entity coverage, and step alignment (prompt in Appendix P).
evaluates
Chain alignment. We evaluate how well the predicted reasoning graph & aligns with the golden reasoning graph G. Each reasoning graph is a sequence of retrieval- auggeette reasoning steps indexed by t, where r. E denotes the evidence retrieved golden step i, and Arr E 9 denotes the evidence retrieved at predicted step t'.
To assess step-level retrieval fidelity, we define the (i) Hit per Step (HPS) metric as the fraction of golden steps whose evidence is successfully recovered in the predicted graph:
H) =
1 g
{ (t,t)) | re Eg for E E Ag. = =
where [g] is the number of golden steps and || on the right is the number of matched steps. Each "hit" corresponds to a predicted step that exactly matches the evidence of golden step, with duplicates counted only once. Beyond exact matching, we further align Ó and through maximun-weight bipartite matching, where nodes correspond steps in the two graphs and edge weights reflect evidence similarity between predicted and golden steps. This alignment enables fine-grained step-wise comparison and better handles cases involving parallel reasoning steps.
To capture structural fidelity, we measure the step-length gap between the predicted and golden reasoning graphs. A large gap indicates under- over-retrieval, while small value suggests closer alignment in reasoning complexity. We refer to this metric as (ii) Rollout Deviation (RD):
RD(3, = 1191 ||||-
AGENTIC MM-RAG PIPELINE AND PROCESS-LEVEL ALIGNMENT
To systematically evaluate and improve MLLMs in search-enhanced reasoning, we introduce unified agentic MM-RAG pipeline for adaptive planning, retrieval for multimodal evidence, and reasoning over evidence, together with SEARCH- ALIGN, process-level alignment framework that leverages verified reasoning chains strengthen open-source models.
3.1 AGENTIC MM-RAG PIPELINE
As illustrated in Figure 2 (Right), our pipeline models multimodal iterative process with three modules: (i) sub-query and action generation, (ii) evidence acquisition, and (iii) iterative reasoning and synthesis. At each iteration, the agent formulates a sub-goal, selects modality-awan retrieval action, integrates the retrieved evidence, and decides whether continue searching or terminate with a final answer. This design supports not only effective reasoning but also fine-grained, step-level evaluation for agentic RAG.
reasoning as an
(i) Sub-query and Action Generation. Given a complex question, the MLLM first predicts the next sub-goal and generates a focused sub-query (steps 1-2 in Figure 2). It then adaptively chooses one of three retrieval actions, corresponding to different ways of accessing multimodal evidence: (a) text search with text query, (b) image search with text query, (c) image search with input image.
5

---

Published as a conference paper at ICLR 2026
(ii) Evidence Acquisition. The selected action is executed over our local multimodal knowledge base The sub-query is embedded by multimodal encoder for dense retrieval in the corresponding modality, returning textual visual evidence. For each sub-query, keep the top-1 evidence by query-answer similarity (step 3) and let the MILMM generate a sub-answer (steps 4-5) after reasoning. Both the chosen modality and retrieved evidence are logged for process evaluation.
(iii) Iterative Reasoning and Synthesis. The sub-answer and its evidence are fed back into the model guide the next planning step (step 6), forming loop of planning, retrieval, and reasoning. The agent dynamically checks whether the accumulated evidence is sufficient; if so, it outputs a final response (step 7). Otherwise, it continues with another sub-query. This iterative design exposes the full reasoning trajectory, enabling moddlitty-waar execution and precise chain-level evaluation.
3.2 SEARCII-ALGGN PROCESS-LEVEL ALIGNMENT
Beyond evaluation, our dataset also serves as resource for improving model capabilities. We introduce SEARCH-ALIGN, conversation-leva alignment framework for open-source MILMs built on HAVE-verified reasoning chains. Unlike conventional SFT that supervises only final answers, SEARCH-ALIGN provides process-level supervision by leveraging step-wise annotated trajectories with sub-questions, retrieval actions, evidence, and intermediate answers.
Training Data Construction. Each reasoning graph is augmented with explicit reasoning thoughts generated by 2eemini-2.5-Flash. which explain how ground reasoning in evidence and connect adjacent hops. This converts ttep-wise annotations into coherent dialogue traces: the assistant poses sub-questions and reasons over evidence, while the user executes retrieval actions and returns results.
Supervised Fine- Fine-Tuning (SFT). We then fine-tune MLLMs on these conversation-style traces, enabling them to learn not only produce correct answers but also plan, choose retrieval modalities, and integrate evidence across steps. This process-level alignment provides a richer training signal that better prepares models for long-horizon multimodal reasoning.
4 EXPERIMENTS
In this section, we evaluate leading MILMSs on the MC-SEARCH benchmark under our unified agentic MM-RAG pipeline, focusing five research questions (RQs): RQ1: How do MLLMs perform across reasoning structures, and does SEARCH-ALIGH improve open-source models? RQ2: How does reasoning chain length affect reasoning performance? RQ3: What are the effects of over- and under-retrieval? RQ4: Do models show inherent modality preferences? RQ5: What are the typical error types in multimodal agentic reasoning?
4.1 EXPERIMENTAL SETUP
We evaluate the following MLLMs using the agentic MM-RAG pipeline introduced in Section 3.1:
Closed-source models: GPT-4o-Mini (Achiam et al., 2023), Gemini-2.5-Flash (Team et al., 2023), Gemini-2.5-PP (Team et al., 2023), and Claude-3.7-Sonnee (Anthropic, 2024), representing the strongest commercially available ccasoning-oriented MLLMs.
Open-source models: InternVL3.5588 (Wang al., 2025b) and Qwen2.5-VL-7B (Bai al., 2025), two state-of-the-arr lasion-language models. We also evaluate their performance after using SEARCII-ALGGN framework.
Evaluation protocol. All models are evaluated under identical conditions. We use the same embeding model for multimodal retrieval (Zhou et al., 2024), prompts (Appendix O), and decoding parameters across models. Each retrieval step is restricted to the top-1 result from the local multimodal knowledge base, and the same maximum number of reasoning iterations is enforced. Performance reported using the six metrics in Section 2.2: answer accuracy (F1, AFI, Golden F1, and LLM-as-a-Judge and chain-level retrieval and planning fidelity (Hit per Step and Rollout Deviation). Additional details on evaluation and the training of SEARCH-ALICE we additionally report fixed-step RAG baselines (I-hop and 2-hop) for the backbone models to complement the main results.
provided in Appendix I. In the Appendix D.I.

---

Published as a conference paper at ICLR 2026
Table 3: Evaluation MLLMs on the MC-SEARCH benchmark under agentic MM-RAC pipeline, reported across five reasoning graphs. Best backbone results are shown in bold, second-best are underlined. and improvements from SEARCH-ALIGN on open-source models are highlighted in bold red.
Answer Accuracy
Chain Alignment
Reasoning g
Model
Golden F1 (1)
F1 (f)
AFI (?)
(T)
HPS (f)
(4)
GPT-io-Mini
36.49
34.18
2.63
27.51
1,46
68,29
CLOSE
Gemini-2.5-Flash
44.10
37.38
3.01
31.46
2.91
72.39
ImitaeInitiated Chain
Gemini-2.5-Pro
47.61
42.76
3.18
25,90
1.05
69.83
Claude-3.7-Somne
37.80
33.09
2.60
27.31
1.18
72.62
Int5mVL3..-8B
39.11
29.49
2.27
22.59
1.58
OPEN
SEARCH-ALIGN
42.27
32.65
2.53
32.49
0.94
63.86
Qwen2.5-VI.-7B
26.30
8,65
1.34
16.51
4.04
60.95
SSEARCH-ALIGN
45.70
28.05
2.23
33.59
0.70
GPT-i-Minii
24.00
19.16
2.13
16.04
1.94
59.46
CLOSE
Gemini-2.55Faas.
36.80
31.89
2.35
13.45
1,66
64.40
Multi-Images Fork
Gemini-2.5-Pro
40.37
36.58
2.76
18.68
1.40
61.29
Claude Monude-3,7-Sorn 3.7-Sonnet
29.78
27.85
2.07
19.43
1.45
62.41
33.08
20.98
2.13
16.02
2.25
OPEN
SEARCH-ALIGE
36.53
24.43
2,49
39.33
1.67
57.65
Qwenn.55VVL-7B
26.13
7.53
1.41
12.58
3.80
57.04
SEARCH-ALIGN
38.01
19.41
2,11
38.01
1.16
GPT-4-Mini
30.11
18.46
2.41
27.18
1.76
49.47
CLOSE
Gemini-2.5-
43.55
26.34
3.30
25,20
1,20
66.27
Text-Initiated Chain
Gemini-2.5
45.30
29.89
3.62
19.51
0.95
55.94
Claude 3.7el--.7-Sonee
36.10
17.36
2.63
25.09
1.12
$7.13
VLternVL3.5-8B
38.34
25.48
2.60
19.06
1.97
OPEN
SEARCH-AALLC
40.87
28.01
2.59
37.82
1.44
38.70
Qwen2.5-VI-7B
31.47
14.34
1.66
17.02
3,80
39.86
SEARCH-ALIGN
45.07
27.94
2.42
33.59
0.87
GPT-o-Mini
21.98
21.65
2.20
15.00
1.71
53.98
CLOSE
Gemini-2.5-Flash
29.92
27.94
2.58
11.43
2.70
$7.99
Parallel Image-Text Fork
Gemini-2.5-Pro
34,83
34.19
2.99
16.74
1.29
53.46
Claude-3.7-Somee
26.43
24.65
1.95
14.32
1.37
58.51
InternVL3.5-88
29.74
23.05
2.13
14.22
1.75
$8.82
OPEN
SEARCH-ALIGN
30.72
24.03
2.29
23.55
1.45
Qwen2.5-VL-7B
22.94
14.32
1.33
10.09
3.93
$5.76
SEARCH-ALIGN
32.73
24.11
1.78
25.07
1.05
GPT-4o-Mini 30.96
30.94
2.58
21.94
1.13
61.90
CLOSE
Gemini-2.5-Flash
34.03
33.96
2.49
20.59
1.04
67.72
Text-Only Chain
Gemini-2.5-Pro
34.47
34.46
2.66
21.59
1.07
62.42
Claude-3.7-Somet
27.37
26.75
2.08
29.64
3.14
66.38
IntemVLL..5888
34.82
27.84
2.45
21.81
2.15
65.72
OPEN
SEARCH-ALIGN
38.45
31.47
2.68
20.54
1.28
Qwen2.5-VL-7B
23.54
14.89
1.36
13.35
4.63
61.57
SEARCH-ALIGH
37.54
28.89
1.99
19.04
4.2 MAIN RESULTS (RQ1)
Backbone Performance and Search-Align Gains. As shown in Table 3, proprietary models such as Gemini-2.5-PPr achieve the highest overall accuracy across most reasoning topologies. Among open-source models, Intern VL3.5-8B is the stronger backbone, while Qwen2.5-VLL 5-VL-7B consistently lags behind. With SEARCH-ALIGN, however, both backbones see substantial gains: Intern VI. improves by about +2.8 F1 and +12.0 HPS on average while reducing RD by 0.6, and Qwen achieves even larger boosts +13.7 Fl and +16.0 HPS with a -3.1 drop RD. After alignment, Qwen2.5-VL-7B nearly matches Gemini-2.5-Prr on text-centric chains, and InternVL shows stable improvements across all topologies. These results indicate that a major weakness of open-source models lies in retrieval planning and step-level reasoning alignment rather than basic perceptual or semantic understanding, and that these capabilities can effectively improved through training on high-quality search-enhanced reasoning data.
7

---

Published as a conference paper at ICLR 2026
Gemini-..5 Pre
Hauh
0.7
Gemini-2 S-Pro
Cluue-33 7-Sonnet
Hop
Gumini-2.5
GPT-leMei
Claute 37 20-5one::
GFT-4c Miri
InterniL 3.5
Owen2
Q 5--78
0.5
0.4
Hop
Hop
0.3
Q
0.2
0.1
0.0
Under-Retrieval
Over-Retrieval
Hop
Hop
AStop
Figure 3: Model F1 across samples with different chain lengths, showing consistent drop in perfor- mance as the chain length increases.
Figure Model AFI vs. A Step (difference in length between generated and golden reasoning chains), where larger positive Asteps indicate more over-retrioval.
Hopology-Specifie Challenges. Across reasoning topologies, we observe Parallel Image-Text Fork is the most challenging, where all backbones reach their lowest F1 and HPS. The difficulty comes from the need to plan and cover both text and image branches simultaneously. so missing either branch directly lowers HPS, and under the top-1 retrieval constraint models often fail to capture the full evidence set. Aligning parallel evidence without strict order further increases the risk of incomplete inconsistent reasoning. of aligning multiple visual evidences with corresponding textual facts, where models often retrieve the right images but misattribute or fail to integrate them coherently. In contrast, Text-Initiated and Text- Text-Only Chains are relatively easier, as they rely primarily on text retrieval and linear reasoning, which align with model strengths. Image-Initiated Chains also tractable: once the initial visual step is correct, subsequent textual validation is linear and more stable than multi-branch reasoning.
Fork is also difficult, mainly due to the challenge
4.3 ANALYSIS
RQ2: Performance vs. Chain Length. Figure 3 reports model F1 across samples with different chain lengths, showing a consistent drop in performance as the chain length increases. Models remain relatively robust on short reasoning chains (1-3 hops), where performance differences across backbones are moderate, with the exception of Qwen2.5-VL-77B which lags behind consistently even at early hops. When the chain length extends to 4 or 5 hops, however, all models experience a sharp degradation, reflecting the compounding difficulty of sustaining long-horizon reasoning.
Among closed-source systems, Gemini-2.5-Pro is the most robust, sustaining the highest performance at longer hops. Fas and Intern VL3.5-88 degrade gradually, while GPT-4o-Mini and Claude-3.7- 3.nde-t drop sharply beyond 3 hops. Qwen2.5-VL. LL-BB remains weakest across all lengths. Overall, the main challenge lies in extended multi-step chains, where compounding retrieval errors and unstable planning reduce accuracy.
RQ3: Over- and Under-Retrievl Analysis. Figure 4 shows AFI across different ASteps, where negative values correspond to under-retrieval and positive values indicate over-retrieval. The results reveal that both extremes are harmful, though in different ways. When models severely under-retrieve (AStep < 22), performance drops sharply as essential evidence is skipped, creating gaps that later reasoning cannot recover. Conversely, moderate over-vetrieval (AStep = 1-2) often improves accuracy: Gemini-2.5-PP Gemini-2.5. 2.5-aah,.,- and Intern VL3.5-8B achieve their highest AF1 here, suggesting that one or two extra turns can help recover from imperfect planning.
However, when over-retrieval is excessive (AStep 2 4), noise and irrelevant context lead to sharp drops across all models. Gemini-2.55Prr is most resilient, Gemini-2.5- remain moderately stable, while Claude-3.7-Somin and GPT-4o-Min are highly vulnerable. This underscores a central tension between capturing sufficient evidence and avoiding over-retrieval.
and VL.3.588
RQ4: Modality Coverage Analysis. As shown in Table 4, which reports step- retrieval coverage (covered vs. gold steps) for both image and text modalities under queries with and without image input, both models maintain strong text coverage (>78%) across all settings, but image coverage is far weaker and highly dependent on explicit image inputs. For example, Gemini-2.5 2.5-Pro drops from 87.5 87.35% with image mentions to 29.50% without, while Intern VLB..-8B falls from 63. 33.84% to 0.66%.
8

---

Published as a conference paper at ICLR 2026
These findings expose a pronounced modality gap, models tend to default to text retrieval and lose visual grounding in the absence of explicit image cues.
Table 4: Stepwise retrieval modality coverage (Cov.). Red high coverage; Blue = low coverage.
Gemini-2.5-Pro
InternVL-335-p5
Type
Modality
Covered/Gold
Cov. (%)
Covered/Gold
Cov. (%)
Image
1,339/1,533
87.35
977/1,533
63.84
Query w/ Image
Text
3,230/4,109
78.61
3,395/4,109
82.67
Image Text
300/1.017
29,50
7/1,017
0.66
Query w/o Image
2,097/2.510
83.55
2,251/2.510
89.78
Image
1.639/2.550
64.27
861/2.550
33.69
Overall
Text
5,327/6,619
80.48
5.688/6,619
85.96
RQ5: Error Analysis. Figure 5 shows the distribution of eight enror types, annotated using Gemini-2.5-Pro as the judge. The most frequent errors are Retrieval-Failue (4). Hallucinated- Entity/Attribute (75.8%), and Step-Omissie (74.3%), revealing that current models often fail to ground reasoning in relevant evidence, introduce unsupported content, or skip key steps. Mid-frequency errors (e.g., modality mis- match, spurious steps) reflect unstable plan- ning, while structural errors (order/depen- dency mistakes, multi-hop failures) are rarer but, given limited retrieval, remain non-trivial for future study. Evidence misinterpretation infrequent, suggesting models can handle retrieved context; major failures arise ear- lier, during planning and evidence acquisi- tion. Overall, these patterns, together with our HPS and over/uunder-retrieaal analyses, identify retrieval fidelity (modality-awand re- trieval, sufficient planning/hop coverage, and calibrated stopping) the central bottleneck for reliable chain-level reasoning.
Retrieva--aaltue Hallcinates Enttyyittrituute
04.70%
75.75%
Stop
74.34%
Hodality Mismaach
Spurious
OrderDapendency-Bee Enror
12.077%
Mriience-Miibeerreeee
9.35%
100
Proportion (%)
Figure 5: Eight-way emorr taxonomy proportions (higher worse), annotated by Gemini-2.5.PP
RELATED WORK
We review existing work MM-RAG benchmarks and Agentic RAG, with full details in Appendix B.
Multimodal RAG Benchmarks. MM-RAG extends retrieval-augmentee generation by incorpo- rating image evidence for cross-modal reasoning (Xia et al., 2024; Chang al., 2022). Existing benchmarks are limited: most use fixed retrieve-then-generaae pipelines (Marino et al., 2019; Hu al., 2025), reduce images to captions (Lerner et al., 2022; Chen al., 2023), restrict reasoning to 1-2 hops (Li et al., 2025b; Liu et al., 2025c), and lack step-wise annotations, making it hard to reasoning. We introduce MC-SEARCH, with HAVE-verifie long reasoning trajectories across diverse topologies, enabling fine fine-grained attribution and analysis.
assess multimodal
Agentic RAG. Recent work reframes retrieval as sequential decision- makiing, where agentic RAG systems decompose queries, adaptively retrieve, and synthesize knowledge (Wang et al., 2025a; Li al., 2025a; Zheng et al., 2025b). While effective, most remain text-only and focus on multihop QA or scientific domains (Trivedi et al., 2022; Rein et al., 2024). Our benchmark fills this gap by extending agentic RAG to multimodal settings with diverse, and step-wise verified reasoning chains.
CONCLUSION
We present MC-SEARCH, benchmark for structured, step-wise multimodal serieval-augmented reasoning, encompassing five diverse reasoning mechanisms. With fine-grained annotations, hop-wise attribution and verification, and new chain-level metrics, MC-SEARCI enables thorough diagnosis of retrieval planning and reasoning quality. Our analyses highlight the importance of adaptive, modality- aware search strategies in agentic MM-RAG systems. Looking ahead, we envision MC- SEARCH as a foundation for advancing multimodal agents and fostering principled evaluation standards for agentic reasoning. we will broaden evaluations to stronger reasoning models and extend the benchmark to additional domains such as science and mathematics.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work supported by NSF (2134079), and IBM-IIlinois Discovery Accelerator Institute. The content of the information in this document does not necessarily reflect the position or the policy of the Govermment, and no official endorsement should be inferred. The U.S. Government is authorized reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.
Ethics Statement. Our benchmark is constructed entirely from publicly available sources (e.g., Wikipedia text and images), without involving private data, or sensitive content. We apply hop-wise attribution and verification (HAVE) process to filter hallucinated or redundant steps, ensuring factual grounding and minimizing risks of misleading information. The dataset is released solely for research purposes to advance multimodal reasoning models and agentic RAG, and not intended for sensitive high-stakes applications.
Reproducibility Statement. We document the benchmark construction pipeline, filtering process, evaluation protocols, and experimental setups in detail within the paper and appendix. To facilitate full reproducibility, we have publicly released the dataset and code after acceptance, enabling independent verification and replication of all reported results.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Shakiba Amirshahi, Amin Bigdeli, Charles LA Clarke, and Amira Ghenai. Evaluating the robustness of retrieval-augmeetee generation to adversarial evidence in the health domain. arXiv preprint arXiv:2509.03787. 2025.
Anthropic. Introducing the next generation of claude, 2024. URL htwph:tt////ttt http:::///...ttttttrtt// com/news, ccl.deeaeeeeeeeeeeee 3- family ly.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 1rXiv:250..13923, 2025.
Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16495-16504, 2022.
Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang. Jeff Z Pan, Wen Zhang, Huajun Chen, Fan Yang, et al. Learning reason with search for Ilms via reinforcement learning. arXiv preprint arXiv:2503. 19470, 2025.
Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang. Can pre- pre-trained vision and language models answer visual information-seeking questions? arXiv preprint arXiv:2302.11713. 2023.
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang. Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey RAG meeting Ilms: Towards retrieval-augmented large language models. In Ricardo Baeza- Yates and Francesco Bonchi (eds.), Proceedings of the 30th ACM SIGKDD Conference August 25-29, 2024, pp. 6491-6501. ACM, 2024. doi: 00.114556335288367771 1470. URL https: //doi. orgiiorg//0 10.100000 1eee11115333355 3671470.
Knowledge Discovery and Data Mining, KDD 2024, Barcelona, Spain,
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. Scal- able multi-hop relational reasoning for knowledge-aware question answering. arXiv preprint wwXiv:2005.00646, 2020.
Robert Friel, Masha Belyi, and Atindriyo Sanyal. Raghench: Explainable benchmark for retrieval- augmented generation systems. arXiv preprint arXiv:2407. 2024.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi. Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. setrieval-augmenne generation for large language models: A survey. CoRR, abs/2312. 2023. doi: 10.48550/ARXIV.3112.000 https: /doii ttto//oor/oo////////////o/////e. .rrg/00 /00. 48550/ arxi 2312.10999
URL
Wenbo Hu, Jia-Chen Gu, Zi-Yi Dou, Mohsen Fayyaz, Pan Lu, Kai-Wei Chang, and Nanyun Peng. MRAG-bench: Vision-centric evaluation for retrieval- -ugmentee multimodal mod- els. In The Thirteenth International Conference on Learning Representations, 2025. URL https: haaps://ppenrrviee. net
Jerry Huang, Siddarth Madala, Risham Sidhu, Cheng Niu, Hao Peng, Julia Hockenmaier, and Tong Zhang. Rag-rl: Advancing retrievll-augmente generation via rl and curriculum learning. arXiv preprint arXiv:2503. 12759, 2025a.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint Nww.:xi:::30074444 2025b.
Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Chaoyou Fu, Guanglu Song, et al. Mmsearch: Benchmarking the potential of large models as multi-modal search engines. arXiv preprint arXiv: /2959, 2024.

---

Published as a conference paper at ICLR 2026
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Serean Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.05516, 2025.
Pankaj Joshi, Aditya Gupta, Pankaj Kumar, and Manas Sisodia. Robust multi model rag pipeline for documents containing text, table & images. 2024 3rd International Conference on Applied Artificial Intelligence and Computing (ICAAIC), pp. 993-999, IEEE, 2024.
Meng-Chieh Lee, Qi Zhu, Costas Mavromatis, Zhen Han, Soji Adeshina, Vassilis N loannidis, Huzefa Rangwala, and Christos Faloutsos. Agent-g: An agentic framework for graph retrieval augmented generation.
Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, José G Moreno, and Jesús Lovón Melgarejo. Viquae, dataset for knowledge-based visual question answering about named entities. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pp.
2022.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024.
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.053366 2025a.
Yangning Li, Yinghui Li, Xinyu Wang, Yong Jiang, Zhen Zhang. Xinran Zheng, Hui Wang. Hai Tao Zheng, Fei Huang. Jingren Zhou, and Philip S. Yu. Benchmarking multimodal retrieval augmented generation with dynamic VQA dataset and self-adaptive planning agent. In The Thirteenth International Conference on Learning Representations, 2025b. URL https: http::://pperee//// openreview. net/forum iidi-dd--VVKKe
Weizhe Lin and Bill Byrne Retrieval augmented visual question answering with outside knowledge. arXiv preprini arXiv:2210003800 2022.
Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. Fine-grained late- interaction modti-modal retrieval for retrieval augmented visual question answering, 2023. URL https:/ / /aaxivv.ivvvvv////////a/////aav/ org/ alss 22309 17133.
Ben Liu, Jihai Zhang, Fangquan Lin, Cheng Yang, Min Peng, and Wotao Yin. Symagent: A seural-symbolic self-learning agent framework for complex reasoning over knowledge graphs. In Proceedings of the ACM on Web Conference 2025, pp. 98-108, 2025a.
Zhenghao Liu, Xingsheng Zhu, Tianshuo Zhou, Xinyi Zhang, Xiaoyuan Yi, Yukun Yan, Yu Gu, Ge Yu, and Maosong Sun. Benchmarking retrieval-augmeenedd generation in multi-modal contexts. CoRR, abs/2502.17297, 2025b. doi: 1048550/ARXX22500011977,, URL https://doi..... org. org/ 10 485444555//0 arxiv 2502.17297 17297.
Zhenghao Liu, Xingsheng Zhu, Tianshuo Zhou, Xinyi Zhang, Xiaoyuan Yi, Yukun Yan, Ge Yu, and Maosong Sun. Benchmarking retrieval-augmented generation in multi-modal contexts. arXiv preprint 1xXi::2502.17297 2025c.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cff conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Zhenyu Pan, Haozheng Luo, Manling Li, and Han Liu. Chain-of-action: Faithful and multimodal question answering through large language models. arXiv preprint arXiv:2403. 17359, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference Language Modeling, 2024.
12

---

Published as a conference paper at ICLR 2026
Zhili Shen, Chenxin Diao, Pavlos Vougiouklis, Pascual Merita, Shriram Piramanayagam, Enting Chen, Damien Graux, Andre Melo. Ruofei Lai, Zeren Jiang, et al. Gear: Graph-enhanced agent for eetrieval-augmenice generation. arXiv preprint arXiv:2412.18331, 2024.
Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. Agentic eetrivallaugmenne generation: survey on agentic rag. arXiv preprint arXiv:2501.099 09116, 2025.
Gemini Team, Rohan Anil, Sebastian Borgeaud. Jean -aan-Baptsste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:23 / 2023.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 47 musique: Multihop questions via single-hop question composition. Trunsactions of the Association for Computational Linguistics, 10:539-55: 2022.
Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, and Furu Wei, Chain- of-retrieval augmented generation. ArXiv, abs/2501. 14342, 2025a. URL https://api. semant www.eicccccc..oooooooooooeeeeeee
27530666944.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Interm133. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint (5ii::200..12665 2025b.
Wenya Wang and Sinno Pan. Deep inductive logic reasoning for multi-hop reading comprehension. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4999-5009, 2022.
Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey emerging trends in multimodal reasoning. arXiv preprint arXiv:2401.068U0 2024.
Peng Xia, Peng Xia, Kangyu Zhu, Haoran Li, Haoran Li, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. Mmed-rag: Versatile multimodal rag system for medical vision language models. ArXiv, 01222410.1308... 2024. URL https:: //api. semanticschiii
org/CorpusID. 273403866.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809 1809:090009000 2018.
Han Zhang, Langshi Zhou, and Hanfang Yang. Learning to retrieve and reason on knowledge graph through active self-refiection. arXiv preprint 2xX1, 2025.
Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, Jie Jiang, and Bin Cui. Retrieval-augmente generation for ai-generated content: survey. arXiv preprint 24X2::2402. 19473, 2024.
Xu Zheng, Ziqiao Weng, Yuanhuiyi Lyu, Lutao Jiang. Haiwei Xue, Bin Ren, Danda Paudel, Nicu Sebe, Lue Van Gool, and Xuming Hu. Retrieval augmented generation and understanding in vision: A survey and new outlook. arXiv preprini arXiv:2503. 18016, 2025a.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL / org/ /203/2443. 13372.
Yuxiang Zheng. Dayuan Fu, Xiangkun Hu, Xiaojie Cai. Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Decpresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint 250: /60 2025b.
Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong. Megapairs: Massive data synthesis for universal multimodal retrieval. arXiv preprint arXiv:2412.1442 2024.
13

---

Appendix
Contents
Use of Large Language Models
15
B Related Work
15
B.1
Multimodal RAG Benchmarks
15
B.2
Agentic Bearch-Enhancee Reasoning
15
B.3
Atwec Benchmarks.
15
Data Example
16
C.I Image-Initiated Chain
16
C,2 Text-Initiated Chain
17
C,3 Text Chain
17
C,4 Parallel Visual-Textual Fork
18
C.5 Multi- Fork
19
D Baseline RAG Performance
20
D.1 Single-Step RAG Baseline
20
D.2 Traditional Multi-modal RAG Baseline
21
Dataset Statistics
22
Dataset Quality Verification
22
G Cross-Model and Human Consistency Analysis
23
G.I Cross-Model LLM-as-Judge Consistency
23
G.2 Human and LLM-as-Judge Agreement
23
G.3 Cross-Model Ground- Truth Consistency
24
H Fine-Grained Improvement Analysis of SEARCH-ALIGN
24
I SEARCH-ALIGN Fine-tuning Details
25
J Soft HPS Evaluation with Thresholded Semantic Matching
26
K Evaluation under Top-k Retrieval
26
K.1 Under- and Reer-Retri under Retrieval @K
26
K.2
SEARCH-ALIGN under Retrieval@K
27
Agentic MM-RAG Case Study
28
L.
Success Case
28
L.2
Failure Case
29
M Theoretical Justification of Topology Designs
30
M.
Problem Formulation
30
M.2
Proof by Exhaustion
30
M.3
Conclusion
30
N Prompt for Data Construction
31
N.I
Image-Initiated Chain
................................
31
N.2
Parallel Tmate-Text Fork
................................
31
N.3
Initiaan Chain
................................
32
N.4
Multi-Imape Fork
32
................................
N.5
Text Chain
32
Prompt Template for Agentic MM-RAG Pipeline
34
Prompt Template for LLM-as-Judge
35
14

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODELS
During the preparation of this paper, we made controlled use of L.Mss, specifically ChatGPT, as an auxiliary writing tool. The LLM was employed solely for stylistic refinement, namely to improve the fluency, grammar, and readability of paragraphs that were originally drafted by the authors. Importantly the scientific content, methodology, experimental design, and main narrative of the paper were fully conceived, written, and validated by the authors without reliance on LLMs. Therefore, LLMs served purely in supportive role for polishing author-written text, and their contribution does not rise to the level of co-authorship.
B RELATED WORK
We organize the related work from three complementary perspectives: (1) multimodal RAG bench- marks that extend retrieval beyond text, (2) agentic RAG systems that model retrieval as a sequential decision process, and (3) structure-aware benchmarks.
B. MULTIMODAL RAG BENCHMARKS
MM-RAG extends Strrevall-uggenn generation beyond text by incorporating image evidence for knowledge-intensing cross-modal reasoning (Xia et al., 2024; Chang et al., 2022; Pan et al., 2024). Existing benchmarks offer valuable first steps but remain limited: most adopt simple QA formats with fixed 2022; Amirshahi al., 2025; Friel al., 2024), perform limited visual seeking that reduces image information to captions (Chang et al., 2022; Lerner et al., 2022; Chen et al., 2023), restrict reasoning to 1-2-hop retrievals in VQA-style tasks (Li al., 2025b; Liu al., 2025c), and lack step-wise annotations or explicit reasoning structures. These constraints hinder diagnosing whether MLLMs genuinely perform structured multimodal reasoning or merely exploit shortcuts. To address this, We introduce MC-SEARCH, which provides HAVE-verified long trajectories across diverse reasoning topologies enabling fine-grained attribution and comprehensive analysis of MI.LMs' long-horizon, search-enhanced reasoning.
pipelines (Marino et al., 2019; Hu et al., 2025; Chang et al.,
B.2 AGENTIC SEARCH-ENHANCED REASONING
Recent work reframes retrieval as a sequential decision process, giving rise agentic RAG systems that decompose queries, adaptively retrieve evidence, and synthesize knowledge from retrieved contexts (Wang et al., 2025a; et al., 2025a; Zheng et al., 2025b; Singh et al., 2025; Pan al., 2024). Supervised fine-tuning (Li et al., 2025a) and reinforcement learning methods (Chen et al., 2025; Jin et al., 2025) further align model behavior with ground-truth reasoning traces to enhance interpretability and grounding fidelity. However, these advances remain largely text-only, often targeting multihop QA (Trivedi et al., 2022; Yang et al., 2018) or scientific domains (Rein et al., 2024), and thus lack systematic study of how multimodal signals affect retrieval planning and step- wise supervision in MLLMs, and how different search-enhanced reasoning topologies influence model behavior. Our work addresses this gap by introducing multimodal benchmark with diverse sarcc-en reasoning structures, step-wise verified long reasoning chains, and process-level metrics, revealing challenges such as error taxonomy, and by proposing SEARCH-ALIGN to improve planning and retrieval fidelity through step-wise supervision.
planning, under- and rver-retrieval, and
B.3 STRUCTURF-AWARE BENCHMARKS
A related line of work investigates reasoning models operating over symbolic KBs or graphs, often with schema-constrained queries or specialized tools (Lee al.; Shen al., 2024; Zhang et al., 2025; Wang & Pan, 2022; Feng et al., 2020; Liu et al., 2025a). These improve symbolic grounding but mainly focus on what to retrieve. In contrast, benchmarking agentic MM-RAG requires supervision over the structure of the retrieval chain itself-covering initiation modality, hop order, and parallel vs. serial patterns. To our knowledge, no prior multimodal dataset provides such structured, step-wise an- notations. Our Search-MM fills this gap by supplying golden reasoning chains with modality-specifff retrieval steps, enabling fine- mneegrained attribution, analysis of overr/under-retrienn retrieval, and evaluation of diverse multimodal reasoning structures.
15

---

Published as a conference paper at ICLR 2026
C DATA EXAMPLE
C.I INAGE-INITITED CHAIN
Example Reasoning Chain Image-Initiated Chain
Main Question. What biblical scene featuring Archangel Gabriel is depicted in the image, where is this painting currently held, when and for how much was it acquired for the collection, and what historical significance did this acquisition represent for the artist regarding American mu- seums?
Final Answer. The painting depicts the Annunciation scene, is held by the Philadelphia Museum of Art, acquired on April 5, 1899 for SI, 750, and was Tanner's 's first work to be purchased by an American museum.
By TiALE
Reasoning Chain (4 steps):
1. (Image) What biblical scene featuring Archangel Gabriel depicted in the image? Image Evidence ID: 30332733 Answer: The Annunciation.
(Text) Which museum holds this painting? Evidence ID:
d5be2ae00dba 1 eecbl e 7 463288e9_0
Answer: Philadelphia Museum of Art.
Caption: Archungel Gabriel from Annunciation
Google Art Project
(Text) When for what price was the painting acquired for the collection?
Text Evidence ID:
d5be2ae00dbal www.ccccc.111168888-8811
Answer: April 1899 $1,750.
4. (Text) What was the historical significance of this acquisition for the artist regarding American museums?
Text Evidence ID:
d5be2ae00dba leeblelllee eecblest 8 111171463288 Answer: It was Tanner's first work to be pur- chased by American museum.
16

---

Published as a conference paper at ICLR 2026
C.2
TEXT-INITIATED CHAIN
Example Reasoning Chain - Iext-Initi Chain
Main Question. Does the image visually confirm Solvang's unique cultural character as descrihed the text?
Final Answer. Yes, the image prominently features large windmill and buildings with distinct European/Danish chitectural styles, aligning with the description Solvang as "a bit of Denmark" in California.
Reasoning Chain (2 steps):
(Text) How is Solvang culturally characterized according the text?
Caption: Solvang California Windmill.
Text Evidence ID:
4Sbd142a0dbbal wtw.eaaaeecceee81146628888888882
Answer:
Solvang described "a bit of Den- mark tucked into this beautiful California."
2. (Image) Does the image visually confirm this cultural characterization, and if so, how?
Image Evidence ID:: 30326437 Answer:
Yes, image prominently displays a large windmill and buildings with distinet Euro- pean/Danish architectural styles, confirming its cultural deseription.
C.3 TEXT CHAIN
Example Reasoning Chain Text Chain
Main Question. What are the key attributes of the Bentley Mulsanne, including its manufacturing period, engine specifications, origin of its name, and notable special editions?
Final Answer. The Bentley Mulsanne, a full-size luxury car manufactured from 2010 2020, named after the Mulsanne Corner of the Le Mans racing circuit. It uses a 6.75 L Bentley L Series V8 engine, and notable special edition is the "W.O. Edition" ", which features a piece of W.O. Bentley's personal crankshaft.
Reasoning Chain (3 steps):
1. (Text) What type vehicle the Bentley Mulsanne, when was it manufactured, and what the origin of its name?
Text Evidence ID: d5bd6ace0dba 1ec.abe111e1111742288888888 _
Answer: The Bentley Mulsanne is full-size luxury that manufactured from 2010 2020 and is named after the Mulsanne Corner of the Mans racing circuit.
2. (Text) What are the engine specifications of the Bentley Mulsanne? Text Evidence ID: dabd6ace0dbal htt.d/aallbbell11111111 61171463228e9
Answer: The Mulsanne uses 6.75 (6.750 co/411 in²) Bentley L Series V8 engine, modified to meet Euro emissions regulations.
3. (Text) What notable special edition of the Mulsanne was introduced and what its unique feature?
Text Evidence ID: wat.aaaccooaaaaccccceeeeeee
Answer: The Mulsanne "W.O. Edition" was presented, featuring a piece of the crankshaft from W.O. Bentley's 's personal car displayed in the arm rest.
17

---

Published as a conference paper at ICLR 2026
C.4 PARALLEL VISUAL-TEXTUAL FORK
Example Reasoning Chain - Parallel Visual- Visual-Textual Textual Fork
Main Question. Given the appearance of the mandolin shown, did Pasquale Vinaccia's 1835 innovation in- volving string material lead to structural changes in the instrument and its lasting impact mandolin design?
Final Answer. Pasquale Vinaccia's 1835 innovation of ing steel wire strings, which are visible on the mandolin, necessitated strengthening the body and deepening the bowl for increased resonance, and this method of stringing ulti- mately became the dominant way for mandolins.
Reasoning Chain steps):
(Image) What type of strings are visible on the mandolin depicted, and how does this visual detail relate Pasquale Vinaccia's 's historical improve- ments the instrument?
Caption; Mandolinl.
Image Evidence ID: 30204742 Answer: The mandolin prominently displays wire/steel strings."
(Text) What structural modifications were subsequently required for the mandolin's 's body due to the adoption of these new wire strings? Text Evidence ID:
dabe6802 Odball Lechleel111 (111ee811114188eee1 15 Answer: The wire strings necessitated strengthen- ing the body and deepening the bowl.
(Text) What specific acoustic quality was enhanced by deepening mandolin's 's bowl a consequence of these structural changes?
Text Evidence ID:
Answer: Deepening the bowl increased tonal resonance.
4. (Text) Considering these improvements, what was the long-term impact of Vinaccia's decision to use steel strings on mandolin design and construction?
Text Evidence ID:
d5be6802 1eb.baalleeee11111111111421
Answer: steel-striggngg approach became the dominant way of stringing mandolins.
18

---

Published as a conference paper at ICLR 2026
C.5
MULTI-IMAGES FORK
Example Reasoning Chain - Multi- Multi-Images Fork
Main Question. Which of the depicted Epeot pavilions, Canada or Morocco, features a prominent tall, slender tower as its main architectural centerpiece, and what is its historical significance regarding its addition to the World Showcase?
The Morocco Pavilion features a prominent tall, slender tower, and was historically significant as the first expansion pavilion added to World Showcase, opening September 1984.
Final Answer.
Reasoning Chain (4 steps):
1. (Image) What is the most prominent feature the roof structure shown on the main building, the Canada Pavilion?
Caption: Canada Pavilion
Image Evidence ID: 30383576 Answer: very steep, multi-tiered green roof with pointed spires.
(Image) What the most prominent architec- tural feature defining the skyline of building in the Morocco Pavilion?
Image Evidence ID: 30021436
Answer: A tall, rectangular tower topped with small dome.
3. (Image) Considering the contrast in primary ver- tical architcctural elements between the Canada Pavilion's 's steep roof and the Morocco Pavilion's 's tower, which of these two pavilions would best described as featuring a prominent tower rather than very steep roof? Image Evidence ID: 30021436 Answer: The Morocco Pavilion.
Caption: Morocco Pavilion
4. (Text) According to the provided text, what significant historical fact is associated with the Morocco Pavilion regarding its status World Showcase addition?
Text Evidence ID:
d.bef (aeef66600bllll (eteeddbaleeeblleeeeeeee
Answer: It the first expansion pavilion be added to World Showcase, opening on September 7, 1984.
19

---

Published as a conference paper at ICLR 2026
D BASELINE RAG PERFORMANCE
D.1 SINGLE-STER RAG BASELINE
Table 5 reports the results of the fixed-step RAG baseline under the single-step retrieval setting. For all experiments, we use the BAAI/ BAAI/bge-large aagee-llg--e--l..... (Zhou et al., 2024) embedding model for dense retrieval. If the input question contains images, the system performs one-step image retrieval to obtain the corresponding caption, which is then concatenated with the query before being fed into the model. If the input is purely textual, the query itself is used for one-step text retrieval. In both cases, only the top-/ retrieved passage is selected as context and appended to the input prompt.
To ensure fair comparison with agentic multimodal RAG, enforce the same evaluation policy: models are only allowed to output an answer when they are sufficiently confident, otherwise they must abstain. This constraint avoids artificially inflating accuracy by forcing guesses and reflects the practical reliability of models in knowledge-grow grounded settings.
Our results reveal clear contrast between model families. More robust agentic models (e-g., proprictary Gemini-2.5-PPr tend to be conservative: when provided with only single retrieved knowledge piece, they often judge the evidence insufficient and refrain from answering. In contrast, most open-source models (e.g.. IntemVL-335. VL-3.5-Pro, Qwen2.5-VL- attempting to generate complete answers even when the retrieved context is too limited. While this behavior leads to superficially higher accuracy under fixed-step step evaluation, it also exposes a weakness of the fixed RAG paradigm: relying on single evidence piece frequently fails to support complex reasoning, producing shallow unsupported answers.
display overconfidence,
Table 5: Evaluation of various models the MC-Search benchmark under single-step RAG.
Answer Accuracy
Chain Alignment
Reason Graphs
Model
Golden Fl (1)
FI
AFI (f)
(t)
HPS (f) RD RD) (1)
GPT-4o-Mini
27.50
27.48
61.90
Gemini-2 Gemini-2.5-Flass 2.5-Flash
10.88
10.81
0.75
67.72
Gemini-2.55 Pro
11.11
11.10
0.70
62.42
Text-Only Chain
19.22
3.00
Claude-3.7-Sume
22.71
22.09
1.69
66.38
Inte5nVL3.558B
30.98
24.00
Qwen2.5-VIL-7B
35.26
26.61
1.81
61.57
GPT-4o-Mini
29.37
27.06
0.83
68.29
Gemini-2.5-FHi
3.63
-3.09
0.22
72.39
Gemini-2.5-Prg
2.34
-2.51
0.13
69.83
Image-Initiate Chain
21.52
3.00
Claude-3.7-Sonnet
8.29
3,58
72.62
Interm
21.10
11.48
0.77
63.86
Qwen2.5-VL-7B
40.31
22.66
1.62
60.95
GPT-4o-Mini
25.53
20.69
1.27
59.46
Gemini-2.5-Flast Fla-s
6.49
1.58
0.56
64.40
Gemini-2.5-Pro
7.26
3,47
0.51
61.29
Multi-Images Fork
15.38
3.00
Claude-3.7-Somme
12.70
10.77
InternVL335588B
33.11
21.01
1.59
57.65
Qwcn2,5-VL-7B
36.96
18.36
1,89
57.04
GPP--4--Mini
15.20
14.87
0.77
$3.98
Gemini-2.5 F.5-aa
4.09
2.11
0.30
$7.99
Gemini-2.5-PP
4.04
3.40
$3.46
Parallel Image-Text Fork
15.06
3.00
Claude-3.7
9.61
7.83
1.22
$8.51
Intern VL3.5-L8
20.63
13.94
0.99
58.82
Qwen2.5-VL--22
29.70
21.08
1.57
55.76
GPT-4o-Mini
24.67
13.02
1.43
49,47 19.47
Gemini- F-saaahh
16.78
-0.43
66.27
Text-Initiated Chain
Gemini-2.5.PP
14.51
-0.90
1.22
26.86
3.00
55.94
Claude 3,7--,nneSttt
17.63
-1.11
1.80
57.13
Intern 5L3 5-8B
28.82
15.96
1.38
38.70
Qwen2.5-VL-70
42.97
25.84
20

---

Published as a conference paper at ICLR 2026
D.2 TRADITIONAL MULTI- RAG BASELINE
To provide a stronger fixed-step baseline, we extend the single-step RAG setup to a two-hop variant. For multimodal queries, follow the standard multimodal RAG pipeline (Lin al., 2023; Joshi et al., 2024; Lin & Bymee 2022; Zheng et al., 2025a): first, we perform image retrieval to obtain the caption of the imput image. The retrieved caption concatenated with the original query, and this combined text is then used to conduct second-step text retrieval. For text-only queries, we directly perform two successive rounds of retrieval. both cases, the final retrieved passage (top- 1) appended as context to the input of the multimodal LLM.
As with the single-step baseline, we require models to output an answer only when sufficiently confident, ensuring fair comparison with agentic multimodal RAG and avoiding inflated accuracy due to forced guesses. Table presents the results the fixed two-hop RAG baseline. Compared to single-step retrieval (Table 5), two-hop retrieval yields consistent improvements across reasoning structures, especially for multimodal cases such as Multi-Images Fork and Text-Initiated chains. This confirms that additional retrieval steps can partially mitigate the information bottleneck of single-hop retrieval by providing more context.
However, performance still lags behind agentic multimodal RAG. While fixed two-hop retrieval provides more evidence, it cannot flexibly adapt the number retrievals or strategically integrate cross-modal knowledge. In contrast, agentic approaches selectively plan multiple hops and decide when to stop, yielding more faithful reasoning chains and substantially higher overall accuracy. These results highlight both the utility and the limitations of fixed two-hop RAG; it stronger than single-step retrieval but remains rigid and less reliable than adaptive agentic multimodal RAG.
Table 6: Evaluation of various models on the MC-Scarch benchmark under traditional MM-RAG.
Answer Accuracy
Chain Alignment
Reason Graphs
Model
Golden (t)
FI (1)
AFl (*)
LLJ (1)
HPS (1)
RD (L)
GPT-4o-Mini
35.61
33,30
1,58
68.29
Gemini-2.5-Faa
7.42
0.70
0.32
72.39
Gemini-2.5-Pro
6.06
1.21
0.25
69.83
Image Initiated Chain
21.11
2.42
Claude-3.7-Soma
21.72
21.10
1.72
72.62
IntemVL3.5588
31.26
21.64
1.24
63.86
Qwen2.5-VL-7B
39.59
21.94
1.79
60.95
GPT-4o-Mini
27.88
23.04
1.04
$9.46
Gemini-2.5-Flash
10.29
5.38
0.56
64.40
Gemini-2.5-Pro
11.03
7.24
0.57
61.29
Multi-Images Fork
19.35
2.76
Claude-3.7 3.7nonntt
19.96
15.25
1.67
62.41
Intem VL3.5VL3
34.55
22.45
1.58
$7.65
Qwcn2.5-VL-7B
36.61
18.01
1.97
57.04
GPT-4o-Mini
19.68
19.35
1.53
$3.98
Gemini-2.5-Flash
5.19
3.21
0.26
$7.99
Gemini-2.55 P.o-Pro
4.03
0.21
$3.46
Parallel Image- Text Fork
10.87
3.18
Claude 3.7-Sonnet
15.80
13.87
1.45
58.51
Intem VL3.55L3.5588
23.83
17.14
58.82
Qwen2.5-VL--77
30.16
21.54
1.61
55.76
GPT-4o-Mini
28.23
16.58
1.53
49.47
Semini-2255-Flan
20.51
3.30
1.19
66.27
Gemini-2.5. Pro
17.60
2.19
1.03
55.94
Text-Initiated Chain
26.72
1.90
Claude-3, 1.lee,3.-Sonnee
25.55
23.77
2.20
57.13
Intem VL3.5-8B
30.80
17.94
1.47
38.70
Qwen2.5 2. VL-7B
41.80
24.67
2.47
39.86
21

---

Published as a conference paper at ICLR 2026
E DATASET STATISTICS
Table 7 provides a detailed breakdown of the MC-Search benchmark. The dataset contains a total of 3,333 multimodal reasoning samples spanning five representative reasoning structures, with the largest portion being Image-Initiated Chains (1,306 samples), followed by Text Chains (945 samples) and Parallel Visual-Textuu Forks (680 samples). Although smaller in size, Text-Initiated Chains (169 samples) and Multi-Images Forks (233 samples) introduce challenging cases that require explicit cross-modal multi-image integration.
The benchmark is grounded large-scale knowledge base containing nearly 390K images and 780K textual documents, ensuring broad coverage across both modalities. On average, questions are 33 tokens long, while answers are longer and entity-rich (47 tokens with 8 entities on average), reflecting the complexity of the tasks.
In terms of reasoning depth, the average number of hops varies by structure, ranging from 3.17 in Text-Initiaten Chains around in Parallel reasoning complexity. Across all samples, the majority of hops rely on text retrieval (79.7%), while image retrieval accounts for roughly one-fifth (20.2%), highlighting a balanced yet text-dominant modality distribution.
Forks, demonstrating the diversity of
Table Statistics MC-Scarch benchmark.
Statistic
Number
Parallel Visual- Textual Fork Samples
680
Innge-Initiated Chain Samples
1,306
Text Chain Samples Text-Initiated Chain Samples
169
Multi-Imagee Fork Samples
233
Total Sumples
3,333
Total Images in the Knowledge Base
389,750
Total Documents in the Knowledge Base
784,473
Avg. Question Length
33.23
Avg. Answer Length
47.40
Avg. Answer Entities
8.03
Avg. Hops in Parallel Visual-Textual Fork
Avg. Hops in Image-Initiated Initiated Chain
3.82
Avg. Hops in Text Chain
3.65
Avg. Hops in Text-Initiated Chain
3.17
Avg. Hops Multi- Multi-Images Fork
3.97
Hops with Text Retrieval
10,063 (79.72%)
Hops with Image Retrieval
2,550 (20.20%)
F DATASET QUALITY VERIFICATION
To assess dataset quality, we employed Gemini-2.5-Pry to score each gold reasoning chain along four dimensions: (i) factual correctness, (ii) step necessity, (iii) clarity, and (iv) multimodal alignment. Scores are integers from (poor) 5 (perfect), with the overall score defined as the average of the four. Table 8 summarizes the results.
Table 8: Average Gemini-2.5-Pro evaluation scores.
Dimension
Average Score
Factual correctness
4.93
Step necessity
4.70
Clarity
4.88
Multimodal alignment
4.95
Overall (mean)
4.87
22

---

Published as a conference paper at ICLR 2026
G CROSS-MODEL AND HUMAN CONSISTENCY ANALYSIS
We report additional consistency study to assess the stability of the LLM-as-Judge (LLJ) evaluation. Agreement across heterogeneous model families and human experts shows that LLJ scoring captures objective and broadly applicable quality criteria, with observable dependence the preferred reasoning patterns of the primary judging model (Gemini-24 2.5-Pro) used in our experiments.
G.1 CROSS-MODEL LLM-AS-JUDGH CONSISTENCY
We randomly sample five non-overlapping subsets (each 10%) of predictions generated by the Gemini- 2.5-Pro backhone in Table 3. Each subset is independently rescored by three heterogeneous model families: Gemini-2.5-Pro, GPT-5, and Ww-n--V-PPlus. For each pair of judges, we compute Pearson correlation, Spearman rank correlation, Bland-Altman mean bias, and limits of agreement (LoA). Table 9 reports the mean and variance of the consistency score across five subsets.
Table 9:
Cross-model LLJ consistency scores, with mean (variance) reported. High correlations and small mean biases indicate strong agreement across evaluator families.
Evaluator Pair
Pearson
Spearman
Mean Bias
LoA
Gemini GPT-5
0.9330 (0.00076)
0.8720 (0.00120)
+0.0288 (0.00051)
0.8438 (0.07937)
Gemini Qwen
0.9147 (0.00116)
0.8145 (0.00360)
-0.0083 (0.00082)
0.9295 (0.08834)
GPT-5/ wwen
0.9147 (0.00131) 0.8259 (0.00374)
-0.0371 (0.00212) 0.9341 (0.09565)
All evaluator pairs show Pearson correlations above 0.91 and Spearman correlations above 0.81. Mean biases remain small in magnitude (< 0.04), and LoA ranges are narrow and stable. These results indicate that the LLJ evaluator generates consistent assessments across model families with diverse training data and architectures. Notably, the scores do not display any preferential tendency toward answers produced by Gemini-2.5-Pro, despite it being the primary judge in our main evaluation.
To further illustrate this, we provide Hexbin correlation plots between Gemini-2.5-PP and GPT-5 across all scoring dimensions (Accuracy, Alignment, Coverage, Coherence, and Entities). The distributions concentrate tightly around the diagonal, confirming that the LLJ scores follow objective, generalizable evaluation criteria rather than model-specific patterns.
Alammert
Coherence
Draifis
5
Figure 6: Hexbin plots of Gemini-2.5-Pro vs. GPT-5 scores across four L.IJ dimensions.
G.2 HUMAN AND ALM-AS--UDGE AGREEMENT
We additionally compare LIJ scores against human annotations collected from computer-science PhD experts 100 randomly sampled predictions. Humans score each sample along the four LLJ dimensions. Table 10 reports correlations and Bland-Altman statistics between human scores and Gemini-2.5-Pro scores for predictions generated by the Gemini-2.5-PP backbone.
Overall, the LLJ scores closely track human judgments across all content-groundee dimensions, with Pearson correlations above 0.92 for Accuracy, Entity Coverage, and the Stacked score. Even for the more subjective Coherence dimension, rank-order agreement remains strong (Spearman = 0.82), confirming that the L.J evaluation provides reliable and human-aligned quality assessments.
23

---

Published as a conference paper at ICLR 2026
Table 10: Human-LLa agreement scores.
Dimension
Pearson
Spearman
Mean Bias
LoA
Accuracy
0.9632
0.9556
+0.0800
0.6606
Entity Coverage
0.9375
0.8691
+0.1200
0.9309
Coherence
0.6061
0,8200
-0.0200
0.6778
Alignment 0.9275
0.7925
+0.2750
0.9982
Stacked Score
0.9323
0.8675
+0.1138
0.8562
G.3 CROSS-MODEL. GROUND- TRUTH CONSISTENCY
To examine whether the agentic MM-RAG pipeline depends model-specific phrasing or syntactic patterns in the reference reasoning chains, we construct an alternative benchmark using ground truth generated by a different model family. For each of the five graph topologies, randomly sample 50 instances and use GPT-5 to regenerate all intermediate sub-answers and the final answer following the construction procedure in Section 2. These GPT-5 chains serve additional set of ground-truth annotations.
We then evaluate Gemini-2.5-Pro and GPT-4o-mini under the full agentic MM-RAG pipeline. As shown in Table 11, both models achieve highly consistent performance across GPT--generatd and Gemini-generatee ground truths. Fl and LLJ scores closely track each other across all topologies, and the performance ordering between models remains stable.
These results indicate that the pipeline robust variations in the source model used generate reference reasoning chains. The evaluation reflects genuine multi-hop reasoning ability rather than dependence on stylistic or syntactic artifacts that may favor specific source model family.
Table Il: Performance of Gemini-2.5- Pro and GPT-4o-mini when evaluated against ground-trute reasoning chains generated by GPT-5 (GPT columns) or by the original Gemini- based construction (Gemini columns).
Fl Score (1)
LLJ Score (t)
Graph Type
Model
GPT
Gemini
GPT
Gemini
Gemini-Pro
32.52
34.42
3,44
3.46
Text Chain
GPT-4o-mini
26.05
26.98
3.01
3.05
Gemini-Pry GPT-4o-mini
41.37
46.57
3.82
3.81
Image-Initiated Chain
35.01
38.96
3.24
3.20
Gemini-Pa
35.49
38.04
3,48
3.66
Multi-Images Fork
GPT-4o-mini
17.97
18.77
2.69
Gemini-PP 36.59
38.44
3.64
3.67
Parallel Image-Text Fork
GPT-4o-mini
21.09
21.97
2.58
2.54
Gemini-Pa 34.91
43.71
3.42
3.82
Text-Initiann Initiated Chain
GPT-4o- mini
22.95
26.01
2.95
3.03
H FINE-GRAINED IMPROVEMENT ANALYSIS OF SEARCH-ALIGN
To examine how SEARCH-ALIGN alters the process behavior of multimodal agents, we analyze IntemVL-335 before and after alignment using fine-gra grained error taxonomy covering planning, retrieval, and modality selection behaviors. For each sample, an external LLM annotates four binary error types: (i) SerievallFailure (ii) Stp-Omission (iii) Order/Dependency Error, and (iv) Modality- Mismatch. We report the proportion of examples exhibiting each error across the MC-SEARCH validation split.
Figure 7 summarizes the results. SEAGN leads to consistent reductions across all four error categories, with the most substantial improvements observed in modality selection and multi-hop
24

---

Published as a conference paper at ICLR 2026
completeness. Modality- ModalittyMismatch decreases from 65.75% to 34.72%, indicating that the model learns to choose visual versus textual evidence more appropriately. Step-Omission also notably reduced (86.15% to 73.66%), suggesting that the agent follows more complete reasoning chains and less likely to skip intermediate steps. Retrieval-Failee and Order//Dpendeency-ee but stable reductions, reflecting improvements in evidence acquisition and hop structuring.
show smaller
Overall, this process-level analysis shows that SEARCH-ALIGN does not simply encourage imitation of annotated reasoning chains; rather, it induces measurable improvements across multiple dimen- sions of planning and multimodal decision-making. These reductions in structured error patterns complement the quantitative gains reported in answer accuracy and alignment metries.
86.15%
Step-Omission
73.66%
65.75%
Indaaity-Mismmathe
34.72%
31.06%
SEARCH-ALIGN InternVL-3.5-88
Onder/Dependeency-Errr
28.18%
90.28%
Retrieval-Failure
76.76%
20
40
60
80
100
Error Proportion (%)
Figure 7: Error proportions before and after SEARCH-ALIGN across four process-level categories.
I SEARCH-ALIGN FINE-TUNING DETAILS
To align the reasoning trajectories of multimodal large language models (MLLMs) with golden search chains, we perform supervised fine-tuning (SFT) using the allama-factoro framework (Zheng al., 2024). All experiments are conducted on NVIDIA A100 GPUs (80GB memory each). We report the detailed syper-paramett configurations for both open-source backbones used in our study, namely Intern -L and Swenn/Qeen.55-v1--75 hyper-parameters in unified format.
Table summarizes all fine-tuning
For clarity, we highlight the following aspects:
1. Backbone Models. InternVL uses the official InternVL3.5-88 checkpoint, while Qwen adopts the Qwven/Qwen2.5-VV--7B- V-L-7----t- variant.
2. Training Setup. Both models are fine-tuned with identical optimizer and scheduler settings, but differ learning rate and epoch count due to their different convergence characteristics.
3. Closed-source Baselines. For proprietary models (e.g., fine-tuning applied. Their internal thinking budgets remain at the default values.
GPT-4.1), no
Table 12: Fine-tuning hyper-parameters for Search-Align alignment on Intem and Seen//wwe2.5-VLL-B--ntttt Both models trained on 4xA100 GPUs.
Hyper-parameter
InternVL33.552
Qwen2.5-VL-7B--nnstee
Optimizer Learning Rate
AdamW
AdamW
le-5
le-4
Training Epochs
2
Warm-up Ratio
0.1
0.1
Scheduler Type
Cosine
Cosine
Train Batch (per device)
Gradient Accumulation Steps
8
25

---

Published as a conference paper at ICLR 2026
J SOFT HPS EVALUATION WITH THRESHOLDED SEMANTIC MATCHING
The Hit per Step (HPS) metric in the main paper evaluates whether the retrieved evidence exactly matches the gold evidence each reasoning hop. Because our maximum-weet weeggtt matching formula- tion naturally supports similarity- based matching, we further examine soft HPS variant in which an evidence pair considered hit its embedding similarity exceeds threshold т. We report results for four thresholds, 7 (1.0, 0.95, 0.90, 0.85). corresponding to exact match and increasingly relaxed semantic similarity.
Table 13 shows that soft-HPS scores for both Gemini-2.5-Pro and Gemini-Flash increase as decreases, indicating that models often retrieve semantically related evidence even when exact matches are missed. The improvements become smaller near = 0.85, suggesting that the remaining errors are driven by suboptimal planning, such as imperfect question decomposition or missing reasoning steps, which prevents the retrieval of the truly relevant evidence.
Table 13: Soft HPS for Gemini-2.55PPr and Gemini- Semini-Flashh under different similarity thresholds T.
Gemini-2.5-Pro Soft-HPS
Remini-Flash Soft-HPS
Graph Type
1=1.0
r=0.95
7=0.90
+=0.5
1=1.0
==0.95
r=0.90
7=0.85
Text Chain
21.59
35.59
38,05
41.26
13.33
26.86
28.78
30.97
Image-Initiated Chain
25.90
38.74
40.48
42.87
25.03
43.19
45.04
47.42
Multi- Images Fork
18.68
25.58
26.49
28.18
24.51
25.35
27.20
Parallel Image-Test Text Fork
16.74
26.25
27.68
30.22
20.73
25.68
27.12
29.69
Text- Leit--nnti Chain
19.51
26.31
27.81
29.88
21.67
29.94
30.55
32.03
K EVALUATION UNDER TOP-k RETRIEVAL
This section evaluates whether the retrieval dynamics in the main analysis hold when each step retrieves multiple candidates. Across top-k retrieval settings (k 1,3, 11,,5), 5), the overall conclusions on under- and over-retrieval remain unchanged. We also evaluate SEARCH-ALIGN under the same settings and find that its improvements are stable, indicating that both empirical insights generalize reliably to broader retrieval regimes.
K. UNDER- AND EVER-RETRIEVAL UNDER RETRIEVAL @K
We extend the over- and under-retrieval analysis to top-k retrieval by allowing Gemini-2.5-Flaast retrieve the top-k evidence candidates each step (1,3,5)) We evaluate AFI across different AStep values, where negative values indicate under-retrieval and positive values indicate retrevetrieval. Figure reports the results.
gemini-fiss-ttop.
gemini-ftash-top
gamini-ftash-tops
0.5
0.3
0.2
Under-Retrieval
Over-Retrieval
0.0
4
-3
-2
-1
0
1
2
AStep
Figure 8: AF1 across AStep values for Gemini-2.5-Flasr under top-k retrieval (k: 1,3, 5).
26

---

Published as a conference paper at ICLR 2026
Across all three settings, the qualitative trend is consistent. Severe under-retrieval (AStep < -3) causes large performance drops because essential evidence is skipped. As increases, the peak performance region shifts slightly toward AStep near - 1 or 2, suggesting that additional retrieved items help cover relevant evidence even with fewer reasoning steps. Larger l: also amplifies the nega- tive effect of over-retrieval: for ASeep > 4, performance declines more steeply due to accumulated noisy information. Thus, moderate deviations around the optimal retrieval depth remain beneficial, while strong under- over-retrieval is consistently detrimental.
K.2 SEARCH-ALIGN UNDER RETRIEVAL @ K
We further evaluate Qwen2.5-VL-7B and its SEARCH-ALIGN variant under the same retrieval settings. Table 14 reports FI, AFI, Hit per Step, and Rollout Deviation.
Before alignment, increasing k yields only small Fl improvements, Hit per Step remains low, and Rollout Deviation stays high (~3-4).. This indicates that broader retrieval alone does not resolve planning issues such as missed misordered reasoning steps.
After applying SEARCH-ALIGN, Qwen2.5 Swen2.5-VL-7B shows consistent improvements across all k: Fl increases by 15-20 points, Hit per Step rises substantially, and Rollout Deviation decreases around 1.0. These gains hold for retrieval@1 1, @33, and @5, and additional candidates provide diminishing but still positive benefits. Overall, SEARCH-ALIGN substantially improves both retrieval accuracy and multi-hop execution, enabling the model to utilize larger candidate sets effectively while maintaining stable reasoning behavior.
Table 14: Performance of Qwen2.5 VL-7B and +SEARCH-ALICS under Retrieval@ @3, @5. report F1, AFI, Hit per Step (HPS), and Rollout Deviation (RD). Best values per topology and metric are highlighted in bold font.
Retrieval
Retrieval
Retrievale
Graph Type
Model
F1
AF1
HPS
RD
F1
AF1
HPS
RD
F1
AF1
HPS
RD
Image- Initiated
Qwen2.5-VL-7B
26.30
8.65
16.51
4.04
29.21
11.56
16.17
3.79
29.35
11.70
16.95
3.79
Chain
*SEARCH-ALIGN
45.70
28.05
33,59
0.70
47.19
29.54
39.45
0.75
47.29
29.64
41.79
0.79
Multi Images
Qwen2.5-VL-7B
26.13
7.53
12.58
3.80
30.41
11.81
12.80
3.35
30.38
11.78
14.15
3.23
Fork
SLEARCI-ALIGN
38.01
19.41
41.20
1.16
40.00
21.40
46.82
1.21
39.80
21.20
48.21
1.28
Pamilell
Quen2.5-VL--B
22.94
14.32
10.09
3.93
26.28
17.66
12.62
3.39
26.41
17:79
15.24
3.26
Maage/Text&a Fork
SSEARCI-ALIGGN
32.73
24.11
25.07
1.05
33.54
24.92
28.43
1.09
32.77
24.15
30.93
1.18
Initiaait
Qwen2.5.V1.-7
31.47
14.34
17.02
3.80
35.54
88.41
18.86
3.20
35.25
18.
18.83
3.14
Chain
+SEARCH-ALIGN
45.07
27.94
33.59
0,87
46.69
29.56
41.06
0.95
46.73
29.60
42.35
0.93
27

---

painting Published as a conference paper at ICLR 2026
L AGENTIC MM-RAG CASE STUDY
L.1 SUCCESS CASE
Agentic Reasoning Prediction
Golden Reasoning Chain
Q: What biblical scona featuring Archangel Gabriel is depicted in image, currently held, when and for how much was acquired for the collection, and what historical significance did this acquisition represent for the artist regarding American museums?
Q: What biblical scene featuring Archangel Gabriel is depicted Image, currontly hald, when and for how much was acquired for the collection, and what historical significance did this acquisition represent for the artist regarding American museums?
painting
TThugghbiddenify the
featuring
<Question>
scone featuring Archangel
Archangel Gabrial in the image. </Thought>
Gebriel is depicted in the image?
<Sub-Questie Question> What biblica scane is depicted
image fealuring Archengel Gabriel?...
Retrieved golden supporting image.....
Image
Input
Image </Search> Retrivved content omilled...
<Thought> image depicts Annunciation scene featuring Archangel Gabriel. -Sub-Questin>> Where the painting currently held?
<Answer> The Annunciation. <Question> Which museum holds this peinting?
Retrieved content amitted...
golden supporting text......
<Thought-The painting held by the Philadelphii Museum Art. <Sub- Suab--ueetio>>> When and for how much this psinting aoquired collection?...
<Answer> Philadelphie Museum of Art. <Question> When and for what price was the peinting acquired collection?
Retrieved content omitte....
Retrieved golden supporting text...
<Thought>Thn painting 1899, $1,750. <Sub-Question>> What historical significance did acquisitinn represent artist regarding museums?
<Answer> April 1899 $1,750. historical significance
this acquistion for the artist regarding American museums?
Retrieved content omitted...
golden supporting text...
<Thought> The acquisition marked Tanner's first work purchased by an American museum.
purchased Amorican museum.
The painting depicts the Annunciation scene feeitrrig Archangel Gebriel, is held Philadelphia
<End> Final Answer:
Golden Answer: painting depicts Annunaation scene, hold by the Philodelphia Museum of Art, was acquired on April 5, 1899 for $1,750, and Tanner's first purchased by an American museum.
acquired on April 5, 1899, for $1,750, and represents Henry Ossewwa Tanner's a firet work purchased by an American
Figure Success case. The reasoning chain produced by the agent aligned with the golden reasoning chain and the final answer includes all key knowledge entities.
28

---

Published as a conference paper at ICLR 2026
L.2 FAILURE CASE
Agentic Reasoning Prediction
Golden Reasoning Chain
Q: What the name the bus operator depicted in the image, its establishment mainline to Francisco what is the city where its headquarters are located?
Q: What the name the depicted in image, establishment year, the start of its maintine service to San Francisco and what the where its headquarters
< <Thought The question asks for the name of the establishment year, service....
<Question>
operator
visible on the vehicle the image?
<Sub Question What the name of bus operator depicted in the image?
Retrieved golden supporting image......
Input Image </Search> Retneved content omitted....
Image
<Answer>
<Thought> The bus operator's 's name is "Sam Trans". indicaled the retrieved image Now <Bub-Queetion> What the establishment year of SamTrans and when did stari...
<Question> In what year was the district Tran establisedd?
information...
Retrieved golden supporting text......
<Search>.... No relevant
information relevantiiformmtion
found....
FAIL
cAnswar> 1976.
<Question How many years after Its estab Sam Trans begin mainlina bus service San
<Thought> nead to gother more information about Sam Trans, speciiiclll its estabitshment year and mainiine
San Francisso.
<Sub-Question>
headquartered?
Ratrieved golden supporting text...
Retrieved content omitted...
<Answer> One year later.
<Question> Where the headquarters of SamTrans
<Toought> SamTrane is meedquartered in San Carfos, California, However, stil need the astablishment year <Sub-Question>
mainiine.
cstabilshment year SamTrans and when did start its mainline service San Franclsco?
Retrieved golden eupporting lext...
information found...
FAIL
<Answer> San Carlos, California.
<Thought> have not been able reli leve year
the mainline
<End>
operator
Trans, It was established in 1976 and began lator, Its headquartars are located in San Carlos, Califomia.
<End> Final Answer:
SamTrans Trans is headquartered providle
year
in San Garlos, Califrnia, ostablishment year
to San Frencisco.
Figure 10: Failure case. While the agent successfully retrieves the first and the last hop information, it fails to retrieve the second and third hop information. Thus, the final answer does not include all key knowledge entities.
29

---

Published as a conference paper at ICLR 2026
M THEORETICAL JUSTIFICATION OF TOPOLOGY DESIGNS
In this section, we demonstrate that the five reasoning topologies defined in MC-SEARCH form a complete canonical basis for the space of agentic multimodal RAG traces considered in this work.
M.1 PROBLEM FORMULATION
We model a reasoning trace as a directed acyclic graph G (V, F)) where:
Each node i; represents retrieval step at hop t.
Each node assigned a modality m(v)) E {T, (Text or Image).
The structure is constrained be either Linear (a single path) or Forked (containing exactly
one parallel branching step that merges into an aggregator).
We define the set of valid topologies T by exhaustively partitioning the space of graphs based on (1)
Initiation Modality and (2) Structural Composition.
M.2 PROOF EXHAUSTION
Case 1: Linear Structures (Chains) Let G be linear chain 01 -> #2 - - The topology
fully determined by the sequence of modalities. We distinguish three mutually exclusive regimes:
Image-Anchored: The chain begins with visual input (m(v)) = I).
Definition: Image-Initiated Chain. (Covers all traces starting with an image).
2. Text-Anchored (Unimodal): The chain begins with text (m(v1) T) and contains only text nodes (Vt, m(v+ = T)).
Definition: Text-Only Chain. (Represents the standard text-RAG baseline).
3. Text-Anchored (Multimodal): The chain begins with text (m{(1) T) but contains at least one subsequent image node (Er > 1, m(x) T).
Definition:
Text-Initiated Chain. (Captures text-to-visual grounding).
Case Forked Structures (Parallelism) Let contain a parallel branch where nodes va and are retrieved independently. The topology is determined by the multiset of branch modalities Sbranch = (va), m.
Heterogeneous Branching: Stranch = {1,T}.
Definition: Parallel Image-Text Fork. (Captures simultaneous cross-modal evidence integration).
2. Homogeneous Visual Branching: Stranch = 11.
Definition:
Multi-Images Fork. (Captures comparative visual reasoning).
Remark on Homogeneous Text Branching (Sbranch {T, The final mathematical possibility is parallel text retrieval. Within the scope of multimodal agentic search, we treat this case as structurally reducible to the linear Text- Only Chain. Since modern LLMs process multiple text documents via context concatenation, a parallel text retrieval step is functionally equivalent to single multi-document retrieval hop. Therefore, we do define distinct topology for text-only branching, preserving the minimality of the set relative to multimodal operations.
M.3 CONCLUSION
The five definitions above cover all permutations initiation modality and distinct multimodal branching structures. Thus, the set is complete under our assumptions.
30

---

Published as a conference paper at ICLR 2026
N
PROMPT FOR DATA CONSTRUCTION
This appendix provides the system prompt template used to generate multimodal reasoning chains with Google Gemini. The annotator is instructed to construct complex, structurally grounded multi-hop questions following of five reasoning graph types: Image-Initiated Initiated Chain, Parallel Visual-Texmna Fork, Text-Initiated Chain, Parallel Multi-Images Fork, and Text Chain. Each generated chain must explicitly build upon previous answers. remain grounded in the provided images and Wikipedia snippets, and output in strict JSON format with unique supporting evidence identifiers.
We provide five reasoning graph templates with guidelines and few-shot examples standardize multimodal question generation.
N.1 IMAGE-INITIATE CHAIN
Structure Description: Reasoning begins with visual recognition or inference from an image, followed by sequential text-based hops grounded on that visual anchor.
Guidelines:
Begin with a visual observation from the image.
Use that entity/fact the anchor for the next step.
Add 1 1-2 textual hops depending on the previous answer.
Example: Question: Where did the football team, whose players wore red-and-yellow vests during 2019 match, originally form, and is that location still part of the city they are associated with today? Answer: The team was originally formed in Partick, which is now part of Glasgow. Sub-questions and answers:
(Image) Which team wore red-and-yellow vests during a 2019 soccer match? > Partick Thistle F.C.
(Text) Where was Partick Thistle originally formed? - Partick
(Text) Is Partick still independent from Glasgow? + No, it is now part of Glasgow.
N.2 PARALLEL MAGE-TEX' FORK
Structure Description:
Reasoning starts with a sub-question that requires jointly leveraging both visual and textual information. These signals must be fused before continuing via text.
Guidelines:
Design sub-question requiring combination of both image and text.
The fused result forms the anchor for continued reasoning.
Add a follow-up text hop grounded in the first answer.
Example: Question: Is the curved architecture of the Petersen Events Center well-suited for its use as modern basketball arena Pitt? Answer: Yes, the curved structure complements its role as large multi-purpose sports venue. Sub-questions and answers:
(Image) Does the Petersen Events Center have curved architectural design suitable for large gatherings? + Yes, the central structure is rounded and accommodates large interior space.
(Text) What is the primary use the Petersen Events Center on the Pitt campus? - is a multi-purpose arena used by the Pitt basketball team.
(Text) Does the architectural design align with its use as a modern sports arena? + Yes, its rounded shape supports audience visibility and modern sporting needs.
31

---

Published as a conference paper at ICLR 2026
N.3 TEXT-INITIATED CHAIN
Structure Description: Reasoning is initiated via text and ends with image confirmation; the final hop depends on visual content.
Guidelines:
Start with a sub-question answerable via text.
Use that answer as the anchor for another textual hop.
The final step should rely on visual confirmation from the image.
Example: Question: Is the area in front of the Torre del Reloj enhanced with any water-related architectural features? Answer: Yes, there is fountain placed in front of the Torre del Reloj. adding a water-related feature the plaza. Sub-questions and answers:
(Text) Where is the Torre del Reloj and what landmark it? + It a historic clock tower in Chiclana's Plaza Mayor.
(Image) Is there any water-related structure in front of the Torre del Reloj? + Yes, fountain is clearly visible.
N.4 MULTI-IMAGES FORK
Structure Description: Reasoning begins by comparing or aggregating information from two images, then follows with text-based interpretation.
Guidelines:
Compare or verify a shared visual pattern across two related images.
Use their joint interpretation the basis for a follow-up textual inference.
Example: Question: What visual marker appears along both Salou Boulevard and the adjacent plaza, and what does it symbolize? Answer: Blue footprints appear in both places, symbolizing a historical walking route. Sub-questions and answers:
(Image) What pattern is painted down the center of Salou Boulevard? + Blue footprints.
(Image) Does the plaza image show the same pattem?? > Yes, the same blue footprints.
(Text) According to local guides, what do blue footprints represent? - They mark historic promenade route.
N.5 TEXT CHAIN
Structure Description: A standard multi-hop reasoning chain entirely over textual content.
Guidelines:
Construct a multi-hop reasoning chain using only text.
Each sub-question must depend explicitly the previous answer.
Avoid visual modality entirely.
Which historical street in Quebec City was known for having unique wooden paving method in the early 20th century? Answer: Little Champlain Street was known for its board paving in 1916. Sub-questions and answers:
Example: Question:
(Text) What notable historic streets exist in Old Quebec? -> Saint-Vallis Est Street and Petit-Champlain (Little Champlain).
(Text) Which of them had unusual wooden paving in 1916? Little Champlain Street had board paving.
(Text) Was board paving common or unique for that era in Quebec City? It was unique/un- common, making the street notable.
32

---

Published as a conference paper at ICLR 2026
Example: Generating Multimodal Reasoning Chains with Gemini
Input Prompt:
You are a multimodal data annotation expert. You are given multihop multimodal QA generation task.
You are provided with:
or images (embedded as Base64 above with captions below)
Several Wikipedia passages
An original 1-hop simple QA pair for reference
Your task is to generate new, complex, coherent, and structurally grounded multi-hop question
and reasoning chain that follows the specified reasoning graph type.
Reasoning Graph Information:
Graph Type: (graph_type)
Structure Description: (graph_struccee structure {
Structure Instruction: (graph_struccuti structure instruction)
Expected Reasoning Hop Count: 4-5 hops
Guidelines:
The question must align with the given graph structure, integrating visual and textual modalities where required.
Each sub-question must explicitly build upon the previous answer, using it as anchor for the next hop.
Each peopporting_fcctee
must unique across the chain.
Answers must be directly grounded in the provided images or text snippets (no halluci- nated facts).
The chain must be coherent and progressive, forming a natural reasoning path.
The final answer should logically follow from the entire chain.
Each sub-question must specify:
modality: "text" or "image"
susporting__act___ d: the of the text snippet or image answer: the grounded answer
Output must be in strict JSON format (not Markdown or free text).
Additional Context:
Original question (for inspiration): [entry['Q") Q']]
Original answer: [entry['A
Image captions: (captions)
Wikipedia snippets: [text_snippets]
Expected Output Format:
"question": "...",
"answer"; "..."
"image_id""
# or "image_ide" ids": [...]
"graph_type"; "ggraph_type)",, "subqa_chain":
"subquestion":
"modality": "text" "image",
"supportiiggface seatttinn_fatttioo id": "...", "answer:: "..."
33

---

Published as a conference paper at ICLR 2026
O PROMPT TEMPLATE FOR AGENTIC MM-RAG PIPELINE
This appendix provides the full system prompt used for agentic MM-RAG pipeline. The assistant is constrained operate strictly on retrieved local contexts from external text and image knowledge bases, and follows a structured reasoning protocol consisting of <Thought>, <Sub Sesb-Quest ion>, <Search>, and <End> stages. This ensures that every generated answer grounded, step-wise, and verifiable.
Gemini Prompt for Multimodal QA
System Prompt:
You are helpful multimodal QA assistant that works with an external text knowledge base and an external image knowledge base. You must use only the knowledge from the external resources provided. You must retrieve by yourself. All answers must be grounded only in the retrieved local context that give to you.
Available retrieval actions:
Image Retrieval with Input Image
Text Retrieval with specific query
Image Retrieval with Text Query
Protocol:
<Thought> Analyze the original question and decide the next minimal sub-question solvable in one step.
Stub-Question Write one sub-question that should be solved in one step (no references).
<Search> Choose exactly one of the retrieval actions above (or "No Retrieval").
Repeat <Thhought>/<uub-uuett ion> <Search>>>>> zero
times.
<Thought> Integrate retrieved content and reason the final answer.
<End> Final Answer: one-sentence answer to the original question.
Extra Notes:
The user will perform the actual search after you provide the query.
2. The local text knowledge base contains short factual passages. Retrieval returns most one passage each time.
3. Answers must be grounded only in retrieved content.
4. If retrieval insufficient, continue with additional steps.
5. Keep steps concise; do not include citations URLs.
6. Do not hallucinate; if not enough information is retrieved, state that you cannot answer.
Input Format: Input Question: [QUESTION)
34

---

Published as a conference paper at ICLR 2026
P PROMPT TEMPLATE FOR LLM-AS-JUDGE
This appendix provides the exact system prompt used to evaluate predicted answers against golden references under an input-only policy. The judge does assess the truthfulness of the golden answer; it scores the prediction's Accuracy, Entity Coverage, Coherence, and Alignment on 0-5 integer scales following the band definitions. When a prediction claims "insufficient information" while a golden answer exists, is penalized. The judge outputs strict JSON containing only the four scores, enabling reliable programmatic aggregation for large-scale evaluation.
Example: Judge System Prompt
Input Prompt:
You are strict, neutral judge. Compare pred answer to gold answer (and optional subchains) for the given question. Use only the provided text; no external knowledge. Do not judge the truth of the gold answer-only the prediction's 's consistency with it. If the prediction claims "insufficient information" while gold answer exists, penalize. Keep the output terse.
Inputs:
question
gold_answer, pred_answer
gold_subehain, pred_subchain
Guidelines (Scoring Dimensions 0-5):
Accuracy: agreement with gold on facts/relations; answers the question; no contradic- tions.
Entity Coverage: presence and correctness of key entities, values, and relations required by gold.
Coherence: clear stepwise reasoning (if any); no leaps or conflict;; conclusion follows logically.
Alignment: uses only the given information; matches gold subchain when relevant; no unsupported additions.
Score Bands: = perfect; minor issues; noticeable gaps; major gaps weak; 1
mostly wrong; 0 = off-topic or none.
Expected Output Format:
"scores";
"accuracy"; 0,
"entities": 0,
"coherence": 0, "alignment.:
35


Reviews:
[{"review_id": "2qgxQjuRcr", "reviewer": "Reviewer_HAWQ", "summary": "This paper is well-motivated, presenting a novel perspective for improving retrievers by tackling hard tasks. We are convinced by the sufficient baselines used to verify the proposed method and believe it inspires future work on more challenging tasks. I have several minor concerns, including potential unreliability in using LLMs for relevance evaluation, the lack of a simpler majority vote baseline, prohibitive computational costs, and some lack of clarity in the agent selection process. We further seek clarification on case analyses for hard samples, the potential benefit of incorporating more specialized agents.", "questions": "1. Line186 & 208. Inconsistency among Code Agent and CoT Agent indicates a hard sample. Could the author give any case analysis?\n2. Related to Weakness1, will it be more trustworthy by incorporating more Agents besides Code Agent and CoT Agent? E.g., agents that specialize searching. Or, the current setup is enough.\n3. Typo? Line243: $y_{i,k}^{t}$ and $y_{i,k}^{t}$.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-motivated. It tries to solve hard tasks in a new perspective of view, improving retriever. Sufficient baseline retrievers adopted to verify the proposed method to improve the retriever. This work inspires the future direction on solving more challenging tasks like BrowseComp.", "weaknesses": "1. Previous work has shown that asking LLMs themselves to evaluate the relevance of queries and documents are not that reliable [[1]](https://arxiv.org/abs/2505.21870). Applying Code Agent and CoT Agent is also within this paradigm. It would be better if there are experiments conducted to verify the relevance between inconsistency (among Code and CoT Agent) and query difficulty.\n2. The initial CoT agent's decision ($y^{g}_0$) is only overturned if all L discussion groups unanimously disagree. Thus, it seems that a simpler majority vote is a more standard and intuitive baseline, but no comparison is offered.\n3. Prohibitive computational cost makes the approach infeasible for generating datasets of any significant size. (1 hard example=~58 API calls and 100k tokens)\n4. Lack of Clarity on Agent Selector and Validator.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DdKpgBP5Ge", "reviewer": "Reviewer_nveM", "summary": "This paper introduces a multimodal retrieval-augmented generation benchmark for agentic paradigm. It has 3333 examples averaging 3.7 hops with sub-questions, retrieval modalities, supporting facts, and intermediate answers ensured by HAVE. It also proposes some process-level metrics to judge the reasoning quality and retrieval performance and planning fidelity.", "questions": "1. how to evaluate and ensure that Qwen2.5-VL-7B is good at judging each hop is both necessary and non-redundant? is there any mannual double-check and fine-tuning methods or?\n\n2. please refer to weakness section for more contents.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. this submission is well-prepared, especially in figures, tables, and appendix.\n\n2. the key contribution of this submission is obvious and makes sense.\n\n3. the process metrics are actually needed things in multi-step reasoning tasks. \n\n4. the experiments and analysis are comprehensive and high-quality.", "weaknesses": "Many related work may help authors to enhance the completeness of the submission:\n\n1. evaluation on robustness of resisting harmful information is also interesting in RAG-based agentic framework [1].\n\n2. \"multi-modality\" may also extend to SQL-based database, query-rewriter-based web-search, and even more [2].\n\n3. token usage (input and output) and the number of retrieval callings are also helpful to enhance the benchmark [3].\n\n\n\n[1] Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain\n\n[2] Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models\n\n[3] RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CM3VCRAG1e", "reviewer": "Reviewer_YFCi", "summary": "This paper introduces MC-SEARCH, a new benchmark for evaluating multimodal agentic Retrieval-Augmented Generation (MM-RAG). The authors argue that existing benchmarks are too simple, focusing on short reasoning tasks. To address this, they make four main contributions: \n\n1. The MC-SEARCH benchmark, a dataset of 3,333 examples featuring long reasoning chains (avg. 3.7 hops) organized into five distinct reasoning topologies. \n\n2. A data filtering process called HAVE (Hop-wise Attribution and Verification of Evidence) to ensure each reasoning step is necessary and non-redundant. \n\n3. New process-level metrics (e.g., Hit per Step, Rollout Deviation) to evaluate intermediate reasoning steps, not just the final answer. \n\n4. The SEARCH-ALIGN framework, a process-supervised fine-tuning method that uses the benchmark's annotations to improve the capabilities of open-source models.", "questions": "How does MC-SEARCH differ empirically and conceptually from Dyn-VQA and MMSearch beyond having predefined topologies? Could the authors quantify the added diagnostic value of these structures?\n\nHow do the authors ensure benchmark neutrality given that Gemini models are used for both generation and evaluation? Have they tested whether non-Gemini models (e.g., GPT-4o, Claude, QwenVL) are unfairly disadvantaged?\n\nHow might the conclusions about SEARCH-ALIGN change under a top-k retrieval setting where irrelevant evidence must be filtered dynamically?\n\nWhat criteria guided the selection of the five reasoning topologies? Are these empirically grounded (e.g., observed in task distributions) or designed heuristically?\n\nCould HPS be complemented by a semantic similarity–based variant, ensuring agents that retrieve equivalent but non-identical evidence are not penalized?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper's primary strengths lie in its thoughtful benchmark design and its focus on process-level evaluation.\n\nStructured Benchmark Design: The introduction of five explicit reasoning topologies is a significant contribution. It moves beyond simply creating \"long\" chains and provides a structured way to diagnose specific model failures (e.g., a model failing on Parallel Forks but succeeding on Linear Chains). This offers a more granular analysis than existing benchmarks.\n\nRigorous Data Curation: The HAVE process is a commendable effort to improve benchmark quality. By programmatically filtering out trivial or redundant reasoning steps, the authors have likely created a more challenging and reliable testbed for genuine reasoning abilities, addressing a common weakness in synthetically generated datasets.\n\nProcess-Oriented Evaluation: The push for metrics beyond final answer accuracy is timely. In complex agentic tasks, intermediate failures are often hidden. Metrics like Hit per Step (HPS) and Rollout Deviation (RD) provide a clearer view of how and where models fail, which is critical for future development.\n\nDemonstrated Training Utility: The inclusion of the SEARCH-ALIGN framework is a strong point. By showing that the detailed annotations can be used for process-supervised fine-tuning to improve open-source models, the authors prove that MC-SEARCH is a valuable resource for both evaluation and model development.", "weaknesses": "The paper's contributions are undermined by significant weaknesses, primarily an overstatement of novelty and key methodological limitations.\n\nNovelty overstatement — The paper's central claim of being the \"first benchmark for agentic MM-RAG with long, step-wise annotated reasoning chains\" is not well-supported. Prior work such as Dyn-VQA benchmark was specifically designed for dynamic, multi-hop questions requiring complex, adaptive retrieval and also introduced a self-adaptive planning agent. The paper must reposition its contribution not as being the first, but as providing uniquely structured and verified reasoning chains, and it needs to properly differentiate itself from Dyn-VQA and other related works like WebQA, MRAG-Bench, and MMSearch.\n\nPotential model bias — While the HAVE pipeline partially mitigates bias via cross-model filtering (Qwen2.5-VL and Gemini-Pro), Gemini models still dominate generation, filtering, and evaluation phases. This may tune the benchmark toward Gemini’s reasoning style.\n\nSimplified retrieval setting — The evaluation uses a top-1 retrieval setup, which does not reflect realistic RAG conditions where irrelevant documents co-exist. The conclusions regarding over/under-retrieval and SEARCH-ALIGN efficacy may not fully generalize to top-k retrieval.\n\nLimited justification for reasoning topologies — Although the five topologies are intuitive, the paper does not provide empirical or theoretical justification for their selection (e.g., frequency in real-world tasks). Including such evidence would improve the framework’s validity.\n\nMetric rigidity — The Hit per Step (HPS) metric relies on exact evidence matching, which could penalize models finding semantically equivalent alternatives. The authors do mention a semantic alignment mechanism for structural comparison, but an integrated soft-matching variant could make evaluation fairer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CQMK91eLqE", "reviewer": "Reviewer_mT9g", "summary": "This paper introduces MC-SEARCH, a comprehensive benchmark designed to evaluate and advance agentic multimodal retrieval-augmented generation (MM-RAG) systems, particularly focusing on structured, stepwise long-horizon reasoning. MC-SEARCH comprises 3,333 high-quality, HAVE-verified examples encompassing five distinct multi-hop reasoning topologies (including serial, parallel, and cross-modal forks), each annotated with granular sub-questions, modalities, evidence, and intermediate answers. The paper also proposes novel process-level evaluation metrics that go beyond traditional answer accuracy, enabling fine-grained analysis of stepwise retrieval, planning, and reasoning. Extensive benchmarking is conducted on six leading MLLMs, and a process-supervised fine-tuning scheme (SEARCH-ALIGN) leveraging the new dataset is presented, showing notable improvements for open-source models.", "questions": "- I’m a bit confused about why the reasoning process is represented as a graph rather than a chain.\n\n- Why didn’t you include ROUGE and MRFS scores [1] in your evaluation?\n\n- Why didn’t you incorporate retrieval-related metrics in your benchmark analysis?\n\n- How does your method Search-ALign perform compared to other RL-based agentic optimization approaches?\n\n\n\n[1] Pan Z, Luo H, Li M, et al. Chain-of-action: Faithful and multimodal question answering through large language models[J]. arXiv preprint arXiv:2403.17359, 2024.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Rigorous Dataset Construction:** MC-SEARCH addresses a clear gap in existing multimodal RAG benchmarks by providing long, structured, and verified reasoning chains spanning five rich reasoning topologies. The dataset construction meticulously filters for non-redundancy and necessity at each reasoning step using the HAVE protocol, resulting in high annotation quality\n\n**Process-Level Evaluation:** The paper moves beyond answer-level benchmarking by introducing stepwise “Hit per Step,” Rollout Deviation, and LLM-as-a-Judge metrics, directly quantifying the chain of reasoning, retrieval fidelity, and error types. This is a step change from prior black-box evaluation paradigms.\n\nNew Method: They propose SEARCH-ALIGN, a process-supervised fine-tuning framework for MLLMs that leverages verified reasoning chains to align model behavior", "weaknesses": "- Although the model coverage is broad, several open-source baselines used are not state-of-the-art for their respective modalities, which may overstate SEARCH-ALIGN’s improvements. The paper does not sufficiently justify the exclusion of other competitive open-source MLLMs or recent RAG architectures such as UniRAG and MIRAGE from direct comparison.\n\n- While Table 3 highlights clear performance gains from SEARCH-ALIGN, the contribution of individual components—planning, retrieval, and modality selection—is analyzed only qualitatively. A more rigorous ablation is needed to determine whether the model genuinely learns better planning strategies or simply imitates supervised reasoning steps.\n\n- Given the use of LLMs for data construction, filtering, and evaluation, to what extent might there be annotation artifacts or overfitting to the verifier’s error modes? How do the authors ensure the robustness of results in light of potential circularity?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)