MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval
Authors: Siyue Zhang
Yuan Gao
Xiao Zhou
Yilun Zhao
Tingyu Song
Arman Cohan
Anh Tuan Luu
Chen Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Retrieval Benchmark
Reasoning
Multimodal LLMs
Primary area: datasets and benchmarks

Abstract: We introduce MRMR, the first expert-level multidisciplinary multimodal retrieval benchmark requiring intensive reasoning. MRMR contains 1,435 queries spanning 23 domains, with positive documents carefully verified by human experts. Compared to prior benchmarks, MRMR introduces three key advancements. First, it challenges retrieval systems across diverse areas of expertise, enabling fine-grained model comparison across domains. Second, queries are reasoning-intensive, with images requiring deeper interpretation such as diagnosing microscopic slides. We further introduce Contradiction Retrieval, a novel task requiring models to identify conflicting concepts. Finally, queries and documents are constructed as image–text interleaved sequences. Unlike earlier benchmarks restricted to single images or unimodal documents, MRMR offers a realistic setting with multi-image queries and mixed-modality corpus documents. We conduct an extensive evaluation of 4 categories of multimodal retrieval systems and 14 frontier models on MRMR. The text embedding model Qwen3-Embedding with LLM-generated image captions achieves the highest performance, highlighting substantial room for improving multimodal retrieval models. Although latest multimodal models such as Ops-MM-Embedding perform competitively on expert-domain queries, they fall short on reasoning-intensive tasks. We believe that MRMR paves the way for advancing multimodal retrieval in more realistic and challenging scenarios.

Published as a conference paper at ICLR 2026
MRMR: A REALISTIC AND EXPERT-LEVEL MUL- TIDISCIPLINARY BENCHMARK FOR REASONING- INTENSIVE MULTIMODAL RETRIEVAL
Siyue Zhang NN NS Yuan Gao
Xiao Zhou
Yilun Zhao Y
Tingyu Song
Arman Cohan
Anh Tuan N
Chen Zhao SC
Nanyang Technological University
Yale University
NNYU Shanghai
Shanghai Jiao Tong University Universityy of the Chinese Academy of Sciences
Center for Data Science, New York University VinUnnivrssity
ABSTRACT
We introduce MRMR, the expert-leee multidisciplinary multimodal retrieval benchmark requiring intensive reasoning. MRMR contains 1,435 queries span- ning 23 domains, with positive documents carefully verified by human experts. Compared to prior benchmarks, MRMR introduces three key advancements. First, it challenges retrieval systems across diverse areas of expertise, enabling fine-grained model comparison across domains. Second, queries are reasoning- intensive, with images requiring deeper interpretation such as diagnosing micro- scopie slides. We further introduce Contradiction Retrieval, a novel task requir- ing models to identify conflicting concepts. Finally, queries and documents are constructed as image-texi interlcaved sequences. Unlike earlier benchmarks re- stricted to single images unimodal documents, MRMR offers realistic set- ting with multi-image queries and mixed-modality corpus documents. We con- duct an extensive evaluation of 4 categories of multimodal retrieval systems and 14 frontier models on MRMR. The text embedding model Qwen3-Embedding with LLM-geerrtted image captions achieves the highest performance, highlight- ing substantial room for improving multimodal retrieval models. Although lat- est multimodal models such as Dos-MMM-Emeeddingg perform competitively on expert-domain queries, they fall short on -intenniii tasks. We believe that MRMR paves the way for advancing multimodal retrieval more realistic and challenging scenarios."
INTRODUCTION
LLM-based agents, such as DeepResearch (OpenAI, 2024; Qiao et al., 2025), have been widely applied in domains including science, engineering, medicine, and finance (Zhao et al., 2025; Tang et al., 2024; Barry et al., 2025; Phan et al., 2025). These systems move beyond the intrinsic knowl- edge of L.MMs by actively retrieving and integrating external information, making a strong and robust retrieval component essential (Chen et al., 2025). In practice, many expert-domain applications rely on multimodal information, underscoring the need for retrieval methods that can handle queries and documents spanning both visual and textual modalities, or even interleaved image-texi content (Zhang et al., 2021: Liu al., 2021; 2023). For instance, given medical image, the agent system should retrieve similar cases or guidelines to support clinical decisions.
While existing multimodal retrieval benchmarks have made progress, they are insufficient to capture the complexity of agentic scenarios. We identify three key limitations: (1) Multidisciplinary expert domains: most multimodal benchmarks are built on Wikipedia text and images, focusing on general- domain knowledge (Hu et al., 2023; Chen et al., 2023; Zhang et al., 2025b). However, state-of-the- art LLMs already demonstrate strong capabilities handling such knowledge (Team et al., 2025), making it essential to develop benchmarks for high-stakes expert domains such as medicine, science,
Our data available https: htw.ghuggnngggg aaef.ce...o co/ datasets http///wwweeeea.....e.e

---

Published as a conference paper at ICLR 2026
Medicine Diagnostics
Science Agriculture
Literabure
Music
Task: Retrieve relevant documents
Task: Retrieve relevant documents
Pharmacy
that help answer the question.
that help answer the question.
Design
Diagnosties
Humanitee
Query: What is the diagnosis for <image 1>
Query: simage
Art Theory
I> What causes
Ciinicl Meatcing
black bulges
Medictne
Art
corn cob?
besic Medicine
Agricuiture
Sclence
Thearom
Cantradiciin
Negation
Theorem- Physics
Contradiction Vehicle Design
Vonicte Design
Task: Retrieve relevant theorems that help solve the problem.
Task: Retrieve design rules that this velicle design is not complaan with.
Geography
Traffic Case
Chemistry
Business
Query: Gate AB
Query: Attached an image that shows
aiology
Engineering
<image 1> 1.2
Phycics
long and 0.8 m compute the the galle.
engineering drawing of the vehicle <amage
Waths
PE
examples, covering 23 domains across 6 disciplines. It is specifically designed to assess multimodal retrieval tasks. Notably, we originally introduce the Contradic- tion Retrieval task in the multimodal setting, which requires retrieving documents that conflict with the user query and features deeper logical reasoning.
Figure 1: Overview of the MRMR benchmark. MRMR includes 1,435
models in expert-level, reasoning-intense
and engineering. (2) Reasoning intensity: existing benchmarks primarily target semantic matching and informnaioon-eckinng tasks, whereas real-world queries often involve expert-domain images and require deeper understanding and logical reasoning over them. (3) Image-text interleaving: prior benchmarks mostly support single-image queries with supplementary text, yet real-world queries and documents typically consist of interleaved text and multiple images (Zhang et al., 2025b).
To address these gaps, we introduce MRMR, comprehensive benchmark measuring retrieval models in expert-level Multidisciplinary and Reasoning-intensive Multimodal Retrieval. Figure presents an overview of our benchmark. MRMR consists of 1,435 expert-annotated examples, cate- gorized into three types of retrieval tasks: (1) Knowledge for retrieving web pages related to queries involving multiple expert-domain images; (2) Theorem for retrieving theorems involved in solving multimodal math problems; and (3) Contradiction for retrieving contradictory statements or rules given a case description. Specifically, we derive complex multidisciplinary queries from established Visual Question Answering (VQA) benchmarks (Yue et al., 2024; 2025) and assign expert annota- tors to collect positive documents from Internet. To build a sizable corpus, we additionally include negative documents from knowledge-inteee intggsvve collections (Wang et al., 2024a; Su et al., 2025). To further elevate the reasoning challenge, we originally introduce Contradiction Retrieval, which re- quires models not only to detect semantic relevance but also perform logical reasoning identify conflicting concepts. To foster a deeper integration visual and textual content, we represent both queries and documents in interleaved multimodal format.
We conduct an extensive evaluation on MRMR across four main categories of multimodal retrieval approaches and 14 representative models. The results reveal that current multimodal retrieval systems consistently underperform text-only retrievers with image captioning on knowledge- and reasoning- intensive multimodal queries. The highest score of 54.1 achieved by the text embed- (Zhang et al., 2025d) combined with -ased image captioning. The perffrrrfonnng multimodal model, Ops-MM Ops-MIM-Embeddin (OpenSearch-AN 2025), trails by 6.0 points, mainly due to its limited reasoning capabilities rather than domain expertise. Its performance drops from 67.4 on Knowledge tasks to 37.4 and 36.6 on Theorem and Contradiction tasks, even though the corpora for these two tasks are much smaller than that of Knowledge. More importantly, the multidisciplinary setup in MRMR reveals substantial performance differences across models and domains. For instance, Ops-MM-Embeddinn surpasses the second-ber best model, MM-Embed (Lin et al., 2025). in the Art discipline, whereas their performances are comparable the Medicine discipline. We hope our benchmark and findings will help progress in multimodal retrieval.
insteningintensivee ding model
2

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of multimodal retrieval benchmarks and MRMR. In the "Modality" column, "T + I" indicates retrieving image documents using a text query. The "#Domain" column reports the number of domains; "Open" denotes datasets built from Wikidata with general domains. The "Ex- pert", "Reason", and "Interleaved" columns indicate whether expert knowledge is required, whether intensive reasoning is involved, and whether data are the interleaved image-text format.
Benchmarks
Modality
Retrieval Type
#Domain
#Query
Expert?
Reason?
Interleaved?
NIGHTS
I-1
Visual Similarity
Open
20K
SciMMIR
T+1
Image Caption
530K
EDIS
T- IT
Image Caption
Open
3,241
X
X
Wiki-SS
T-I
Document QA
Open
3,610
X
X X X X X X X X
WebQA
Document QA
Open
2,511
X
ViDoRe
T- IT
Document QA
10
3,810
X
MMDoc1R
T- IT
Document QA
10
1,658
X
FashionIQ
-
Composed Image
12,238
X
CIRR
IT -
Composed Image
Open
4,148
X
CIRCO
IT -+1
Composed Image
Open
1,020
InfoSeek
IT - IT
VQA
Open
1.35M
OVEN
IT - IT
VQA
Open
18,341
X
wikiHow-
IT IT
VQA
Open
7,654
MRMR
IT + IT
VQA
23
1,435
2
RELATED WORK
Benchmarking multimodal retrieval. As illustrated Table 1, existing multimodal retrieval datasets mainly focus semantic matching information-seeking tasks. Early semantic matching benchmarks are built from paired image-tex text data, where the text is semantically aligned with the image (Liu et al., 2023; Wu al., 2024; Xiao al., 2025: Jiang et al., 2025c), and the task is to retrieve the corresponding modality. Composed Image Retrieval (CIR) emerges as challenging task that allows users to search for target images using multimodal query, comprising reference image and modification text specifying the user's 's desired changes to the reference image (Zhang al., 2021: Liu al., 2021; Baldrati al., 2023; Zhang et al., 2024). Information-seeue seeionngn-eeigggn bench- marks either retrieve supporting evidence for visual questions (Hu et al., 2023; Chen et al., 2023) or retrieve multimodal documents for textual queries (Ma al., 2024; Macé et al., 2025; Dong al., 2025). As all prior studies focus single-image inputs. TIR (Zhang et al., 2025b) proposes more realistic setup which the query and document consist of interleaved text-image sequences sup- porting multiple images. However, it is limited to searching general-domain wikiHow tutorials. To further advance multimodal retrieval, we construct MRMR, the first benchmark comprising complex multidisciplinary queries that require in-depth reasoning in the interleaved text-image format.
Multimodal retrieval models and multimodal retrieval augmented generation. State-of-the- art multimodal retrieval models commonly rely on large pre-trained encoders such CLIP (Rad- ford et al., 2021) and BLIP (Li et al., 2023), which map images and texts into a shared embedding space. Their outputs often combined using fusion strategies (e.g., score fusion) integrate in- formation across modalities (Wei et al., 2024). More recent works adapt multimodal large language models (MLLMs) for universal multimodal embeddings by fine-tuning them diverse retrieval tasks (Jiang et al., 2025b; Zhang et al., 2025c; Jiang et al., 2025c: Lin et al., 2025). In these ap- proaches, multimodal queries are processed through the MLLM, and the hidden states from the final transformer layer, typically the last token representation, are used as the dense embedding for retrieval. In this work, we benchmark a diverse set of multimodal retrieval approaches, including text retrievers with image captioning, text and image two-stream models with vector fusion, and multimodal retrievers. Additionally, thanks to advances in both retriever and generative models, multimodal retrieval-augmennee generation (MM-RAG) has emerged as a key application (Hu et al., 2025; Jiang et al., 2025a; Wu et al., 2025b; Zhan et al., 2025; Wasserman et al., 2025). While vari- MM-RAG benchmarks have been introduced, most focus evaluating response generation and lack evidence- relevance annotations, making impractical to assess retrieval performance and its contribution within MM-RAG (Chen et al., 2025).

---

Published as a conference paper at ICLR 2026
Table Data statistics of MRMR. For each dataset, we show the number of queries (Q) and docu- ments (D), the average number of positive documents (D+) per example, the average number of text tokens of queries and documents (measured by the GPT-2 tokenizer (Radford al., 2019), not cluding task instruction text), the average number of images in queries and documents, and sources of queries and documents. Knowledge datasets share common retrieval corpus, while Theorem datasets share another. Examples for each dataset can be found in Appendix G.
Total Number
Avg. #Text
Avg. #lmages
Source
Ex.
Dataset
Q
D
D
Q
Q
D
Knowledge
Art
157
26,223
1.8
15.4
421.6
Fig.
MMMU-Pro knowledge question
Medicine
167
26.223
2.2
32.0
421.6
PIN-14M. Web pages
Fig.
Science
137
26,223
1.8
32.1
421.6
0.72
Fig.
Humanities
26,223
1.9
54.5
421.6
0.72
Fig.
Theorem
Math
60
14,257
2.1
64.6
364.3
1.1
0.001
Fig.17
MMMU-Pro calculation question
Physies
104
14,257
2.1
56.2
364.3
1.0
0.001
BRIGHT, Web pages
Fig.18
Enggnnering
190
14,257
2.0
53.5
364.3
1.0
0.001
Fig.
Business
158
14.257
3.2
64.2
364.3
1.0
0.001
Fig.20
Contrudiction
Negation
200
4
1.0
0.0
12.8
1.0
0.00
COCO DesignQA
Synthetic Design Rules
Fig.21
Vehicle Design
88
700
1.0
152.5
107.5
1.0
0.04
Fig.22
Driving Handbook
Traffic Case
80
796
1.8
19.5
Synthetic
Fig.23
Reasoning-intensive retrieval. Beyond keyword- and semantic-based information retrieval, BRIGHT (Su et al., 2025) has introduced the first benchmark in the text domain that requires inten- sive reasoning to identify relevant documents. For example, given new math or physics problem, the retrieval system is expected to provide previously solved problems using same theorems or relevant theorem statements. To tackle this challenge, recent methods train the text retrievers using synthetic datasets containing complex queries and hard negatives (Weller et al., 2025; Das et al., 2025; Zhang et al., 2025a; Long et al., 2025; Shao al., 2025; FlagEmbeddding 2025). Our work extends reasoning-intensse ing expert-level queries from the multimodal understanding and reasoning benchmark MMMU-Pro (Yue et al., 2025), collecting image-text interleaved documents from web pages, and obtaining rele- vance annotations from human experts.
retrieval into the multimodal domain. MRMR constructed by sourc-
MRMR BENCHMARK
3.1 TASK FORMULATION
We define the task of multimodal retrieval as follows. Let Q 1q1....,44 In) be the set of queries and D {dz...., dm) the document corpus. Each query g and document d is represented sequence of segments (ZI..... p&), where each segment æ can be either text or an image. For query y, a document can be either positive document d., (relevant) or a negative document d- (non-relevant). In reasoning-intensive retrieval, a document is considered relevant if it provides principles or theorems that support the reasoning chain required to answer query (Su et al., 2025). Unlike prior studies (Xiao et al., 2025; Dong al., 2025), we do not constrain the corpus to uniform data types, reflecting more realistic retrieval scenarios. To evaluate diverse reasoning capabilities, design three types of retrieval tasks in MRMR:
Knowledge. emphasizes reasoning over broad expert domain knowledge. For multimodal query, a document is relevant if expert annotators confirm that it contributes to reasoning about the query by providing critical concepts theoretical foundations.
Theorem. It targets the theorem-base based reasoning over calculation problems. For multimodal calculation query, a document relevant if it conveys the same underlying theorem or formula needed to solve the problem.
4

---

Published as a conference paper at ICLR 2026
Contradiction. It requires logical reasoning detect conflicting inconsistent concepts. For multimodal case description query, a document is relevant it provides the rule or requirement that the query violates.
3.2 KNOWLEDGE: RETRIEVING WEB PAGES THAT HELP ANSWER QUESTIONS
MMMU (Yue et al., 2024) is of the most widely used benchmark for evaluating multi-discipline multimodal understanding in MLLMs. Its robust version, MMMU-Pro (Yue et al., 2025), excludes questions solvable by text-only models, expands the candidate options, and provides verified correct answers. We repurpose the knowledge- and reasoning intensiive questions in MMMU-Pro as queries Q and construct a corpus D of image-text interleaved documents. The positive documents D+ are scraped from relevant websites referenced by the GPT-Search" model (OpenAI, 2024) and verified by human experts; while negative documents D_ are augmented by sampling from the multimodal collection PIN-14M (Wang al., 2024a) (see Figure 2).
Selecting questions. We prompt GPT-5² to categorize MMMU- questions into two groups, i.e,, knowledge-based and calculation questions. We adopt calculation questions for the Theorem subset Section 3.3. For knowledge questions, we then instruct GPT-5 filter out questions that require only superficial reasoning over text and images, without reliance on external domain expertise. For the remaining questions, we generate detailed descriptions for each associated image using GPT-5, which we include part of the input context for subsequent steps.
Constructing positive and hard negative documents. Unlike keyword- or semantic-based mul- timodal retrieval benchmarks, collecting positive documents for our queries is more time-consuming because requires identifying and validating multimodal sources that support the query's answer. To address this, design semi-automated pipeline with human expert annotators. Specifically, for each query, given the GPT-5-generated image descriptions and ground-truth answer, we prompt GPT-Search reason over the question and generate explanation for the correct answer with reference web links pointing to diverse materials such as Wikipedia, books, academic papers, and blogs. To preserve the completeness of multimodal content, we capture these webpages as PDFs, apply MonkeyOCR (Li et al., 2025) to extract interleaved text and images, and split the content into chunks while preserving image references. Resulting documents are then screened by GPT-5 and validated by human experts about whether they support the correct answer. Documents with GPT- human agreement on relevance are retained as positives, those agreed irrelevant hard negatives, while ambiguous cases 600% across domains) are discarded. In cases where GPT-Search fails retrieve relevant documents (38.2% of questions), expert annotators are instructed to search the web and create one supporting document, optionally including image links within the text. Due to the complexity of the questions, the number of positive documents per query is typically fewer than four. We annotate data anonymously through the Turkle platform (HLT-COEF CHU,JHU, 2025), with detailed guidelines provided in Appendix B.
Constructing additional negative documents. After the previous step, we obtain 993 cleaned and annotated documents for 555 queries. To construct a sizable retrieval corpus comparable to (Xiao et al., 2025; Su et al., 2025), we supplement these with negative documents sampled from the large-scale multimodal collection PIN-14M (Wang et al., 2024a), which contains knowledge- intensive resourees such medical articles from PubMed Central (PMC)* and web content from OBELICS (Laurençon et al., 2023). Given the wide topic coverage and large number of documents in PIN-14M assume low probability of false negatives for sampled documents. We validate this assumption through manual error analysis in Section 5. In total, we curate a corpus of 26,223 documents, including text only, image only, and text-image interleaved.5
GGPT-Search refers to version
00-111133111 throughout this work. 'GPT-5 refers the version 1ottt---225--0-2-- throughout this work.
'https: / www. nobi. m1m. n h. ppov/pmc.
'The corpus could he further expanded by sampling additional expert-domain documents, which naturally increases retrieval difficulty and the probability of false negatives. We leave future work.
5

---

Published as a conference paper at ICLR 2026
Knowledge-
Relevant Pages and Annotate Positive Documents
(3) Supplement Negative Documents
Reasoning-intemsin Multimodal Query
MMMU-Pro
SearchGPT
Cleaning Chunking
Positive Docs
Armotated Web Pages
subtraction <image 1s, 12 weighted <image 2> and weighted axial screening ng breast MRL What the etiology of the finding in the breast?
Fat necrosis is a benign condition that that occurs when fatty breast [Web] signal, making the necrotic area stand
Human Crested Docs
Judge
<image 1>
...
Relevance
Negative Docs
Review Relevance
Cyst
B: Fat necrosis
out [Web] ...
Arnotated Web Pages
C: Lipoma
D: Fibroadenoma
K Multimodal Collections PIN-14M
<image 2>
Wikiped
Book
Paper
Blog
Figure 2: An overview of the data construction workflow for MRMR (Knowledge). We select and convert knowledge- and sessoiing-nnteniii questions from MMMU-Pro (Yue et al., 2025) into retrieval queries. Web pages such as Wikipedia, blogs, and papers referenced by the GPT-Search model during reasoning are processed into documents through screen capturing, OCR (Li et al., 2025), and chunking. The relevance of resulting documents is first evaluated by GPT and then verified by expert annotators. Lastly, we source negative documents from the knowledge-intenss multimodal collection PIN-14M (Wang et al., 2024a) to construct sizable corpus.
3.3 THEOREM: RETRIEVING RELEVANT THEOREMS THAT SOLVE PROBLEMS
As introduced by Su et al. (2025), retrieving relevant theorem statements can assist users in solving new math or physics problems. We extend this formulation to the multimodal domain by leveraging challenging calculation problems from MMMU-Pro. calculation problem, and the corpus D consists of theorem descriptions across domains such as mathematics, physics, engineering. and business. A document d regarded as positive if describes a theorem applicable to solving the query problem.
this setting, the query q is a image-centric
Selecting questions. From the calculation questions in MMMU-Pro, we first use GPT-5 exclude questions that explicitly state the required theorem in the text. The remaining questions then organized into four major domains: Math, Physics, Engineering, and Business. The Engineering domain further includes areas such Mechanical Engineering, Computer Science, and Electronics, while the Business domain covers Finance, Economics, Marketing, and related fields. Then, we prompt GPT-5 to reason through each multimodal question, produce a final answer, and summarize the key theorems used in the solution. We exclude questions for which GPT-5 produces incorrect answers, with final of 512 questions.
Constructing positive and negative documents. We adopt the theorem statements from BRIGHT (Su et al., 2025) as the primary retrieval corpus (~13.8k documents), reflecting the realistic set- ting where most theorems are expressed in text. For each question, the summarized key theorems are used as queries to retrieve the top-10 candidate statements from the corpus with the Qwen3- Embedding model (Zhang al., 2025d). Among these candidates, GPT-5 identifies the most rele- vant theorem statements, which are retained as positive documents, while the rest serve as negatives.
Constructing additional positive documents. Not all theorems have relevant counterparts in BRIGHT. To address this, we scrape additional theorem statements, optionally accompanied by illustrative images, from webpages such as Wikipedia, following the OCR pipeline described in Section 3.2. GPT-5 then rewrites these theorems to match the format of BRIGHT statements. Fi- nally, we deduplicate the scraped documents to ensure consistent and complete retrieval corpus. Consequently, 63.6% of the positive documents are sourced from webpages, with the remainder drawn from the BRIGHT corpus. More details are presented in Appendix C.
3.4 CONTRADICTION: RETRIEVING CONTRADICTORY RULES AND REQUIREMENTS
Most existing datasets emphasize retrieving positively supporting evidence for query (Xiao al., 2025; Chen et al., 2023; Dong et al., 2025). However, retrieving contradictory information could be of great importance especially in expert domains. For example, user may provide case description

---

Published as a conference paper at ICLR 2026
and seek evidence of violation of laws, policies, guidelines, shown in Figure 23. In this setting, the query q is a case description (e.g.. traffic or vehicle design cases), while the corpus D comprises mandated rules (e.g., driving theory handbooks design requirements). A document d is considered positive it contains the statement or rule contradicting the query case. Unlike traditional retrieval tasks, this new formulation requires not only semantic matching between query and document but also deep logical reasoning to identify conflicting concepts.
Negation. study contradiction retrieval, we first design a synthetic task inspired by the negation benchmark NegBench (Alhamoud et al., 2025). Given an image from COCO (Lin et al., 2014) with ground truth object annotations, we synthesize four candidate text descriptions: three accurately reflecting the objects and one containing contradiction, either by asserting the existence of a non- existent object or the absence of an existent one. The models are required to pinpoint the text description conflicting with the given image in a multi-choice setup. For example, in Figure 21, the query image shows a keyboard on the table, while the positive document explicitly states that none present, revealing contradiction. More details are provided in Appendix D.I.
Vehicle Design. To evaluate contradiction retrieval in engineering documents, we construct a ve- hicle design task by leveraging the Formula SAE Rulebook and design cases from the DesignQA dataset (Doris et al., 2025). In industrial product design, designers must review hundreds of pages of requirement documents to ensure their designs comply with specifications. To assist designers, retrieval systems are expected to identify the specific sections that a design case fails to satisfy. For example, in Figure 22, the vehicle's 's wheelbase in the design is shorter than the required minimum, indicating a contradiction. During data preparation, chunk the lengthy design document, as detailed in Appendix D.2.
introduce variations to the design cases and
Traffic Case. Retrieval systems have been applied to legal documents to assist legal profession- als in preparing arguments and citations (Feng et al., 2024). To evaluate this capability in multi- modality, we construct a traffic case task to assess whether retrievers can identify which driving rules violated in traffic cases. We build the corpus by chunking official driving handbooks (Sin- gapore Police Force, 2017) into sections. Meanwhile, we build the query set by selecting dozens of driving rules, each linked to several annotated violation cases. We augment these violation cases by replacing key textual elements with Al-generated images using Owen-Image (Wu et al., 2025a). For example, as shown in Figure 23, car is driving only meters behind the vehicle ahead significantly less than the required safe distance. Further details are provided in Appendix D.3.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We evaluate types of multimodal retrieval setups with 14 frontier models, as follows: (1) Text models with image caption (T2T): We assess text retrievers, namely BGF-M3 (Chen et al., 2024), NV-Embed-V2 (Lee et al., 2025), and
(Zhang et al., 2025d), by pairing with MLLM-generated image captions (see Appendix E. for details). (2) Text and image two-stream models with vector fusion (IT2IT): We evaluate CLIP-style two-stream models, including EVA- CLIP (Sun al., 2023), SigLIP (Zhai et al., 2023), OpenCLIP (Cherti al., 2023), and JinaCLIP (Koukounas et al., 2024), by simple vector-fusion strategy. Given an input sequence, we con- catenate all text chunks for one text embedding while all images are concatenated vertically for another image embedding i. Following MTEB (Xiao et al., 2025), the final score is computed using the fused embedding E + (3) Multimodal models with merged image (IT2IT): We evaluate multimodal retrievers including VISTA (Zhou et al., 2024), ES-V (Jiang et al., 2025b), MM-Embed (OpenSearch-Al,, 2025) and GME-Qwen2- VI. (Zhang et al., 2025c). Since these models support only single-image input, multiple images are concatenated in the same way as for two-stream models. (4) Multimodal mod- els with document as image (T2I): We also include the document retrieval paradigm that receives text-only query and encode entire multimodal documents as screenshot images, such as ColPali (Faysse et al., 2025). Because these models are trained for text queries, query images replaced with -egerraed captions, similar to the text retriever setup. Besides, we note that a native image-text interleaved model, THR (Zhang al., 2025b), has been introduced and is expected to
(Lin et al., 2025), VLM2Vec (Jiang et al., 2025c), OPS-MMMEmbeddinn
7

---

Published as a conference paper at ICLR 2026
Table The performance of retrieval models on MRMR. We report nDCG@ 10 for all subtasks ex- cept Negation, for which we use Hit@ I: Art, Medicine (Med.), Science (Sci.), Humanities (Hum.), Math, Physics (Phy.), Engineering (Eng.), Business (Bus.), Negation (Neg.). Design, and Traffic. Avg. denotes the average score across II subtasks. The best score on each subtask is highlighted in bold, and the second best is underlined.
Knowledge
Theorem
Contradiction
Avg.
Model
Art
Med.
Sci.
Hum.
Math
Phy.
Eng-
Bus.
Neg.
Design
Traffic
Text Models with Image Caption (T2T)
BGE-M3
48.6
30.0
42.4
45.6
16.5
19.5
21.6
16.0
25.9
17.4
29.3
NV-Embed--2
70.7
46.8
65.7
66.6
26.4
35.2
3299
52.2
12.5
42.1
42.2
44.8
Fomeedinn
71.9
53.2
72.5
74.4
37.7
50.2
42,9
58.3
12.0
67.8
54.2
54.1
Text and Image Two-Stream Models with Vector Fusion (IT2IT)
EVA-CLIP
10.2
13.5
26.1
12.9
12.2
10.7
17.4
4.4
5,4
SigLIP
26.7
12.3
12.5
13.5
4.9
OpenCLIP
56.0
22.0
6.6
7.3
14.0
13.0
8.1
12.4
18.0
JinaCLIP
21.4
16.8
27.1
10.7
10.9
7.5
13.7
10.5
16.5
9.7
14.0
Multimodal Models with Merged Image (IT2IT)
VISTA
21.3
27.8
32.6
17.0
18.8
17.1
17.3
28.6
20.0
20.2
9.4
20.9
ES-V
25.1
10.8
3.4
11.5
2.1
MM-Embed
65.6
53.0
63.5
62.8
23.6
30.8
27.4
44.9
23.8
34.9
39.8
VLM2Vec
53.5
22.4
36.7
24.0
2.1
2.8
2.8
2.9
11.5
5.6
18.3
18.1
CMF-Qwen2-VI
54.3
40.1
46.8
45.6
28.8
36.0
30.2
45.1
15.0
26.3
29.6
36.2
Ops-MM Op--MM-Embeee M-bedddddn
79.3
52.5
70.0
67.8
27.7
39.5
30.1
52.3
8.0
55.9
45.8
48.1
Multimodal Models with Document Image (T21)
GME-Qwen2-VL
$4.0
40.7
59.0
50.1
21.2
22.1
27.0
45.3
14.5
56.1
40.1
39.1
Dps-MM-Embedddne
67.7
48.8
67.7
63.9
25.0
34.0
29.2
49.0
10.5
59.8
46.3
45.6
ColPali
36.1
29.9
42.7
29.2
7.3
17.5
13.5
34.6
28.5
19.4
18.2
25.2
best fit the interleaved format of MRMR; however, it is not publicly available. We provide details of each model in Appendix 1. Following prior work (Xiao al., 2025; Su et al., 2025), nDCG@ 10 as the main evaluation metric except Negation. Since each query in Negation has exactly one gold document among four candidates, we adopt Hit@| the main metric for this task.
4.2 MAIN RESULTS
Multimodal retrieval systems lag behind text retrieval-based approaches on knowledge- and ceasoning-intensiin images. As shown in Table the text retriever Qwen3-Fmbeddding combined with LLM-based image captioning achieves the highest performance (54.1 nDCG@ 10). Although captions may omit certain visual details, they provide rich contextual descriptions and additional knowledge that substantially benefit retrieval. In contrast, multimodal systems struggle with the expert-level query images in MRMR, which often require deep reasoning, such diagnosing mi- croscopic tissue sections (Figure l). CLIP-style two-stream models are particularly limited, as their training emphasizes alignment of superficial text-image semantics and model sizes are relatively small. The most recent MLLM-based embedding models, such as Ops-MM--Embedee promising results under both interleaved text-image and document-as -s-iage paradigms, indicating the effecttivnness of unified training diverse retrieval tasks.
show
Multimodal retrieval systems perform particularly poorly on reasoning-intensive tasks. While Mi--MMM-mmbddinggg. achieves a solid 67.4 nDCG@ 10 on Knowledge subtasks, its perfor- mance drops sharply to 37.4 and 36.6 on Theorem and Contradiction, respectively. Models such as E5-V and VLM2Vec perform even worse, essentially failing on these tasks. This gap highlights the difficulty of extracting abstract concepts from practical problems, for example linking an image- based physics question to the relevant theorem in Figure 1. Notably, Hit I scores for most models on the synthetic Contradiction task Negation remain below 25% equivaleet to random guessing given four candidates per query. As illustrated in the Negation example Figure 21, humans can read- ily detect conflicting concepts embedded within supporting evidence, yet retrieval models struggle even for strong text embedding models. Although the candidate corpora for the Design and Traffic
8

---

Published as a conference paper at ICLR 2026
subtasks much smaller than those of standard knowledge bases (Su et al., 2025; Dong et al., 2025), models still struggle to identify the underlying contradictions. Nevertheless, surface-level semantic matching remains useful in these settings, as it allows models to locate relevant documents without fully resolving the conflicting concepts (e.g. a query concerning driving speed matched with document specifying the speed limit). These findings suggest that current retrieval models possess strong capabilities in semantic matching and information seeking, but remain fundamentally limited in their reasoning ability.
Substantial differences in performance are evident across models and domains. Across all four multimodal retrieval settings, we observe wide performance difference between models. For instance, among multimodal models with merged image, the weakest model, E5-V, achieves only 8.6 nDCG@ 10, whereas Ops-MM-Embedde ological differences. As MRMR is the first multidisciplinary multimodal retrieval benchmark, it en- ables fine- grne-graied domain- level evaluation. For example, as shown in the breakdown performance Table 7, MM-Embed performs competitively with Jps-MM-Embeddiigg in medical domains such as Clinical Medicine and Diagnostics, yet lags behind in art-related tasks. We also observe pronounced variation in retrieval difficulty across domains. In the Art subtasks, systems can often succeed by matching query images to visually identical or similar artworks, which narrows the search space. However, in medical imaging, such overlap is rare, and models are required to identify underlying pathological and radiological features rather than relying on superficial visual similarity.
reaches 4 nDCG@ 10, revealing substantial method-
ANALYSIS
5.1 QUALITATIVE ANALYSIS
To understand model limitations, we conduct 30 case studies by manually reviewing their top-10 10 documents retrieved by Ops-MM- Dra-MMMEmbedding. There are two major failure patterns that we have observed. (1) Visual bias over contextual relevance: in the Agriculture case as shown in Figure 11, the model ranks negative document higher because it contains nematode SEM image resembling the earthworm image the query, even though the positive document provides a detailed discussion of the key topic Fauna. Similar errors occur in Medicine, where visually similar eye images from different diseases mislead the model. (2) Failure of higher-level deduction: in the Traffic case as shown in Figure 12, the model assigns a higher score to a negative document than to positive one because both depict cars, tunnels, and lane markings. However, it fails to infer that the car is crossing the line, which contradicts the positive document's instruction to "Stay in lane". Although multimodal retrievers exhibit these shortcomings and lag behind text-only retrievers with image captions, we believe they remain essential because many real-world queries inherently span across modalities. Fundamentally, textual descriptions alone cannot fully capture the nuanced information in images, especially when MLLMs lack the required visual knowledge.
5.2 TEST-TIME SCALING IN RETRIEVAL
Query expansion is a widely used technique, recently framed as test-time scaling in retrieval (Shao al., 2025). Prior work (Su et al., 2025) demonstrates that incorporating explicit reasoning substan- tially improves performance on saaoning-intensiiv text retrieval tasks. Motivated by this, we have conducted comparative experiments evaluate the effectiveness for multimodal retrieval. Specif- ically, we prompt MLLMs, including Qwen2-VL-2B -n-tt---tttttt
Instruct (Wang et al., 2024b) and Qwen2.55 VL--72B-Instruut (Bai al., 2025), to generate reasoning traces, including question summarization and chain-of-thouglh reasoning, following (Su et al., 2025). As shown in Table 9, replacing the orig- inal queries with +5.1 for Qwen2-VL-2B and +14.8 for Qwen2.5-VL-72BB The improvements are particularly pro- nounced on Knowledge tasks, whereas Theorem tasks benefit to a lesser extent. Meanwhile, we produces
reasoning traces leads to substantial performance improvements:
observe that, without constraining output length, the larger model Qwen2.5- average 20% and 66% more tokens than Qwen2-VL-22 VL-2B in Knowledge and Theorem respectively, trading higher inference cost for larger performance gains (see more details in Appendix F.3).

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
We introduce MRMR, realistic, multidisciplinary. reasoning-intennn intensive multimodal retrieval bench- mark. We leverage knowledge- and rassoningiintensig questions from MMMU-Pro and build sizable multimodal corpus with positive documents verified by human experts. In addition, we introduce Contradiction Retrieval for evaluating models' logical reasoning capabilities to identify conflicts. Comprehensive evaluation shows that multimodal retrieval systems lag behind their text- retrieval counterparts, indicating substantial room for improvement. Although state-of of-the-art multi- modal models excel in Knowledge domains, they drop nearly 30 points on inssoninn-intessvee tasks. We hope MRMR facilitates identifying model limitations and advancing multimodal retrieval.
CODE OF ETHICS AND ETHICS STATEMENT
All data used in constructing MRMR sourced from publicly available materials and ployed solely for academic research, not commercial use. We have carefully ensured that the dataset contains private information harmful content, such discriminatory, violent, or unethical ma- terial. Our goal is to support socially beneficial research. Following the practice of MMMU (Yue al., 2024), our annotators and validators are instructed avoid using materials from websites that prohibit copying or redistribution when reviewing MRMR documents. Consequently, most doc- uments are derived from sources that are free of copyright restrictions, such as Wikipedia pages, government reports (e.g., National Institutes of Health and Singapore Police Force), and PubMed Central (PMC). The datasets we build upon also carry permissive public licenses, including MMMU (Apache-2.0)) PIN-I4M (CC-BY-4.0), COCO (CC-BY-4.0), and BRIGHT (CC-BY-4.0). For test- time scaling, we primarily focus on text expansion rather than image resizing and process as the text expansion has shown more significant impacts.
REPRODUCIBILITY
Our datasets and annotation process are introduced in Section 3, and the experimental settings are described in Section Specific implementation details are provided in Appendix E.1. To facil- itate the reproduction of our experiments, the data available https: btstps:/huugginn htt.g/ggggggacceeee////// co com/rebeccaz4//RR
and the evaluation code provided https: /
ACKNOWLEDGMENTS
This research is supported by the RIE2025 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) (Award 12301 administered by A*STAR, as well supported by Al- ibaba Group and NTU Singapore through Alibaba-NTU Global e-Sustainability CorpLab (ANGEL). Siyue Zhang and Chen Zhao were supported by NYU Shanghai Center for Data Science. This work was supported in part through the NYU IT High Performance Computing resources, services, and staff expertise.
REFERENCES
Kumail Alhamoud, Shaden Alshammari, Yonglong Tian, Guohao Li, Philip Torr, Yoon Kim, and Marzyeh Ghassemi. Vision-language models do not understand negation. In Proceed- ings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), thecvf.e.. cwm/ ceatteen//vRR222255255 LaaioonnLannuggee Models Bacii__D___MM__a Do_ NtttNootnndustaann Lww.eessttaddeegeaionnnnnn.oon CVPR 2025 waper.
2025. URL https::/
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu. Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. 13923.
URL https::/// // /arxiv org/
10

---

Published as a conference paper at ICLR 2026
Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. Zero-shot com- posed image retrieval with textual inversion. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2023. URL /ttps::/openaccess. theovf. Sorrattieerrr-shtttooooppsssssep
Mariam Barry, Gaetan Caillaut, Pierre Halftermeyer, Raheel Qader, Mehdi Mouayad, Fabrice Le Deit, Dimitri Cariolaro, and Joseph Gesnouin. GraphRAG: Leveraging graph-based ef- ficiency to minimize ballucinations in LLM-driven RAG for finance data. In Proceedings of the Workshop on Generative Al and Knowledge Graphs (GenAIK), 2025. URL https:
//aclanthologyy wttg:/org///20/ gena k...i//k.6.6...
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionallity, multi-granularity text embeddings through self-knowledge dis- tillation, 2024. URL https::tp/...m http///hhgggggca..ee/////// co/ /BAII AAAI/bbgmm3
Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-War Beig-Wei Chang. Can pre-trained vision and language models answer visual information-seeking questions? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: / aclanthlooyy org. /2233.222.22 monnpp-main in. 925/.
Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Sahel Sharifymoghaddam, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, and Jimmy Lin. Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent, 2025. URL https: htwpxxivvv.oo./ org/ abs /btp////....2 2508. 06600.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. URL https: //tthtttttug/gge///g www. httmmmfffoumm wowmm//oooddaiinnnoo.ooo open_clip.
Chroma. Chromadb: An open-source vector embedding database, 2025. URL https://
github. com/ cwwoca-coooooooooo www.ooaaacccreeeeeeeeeeee chroma. Apache 2.0 license.
Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Process- ing Systems (NeurIPS), 2022.
Debrup Das, Sam O' Nuallain, and Razich Rahimi. Rader: Reasoning- -warr dense retrieval models, 2025. URL https: //axivv. org/ /bss////////// abbs/2505.1. 18405.
arxiv.
Kuicai Dong. Yujing Chang, Xin Deik Goh, Dexun Li. Ruiming Tang, and Yong Liu. Mmdocir: Benchmarking multi-modal retrieval for long documents, 2025. URL peattss:/arxiv.... abs/2501 2501. 08828.
Anna Doris, Daniele Grandi, Ryan Tomich, Md Ferdous Alam, Mohammadmehdi Ataei, Hyun- min Cheong, and Faez Ahmed. Designqa: A multimodal benchmark for evaluating large language models' understanding of engineering documentation. Journal of Computing and Information Science in Engineering. 2025.
Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models. In International Conference on Learning Representations (ICLR), 2025. URL https:/ / /ppnrevieww / forum?id=ogjjp28888
Yi Feng, Chuanyi Li, and Vincent Ng. Legal case retrieval: A survey of the state of the art. In Pro- ceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Longg Papers), 2024. URL https: /att://acla /aclaaaannntooyoo.eccaaaaa wwwpannthhooooooogo///oe
www.ey.rrg////0 2024 scl--ong.... 3500
FlagEmbedding. Bge-reasoner: Towards end-to-end reasoning-intensiv information retrieval. /ettss//githhe cwm/ httppp///laaggbeddingggeeeeeeeee BGE, Reacnees,,er,, 2025. Accessed: 2025-09-12.

---

Published as a conference paper at ICLR 2026
Lixue Gong, Xiaoxia Hou, Fanshi Li, Liang Li, Xiaochen Lian, Fei Liu, Liyang Liu, Wei Liu, Wei Lu, Yichun Shi, Shiqi Sun, Yu Tian, Zhi Tian, Peng Wang, Xun Wang, Ye Wang, Guofeng Wu, Jie Wu, Xin Xia, Xuefeng Xiao, Linjie Yang, Zhonghua Zhai, Xinyu Zhang, Qi Zhang, Yuwei Zhang, Shijia Zhao, Jianchao Yang, and Weilin Huang. Seedream 2.0: A native chinese-english bilingual image generation foundation model, 2025. https : /ttl://aaxiie aaaaiiivvvv. wttpddv.or//bb///////
07703.
HLT-COE@JHU Turkle: An open-source clone of amazon mechanical turk, https: https://gáthub. com/h1
2025.
Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Wei Chang. Open-domain visual entity recognition: Towards recogniz- ing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. URL htttps://arxtta .rgg abs/ /2t///23.2222 11....1155. 1154.
Wenbo Hu, Jia-Chen Gu, Zi-Yi Dou, Mohsen Fayyaz, Pan Lu, Kai-Wei Chang, and Nanyun Peng- Mrag-bench: Vision-centric evaluation for retrieval-augmeented multimodal models. Proceedings of The International Conference Learning Representations (ICLR), 2025. URL https:// id-Uskk1i Usk.1i4MMe...
openreview www.riewwwee//oooeee.e
Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Chaoyou Fu, Guanglu Song, Peng Gao, Yu Liu, Chunyuan Li, and Hongsheng Li. Mmsearch: Benchmarking the potential of large models as multi-modal search engines. In International Conference on Learning Representations (ICLR), 2025a. URL https: /tce://arxee /ataivivvvv.///////eaa//aaa org/ abs/ 2409. 12959.
Ting Jiang, Shaohan Huang, Minghui Song, Zihan Zhang, Haizhen Huang. Liang Wang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, deqing wang, and Fuzhen Zhuang. E5-v: Universal embed- dings with multimodal large language models, 2025b. URL https: http::s////pnrevie... Forum?id=:D6L0agee
Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingho Zhou, and Wenhu Chen. V1m2vec: Training vision-languee models for massive multimodal embedding tasks. Proceedings of The International Conference on Learning Representations (ICLR), 2025c. URL https:
Andreas Koukounas, Georgios Mastrapas, Bo Wang, Mohammad Kalim Akram, Sedigheh Eslami, Michael Günther, Isabelle Mohr, Saba Sturua, Scott Martens, Nan Wang, and Han Xiao. jina-clip- v2: Multilingual multimodal embeddings for text and images, 2024. URL https:://aaa org/ org/abs// 2412. 2. 08802.
Hugo Laurençon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, Matthieu Cord, and Vic- Sanh. OBELICS: An open web-scale filtered dataset of interleaved image-text documents. In seirth- Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: wwwp//weeneeeeeeeeee/ee net / forum? 1d-SKN2hf1BII..
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Nv-embed: Improved techniques for training Iims generalist embedding models, 2025. URL https: http:///axvvv..o org/ abs /2s////055.0 8
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: bootstrapping language-iman -inaaege--mggeeaeoe pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference Machine Learning, 2023.
Zhang Li, Yuliang Liu, Qiang Liu, Zhiyin Ma, Ziyang Zhang, Shuo Zhang, Zidun Guo, Jiarui Zhang. Xinyu Wang, and Xiang Bai. Monkeyocr: Document parsing with a struecure-rccogniiioo-raala triplet paradigm, 2025. URL https: http://aaxvvvv.//oe abs 2 16.0666.000 05218.
Sheng-Chin Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping- Mm-embed: Universal multimodal retrieval with multimodal 1lms. In International Conference on Learning Representations (ICLR), 2025. URL https:/ /pwnrevieeeeeeeee////ee net forum?
id= 4540b21k2
12

---

Published as a conference paper at ICLR 2026
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Pi- otr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. Euro- pean conference computer vision, 2014. URL https: http://///oooaaaasttteeoo org htt.aetgee..///maa//////e images/ coco- pacco-pppr..... png.
Siqi Liu, Weixi Feng, Tsu jui Fu, Wenhu Chen, and William Yang Wang. EDIS: Entity-driven image search over multimodal web content. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, 2023. URL https: aaaaiiiivvv.....a org/ abs 2305. 13631.
Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. Image trieval on real-life images with pre-trained lision-ndd-aanguae models. In Proceed- ings of the IEEE/CVF International Conference URL hww.amaaeeeeeeaaaaeeeeeaee Lea__ooo__Raaa_a_eeaaaaaa Real Balfffe_mmma____e Liu Liu__mmage__a
Computer Vision (ICCV), 2021.
https: /ettps://ppeaacceett www.ss.......cc.
com/
Vision-and- Lag.oo--arr-LLauauaeeeeeeee
Meixiu Long, Duolin Sun, Dan Yang. Junjie Wang, Yue Shen, Jian Wang, Peng Wei, Jinjie Gu, and
Jiahai Wang. Diver: A multi-stage approach for reasoning-intension information retrieval, 2025.
URL https: /arxivv wwwp:iivvorrroooooooooooooooo abs/2508. 07995.
Xueguang Ma, Sheng-Chieh Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multimodal retrieval via document screenshot embedding. In Proceedings of the 2024 Conference Empir- ical Methods in Natural Language Processing, 2024. URL https: / aclanthoooggyy org/ 2024. emn 1-mml .
Quentin Macé, António Loison, and Manuel Faysse. Vidore benchmark v2: Raising the bar for org./
visual retrieval, 2025. URL http:////aax..oo/aa//2/55.. hltgg//b.///5.....//
171666
MediaWiki. Apissearch - mediawiki,, 2024. URL https://wwa mediawiki. wwdllawii.orrwww/// index. seadex.php?ittee 2025].
[Online; accessed 25-September-
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive text embed- ding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, 2023. URL https: /acl anthelogy. wrw.yorrg/2220 2023.
eacl-maing 148/.
OpenAl.
Introducing chatgpt search.
2024. Accessed: 2025-09-17.
OpenSearch-AL.
Openssaccaaiopp-mmmmmmmmdddeeeee //huggingface. http:///w...........aaoao//aaaae
2025.
URL https:
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, et al. Humanity's 's last exam, 2025, URL https: /ttp///axvvv////////a////e//aaa/ org/ /b50/
Zile Qiao, Guoxin Chen, Xuanzhong Chen, Donglei Yu, Wenbiao Yin, Xinyu Wang, Zhen Zhang, Baixuan Li, Huifeng Yin, Kuan Li, Rui Min, Minpeng Liao, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou. Webresearcher: Unleashing unbounded reasoning capability in long-horizon agents, 2025. URL https://araat
orgo abs//b////2 /200e///59..0 13309.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask learners. 2019. URL https: htt::///mmfc/arr/ wtwoell//agguuggeeeeeeeeeee anguage mode 1 l...des....f.............. .dff.
nwt/better-eeeeeeeeeeeeeeeeeeeee language mode 1. l
13

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamila Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Pro- ceedings of the 38th International Conference on Machine Learning (ICML), 2021. URL. https: happ:://procediiie ngs ml press v
httpr/222rrraaor221111 pdd..
Gerard Salton and Christopher Buckley. Tern-weighting approaches in automatic text retrieval. In- formation Processing Management, 1988. URL https: /ttw///www///// sciencedirect com/
Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan
Kian Hsiang Low, Sewon Min, Wen tau Yih, Pang Wei Koh, and Luke Zettlemoyer. Reasonir: Training retrievers for reasoning tasks. Proceedings of Conference on Language Modeling, 2025. URL https:ttph...mm //aaav/aaaivv.v/aa org. wwtp://gggbbbbaaaoeoo..o 205955
Singapore Police Force. Basic theory of driving, 2017.
UtL.ttps:/www.pplccc..u http:////w...ooaaa...aoaeaaannnn 20ecition pdd. Accessed: 2025-09-21.
"
Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang. Ruoxi Sun, Jinsung Yoon, Sercan O. Arik, Danqi Chen, and Tao Yu. Bright: A realistic and challenging benchmark for reasoning-intensive retrieval. In International Conference on Learning Representations (ICLR), 2025. URL https: // //openreview net./ forum? Sertumidd=ylee deymmm?idyyuccs331111
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale, 2023. URL https: //axivv www.orgg//bao.. abs//////////// 2303 15389.
Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. MedAgents: Large language models as collaborators for zero-shot medical reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, 2024. URL https: accccaaaaa aellanthhlogy org/2024. findings-aci....
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534 2025.
Junjie Wang. Yuxiang Zhang, Minghao Liu, Yin Zhang, Yatai Ji, Weihao Xuan, Nie Lin, Kang Zhu, Zhiqiang Lin, Yiming Ren, Chunyang Jiang. Yiyao Yu, Zekun Wang, Tiezhen Wang. Wenhao Huang, Jie Fu, Qunshu Lin, Yujiu Yang, Ge Zhang, Ruibin Yuan, Bei Chen, and Wenhu Chen. PIN: A knowledge-intensive dataset for paired and interleaved multimodal documents. 2024a. URL https::p/// htapps://hhuggaaa http///gggggggg..eege/ co/ co/datasets..eeecocccccossssssse /m- p/P PIN-14M.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024b. URL ht //apiiv/////ivaa//aa/aae/aa wwwpsiiiv.orrroooooooooe hbsgg//bb//45 12191 21.
org9abs/2499
Navve Wasserman, Roi Pony, Oshri Naparstek, Adi Raz Goldfarb, Eli Schwartz, Udi Barzelay, and Leonid Karlinsky. Real-mm-rag: A real-worke multi-modal retrieval benchmark, 2025. URL https: /ttp::///axvvvv. org/ abs 2502 12342.
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. In The European Conference on Computer Vision (ECCV), 2024. URL https: //ww: ecva .nt/ /eters/ sccv_2024 2ccw_2024/papers___ BaweksssEcV//ppppp /11.22
Orion Weller, Kathryn Ricci, Eugene Yang. Andrew Yates, Dawn Lawrie, and Benjamin Van Durme. Rankl: Test-time compute for reranking in information retrieval, 2025. URL https: //aax/iv.rr//ab///a/./.... 8418. 8.
14

---

Published as a conference paper at ICLR 2026
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang. Hao Meng. Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng, Lin Qu, Minggang Wu, Peng Wang, Shuting Yu, Tingkun Wen, Wensen Feng, Xiaoxiao Xu, Yi Wang, Yichang Zhang. Yonggiang Zhu, Yujia Wu, Yuxuan Cai, and Zenan Liu. Qwen-image technical report, 2025a. URL https: http://axxvvv./o/eee.o /bb// 2508. 02324.
Jinming Wu, Zihao Deng, Wei Li, Yiding Liu, Bo You, Bo Li, Zejun Ma, and Ziwei Liu. Mmsearch- rl: Incentivizing Imms to search, 2025b. URLhttpa.
www.i.org/aaa........ 2506. 20670.
Siwei Wu, Yizhi Li, Kang Zhu, Ge Zhang, Yiming Liang, Kaijing Ma, Chenghao Xiao, Haoran Zhang, Bohao Yang, Wenhu Chen, Wenhao Huang, Noura Al Moubayed, Jie Fu, and Chenghua Lin. SciMMIR: Benchmarking scientific multi-modal information retrieval. In Findings of the As- sociation for Computational Linguistics: ACL 2024, 2024. URL https: http://aalaattlllcccccc org/2024, wtt.ffiniiio findings-acl.77 746/.
Chenghao Xiao, Isaac Chung, Imene Kerboua, Jamie Stirling, Xin Zhang, Márton Kardos, Roman Solomatin, Noura Al Moubayed, Kenneth Enevoldsen, and Niklas Muennighoff. Mieb: Massive image embedding benchmark, 2025. URL /eltts://arxx.. org/ abs/ 2504.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. MMMU-pro: A more robust multi-discipline multimodal understanding benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), 2025. URL https:/
org/ 2
736/.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference computer vision, 2023. URL httpt:tt//// //apss://rxxivv www.wi...rrgggg.... wttpp/d/////////oo/ooee/e// 15343.
Zaifu Zhan, Jun Wang, Shuang Zhou, Jiawen Deng, and Rui Zhang. Mmrag: Multi-mode retrieval- augmented generation with large language models for biomedical in-context learning, 2025. URL https:/ / arxiv. / abs/ 2502 15954.
Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming- Miig-Wei Chang. Magiclens: Self-supervised image retrieval with open-ended instructions. In The Forty- first International Conference on Machine Learning (ICML), 2024. URL org/abs/ 22222443..0 19651..
org/ abs
Siyue Zhang, Yilun Zhao, Liyuan Geng, Arman Cohan, Anh Tuan Luu, and Chen Zhao. Diffusion vs. autoregressive language models: A text embedding perspective, 2025a. URL https:// arxiv.org/abs//250 15045.
Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Fashion iq: A new dataset towards retrieving images by natural language feedback. In The IEEE CVF Computer Vision and Pattern Recognition Conference (CVPR), 2021. URL https: / oppp/cccsssss...////pp///////ssa teeenacceess..eeef....e CVPR2021/papeze /ta////aaaaaaaaaaaeeaaaaaaaaaaaa Beaiiion_n_______ Le__AA/eeDaatttte__ttoooeaee thecvf wwt. content/ Images by Natural CVPR 2021 11121_papee____ pdf.
Xin Zhang, Ziqi Dai, Yongqi Li, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, Jun Yu, Wenjie Li, and Min Zhang. Towards text-image interleaved retrieval. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025b. URL https::///// wwweeannthlla.o ogy 2025 acl long.214/
15

---

Published as a conference paper at ICLR 2026
Xin Zhang, Yanzhao Zhang. Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang. Wenjie Li, and Min Zhang. Gme: Improving universal multimodal retrieval by multimodal 1lms. In The IEEE / CVF Computer Vision and Puttern Recognition Conference
httpa//n..rrroggggoooooaoaaaeeee /ttp////ww..co//ooot/t/t//////ee http:///connntttttccctccet CCVRR2025/ (CVPR), 2025c. URL.
www.:/ccccses.ccccccccoc.oceee com
Yanzhao Zhang, Mingxin Li, Dingkun Long. Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang. Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advanc- ing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025d. URL https: /arxiv.v.v.. org/ abs/2506 05176. 6.
Xuejiao Zhao, Siyan Liu, Su-Yin Yang, and Chunyan Miao. Medrag: Enhancing retrieval- augmented generatonn with knowledge graph-elicited reasoning for healthcare copilot, 2025. URL attg://b///// /202 04413.
https::/arxiv..org/
Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, and Yongping Xiong. VISTA: Visualized text embedding for universal modal retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), 2024. URL https: acl //a a wwnttholoy..rrg//200 200024accllonn -long.ong.... 175/.
16

---

Appendix Contents
A The Use of Large Language Models
18
Dataset Construction: Knowledge
18
B.1
Annotator Biography
18
B.2 Annotation Guideline and Interface
18
B.3 Data Annotation Payment
18
B.4 Dataset Construction Prompts
19
C Dataset Construction: Theorem
19
C.1 Theorem Database Construction
19
C.2 Wikipedia Content Processing Pipeline
19
C,3 Document Deduplication
19
Dataset Construction: Contradiction
20
D.I Negation
20
D.2
Vehicle Design
20
D.3
Traffic Case
20
E Experiment Details
23
E. Models and Instructions.
23
E.2
Implementations and Machines
23
E.3
Detailed Results
24
F
Analysis Details
25
F.1
False Positive and False Negative Analysis
25
F2
Error Case Studies
25
F3
Test-Time Scaling in Retrieval
29
Data Examples
30
17

---

A THE USE OF LARGE LANGUAGE MODELS
In this work, large language models (LLMs) are employed solely as tools for data generation, as described in the main paper. Importantly, parts of the manuscript generated by LLMs. Hence, there are no concerns of plagiarism or scientific misconduct related to text generation.
B DATASET CONSTRUCTION: KNOWLEDGE
B. ANNOTATOR BIOGRAPHY
The detailed biographies of the annotators involved in MRMR construction are presented in Table 4. All annotators are from universities ranked in the Top 500 the 2025 QS Global Rankings' and fluent in English. Annotators assess document-que queey relevance by judging whether document facilitates answering the query. To ensure quality, independent validators conduct an additional round of verification.
Table 4: Biographies of 24 annotators involved in MRMR construction (Author biographies hidden to protect identity confidentiality).
Year
Major
Assigned Subject(s)
Author?
Validator?
3rd year Undergraduate
Biological Engineering
Biology
1st year Master
Biological Engineering
Biology
1st year Master
Biomedical Lingineering
Biology. Pharmacy
X
2nd year Master
Bionedical Engincering
Biology. Pharmacy
X
X
1st year Master
Biomedical Engineering
Biology. Pharmacy
X
X
Ist year PhD
Chemistry
Chemistry
X
X
2nd year Master
Chemistry
Chemistry
X
3rd year PhD
Medicine
Basic Medicine
X
X
3rd year Undergraduate
Clinical Medicine
Clinical Medicine, Diagnostics
X
3rd year Undergraduate
Medicine
Basic Medicine
2nd year Master
Clinical Medicine
Clinical Medicine, Diagnosties
X
12
2nd year Master
Clinical Medicine
Clinical Medicine, Diagnosties
X
13
3rd Undergraduate
Pharmacology
Pharmacy
X
14
4th Undergraduate
Pharmacology
Pharmacy
X
15
1st year Master
Music
Music
X
16
1st year Master
Clinical Medicine
Clinical Medicine
X
17
1st year PhD
Sociology
Sociology. Psychology
X
1st year Master
Bioinformaties
Biology
X
2nd year PhD
Agricultural
Agriculture
Biosystems Engineering
4th year Undergraduate 3rd year Undergraduate
Literature
History, Literature Geography
X
21
Geography and Environmental Studies
x
22
4th year PhD
Computer Science Computer Science
23
4th year Undergraduate
24
3rd year Undergraduate
Electronic Engincering
B.2 ANNOTATION GUIDELINE AND INTERFACE
To facilitate data annotation, develop the following interface based on Turkle (HLT-COE @HDD@JHHH JHU, 2025), an open-source clone of Amazon's Mechanical Turk. The annotation guideline and interface detailed Figure 3. Figure and Figure
B.3 DATA ANNOTATION PAYMENT
The annotation and validation process for MRMR spanned three months. Each annotator was as- signed approximately 50 questions aligned with their academic major. After annotation, validators independently assessed the quality of the labels. We provided base rate of 7 USD per hour, with a quality adjustment of about 10%. On average, annotating a single question required 10 minutes, while validation took 4 minutes. This compensation scheme ensured that annotators received wages competitive with the average teaching assistant salary at their universities. To maintain a manageable workload and reduce pressure, recommended maximum of 10 questions per day.
18
Back to Appendix Table of Contents

---

Turkle
Loggedi ennotator Change Paawert
logout
Project tharmeyy mm0
Auto acsept
Skip Tack
Dypire
Document Relevance Verification
given
IDc detion Pharmacy.20 20
Al Explanation:
Quentions cumpunnd picture belom, ideasify fanutimall group and name the componnd, The red stoms repissent ceygen. vimago 1x <haage
compoind in quetton conts ns carbeay: funchoonal group, deneted es CDOss which a chanaderiatic carbesyfi aciss Thi fancliionll comiiio of cartor
(C=00 atteched addile
hydtood 20ng.
of this
propertes
pttos//cchmmmmyyo youk orgicarboyy ic-addd funttftnntionnone groups" utm, ooarnt----eettt
you given is
"Wrong"
Optiurs:
one supparting decument the last section.
At maggen attached neo ally!
B: CCOM add
gicepeckentyllin wither
dousle segen,, Ft 2112 uthanari ide
Corred Whrnn
OH methunal
C: C-O, proparene
H. Aldotioli caloony butanal keture, wnloonllaroeee sore
Inn, correct srower?
atudlre to teu dlly
Lnter the mecn snower (k-)) any sec..
direttty ether
Answer:
Figure 3: Annotation Interface Step 1: Question Understanding. Annotators first shown the question, associated images, candidate options, the correct answer, and an Al-generated expla- nation. The explanation is provided to aid understanding, though annotators are informed it may be incorrect. In this step, they judge whether the given answer is correct based on their own knowledge.
B.4 DATASET CONSTRUCTION PROMPTS
The dataset construction prompts are presented in Figure 6, Figure 7, Figure 8, and Figure 9.
C DATASET CONSTRUCTION: THEOREM
C.I THEOREM DATABASE CONSTRUCTION
The BRIGHT theorem corpus was embedded using Qwen3-Embedding (Zhang et al., 2025d) and indexed ChromaDB, which supports efficient semantic search via HNSW (Chroma, 2025). Each entry retains unique theorem em i and the original text, enabling fast, semantics-aware retrieval with full traceability to the source.
C.2 WIKIPEDIA CONTENT PROCESSING PIPELINE
We retrieved Wikipedia content by querying the MediaWiki Search API (MediaWiki, 2024) using theorem names as search keys. For supplementary sources in PDF format, we employed Monkey- OCR (Li et al., 2025) to convert scanned documents into Markdown. The resulting text was then processed through a structured extraction prompt (Figure 10) using GPT-5 to perform final cleaning, normalization, and precise theorem statement extraction.
C.3 DOCUMENT DEDUPLICATION
All theorems extracted from Wikipedia were deduplicated prior to inclusion in the corpus. Dedu- plication was performed in two stages: first by theorem name, and then by semantic content using TF-IDF- cosine similarity (Salton & Buckley, 1988). Specifically, we employed tFidEVectorizes rier to compute TF-IDF vectors for all theorem statements (Pedregosa et al., 101 followed by pairwise cosine similarity. Entries with near-identical content (cosine similarity 0.85) were collapsed into single representative instance.
19
Back to Appendix Table of Contents

---

Candidate Document Evaluation
"Reeoant" zoase cn couditbre dacunent toatt kninont: You reed evauste the cancabite cocamert aqplams
The why
Document
/aaticylionnaallddddd Ptamaayyaa cttpppptppppppp.pp.ggggg. Open lini In new 1ab EZ
Relevant Not Relevont
you find conee question image decument opp SYME here
Document
ssalicylloonhheldddoi Pharmacy doc2 split Open link in tab
Relevent
Not Relevent
some quection image deaument hype her
Document S:
/Matiizviionnalllleii Pharmary 28 rioct xpli 2,ng
Opan Evvess lab E
Relowant
Not
Figure 4: Annotation Interface - Step 2: Candidate Document Evaluation. After understand- ing the question, annotators are instructed to review candidate documents individually and judge whether each can facilitate correctly answering the question. Documents are shown image for- mat, with up eight candidates presented. Document relevance definition has been explained to annotators before the annotation process.
D DATASET CONSTRUCTION: CONTRADICTION
D.1 NEGATION
First, we randomly select 200 samples from the COCO (Lin al., 2014) dataset, each containing at least three positive objectives. For each entry, construct description using the template, "The image includes a, b, c, but no d." the positive description, we randomly select three positive objectives replace a, b, and and select negative objective to replace d. For the negative de- scription, we generate two variations: one where all four objectives (a, b, c, d) are selected from the positive objectives, and another where one of a, b, is replaced by randomly selected negative objective. The image from each sample is used as the query, and the three positive descriptions and one negative description are used the corpus. Finally, corresponding gold documents to ensure that the contradictory descriptions are identifiable by hu- mans, and revise any ambiguous queries for clarity. No LLM prompting involved in constructing the Negation task.
manually review the 200 queries and
D.2 VEHICLE DESIGN
On hand, construct the queries, design cases from the DesignQA dataset (Doris al., 2025) and augment them through appropriate modifications, such as altering numerical values and introducing variations in image elements. On the other hand, to construct the corpus, we apply MonkeyOCR (Li et al., 2025) to extract and segment the Formula SAE Rulebook into 700 files, organized by rule ID. Finally, we review all the queries ensure they represent incorrect designs.
D.3 TRAFFIC CASE
First, we select a set of traffic rules and, based on these rules, create traffic violation cases by crafting relevant stories. These stories are then used prompts generate 12 images for each story using
20
Back to Appendix Table of Contents

---

Create Your Own Document
-ithem dacanam abose rumiaar shoul
wleart the below
Enagos grovic thai intu troct ripod
-caly witte in English youron dueen for ror dentinn yourcn nre the sieort section and prapaephs from rtlpeelss websteeen etc Ue sene deno
Additional hora, Referenge
within
cimage
Relevant link for <Image 2+1
thii secorel webeent imagee Eittonel..
S.bmiit
Figure 5: Annotation Interface - Step 3: Create Relevant Document. If none of the candidate documents are deemed relevant, annotators are required to search for suitable web page and pro- vide the gold evidence content. They are encouraged to include images from the source, and the final document written an interleaved image-text format.
Your task is to determine whether a question with images requires expert knowledge, such as about a historical event, scientific concept, economic theory, medical disease. The last line of your response should be of the following format: "Result: YES OR NO" (without quotes). If the answer be ob- tained easily by reading the question text and image content without the need expert knowledge, say NO. Think step by step before answering. Here are some examples:
(example_] 11
(example_2
Now please determine whether this new question requires expert knowledge:
answer)
Figure 6: The prompt for determining whether the question is knowledge-based
GPT-5. Afterwards, we manually review all the generated images and use Doubao (Gong et al., 2025) to refine and enhance them for better clarity and relevance. Additionally, we leverage Doubao to generate specific objectives from the queries in order to construct image-text interleaved queries. For the corpus, we use MonkeyOCR to split Basic Theory of Driving and Final Theory of Driving (Singapore Police Force, 2017), two official driving handbooks in Singapore, into separate files, which are then organized and used the corpus. Finally, conduct manual review of all the queries, ensuring that any additional corpus IDs caused by excessive image details are properly incorporated into the queries.
21
Back to Appendix Table of Contents

---

Is the multimodal question testing theorem, formula, equation, or algorithm in domains such as physics, economics, finance, computer science, and math? Answer YES NO directly.
(quest investic_aande jin_and_answer"
Figure 7:
The prompt for determining whether the question is theorem-based.
Explain the answer the question in clear and detailed manner. Include citation web links to support the explanation relevant Wikipedia pages whenever possible. If Wikipedia page not available, use other reliable sources.
(quest and _o__ad__asseee answer)
Figure 8: The prompt for searching relevant web pages using GPT-Scarch.
Question:
(questicn and answer)
Document:
[document)
You a document analysis assistant. Your task is to determine whether the given document above answers the given question and supports the given answer. Instructions:
1. If the answer can be derived or inferred from the text and images in the document, respond YES; otherwise, respond NO.
2. document discusses related topics but does not directly answer the given question, re- spond NO.
3. If the document only provides reference paper titles without substantive content that supports answer, respond NO.
First, think step by step and explain your reasoning. In the last separate line, directly respond YES NO without quotes.
Figure 9: The prompt for judging whether the document is relevant to the question.
22
Back to Appendix Table of Contents

---

are given a markdown document. task to extract specific theorem, formula, equation, algorithm, or concept named "[theorem_name) from this document. Instructions:
1. Carefully locate the section that describes theorem "(theorem, ramme] )".
2. Extract the complete definition, explanation, and any associated formulas equations.
3. Remove reference citations.
4. If there are referenced images in the content, preserve the image references exactly they appear.
5. Your response MUST follow the following LaTeX-style format:
\begin definition] (theorem name Complete definition and explanation, preserving mathematical notation. Include examples if present. \end medddefinntion.
Here is the document content: (markdown Sharkdown_contente
Figure 10: The prompt for cleaning the theorem content.
E
EXPERIMENT DETAILS
E.I MODELS AND INSTRUCTIONS
Table 5: Details of the multimodal retriever models evaluated MRMR.
Model
Size
Version
BGE-M3 (Chen et al., 2024)
600M
BAAl/hge-m3
NE-Embed-V2 (Lee al., 2025)
nvidiu/VV-Embxed-22
Qwen3-Embedding (Zhang et al., 2025d)
Owen//Qwen--Emmedddme
EVA-CLIP (Sun et al., 2023)
400M
QuanSu//EVAO2-CIIPP-I
.P (Zhai et al., 2023)
650M
googlesiglip-llree--aahhh--22222
JinaCLIP (Koukounas et al., 2024)
860M
jnaaifjin--lipp-2
OpenCLIP (Cherti et al., 2023)
1.4B
VISTA (Zhou al., 2024)
200M
BAAllhge-visualized-m3
VLM2Vec (Tiang al., 2025c)
4B
LTGER-LLb/VLM2Vee=ulll
GME-Qwen2-V LL (Zhang et al., 2025c)
7B
Ops-MM-Embeddinn (OpenSearch-Al, 2025)
7B
ES-V (Jiang al., 2025b)
8B
royokong/e5-v
MM-Embed (Lin al., 2025)
nvidia/MM-EEheed
ColPali (Faysse et al., 2025)
3B
vidore/colpali-vl.3
Following TII,, we evaluate text retrievers on multimodal retrieval tasks by replacing images with captions generated by an LLM. To simulate real-time inference, we apply the standardized prompt "Describe the image" and use Qwen2-Vi
to produce the captions.
E.2 IMPLEMENTATIONS AND MACHINES
The MRMR dataset is constructed following the conventions MTEB (Muennighoff et al., 2023), including data format and evaluation pipeline, with modifications to support mixed-modality inputs during evaluation. All experiments are conducted on NVIDIA A100, A6000, or H100 GPUs. The runtime of full evaluation depends the model, but with the limited corpus size for efficiency, one complete run can be completed within 4 hours on a single A100 GPU for open-source dense models. To further accelerate dense model evaluation, employ FlashAttention (Dao al., 2022).
23
Back to Appendix Table of Contents

---

Table 6: Instruction prompts used during model evaluation in MRMR.
Task
Modality
Prompt
Multimodal
Knowledge
Retrieve relevant documents that help the question.
Text
Multimodal
in salving the problem.
Text
Multimodal
Given image, relrieve descriptions that have contradictury information with image.
Nogation
Text
Given image caption, retrieve descriptions that have contradictory information with the image caption.
Multimodal
Given vehicle design, retrieve the design requirements that violates.
Vehicle Design
Text
Given vehiele design description, retrieve the design requirements that violates.
Multimedal
traftic
driving documents violates. traftic description, driving documents
Traffic Case
E.3 DETAILED RESULTS
Table 7: Detailed performance of retrieval models on MRMR (Knowledge).
Knowledge
Avg
Model
Music
Design
Theo.
Hist.
Soci.
Phann.
Clinic
Bask:
Apri.
Geo.
Chem.
Tear Models with Image Captton
RGF-MB
41.4
44.0
49.4
57.2
47.7
19.5
52.2
158
58.5
11.2
18.2
36.2
38.7
48.6
37.6
48.3
41.0
NV-Embed-22
61.8
70.1
86.8
70.6
64.3
59.7
95.8
78.0
19.8
46.0
59.0
65.3
63.3
70.0
63.6
64.9
QuenS-Fmbeediing
62.8
62,1
74.8
87.3
76.1
74.0
69.3
97.8
83.1
348
47.0
64.0
69.5
76.5
74.0
72.6
70.4
Image Pan-Strvam Madels
Fanion
EVA-CLIP
30.5
1.5
3.5
7.5
16.7
5.5
16.3
0.00
22.7
10.3
10.0
16.4
41.6
15.4
20.4
18.5
14,8
SigLIP
26.2
138
9.7
15.6
26.7
OPeSCI.PP
20.9
50.7
62.9
86.4
35.8
10.2
15.1
22.7
11.1
10.6
30.8
25.8
34.1
45.8
23.9
34.3
31.9
21.1
Mullimoda Modela wilhi Merged Imauge
12.3
18.2
ES-V
13.0
23.4
17.6
46.1
15.6
4.3
10.8
7.7
12.5
7.1
13.5
13.7
18.3
13.1
23.3
10.0
15.6
MM-FFmbal
35.1
VLM22ec
34.4
44.0
49.6
$4.8
36.4
12.3
19.3
19.2
13.4
13.6
23.7
33.1
29.0
40.7
37.5
30.8
33.5
GME-Qwart
Ops MM Embedding
58.5
75.6
84.2
96.8
71.4
71.1
$9.7
73.7
76.1
30.9
50.7
66.5
58.7
78.5
80.4
69.0
68.7
Multinndul
Image
GIME-Owa2-VL.
58.2
46.5
33.6
58.4
$2.5
48.5
48.2
52.1
72.9
1688
31.7
40.2
49.7
09.0
53.8
45.4
49.8
Ops MM Embedding
684
82.4
63.0
74.3
29.3
ColPali
25.1
27.7
46.4
43.7
31.7
19.4
38.5
01.0
64.1
10.6
23.0
60.1
36.7
32.6
67.6
56.3
36.5
24
Back to Appendix Table of Contents

---

F ANALYSIS DETAILS
F.1 FALSE POSITIVE AND FALSE NEGATIVE ANALYSIS
Although queries and their relevant documents carefully validated by human annotators, false positives and false negatives may still arise when aggregating documents across queries or sampling from external corpora. We explicitly instructed annotators and validators to identify similar lated queries and to cross-annotate documents accordingly. As a result, some queries share the same relevant documents.
To quantitatively assess the prevalence of such labeling errors, we conducted a human audit of the top-retrieved documents retrieved by the best-performing model Dos-MM-Embedding Table 8, the audit revealed zero false positives and false negative rate of only 2.5% for Knowledge tasks for sampled 120 documents, Similarly, for Theorem tasks, the combined error rate was minimal at approximately 5.8%, comprising 3.3% false negatives and 2.5% false positives. These results suggest that label noise is insignificant, thereby supporting the reliability of the benchmark.
As shown in
For Contradiction tasks, the dataset is relatively small and predominantly constructed manually by annotators and validators. Given the quality control in the construction process, no additional human evaluation was deemed necessary.
Table 8: Human audit of document relevance annotations for the top-retrieved documents produced by the perfforrf multimodal model, Ops-MM-Embeddingg false negative is relevant doc- ument incorrectly labeled as irrelevant by our method, and a false positive is an irrelevant document incorrectly labeled relevant.
False Negatives
False Positives
Dataset
Documents Checked
Count
Ratio
Count
Ratio
Theorem
120
4
3.3%
3
2.5%
Knowledge
120
3
2.5%
0
0.0%
F.2 ERROR CASE STUDIES
In this section, we present case studies for the Ops-MM-Emmedding model in different domains such Biology (Figure 11) and Traffic Case (Figure 12). The error case analysis for Theorem tasks are exemplified as follows.
A recurring issue in Engineering and Geometry tasks is the model's tendency to perform coarse- grained matching based on shape and keywords, while ignoring the specific geometric conditions or physical constraints defined in the query.
Case Study: Engineering (Geometry)
Query ID: validation wrritinrrrrrrrrreeeeeeeee 5
Query Content: An image showing pentagon with internal angles and a specific coordi- nate bearing angle (a12 = 30"), asking to calculate other bearing angles.
Retrieved Negative (Top-1): "Inscribing Circle in Regular Pentagon".
The model correctly iden- tifies the visual object pentagon) and the domain (geometry/angles) However, it retrieves a document about inscribing dircles-likely because the dense geometric keywords and the visual of a polygon strongly correlate the embedding space. The model fails to attend to the specific logical task (calculating bearing angles) and instead prioritizes the dominant visual features (the pentagon shape).
Analysis:
This represents keyword and shape hallucination.
Case Study: Engineering (Statics) Query ID: test Architecture and Engineering 214
25
Back to Appendix Table of Contents

---

Query Content: A floor plan asking to compute the "tributary areas" for specific floor beam B1.
Retrieved Negative (Top-1); "Static equilibrium".
The retrieved document discusses the static equilibrium of beams. While mantically related to the domain (structural engineering and beams), it is conceptual mismatch. The model retrieves a theoretical concept (statical indeterminacy) rather than the procedural knowledge required for area calculation. This suggests that the retriever struggles to distinguish between theoretical concepts and practical calculation tasks when visual cues (schematic diagrams of beams) are similar.
Analysis:
In scientific domains, the model often exhibits "partial understanding". where correctly identifies the strict sub-domain or topic but fails retrieve the document addressing the specific variable or relationship queried.
Case Study:
Physics (Thermodynamies)
Query ID: test Physics 74
Query Content: P - V (Pressure- Volume) graph showing a cyclic process, asking to identify the point of highest temperature.
Retrieved Negative (Top-1): "Isothermal process".
The retrieved document explains isothermal processes (constant temperature), which frequently utilize p diagrams similar the query image. The retrieval is plausible but incorrect; the model latched onto the visual graph type (P - V curve) but failed to deduce the specific relationship (PV nRTT required find the maximum temperature. This confirms the limitation regarding higher-level deduction: the model recognizes the graphical language but not the specific physical implication.
Analysis:
Case Study: Math (Data Interpretation)
Query ID: test_Matte Mat__MMhhh4000
Query Content: A histogram/bar chart showing student distances from school, asking for the percentage of students whose distance falls within the 5 km to 10 km range.
Retrieved Negative (Top-1): "Generic statistical methods".
The retrieved document discusses general statistical techniques and data repre- sentation, which often involve histograms similar the query image. The retrieval appears contextually related due to the presence of histogram but is ultimately incorrect; the model associated the visual format with broad statistical category but failed to extract or interpret the specific numerical data (bin counts and ranges) needed to compute the re- quired percentage. This highlights deficiency in nusurillu alignment: the model recognizes the chart type but does not connect it to the mathmatical reasoning.
Analysis:
26
Back to Appendix Table of Contents

---

Science Biology
ID: validation Agriculture 6
Task:
Retrieve relevant documents that help answer the question.
Query: The picture below shows common soil organism. How should this organism be classified in terms of Flora vs. Fauna and its size category? <image 1>
Answer: macro-fauma.
Negative Documents
Positive Documents
Soybean eyst nematede
ponecc socondty san Neppoa the
coperity comer calor
zar encopsed lenars fom ando codl
seroreee see drggerentt pegirr misee
yora romer enter tyroe cale
tendee
lomen cager pirtt oat
enoee snce Jeat ahe bcod morkin eneiren rnbiesscco schved
Acnde
Figure 11: Error case example in Agriculture where the multimodal embedding model Ops-MM- Embedding prioritizes the negative document in the left over the positive document in the right.
27
Back to Appendix Table of Contents

---

Contradiction Traffic Case
ID:
Given traffic case, retrieve driving rule documents that violates.
Task:
Query: was going through location shown in the picture on Tuesday. <image>
Negative Documents
Positive Documents
PART teated during Banic Theery Teet)
PART tested during Test)
DRIVING IN TUNNELS
DRIVING IN TUNNELS
DAILY DRIVING RULES
sorherto codeoaattea soa Maro Hameennn yn seteett
denalann Seeaer die model valummu mesred onter 3o
Figure 12: Error case example in Traffic where the multimodal embedding model Ops-MM- Embedding prioritizes the negative document in the left over the positive document the right.
28
Back to Appendix Table of Contents

---

F.3 TEST- TIME SCALING IN RETRIEVAL
We conducted query expansion experiments using both weak and strong vision-language models Qwen2-VI-2B and Qwen2.5-VL-72B- weak and strong multimodal retriev- ers (i.e., GME-Qwen2-VL and (Ds-MMMEEmedddnng) As shown in Tables and 10, query expan- sion generally effective for weak retriever models. However, for stronger retrievers, the quality of the query expansion becomes critical: expansions generated by the weaker VLM actually degrade the performance of the stronger retriever.
With query expansion by strong LLM (Qwen2.5-VL-72B) the expansion technique is effective for improving both strong and weak retriever models. However, they are still far from perfect on this benchmark. For example, the best retriever with strong query expansion only achieves 55.9 for medical queries and 31.8 8 for math queries.
Table 9: nDCG@ 10 scores of the multimodal retriever CME-Qwen2-Ve on MRMR Knowledge and Theorem tasks, comparing the original queries with query expansions generated by Qwen2-VL- 2B-Instruct and Qwen2.5- expansion is reported as the number of tokens measured by the GPT-2 tokenizer.
The average query length (Q # #Text) before and after
Knowledge
Theorem
Avg.
Model
Q #Text
Art
Med.
Sci.
Hum.
Q #Text
Math
Phy.
Eng.
Bus.
Original
31.4
54.3
40.1
46.8
45.6
58.6
28.8
36.0
30.2
45.1
40.9
Qwen2-VL-2B
699.6
64.9
49.6
64.6
48.9
735.9
23.5
36.5
31.5
48.3
46.0
Qwen2.5-VL-72B
843.8
76.9
61.8
77.0
72.2
1218.4
33.3
36.9
32.7
55.0
55.7
Table 10: nDCG@ 10 scores of the multimodal retriever Ops-MM--mbeddding on MRMR Knowl- edge and Theorem tasks, comparing the original queries with query expansions generated by Qwen2- VL-2B-Instruct and -e--n---VL-72B--mss- expansion is reported as the number of tokens measured by the GPT-2 tokenizer.
The average query length (Q #Text) before and after
Knowledge
Theorem
Avg.
Model
Q #TTxl
Art
Med.
Hum.
Q #Texl
Math
Phy.
Eng.
Original
31.4
79.3
52.5
70.0
67.8
58.6
27.7
39.5
30.1
52.3
52.4
Qwen2-VL. -BBB2
699.6
77.2
45.7
67.1
58.0
735.9
24.7
35.6
29.5
50.2
48.5
Dwen2.55VVL-2B
843.8
80.5
55.9
73.1
64.6
1218.4
31.8
39.4
35.3
53,9
54.3
29
Back to Appendix Table of Contents

---

G DATA EXAMPLES
Art Music
ID: test Music 327
Task: Retrieve relevant docimeents that help anssve the question.
Query: Determine True False: This the dominant in minor. <image 1>
Answer: True
Positive Document
the ennor offtna dim cableeder.
DE
G
Depres
Raperbetc Ofll Medl fateeemnn EDvement Cob medurt Gusbeen Ostave
The mirde mair of mihor is major. Minor koys made wart maje make of samo rolen.
The misor ecaln as
Cc and Ersapo raale Ins
viloe czale
while rap hall ena vhok ris whole map are b- -an 3- un ettole top roppe
crp moleroo bey (iens withitha &he manor scatls and roune
whele stap kis
Figure 13: Music example.
30
Back to Appendix Table of Contents

---

Medicine Clinical Medicine
ID: test Clinical Medicine 283
Task: Retrieve relevont documents thathelp answer the question.
Query: 61 61-year-old woman in the hospital 2 weeks with bronchopueumonia following surgery for endometrial ademocarcmomaa She then becomes suddenly short of breath. This microscopic appearance from her hung most typical for which of the following pathologic abnommalitess? <imnage 1>
Answer: Thromboenbbcss
Positive Document
Risk Factors
Signa and symptoms
lnd und
Rlakk Eactors
Underll ing COUDE5
Figure 14: Clinic Medicine example.
31
Back to Appendix Table of Contents

---

Science Biology
ID:: test test_eestiiiiee Agricultureee 207
Task: Retrieve relevont documents thothelp answer the question.
Query: <image With which aroup are leaf galls like the ones on gripevines, caused by the uphid-hite insect called Grape phy dhylloxera which las root and leaf feeding stages its lifecycle and considered an extremely atypical symptom for this insect group- more commonly associared?
Answer: Mates
Positive Document
Diptesan flies
pel stpeess
Nematodes
Figure 15: Biology example.
32
Back to Appendix Table of Contents

---

Humanities Psychology
ID: validatioo_Sociinnee 1
Task: Retrieve relevant documents that thet answer the question.
In 1946, the person in <image I> was arrested for refusingto to sit in the blacks-only section ofe cinemu in Nova Scotia. This is example of
Query:
Answer: conflict crime
Positive Document
Conffict criminology
cito wboue co dloung recommoiee coweree colscciroa otoum and the siecs.
Hemberr rerpinsieed copeereed
ger nataass by srong rijultty inppilate watee was the covernue cocal undk betheer siate ottoors
Discussion
thery awoures coanty vatpettec piocees dieusa change and Battii Vorkd seilnciee tiee
the cesaiaaal
parciuemaa procksttur then pusters
omso Pirm tree tutte eac deeneert contlaction may peotieur regatins cirediadiive vo
popyent buk the iessss aropestle grop het sccess pove office map eneence
idraffes
essemaai
sonaters mary
diass light
sobe making propraale
Figure 16: Psychology example.
33
Back to Appendix Table of Contents

---

Theorem Math
ID: test Math 16
Task: Retrieve relevant theorems that involved in solving the problem.
Query: <image 1> The radius of the circle above is and 45=45.. What the area of the shaded scction of the circle?
Answer: 2xT
Positive Document
Area of
Area Circle
Taga Clickes Soveeny,Arruc Semor, kornehe
Tage Gommutyy
Thowerss
Theeeer
Airand
shache oe where wnce henvers
Thes given
Preef
diftsteeeeegg offee ofthon
Proof
Sove rephered serethigg calasnr.
From ane Oretu the trea le oud vithin In.
ridarpueee tattin
Cluiiiis Bisnucts BACC
Apphting adectsttio
(ready
Figure 17: Math example.
34
Back to Appendix Table of Contents

---

Theorem Physics
lest Physics_26 265
Task: Retrieve relevant theorems that are involved in solving the problem.
Query: <image 1>The graph above represents position versus time an object being acted on by constant force. The average speed during the interval between Is und 2s must nearly
100
Answer: m/s
Positive Document
Length a Real Interval
Definition
any of the fellening ckeatec real inceal:
Closed interval:
Js-A
Lilll-pee open mecol Dights 6.
Displacement
The langth
This definiti
Digiboreeee
Speed
mang
Definstion
ingattode
Uecaus durugard dimetionn spend axalez quancity.
Mathanatiall,, it'ss the velucity veetar of bocy, then speed li given by:
[[]]
magaritude of
Figure 18: Physics example.
35
Back to Appendix Table of Contents

---

Theorem Engineering
ID: validation Electronics 20
Task: Retrieve relevant theorems that involved in solving the problem.
0-10
Query: In <image 1>, = sin(2nr) Find expression for i and calculate at the instants = 0.
Fig.
Answer: 2TT 10 A
Positive Document
Current in Electric L, R, C in Series
Toge
mutioor
Thooren
deetriatiiioa concle
ammditance
U'ing fasdanerral drrent
cipesitance C,
connected snlecutt sounce el Alj) Fauction
Copectior vottoge: Qusherr
The
chargras
Subattruingg theie laro Kirchwath
E- C- 2
Becall cureeu tine of
40
cetvenetrrve teth equatio with reppett
de
(5)
Figure 19: Engineering example.
36
Back to Appendix Table of Contents

---

Theorem Business
ID: test Economics 157
Task: Retrieve relevant theorems that involved in solving the problem.
Query: Presuming that the Marginal Propensity to Consume (MPC) 0.8, could you calculate monetary value signified by the interval A-B the aforementioned diugram <image I>?
C+1-0+X
millioe
Answer: $25 million.
Positive Document
Maliiiar
Magital Save Relatooshipp
Grocrii for diglle codeisae
Figure 20: Business example.
37
Back to Appendix Table of Contents

---

Contradiction Negation
ID: 340894
Task: Retrieve text that contradictory information to the image.
Query: <image 1>
Positive Document
This image includes mouse, dining table. book but no key board.
Negative Documents
This image includes mouse, tv, laptop but bottle.
This image includes mouse, cell phone, laptop but no refrigerator.
his image inchudes cell phone, person, chair but no bottle.
Figure 21: Negation example.
Contradiction Vehicle Design
ID: O
Task: Given vehicle design as the query, retrieve the design reguirements that irviolates.
Query: Atlached is am image that shows an engineering drawing of the vehicle ... All units displayed in the engineering drawing have units of mfm <image>
Positive Document
VEHICLE REQUIREMENTS
CONFIGURATION
The velidd must open sooughht Ins.
spenwitead witeetedd and cocipt fommle anyl body with four wheels that arerot
V.1.2 Wheelbase
velidd wheebased 125
Figure 22: Vehicle Design example.
38
Back to Appendix Table of Contents

---

Contradiction Traffic Case
17:
Task: Given maffic case the query, retrieve driving rule document violates.
Query: In Singapore, Ginny was driving with the speed of 64 km/h, keeping 3-meter gap behind the silver car. as shown in the picture. <image>
Positive Document
DRIVING IN TRAFFIC
Bito be tested Basic Theory Test)
THE VEHICLE IN FRONT
CONDUCT
lenghh Wonch seed.
Telh diaee oprapiic poreeee
Figure 23: Traffic Case example.
39
Back to Appendix Table of Contents


Reviews:
[{"review_id": "qhVYRtAHJj", "reviewer": "Reviewer_u6P3", "summary": "The authors introduce a dataset focusing on 'Multidisciplinary and Reasoning-intensive Multimodal Retrieval' MRMR with many key contributions with respect to existing multi-modal retrieval dataset. From the perspective of somebody likely to utilize this dataset, there are two practical high-level contributions: (1) MRMR provides validated retrieval results for a subset of the MMMU-Pro question source based on high-quality datasources (PIN-14M + web pages for {Knowledge: Art, Medicine, Science, Humanities} and Bright + web pages for {Theorem: Math, Physics, Engineering, Humanities}; and (2) MRMR develops an interesting reasoning-intensive \"contradiction\" problem and dataset that is applicable to agentic LLM settings (and currently has low performance with existing systems). In contrasting this work with existing MMIR benchmarks, some key contributions of MRMR (also pointed out by the authors) include: (1) multi-disciplinary expert-level questions (most benchmarks focus on the general knowledge of Wikipedia), (2) reasoning-focus with more complex IR instructions (most benchmarks focus on semantic matching or text IR/QA), and (3) image-text interleaving in queries and data (most MM benchmarks are VQA-focused with a single image). \n\nFor each question type {Knowledge, Theorem, Contradiction}, the authors first describe their method for selecting questions, collecting/validating positive and hard negative results, and adding a background set of reasonable negative examples to construct the corpus. Based on the MRMR dataset, the authors conduct a large scale ablation study over four multi-modal retrieval setups (Text models with image captions, Text and image two-stream models with vector function, Multimodal model with merged images, and Multimodal models with document as image) with 14 state-of-the-art candidate models, demonstrating: (1) text-retrieval based approaches presently have the best performance for these knowledge/reasoning-required tasks, (2) there is ample room for improvement across the board, and (3) there is large variance in performance across different tasks. This study was able to power a reasonable failure-space analysis to point toward research directions that may result in tangible improvements.", "questions": "- It would be useful for you to clarify any licensing issues for academic work in commercial settings.\n- In terms of the ablation study, it would also be interesting in the discussion to point out any alignment or contradictions for similar systems on related datasets (e.g., wikiHow).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Some strengths of this work include:\n- As primarily a resources paper, the authors clearly defined what type of questions they targeting, clearly motivated why this would result in a useful benchmark, and demonstrated that a large-scale ablation with the benchmark surfaces interesting (albeit not entirely surprising) conclusions.\n- The authors clearly contrast MRMR with related datasets (Table 1) and clearly describe the collection procedure in the various settings (Table 2 and Sections 3.2-3.4) -- which are generally sensible and required a significant amount of work to collect with high-quality.\n- MMMU-Pro is a good dataset to derive challenging IR problems with.\n- The 'contradiction' dataset is innovative and I expect will inspire additional work in other challenges IR cases that require specific instructions and will lead to more interesting agentic LLM responses.\n- A good benchmark paper introduces a non-trivial dataset, makes it easy for others to build on, and provides non-trivial baseline performance results -- which MRMR does.", "weaknesses": "Some weaknesses of this work include:\n- The 'contradiction' family of questions is the most interesting from an innovation perspective, but underdeveloped in my opinion. To begin with, even if I am being a bit pedantic, these technically aren't all contradictions (e.g., Figure 1 vehicle design is 'non-compliance'). Also, in looking through the specific cases, 'negation' is a bit contrived (albeit shown to perform poorly in the empirical results) and I can think of other cases and possibly a general hierarchy of \"matching a description that isn't there\". Basically, it is somewhat preliminary and likely deserves its own study.\n- The dataset is high-quality, but relatively small as compared to other related datasets in Table 1. Minimally, I would recommend discussing why a smaller number of high-quality examples is more valuable with respect to detecting improvements.\n- Based on the process used, it wasn't clear if there would be any license issues with respect to using this in commercial settings (even if only for academic purposes). I am assuming this isn't an issue, but it would likely affect its impact.\n- The analysis in Section 5 is underdeveloped.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x2jtZXUMtv", "reviewer": "Reviewer_8xXt", "summary": "Summary:\nAuthors repurpose MMMU-Pro and use other synthetic methods to generate a multimodal retrival dataset that has expert domain, reasoning intensive and contradition style questions.", "questions": "-", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Strengths:\n- Addresses gaps in the present day benchmarks which mostly rely on surface level/minimal reasoning.\n- Clear writing on how each datapoint was created. \n- Also provide hard negatives which can be useful to train models.", "weaknesses": "Weakness:\n- Positives of each question are curated independetly. Positives of all documents are put together to construct the corpus. How was it ensured that positives of other queries are not positives of a given query.\n  - What were false negatives filtered?\n  - Similarly, negatives mined for other queries either by GPT-Search/Human mismatch or by PIN14 can turn out to be potential positves of other queries.\n- Smaller dataset compared to sme of the MTEB/MMEB datasets.\n- Evaluation only benchmark. It would have been significantly more useful had a training dataset been provided.\n- Query expansion results for stronger retriever models missing. It is important to understand if query exapansion with multimodal/text retriver models still falls short on this benchmark.\n- Weak qualitative analysis. A category wise qualitative analysis would have been much more helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eOBKDLVZ8t", "reviewer": "Reviewer_8qw3", "summary": "This paper introduces MRMR (Multidisciplinary Reasoning-intensive Multimodal Retrieval), a novel benchmark designed to evaluate retrieval systems in expert-level, reasoning-intensive, and multimodal tasks. Unlike previous benchmarks that focus on general domains, MRMR spans 23 domains, including medicine, engineering, science, and business, and features 1,502 expert-annotated queries. The tasks are categorized into three types: Knowledge Retrieval, Theorem Retrieval, and Contradiction Retrieval. MRMR challenges retrieval models not only to identify semantically relevant documents but also to perform deep reasoning and logical deduction, such as identifying contradictory concepts or retrieving relevant theorems for complex calculations. The dataset includes interleaved image-text sequences, reflecting the real-world scenario where queries and documents are multimodal. The paper evaluates a variety of state-of-the-art multimodal retrieval systems and finds significant room for improvement, particularly in reasoning-intensive tasks.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* MRMR provides a unique and comprehensive framework to evaluate multimodal retrieval systems on expert-level, reasoning-intensive tasks across multiple disciplines. It addresses the gap in existing benchmarks that fail to capture the complexity of real-world, domain-specific applications, such as medical diagnoses and engineering design.\n* The benchmark spans 23 domains, providing a holistic evaluation of retrieval systems across various expert fields. This diversity helps to assess the models' generalizability and specialization across disciplines like medicine, business, engineering, and the sciences.\n* By using interleaved image-text sequences, MRMR more accurately reflects real-world retrieval tasks, where queries and documents are rarely confined to a single modality. This design enhances the practical relevance of the benchmark.", "weaknesses": "1. Have the authors considered the scenario where the documents highly relevant to a given query come from multiple different disciplines? For example, the user’s question might require the synthesis of documents from multiple disciplines to provide a comprehensive answer. This kind of situation frequently arises in real-world applications. Did the authors consider this when designing the benchmark?\n\n2. In the overall design of the benchmark, the authors separated documents by discipline for the experiments. However, creating a large database by mixing documents from all disciplines and performing retrieval might present a more challenging setting. This cross-disciplinary retrieval scenario could provide an even more difficult test for the models.\n\n3. The authors discuss retrieval within specific disciplines, which is an interesting topic, but they have not addressed cross-disciplinary retrieval. For instance, if the query includes an image of a sculpture from the art domain and the user wants to retrieve mathematical theorems (from the math domain) related to the geometry of the sculpture, this is a very interesting problem. I look forward to the authors' response and discussion on this question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6kcaEUbOak", "reviewer": "Reviewer_9XX3", "summary": "This paper introduces a multimodal retrieval benchmark with an emphasis on reasoning-intensive retrieval across 23 domains.\nIn the proposed benchmark, the authors introduce three distinct types of retrieval tasks -- knowledge retrieval, theorem retrieval, and contradiction retrieval.\nThe paper evaluates various multimodal retrieval setups (e.g., text-only, two-stream, multimodal, and document-as-image models) covering several frontier models on the MRMR benchmark. The results suggest that existing multimodal retrievers struggle on reasoning-intensive and contradiction retrieval tasks compared to text-only retrievers.\nOverall, this paper presents a meaningful contribution to the community, though its technical contribution is limited.", "questions": "1. In table 2, the third last row (L181), what does it mean that negation task has 4 documents in total (D=4), shouldn't it be 200*4=800?\n2. L220, should \"solvable\" be \"unsolvable\"?\n3. Can you discuss similar works such as $\\text{MR}^2$ bench [1] on how MRMR differentiates from them?\n4. Can you add prompts in the appendix that you use to synthesize the dataset (e.g., synthesize negation candidates for the contradiction retrieval task)?\n\n[1] Zhou, Junjie, et al. \"MR $^ 2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval.\" arXiv preprint arXiv:2509.26378 (2025).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. Wide domain coverage: MRMR spans 23 domains across six disciplines.\n2. Integration of contradiction retrieval: adding the negation/contradiction retrieval testbed is practically useful and can be a good initiative for the community.\n3. Comprehensive evaluation: the experiments cover a wide range of retrieval architectures and models.\nHuman-verified annotation and corpus quality control: All positive documents are validated by human experts, and the corpus is cleaned under human supervision, making MRMR more realistic compared to fully synthetic benchmarks.", "weaknesses": "1. Limited technical contribution beyond task composition: it's unclear about the main technical contribution beyond composing tasks (e.g., reasoning-intensive, image-text interleaving, multidisciplinary) from prior works.\n2. Limited dataset size: as a benchmark spanning 23 domains, size of the dataset (1502 queries) is relatively small and therefore may limit statistical robustness.\n3. Writing clarity: Certain sections could be written more clearly to improve readability and logical flow, particularly in the methodology and evaluation descriptions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)