REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark
Authors: navve wasserman, roi pony, Oshri Naparstek, Adi Raz Goldfarb, Eli Schwartz, Udi Barzelay, Leonid Karlinsky
Accepted at: acl25

REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark

Navve Wasserman1,2, Roi Pony1, Oshri Naparstek1, Adi Raz Goldfarb1
Eli Schwartz1, Udi Barzelay1, Leonid Karlinsky1
1IBM Research Israel   2Weizmann Institute of Science
navve.wasserman@weizmann.ac.il

Abstract
Accurate multi-modal document retrieval is crucial for Retrieval-Augmented Generation (RAG), yet existing benchmarks do not fully capture real-world challenges with their current design. We introduce REAL-MM-RAG, an automatically generated benchmark designed to address four key properties essential for real-world retrieval: (i) multi-modal documents, (ii) enhanced difficulty, (iii) Realistic-RAG queries and (iv) accurate labeling. Additionally, we propose a multi-difficulty-level scheme based on query rephrasing to evaluate models’ semantic understanding beyond keyword matching. Our benchmark reveals significant model weaknesses, particularly in handling table-heavy documents and robustness to query rephrasing. To mitigate these shortcomings, we curate a rephrased training set and introduce a new finance-focused, table-heavy dataset. Fine-tuning on these datasets enables models to achieve state-of-the-art retrieval performance on REAL-MM-RAG benchmark. Our work offers a better way to evaluate and improve retrieval in multi-modal RAG systems while also providing training data and models that address current limitations. Our benchmark is available at this project page.

1 Introduction
Accurate retrieval of relevant documents is a cornerstone of modern natural language processing (NLP) applications, whether used alone or in advanced pipelines like Retrieval-Augmented Generation (RAG). RAG (Lewis et al., 2020) has emerged as a powerful approach wherein models retrieve external information before generating answers or content, enabling operation over large document collections. Multi-modal RAG extends this to real-world scenarios involving text, figures, tables, and potentially entire page images.

Successful retrieval is crucial for RAG, as retrieving the wrong page or document inevitably hinders the final generated response. Therefore our analysis will focus on the retrieval part. Although research on RAG is advancing, the field still lacks a complete understanding of how models perform in realistic setups, both for evaluating performance and identifying current weaknesses to overcome. This gap arises from a shortage of benchmarks that thoroughly assess real-world retrieval challenges.

We identify four essential properties for a real-world document retrieval benchmark, particularly in multi-modal contexts: (i) Multi-modal documents: The dataset should include pages with text, figures, tables, and other visual elements to reflect the complexity of real-world materials. (ii) Enhanced difficulty: Queries should require more than simple keyword matching and involve a large corpus of contextually similar pages to ensure challenging evaluations. (iii) Realistic-RAG queries: Questions must be posed naturally, without explicit references to pages—reflecting queries a person might ask when seeking information without knowing the answer’s location (in contrast to generic question-answering setups). (iv) Accurate labeling: All documents relevant to a query must be correctly and exhaustively labeled to prevent underestimation of retrieval performance and to avoid false negatives.

Although a few recent benchmarks touch on some of these aspects (Faysse et al., 2024; Ma et al., 2024a,b), most fail to fully capture them, limiting their usefulness for understanding and improving multi-modal retrieval models. We introduce REAL-MM-RAG-Bench (Real-World Multi-Modal Retrieval-Augmented Generation Benchmark), a benchmark designed to satisfy the properties above:

Multi-modal documents (Property i): Our dataset comprises slides and documents with text, figures, tables, and images, requiring systems to handle combined textual and visual data.
Realistic-RAG Queries
A query one may ask without seeing the relevant document.

Previous Benchmarks
✗ What is the metric of the x-axis in the graph?
✗ How many faces appear in the slide?
✗ Where was the highest revenue growth?

Real-MMRAG – Rag Tailored Generation and Filtering
✔ How can Watson AI help build a user-centric AI apps?
✔ How many PCI adapters can be installed in an IBM FlashSystem 5300?
✔ How much did IBM’s long-term debt increase in 2016?

Enhanced Difficulty
Previous Benchmarks
Solvable by Keyword Matching
What was IBM’s pre-tax income from global business services in 3Q19?

Real-MMRAG - Rephrasing
Requires Semantic Understanding
What were IBM’s earnings before tax from international business services in the third quarter of 2019?

<table>
  <tr>
    <th>Segment</th>
    <th>2019</th>
    <th>Yr/Yr</th>
  </tr>
  <tr>
    <td>Revenue (External)</td>
    <td>$4.1</td>
    <td>2%</td>
  </tr>
  <tr>
    <td>Gross Profit Margin (External)</td>
    <td>31.1%</td>
    <td>1.1 pts</td>
  </tr>
  <tr>
    <td>Pre-Tax Income</td>
    <td>$0.6</td>
    <td>1%</td>
  </tr>
  <tr>
    <td>Pre-Tax Income Margin</td>
    <td>13.7%</td>
    <td>0.1 pts</td>
  </tr>
</table>

Figure 1: Proposed Real-MM-RAG Benchmark

Enhanced difficulty (Property ii): Instead of relying on isolated pages or trivial queries, we focus on long documents from specialized domains (e.g., IBM finance reports, FlashSystem technical materials). This makes retrieval significantly more challenging, as models must differentiate between highly similar content within the same domain. Additionally, we incorporate a rephrasing step to ensure that query wording and order are not identical to the page content, requiring semantic understanding rather than simple keyword matching.

Realistic-RAG queries (Property iii): We use a two-step process to generate queries: vision-language models (VLMs) create retrieval-focused queries, and large language models (LLMs) filter them to ensure natural phrasing and realistic user intent. Unlike many existing datasets, our queries avoid direct references to specific pages, reflecting authentic retrieval scenarios.

Accurate labeling (Property iv): Ensuring all pages answering a query are correctly identified is crucial, especially in benchmarks with many similar pages. Existing benchmarks often mislabel valid documents as incorrect, leading to false negatives. To address this, we employ an automated pipeline using VLMs to verify query relevance to each page. While computationally intensive, this approach enhances labeling reliability, particularly for closely related pages.

Our benchmark enables reliable evaluation of current retrieval models and uncovering some of their weaknesses. Additionally, it incorporates two essential properties that expose specific retrieval challenges:

Rephrasing Robustness Evaluation: In real-world scenarios, users rarely phrase their queries exactly as they appear in documents. However, both VLMs and human annotators tend to generate queries that closely mirror the source material, often using similar words and sentence structures (Smeaton and Kelledy, 1998; Zhu et al., 2024). This fails to reflect natural user behavior, where users are not directly exposed to the document page when forming queries. To address this, we introduce a multi-level rephrasing benchmark, modifying queries at three distinct levels—ranging from slight rewording to significant structural changes. Our experiments show that current retrieval models struggle to maintain performance across these variations, highlighting a critical weakness in their semantic understanding.

Table-Focused Scenarios: Table-heavy documents (e.g. financial reports) often contain dense tabular data, posing a major challenge for retrieval models. By incorporating table-heavy documents into our benchmark, we expose key deficiencies in table comprehension that significantly impact model performance. These properties allow us to demonstrate that all current retrieval models exhibit weaknesses in handling both rephrased queries and table-heavy financial documents.

To address these shortcomings, we leverage in-
sights from our benchmark to enhance retrieval performance. Specifically, we introduce two targeted training strategies: (i) a rephrased training dataset, generated by rephrasing the ColPali training dataset (Faysse et al., 2024), and (ii) a finance-table-heavy training set, designed to improve retrieval in tabular contexts. Fine-tuning the current best model on these datasets achieves state-of-the-art retrieval performance on our benchmarks. This demonstrates how systematic evaluation through our benchmark can informs effective training strategies, leading to more robust and adaptable retrieval models.

The contributions of this paper are as follows:
• Defining properties of a real-world retrieval benchmark and highlighting shortcomings in existing ones.
• Introducing a high-quality multi-modal retrieval benchmark with an automated pipeline for query generation, filtering, and labeling verification.
• Establishing a rephrasing robustness evaluation framework via multi-level query rephrasing.
• Providing two specialized training datasets: (i) a rephrased dataset and (ii) a finance-table-heavy dataset, where fine-tuning on them significantly enhances retrieval performance.

2 Related Work

2.1 Text-Based Retrieval
Text-based retrieval methods identify relevant documents given a query and are widely used in RAG systems. Lexical matching techniques like BM25 (Robertson et al., 1994) and TF-IDF (Sparck Jones, 1972) are efficient but lack semantic understanding. Sparse models like SPLADE (Formal et al., 2021) improve retrieval by expanding queries into high-dimensional sparse representations but struggle with deep contextual meaning. Dense retrieval models, leveraging transformers like BERT (Devlin, 2018), T5 (Raffel et al., 2020), and DPR (Karpukhin et al., 2020), map queries and documents into a continuous vector space, enhancing recall but demanding significant computational resources for training and inference. Hybrid methods, that combine lexical and dense retrieval, such as ColBERT (Khattab and Zaharia, 2020) and ANCE (Xiong et al., 2020), often achieving state-of-the-art performance. A recent model, M3-Embedding (Chen et al., 2024), unifies dense, sparse, and multi-vector embeddings, achieving strong retrieval performance. Despite advancements, text-based retrieval struggles with multi-modal content, particularly in scenarios where visual cues enhance contextual understanding.

2.2 Multi-Modal Retrieval
Until recently, multi-modal retrieval primarily relied on Optical Character Recognition (OCR) to extract textual information from documents, including text within visual elements. More recent approaches detect visual components and process them in one of two ways: (i) Captioning-based retrieval, where a VLM generates textual descriptions of visual elements, enabling standard text-based retrieval (Ramos et al., 2023); or (ii) Direct embedding, where visual elements are embedded either using VLMs directly or through contrastive Vision-Language Models that align separate visual and text encoders via contrastive losses (Radford et al., 2021; Zhai et al., 2023).

A more recent line of work leverages the strong performance of VLMs in analyzing full document images by embedding entire document pages instead of relying on OCR-based extraction. Methods such as VISRAG (Yu et al., 2024) and DSE (Ma et al., 2024a) generate dense embeddings directly from document images. Similarly, ColPali (Faysse et al., 2024) generates multi-vector embeddings for ColBERT-style late interaction retrieval, using PaliGemma (Beyer et al., 2024) or in a newer variant, ColQwen, utilizes Qwen2-VL (Wang et al., 2024a). These methods have demonstrated significant improvements over earlier text-based and OCR-dependent retrieval approaches. Our benchmark provides a rigorous evaluation framework for both text-based and visual-based multi-modal retrieval.

2.3 Query Rephrasing
Retrieval models known to be highly vulnerable to query rephrasing (Zuccon et al., 2016; Bailey et al., 2017; Sidiropoulos and Kanoulas, 2022; Penha et al., 2022; Hagen et al., 2024), often leading to significant performance degradation. However, only few works have provided accessible and reliable evaluation frameworks for model robustness. An early study (Bailey et al., 2016) introduced a basic Query Variability dataset, while more recent works (Benham et al., 2018; Lu et al., 2019; Penha et al., 2022) focus on automatically generating query variations. Yet, no prior research has established a standardized benchmark for retrieval
robustness, nor a dedicated RAG robustness benchmark—especially for multi-modal retrieval. In contrast, we leverage LLMs to generate multi-level query rephrasing, enabling structured comparative evaluation for multi-modal documents retrieval.

2.4 Multi-Modal Retrieval Benchmarks

Despite the growing importance of document retrieval in multi-modal RAG systems, only a few evaluation benchmarks exist, all of which fall short in key aspects crucial for real-world scenarios. We review recent efforts and highlight their limitations (with further comparison in Table 1 and section 4). While many question-answering benchmarks exist (Mathew et al., 2021; Zhu et al., 2022; Masry et al., 2022; Islam et al., 2023; Ding et al., 2024), they are largely unsuitable for RAG. Their queries assume exposure to a specific page (unlike RAG), and their tendency toward high variability make retrieval easier. Some benchmarks, such as MMLong-Bench (Ma et al., 2024b) (based on 130 lengthy PDFs) and SlideVQA (Tanaka et al., 2023), are partially relevant but not suited for RAG (see section 4).

A notable benchmark is WIKI-SS-NQ (Ma et al., 2024a), generated from Wikipedia screenshots with real human queries—the only dataset providing mostly valid retrieval queries. However, it is not multi-modal benchmark, consists of mainly text-based documents, and has narrow sub-domain coverage. The ViDoRe benchmark (Faysse et al., 2024), introduced in the ColPali paper, comprises both QA datasets and domain-specific documents with generated queries filtered by human annotators. While the QA datasets are unsuitable for RAG, the domain-specific queries are better tailored but suffer from trivial difficulty (e.g., synthetic datasets reporting an NDCG@5 > 95). This occurs because pages often differ significantly, and VLM-generated questions closely mirror the original page wording, making retrieval easy.

Our REAL-MM-RAG-Bench is the first multi-modal retrieval benchmark incorporating all essential properties for real-world RAG. It features a challenging setup with broad sub-domain coverage, long documents, RAG-tailored rephrased queries, and accurate labeling. Additionally, it is the first to offer a robustness evaluation through multi-level query rephrasing.

3 REAL-MM-RAG-Bench

Creating a high-quality benchmark manually is both exhaustive and error-prone, limiting its size and reliability. To address this, we propose an automated generation and verification pipeline tailored for Retrieval-Augmented Generation (RAG) evaluation. Our benchmark introduces robustness evaluation through multi-level query rephrasing, further improving upon previous benchmarks. The benchmark construction begins with document collection, followed by four key steps: (1) Query Generation, (2) Query Verification, (3) Query Rephrasing, and (4) False Negative Verification.

3.1 Document Collection

To reflect real-world retrieval challenges, we focus on long documents rather than isolated pages, and also ensuring many pages within the same sub-domain by focusing on a single company data (IBM). Our dataset consists of 8000 pages across four sub-domains, forming four specialized benchmarks (see Table S1 for details). For each page, we added the document name to the page image to provide context. FinReport: Financial reports (2005–2023), totaling 19 documents and 2687 pages, with a mix of text and tables. FinSlides: Quarterly financial presentations (2008–2024), totaling 65 presentations and 2280 pages, primarily table-heavy. TechReport: 17 Technical documents on FlashSystem, totaling 1674 pages, text-heavy with visual elements and tables. TechSlides: 62 Technical presentations on business and IT automation, totaling
<table>
  <tr>
    <th rowspan="2">Benchmark</th>
    <th colspan="2">Statistics</th>
    <th colspan="2">Multi-Modal</th>
    <th colspan="2">Enhanced Difficulty</th>
    <th colspan="2">Realistic-RAG Queries</th>
    <th>Accurate Labels</th>
  </tr>
  <tr>
    <th># Pages</th>
    <th># Queries</th>
    <th>MM Pages</th>
    <th>Long Docs</th>
    <th>Sub domain Cover</th>
    <th>Queries Rephrasing</th>
    <th>RAG Tailored Gen.</th>
    <th>RAG Query Verif.</th>
    <th>Neg. Verif.</th>
  </tr>
  <tr>
    <td>SlideVQA</td>
    <td>52k</td>
    <td>14.5k</td>
    <td>✓</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>MMLONG</td>
    <td>7k</td>
    <td>1k</td>
    <td>✓</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>WIKI-SS-NQ</td>
    <td>4k</td>
    <td>4k</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>ViDoRe</td>
    <td>8k</td>
    <td>4k</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>8k</td>
    <td>5k</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
</table>

Table 1: Document Retrieval Benchmarks Comparison.

1963 pages, with significant visual content.

3.2 Query Generation & Filtering

Generation. We aim to generate queries that are both answerable by a specific document and RAG-suitable, meaning they reflect natural user inquiries without prior knowledge of the exact page or answer location (unlike traditional Q/A datasets tied to specific pages). To achieve this, we employed a Pixtral-12B VLM (Agrawal et al., 2024), prompting it to generate RAG-specific questions (see Fig. S4). Each document page was fed into the VLM, which produced 10 query-answer pairs per page, later keeping only a subset that met the benchmark’s quality criteria after filtering. Each retained query-answer pair is labeled with the corresponding page it was generated from.

Verification. Although the VLM is instructed to generate RAG-specific queries, many still do not fully align with our requirements. To systematically classify them, we use Mixtral-8x22B-v0.1 LLM (Jiang et al., 2024), which evaluates each generated query and determines whether it is suitable as retrieval query (see prompt in Fig. S5). Queries that are well-formed for RAG are those that a user might ask without prior knowledge of the document’s structure, ensuring they are neither too general nor overly specific to a single page. Queries that fail this criterion fall into two categories: those with explicit page references, such as "in Figure 5" or "the title of the page", and those that are too broad, like "What is the net revenue in 2020?" instead of "What is IBM’s net revenue in 2020?".

3.3 Query Rephrasing

In real-world retrieval, a user formulating a query does not have direct access to the document’s content and will naturally phrase their question without mirroring the exact wording from the source. However, VLMs often generate queries by copying phrases directly, leading to an over-reliance on keyword matching rather than true semantic retrieval. To address this, we introduce a rephrasing step that preserves query meaning while reducing dependence on specific document wording. Each query is processed by Mixtral-8x22B-v0.1 with a dedicated prompt designed to alter phrasing while maintaining intent. The rephrased query is then verified by the LLM using a validation prompt (Fig. S6), along with the original query and answer, to ensure it retains the original meaning and still corresponds to the known answer in the labeled page.

To enable deeper evaluation, each query undergoes three levels of rephrasing using distinct prompts (Fig. S6). The first level introduces minor word changes while maintaining structure. The second modifies word choice and sentence order, making the phrasing more distinct. The third involves significant word rephrasing and sentence restructuring while preserving meaning. At the end of this process, each query exists in four versions: the original and three progressively rephrased forms, all linked to the same document page (see examples in Figs. S1 and S2).

3.4 Accurate Labeling

The final step in preparing our benchmark is verifying the correctness of negative labels. This is especially crucial for our challenging benchmarks, where many pages share highly similar content within the same sub-domain. Each query is systematically tested against all benchmark pages. Though computationally expensive, this step prevents false negatives and ensures reliable evaluation. Queries together with each page are processed using Pixtral-12B, which determines whether a page contains an answer to the query. Every query is then explicitly linked to all relevant pages. For
simplicity, our final benchmark retains queries whose only the originally assigned page is verified to contain the correct answer. This results in a high-quality dataset of triplets: a page image, a query, and its corresponding answer. Note that our benchmark includes pages without corresponding queries. These are pages whose queries were filtered out at some stage, either because they were not suitable for RAG-style questions in general (e.g., title pages) or because the specific generated queries were not suitable for RAG.

4 Benchmarks Quality Evaluation

A high-quality benchmark for multi-modal retrieval is essential, yet few existing benchmarks are designed for this purpose, and none comprehensively define or implement the necessary properties. Table 1 compares our benchmark with other prominent ones, which suffer from limitations such as poor alignment with real-world queries, high false-negative rates, and trivial difficulty.

Accurate Labeling. Many perceived retrieval errors in existing benchmarks are actually false negatives, meaning pages that correctly answer the query but were mislabeled as irrelevant. To mitigate this, we introduce a false-negative verification process that exhaustively labels all valid pages. Human Evaluation. We sampled 50 top-1 retrieval errors of ColQwen on Vidore, MMlongbench, and our benchmark. Annotators reviewed the query and retrieved page (labeled as negative) to determine if it could answer the query (Fig. S8). A total of 234 responses from 5 annotators were collected.

<table>
  <tr>
    <th></th>
    <th>Vidore</th>
    <th>MMLong</th>
    <th>Ours</th>
  </tr>
  <tr>
    <th>False Negative (%) ↓</th>
    <td>86.9</td>
    <td>77.8</td>
    <td>31.9</td>
  </tr>
</table>

The table shows that Vidore and MMlongbench had a high rate of false negatives, whereas our benchmark, despite its challenging design with similar sub-domain pages, had significantly fewer, proving the effectiveness of accurate labeling.

Enhanced Difficulty. A strong benchmark must pose real challenges. Existing ones fall short by offering too few relevant candidates or allowing retrieval via simple keyword matching rather than true semantic understanding. For example, having 1,000 financial pages from different companies is insufficient, as knowing the company name narrows the candidates to a few dozen. The ColQwen model achieves an NDCG@5 of around 90 on Vidore. Other sub-datasets, although reporting lower performance, contain many errors that are actually false negatives, as demonstrated by our human evaluation presented above. We address this issue through accurate labeling and by incorporating long documents and extensive sub-domain coverage. This provides many similar pages, making retrieval more challenging and better reflecting real-world scenarios. Moreover, we prevent trivial keyword-based retrieval by introducing the first rephrasing benchmark for multi-modal document RAG, ensuring robustness to query variations and promoting semantic learning.

Realistic-RAG Queries. To reflect real RAG use cases, queries must resemble natural information-seeking questions. Our benchmark ensures this through a two-step RAG-tailored pipeline: generation and filtering. Human Evaluation. We randomly sampled 500 queries from Vidore, MMLongBench, and our benchmark. Annotators, unaware of the source benchmark or study goal, evaluated whether each query could reasonably be asked by a real user (Fig. S7). A total of 2578 responses were collected from 10 annotators.

<table>
  <tr>
    <th></th>
    <th>Vidore</th>
    <th>MMLong</th>
    <th>Ours</th>
  </tr>
  <tr>
    <th>Realistic-RAG Queries (%) ↑</th>
    <td>32.9</td>
    <td>35.8</td>
    <td>88.0</td>
  </tr>
</table>

The table shows that most Vidore/MMLongBench queries were labeled as unrealistic RAG queries (see some examples in Fig. S3), whereas 85% of ours were validated as realistic, highlighting shortcomings in existing benchmarks and the effectiveness of our query generation and filtering process.

Summary. Our benchmark enhances multi-modal retrieval evaluation by introducing non-trivial difficulty with long documents and broad sub-domain coverage. It ensures RAG-aligned queries and promotes semantic retrieval over keyword matching through query rephrasing, addressing key limitations of existing benchmarks.

5 Model Evaluation & Enhancement

Evaluation Models and Metrics. We evaluate multiple models on our benchmark, covering both text and vision-based approaches. We use the ColPali benchmark code (ViDoRe) to assess our text-based models and the vision-based models ColPali and ColQwen. For the text-based methods, we follow the framework suggested in ColPali, which employs Unstructured in a high-resolution configuration with OCR engine to parse PDFs. For each document, Unstructured produces text chunks and
<table>
  <tr>
    <th>Benchmark</th>
    <th>FinReport</th>
    <th>FinSlides</th>
    <th>TechReport</th>
    <th>TechSlides</th>
  </tr>
  <tr>
    <th colspan="5">Text</th>
  </tr>
  <tr>
    <td><i>BM25 (OCR)</i></td>
    <td>21.7</td>
    <td>5.9</td>
    <td>35.1</td>
    <td>31.2</td>
  </tr>
  <tr>
    <td><i>BGE-M3 (OCR)</i></td>
    <td>36.5</td>
    <td>11.4</td>
    <td>37.1</td>
    <td>49.7</td>
  </tr>
  <tr>
    <td><i>BM25 (Captioning)</i></td>
    <td>25.3</td>
    <td>9.9</td>
    <td>37.2</td>
    <td>36.1</td>
  </tr>
  <tr>
    <td><i>BGE-M3 (Captioning)</i></td>
    <td>35.9</td>
    <td>13.8</td>
    <td>37.5</td>
    <td>51.7</td>
  </tr>
  <tr>
    <th colspan="5">Vision</th>
  </tr>
  <tr>
    <td>ColPali</td>
    <td>34.5</td>
    <td>27.6</td>
    <td>62.0</td>
    <td>75.8</td>
  </tr>
  <tr>
    <td><i>RobColPali</i></td>
    <td>47.1 <i>+12.6</i></td>
    <td>48.4 <i>+20.8</i></td>
    <td>66.6 <i>+4.60</i></td>
    <td>82.8 <i>+7.0</i></td>
  </tr>
  <tr>
    <td><i>TabColPali</i></td>
    <td>50.5 <i>+16.0</i></td>
    <td>41.5 <i>+13.9</i></td>
    <td>61.3 <i>+0.7</i></td>
    <td>77.6 <i>+1.8</i></td>
  </tr>
  <tr>
    <td><i>RobTabColPali</i></td>
    <td>63.2 <i>+28.7</i></td>
    <td>58.3 <i>+30.7</i></td>
    <td>70.7 <i>+8.7</i></td>
    <td>83.3 <i>+7.5</i></td>
  </tr>
  <tr>
    <td>ColQwen</td>
    <td>41.8</td>
    <td>31.1</td>
    <td>66.9</td>
    <td>78.1</td>
  </tr>
  <tr>
    <td><i>RobColQwen</i></td>
    <td>47.5 <i>+5.7</i></td>
    <td>44.3 <i>+13.2</i></td>
    <td>69.5 <i>+2.6</i></td>
    <td>83.0 <i>+4.9</i></td>
  </tr>
  <tr>
    <td><i>TabColQwen</i></td>
    <td>54.0 <i>+12.2</i></td>
    <td>49.6 <i>+18.5</i></td>
    <td>65.9 <i>+1.0</i></td>
    <td>78.9 <i>+0.8</i></td>
  </tr>
  <tr>
    <td><i>RobTabColQwen</i></td>
    <td><b>67.1 <i>+25.3</i></b></td>
    <td><b>61.6 <i>+30.5</i></b></td>
    <td><b>73.2 <i>+6.3</i></b></td>
    <td><b>85.0 <i>+6.9</i></b></td>
  </tr>
</table>

Table 2: Performance of Different Models on Our Benchmark. We evaluate various models, including text- and vision-based approaches, across our four benchmarks. Results, measured using NDCG@5, are reported on our final benchmark with queries rephrased at the highest level (Level 3). We also present results for our fine-tuned models trained on our proposed datasets: Rob – trained on a rephrased dataset, Tab – trained on a table-heavy dataset, and RobTab – incorporating both.

visual chunks (e.g., tables, figures, images). We consider two text-based variants: (1) OCR, where visual data is processed through an OCR engine, and (2) Captioning, where visual elements are described using a Vision Language Model (Qwen2-VL-72B-Instruct (Wang et al., 2024b)). We then evaluate two retrieval methods: Okapi BM25 and BGE-M3 (Chen et al., 2024) (see appendix A.1 for more details). We report NDCG@5 as our primary ranking metric, which evaluates how well relevant items are ranked within the top 5 results, giving higher importance to those appearing earlier. Additional metrics and details provided in A.1 & A.5.

5.1 Results Analysis

In Table 2, we report NDCG@5 performance for different models across our four benchmarks with high rephrasing levels, which better reflect real-world scenarios. We first present observations about the vanilla models, including text-based models, ColPali, and ColQwen: Visual vs. Text-Based Models. Vision-based models, which use VLMs on page images, significantly outperform text-based models across all benchmarks. This supports the notion that visual information is essential for our benchmark and that these models can effectively utilize it. Non-Trivial Difficulty.

Performance is generally low, especially compared to Vidore, where ColQwen achieves nearly 90% on average. Rephrasing Effects on Performance. Some of the drop in performance is due to rephrasing. In Table 3, we analyze the impact of rephrasing level, showing a clear performance drop as rephrasing intensity increases. BM25 suffers the most, as expected for a lexical-based model, while dense retrieval models are more resilient. Table-Heavy Finance Benchmarks Are Harder. Our financial benchmarks (FinReport, FinSlides), which are table-heavy, are significantly more challenging than text/visual-based ones (see table vs. non-table analysis in Table S5).

5.2 Table and Finance Focused Training

To address the challenges in table-heavy datasets, we curated a table-focused finance dataset using FinTabNet (Zheng et al., 2021), which contains complex tables from S&P 500 company reports. Through the pipeline in section 3, we generated 46,000 query-answer-page triplets to enhance retrieval for table-heavy financial data (see Table S7 for a VLM ablation study). We fine-tuned ColPali and ColQwen for one epoch on this dataset while incorporating the ColPali training set, producing the TabCol models.

As shown in Fig. 3 and Table 2, TabCol models significantly improve performance on financial benchmarks, effectively addressing table-heavy dataset challenges. Importantly, this enhancement does not come at the cost of generalization, as TabCol models continue well across the two other benchmarks.

![Bar chart comparing NDCG@5 performance for ColPali, TabColPali, RobColPali, ColQwen, TabColQwen, RobColQwen, FinReport, and FinSlides](page_1012_1342_388_246.png)

Figure 3: Table-Focused Training Improves Financial Benchmarks. Fine-tuning with our proposed table-heavy training set, combined with the ColPali training set (both in their original and rephrased versions) significantly enhances performance on financial benchmarks (results shown for rephrasing level 3).
<table>
  <tr>
    <th>Rephrasing Levels</th>
    <th>0</th>
    <th>1</th>
    <th>2</th>
    <th>3</th>
  </tr>
  <tr>
    <td>BM25 (Captioning)</td>
    <td>52.7</td>
    <td>41.6</td>
    <td>31.3</td>
    <td>27.1</td>
  </tr>
  <tr>
    <td>BGE-M3 (Captioning)</td>
    <td>43.3</td>
    <td>39.2</td>
    <td>36.5</td>
    <td>34.7</td>
  </tr>
  <tr>
    <td>ColPali</td>
    <td>71.3</td>
    <td>66.1</td>
    <td>56.0</td>
    <td>50.6</td>
  </tr>
  <tr>
    <td><b>RobColPali</b></td>
    <td>76.7</td>
    <td>73.9</td>
    <td>66.0</td>
    <td>61.2</td>
  </tr>
  <tr>
    <td><b>RobTabColPali</b></td>
    <td>80.8</td>
    <td>77.8</td>
    <td>72.5</td>
    <td>68.9</td>
  </tr>
  <tr>
    <td>ColQwen</td>
    <td>73.9</td>
    <td>67.5</td>
    <td>59.6</td>
    <td>54.5</td>
  </tr>
  <tr>
    <td><b>RobColQwen</b></td>
    <td>74.1</td>
    <td>70.2</td>
    <td>64.4</td>
    <td>61.1</td>
  </tr>
  <tr>
    <td><b>RobTabColQwen</b></td>
    <td>83.9</td>
    <td>80.6</td>
    <td>75.1</td>
    <td>71.7</td>
  </tr>
</table>

Table 3: <b>Query Rephrasing Effect.</b> This table presents the NDCG@5 scores averaged across all benchmarks for different models and rephrasing levels. '0' represents no rephrasing, while '3' indicates significant rephrasing (see Table S2 for full results).

5.3 Rephrasing Robustness Training

Our benchmark reveals that current models struggle with rephrasing, suggesting that training and evaluation queries often closely match the phrasing of their retrieved pages. To address this, we augmented the ColPali training set by rephrasing half of its queries, randomly selecting one of three rephrasing levels. This was done using LLaMA-3-70B\footnote{https://huggingface.co/meta-llama/Llama-3-70B-Instruct}, a different LLM than the one used for the benchmark. This dataset forces models to learn semantics rather than relying on keyword matching. We fine-tuned ColPali-v1.2 and ColQwen2-v1.0\footnote{https://huggingface.co/vidore/colqwen2-v1.0} (using the ColPali code) for one epoch, producing the RobCol models.

As demonstrated in Fig. 4 and Table 3 (and in Tables 2 and S2), RobCol significantly outperforms baselines on rephrased queries while maintaining comparable performance on non-rephrased cases, achieving an average NDCG@5 improvement of 11.1 at rephrasing level 3. The gains are stronger on financial benchmarks, where the task is more complex, suggesting that enhanced semantic understanding through rephrasing robustness is particularly beneficial. In Table S6, we show that fine-tuning on the original data (rather than the rephrased version) provides much lower improvement for ColPali and leads to a decrease in performance for ColQwen.

Additionally, we created a rephrased version of the tabled-focused dataset and fine-tuned both models on it, along with the rephrased ColPali training set, producing RobTabCol models. RobTabCol consistently outperforms all models across nearly all benchmarks and rephrasing levels, achieving a 25–30 NDCG@5 improvement over base models on rephrased finance benchmarks and 6–9 on non-financial ones. We further show in appendix A.6 that our RobTab models outperform the baselines on the newly released ViDoRe V2 benchmark.

![Line plots comparing NDCG@5 scores for baseline and fine-tuned models across rephrasing levels](page_1012_180_420_312.png)

Figure 4: <b>Fine-Tuning on Rephrased Training Set.</b> We compare the NDCG@5 scores across rephrasing levels for baseline models (ColPali and ColQwen) against our fine-tuned models (RobCol). The results demonstrate that fine-tuning with our rephrased training data significantly enhances rephrasing robustness for both ColPali and ColQwen.

all benchmarks and rephrasing levels, achieving a 25–30 NDCG@5 improvement over base models on rephrased finance benchmarks and 6–9 on non-financial ones. We further show in appendix A.6 that our RobTab models outperform the baselines on the newly released ViDoRe V2 benchmark.

6 Conclusions

We introduced <i>REAL-MM-RAG-Bench</i>, a real-world multi-modal retrieval benchmark designed to evaluate retrieval models in <i>reliable, challenging, and realistic</i> settings. Our benchmark addresses key properties essential for evaluating retrieval systems in real-world applications, which prior benchmarks often fail to capture. An important contribution of our work is the introduction of a <i>multi-level rephrasing evaluation</i>, which assesses models under increasing linguistic variation, highlighting their limitations in generalizing beyond surface-level text matching.

Our findings reveal two major weaknesses in current models: (i) retrieval over table-heavy financial documents and (ii) sensitivity to query rephrasing. To address these, we proposed dedicated training sets: a <i>finance-table-heavy dataset</i> to improve retrieval on tabular content and a <i>rephrased dataset</i> to enhance model robustness to query variations. Fine-tuning on these datasets yields significant improvements across benchmarks, demonstrating the impact of targeted training data. <i>REAL-MM-RAG-Bench</i> and our proposed solutions establish a foundation for future research, paving the way for robust and effective retrieval models in real-world multi-modal retrieval scenarios.
Limitations

While REAL-MM-RAG-Bench presents a comprehensive and realistic evaluation framework for multi-modal retrieval, several limitations remain:

Query Variability: Our queries are generated using a Vision-Language Model (VLM), which, while effective, may not fully capture the full range of plausible user queries.

LLM and VLM Limitations: Despite the strength of modern Large Language Models (LLMs) and VLMs, our filtering strategies and labeling process remain subject to their limitations. While our human evaluation confirms the effectiveness of our approach, errors in labeling and query selection may still occur. As LLMs and VLMs continue to improve, future benchmarks could leverage more accurate models to refine dataset construction further.

Multi-Page Reasoning Queries: Our benchmark is designed to best evaluate the retrieval component of Retrieval-Augmented Generation (RAG). While the dataset can be used for the generation step as well, it does not explicitly assess multi-page reasoning. Future work could explore automated query generation that combines multiple pages using LLMs and/or VLMs to construct multi-page reasoning tasks, enhancing the benchmark’s ability to evaluate complex retrieval scenarios.

REAL-MM-RAG-Bench provides a realistic, reliable, and challenging retrieval benchmark, helping to identify critical weaknesses in current multi-modal retrieval models and paving the way for future improvements in both evaluation and model development.

References

Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Moncault, Saurabh Garg, Theophile Gervet, et al. 2024. Pixtral 12b. arXiv preprint arXiv:2410.07073.

Peter Bailey, Alistair Moffat, Falk Scholer, and Paul Thomas. 2016. Uqv100: A test collection with query variability. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 725–728.

Peter Bailey, Alistair Moffat, Falk Scholer, and Paul Thomas. 2017. Retrieval consistency in the presence of query variations. In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval, pages 395–404.

Rodger Benham, J Shane Culpepper, Luke Gallagher, Xiaolu Lu, and Joel M Mackenzie. 2018. Towards efficient and effective query variant generation. In DESIRES, pages 62–67.

Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. 2024. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726.

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216.

Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Yihao Ding, Kaixuan Ren, Jiabin Huang, Siwen Luo, and Soyeon Caren Han. 2024. Mvqa: A dataset for multimodal information retrieval in pdf-based visual question answering. arXiv preprint arXiv:2404.12720.

Manuel Faysse, Hugues Sibille, Tony Wu, Bilal Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2024. Colpali: Efficient document retrieval with vision language models. arXiv preprint arXiv:2407.01449.

Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2288–2292.

Tim Hagen, Harrison Scells, and Martin Potthast. 2024. Revisiting query variation robustness of transformer models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4283–4296.

Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. Financebench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.11944.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088.

Vladimir Karputhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.

Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd
International ACM SIGIR conference on research and development in Information Retrieval, pages 39–48.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Xiaolu Lu, Oren Kurland, J Shane Culpepper, Nick Craswell, and Ofri Rom. 2019. Relevance modeling with multiple query variations. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, pages 27–34.

Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhui Chen, and Jimmy Lin. 2024a. Unifying multimodal retrieval via document screenshot embedding. arXiv preprint arXiv:2406.11251.

Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Liu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. 2024b. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations. arXiv preprint arXiv:2407.01523.

Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244.

Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209.

Gustavo Penha, Arthur Câmara, and Claudia Hauff. 2022. Evaluating the robustness of retrieval pipelines with query variation generators. In European conference on information retrieval, pages 397–412. Springer.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.

Rita Ramos, Desmond Elliott, and Bruno Martins. 2023. Retrieval-augmented image captioning. arXiv preprint arXiv:2302.08268.

S Robertson, Steve Walker, Susan Jones, and MHB GATFORD. 1994. Okapi at 3. In Proceedings of the 3rd Text REtrieval Conference (-3), pages 109–126.

Georgios Sidiroopoulos and Evangelos Kanoulas. 2022. Analysing the robustness of dual encoders for dense retrieval against misspellings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2132–2136.

Alan F Smeaton and Fergus Kelly. 1998. User-chosen phrases in interactive query formulation for information retrieval. In 20th Annual BCS-IRSG Colloquium on IR. BCS Learning & Development.

Ray Smith. 2007. An overview of the tesseract ocr engine. In Ninth international conference on document analysis and recognition (ICDAR 2007), volume 2, pages 629–633. IEEE.

Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11–21.

Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. 2023. Slidevqa: A dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13636–13645.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024a. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808.

Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. 2024. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. arXiv preprint arXiv:2410.10594.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986.

Xinyi Zheng, Doug Burdick, Lucian Popa, Peter Zhong, and Nancy Xin Ru Wang. 2021. Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context. Winter Conference for Applications in Computer Vision (WACV).
Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua. 2022. Towards complex document understanding by discrete reasoning. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4857–4866.

Hongyi Zhu, Jia-Hong Huang, Stevan Rudinac, and Evangelos Kanoulas. 2024. Enhancing interactive image retrieval with query rewriting using large language models and vision language models. In Proceedings of the 2024 International Conference on Multimedia Retrieval, pages 978–987.

Guido Zuccon, Joao Palotti, and Allan Hanbury. 2016. Query variations and their effect on comparing information retrieval systems. In Proceedings of the 25th ACM international on conference on information and knowledge management, pages 691–700.
A Appendix

A.1 Models Evaluation

Beside the models from ColPali, we have evaluated on text based methods. Following the ColPali framework, we adopt Unstructured as our PDF parser in its high-resolution configuration, which relies on the Tesseract (Smith, 2007) OCR engine. Unstructured processes each document into two main types of chunks: text chunks and visual chunks (e.g., tables, figures, images). We then construct two text-based variants of our benchmark, differing in how visual chunks are converted into text:

1. OCR: Text is retained; tables, figures, and images undergo OCR extraction.

2. Captioning: Text remains unchanged; visual elements are described using Qwen2.5-VL-72B-Instruct.

Retrieval Methods. We evaluate two retrieval approaches:

• Okapi BM25: A sparse statistical baseline.

• BGE-M3 (multi-vector): A state-of-the-art embedding model.

In line with ColPali, chunks are embedded and scored independently, and page-level scores are then obtained via maximum pooling across all chunks for a given page.

A.2 Data Generation, Filtering and Validation

To construct a high-quality benchmark, we applied a conservative filtering strategy that prioritizes precision over recall—favoring the exclusion of low-quality queries even at the cost of discarding valid ones. Setting a precise confidence threshold for VLMs/LLMs is non-trivial, so we experimented with multiple prompt templates for filtering. For each prompt, we sampled 150 queries, manually labeled them as good or bad, and selected the prompt that minimized the percentage of bad queries that passed the filter. Since the original pool of generated queries was large, our emphasis was on reducing false positives rather than preserving query volume.

We then manually reviewed queries that passed the filter and added representative errors as negative examples to the prompt context, improving robustness through iterative refinement. This process led to fewer recurring mistakes and was validated through human evaluation, where 85.0% of our queries were rated as good compared to 43.6% for ViDoRe.

Validating label correctness at scale is more challenging, as it would require exhaustively comparing each query to all documents. Instead, we qualitatively inspected queries with known answer pages to tune the labeling prompt. As shown in section 4, our benchmark yields significantly fewer false negatives than competing datasets.

We also conducted a false positive validation to assess label quality. Specifically, we randomly sampled 200 query–page pairs from each of the three benchmarks: ViDoRe, MMLong, and ours. Annotators were shown the page image alongside its corresponding query and asked whether the page contained an answer to the query. The results show 0% false positives in both our benchmark and ViDoRe, while MMLong exhibited a high false positive rate of 38.1%. These findings highlight the effectiveness of our benchmark generation pipeline in preventing false positive examples.

A.3 Proposed Training sets and Fine-tuning

Rephrasing Augmentation Training. We aimed to fine-tune a trained model with a short training phase to improve robustness to rephrasing. To achieve this, we created a rephrased training set based on the ColPali training data. Specifically, we used approximately half of the full training set (56k queries) and generated rephrased versions. The rephrasing was performed using LLaMA-3-70B, a different LLM than the one used for filtering and rephrasing in the benchmark.

Each query was randomly rephrased using one of three rephrasing levels (different prompts, see Fig. S6). To ensure semantic consistency, we used a secondary LLM verification step: the original query and its corresponding answer were fed alongside the rephrased query, and if the meaning was not preserved, the original query was retained in the rephrased training set.

For fine-tuning, we used the full ColPali training set, incorporating the rephrased queries in approximately half of it. The model was trained for one epoch with configurations similar to those used in the original ColPali/ColQwen training.

Table and Finance-Focused Training. As we observed that current models performed signifi-
cantly worse on our table-heavy finance benchmark, we curated a dedicated training set to address this gap. We leveraged the publicly available FinTabNet dataset, a large-scale resource designed for financial table recognition and structure extraction. FinTabNet consists of pages from annual reports of S&P 500 companies, featuring complex tables.

We used the page images from FinTabNet to generate queries and answers using our automated pipeline, which includes a filtering process. Additionally, we explored using Qwen2-VL-72B for generating this training data, with results reported in Table S7. This process resulted in approximately 46k triplets of page images, queries, and answers.

For fine-tuning, we trained both ColPali and ColQwen on our table-focused training data, combined with the original ColPali training set, for one epoch, producing the TabCol models. For RobTab models, we incorporated LLaMA-3-70B for rephrasing, where half of the newly generated training set was randomly rephrased using three different rephrasing levels. These models were then fine-tuned for one epoch, together with the ColPali rephrased dataset.

All fine-tuning procedures followed the same configurations as the official ColPali training pipeline, utilizing ColBERT in-batch loss. The specific configurations for ColPali and ColQwen can be found at GitHub. Each model was fine-tuned starting from its respective base version: ColPali from ColPali-1.2 and ColQwen from ColQwen2-1.0. We used a batch size of 64 per GPU, resulting in a total effective batch size of 256. The training maintained the same initial learning rate, warmup steps, learning rate schedule, and LoRA configuration as the original pipeline. All training runs were conducted on four A100 80GB GPUs, with each fine-tuning session taking approximately three hours to complete.

A.4 Benchmark Document Examples

Our benchmarks include a diverse set of pages containing different types of information, including full-text pages, table-heavy pages, and slides with both tables and other visual elements. Table S1 presents the benchmark statistics for the different datasets. Additionally, for each page, we provide four types of queries: the original VLM-generated query and three levels of rephrasing. Examples of pages along with all query versions are shown in Figs. S1 and S2. In Fig. S3, we present examples from previous benchmarks, highlighting that many of their queries are not well-suited for RAG, as they often reference specific pages (i.e., QA-style queries) rather than general information-seeking queries.

A.5 Additional Results and Ablations

In our main paper, we focused on reporting the NDCG@5 metric for a subset of models and benchmarks across different rephrasing levels. In Tables S2 to S4, we provide the full results for Recall@1, Recall@5, and NDCG@5. Additionally, Table S5 presents model performance across different evidence source types. A Vision-Language Model (VLM), Pixtral-12B, was used to classify each query based on the type of evidence source from which the answer was retrieved on the corresponding page. We analyze performance across three different evidence types: Text, Table, and Visual. The reported results show NDCG@5 scores on the non-rephrased version, averaged across our four benchmarks. These results highlight a significant weakness in handling tables. However, after fine-tuning on the table-focused dataset, we observe improvements across all evidence types, with tables showing the most substantial gains.

We further provide two ablation experiments. The first, shown in Table S6, aims to demonstrate that the observed improvements after fine-tuning with our proposed datasets are due to the tailored data rather than fine-tuning itself. To verify this, we performed an additional fine-tuning run using the same training set as the RobCol models but without rephrasing, maintaining the exact number of training examples and identical training configurations.

As seen in the results, ColPali gains some improvement from fine-tuning alone, but the gains are significantly lower compared to fine-tuning with our rephrased training set. This gap is even more pronounced when evaluating on the rephrased benchmark (Level 3). For ColQwen, the baseline fine-tuning without rephrasing leads to a decrease in performance, whereas our fine-tuned models show substantial improvements on rephrased queries, as expected when training with our rephrased dataset.

The second ablation (see Table S7) aims to show that the improvements from the table-heavy training set are general and not specific to the Vision-Language Model (VLM) used for question generation. To test this, we generated an alternative
version of the training set using a different VLM, Qwen2-VL-72B-Instruct, and trained models both with and without rephrasing (using LLaMA-3-3-70B). After filtering, the dataset size was slightly smaller—40k examples compared to 46k with Pixtral generation—likely due to Qwen generating more queries that were filtered out.

While results show a slight decrease in performance compared to using Pixtral-generated data, the fine-tuned models still significantly outperform the base ColPali and ColQwen models. This confirms that the effectiveness of our data is not limited to a specific VLM and that training on a table-heavy dataset remains highly beneficial.

A.6 Our Models’ Results on ViDoRe V2
We report NDCG@5 performance on the newly released ViDoRe V2 benchmark, both before and after fine-tuning. As shown in Table S8, we compare base models (ColPali and ColQwen), models fine-tuned on the rephrased ColPali training set (Rob variants), and models fine-tuned with our table-focused and rephrased training set (RobTab variants).

Across 4 out 5 English benchmarks, we observe consistent improvements from fine-tuning, with particularly large gains for ColPali-based models. Among all methods, RobTabColQwen achieves the highest average NDCG@5 score, highlighting the effectiveness of our combined rephrasing and table-focused training strategy.

We further evaluate performance on the multilingual subset of ViDoRe V2 (Table S9). Here, fine-tuning also improves performance over the base models, with an even greater margin between RobTabColQwen and ColQwen. This suggests that rephrasing augmentation enhances the model’s ability to capture semantic similarity beyond surface-level token overlap, which is especially beneficial for cross-lingual retrieval.

Overall, these results demonstrate the generalization ability of our fine-tuned models across both diverse content types and languages.

A.7 Licensing and Additional General Information
All models and datasets used in this work comply with their respective licenses. Qwen2-VL (ColQwen2) is licensed under Apache 2.0, with adapters under MIT. PaliGemma (ColPali) follows the Gemma license, with adapters under MIT. Pixtral-12B-2409 (mistralai) and Mixtral-8x22B are both under Apache 2.0, allowing unrestricted use, modification, and distribution. LLaMA 3.3 70B is licensed under the LLaMA 3.3 Community License Agreement. All datasets used are in English. The Colapli training set consists of sub-sampled academic datasets redistributed under their original licenses. It also includes synthetic datasets generated from publicly available internet data and VLM-generated queries, which are released without usage restrictions. Benchmark datasets are derived from openly available documents and images, with owner approval for publication. Fine-tuning data (Fintabnet) is collected from publicly available sources and processed for compatibility with our models.

For human evaluation, we published an online form alongside a request for participation in annotating queries for evaluating data intended for publication. Throughout the paper, an AI assistant (ChatGPT) was used for minor grammar and sentence structure edits.
Table S1: Benchmark Statistics.

<table>
  <tr>
    <th rowspan="2">Benchmark</th>
    <th colspan="3">Documents</th>
    <th colspan="3">Queries</th>
    <th colspan="3">Evidence Label</th>
  </tr>
  <tr>
    <th># Pages</th>
    <th># Docs</th>
    <th>Avg. Len</th>
    <th># Queries</th>
    <th>Text</th>
    <th>Table</th>
    <th>Visual</th>
  </tr>
  <tr>
    <td>FinReport</td>
    <td>2687</td>
    <td>19</td>
    <td>141</td>
    <td>853</td>
    <td>75%</td>
    <td>24%</td>
    <td>1%</td>
  </tr>
  <tr>
    <td>FinSlides</td>
    <td>2280</td>
    <td>65</td>
    <td>35</td>
    <td>1052</td>
    <td>12%</td>
    <td>83%</td>
    <td>5%</td>
  </tr>
  <tr>
    <td>TechReport</td>
    <td>1674</td>
    <td>17</td>
    <td>98</td>
    <td>1294</td>
    <td>81%</td>
    <td>12%</td>
    <td>7%</td>
  </tr>
  <tr>
    <td>TechSlides</td>
    <td>1963</td>
    <td>62</td>
    <td>32</td>
    <td>1354</td>
    <td>66%</td>
    <td>6%</td>
    <td>28%</td>
  </tr>
</table>

What is the main role of GenAI in the context of IBM Workload Automation?
L1 What is the primary function of GenAI within the framework of IBM Workload Automation?
L2 How does GenAI function within the framework of IBM Workload Automation, and what primary purpose does it serve?
L3 How does Generative Artificial Intelligence function as a key component within the framework of IBM’s automated workload management systems?

The Future of Work: Human-AI Collaboration

• GenAI is a powerful tool for augmenting automation and enhancing human capabilities.
• Human-AI collaboration is the key to unlocking the full potential of GenAI in the workplace.
• By working together, humans and AI can achieve greater efficiency, productivity, and innovation.

What was the total amount of signings for IBM’s Global Services in the second quarter of 2012?
L1 What was the overall value of contracts for IBM’s global services in the second quarter of 2012?
L2 How much were the total contract signings valued for IBM’s Global Services division during Q2 of 2012?
L3 How much was the cumulative value of contracts secured by IBM’s Global Services division during the three-month period ending June 30, 2012?

Supplemental Segment Information – 20 2012
Supplemental Revenue Information
<table>
  <tr>
    <th></th>
    <th>Accrued Growth</th>
    <th>2m Billion</th>
    <th>2012</th>
    <th>2011</th>
    <th>2010</th>
  </tr>
  <tr>
    <td>GTS Outsourcing</td>
    <td>(1%)</td>
    <td>816</td>
    <td>1,016</td>
    <td>1,016</td>
    <td>1,016</td>
  </tr>
  <tr>
    <td>Integrated Tech Services</td>
    <td>Flat</td>
    <td>4%</td>
    <td>1,016</td>
    <td>1,016</td>
    <td>1,016</td>
  </tr>
  <tr>
    <td>Maintenance</td>
    <td>(1%)</td>
    <td>2%</td>
    <td>Quarter-to-Quarter</td>
    <td>(8)</td>
    <td>(8)</td>
  </tr>
  <tr>
    <td>Total GTS</td>
    <td>(1%)</td>
    <td>2%</td>
    <td>1,016</td>
    <td>1,016</td>
    <td>1,016</td>
  </tr>
  <tr>
    <td>GBS Outsourcing</td>
    <td>(1%)</td>
    <td>1%</td>
    <td>2m Billion</td>
    <td>2012</td>
    <td>2011</td>
    <td>2010</td>
  </tr>
  <tr>
    <td>GBS Maintenance</td>
    <td>(1%)</td>
    <td>1%</td>
    <td>2m Billion</td>
    <td>2012</td>
    <td>2011</td>
    <td>2010</td>
  </tr>
  <tr>
    <td>Total GBS</td>
    <td>(1%)</td>
    <td>1%</td>
    <td>2m Billion</td>
    <td>2012</td>
    <td>2011</td>
    <td>2010</td>
  </tr>
  <tr>
    <td>Total Outsourcing</td>
    <td>(1%)</td>
    <td>1%</td>
    <td>2m Billion</td>
    <td>2012</td>
    <td>2011</td>
    <td>2010</td>
  </tr>
  <tr>
    <td>Total Transactional</td>
    <td>(1%)</td>
    <td>1%</td>
    <td>2m Billion</td>
    <td>2012</td>
    <td>2011</td>
    <td>2010</td>
  </tr>
  <tr>
    <td>Maintenance</td>
    <td>(1%)</td>
    <td>2%</td>
    <td>2m Billion</td>
    <td>2012</td>
    <td>2011</td>
    <td>2010</td>
  </tr>
  <tr>
    <td>Total Signings</td>
    <td></td>
    <td></td>
    <td>$13.7</td>
    <td>(1%)</td>
    <td>Flat</td>
  </tr>
</table>

Note: Actual booking illustrated using a mix of currency split off.

Figure S1: Real-MM-RAG Benchmark Examples with Rephrasing. On the right: FinSlides—financial quarterly presentations. On the left: TechSlides Technical slides about business and IT automation. Questions are listed from the original query to Level 3 rephrasing.
What types of benefits are provided under the Non-U.S. Retirement Plan at IBM?

L1 What are the specific benefits included in IBM’s non-US retirement plan?
L2 What advantages or perks does IBM’s non-US retirement program offer to its participants?
L3 At IBM, what advantages and perks can employees expect to receive as part of their retirement package outside of the United States?

Financial Report of the company IBM on 2019

Notes to Consolidated Financial Statements
International Business Machines Corporation and Subsidiary Companies

NOTE V. RETIREMENT-RELATED BENEFITS
Description of Plans
IBM sponsors the following retirement-related plans/benefits:

<table>
  <tr>
    <th>Plan</th>
    <th>Eligibility</th>
    <th>Funding</th>
    <th>Benefit Calculation</th>
    <th>Other</th>
  </tr>
  <tr>
    <td>U.S. Defined Benefit Pension Plan (PP)</td>
    <td>Qualified Personnel</td>
    <td>U.S. regular, full-time, non-union employees hired after December 7, 2005</td>
    <td>Company contributes based on salary and benefit years</td>
    <td>Vary based on the five-year, final pay formula</td>
    <td>Benefit accrued through December 31, 2007</td>
  </tr>
  <tr>
    <td>Excess Personal Savings Plan (EPS)</td>
    <td></td>
    <td>Unfunded, provides benefit based on salary and benefit years</td>
    <td></td>
    <td>Cash balance formula</td>
    <td></td>
  </tr>
  <tr>
    <td>Supplemental Executive Retirement Plan (SERP)</td>
    <td>Eligible U.S. executives</td>
    <td>Voluntary</td>
    <td>Based on average earnings, years of service and age at benefit calculation date</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>U.S. Defined Contribution (401A) Plan</td>
    <td>U.S. regular, full-time and part-time employees</td>
    <td>All contributions are made in cash and are matched by IBM with participant’s contribution</td>
    <td>Defers the dollar match, generally 50 percent of the employee’s contribution, with automatic matching of 1:1 for up to 6% of base salary, subject to IRS limits</td>
    <td>Employees generally receive contribution matching as service</td>
    <td></td>
  </tr>
  <tr>
    <td>Excess 401A(1) Plan</td>
    <td>U.S. employees, compensation is not subject to qualified plan limits</td>
    <td>Unfunded, non-deferred are recorded in trust fund, funded in trust for the benefit of the employee and invested in accounts (under the 401A(1) Plan only)</td>
    <td>Company match and individual contributions (at the same rate under the U.S. 401A Plan) are credited to the employee’s account on compensation earned during the year. The percentage varies based on years of service and years of service</td>
    <td>Employees generally receive contribution matching after one year of service</td>
    <td></td>
  </tr>
  <tr>
    <td>Nepension Retirement Benefits Plan</td>
    <td>Medical and dental benefits for eligible dependents, including U.S. retirees</td>
    <td>Company contributions to a reserve must be available for benefit payments to participants and beneficiaries</td>
    <td>Varies based on plan design, formula and eligibility requirements</td>
    <td>Benefit accrued through December 31, 2006, new hires may accrue these benefits</td>
    <td></td>
  </tr>
  <tr>
    <td>Non-U.S. Plans DB or DC</td>
    <td>Eligible regular, full-time non-U.S. subsidiaries of IBM</td>
    <td>Company deposits based on February type compensation, under group contracts for these plans</td>
    <td>Based on the years of service and compensation (generally defined as the highest annual compensation or annual equivalent in normal service)</td>
    <td>In certain countries, benefit accrual has ceased earlier than in the U.S. due to various factors</td>
    <td></td>
  </tr>
  <tr>
    <td>Nepension Retirement Benefits Plan</td>
    <td>Medical and dental benefits for eligible dependents, including U.S. retirees</td>
    <td>Primarily unfunded, based on country where the employee is located, subject to funding requirements by country</td>
    <td>Varies based on plan design, formula and eligibility requirements by country</td>
    <td>Most non-U.S. plans are administered by local government and/or national programs</td>
    <td></td>
  </tr>
</table>

* Matching and automatic contributions are made once at the end of the year for employees that are employed as of December 14 of the plan year. Contributions may be made for certain types of separations that occur prior to December 15.

What types of adapters are supported in the new release of IBM Storage Virtualize V8.6?

L1 Which adapter varieties are compatible with the latest IBM Storage Virtualize V8.6 version?
L2 Which adapter types are compatible with the latest version of IBM Storage Virtualize, specifically V8.6?
L3 In the latest version 8.6 of IBM Storage Virtualize, which adapter formats are compatible and can be utilized effectively?

IBM Storage FlashSystem 9500 Product Guide for IBM Storage Virtualize 8.6

IBM Storage FlashSystem 9500 Product Guide

This IBM® Redbooks® Product Guide describes the IBM Storage FlashSystem 9500 (IBM FlashSystem 9500) solution, which is a next-generation IBM Storage FlashSystem control enclosure designed to deliver high performance, high availability, and virtualized access (VMM)-optimized architecture with the reliability and innovation of IBM FlashCore™ technology and the rich feature set and high availability (HA) of IBM Storage Virtualize.

Often, applications exist that are foundational to the operations and success of an enterprise. These applications are often mission-critical, and they must perform important tasks, or provide crucial business intelligence, among many other jobs. Whatever their function, these applications must operate reliably, consistently, and at the highest level of performance, functionality, security, and availability. They must also be protected against cyberattacks.

To support such mission-critical applications, enterprises of all types and sizes turn to the IBM System x9500.

IBM FlashSystem 9500 provides a rich set of software-defined storage (SDS) features that are delivered by IBM Storage Virtualize, including the following examples:
• Data reduction and deduplication
• Dynamic tiering
• Thin-provisioning
• Snapshots
• Cloning
• Replication and data copy services
• Cyber resilience
• Transparent cloud tiering
• IBM HyperSnap® including 3-site replication for HA
• Peer-out and scale-up configurations that further enhance capacity and throughput for better business results

With the release of IBM Storage Virtualize V8.6, extra functions and features are available, including support for new generation IBM FlashCore Modules NVMe-type drives within the IBM FlashSystem 9500, 100 Gbps Ethernet connectivity, support for Remote Direct Memory Access (RDMA) options, new software features that include multi-factor authentication (MFA) and single sign-on (SSO).

© Copyright IBM Corp. 2023. All rights reserved.
ibm.com/redbooks

Figure S2: Real-MM-RAG Benchmark Examples with Rephrasing. Left: FinReport—financial annual reports. Right: TechReport—FlashSystem technical reports. Queries are listed from the original to Level 3 rephrasing.
ViDoRe / infovqa
How many infrastructure in this diagram are new?

MMLongBench
What does the map in the report shows?

SlidesVQA
Where is this market research based on?

Figure S3: Examples from Previous Benchmarks. These examples illustrate common query types in these benchmarks. Many queries are generated for question answering and refer to a specific page rather than resembling real user queries, which are typically asked without prior knowledge of a specific page.
Table S2: Impact of Rephrasing Levels on Document Retrieval Benchmarks. This table shows NDCG@5 performance variations across rephrasing levels (0-3) for different benchmarks and models.

<table>
  <tr>
    <th rowspan="2">Rephrasing Level</th>
    <th colspan="4">FinReport</th>
    <th colspan="4">FinSlides</th>
    <th colspan="4">TechReport</th>
    <th colspan="4">TechSlides</th>
  </tr>
  <tr>
    <th>0</th><th>1</th><th>2</th><th>3</th>
    <th>0</th><th>1</th><th>2</th><th>3</th>
    <th>0</th><th>1</th><th>2</th><th>3</th>
    <th>0</th><th>1</th><th>2</th><th>3</th>
  </tr>
  <tr>
    <td><i>BM25 (OCR)</i></td>
    <td>48.8</td><td>38.4</td><td>26.6</td><td>21.7</td>
    <td>13.6</td><td>13.0</td><td>7.1</td><td>5.9</td>
    <td>66.6</td><td>48.0</td><td>38.7</td><td>35.1</td>
    <td>58.7</td><td>45.7</td><td>35.7</td><td>31.2</td>
  </tr>
  <tr>
    <td><i>BGE-M3 (OCR)</i></td>
    <td>47.2</td><td>40.3</td><td>37.6</td><td>36.5</td>
    <td>16.9</td><td>16.3</td><td>14.6</td><td>11.4</td>
    <td>44.7</td><td>40.9</td><td>37.8</td><td>37.0</td>
    <td>60.5</td><td>56.3</td><td>51.9</td><td>49.6</td>
  </tr>
  <tr>
    <td><i>BM25 (Captioning)</i></td>
    <td>54.4</td><td>43.0</td><td>30.6</td><td>25.3</td>
    <td>20.9</td><td>19.0</td><td>11.0</td><td>9.9</td>
    <td>69.0</td><td>50.8</td><td>41.8</td><td>37.2</td>
    <td>66.4</td><td>53.4</td><td>41.7</td><td>36.1</td>
  </tr>
  <tr>
    <td><i>BGE-M3 (Captioning)</i></td>
    <td>46.4</td><td>39.0</td><td>36.9</td><td>35.9</td>
    <td>19.5</td><td>18.7</td><td>16.3</td><td>13.8</td>
    <td>44.6</td><td>40.9</td><td>38.4</td><td>37.5</td>
    <td>62.6</td><td>58.3</td><td>54.5</td><td>51.7</td>
  </tr>
  <tr>
    <td>ColPali</td>
    <td>52.7</td><td>47.2</td><td>40.8</td><td>36.8</td>
    <td>62.2</td><td>59.4</td><td>37.6</td><td>27.6</td>
    <td>80.6</td><td>72.9</td><td>66.5</td><td>62.0</td>
    <td>89.7</td><td>85.0</td><td>79.2</td><td>75.8</td>
  </tr>
  <tr>
    <td>RobColPali</td>
    <td>59.3</td><td>57.4</td><td>51.4</td><td>47.1</td>
    <td>76.8</td><td>75.1</td><td>58.3</td><td>48.4</td>
    <td>80.1</td><td>74.7</td><td>70.0</td><td>66.6</td>
    <td>90.5</td><td>88.2</td><td>84.2</td><td>82.8</td>
  </tr>
  <tr>
    <td>TabColPali</td>
    <td>70.5</td><td>63.5</td><td>56.4</td><td>50.5</td>
    <td>74.5</td><td>70.7</td><td>54.2</td><td>41.5</td>
    <td>82.7</td><td>73.8</td><td>66.8</td><td>61.3</td>
    <td>90.8</td><td>86.5</td><td>80.4</td><td>77.6</td>
  </tr>
  <tr>
    <td>RobTabColPali</td>
    <td>71.0</td><td>68.5</td><td>65.0</td><td>63.2</td>
    <td>80.9</td><td>79.5</td><td>68.0</td><td>58.3</td>
    <td>80.8</td><td>76.2</td><td>72.6</td><td>70.7</td>
    <td>90.5</td><td>87.1</td><td>84.3</td><td>83.3</td>
  </tr>
  <tr>
    <td>ColQwen</td>
    <td>60.8</td><td>54.5</td><td>46.7</td><td>41.8</td>
    <td>59.3</td><td>54.8</td><td>39.1</td><td>31.1</td>
    <td>84.2</td><td>74.9</td><td>71.8</td><td>66.9</td>
    <td>91.3</td><td>85.9</td><td>80.6</td><td>78.1</td>
  </tr>
  <tr>
    <td>RobColQwen</td>
    <td>58.4</td><td>54.5</td><td>49.7</td><td>47.5</td>
    <td>65.8</td><td>63.0</td><td>52.0</td><td>44.3</td>
    <td>81.9</td><td>75.4</td><td>71.8</td><td>69.5</td>
    <td>90.1</td><td>87.8</td><td>84.0</td><td>83.0</td>
  </tr>
  <tr>
    <td>TabColQwen</td>
    <td>78.2</td><td>69.0</td><td>61.5</td><td>54.0</td>
    <td>77.1</td><td>73.9</td><td>58.1</td><td>49.6</td>
    <td>83.6</td><td>75.1</td><td>70.8</td><td>65.9</td>
    <td>92.4</td><td>87.7</td><td>82.5</td><td>78.9</td>
  </tr>
  <tr>
    <td>RobTabColQwen</td>
    <td>79.7</td><td>74.8</td><td>69.4</td><td>67.1</td>
    <td>79.6</td><td>78.5</td><td>69.1</td><td>61.6</td>
    <td>83.7</td><td>79.3</td><td>75.5</td><td>73.2</td>
    <td>92.5</td><td>89.9</td><td>86.3</td><td>85.0</td>
  </tr>
</table>

Table S3: Impact of Rephrasing Levels on Document Retrieval Benchmarks (Recall@1). This table shows Recall@1 performance variations across rephrasing levels (0-3) for different benchmarks and models.

<table>
  <tr>
    <th rowspan="2">Rephrasing Level</th>
    <th colspan="4">FinReport</th>
    <th colspan="4">FinSlides</th>
    <th colspan="4">TechReport</th>
    <th colspan="4">TechSlides</th>
  </tr>
  <tr>
    <th>0</th><th>1</th><th>2</th><th>3</th>
    <th>0</th><th>1</th><th>2</th><th>3</th>
    <th>0</th><th>1</th><th>2</th><th>3</th>
    <th>0</th><th>1</th><th>2</th><th>3</th>
  </tr>
  <tr>
    <td><i>BM25 (OCR)</i></td>
    <td>33.9</td><td>25.4</td><td>17.1</td><td>14.0</td>
    <td>7.7</td><td>7.5</td><td>3.7</td><td>3.0</td>
    <td>53.4</td><td>34.1</td><td>26.1</td><td>23.1</td>
    <td>45.9</td><td>33.4</td><td>25.4</td><td>21.3</td>
  </tr>
  <tr>
    <td><i>BGE-M3 (OCR)</i></td>
    <td>34.0</td><td>28.6</td><td>25.8</td><td>25.0</td>
    <td>11.0</td><td>9.5</td><td>9.3</td><td>7.2</td>
    <td>34.4</td><td>31.2</td><td>28.6</td><td>27.8</td>
    <td>48.6</td><td>44.2</td><td>40.0</td><td>37.7</td>
  </tr>
  <tr>
    <td><i>BM25 (Captioning)</i></td>
    <td>40.1</td><td>28.0</td><td>19.8</td><td>17.5</td>
    <td>12.2</td><td>10.9</td><td>6.2</td><td>5.7</td>
    <td>56.2</td><td>36.8</td><td>28.8</td><td>24.6</td>
    <td>54.0</td><td>40.8</td><td>31.0</td><td>25.2</td>
  </tr>
  <tr>
    <td><i>BGE-M3 (Captioning)</i></td>
    <td>33.4</td><td>27.7</td><td>25.1</td><td>25.3</td>
    <td>12.5</td><td>11.5</td><td>9.6</td><td>8.9</td>
    <td>34.7</td><td>31.1</td><td>28.8</td><td>27.7</td>
    <td>50.9</td><td>45.8</td><td>41.8</td><td>40.1</td>
  </tr>
  <tr>
    <td>ColPali</td>
    <td>40.3</td><td>35.4</td><td>29.1</td><td>25.9</td>
    <td>45.6</td><td>41.6</td><td>23.3</td><td>15.5</td>
    <td>68.1</td><td>57.6</td><td>51.5</td><td>45.8</td>
    <td>82.4</td><td>75.3</td><td>68.4</td><td>63.5</td>
  </tr>
  <tr>
    <td>RobColPali</td>
    <td>44.4</td><td>42.1</td><td>36.5</td><td>32.0</td>
    <td>60.4</td><td>57.4</td><td>39.8</td><td>30.4</td>
    <td>68.0</td><td>60.2</td><td>53.8</td><td>51.2</td>
    <td>82.8</td><td>79.7</td><td>74.1</td><td>73.1</td>
  </tr>
  <tr>
    <td>TabColPali</td>
    <td>55.1</td><td>50.3</td><td>40.9</td><td>36.0</td>
    <td>55.9</td><td>52.0</td><td>35.0</td><td>22.8</td>
    <td>70.3</td><td>58.8</td><td>50.3</td><td>44.8</td>
    <td>83.9</td><td>77.8</td><td>69.4</td><td>66.5</td>
  </tr>
  <tr>
    <td>RobTabColPali</td>
    <td>56.5</td><td>53.9</td><td>49.4</td><td>48.4</td>
    <td>64.0</td><td>61.7</td><td>48.4</td><td>35.8</td>
    <td>67.9</td><td>61.2</td><td>57.4</td><td>54.3</td>
    <td>83.4</td><td>78.4</td><td>75.2</td><td>73.1</td>
  </tr>
  <tr>
    <td>ColQwen</td>
    <td>44.0</td><td>39.6</td><td>32.6</td><td>28.5</td>
    <td>44.7</td><td>40.8</td><td>26.7</td><td>18.7</td>
    <td>73.0</td><td>60.1</td><td>56.5</td><td>51.4</td>
    <td>84.2</td><td>77.4</td><td>70.3</td><td>66.5</td>
  </tr>
  <tr>
    <td>RobColQwen</td>
    <td>41.4</td><td>37.7</td><td>34.8</td><td>33.1</td>
    <td>49.1</td><td>46.6</td><td>37.2</td><td>29.3</td>
    <td>68.5</td><td>60.4</td><td>56.1</td><td>54.2</td>
    <td>83.3</td><td>79.5</td><td>74.6</td><td>72.1</td>
  </tr>
  <tr>
    <td>TabColQwen</td>
    <td>62.7</td><td>53.2</td><td>44.9</td><td>37.5</td>
    <td>58.7</td><td>55.9</td><td>41.2</td><td>33.1</td>
    <td>71.0</td><td>59.8</td><td>54.9</td><td>49.7</td>
    <td>85.9</td><td>78.6</td><td>71.1</td><td>66.9</td>
  </tr>
  <tr>
    <td>RobTabColQwen</td>
    <td>58.1</td><td>52.5</td><td>50.2</td><td>45.8</td>
    <td>62.2</td><td>60.7</td><td>51.4</td><td>41.8</td>
    <td>70.3</td><td>62.2</td><td>59.0</td><td>56.3</td>
    <td>85.0</td><td>80.9</td><td>76.0</td><td>74.9</td>
  </tr>
</table>
Table S4: Impact of Rephrasing Levels on Document Retrieval Benchmarks (Recall@5). This table shows Recall@5 performance variations across rephrasing levels (0-3) for different benchmarks and models.

<table>
  <tr>
    <th rowspan="2">Rephrasing Level</th>
    <th colspan="4">FinReport</th>
    <th colspan="4">FinSlides</th>
    <th colspan="4">TechReport</th>
    <th colspan="4">TechSlides</th>
  </tr>
  <tr>
    <th>0</th><th>1</th><th>2</th><th>3</th>
    <th>0</th><th>1</th><th>2</th><th>3</th>
    <th>0</th><th>1</th><th>2</th><th>3</th>
    <th>0</th><th>1</th><th>2</th><th>3</th>
  </tr>
  <tr>
    <td><i>BM25 (OCR)</i></td>
    <td>62.0</td><td>49.9</td><td>35.3</td><td>29.0</td>
    <td>19.7</td><td>18.6</td><td>10.4</td><td>8.7</td>
    <td>77.6</td><td>59.8</td><td>50.0</td><td>45.9</td>
    <td>70.3</td><td>57.1</td><td>45.2</td><td>40.6</td>
  </tr>
  <tr>
    <td><i>BGE-M3 (OCR)</i></td>
    <td>58.4</td><td>50.5</td><td>47.6</td><td>47.0</td>
    <td>22.9</td><td>22.8</td><td>19.8</td><td>15.6</td>
    <td>53.5</td><td>49.2</td><td>45.7</td><td>45.2</td>
    <td>70.6</td><td>66.5</td><td>62.3</td><td>60.2</td>
  </tr>
  <tr>
    <td><i>BM25 (Captioning)</i></td>
    <td>66.9</td><td>56.3</td><td>40.4</td><td>32.5</td>
    <td>29.4</td><td>27.2</td><td>15.5</td><td>13.8</td>
    <td>79.7</td><td>63.5</td><td>53.5</td><td>49.0</td>
    <td>77.2</td><td>64.8</td><td>51.3</td><td>46.1</td>
  </tr>
  <tr>
    <td><i>BGE-M3 (Captioning)</i></td>
    <td>57.0</td><td>48.8</td><td>46.7</td><td>45.5</td>
    <td>25.9</td><td>25.1</td><td>22.7</td><td>18.3</td>
    <td>52.9</td><td>49.3</td><td>46.6</td><td>46.2</td>
    <td>72.6</td><td>68.8</td><td>65.7</td><td>62.0</td>
  </tr>
  <tr>
    <td><i>ColPali</i></td>
    <td>64.0</td><td>57.8</td><td>51.7</td><td>47.0</td>
    <td>76.9</td><td>74.7</td><td>50.4</td><td>38.7</td>
    <td>90.7</td><td>85.7</td><td>79.4</td><td>76.1</td>
    <td>95.2</td><td>92.4</td><td>87.8</td><td>85.7</td>
  </tr>
  <tr>
    <td><i>RobColPali</i></td>
    <td>73.0</td><td>71.3</td><td>64.7</td><td>60.6</td>
    <td>90.6</td><td>89.8</td><td>74.1</td><td>64.4</td>
    <td>89.8</td><td>86.5</td><td>83.7</td><td>79.5</td>
    <td>96.1</td><td>94.4</td><td>91.9</td><td>90.1</td>
  </tr>
  <tr>
    <td><i>TabColPali</i></td>
    <td>83.8</td><td>75.5</td><td>70.1</td><td>63.3</td>
    <td>90.3</td><td>87.1</td><td>71.7</td><td>58.3</td>
    <td>92.7</td><td>86.0</td><td>80.7</td><td>75.1</td>
    <td>96.0</td><td>93.2</td><td>88.9</td><td>86.7</td>
  </tr>
  <tr>
    <td><i>RobTabColPali</i></td>
    <td>83.1</td><td>80.8</td><td>78.0</td><td>76.0</td>
    <td>95.0</td><td>94.2</td><td>85.0</td><td>77.7</td>
    <td>91.2</td><td>88.5</td><td>85.1</td><td>84.3</td>
    <td>95.6</td><td>93.7</td><td>91.2</td><td>91.3</td>
  </tr>
  <tr>
    <td><i>ColQwen</i></td>
    <td>75.3</td><td>68.0</td><td>59.6</td><td>54.0</td>
    <td>71.8</td><td>66.8</td><td>50.3</td><td>42.5</td>
    <td>93.1</td><td>86.8</td><td>84.5</td><td>79.7</td>
    <td>96.4</td><td>92.3</td><td>88.9</td><td>87.4</td>
  </tr>
  <tr>
    <td><i>RobColQwen</i></td>
    <td>73.0</td><td>69.5</td><td>63.0</td><td>60.7</td>
    <td>79.8</td><td>76.5</td><td>65.6</td><td>57.8</td>
    <td>92.6</td><td>87.9</td><td>84.8</td><td>82.7</td>
    <td>95.1</td><td>94.1</td><td>91.3</td><td>91.4</td>
  </tr>
  <tr>
    <td><i>TabColQwen</i></td>
    <td>90.7</td><td>82.6</td><td>75.5</td><td>68.0</td>
    <td>92.4</td><td>88.7</td><td>72.8</td><td>64.3</td>
    <td>93.3</td><td>87.2</td><td>83.8</td><td>79.6</td>
    <td>97.2</td><td>94.7</td><td>91.3</td><td>88.6</td>
  </tr>
  <tr>
    <td><i>RobTabColQwen</i></td>
    <td>86.3</td><td>80.1</td><td>76.8</td><td>74.4</td>
    <td>92.5</td><td>92.1</td><td>82.0</td><td>76.2</td>
    <td>92.6</td><td>88.8</td><td>86.4</td><td>84.7</td>
    <td>96.9</td><td>95.3</td><td>93.3</td><td>92.6</td>
  </tr>
</table>

Table S5: Model Performance Across Different Evidence Source Types A Vision-Language Model (VLM) was used to classify each query based on the type of evidence source from which the answer was retrieved on the corresponding page. We present the division of performance across three different source types: Text, Table, and Visual. The reported results are the NDCG@5 scores on the non-rephrased version, averaged across our four benchmarks.

<table>
  <tr>
    <th>Benchmark</th>
    <th>Text</th>
    <th>Tables</th>
    <th>Visual</th>
  </tr>
  <tr>
    <td>ColPali</td>
    <td>75.8</td>
    <td>58.6</td>
    <td>84.5</td>
  </tr>
  <tr>
    <td>ColQwen</td>
    <td>79.2</td>
    <td>59.9</td>
    <td>86.8</td>
  </tr>
  <tr>
    <td>TabColQwen</td>
    <td>84.5</td>
    <td>78.5</td>
    <td>88.5</td>
  </tr>
</table>

Table S6: Ablation of Fine-Tuning Without Rephrasing. To demonstrate that the performance improvement is not solely due to fine-tuning, we fine-tune the models on the original ColPali dataset without rephrasing, using the exact same fine-tuning configuration.

<table>
  <tr>
    <th rowspan="2">Rephrasing Level</th>
    <th colspan="2">FinReport</th>
    <th colspan="2">FinSlides</th>
    <th colspan="2">TechReport</th>
    <th colspan="2">TechSlides</th>
  </tr>
  <tr>
    <th>0</th><th>3</th>
    <th>0</th><th>3</th>
    <th>0</th><th>3</th>
    <th>0</th><th>3</th>
  </tr>
  <tr>
    <td>ColPali</td>
    <td>52.7</td><td>34.5</td>
    <td>62.2</td><td>27.6</td>
    <td>80.6</td><td>62.0</td>
    <td>89.7</td><td>75.8</td>
  </tr>
  <tr>
    <td><i>ColPali Baseline FT</i></td>
    <td>57.1</td><td>39.1</td>
    <td>69.1</td><td>35.7</td>
    <td>80.0</td><td>61.4</td>
    <td>90.0</td><td>77.8</td>
  </tr>
  <tr>
    <td><i>RobColPali</i></td>
    <td>59.3</td><td>47.1</td>
    <td>76.8</td><td>48.4</td>
    <td>80.1</td><td>66.6</td>
    <td>90.5</td><td>83.3</td>
  </tr>
  <tr>
    <td>ColQwen</td>
    <td>60.8</td><td>41.8</td>
    <td>59.3</td><td>31.1</td>
    <td>84.2</td><td>66.9</td>
    <td>91.3</td><td>78.1</td>
  </tr>
  <tr>
    <td><i>ColQwen Baseline FT</i></td>
    <td>58.1</td><td>39.3</td>
    <td>53.4</td><td>27.6</td>
    <td>79.7</td><td>62.1</td>
    <td>90.5</td><td>77.4</td>
  </tr>
  <tr>
    <td><i>RobColQwen</i></td>
    <td>58.4</td><td>47.5</td>
    <td>65.8</td><td>44.3</td>
    <td>81.9</td><td>69.5</td>
    <td>90.1</td><td>83.0</td>
  </tr>
</table>
Table S7: Comparison of different queries generation models. This table compares the NDCG5 performance of the ColQwen model fine-tuned with the original data of ColPali and our generated table-focused data from the FinTabNet dataset. The evaluation is conducted for two query generation approaches: one using Pixtral and the other using Qwen. The rephrasing for the benchmarks was performed using LLaMA-3-3-70B. Results are presented across rephrasing levels (0 and 3) for our retrieval benchmarks.

<table>
  <tr>
    <th rowspan="2">Rephrasing Level</th>
    <th colspan="3">FinReport</th>
    <th colspan="3">FinSlides</th>
    <th colspan="3">TechReport</th>
    <th colspan="3">TechSlides</th>
  </tr>
  <tr>
    <th>0</th><th>3</th><th></th>
    <th>0</th><th>3</th><th></th>
    <th>0</th><th>3</th><th></th>
    <th>0</th><th>3</th><th></th>
  </tr>
  <tr>
    <td>ColPali</td>
    <td>52.7</td><td>34.5</td><td></td>
    <td>62.2</td><td>27.6</td><td></td>
    <td>80.6</td><td>62.0</td><td></td>
    <td>89.7</td><td>75.8</td><td></td>
  </tr>
  <tr>
    <td>ColQwen</td>
    <td>60.8</td><td>41.8</td><td></td>
    <td>59.3</td><td>31.1</td><td></td>
    <td>84.2</td><td>66.9</td><td></td>
    <td>91.3</td><td>78.1</td><td></td>
  </tr>
  <tr>
    <td><i>ColTab (Pixtral Queries Gen.)</i></td>
    <td>78.2</td><td>54.0</td><td></td>
    <td>77.1</td><td>49.6</td><td></td>
    <td>83.6</td><td>65.9</td><td></td>
    <td>92.4</td><td>78.9</td><td></td>
  </tr>
  <tr>
    <td><i>ColTab (Qwen Queries Gen.)</i></td>
    <td>74.8</td><td>49.5</td><td></td>
    <td>74.3</td><td>41.5</td><td></td>
    <td>83.8</td><td>66.8</td><td></td>
    <td>92.6</td><td>79.5</td><td></td>
  </tr>
  <tr>
    <td><i>ColRobTab (Pixtral Queries Gen.)</i></td>
    <td>79.7</td><td>67.1</td><td></td>
    <td>79.6</td><td>61.6</td><td></td>
    <td>83.7</td><td>73.2</td><td></td>
    <td>92.5</td><td>85.0</td><td></td>
  </tr>
  <tr>
    <td><i>ColRobTab (Qwen Queries Gen.)</i></td>
    <td>73.5</td><td>61.1</td><td></td>
    <td>78.9</td><td>60.0</td><td></td>
    <td>82.8</td><td>71.8</td><td></td>
    <td>91.8</td><td>84.7</td><td></td>
  </tr>
</table>

You are given a single page from a document. This page may contain text, figures, tables, and diagrams. The document is about {Document_short_description}. Make sure to mention the company or product name when the question pertains to specific information about it.

Your task is to produce 10 question-answer pairs. Each question should represent a plausible inquiry that a person (who has not seen the page) might ask about the information uniquely presented on this page. The questions should not reference this specific page directly (by page number, pointing to a specific paragraph or figure, and never refer to the document using phrases like 'in the document'), nor should they quote the text verbatim. They should use natural language reflecting how someone might inquire about the page's content without direct access.

Please focus mainly on tables and figures rather than just text. Try to formulate questions that require data from multiple locations on the page. The answer should be uniquely supported by the content of this page (i.e., the user could not find the answer elsewhere in the report or infer it without seeing this particular page). Ensure that the questions are not too general. For example, refer to the company name instead of using 'the company' generically, as different documents may refer to different companies.

Good questions:
What types of properties does Wihlborgs specialize in?
How many women are in Wihlborgs’ group management?
What was the occupancy rate of Wihlborg properties in 2021?
Was the net letting positive in the first quarter of 2022?
Which key factor contributed the most to the change in property value in 2022?
What’s the difference in the number of properties in South Copenhagen between 2021 and 2022?
What is the proportion of women on the board of directors?
How many dividends were paid in 2020 in total equity?
What portion of Wihlborg’s sponsorship in 2018 was community-oriented?

Bad questions:
What is the title of the page?
What is the main purpose of this page?
What can you learn from the figure?
What is the name of the CEO mentioned in the document?
What is IBM’s clear path to growth according to the document?
What is the basic earnings per share for 2023?

Make sure to include references to the company and product names when needed.

Output format:
Return as a list of objects:
[
  { "query": "...", "answer": "..." },
  { "query": "...", "answer": "..." },
  { "query": "...", "answer": "..." },
  { "query": "...", "answer": "..." },
  { "query": "...", "answer": "..." }
]

Figure S4: Query Generation Prompt
You have a large collection of document pages. Your goal is to determine, for each incoming question, whether that question would be suitable for retrieving a specific page from this collection.
Classify each question into one of two categories:

Category #A:
A plausible inquiry about the unique information on a page, posed as though the reader does not have direct access to that page.
The question does NOT explicitly reference the page number, figure label, or phrases like “in the document” or “on page X.”
Instead, it uses natural language to reflect how someone would genuinely ask about the content.

Category #B:
Any question that explicitly references the page, figure, table, or paragraph (e.g., “on page 3,” “in figure 2,” “in the document”).
Or is too vague (e.g., “what is the name of the company?” without context).
Or otherwise, does not provide enough context to be meaningfully answered from a single page.

Below are examples for both categories:

"Which climate factors had the greatest impact on ABC Farming’s wheat yield in 2022?" # A
"How are topic scores calculated for each topical subset in the processing step S2657?" # B
"How many corporate partners joined Apple Inc.’s technology incubator program last year?" # A
"Can you show me the figure on page 12 that references the new regulation?" # B
"Which strategic moves led to a 15% revenue increase for Toyota in Q2 2022?" # A
"List the bullet points mentioned in the second paragraph." # B
"What percentage of employees in Google’s marketing department are working remotely?" # A
"What is the heading of Table 2 in the document?" # B
"How has the demand for Tesla’s electric vehicles changed over the past five years, based on the company’s data?" # A
"Who is labeled as the second co-author on page 7?" # B
"By how many degrees did the average global temperature rise according to NASA’s 2020-2022 research?" # A
"Which lines in the document discuss the formula for carbon emissions?" # B
"Why did the organization decide to shift to renewable energy solutions?" # B (No specific organization)
"Which historical events contributed most to the British government’s policy reforms in 1985?" # A
"How many employees does the company have?" # B (Too vague; no company mentioned)
"Did Walmart notice any major changes in consumer spending habits in Q4 2021?" # A
"What is the official job title of the lead manager in charge?" # B (Unclear which manager or context)
"Which country introduced new tax incentives for renewable energy projects in 2022, according to the IEA’s annual report?" # A
"Could you summarize the entire document for me?" # B (Overly broad; not tied to specific info)

When I give you a question, you should respond with ONLY 'A' or 'B'.

Figure S5: Query Verification Prompt
Rephrasing Level 1

""" Your task is to rephrase questions by changing only a few words without altering the sentence structure while maintaining the same intent and focus.
Question: "{query}"
Write only the rephrased question."""

Rephrasing Level 2

""" You are an expert at rephrasing questions while retaining their meaning.
Your task is to rewrite the following question using different words, maintaining the same intent and focus:
Question: "{query}"
Write only the Rephrased Question. """

Rephrasing Level 3

""" You are an expert at rephrasing questions while retaining their meaning.
Your task is to significantly rewrite the following question using different words and sentence order, maintaining the same intent and focus while changing a lot:
Question: "{query}"
Write only the Rephrased Question. """

Rephrasing Verification

You are given an original query, its rephrased version, and the answer. Verify if the rephrased query maintains the same intent and focus as the original query.
If the rephrased query is correct, answer "Yes." If not, answer "No."
Original Query: "{original_query}"
Rephrased Query: "{rephrased_query}"
Answer: "{answer}"
Response: ""

Figure S6: Rephrasing Generation and Verification Prompts

<table>
  <tr>
    <th colspan="2">Classify each question as bad or good based on those definitions:</th>
  </tr>
  <tr>
    <th>Category Good</th>
    <td>
      - Written as a genuine information-seeking question.<br>
      - Natural language inquiry about specific information.<br>
      - Includes proper context (company names, years, etc.)<br>
      - Avoids references to document structure
    </td>
  </tr>
  <tr>
    <th>Category Bad</th>
    <td>
      - Explicitly references page numbers, figures, or document parts<br>
      - Too vague or missing crucial context<br>
      - Asks about generic document properties
    </td>
  </tr>
  <tr>
    <td colspan="2">
      Example of Good: "How many corporate partners joined Apple Inc.'s technology incubator program in 2023?"<br>
      Example of Bad: "What is the heading of Table 2 in the document?"
    </td>
  </tr>
</table>

Figure S7: Human Evaluation of Query Alignment to RAG. This figure shows the instructions presented to human annotators along with randomly sampled queries from different benchmarks.
You are given a document page and a query. Determine if the query can be perfectly answered by the information in the document.
Query: What was the year-over-year change in the Gross Margin for IBM's Global Business Services Segment in the third quarter of 2018?

IBM financial presentation for the third quarter of the year 2019

Global Business Services Segment

• Revenue growth driven by application modernization for the cloud and next-generation enterprise apps
• Early progress with Red Hat with over 20 deals signed in 3Q19
• Gross margin expansion reflects shift to higher value, leveraging global delivery footprint and productivity actions

<table>
  <tr>
    <th>Segment Results Overview</th>
    <th colspan="2">B/(W)</th>
  </tr>
  <tr>
    <th>Segment</th>
    <th>3Q19</th>
    <th>Yr/Yr</th>
  </tr>
  <tr>
    <td>Revenue (External)</td>
    <td>$4.1</td>
    <td>2%</td>
  </tr>
  <tr>
    <td>Gross Profit Margin (External)</td>
    <td>31.1%</td>
    <td>1.1 pts</td>
  </tr>
  <tr>
    <td>Pre-Tax Income</td>
    <td>$0.6</td>
    <td>1%</td>
  </tr>
  <tr>
    <td>Pre-Tax Income Margin</td>
    <td>13.7%</td>
    <td>0.1 pts</td>
  </tr>
  <tr>
    <td>Cloud Revenue</td>
    <td>$1.3</td>
    <td>10%</td>
  </tr>
</table>

Segment Revenue Elements

Consulting +5% Y/Y
Application Management Flat W/Y/Y
Global Process Services (3%) W/Y/Y

Figure S8: Human Evaluation of False Negatives. This figure presents an example of the image shown to human annotators, including the instructions, the query, and the negative page retrieved for the given query using ColQwen. The query and page are from our FinSlides benchmark and illustrate a case where the model incorrectly retrieved the wrong year.
Table S8: ViDoRe V2 Results (English Benchmarks). This table presents NDCG@5 scores for each domain and the overall average on the newly released ViDoRe V2 benchmark.

<table>
  <tr>
    <th>Model</th>
    <th>Biomedical</th>
    <th>Macroeconomics</th>
    <th>Restaurant (RSE)</th>
    <th>Restaurant (ESG)</th>
    <th>Insurance (AXA)</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>ColPali</td>
    <td>59.1</td>
    <td>51.4</td>
    <td>50.0</td>
    <td>57.9</td>
    <td>53.2</td>
    <td>54.3</td>
  </tr>
  <tr>
    <td>RobColPali</td>
    <td>59.5</td>
    <td>57.5</td>
    <td>58.4</td>
    <td>66.5</td>
    <td>65.0</td>
    <td>61.4</td>
  </tr>
  <tr>
    <td>RobTabColPali</td>
    <td>61.0</td>
    <td>60.3</td>
    <td>55.3</td>
    <td>64.6</td>
    <td>60.0</td>
    <td>60.2</td>
  </tr>
  <tr>
    <td>ColQwen</td>
    <td><b>61.1</b></td>
    <td><b>61.5</b></td>
    <td>55.8</td>
    <td><b>63.0</b></td>
    <td>65.3</td>
    <td>61.3</td>
  </tr>
  <tr>
    <td>RobColQwen</td>
    <td>61.0</td>
    <td><b>57.0</b></td>
    <td><b>60.6</b></td>
    <td>62.0</td>
    <td><b>67.2</b></td>
    <td><b>61.6</b></td>
  </tr>
  <tr>
    <td>RobTabColQwen</td>
    <td><b>62.1</b></td>
    <td>56.5</td>
    <td><b>61.7</b></td>
    <td><b>66.0</b></td>
    <td><b>69.7</b></td>
    <td><b>63.2</b></td>
  </tr>
</table>

Table S9: ViDoRe V2 Results (Multilingual Benchmarks). This table presents NDCG@5 scores for each domain and the overall average on the multilingual subset of the newly released ViDoRe V2 benchmark.

<table>
  <tr>
    <th>Model</th>
    <th>Biomedical</th>
    <th>Macroeconomics</th>
    <th>Restaurant (RSE)</th>
    <th>Insurance (AXA)</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>ColPali</td>
    <td>55.9</td>
    <td>47.2</td>
    <td>52.5</td>
    <td>47.4</td>
    <td>50.8</td>
  </tr>
  <tr>
    <td>RobColPali</td>
    <td>58.4</td>
    <td>53.2</td>
    <td>57.9</td>
    <td>57.5</td>
    <td>56.8</td>
  </tr>
  <tr>
    <td>RobTabColPali</td>
    <td>57.7</td>
    <td><b>55.0</b></td>
    <td>54.8</td>
    <td>54.4</td>
    <td>55.5</td>
  </tr>
  <tr>
    <td>ColQwen</td>
    <td>56.3</td>
    <td>52.8</td>
    <td>56.9</td>
    <td>56.5</td>
    <td>55.6</td>
  </tr>
  <tr>
    <td>RobColQwen</td>
    <td><b>58.3</b></td>
    <td>53.0</td>
    <td><b>60.6</b></td>
    <td><b>59.1</b></td>
    <td><b>57.8</b></td>
  </tr>
  <tr>
    <td>RobTabColQwen</td>
    <td><b>59.1</b></td>
    <td><b>55.1</b></td>
    <td><b>60.2</b></td>
    <td><b>61.8</b></td>
    <td><b>59.1</b></td>
  </tr>
</table>

Reviewer paper summaries: The paper presents a realistic benchmark for the retrieval component of multi-modal Retrieval-Augmented Generation (RAG). This benchmark is based on four key properties of real-world scenarios: Multi-modal Mocuments, Enhanced Difficulty, Realistic-RAG Queries, and Accurate Labeling. Experiments show that the constructed pipeline is not only effective but also outperforms existing benchmarks.

This paper introduce REAL-MM-RAG, an automatically generated benchmark designed to address four key properties essential for realworld retrieval: (i) multi-modal documents, (ii) enhanced difficulty, (iii) Realistic-RAG queries and (iv) accurate labeling. Additionally, it propose a multi-difficulty-level scheme based on query rephrasing to evaluate models’ semantic understanding beyond keyword matching.
Summary of strengths: 1.	The proposed benchmark effectively addresses the limitations of existing benchmarks by being more aligned with real-world conditions.
2.	The writing is easy to follow, and the experiments are thorough.
3.	It clearly defines four fundamental and important attributes for real-world retrieval tasks, providing a solid foundation for future research.

1. this paper integrate multi-modal documents (text, tables, figures) with realistic, non-referential queries, addressing a critical gap in existing RAG evaluations. The inclusion of domain-specific, contextually dense documents ensures enhanced difficulty, forcing models to distinguish fine-grained semantic differences rather than relying on keyword matching. 
2. The multi-level query rephrasing mechanism systematically tests semantic understanding and robustness, exposing model weaknesses (e.g., sensitivity to phrasing changes, poor table comprehension). This provides actionable insights for improving retrieval systems.
3. The proposed rephrased training set and finance-table-heavy dataset demonstrate significant performance gains, offering concrete strategies to address identified weaknesses.
Summary of weaknesses: 1.	Figures and Tables: The layout of Table 1 could be optimized for better aesthetics. In Figure 2, the text in the small left-hand side graph can be replaced with appropriate diagrams or captions if detailed reading isn't necessary. The meanings of {N Q/A Pairs} and {M Q/A Pairs} are unclear; adding explanations or descriptions in the caption would help clarify, especially since {M Q/A Pairs} appears twice with different operations in between. Highlighting these differences would improve clarity.
2.	Formatting Issues: The paper does not strictly follow the ACL template format, particularly regarding paragraph indentation and the style of the "Limitations" section title (the section number is not required).
3.	Sample Size: The benchmark includes 5k queries and Vidore has 4k queries. Only 50 queries were sampled for testing, which is statistically small and introduces randomness. Although the sampled results show significant differences, a larger sample size would provide more reliable result.

1. The benchmark’s heavy reliance on IBM-centric documents (financial/technical reports) introduces domain bias. Performance on heterogeneous domains (e.g., medical, legal) or cross-enterprise retrieval remains unvalidated, raising concerns about broader applicability.
2. Generated queries lack explicit stratification by semantic depth (factual vs. analytical) or linguistic diversity (formal vs. colloquial). This limits insights into how models handle real-world variations like incomplete sentences or implicit context.
3. While the automated annotation framework is highlighted, the paper inadequately addresses critical aspects such as setting confidence thresholds for VLMs/LLMs, resolving annotation conflicts (e.g., model discrepancies in judging query-page relevance), and managing edge cases (e.g., partially relevant content). Additionally, the absence of error case analysis undermines the assessment of the annotation system’s robustness.
Comments, suggestions and typos: 1.	Benchmark Validation: Accurate labeling validation checks for false negatives, but why not validate false positives as well? Both false negatives and false positives are crucial for improving the reliability of the benchmark data.

None
Overall assessment scores: 3

3.5
Metareview score: 3.5
Metareview: The paper introduces REAL-MM-RAG-Bench, a multimodal RAG benchmark designed to tackle the shortcomings of existing datasets, which are often too simplistic and poorly suited for real-world use. The authors create two specialized datasets, one with rephrased queries requiring semantic understanding, and another focused on finance-heavy tables. Using a VLM-LLM pipeline, they generate and refine queries to increase complexity and realism. The main strengths are its creation of multimodal datasets and its focus on semantic complexity. Reviewers report that the benchmark is well-designed, the writing is clear, and the experiments are thorough. However, the study lacks exploration of biases from fine-tuning on their own data. And its focus on IBM-centric documents shows issues about generalizability. The work could also benefit from clearer visual presentation, stricter formatting, and a larger evaluation sample with revision. Additional tests on external datasets and broader error analysis would be beneficial.
Summary of reasons to publish: This paper makes a meaningful contribution to the field of multimodal document retrieval. The authors provide well-constructed datasets that better reflect real-world complexity, especially with table-heavy financial documents. Their query generation pipeline and human evaluation demonstrate clear improvements in dataset quality and model performance. Despite some limitations in domain diversity and evaluation scope, the work is timely, valuable for multimodal RAG systems. I recommend borderline conference.