Venue: NAACL 2025

# VisDoM: Multi-Document QA with Visually Rich Elements

Using Multimodal Retrieval-Augmented Generation

Manan Suri , Puneet Mathur\({}^{\circledR}\)1 , Franck Dernoncourt\({}^{\circledR}\)

Kanika Gowswami , Ryan A. Rossi , Dinesh Manocha

University of Maryland, College Park \({}^{\circledR}\)Adobe Research \({}^{\circledR}\)IGDTUW manans@umd.edu, puneetm@adobe.com

Footnote 1: Primary Research Mentor

###### Abstract

Understanding information from a collection of multiple documents, particularly those with visually rich elements, is important for document-grounded question answering. This paper introduces VisDoMBench, the first comprehensive benchmark designed to evaluate QA systems in multi-document settings with rich multimodal content, including tables, charts, and presentation slides. We propose VisDoMRAG, a novel multimodal Retrieval Augmented Generation (RAG) approach that simultaneously utilizes visual and textual RAG, thereby combining robust visual retrieval capabilities with sophisticated linguistic reasoning. VisDoMPMAG employs a multi-step reasoning process encompassing evidence curation and chain-of-thought reasoning for concurrent textual and visual RAG pipelines. A key novelty of VisDoMRAG is its consistency-constrained modality fusion mechanism, which aligns the reasoning processes across modalities at inference time to produce a coherent final answer. This leads to enhanced accuracy in scenarios where critical information is distributed across modalities and improved answer verifiability through implicit context attribution. Through extensive experiments involving open-source and proprietary large language models, we benchmark state-of-the-art document QA methods on VisDoMBench. Extensive results show that VisDoMRAG outperforms unimodal and long-context LLM baselines for end-to-end multimodal document QA by 12-20%.

## 1 Introduction

In today's information-rich landscape, PDF documents play a crucial role in storing and disseminating information across various domains, including finance, legal, scientific research, and more. These documents often contain a rich blend of textual, visual, and tabular data, making them a unique challenge for information retrieval systems. Unlike structured formats like databases, PDFs are inherently unstructured, with diverse layouts combining paragraphs, images, charts, and tables. This complexity demands sophisticated multimodal processing techniques capable of interpreting both the textual and visual content. Effective handling of multimodal content from PDFs is essential for downstream tasks such as question-answering Ding et al. (2022); Mathew et al. (2021), summarization Pang et al. (2023), and knowledge extraction Pal et al. (2023), where accurate and context-aware data extraction can significantly enhance decision-making processes. As a result, developing advanced methods that can fully leverage the multimodal nature of PDF documents has become a critical research challenge.

In real-world document QA systems, queries are often directed over a collection of source documents rather than a single source, requiring the system to identify the document that contains the relevant answer. This reflects common scenarios in domains such as finance, science, and policy analysis, where users interact with large, varied document sets to find specific information. In these cases, the challenge lies in effectively localizing context relevant to the query, from a large volume

Figure 1: Multi-document QA systems require inferring relevant context from a large volume of unstructured data, inherently making it a more challenging task than single-document QA.



of information distributed across multiple documents (akin to finding a "needle in a haystack" Wang et al. (2024)).

Multi-document QA datasets are scarce, with existing multi-document benchmarks Bai et al. (2023); Wang et al. (2024), predominantly focused on textual information, often overlooking the diverse content forms found in real-world documents, such as tables, charts, and visual elements. Visually rich elements, such as tables, charts, and slides, provide structured data and visual summaries that are critical for answering certain types of questions. Tables often present dense, organized information that cannot be captured through plain text. At the same time, charts and slides can visually depict trends, relationships, or distributions that require interpretation beyond textual descriptions. The absence of datasets that include these modalities limits the ability of current QA models to address complex, multimodal questions. For instance, answering a financial or scientific question may require interpreting both numerical data in tables and trends in graphs alongside the surrounding text.

In the context of visually rich content-based documents, existing RAG systems face a critical limitation due to their reliance on a singular modality (either text or vision) for retrieval. Text-based systems are proficient in linguistic reasoning but often overlook vital visual elements, such as tables and figures, that may contain key information. Conversely, multimodal RAG Chen et al. (2022) systems that leverage vision-based retrieval can effectively extract visual data but are often constrained in end-to-end performance by the LLM's visual reasoning abilities, as text often performs better than visual input when given the same context Deng et al. (2024), which can be attributed to language bias in visual LLMs Niu et al. (2021); Wang et al. (2024), and visual hallucination Ghosh et al. (2024).

**Main Results:** We introduce VisDoMRBench, the first multi-document, multi-modal QA dataset specifically designed to address rich visual content, including tables, charts, and slides. VisDoMBench encompasses a diverse range of complex content and question types, along with annotated evidence, allowing for a comprehensive evaluation of multimodal QA systems. In this work, we benchmark the performance of various visual and textual retrieval methods on VisDoMBench, providing insights into their effectiveness in handling visually rich, multi-document queries.

Further, we propose VisDoMRAG, a novel multimodal RAG approach that effectively performs modality fusion over textual and visual RAG pipelines, benefiting from the inherent strengths of both these approaches, unlike contemporary approaches, which perform only-text or only-vision-based retrieval. VisDoMRAG employs parallel RAG pipelines for text and visual elements, each with a multi-step reasoning process involving evidence curation, chain-of-thought reasoning, and answer generation. The system then integrates the outputs from both pipelines using modality fusion, which imposes a consistency constraint on the reasoning chains, ensuring inference-time alignment across the modalities' reasoning processes to produce the final answer. VisDoMRAG offers several significant advantages over traditional unimodal or simpler multimodal systems. Firstly, it ensures comprehensive information utilization by fully leveraging both textual and visual cues, leading to more accurate and complete answers, particularly in scenarios where critical information is distributed across different modalities. Moreover, the evidence curation step provides an additional advantage of answer verifiability, since context attribution is built into our approach. We conduct experiments utilizing various open-source and closed-source LLMs, comparing multiple strategies such as long-context processing, textual RAG, and visual RAG, with our proposed system. We find that our VisDoMRAG improves end-to-end QA performance on our benchmarks, with performance gains in the range of 12%-20%. Our **main contributions** are:

* **VisDoMBench**1, **a novel multi-document, multimodal QA benchmark** designed to address QA tasks across visually rich document content such as tables, charts, and slides, allowing for a comprehensive evaluation of multimodal document QA systems. Footnote 1: [https://github.com/MananSuri27/VisDoM/](https://github.com/MananSuri27/VisDoM/)
* **VisDoMRAG, a novel multimodal RAG** approach that effectively paralellly performs textual and visual RAG via Evidence Curation and Chain-of-Thought reasoning. The output reasoning chains from both the modalities are aligned using consistency analysis and resultant answers are ensembled together via LLM-based modality fusion to enhance visually-rich document QA. Footnote 1: [https://github.com/MananSuri27/VisDoM/](https://github.com/MananSuri27/VisDoM/) 

ing, textual RAG, and visual RAG on the VisDoMBench corpus by 12-20% across various open and closed-source LLM settings.

## 2 Related Work

**Retrieval Augmented Generation** While Large Language Models (LLMs) have achieved significant advancements, they still encounter challenges in integrating external knowledge and adapting to new, unseen data. Retrieval Augmented Generation (RAG) addresses these gaps by incorporating external information, enhancing the precision and reliability of LLM responses Lewis et al. (2020). RAG is utilized across various downstream unimodal NLP tasks, including machine translation Gu et al. (2018); He et al. (2021), dialogue generation Cai et al. (2018), abstractive summarization Peng et al. (2019), and knowledge-intensive generation Izacard and Grave (2020); Lewis et al. (2020). In visual question answering (VQA), Lin and Byrne (2022) addresses open-domain challenges by using object detection, image captioning, and optical character recognition (OCR) to transform target images into textual data. Moving beyond text-only contexts, MuRAG retrieves both text and image data, incorporating images as visual tokens Chen et al. (2022). RAMM enhances performance by retrieving and encoding similar biomedical images and their captions through distinct networks Yuan et al. (2023).

**Long Context Document Benchmarks** The comparison of long context document question-answer benchmarks (Table 1), highlights the diversity in content types, multi-document capabilities, and domains. Existing benchmarks such as L-Eval An et al. (2023), Marathon Zhang et al. (2023), and LooGLE Li et al. (2023) primarily focus on text-based content from multi-domain sources but do not support multi-document inputs. LongBench Bai et al. (2023) and Loong Wang et al. (2024c) extend their evaluations to include multi-document settings, although they remain text-centric.

**Comparison with existing datasets**: Certain benchmarks like MPDocVQA Titto et al. (2023), UDA Hui et al. (2024), and MMLONGBENCHDOC Ma et al. (2024) expand the content spectrum by incorporating tables, charts, and slides, but they are limited to single-document question answering. In contrast, VisDoMBench supports multi-document question answering across various content types, including text, tables, charts, and slides, offering a more comprehensive multi-domain evaluation framework.

## 3 Problem Formulation

Given a query \(q\), we have a collection of \(M\) documents \({\cal D}=\{d_{1},d_{2},\ldots,d_{M}\}\), wherein each document \(d_{i}\) may consist of a set of \(N_{i}\) pages represented by \(P^{i}=\{p_{1}^{i},p_{2}^{i},\ldots,p_{N_{i}}^{i}\}\). We aim to generate text \(\hat{a}\) for each query \(q\) that accurately answers the user query. The answer generation relies on retrieving relevant evidence context from one or more documents. Each query \(q\) may require information spread across different pages from one or more of the associated documents in \(D\).

We aim to propose a framework that can accurately answer questions over a collection of multi-page documents where the system first retrieves relevant evidence at the level of individual pages, paragraphs or text chunks, followed by using the retrieved context to generate answer text.

## 4VisDoMBench

Every data point in VisDoMBench can be expressed as triple \((q,D,\hat{a})\), where a question \(q\) is posed to a set of documents \(D\), with ground-truth answer \(\hat{a}\). We re-purpose five existing document-QA datasets to form our benchmark. Table 2 summarises different data splits present in VisDoMBench, including summary statistics, QA type, and content type.

### VisDoMBench

**Data Sourcing**: In the curation of document question-answering datasets, we adhered to the following criteria: (1) the inclusion of visually rich content, encompassing tables, charts, and presentation slides; (2) the utilization of publicly accessible source documents; and (3) the presence of grounded evidence. These parameters were established to ensure the datasets' relevance to multimodal information retrieval and their applicability to real-world question-answering tasks. Our corpus comprises test/eval sets sourced from several established datasets. We incorporated the PaperTab and FeTaTab splits from the UDA BenchmarkHui et al. (2024), which in turn sourced these datasets from QASPERDasigi et al. (2021) and FeTaQANan et al. (2022), respectively. For chart-based question-answering samples, we drew from SciGraphQA Li and Tajbakhsh (2023), which is multi-turn QA dataset on charts from scientific pa 

pers, and SPIQA(Pramanick et al., 2024), a chart and table QA dataset system sourced from Dasigi et al. (2021). Additionally, we included SlideVQA Tanaka et al. (2023), a multi-image, multi-hop QA dataset centered on presentation slide decks.

**Data Sampling**: Sourced QA pairs need to be sampled to retain high quality samples. To maintain the integrity and uniqueness of our benchmark, we meticulously removed overlapping samples between PaperTab and SPIQA and implemented rigorous de-duplication of QA pairs across all included datasets. Further, we also perform question-level de-duplication to ensure similar questions are not repeated across different document collections. This ensures that QA systems are not rewarded disproportionately for better handling particular question types. For SciGraphQA, we filter out trivial questions related to layout and document metadata. From the remaining questions, we randomly sample 500 questions from the top 50%-ile of questions by length. The rationale for filtering on answer length filter is based on the heuristic that longer questions tend to be more specific, making them better suited for multi-document QA tasks, where specificity is crucial. For SlideVQA, we exclude single-hop questions, as they are generally non-specific and may have more than one correct answer from the document collection. We heuristically observe that multi-hop questions in this dataset are more likely to reference content from specific documents, thus making them a better fit for multi-document setups. SciGraphQA and SPIQA contain questions specific to charts or tables extracted from scientific papers. We use the ArXiv API2 to extract full document PDFs.

Footnote 2: [https://info.arxiv.org/help/api/index.html](https://info.arxiv.org/help/api/index.html)

**Document Augmentation**: To simulate realistic multi-document settings, we augment each question across all data splits with varying number of distracting documents, (\(|{\cal D}_{i}=M|\)). We intend to keep the expected number of total pages per query between 50 to 200 to ensure that there is sufficient distracting content while maintaining the practical feasibility of contemporary long-context models. Hence, based on the average number of pages per document \(P_{avg}\), we randomly sample the number of distracting documents \(l\) to lie between the range \([\lfloor\frac{50}{P_{avg}}rfloor,\lfloor\frac{200}{P_{avg}}rfloor]\). Randomly sampling \(l\) ensures that each benchmark instance contains a diverse degree of multi-document evidence, allowing for a more thorough evaluation of the QA model's retrieval and reasoning capabilities.

**Query Augmentation**: To address the challenge of ambiguous questions in datasets such as SciGraphQA, and PaperTab, we implement a query augmentation procedure to create a one-to-one mapping between a given question and the document(s) that exclusively answer it. Given an original question and the document containing answer, we utilize GPT-4o to generate more specific variations of the question, ensuring that the generated question can only be answered by the corresponding document. To maintain consistency, we constrain the LLM such that the answer to the generated question must match the provided an

\begin{table}
\begin{tabular}{p{86.7pt} p{91.1pt} c c} \hline \hline
**Benchmark** & **Content Type** & **Multi Document** & **Domain** \\ \hline L-Eval An et al. (2023) & Text & ✗ & Multi-domain LongBench Bai et al. (2023) & Text & ✓ Wikipedia Marathon Zhang et al. (2023) & Text & ✗ Multi-domain LogGLE Li et al. (2023) & Text & ✗ Multi-domain MPDocVQA Tito et al. (2023) & Text, Tables Charts & ✗ Multi-domain \\ coBench Zhang et al. (2024) & Text & ✗ & ✗ Multi-domain Ruler Hüsi et al. (2024) & Text & ✗ Wikipedia Loong Wang et al. (2024c) & Text & ✓ Multi-domain UDA Hui et al. (2024) & Text, Tables & ✗ Multi-domain NarrativeQA Kotskiy et al. (2018) & Text & ✗ Movies and Shows MMILONGENCH-DOC Ma et al. (2024) & Text, Tables Charts, Slides & ✗ Multi-domain \\ \hline \multicolumn{4}{p{86.7pt}}{**VisbotBench (Ours)**} & **Text, Tables, Charts, Slides** & ✓ & **Multi-domain** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of long context document QA benchmarks with VisbQMBench.

\begin{table}
\begin{tabular}{l l l l l l l l l l} \hline \hline
**Dataset** & **Domain** & **Content Type** & **Queries** & **Decoy** & **Avg. Question Length** & **Avg. Doc Length Group** & **Avg. Doc Length Group** & **Avg. Prozos per Query** & **Avg. Prozos per Query** \\ \hline
**PaperTab** & Wikipedia & Tables; Text & 377 & 297 & 29.44 \(\pm\)3.3 & 10.95 \(\pm\)4.3 & 10.82 \(\pm\)4.4 & 113.10 \(\pm\)2.4 \\
**Feisha** & Socurciens Papers & Tables & 350 & 300 & 1.29 \(\pm\)1.6 & 15.77 \(\pm\)3.2 & 7.77 \(\pm\)1.1 & 124.33 \(\pm\)8.01 \\ SoftGraphQA & Sacericius Papers & Charles & 407 & 319 & 16.80 \(\pm\)1.5 & 22.75 \(\pm\)3.1 & 5.91 \(\pm\)2.0 & 129.71 \(\pm\)1.87 \\
**SPQ3** & Sacericius Reports & Tables; Chart & 586 & 117 & 16.60 \(\pm\)6.6 & 14.08 \(\pm\)7.9 & 5.91 \(\pm\)3.5 & 135.85 \(\pm\)3.2 \\
**SlideVQA** & Permutation Books & Silboq & 551 & 244 & 22.39 \(\pm\)5.4 & 200 \(\pm\)0.0 & 6.99 \(\pm\)2.0 & 139.71 \(\pm\)4.06 \\ \hline
**VieldbBench** & Combined & Tables; Clark, Slide, Text & 2271 & 127 & 19.11 \(\pm\)4.1 & 16.43 \(\pm\)4.5 & 8.36 \(\pm\)3.0 & 128.69 \(\pm\)0.7 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Summary of data splits included in ViSbOMBench.

 

swer. Once the augmented queries are generated, a human annotator reviews them using a predefined rubric. The rubric guides the annotator to either select one of the five generated questions, retain the original question, or mark all questions (synthetic and actual) as ambiguous, in which case, the data point is discarded. The annotator is tasked with ensuring that the question is sufficiently specific by cross-referencing the localized evidence. Additionally, the annotator performs a simple search across the entire document collection to verify that the question cannot be ambiguously answered by any other document. Experimental validation of one-to-one mapping of query with respect to the source document is given in the Appendix.

## 5 VisDoMRAG

VisDoMRAG (Fig 2) is a multimodal RAG approach for visually rich document QA consisting of two steps: (i) parallel evidence-driven unimodal (vision and textual) RAG pipelines, and (ii) Modality Fusion, which imposes consistency constraints to combine unimodal reasoning chains and arrive at a final answer.

### Evidence-driven Parallel Unimodal RAG

Textual Retrieval PipelineThe textual RAG pipeline commences with the extraction of text from the set of documents utilizing Optical Character Recognition (OCR), followed by the segmentation of the extracted text into smaller, indexable chunks. Metadata indicating the source document and page number is preserved to facilitate traceability. These chunks are then indexed using a text embedding model, enabling efficient retrieval. Relevant chunks are subsequently retrieved in relation to the specified query by a text retrieval model and provided as contextual input to the LLM along with the query to generate textual answer response.

Visual Retrieval PipelineSimultaneously, the visual RAG pipeline is dedicated to the extraction and analysis of graphical elements, including images, charts, and diagrams. For a given set of PDFs, a visual embedding model generates an index at the page-level granularity for all documents. Relevant pages are then retrieved by a visual retrieval model based on the specified query, and these pages are supplied to multimodal LLMs as visual context. This approach ensures that the model has access to critical visual information, employing its multi-modal capability to utilize visual cues from document layout and graphical structures such as charts, diagrams and infographics.

Prompting StrategyBoth the textual and visual pipelines employ a sophisticated three-step prompting strategy. Given a set of context artifacts (page images or textual chunks), and a query, the LLM is prompted with the following steps:

**1. Evidence Curation**: As a first step, we prompt the LLM to extract relevant evidence from the retrieved context. The LLM must isolate key sections, such as paragraphs, tables, and figure details, that are most likely to address the query and verbalize them in a structured form. This curation is crucial in a multi-document setup, where non-uniform sources introduce irrelevant, distracting, or adversarial content. Accurately identifying relevant information enhances the model's reasoning abilities by filtering out noise and helps mitigate LLM hallucinations.

**2. Chain of Thought Reasoning**: Extracting rea

Figure 2: Vi3DoMRAG: Given a set of documents, VisDoMRAG parallelly performs evidence-driven Visual RAG and Textual RAG, prompting the LLMs to answer a query based on the respective retrieved context via Evidence Curation and Chain-of-Thought reasoning. The reasoning chains, and answers from the text and visual pipeline are ensembled together via Modality Fusion, where the outputs of both the modalities are aligned using consistency analysis on their reasoning chain to arrive at the final answer.

 

soning chains from multi-document artefacts can help contextualize curated evidence for final answer generation. We utilize Chain-of-Thought (CoT) Wei et al. (2022) reasoning to link individual pieces of evidence that form a coherent step-by-step narrative, ensuring that the answer is not only accurate but also logically derived from the evidence, leading to more robust and reliable responses.

**3. Answer Generation**: By leveraging insights from curated, contextually relevant evidence and applying CoT reasoning processes, the answer generation step produces responses that are both precise and well-justified. Additionally, we use targeted prompts to guide the LLM about the appropriate format for answer generation as per the question type.

### Modality Fusion

The modality fusion stage is a key contribution in VisDoMRAG which differentiates it from simpler multimodal approaches. This stage takes as input the outputs from both the textual and visual pipelines, including the curated evidence, reasoning chains, and generated answers. The fusion process is orchestrated by prompting an LLM to evaluate the consistency between the reasoning chains produced by the textual and visual pipelines. This idea is inspired by self-consistency in CoT Wang et al. (2023), which leveraged multiple thoughtchains and derives an answer based on the consistency of the individual chains' results. Consistency constraint prompting is crucial for identifying and resolving any discrepancies, contradictions and filling in reasoning gaps that may arise from the separate processing of different modalities. When inconsistencies are detected, the LLM is tasked with reconciling the differences, potentially by re-evaluating the evidence or adjusting the reasoning steps. This process ensures that the final answer integrates information from both modalities in a coherent and logically consistent manner.

## 6 Experiments

In our experiments, we first evaluate different retrieval and indexing models on our benchmark, followed by end-to-end QA evaluation using the identified optimal retrieval models with different LLMs. The experiments, baselines and evaluation are discussed below:

### Retrieval

#### Baselines:

We use popular text based retrieval models: BM25 Robertson et al. (1995) a statistical baseline, and , MPNet Song et al. (2020), MiniLM Wang et al. (2020), and BGE-1.5 Xiao et al. (2023), which represent SoTA dense retrieval baselines. Text extraction from PDF documents is performed using PyTensoract. The extracted text is then segmented into 3000-character chunks using the recursive-split method Sarmah et al. (2023), with a 10% overlap to mitigate information loss.

For visual retrieval, we utilize recent advances late interaction based multi-vector retrieval models built on top of LLMs Faysse et al. (2024), namely ColPali and ColQwen2, which have PaliGemma Beyer et al. (2024) and Qwen2 Yang et al. (2024) as their base LLMs. Readers are encouraged to refer to the appendix for further details of these models.

**Evaluation:** Evidence extraction is assessed using ANLCS between ground truth evidence and retrieved chunks/pages. Document identification evaluates the retrievers' ability to select the correct source document in a multi-document setup. We report the rate of instances where the ground truth document is the source of the majority of the retrieved context.

### End-to-End QA

We use the best text and visual retrieval models from the retrieval experiments for End-to-End QA evaluation.

**Baselines:** We benchmark our method using LLMs capable of handling multi-image inputs and long context. To this extent, we include two off-the-shelf models Gemini-1.5-Flash Reid et al. (2024), and ChatGPT-4o OpenAI (2024), as well as Qwen2-VL-7B-Instruct Yang et al. (2024), an open-source LLM with visual and long context capabilities. We evaluate these LLMs in four approaches: 1. Long Context: where text content of all documents queries for a sample is passed as context, and 2. TextualRAG, 3. VisualRAG, and, 4.VisDoMRAG as described in Section 5.

**Evaluation:** For PaperTab, we borrow the modified implementation of Word Overlap F1 from Hui et al. (2024), which takes into account different answer types (binary, short text). For all other datasets, we report the Word Overlap F1, which serves as a flexible metric to evaluate different answer types 



## 7 Results

### Retrieval Evaluation on VisDoMRBench

Fig. 3 presents the performance of various retrieval models in extracting evidence from documents, evaluated using the Averaged Normalized Longest Common Subsequence (ANLCS) between retrieved evidence and ground truth evidence, for different context window lengths (\(k=[1,5,10,20]\)). Based on a threshold of ANLCS = 0.7, we use a context window of \(k=5\), \(k=7\) for Visual RAG and Textual RAG, respectively, with ColQwen2 and BGE-1.5 as the visual and textual retrievers. ColQwen2 outperforms other retrieval baselines across different datasets due to the presence of a strong LLM backbone (Qwen2).

Table 4 evaluates the retriever performance in identifying the correct source document, presenting the proportion of queries with accurate document retrieval for \(k=5\). A document is considered correctly retrieved if at least \(\lceil k/2\rceil\) of the retrieved documents correspond to the ground truth source documents. We observe that ColQwen2 is better than the next closest BGE1.5 model by 4.5%. Notably, we observe a substantial performance gap in this metric for SlideVQA, with visual models significantly outperforming text-only models. BM25 exhibits better performance than text-only models in this case, as slides typically contain sparse text, often comprising keywords that directly match between the query and context. Conversely, neural models struggle to capture semantic information effectively, as the textual content lacks complete sentences, limiting their ability to exploit contextual meaning.

### End-to-End Evaluation

Table 3 presents the comparative performance of VisDoMRAG against Visual GAG, Textual RAG, and Long Context methods across multiple LLMs, including Qwen2VL (7B), Gemini Flash, and GPT-4. The results indicate that VisDoMRAG consistently achieves superior performance over the baseline methods across datasets, with performance gains ranging from 2.1-21.6% (PaperTab), 0.67-36.14% (FetaTab), 0.24-11.24% (SciGraphQA), 0.81-32.87% (SPIQA), 0.40-52.16% (SlideVQA). Additionally, within each baseline method for most datasets, we observe a positive correlation between model size and performance, which aligns with

\begin{table}
\begin{tabular}{l c c c c c c} \hline
**Retriever** & **PaperTab** & **FetaTab** & **SciGraphQA** & **NPQA** & **SlideVQA** & **Average** \\ \hline MIQE & 65.51 & 84.00 & 72.73 & 82.83 & **98.53** & 81.30 \\ \hline MisalM & 65.51 & 88.85 & 91.65 & 61.06 & 0.73 & 61.56 \\ MISU & 90.18 & 98.97 & 91.40 & 95.84 & 0.73 & 73.57 \\ BCPLS & 96.81 & 94.00 & 90.91 & **98.43** & 81.85 & 92.40 \\ CalPal & 96.93 & **97.71** & 95.28 & 93.17 & 97.64 & 96.15 \\ ColQuemal & **97.61** & 96.56 & **95.58** & 96.85 & 97.82 & **96.94** \\ \hline \end{tabular}
\end{table}
Table 4: Comparison of performance in source document identification, at \(k=5\).

Figure 3: Comparison of retrieval performance across datasets, for benchmarked retrievers (BM25, MiniLM, MPNet, BGE1.5, ColPali, ColQwen), at different context window lengths, varying \(k\in[1,5,10,20]\).



established expectations in LLM scaling behavior (Hestness et al., 2017).

**Textual vs Visual RAG**: In comparing the performance of textual RAG vis-a-vis visual RAG, we observe that visual RAG consistently outperforms textual RAG. This behaviour can be explained on the basis of our dataset composition which predominantly consists of visually-rich content, and visual RAG is able to leverage visual information directly. However, the performance difference is less pronounced in scientific figure datasets such as SciGraphQA and SPIQA due to the text-rich nature of scientific papers, where figures are often accompanied by detailed descriptions within the text and captions, particularly emphasizing key results and structural details. In contrast, we see a substantial performance gap between textual and visual RAG for SlideVQA, as slides typically lack extensive textual descriptions of visualizations, forcing the visual modality to be the primary source for answering questions. Additionally, we find that Gemini often performs better in the textual modality compared to the visual modality across most datasets. This disparity could be attributed to factors such as linguistic bias Niu et al. (2021); Wang et al. (2024) or visual hallucination Ghosh et al. (2024), where the model's visual perception may be less reliable than its linguistic capabilities.

**Effect of Long-Context LLMs**: We observe that VisDoMRAG has the ability to significantly enhance the performance of smaller models, as seen from Qwen2VL. This improvement can be attributed to its ability to integrate visual and textual reasoning, compensating for the weaker long-context understanding and visual perception. The long-context LLM baselines prove to be less effective in our setup due to the high token count and the nature of the task, which requires retrieval of specific, localized evidence--essentially a needle-in-the-haystack problem. The combination of modalities in VisDoMRAG mitigates these challenges, resulting in more robust answer generation, as reflected in the results.

**Effect of Increasing Page Count**: Figure 4 evaluate the performance of different approaches averaged across LLMs, segmented by the volume of pages associated with each query. As anticipated, long-context models exhibit significant performance drop with increasing number of pages in the collection. Contrastively, our multimodal RAG approach shows consistent QA performance even at high page counts as it is able to constrain the amount of context the LLM needs to process to answer the question effectively.

**Qualitative Examples**: Fig 5 represents a qualitative example from the PaperTab dataset, where VisDoMRAG effectively uses reasoning chains and answers from unimodal RAG outputs to synthesize the correct answer. More qualitative results are presented in the Appendix.

### Ablations

We conducted ablation studies with ChatGPT4o to evaluate the effectiveness of various components in our proposed VisDoMRAG framework, as well as to compare early fusion and late fusion strategies for modality integration. The results are summarized in Table 5.

**Early Fusion vs. Late Fusion:** In our experiments, early fusion, where text extracted from document pages retrieved by the visual retriever is directly appended to the visual RAG context and used as input to the LLM, demonstrated suboptimal performance compared to the late fusion strategy em

Figure 4: Comparative performance between Long Context and VisDoMRAG (averaged across LLMs) evaluated on different ranges of number of pages \(\bar{p}=\sum_{d\in\mathcal{D}}|d|\), with Low (\(\bar{p}\leq 100\)), Medium (\(100<\bar{p}\leq 150\)), and High (\(150\leq\bar{p}\)) volumes.

Figure 5: Qualitative example from the PaperTab dataset, comparing VisDoMRAG with Unimodal RAG strategies.

 

ployed in VisDoMRAG. Specifically, early fusion struggled to integrate visual and textual evidence effectively, particularly in cross-modal reasoning, resulting in an average score of 43.63 across datasets. This limitation is likely due to the lack of independent processing for each modality, which led to weaker contextual understanding and reasoning. In contrast, late fusion--where each modality is processed independently before aggregating--proved more effective. This performance gap highlights the importance of preserving modality-specific representations before combining them, particularly when reasoning requires nuanced cross-modal evidence integration.

**Prompt Ablation:** The ablation of our proposed prompting strategies also revealed the significance of Evidence Curation, Chain-of-Thought (CoT) prompting, and Reasoning Consistency. By replacing these components with simplified prompts that employ a basic structure where the model directly generates an answer based on the question and retrieved context, without leveraging evidence curation, chain-of-thought (CoT) prompting, or reasoning consistency mechanisms. For instance, removing these prompting strategies led to an average score drop from 37.33 to 34.68 in the text-only setting and from 49.02 to 43.93 in the vision-only setting, highlighting the importance of structured prompts.

For the VisDoMRAG setting, prompt ablation led to an average performance reduction from 50.01 to 45.98, with the most notable declines observed in datasets requiring complex reasoning, such as SPIQA and SlideVQA. The simplified prompts appeared insufficient for handling the intricacies of cross-modal evidence alignment and aggregation, leading to degraded performance in these scenarios.

## 8 Conclusion and Future Work

In this work, we introduced VisDoMRBench, the first QA dataset designed to evaluate multi-document systems incorporating visually rich elements such as tables, charts, and slides. By targeting documents that require both textual and visual comprehension, VisDoMRench offers a novel benchmark to assess the capability of multimodal retrieval systems. We also presented VisDoMRAG, a multimodal Retrieval-Augmented Generation approach that fuses visual and textual pipelines using consistency-constrained modality fusion. This method demonstrated a significant improvement over traditional long context, textual, and visual RAG by 12-20%. While the current work focuses on RAG in multimodal multi-doc settings, future work will extend this approach to include reasoning through end-to-end trained models, especially in low-resource settings.

## 9 Ethics Statement

We use publicly available datasets in this research. The identities of human evaluators remain confidential, and no personally identifiable information (PII) is used at any stage of our experiments. Our work is solely intended for document QA applications. For a deeper understanding of potential risks and mitigation strategies in LLM safety, we direct users to relevant works by Kumar et al. (2024); Cui et al. (2024); Luu et al. (2024).

## 10 Limitations

Despite the advancements presented in this study, several limitations warrant consideration:

**(1) Text Extraction and Document Parsing**: A key argument for the efficacy of visual retrieval methods is the elimination of text extraction and document parsing pipelines Faysse et al. (2024). However, our approach retains this overhead, which may introduce additional complexity and processing time.

**(2) Multiple LLM calls**: Our methodology necessitates multiple LLM calls; specifically, we make three LLM calls per query. While this approach may not be optimal, it is still more cost-effective than utilizing long-context models.

**(3) Hallucinations**: As with all works involving large language models (LLMs), our approach is subject to inherent limitations related to AI safety and the risk of hallucination. These issues can affect the reliability and accuracy of the generated outputs and underscore safety risks, highlighting the need for ongoing research and refinement in the field of AI to mitigate these challenges.

Additionally, unlike previous visual QA research, which typically required models to answer ques

\begin{table}
\begin{tabular}{l l c c c c c c} \hline \multicolumn{1}{c|}{_Baseline_} & **Experiment** & **TargetPath** & **Scientific reports/graphs** & **Avgress** & **SlideVQA** & **Average** \\ \cline{2-9} Test & Ouni & 73.34 & 60.02 & 73.94 & 73.40 & 62.19 & 73.93 & 23.63 \\ \multirow{2}{*}{Test} & Penny Absortions & 37.28 & 56.81 & 31.96 & 75.81 & 73.32 & 44.68 \\  & Onus & 60.01 & 61.08 & 31.12 & 32.55 & 62.62 & 63.62 \\  & Dense Absortions & 34.52 & 99.96 & 33.31 & 32.55 & 63.44 & 43.39 \\  & Dense Absortions & **43.11** & **43.05** & **43.34** & **43.09** & **43.22** & **58.96** \\ \multirow{2}{*}{**VisDoMRAG**} & Penny Absortions & 88.34 & 62.68 & 27.88 & 76.75 & 46.53 & 45.96 \\  & Density Ensuring & 37.37 & 61.29 & 27.94 & 38.45 & 89.12 & 63.63 \\ \hline \end{tabular}
\end{table}
Table 5: Performance comparison of baseline approaches with ablations on VisDoMRench.

 

tions based solely on visual data, our framework incorporates document context. This inclusion allows for relevant textual information from other sections of the paper to contribute to the query response. However, this reliance on document context represents a limitation common to all visually rich document QA datasets, as it challenges the isolation of visual performance testing. Nonetheless, this characteristic may not be entirely detrimental; in fact, it more accurately reflects the complexity of real-world systems where multimodal information is often interdependent.

## References

* An et al. (2023) Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-eval: Instituting standardized evaluation for long context language models. _arXiv preprint arXiv:2307.11088_.
* Bai et al. (2023) Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. _arXiv preprint arXiv:2308.14508_.
* Beyer et al. (2024) Lucas Beyer, Andreas Steiner, Andre Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenchlos, Riishhab Kabra, Matthias Bauer, Matko Bosnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. 2024. Paligemma: A versatile 3b vlm for transfer. _Preprint_, arXiv:2407.07726.
* Cai et al. (2018) Deng Cai, Yan Wang, Victoria Bi, Zhaopeng Tu, Xiaojiang Liu, Wai Lam, and Shuming Shi. 2018. Skeleton-to-response: Dialogue generation guided by retrieval memory. _arXiv preprint arXiv:1809.05296_.
* Chen et al. (2022) Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W Cohen. 2022. Murag: Multimodal retrieval-augmented generator for open question answering over images and text. _arXiv preprint arXiv:2210.02928_.
* Cui et al. (2024) Tianyun Cui, Yanling Wang, Chuanpu Fu, Yong Xiao, Sijia Li, Xinhao Deng, Yunpeng Liu, Qinglin Zhang, Ziyi Qiu, Peiyang Li, Zhixing Tan, Junwu Xiong, Xinyu Kong, Zwije Wen, Ke Xu, and Qi Li. 2024. Risk taxonomy, mitigation, and assessment benchmarks of large language model systems. _Preprint_, arXiv:2401.05778.
* Dasigi et al. (2021) Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. _arXiv preprint arXiv:2105.03011_.
* Deng et al. (2024) Naihao Deng, Zhenjie Sun, Ruqi He, Aman Sikka, Yulong Chen, Lin Ma, Yue Zhang, and Rada Mihalcea. 2024. Tables as images? exploring the strengths and limitations of llms on multimodal representations of tabular data. _arXiv preprint arXiv:2402.12424_.
* Ding et al. (2022) Yihao Ding, Zhe Huang, Runlin Wang, YanHang Zhang, Xianru Chen, Yuzhong Ma, Hyunsuk Chung, and Soyeon Caren Han. 2022. V-doc: Visual questions answers with documents. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 21492-21498.
* Faysse et al. (2024) Manuel Faysse, Hugues Sibille, Tony Wu, Gautier Viaud, Celine Hudelot, and Pierre Colombo. 2024. Collapali: Efficient document retrieval with vision language models. _arXiv preprint arXiv:2407.01449_.
* Ghosh et al. (2024) Sreyan Ghosh, Chandra Evuru, Sonal Kumar, Utkarsh Tyagi, Oriol Nieto, Zeyu Jin, and Dinesh Manocha. 2024. Vdgd: Mitigating llvm hallucinations in cognitive prompts by bridging the visual perception gap.
* Gu et al. (2018) Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. 2018. Search engine guided neural machine translation. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 32.
* He et al. (2021) Qiusxiang He, Guoping Huang, Qu Cui, Li Li, and Lemao Liu. 2021. Fast and accurate neural machine translation with translation memory. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3170-3180.
* Hestness et al. (2017) Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Frederick Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. 2017. Deep learning scaling is predictable, empirically. _ArXiv_, abs/1712.00409.
* Hsieh et al. (2024) Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. Ruler: What's the real context size of your long-context language models? _arXiv preprint arXiv:2404.06654_.
* Hui et al. (2024) Yulong Hui, Yao Lu, and Huanchen Zhang. 2024. Uda: A benchmark suite for retrieval augmented generation in real-world document analysis. _arXiv preprint arXiv:2406.15187_.
* Izcard and Grave (2020) Gauter Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. _arXiv preprint arXiv:2007.01282_.
* Kocisky et al. (2018) Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gabor Melis, and Edward Grefenstette. 2018. The narrativeqa reading 

comprehens challenge. _Transactions of the Association for Computational Linguistics_, 6:317-328.
* Kumar et al. (2024) Ashutosh Kumar, Sagarika Singh, Shiv Vignesh Murty, and Swathy Raspupathy. 2024. The ethics of interaction: Mitigating security threats in llms. _Preprint_, arXiv:2401.12273.
* Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. _Advances in Neural Information Processing Systems_, 33:9459-9474.
* Li et al. (2023) Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2023. Google: Can long-context language models understand long contexts? _arXiv preprint arXiv:2311.04939_.
* Li and Tajbakhsh (2023) Shengzhi Li and Nima Tajbakhsh. 2023. Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs. _arXiv preprint arXiv:2308.03349_.
* Lin and Byrne (2022) Weizhe Lin and Bill Byrne. 2022. Retrieval augmented visual question answering with outside knowledge. _arXiv preprint arXiv:2210.03809_.
* Luu et al. (2024) Quan Khanh Luu, Xiyu Deng, Anh Van Ho, and Yorie Nakahira. 2024. Context-aware llm-based safe control against latent risks. _Preprint_, arXiv:2403.11863.
* Ma et al. (2024) Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Yinyuan Lu, Ziyiu Liu, Yan Ma, Xiaoyi Dong, et al. 2024. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations. _arXiv preprint arXiv:2407.01523_.
* Mathew et al. (2021) Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_, pages 2200-2209.
* Nan et al. (2022) Lingyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryscinski, Hailey Schoelkopf, Riley Kong, Xiangru Tang, et al. 2022. Fetaqa: Free-form table question answering. _Transactions of the Association for Computational Linguistics_, 10:35-49.
* Niu et al. (2021) Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. 2021. Counterfactual vqa: A cause-effect look at language bias. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 12700-12710.
* OpenAI (2024) OpenAI. 2024. Hello, gpt-4o! [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/).
* Pal et al. (2023) Vaishali Pal, Andrew Yates, Evangelos Kanoulas, and Maarten de Rijke. 2023. MultiTabQA: Generating tabular answers for multi-table question answering. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 6322-6334, Toronto, Canada. Association for Computational Linguistics.
* Pang et al. (2023) Bo Pang, Erik Nijkamp, Wojciech Kryscinski, Silvio Savarese, Yingbo Zhou, and Caiming Xiong. 2023. Long document summarization with top-down and bottom-up inference. In _Findings of the Association for Computational Linguistics: EACL 2023_, pages 1267-1284, Dubrovnik, Croatia. Association for Computational Linguistics.
* Peng et al. (2023) Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. _Preprint_, arXiv:2309.00071.
* Peng et al. (2019) Hao Peng, Ankur P Parikh, Manaal Faruqui, Bhuwan Dhingra, and Dipanjan Das. 2019. Text generation with exemplar-based adaptive decoding. _arXiv preprint arXiv:1904.04428_.
* Pramanick et al. (2024) Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. 2024. Spiqa: A dataset for multimodal question answering on scientific papers. _arXiv preprint arXiv:2407.09413_.
* Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. _arXiv preprint arXiv:2403.05530_.
* Robertson et al. (1995) Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beulieu, Mike Gateford, et al. 1995. Okapi at trec-3. _Nist Special Publication Sp_, 109:109.
* Sarmah et al. (2023) Bhaskarjit Sarmah, Tianjie Zhu, Dhagash Mehta, and Stefano Pasquali. 2023. Towards reducing hallucination in extracting information from financial reports using large language models. _Preprint_, arXiv:2310.10760.
* Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Ti-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. _CoRR_, abs/2004.09297.
* Tanaka et al. (2023) Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. 2023. Slidevqa: A dataset for document visual question answering on multiple images. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 37, pages 13636-13645.
* Tito et al. (2023) Ruben Tito, Dimosthenis Karatzas, and Ernest Valveny. 2023. Hierarchical multimodal transformers for multipage docvqa. _Pattern Recognition_, 144:109834.
* Wang et al. (2024a) Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Munao Chen. 2024a. mdpo: Conditional preference optimization for multimodal large language models. _arXiv preprint arXiv:2406.11839_ 

Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, and Hao Wang. 2024b. Multimodal needle in a haystack: Benchmarking long-context capability of multimodal large language models. _arXiv preprint arXiv:2406.11230_.
* Wang et al. (2024c) Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. 2024c. Leave no document behind: Benchmarking long-context llms with extended multi-doc qa. _Preprint_, arXiv:2406.17419.
* Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. _Preprint_, arXiv:2002.10957.
* Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Akanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. _Preprint_, arXiv:2203.11171.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in neural information processing systems_, 35:24824-24837.
* Xiao et al. (2023) Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources to advance general chinese embedding. _Preprint_, arXiv:2309.07597.
* Yang et al. (2023) An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzhenge He, Junyang Lin, Kai Deng, Kemju Lu, Kejian Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruiz Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xunchenen Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024. Qwen2 technical report. _Preprint_, arXiv:2407.10671.
* Yuan et al. (2023) Zheng Yuan, Qiao Jin, Chuanqi Tan, Zhengjun Zhao, Hongyi Yuan, Fei Huang, and Songfang Huang. 2023. RAMm: Retrieval-augmented biomedical visual question answering with multi-modal pre-training. In _Proceedings of the 31st ACM International Conference on Multimedia_, pages 547-556.
* Zhang et al. (2023) Lei Zhang, Yunshui Li, Ziqiang Liu, Junhao Liu, Min Yang, et al. 2023. Marathon: A race through the realm of long context with large language models. _arXiv preprint arXiv:2312.09542_.
* Zhang et al. (2024) Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. 2024. infty bench: Extending long context evaluation beyond 100k tokens. _arXiv preprint arXiv:2402.13718_.

## Appendix A Appendix

### Baselines

#### a.1.1 Retrieval Models

**BM25** BM25 Robertson et al. (1995) is a widely adopted term-based ranking function based on the probabilistic information retrieval model. It calculates the relevance of a document to a given query by considering term frequency, inverse document frequency, and document length normalization. BM25 is effective for sparse text retrieval tasks, making it a standard baseline in information retrieval evaluations. We use the Python rank_bm25 implementation for our experiments.

**MiniLM** MiniLMWang et al. (2020) is a lightweight, transformer-based model designed for efficient knowledge distillation. It compresses the knowledge of larger pre-trained models into a smaller architecture while maintaining competitive performance in natural language understanding tasks. MiniLM is used in retrieval tasks due to its ability to balance computational efficiency and accuracy. We use the sentence-transformers/all-MiniLM-L6-v2 implementation in our experiments.

**MPNet** MPNetSong et al. (2020) is a transformer-based model that leverages permuted language modeling for pre-training, which helps it capture contextual information more effectively than traditional masked language models. It excels in a variety of natural language processing tasks, including text retrieval, due to its robust contextual embeddings and representation learning capabilities. We use the sentence-transformers/all-mpnet-base-v2 implementation in our experiments.

bce.1.5 The BGE model family is based on a BERT-like architecture and a three-stage training process, which collectively enhance its adaptability and generalization capabilities. Pre-training is performed on large-scale plain text corpora using a tailored MAE-style approach, effectively encoding polluted text and reconstructing the clean version. The model then undergoes contrastive learning with 

in-batch negative sampling, leveraging large batch sizes to improve embedding discriminativeness. Finally, task-specific fine-tuning is employed using labeled datasets, applying instruction-based prompts and advanced negative sampling techniques to better accommodate diverse task types. We use the BAAI/bge-base-en-v1.5 model in our experiments, which is their large english model, version 1.5.

ColPali, ColQwen2ColPali (Faysse et al., 2024) performs late interaction retrieval on document embeddings generated directly from document page images using Vision-Language Models (VLMs). By passing the document images through PaliGemma (Beyer et al., 2024), ColPlai uses the projected token embeddings to index the document pages, eliminating the need for OCR or document parsing. The multimodal alignment learned by VLMs allows both text queries and document image embeddings to exist in a shared semantic vector space, enabling more precise and efficient retrieval. ColQwen2 is a similar model with Qwen2 (Yang et al., 2024) as the base VLM. We used the vidore/colpali-v1.2, vidore/colqwen2-v0.1 implementations for our experiments.

#### Llms

We used Qwen/Qwen2-VL-7B-Instruct, chatgpt-4o-latest and gemini-1.5-flash in our experiments. For Chat GPT4o and Gemini, we set the temperature as 0.5, and use the default hyperparameters. For Qwen2-VL, the pixel range is set to \([256\times 28\times 28,640\times 28\times 28]\). For Long Context evaluation, we use Qwen/Qwen2-7B-Instruct because of the implementation availability of long context inference using YaRN (Peng et al., 2023). We report results on a single run of experiments.

### Datasets

The datasets use in our benchmark are described below. Fig 6-10 represent the distribution of pages per query in all the data splits.

FetaTabFetaTab is derived from UDA (Hui et al., 2024), which sources its data from FetaQA (Nan et al., 2022). Many source datasets provide only segmented and partial content, lacking complete documents. To resolve this, UDA conducted a thorough source-document identification process, verifying and collecting the complete original document files based on metadata or content fragments. This was followed by rigorous matching and reorganization to form complete triplet data pairs consisting of document-question-answer. Additionally, UDA categorizes queries based on the source of factual evidence, filters out Q&As without available answers, converts token-based data patterns to natural language, unifies data formats and structures across datasets, and designs specific LLM prompts tailored for each dataset after experimental trials. FetaTab is licensed under the CC-BY-SA-4.0 license.

PaperTabPaperTab is also sourced from UDA Hui et al. (2024), which obtains its data from the QASPER (Dasigi et al., 2021) dataset. Similar to the process described for FetaTab, UDA emphasizes the necessity of ensuring the integrity of original documents for effective document analysis. This involves a comprehensive process of identifying, verifying, and collecting complete original document files, followed by matching and re-organization to create document-question-answer triplets. UDA also categorizes queries, filters out unanswered Q&As, converts data patterns to natural language, unifies data formats, and designs specific LLM prompts for each dataset based on experimental evaluations. PaperTab is released under the CC-BY-SA-4.0 license.

SpiqaSPIQA (Pramanick et al., 2024) is a large-scale and challenging question-answering dataset that focuses on figures, tables, and text paragraphs extracted from scientific research papers across various computer science domains. The dataset encompasses a diverse array of visual elements, including plots, charts, schematic diagrams, and result visualizations. SPIQA consists of 270K questions divided between training, validation, and three different evaluation splits. To ensure the highest quality and reliability, SPIQA employs both au

Figure 6: Distribution of pages per query for FetaTab.

 

tomatic and manual curation methods. The dataset is released under the CC-BY-SA-4.0 license, allowing for broad use while ensuring proper attribution.

SciGraphQASciGraphQA Li and Tajbakhsh (2023) is a synthetic multi-turn question-answer dataset centered on academic graphs, representing a significant advancement in the field of visual question answering. At 13 times larger than the previous largest dataset, ChartVQA, it stands as the largest open-sourced chart VQA dataset with non-synthetic charts. The dataset was constructed from 290,000 Computer Science and Machine Learning papers published on ArXiv between 2010 and 2020, with the help of Palm-2 generating 295,000 samples of open-vocabulary multi-turn question-answer dialogues about the graphs. Each dialogue is contextualized with the paper title, abstract, relevant paragraphs, and rich contextual data from the graphs, achieving an average of 2.23 questionnaire surrus per graph. SciGraphQA is released under the MIT license.

SlidevQAL SlidevQA Tanaka et al. (2023) is a multi-image document VQA dataset that contains over 2,600 slide decks, comprising more than 52,000 slide images and 14,500 questions regarding the slide content. This dataset requires complex reasoning skills, including single-hop, multi-hop, and numerical reasoning. It also provides annotated arithmetic expressions for numerical answers, enhancing numerical reasoning capabilities. More details about the dataset can be found under the license at this link.

#### a.2.1 Distracting Documents

Distracting documents are introduced as additional, irrelevant documents within the retrieval set to simulate real-world scenarios where the task is to find the most relevant context among multiple documents. These distracting documents are selected randomly from the in-domain documents of a given dataset, ensuring that they are contextually similar but not directly relevant to the query.

To validate the effectiveness of the one-to-one mapping and evaluate the robustness of the retrieval system in the presence of distracting documents, we conducted an experiment where we removed the oracle document (i.e., the ground truth document) from the retrieval set. In this setup, we provided GPT-4 with the option to refuse to answer the query if it deemed the provided context insufficient for

Figure 8: Distribution of pages per query for SPIQA.

Figure 10: Distribution of pages per query for Slide-VQA.

Figure 7: Distribution of pages per query for PaperTab.

Figure 9: Distribution of pages per query for SciGraphQA.

 

answering the query. The refusal rate was then measured in both the default setting (with the oracle document included) and without the oracle document.

The results, shown in Table 6, reveal a significant increase in refusal rates when the oracle document is removed. In the default setting, the refusal rate is relatively low across the datasets, with PaperTab and FetaTab having 26% and 4% refusal rates, respectively, indicating that GPT-4 was able to find sufficient context for answering the queries. However, when the oracle document is excluded, the refusal rate jumps dramatically, with all datasets showing refusal rates between 94% and 98%. This increase highlights the importance of having the correct document in the retrieval set, as the model struggles to generate answers without access to the relevant context.

This experiment underscores the critical role of the oracle document in ensuring that the retrieval system can effectively answer queries and demonstrates how distracting documents can hinder retrieval performance when they introduce irrelevant or insufficient context. The results validate our approach in testing the one-to-one mapping of queries to documents and emphasize the importance of ensuring that the retrieval system can maintain performance in the presence of distracting documents.

### Examples

#### a.3.1 Query Augmentation

Tables 7 and 8 represent examples of query augmentation during dataset construction for PaperTab and SciGraphQA.

 



### Computational Resources

Table 9 describes the Computational Resources used for running this paper's experiments.

\begin{table}
\begin{tabular}{|c||c|c|c|c|c|c|c|} \hline \hline Dataset & Final & Final & Final & Final & None & Wei & Strb \\ \hline \hline
20146 & -0.518 & -0.203 & -0.208 & -0.53 & 1.42 & 0.35 \\
2017151 & - & -1.050 & 590 & 9.64 & 27.4 & 98.54 \\
2017151 & - & -1.050 & 590 & 9.64 & 27.4 & 98.54 \\
2017151 & - & -1.050 & 590 & 11.35 & 48.53 \\
2017151 & - & -1.050 & 100 & 10.75 & 15.43 & 45.59 \\
2017151 & - & -1.050 & 100 & 2.71 & 47.59 & 97.58 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of various datasets, Mean and Var indicate the mean and variance of target phrase numbers, “Prre denotes percentage of present keyphrases.

Figure 11: Qualitative example from the PaperTab dataset, comparing VisDoMRAG with unimodal RAG strategies, with Qwen2VL as the base LLM.





**Who started their term as Chief Minister in 1995, and what party did he belong to?**

\begin{table}
\begin{tabular}{l l} \hline
**Original Query** & **Augmented Query** \\ \hline What baselines did they consider? & What baseline approaches using state-of-the-art PDTB taggers were employed for the evaluation of causality prediction in the automatic causal explanation analysis pipeline? \\ \hline What is the average length of the claims? & What is the average token count of claims as reported in Table 2 of the PERSSPCTRUM dataset! \\ \hline What was the performance on the self-collected corpus? & What F1 scores did the CM-Net achieve for intent detection and slot filling on the CAIS dataset as detailed in Table 6? \\ \hline Do they test their framework performance on commonly used language pairs, such as English-to-Germanization in a simulated under-resourced scenario & Does the paper report results for English-to-German translation in a simulated under-resourced scenario using their proposed multilingual NMT framework? \\ \hline \end{tabular}
\end{table}
Table 7: Example of query augmentation from PaperTab dataset.

Figure 12: Qualitative example from the FeetaTab dataset, comparing VisDoMRAG with unimodal RAG strategies, with Gemini as the base LLM.



What observation about aggregate gap in relation to DBS number is presented in the paper's Fig. 4?

[FIGURE:S5.F13][E

Figure 13: Qualitative example from the ScigraphQA dataset, comparing VisDoMRAG with unimodal RAG strategies, with Qwen2VL as the base LLM.

\begin{table}
\begin{tabular}{l l} \hline
**Metric** & **Details** \\ \hline GPU Hours & 100 \\ GPU Specification & RTXA600 \\ Number of GPU(S) & 1 \\ Max Model Parameters & 7B \\ \hline \end{tabular}
\end{table}
Table 9: Computationa Resources for VisDoM RAG experiments.

 

* [145] M. C. Bader, A. D. Krizhevsky, and S. Ghemawat, "A review of the state-of-the-art deep learning framework for large scale visual recognition," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 2698 of _Lecture Notes in Computer Science_, vol. 307 of _Lecture Notes in Computer Science_, vol.

 

* [15] M. C. Bader, A. D. Krizhevsky, and S. Ghemawat, "A simple model for the classification of social networks in a large scale," _Journal of Machine Learning Research_**2006**, no. 3, pp. 47-89, 2007.
* [16] J. L. Friedman, R. Tibshirani, and T. Salakhutdinov, "The k-means clustering algorithm: A survey on machine learning," _IEEE Transactions on Knowledge Discovery and Data Mining_**2006**, no. 1, pp. 1-10, 2006.
* [17] P. F. Hastie, N. H. Lee, and Y. Bengio, "Deep learning with deep convolutional neural networks," _arXiv preprint arXiv:1502.00987_**, 2015.
* [18] E. Hinton, I. Guyon, and V. Vanhoucke, "Learning to improve data quality by reducing internal covariate information," _Advances in Neural Information Processing Systems_**2006**, no. 1, pp. 1-10, 2006.
* [19] W. Tehgaler, "An introduction to boosting deep belief nets," _Neural Networks_**2006**, no. 1, pp. 1-10, 2006.

[MISSING_PAGE_POST]

 

You are given a question about a figure in a research paper. Make that question more specific to the research paper, such that it can only be answered by that paper.

Figure 16: Prompt Template used for Query Augmentation.

Figure 17: Prompt Template used for Unimodal RAG and Long Context experiments.

Figure 18: Prompt Template used for VisDoMRAG.

 



## Instructions for Human Evaluator: Selecting the Optimal Query

When evaluating a list of AI-generated queries, its essential to consider not only how well each query fits the context but also how natural it sounds and whether it maintains a balance between clarity and specificity. As you review the queries, your goal is to select the one that best matches the context, feels human-like, and doesn't over-include information from the answer.

In some cases, none of the queries may be optimal. Therefore, if you find that all the AI-generated queries are inadequate, you also have the option to **reject all of them** and instead return the **original query**. This ensures that you always have a fallback and aren restricted to using the AI-generated versions.

To guide your decision-making, the table below outlines specific criteria and a rubin that will help you evaluate each query on its quality. You'll weigh the mathematics of the query, its relevance to the context, whether its overly detailed, and its adaptability for reuse across different contexts.

### Step-by-Step Evaluation Process:

1. [noitemsep]
2. **Review the Context:** Begin by understanding the context or document that the queries reference, as this will be key to judging relevance and specificity.
3. **Score Each Query:** Evaluate each of the five AI-generated queries based on the rubin above. Assign a score from 1 to 5 for each criterion.
4. **Sum the Scores:** Add up the scores for each query across all four criteria.
5. **Select or Reject:** Choose the highest-scoring query, but if none of the queries perform satisfactorily (e.g., if at are under a threshold like 12/20), you have the option to **reject all queries** and refer back to the original.

[MISSING_PAGE_POST]

Figure 19: Brief of Reviewer Instructions, including the Evaluation Rubric.