M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework

## Abstract

Abstract The ability to understand and answer questions over documents can be useful in many business and practical applications. However, documents often contain lengthy and diverse multimodal contents such as texts, figures, and tables, which are very time-consuming for humans to read thoroughly. Hence, there is an urgent need to develop effective and automated methods to aid humans in this task. In this work, we introduce M-LongDoc, a benchmark of 851 samples, and an automated framework to evaluate the performance of large multimodal models.
We further propose a retrieval-aware tuning approach for efficient and effective multimodal document reading. Compared to existing works, our benchmark consists of more recent and lengthy documents with hundreds of pages, while also requiring open-ended solutions and not just extractive answers. To our knowledge, our training framework is the first to directly address the retrieval setting for multimodal long documents. To enable tuning open-source models, we construct a training corpus in a fully automatic manner for the question-answering task over such documents.
Experiments show that our tuning approach achieves a relative improvement of 4.6% for the correctness of model responses, compared to the baseline open-source models. Our data, code, and models are available at https://multimodal-documents.github.io

## 1 Introduction

The ability to comprehend long and complex multi-modal documents and respond to user queries about them is crucial in various practical applications such as business intelligence analysis, academic literature review, and legal research (Mathew et al., ).
Recently, large multimodal models such as GPT-4V (OpenAI, ) have shown great potential in processing and analyzing diverse types of information, including text, images, and even structured data (Huang et al., ).
These models offer the promise of automating tasks that traditionally required extensive human effort, such as document analysis, information retrieval, and question-answering (Fujitake, ).
However, real-world documents often present significant challenges due to their length, complexity, and multimodal nature, containing a mix of text, figures, tables, and charts (Faysse et al., ).
Thus, it is not clear whether current models are capable of an in-depth understanding of lengthy multimodal documents.
On the other hand, while existing benchmarks have fostered great progress in document understanding, they often fall short in representing these challenges, typically focusing on documents with less than 50 pages,
and limited to simpler extraction-based questions (Ma et al., ).

To address these limitations, we introduce M-LongDoc, a comprehensive benchmark consisting of
851
samples specifically designed to evaluate the performance of large multimodal models on lengthy and diverse documents.
Unlike previous datasets (Mathew et al., ; Liu et al., ; Ma et al., )
that mainly contain short documents, M-LongDoc features recent documents *spanning hundreds of pages, encompassing a wide range of topics and document structures* as shown in Figures 2
and 2.
In addition, as shown in Figure 3, our benchmark *goes beyond simpler extractive questions, requiring models to provide open-ended solutions* that demonstrate in-depth understanding of the document content (Fan et al., ).
M-LongDoc poses a question answering task where models have to analyze and reason over texts, figures, or tables in each multimodal long document.
We aim for this benchmark to serve as a valuable resource for researchers and practitioners, enabling more rigorous testing and development of multimodal document understanding systems.

Figure: Figure 1: Data distribution of document topics in our M-LongDoc benchmark.
Refer to caption: extracted/5987183/assets/documodal-data-pie.png

Another challenge we have to overcome is that due to the lengthy content of multimodal documents and the in-depth solutions, the evaluation of the open-ended question-answering task becomes tricky.
To assess such open-ended solutions in a scalable and standardized manner, we design an automated evaluation framework that does not require reference answers or human annotation.
Inspired by previous works in model-based evaluation (Zheng et al., ; Zhao et al., ; Liu et al., ), our evaluation framework leverages a detailed evaluation guide and multiple judge models to score the correctness of each generated solution.

With our proposed M-LongDoc and evaluation framework, we conducted
preliminary study on existing models and the results show that they struggle with figure and table-based questions compared to text-based questions, revealing their multimodal bias and weaknesses (Chen et al., ).
Furthermore, we observed that the models can be easily distracted by irrelevant content in the document pages (Shi et al., ), even with the aid of retrieval-augmented generation.
To enhance the robustness of multimodal models against potentially irrelevant retrieved content,
we propose a retrieval-aware tuning approach for multimodal document reading.
This framework unifies supervised fine-tuning and retrieval augmented generation by including distracting content from other modalities and pages in each document.
Thus, we adapt models to effectively incorporate the domain knowledge in multimodal documents while ignoring the content irrelevant to the given query.
Compared to existing training methods, ours is the first to address retrieval-augmented multimodal training for rich document layouts.
To support this training framework and the enhancement of open-source models, we further construct a training corpus of 10,070 samples for question-answering over multimodal long documents.
Experiments show that our approach achieves a 4.6% relative improvement in the correctness of model responses, compared to the baseline model.

Figure: Figure 3: Example questions in different multimodal document question answering benchmarks. For illustration, we include content from the relevant page in the original document. The example question from M-LongDoc is more complex than those from other benchmarks, as it requires an explanatory answer rather than an extraction of a short text span. Furthermore, it requires the model to understand the semantics of both image and text. Please note that in our benchmark setting, the model is provided with all page contents from the document, and not only the relevant page.
Refer to caption: x5.png

The key contributions of this work are threefold:
1) We introduce M-LongDoc, a multimodal benchmark that more accurately represents the challenges of real-world document understanding tasks. Our automated evaluation framework enables scalable and standardized assessment of open-ended solutions.
2) Our evaluation of leading models indicates that most models struggle with figure and table-based questions compared to text-based questions, revealing their multimodal bias.
3) We propose a retrieval-aware tuning framework that together with our large-scale training corpus, significantly improves the efficiency and effectiveness of multimodal document reading.

Thus, we believe that this work contributes to the field of document understanding and paves the way for more capable and practical applications of large multimodal models in real-world scenarios. To accelerate the studies in our community, we will make the M-LongDoc benchmark, the training corpus for multimodal document reading, and our source code publicly available.

## 2 M-LongDoc Benchmark

To evaluate the multimodal long document understanding ability of existing models, we present M-LongDoc, a challenging and diverse benchmark.
Notably, the benchmark focuses on open-ended questions that require in-depth solutions and analysis over very long documents with more than 200 pages on average.
For diversity, the questions cover the academic, financial, and product domains, with multiple topics in each domain.

### 2.1 Data Collection

To support our evaluation benchmark, we manually source high-quality multimodal documents from publicly accessible sources.
Concretely, we source research papers(^1^11[https://arxiv.org](https://arxiv.org)), company reports(^2^22[https://www.annualreports.com](https://www.annualreports.com)) and product instruction manuals(^3^33[https://www.manualslib.com](https://www.manualslib.com)) for the academic, financial, and product domains respectively.
Thus, the dataset covers a range of document formats and domains.
As research papers often require domain expertise, we constrain the academic domain to computer science topics.
To reduce the risk of data contamination or memorization when evaluating existing models (Dong et al., ), we constrain the documents to be published in January 2024 or later.
As most existing models are unable to process raw PDF files, we conduct a simple data processing to extract the texts and relevant images from each document.
Specifically, we use the PyMuPDF(^4^44[https://pymupdf.readthedocs.io](https://pymupdf.readthedocs.io)) tool to automatically extract the text from each page.
To extract the figures and tables from each page, we leverage an existing object detection model (Pfitzmann et al., ).
Thus, the processed documents consist of interleaved textual and visual content, where the visual contents are extracted images of figures and tables.

Figure: Figure 4: Overview of our data construction process with question verification stages. For brevity, we shorten the checklist prompts and include the full details in Appendix A.1.
Refer to caption: x6.png

### 2.2 Question Generation

To construct diverse and challenging open-ended questions, we leverage a semi-automated pipeline.
Concretely, as shown in Figure 4, given a specified content category, we first randomly select a page from the document that contains the specific content category, such as texts, tables, or figures.
Consequently, we randomly select a question generator from a pool of leading multimodal models and instruct it to generate a challenging question based on the document page.
To ensure that the question generator has sufficient context, we also provide the previous page and subsequent page as additional inputs during the question generation process.

To improve the quality of the generated questions, we conduct an automated verification process as a preliminary filter for unsuitable questions.
Concretely, the question generator is also instructed to reflect on the generated question and follow a multi-step checklist to validate the question.
For example, the checklist includes checking if the question is relevant to the document page, if the specified content category is required to answer the question, and whether the question is answerable.
The question is rejected if it does not satisfy any condition in the checklist.
Lastly, we employ a team of annotators to conduct final validation for each question.
We employ expert annotators who are Ph.D. students and above in computer science for the academic domain, and professional annotators for the finance and product domains.
To be consistent, we provide a similar checklist and instruction as our automated verification stage, and the annotation details are included in Appendix A.1.
We found that 80.1% of the generated questions satisfied the automated verification. Of these questions that passed automated verification, 80.9% also satisfied the human verification. Thus, we only retain 851 questions that satisfied both the automated and human verification.

The statistics of our benchmark dataset are shown in Table 1, where we ensure a balanced distribution of questions and documents from each domain and question category.
In this work, we focus on questions that require a single page of content to answer,
and only retain answerable questions.
Please also note that while each question focuses on a specific category in a document page, the page may contain content from other categories as context.
For instance, a table-based question may also require comparisons to other tables or texts from the same page.

Compared to the existing benchmarks in Figure 2, M-LongDoc poses a greater challenge in two main aspects. Firstly, the significantly greater number of pages and tokens in each multimodal document poses extreme computational costs and opportunities to be distracted by irrelevant content (Shi et al., ). While this challenge may be mitigated by retrieval-augmented generation (Chen et al., ), our preliminary study in Section 2.4 shows that existing models are still hindered by their multimodal bias (Chen et al., ). In addition, our benchmark poses challenging open-ended questions as shown in Figure 3, requiring models to produce in-depth analyses in their solutions.
Thus, we believe M-LongDoc is a more realistic and challenging benchmark compared with existing datasets focusing on short answers that can often be extracted directly from the source document.

### 2.3 Automated Evaluation

Figure: Figure 5: Our automated evaluation framework to assess the correctness of open-ended solutions for multimodal question answering. The full evaluation guide is included in Appendix A.3.
Refer to caption: x7.png

Given the challenging nature of our multimodal long document benchmark, it is crucial to have a scalable and standardized evaluation method.
However, it is less feasible to conduct comprehensive human evaluation due to high labour costs and lack of reproducibility (Clark et al., ).
Thus, inspired by previous works in automatic evaluation
(Zheng et al., ; Zhao et al., ; Liu et al., ), we propose an evaluation framework based on a committee of multimodal judges.
Concretely, we leverage multiple leading multimodal models to score each answer to a question based on the criteria of correctness.
To provide a clear guideline for evaluation, we define the task introduction and criteria as shown in Figure 5.
To provide more detailed evaluation instruction beyond the basic definitions above, we further construct detailed evaluation steps, based on the task and criteria.
Thus, the finalized evaluation guide for each judge model consists of the task introduction, criteria, and fixed evaluation steps.

To provide a more reliable evaluation and reduce intra-model bias (Verga et al., ), we leverage multiple judges to evaluate each candidate answer.
Specifically, each judge model $M_{j}$ is provided with the evaluation guide $g$, ground-truth evidence page as context $c$, question $q$, and candidate answer $\hat{a}$, and instructed to assign a correctness score from 1 to 5.
However, we observe some variance in the output scores, even with the same judge model and inputs.
Thus, we sample multiple scores from each judge model $M_{j}$ and aggregate the scores to obtain a fine-grained, continuous score that better reflects the quality of the candidate answer:

$$ $\displaystyle\text{Score}=\frac{1}{J\cdot K}\sum_{j=1}^{J}\sum_{k=1}^{K}s_{j,k }\sim M_{j}(g,c,q,\hat{a})$ (1) $$

where $J=3$ is the number of judge models and $K=5$ is the number of sampled scores per judge model.
While there may be some degree of subjectiveness in our framework, our analysis in later sections shows that it largely agrees with human preferences with minimal bias (Zheng et al., ).
Thus, we believe this automated evaluation framework is reliable and more scalable.

**Table 1: Benchmark dataset statistics with respect to each domain.**
|  | Academic | Product | Financial | All |
| --- | --- | --- | --- | --- |
|  | Paper | Manuals | Report |  |
| Documents | 60 | 60 | 60 | 180 |
| Questions | 311 | 279 | 261 | 851 |
| Text-based questions | 95 | 95 | 81 | 271 |
| Figure-based questions | 114 | 93 | 76 | 283 |
| Table-based questions | 102 | 91 | 104 | 297 |
| Average pages per document | 201.2 | 277.8 | 153.4 | 210.8 |
| Average text tokens per document | 114,129.8 | 109,745.0 | 139,089.3 | 120,988.0 |
| Average figure images per document | 90.8 | 368.3 | 24.1 | 161.13 |
| Average table images per document | 34.9 | 96.6 | 83.8 | 71.8 |

### 2.4 Preliminary Study

**Table 2: Preliminary study on M-LongDoc for open-source and close-source models. We report the correctness score out of 5 for text-based, figure-based, table-based, and all questions respectively.**
|  | Text | Figure | Table | All |
| --- | --- | --- | --- | --- |
| Gemini-1.5-pro-002 |  |  |  |  |
| w/ top $k=1$ pages | 4.38 | 3.73 | 4.16 | 4.11 |
| w/ top $k=5$ pages | 4.60 | 4.31 | 4.54 | 4.49 |
| w/ top $k=10$ pages | 4.61 | 4.29 | 4.62 | 4.51 |
| w/ top $k=20$ pages | 4.63 | 4.33 | 4.38 | 4.46 |
| Qwen2-VL-7B-Instruct |  |  |  |  |
| w/ top $k=1$ pages | 4.05 | 3.25 | 3.36 | 3.57 |
| w/ top $k=5$ pages | 4.17 | 3.67 | 3.46 | 3.78 |
| w/ top $k=10$ pages | 4.08 | 3.62 | 3.19 | 3.65 |
| w/ top $k=20$ pages | OOM | OOM | OOM | OOM |

To investigate the limitations of existing models, we conduct a preliminary study on a subset of 100 random samples from our M-LongDoc benchmark.
Concretely, we select Gemini (Google, ) and Qwen2-VL (Wang et al., ) to represent highly capable models for the close-source and open-source settings respectively.
While large multimodal models have shown impressive capabilities and support longer input contexts,
they often struggle with understanding very long documents and may incur great computational costs (Dingjie et al., ).
Thus, we focus our study on the retrieval-augmented generation paradigm (Lewis et al., ), which leverages a retriever to select only the most relevant content, and the retrieved content is used to augment the generator model inputs for question answering.
Concretely, we use ColPali (Faysse et al., ) as a state-of-the-art multimodal retriever and leverage the top $k$ pages of multimodal content as context.
We include more details on the implementation and evaluation of retriever performance in Appendix A.2.

Notably, as shown in Table 2, we observe significantly lower performance for figure-based and table-based questions, as compared to text-based questions.
We believe that this discrepancy suggests that current models are weaker in processing image-based contents in multimodal documents, or may be biased towards the textual content, even when they are trained on interleaved multimodal data (Chen et al., ).
Furthermore, we find that increasing the amount of retrieved content may not improve overall performance, and may even lead to worse performance or out-of-memory (OOM) issues.
This indicates that the multimodal generator models may be easily distracted by irrelevant content in longer contexts (Shi et al., ).
Thus, we believe it is crucial to address these challenges of existing models for processing multimodal long documents.

Additionally, to verify the reliability of our automated evaluation, we conduct manual human scoring based on the same evaluation guide.
For the samples in this preliminary study, we observed a Pearson correlation of 88.9% with $p<0.001$ between the final aggregated score from the judge models, and the human annotator.
Thus, we believe that our evaluation framework can achieve a very high agreement with human preferences despite the open-ended and in-depth nature of the answers.

## 3 Retrieval-Aware Multimodal Tuning

Figure: Figure 6: Our retrieval-aware multimodal tuning framework to enhance the ability of models to identify and utilize pertinent content in multimodal documents. At training time, the model is provided with more relevant pages retrieved from the document, which may contain both the gold evidence page and multiple ‘distractor’ pages.
Refer to caption: x8.png

While current multimodal models are often trained on interleaved inputs with multiple pieces of texts and images (Liu et al., ), they may not be well-optimized for multimodal documents. Specifically, multimodal documents are more challenging to understand as they contain diverse multimodal content including unstructured texts, and images representing structured tables and figures.
Thus, models require a more fine-grained understanding and perception of the multimodal document content.
Furthermore, the retrieval setting requires models to ground their outputs in the additional context by identifying and utilizing
only the pertinent content.
However, previous works (Shi et al., ) and our preliminary study have shown that they may still be easily misled by some irrelevant information in the retrieved content.
To our knowledge, there is no open-source model that can address these challenges in multimodal long documents.

To this end, we propose a simple and effective retrieval-aware multimodal document tuning approach.
Inspired by previous works in retrieval augmentation (Chen et al., ; Zhang et al., ), we include both retrieval context from the ground-truth evidence page as well as potentially irrelevant pages during training.
Our approach as shown in Figure 6 presents a training paradigm that is more realistic and similar to the challenges faced during test-time retrieval of multimodal content.
Thus, the model learns to handle potentially noisy retrieval contexts, while improving its text understanding and visual perception capabilities to utilize the most relevant document content.
To construct the training data, we leverage the same process as shown in Figure 4 to construct a training corpus of 10,070 samples across 300 documents, and leverage the respective question generator models to also produce a high-quality answer based on each ground-truth evidence page.
We omit the human verification stage for scalability and cost-efficiency, as majority of the automatically verified samples also satisfied human verification.
To assess the quality of the generated training solutions, we evaluated using our automated scoring framework on a random subset of 100 samples.
We observed an average correctness score of 4.82, indicating very high quality of answers in the training data.
To avoid data leakage, we ensure that the documents used to construct the training corpus do not overlap with the evaluation set.
For example, we ensure that the training documents are from different companies and products, and are published in an earlier time period.

## 4 Experiments

### 4.1 Task Setting

To ensure a practical task setting, we focus on the retrieval-based paradigm, which avoids the exorbitant cost to process the full document.
Based on our preliminary study in Section 2.4, we use the top $k=5$ pages ranked by the retriever as a reasonable amount of context for each question.
Thus, each model is provided with the retrieved context and question as input, and required to provide an open-ended solution as output.
As discussed in Section 2.3, we leverage an automated framework with multiple judge models to score the correctness of each output solution, on a scale of 1 to 5.

### 4.2 Models

To provide a more comprehensive investigation of current models, we use both open-source and close-source models in this work.
Concretely, we select GPT-4o (gpt-4o-2024-05-13)(^5^55[https://openai.com/index/gpt-4o-system-card/](https://openai.com/index/gpt-4o-system-card/)), Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)(^6^66[https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet)) and Gemini 1.5 Pro (gemini-1.5-pro-002) (Google, ) due to their leading performance on multimodal benchmarks (Yue et al., ).
Regarding open-source models, we specifically select models which support interleaved multimodal inputs with multiple images, and fine-grained visual perception of document content.
Thus, we mainly focus on LLaVA-OneVision-7B (Li et al., ) and Qwen2-VL-7B-Instruct (Wang et al., ).
We plan to expand our investigation to other capable open-source models as they are released.

### 4.3 Hyperparameters

For all models, we use greedy decoding with temperature $T=0$ to reduce variance.
In our training framework, we set the number of training epochs to be 1, batch size as 16, and learning rate as 1e-4.
To reduce the training cost due to limited computational resources, we leverage LoRA (Hu et al., ) training with rank as 64 and alpha as 32.
Due to training instabilities with other open-source models, we mainly focus the training experiments on the Qwen2-VL-7B-Instruct model, which demonstrates leading performance compared to similar-sized models.

**Table 3: Evaluation of model performance for proprietary and open-source multimodal models. We report the correctness on our benchmark across different document domains and question categories. We bold the highest scores obtained by open-source models.**
|  |  | Domain | Question Category |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Size | Academic | Product | Finance | Text | Figure | Table | All |
| Proprietary Models |  |  |  |  |  |  |  |  |
| GPT-4o | - | 4.56 | 4.38 | 4.51 | 4.55 | 4.38 | 4.53 | 4.49 |
| Claude 3.5 Sonnet | - | 4.59 | 4.43 | 4.51 | 4.57 | 4.42 | 4.54 | 4.51 |
| Gemini 1.5 Pro | - | 4.66 | 4.43 | 4.43 | 4.59 | 4.43 | 4.52 | 4.51 |
| Open-Source Models |  |  |  |  |  |  |  |  |
| LLaVA OneVision | 7B | 3.71 | 3.74 | 3.39 | 4.03 | 3.57 | 3.30 | 3.62 |
| Qwen2-VL | 7B | 4.03 | 3.88 | 3.56 | 4.08 | 3.83 | 3.62 | 3.84 |
| w/ Retrieval Tuning | 7B | 4.17 | 4.01 | 3.86 | 4.31 | 4.00 | 3.77 | 4.02 |

## 5 Results

### 5.1 Main Results

To assess the effectiveness of our approach and the holistic performance of existing models, we report the main evaluation results in Table 3.
First, we find that our retrieval-aware multimodal tuning significantly and consistently enhances the performance of Qwen2-VL, representing a relative improvement of 4.6% in answer correctness. Thus, we view the proposed training approach as a promising strategy to enhance multimodal long document understanding ability, and reduce the gap between open-source and proprietary models.
Second, we observe that open-source models have worse performance in answering table-related questions compared to other question categories. This discrepancy highlights the need for more efforts to enhance the table understanding capability of open-source multimodal models.
We include further qualitative analysis of the model predictions in the Appendix A.4.

**Table 4: Analysis on alternative settings for our benchmark, including removing images from model inputs, and using only the render image of each page as document context, without text extraction.**
|  | Question Category |  |  |
| --- | --- | --- | --- |
| Model | Text | Figure | Table |
| Qwen2-VL | 4.08 | 3.83 | 3.62 |
| w/o Image Inputs | 4.22 | 3.37 | 3.38 |
| w/ Render Page as Inputs | 3.99 | 3.70 | 3.39 |

### 5.2 Effect of Alternative Settings

While we mainly focus on the multimodal setting with extracted texts and images, we believe it is also important to explore other settings in practice.
Our main data setting as discussed in Section 2.1 first extracts the texts, figures, and tables separately, with the figures and tables represented as individual images.
As shown in Table 4, we find a significant decrease in performance for figure-based and table-based questions when the image inputs are removed, although the model may still be able to answer the questions to an extent as the extracted text may contain partial information about the tables and figures.
We believe that this underscores the importance of leveraging multimodal content in documents, even though many documents may contain a majority of the content as texts.
The performance increases slightly for text-based questions when image inputs are removed, suggesting that the images may mislead the model in rare cases.
However, we believe this is acceptable as questions may cover a wide variety of multimodal content in practice.
On the other hand, we observe that it may be less optimal to use only rendered images of document pages as inputs, instead of separately extracting the texts, tables, and figures as in our main setting.
While the rendered page image does contain the original information and layout of the document, including texts, tables, and figures, the model may be less capable of distinguishing the content between texts and tables.

## 6 Related Work

### 6.1 Large Multimodal Models

In recent years, large multimodal models have demonstrated their capability to process and comprehend data across various formats.
Close-source models such as GPT-4o (AI, ) can reason across audio, vision and text.
Claude 3.5 Sonnet (Anthropic, ) shows marked improvement on tasks that require visual reasoning like interpreting charts and graphs.
On the other hand, Gemini 1.5 Pro (Google, ) is capable of reasoning over multiple long documents and hours of video and audio.
Open-source models such as Llava (Liu et al., ),
Idefics (Laurençon et al., ),
Otter (Li et al., ),
InternVL (Chen et al., ),
CogVLM (Wang et al., ), have also shown the potential over various types of multi-modal content including document images (Mathew et al., ), slides (Tanaka et al., ), and charts (Huang et al., ).
However, the benchmark performance of open-source models tends to lag behind that of close-source models (Yue et al., ), prompting an urgent need to bridge the gap.
In this work, we introduce a retrieval-aware multimodal tuning framework which can significantly improve the multimodal long document understanding ability of models.

### 6.2 Document Understanding Datasets

Given the practical and business applications of document understanding, researchers have devoted significant effort to this area by introducing new datasets and methods.
SearchQA (Dunn et al., ), NarrativeQA (Kočiskỳ et al., ), QuALITY (Zhu et al., ) are reading comprehension datasets over purely textual data with an average length ranging from 1850 to 60k tokens.
FinQA (Chen et al., ), DocFinQA (Reddy et al., ) are introduced in the financial domain.
MarkQA (Huang et al., ) tackes QA over knowledge bases with numerical reasoning.
DocVQA(Mathew et al., ) presents a visual question answering dataset on document images.
VisualWebBench (Liu et al., ) is a multimodal benchmark over single-page documents focusing on various QA-style tasks.
MMLongBench
(Ma et al., ) is a multimodal document understanding dataset with an average of 47.5 pages and 21k textual tokens.
Methods such as PDFTriage (Saad-Falcon et al., ) enables models to retrieve the context from long and structured documents.
TAT-LLM (Zhu et al., ) addresses QA over a hybrid of tabular and textual data.
ChartQA (Masry et al., ) is a benchmark of extractive questions-answering task over a chart image, while the Chocolate dataset (Huang et al., ) annotates the types of factual errors in machine-generated chart captions.
Compared to the datasets above, our M-LongDoc benchmark contains more lengthy documents with hundreds of pages and focusing on open-ended questions which require in-depth solutions.
We further propose an automated and reliable evaluation framework to assess the correctness of model answers, which demonstrates very high agreement with human preferences.

### 6.3 Retrieval-Augmented Generation

While recent multimodal models have shown impressive capability in many tasks, applying them directly to long document understanding tasks may face several challenges to the diverse multimodal content.
Additionally, processing entire documents with large models is often impractical, as the text alone may contain millions of tokens, leading to substantial computational costs.
Therefore, researchers have designed various retrieval augmented generation (Lewis et al., ; Chen et al., ) methods to address the issues.
In this work, we have investigated multiple retrieval methods optimized for document page retrieval, including JINA-CLIP (Xiao et al., )
BM25 (Robertson & Zaragoza, ), BGE-M3 (Chen et al., ) and ColPali (Faysse et al., ).
However, despite using retrieval, existing multimodal models are constrained by their multimodal biases (Chen et al., ) and susceptibility to irrelevant content that is inherent in the retrieved context (Shi et al., ).
Thus, we further proposed a retrieval-aware tuning framework to enhance the performance of models when leveraging retrieval for multimodal long documents.

## 7 Conclusion

In this work, we introduce M-LongDoc, a benchmark dataset consisting of 851 samples and an automated framework to evaluate the performance of large multimodal models on document question answering tasks. This benchmark is specifically designed for long and diverse document formats containing text, figures, and tables, aligning with the demands of real-world applications. Unlike existing benchmarks, M-LongDoc features more recent and lengthy documents, often hundreds of pages long, and requires open-ended solutions rather than just extractive answers.

We also propose a retrieval-aware tuning approach designed for the efficient and effective processing of multimodal long documents. To our knowledge, this is the first training framework and model to directly address the retrieval setting for such documents.
Experimental results demonstrate that our method achieves a relative improvement of 4.6% in the correctness of model responses compared to baseline open-source models. This improvement showcases the effectiveness of our approach in handling lengthy and complex multimodal documents, potentially aiding humans in various business and practical applications that require understanding and answering questions over such documents.

## Acknowledgment

This work was substantially supported by DAMO Academy through DAMO Academy Research Intern Program.

## Ethics Statement

We will release the benchmark and training dataset publicly to facilitate further research in this area.
To observe copyright rules, we do not release the documents directly, but instead the links to download each document.
All annotators in this work were volunteers.
While we focus on how models may answer questions based on multimodal documents, it is still possible for them to hallucinate information that is false or not verifiable.

## Reproducibility Statement

In this work, we have included the details of our training framework and hyperparameters in Section 3 and 4.
As discussed above, our benchmark dataset and questions will be released under a public licence.
For reproducibility, our code will be found at [https://anonymous.4open.science/r/private-multimodal-documents-B2CF/](https://anonymous.4open.science/r/private-multimodal-documents-B2CF/).

## Appendix A Appendix

### A.1 Data Verification

To verify each question in our data construction process, we use the following guide to prompt the question generator models for automated verification.
Similarly, we use the same guide for human annotation in the human verification stage.

Based on the document content and question, answer yes or no only to the following questions:

1. Does the content contain any {category}?
2. Does the question require information from the {category}?
3. Is the question clear and answerable based on the {category}?
4. Is the question of reasonable difficulty and answer cannot be simply copied?

Where {category} refers to table or figure or text, which is denoted with the question.

Note:
If questions require general knowledge or commonsense in addition to the content, it is still acceptable.
In the document PDF file, each question is shown with the ID corresponding to excel sheet, and the document page as image
In the excel sheet, indicate “yes” or “no” for each check.

### A.2 Retrieval Methods

To support our retrieval-based document question answering setting, we currently include four state-of-the art methods to retrieve relevant pages based on each question. They include text-based sparse methods such as BM25 (Robertson & Zaragoza, ) embedding-based methods such as BGE-M3 (Chen et al., ), multimodal piece-wise embedding methods such as JINA-CLIP (Xiao et al., ), and multimodal page-wise embedding methods such as ColPali (Faysse et al., ).
Note that piece-wise embedding methods separate encode each piece of text, table image, or figure image, whereas page-wise methods can encode the entire page content as a single image.
Thus, we rank each page in the document based on the similarity score or relevance score of that page with respect to the given question.
As each page may have multiple pieces of content, we consider the highest score of all pieces in a page to be the page-wise relevance score.
To compare the effectiveness of each method, we implement a standardized MRR score which refers to the mean reciprocal rank of the gold evidence page for each question.
Based on the results in Table 5, we find that ColPali which encodes each page as single image shows the best performance.
Thus, we select ColPali as the preferred retrieval method in our main experiments.

**Table 5: Retriever performance comparison.**
| Retriever | Text | Figure | Table | All |
| --- | --- | --- | --- | --- |
| BM25 | 56.2 | 31.2 | 42.0 | 43.1 |
| CLIP | 57.1 | 37.9 | 50.4 | 48.5 |
| BGE-M3 | 66.4 | 36.4 | 53.6 | 52.1 |
| ColPali | 68.7 | 67.5 | 65.9 | 67.4 |

### A.3 Evaluation Guide

To evaluate each model answer, we use the following scoring guide. Similarly, we use the same guide for human annotation in our analysis.

You will be given one response to a question based on a multimodal document containing texts, figures, or tables. Your task is to rate the response on correctness using a 1-5 scale. Please read and understand these instructions carefully, and keep them open for reference while reviewing.

Correctness (1-5) refers to how accurately, comprehensively, and appropriately the response addresses the question based on the information in the document.

5 - Fully Correct: Completely accurate, comprehensive, fully integrates relevant information from all parts of the document, and provides a coherent answer.

4 - Mostly Correct: Largely accurate with only minor errors or omissions, addresses most main points, and integrates information well.

3 - Partially Correct: Contains a mix of accurate and inaccurate information, addresses some key points but misses others, and partially integrates information.

2 - Mostly Incorrect: Has multiple inaccuracies, addresses only a small portion correctly, and shows minimal integration of information.

1 - Completely Incorrect: Contains significant errors, is irrelevant, or fails to address the question based on the document.

Evaluation Steps:
1. Thoroughly review the multimodal document and question.
2. Carefully read the response, comparing it to the document information.
3. Assess the response’s accuracy, comprehensiveness, and relevance.
4. Assign a correctness score from 1 to 5 based on the criteria.

Question: question
Response: answer

Evaluation Form (score only without explanation)
Correctness:

### A.4 More examples

#### A.4.1 Example of M-LongDoc

Table 6 illustrates an example of a challenging question in our M-LongDoc benchmark. This question tests the ability of the model to identify and analyze trends across different charts and draw meaningful comparisons.

#### A.4.2 Case study of Retrieval-aware Tuning

Table 7 displays a sample question in M-LongDoc and the answers generated by Qwen2-VL and Qwen2-VL w/ Retrieval-aware Tuning.
The answer generated by Qwen2-VL states that the Cosine method consistently shows the highest latent cosine similarity across all datasets, which is incorrect. In fact, the zero-shot stitching experiment does not involve the Cosine method. It appears that Qwen2-VL may have been misled by the keyword ”cosine” appearing elsewhere in the retrieved context.
In contrast, the answer generated by Qwen2-VL w/ Retrieval-aware Tuning correctly identifies that the affine method consistently obtains the highest latent cosine similarity (lcos) across all datasets. This demonstrates the effectiveness of our Retrieval-aware Tuning method in improving the model’s capability to comprehend retrieved context.

**Table 6: An example of a challenging question from M-LongDoc that requires the model to compare the trends of two charts in a document.**
| Question: |
| --- |
| How does the relationship between reference length percentile and the percentage of empty modes differ from the relationship between reference sentence length percentile and the probability of empty context? Explain the key differences in the trends shown by these two graphs. |
| Relevant page (truncated): |
|  |

**Table 7: Sample answers generated by Qwen2-VL and Qwen2-VL w/ Retrieval-aware Tuning, respectively.**
| Question: |
| --- |
| Which method consistently shows the highest latent cosine similarity (lcos)<br>across all datasets (MNIST, F-MNIST, CIFAR-10, CIFAR-100) in the zero-shot stitching<br>experiment, and what might this imply about the nature of transformations between latent spaces in<br>autoencoders? |
| Answer generated by Qwen2-VL: |
| The method that consistently shows the highest latent cosine similarity (lcos)<br>across all datasets (MNIST, F-MNIST, CIFAR-10, CIFAR-100) in the zero-shot stitching experiment<br>is the ”Cosine” method. This implies that the ”Cosine” method is able to capture the most similar<br>latent spaces across different architectures, modalities, and datasets. This suggests that the nature<br>of transformations between latent spaces in autoencoders is not exclusively correlated by<br>orthogonal transformations, but rather by a class of transformations larger than orthogonal<br>transformations. This finding highlights the importance of understanding and modeling the particular<br>class of transformations that control the correlation between NNs during image autoencoding. |
| Answer generated by Qwen2-VL w/ Retrieval-aware Tuning: |
| The method that consistently shows the highest latent cosine similarity (lcos)<br>across all datasets (MNIST, F-MNIST, CIFAR-10, CIFAR-100) in the zero-shot stitching experiment<br>is the ”affine” method. This method achieves the highest lcos values for all datasets, with scores of<br>0.94 for MNIST, 0.94 for F-MNIST, 0.96 for CIFAR-10, and 0.96 for CIFAR-100. This consistency in<br>high lcos values suggests that the affine transformation is particularly effective in aligning the latent<br>spaces of different autoencoders. The high lcos indicates a strong alignment between the latent<br>representations of the autoencoders, which implies that the affine transformation is capable of<br>capturing the underlying structure and relationships between the latent spaces more accurately<br>than other methods. The results imply that the nature of transformations between latent spaces in<br>autoencoders is not exclusively controlled by orthogonal transformations. The affine transformation,<br>which includes scaling, rotation, and translation, appears to be a more versatile and effective<br>method for aligning these spaces. This finding suggests that the latent spaces of image<br>autoencoders are related by a broader class of transformations, including those that are not strictly<br>orthogonal, which is a significant insight into the nature of these transformations and their<br>implications for model stitching and zero-shot learning. |
| Relevant page (truncated): |
|  |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Does your dataset consider the spanning-page answer setting?
- Is there a dataset structure parsing quality checking procedure?
- Is there any analysis or comparison before and after human checking automatically generated QA pairs?
- Why does the number of document pages per document look wired, especially for academic papers? It might be too long for an academic paper.
soundness: 3
strengths: - Research problem is challenging and necessary for this direction and various domain applications.
- A new dataset is proposed for multimodal long document understanding.
- A LLM-based auto-evaluating framework is introduced.
summary: This paper introduces a dataset for long document understanding challenges with an automatic evaluation approach. The authors also propose a retrieval-aware tuning framework to mitigate the limitations of current LLMs dealing with multimodal long documents.
weaknesses: - The scope of this paper limited the question only focusing on specific pages ignoring the more natural cases of answers distributed spanning pages. And the author mentioned the in-depth but there is no supporting analysis and results to show why the dataset shows more in-depth. 
- The dataset generation workflow uses off-the-shelf tools and models to extract the document structure which should be verified as the accumulated errors may occur when moving to the automatic QA generation stage. 
- More dataset analyses are expected including question length and focusing topics. Simple statistics can not show more insight of your datasets. 
- The proposed evaluation metrics may need more detailed analysis to show robustness. Current average weighting looks too simple ignoring the difference between specific models dealing with specific types of questions. Some penalty or reward terms may need to be considered.
- As the metrics are unexplored the results may not be comprehensive and reliable. Lack of quantitative analysis show different domain, question types performance.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: All clear to me
soundness: 3
strengths: Useful new eval dataset, training dataset, eval framework and interesting model for multimodal RAG-QA on long docs.
summary: The paper presents M-LongDoc, a new benchmark for evaluating the ability of large multimodal models to understand and answer open-ended questions over lengthy and diverse documents containing text, figures, and tables. M-LongDoc comprises 851 samples across academic, financial, and product domains, featuring documents significantly longer and more structurally complex than those in existing benchmarks. The authors also proposes a novel retrieval-aware tuning approach that specifically trains models to handle potentially irrelevant retrieved content, leading to a 4.6% relative improvement in answer correctness compared to baseline open-source models. Lastly, the authors contribute a large-scale training corpus of 10,070 samples and an automated evaluation framework based on a committee of multimodal judges to assess the correctness of open-ended solutions.
weaknesses: This paper doesn't really have any major weaknesses. In particular, the paper presents its contribution as being primarily a dataset paper, so there's understandably not much novelty with the models.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How would direct text extraction and retrieval perform as an alternative to solve the problem? What are the performance on text-only questions? What are the performance on multimodal questions?

2.  Retrieval tuning appears to yield only marginal performance gains. What specific challenges in retrieval are contributing to this limited improvement, and how do they align with the unique requirements of M-LongDoc?
soundness: 3
strengths: 1. M-LongDoc provides a new benchmark. It is different from existing multimodal document visual question answering benchmarks, such as MP-DocVQA and DUDE, which is much longer (210 pages in average), and they are all open-ended questions (not extractive QA or short answer QA). 

2.  The paper proposes a retrieval-aware tuning to improve multimodal model performance on M-LongDoc benchmarks, a strategy that could benefit applications requiring nuanced document comprehension.
summary: The paper introduces M-LongDoc. M-LongDoc is a novel benchmark dataset to evaluating multimodal long document (210 pages in average) understanding. M-LongDoc features 851 samples that challenge existing multimodal models / systems to answer open-ended questions requiring in-depth understanding of complex documents (including financial reports and academic papers). 

Besides the benchmark datasets, the paper offers a retrieval-aware tuning framework to explore the solutions to solve the problem. The retrieval-augmented tuning approach improves model performance by guiding attention to relevant content while ignoring distractions.
weaknesses: 1. All questions in the benchmark are synthetically generated by multimodal LLMs, which may limit the benchmark's reflection of real-world scenarios. Human annotations are not involved in the benchmark creation process.

2. The dataset's scale is relatively modest (only 852 samples), potentially insufficient for capturing a wide range of perspectives and real-world scenarios.

3. Evaluation relies on proprietary LLMs, introducing potential variability due to different checkpoints or versions. 

4. Some related works are missing [1], the differences are not discussed. 

[1] DocBench: A Benchmark for Evaluating LLM-based Document Reading Systems

Meta-review: **Summary:**
The paper introduces M-LongDoc, a benchmark for evaluating multimodal models on lengthy documents containing text, figures, and tables, with open-ended questions requiring deep analysis. It proposes a retrieval-aware tuning framework to improve model performance by training with both relevant and distracting content. The framework show reasonable improvement in correctness, addressing multimodal biases and challenges in long document understanding​

**Strength:**
- It tackles a challenging and necessary research problem with applications across various domains.
- It introduces a new dataset, M-LongDoc, for multimodal long document understanding, having lengthy pages per document with open-ended questions.
- It provides valuable resources, including a new evaluation dataset, training dataset, evaluation framework, and multimodal RAG-QA model.

**Weakness:**
- Questions focus on specific pages, ignoring answers spanning multiple pages; lacks evidence to support claims of in-depth analysis.
- The average weighting approach is overly simplistic, ignoring model-specific differences and lacking penalty/reward mechanisms.
- Evaluation depends on proprietary LLMs, introducing variability due to differing checkpoints or versions.
- Missing discussion of some related works and insufficient differentiation from them.

Decision: Reject