M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework

Authors: Yew Ken Chia, Liying Cheng, Hou Pong Chan, CHAOQUN LIU, Maojia Song, Mahani Aljunied, Soujanya Poria, Lidong Bing
Primary area: datasets and benchmarks
Keywords: multimodal, document, benchmark, retrieval

Abstract:
The ability to understand and answer questions over documents can be useful in many business and practical applications. However, documents often contain lengthy and diverse multimodal contents such as texts, figures, and tables, which are very time-consuming for humans to read thoroughly. Hence, there is an urgent need to develop effective and automated methods to aid humans in this task. In this work, we introduce M-LongDoc, a benchmark of 851 samples, and an automated framework to evaluate the performance of large multimodal models. We further propose a retrieval-aware tuning approach for efficient and effective multimodal document reading. Compared to existing works, our benchmark consists of more recent and lengthy documents with hundreds of pages, while also requiring open-ended solutions and not just extractive answers. To our knowledge, our training framework is the first to directly address the retrieval setting for multimodal long documents. To enable tuning open-source models, we construct a training corpus in a fully automatic manner for the question-answering task over such documents. Experiments show that our tuning approach achieves a relative improvement of 4.6% for the correctness of model responses, compared to the baseline open-source models.

Decision: reject
Reviewer scores: 5, 8, 5

Reviews and discussion:
<thread>
creation date:
1730261633554

thread_type:
official_review

<thread_object>
creator: reviewer_t1tm

<content>
summary:
The paper introduces M-LongDoc. M-LongDoc is a novel benchmark dataset to evaluating multimodal long document (210 pages in average) understanding. M-LongDoc features 851 samples that challenge existing multimodal models / systems to answer open-ended questions requiring in-depth understanding of complex documents (including financial reports and academic papers). 

Besides the benchmark datasets, the paper offers a retrieval-aware tuning framework to explore the solutions to solve the problem. The retrieval-augmented tuning approach improves model performance by guiding attention to relevant content while ignoring distractions.

strengths:
1. M-LongDoc provides a new benchmark. It is different from existing multimodal document visual question answering benchmarks, such as MP-DocVQA and DUDE, which is much longer (210 pages in average), and they are all open-ended questions (not extractive QA or short answer QA). 

2.  The paper proposes a retrieval-aware tuning to improve multimodal model performance on M-LongDoc benchmarks, a strategy that could benefit applications requiring nuanced document comprehension.

weaknesses:
1. All questions in the benchmark are synthetically generated by multimodal LLMs, which may limit the benchmark's reflection of real-world scenarios. Human annotations are not involved in the benchmark creation process.

2. The dataset's scale is relatively modest (only 852 samples), potentially insufficient for capturing a wide range of perspectives and real-world scenarios.

3. Evaluation relies on proprietary LLMs, introducing potential variability due to different checkpoints or versions. 

4. Some related works are missing [1], the differences are not discussed. 

[1] DocBench: A Benchmark for Evaluating LLM-based Document Reading Systems

questions:
1. How would direct text extraction and retrieval perform as an alternative to solve the problem? What are the performance on text-only questions? What are the performance on multimodal questions?

2.  Retrieval tuning appears to yield only marginal performance gains. What specific challenges in retrieval are contributing to this limited improvement, and how do they align with the unique requirements of M-LongDoc?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730596306987

thread_type:
official_review

<thread_object>
creator: reviewer_jfm1

<content>
summary:
The paper presents M-LongDoc, a new benchmark for evaluating the ability of large multimodal models to understand and answer open-ended questions over lengthy and diverse documents containing text, figures, and tables. M-LongDoc comprises 851 samples across academic, financial, and product domains, featuring documents significantly longer and more structurally complex than those in existing benchmarks. The authors also proposes a novel retrieval-aware tuning approach that specifically trains models to handle potentially irrelevant retrieved content, leading to a 4.6% relative improvement in answer correctness compared to baseline open-source models. Lastly, the authors contribute a large-scale training corpus of 10,070 samples and an automated evaluation framework based on a committee of multimodal judges to assess the correctness of open-ended solutions.

strengths:
Useful new eval dataset, training dataset, eval framework and interesting model for multimodal RAG-QA on long docs.

weaknesses:
This paper doesn't really have any major weaknesses. In particular, the paper presents its contribution as being primarily a dataset paper, so there's understandably not much novelty with the models.

questions:
All clear to me
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709755864

thread_type:
official_review

<thread_object>
creator: reviewer_rjzp

<content>
summary:
This paper introduces a dataset for long document understanding challenges with an automatic evaluation approach. The authors also propose a retrieval-aware tuning framework to mitigate the limitations of current LLMs dealing with multimodal long documents.

strengths:
- Research problem is challenging and necessary for this direction and various domain applications.
- A new dataset is proposed for multimodal long document understanding.
- A LLM-based auto-evaluating framework is introduced.

weaknesses:
- The scope of this paper limited the question only focusing on specific pages ignoring the more natural cases of answers distributed spanning pages. And the author mentioned the in-depth but there is no supporting analysis and results to show why the dataset shows more in-depth. 
- The dataset generation workflow uses off-the-shelf tools and models to extract the document structure which should be verified as the accumulated errors may occur when moving to the automatic QA generation stage. 
- More dataset analyses are expected including question length and focusing topics. Simple statistics can not show more insight of your datasets. 
- The proposed evaluation metrics may need more detailed analysis to show robustness. Current average weighting looks too simple ignoring the difference between specific models dealing with specific types of questions. Some penalty or reward terms may need to be considered.
- As the metrics are unexplored the results may not be comprehensive and reliable. Lack of quantitative analysis show different domain, question types performance.

questions:
- Does your dataset consider the spanning-page answer setting?
- Is there a dataset structure parsing quality checking procedure?
- Is there any analysis or comparison before and after human checking automatically generated QA pairs?
- Why does the number of document pages per document look wired, especially for academic papers? It might be too long for an academic paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733313485486

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks to all reviewers and general summary

comment:
Dear reviewers and chairs,

We are delighted to note that reviewer t1TM and rjZp felt their concerns were addressed following our clarifications and revisions, reviewer rjZp increased their score, and reviewer Jfm1 gave a very positive feedback with no major weaknesses. In general, the reviewers noted our challenging research problem (rjZp), strengths of the dataset and evaluation framework for multimodal long documents (rjZp, Jfm1), and novel retrieval-aware tuning approach (jfm1, t1TM). Reviewers t1TM and Jfm1 also noted the novelty of our dataset, which has much longer documents and open-ended answers compared to prior works.

We would like to highlight our key contributions and summarize the enhancements we have made to our submission following the insightful suggestions of the reviewers. Our unique contributions are:

1. A comprehensive baseline for in-depth multimodal understanding of individual pages in long documents, isolating text-based, figure-based, and table-based questions across diverse domains.
2. A focus on open-ended questions requiring longer answers, incorporating aspects such as analytical reasoning, technical analysis, commonsense knowledge, and mathematical reasoning.
3. An automated and reliable evaluation framework for open-ended multimodal document question answering.
4. Detailed analyses of leading models, revealing multimodal biases and practical challenges in leveraging retrieval methods.
5. A novel retrieval-aware tuning approach that improves model performance by up to 4.6%, applicable to many long multimodal document applications.

Following the insightful suggestions of the reviewers, we have made the following enhancements to revise our submission:

1. Detailed breakdown of question types and examples, covering analytical reasoning, commonsense knowledge, and more (t1TM, rjZp).
2. More detailed clarification of text-only vs multimodal results (t1TM).
3. Discussion of DocBench in related works section (t1TM).
4. Comparison showing that our questions and answers are much longer than previous multimodal document benchmarks (rjZp).

Thank you all once again for the valuable feedback and for facilitating the improvement of our work.

Best regards, Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary:**
The paper introduces M-LongDoc, a benchmark for evaluating multimodal models on lengthy documents containing text, figures, and tables, with open-ended questions requiring deep analysis. It proposes a retrieval-aware tuning framework to improve model performance by training with both relevant and distracting content. The framework show reasonable improvement in correctness, addressing multimodal biases and challenges in long document understanding​

**Strength:**
- It tackles a challenging and necessary research problem with applications across various domains.
- It introduces a new dataset, M-LongDoc, for multimodal long document understanding, having lengthy pages per document with open-ended questions.
- It provides valuable resources, including a new evaluation dataset, training dataset, evaluation framework, and multimodal RAG-QA model.

**Weakness:**
- Questions focus on specific pages, ignoring answers spanning multiple pages; lacks evidence to support claims of in-depth analysis.
- The average weighting approach is overly simplistic, ignoring model-specific differences and lacking penalty/reward mechanisms.
- Evaluation depends on proprietary LLMs, introducing variability due to differing checkpoints or versions.
- Missing discussion of some related works and insufficient differentiation from them.