SV-RAG: LoRA-Contextualizing Adaptation of  MLLMs for Long Document Understanding

Authors: Jian Chen, Ruiyi Zhang, Yufan Zhou, Tong Yu, Franck Dernoncourt, Jiuxiang Gu, Ryan A. Rossi, Changyou Chen, Tong Sun
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Large Multimodal Models

Abstract:
Multimodal large language models (MLLMs) have recently shown great progress in text-rich image understanding, yet they still struggle with complex, multi-page visually-rich documents. Traditional methods using document parsers for retrieval-augmented generation suffer from performance and efficiency limitations, while directly presenting all pages to MLLMs leads to inefficiencies, especially with lengthy ones.  In this work, we present a novel framework named **S**elf-**V**isual **R**etrieval-**A**ugmented **G**eneration (SV-RAG), which can broaden horizons of *any* MLLM to support long-document understanding. We demonstrate that **MLLMs themselves can be an effective multimodal retriever** to fetch relevant pages and then answer user questions based on these pages. SV-RAG is implemented with two specific MLLM adapters, one for evidence page retrieval and the other for question answering. Empirical results show state-of-the-art performance on public benchmarks, demonstrating the effectiveness of SV-RAG.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729434878256

thread_type:
official_review

<thread_object>
creator: reviewer_nkcg

<content>
summary:
The paper aims to solve the problem of long document understanding. It proposes a method based on the LoRA technique that uses latent embeddings from LLMs to perform evidence retrieval and question answering at same time. To demonstrate the effectiveness of the method, the paper introduces a new dataset, LoCAL-Bench, and conducts experiments using multiple benchmarks.

strengths:
1.	The paper is well-motivated, and the experiments demonstrate that the proposed method effectively addresses the challenge of long document understanding. 
2.	The structure of the LoCAL method is well-designed and compatible with multiple existing LLM models. Furthermore, sharing LLM parameters fully leverages their linguistic capabilities and enhances memory efficiency, thereby extending applicability. The proposed solution in the article is simple, effective, and insightful.
3.	The experiments are well-organized, and comparisons with multiple baselines, including LLM-based and non-LLM-based methods, are sound and the results convincing. 
4.	The paper is well-written, fluent, and highly readable.

weaknesses:
1.	The LoCAL method primarily aims to solve the problem of multi-page long document understanding. However, the LoCAL-Bench dataset does not appear to feature the characteristics of long document length, which makes it hard to examine the real performance of the method. An ideal dataset should include a greater number of document pages than existing benchmarks. The authors should clarify whether the LoCAL-Bench dataset is necessary to demonstrate the effectiveness of the proposed method.
2.	Although the method achieves comparable performance to proprietary models such as Claude-3 Opus and Gemini-1.5-Pro, these proprietary models still exhibit strong abilities in other downstream tasks. However, the generalization ability of the method remains untested, which is crucial for LLM-based methods. Additionally, some work such as Textmonkey (Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473, 2024d.) also test their performance on multiple downstream tasks. Therefore, the author should conduct the experiments on the generalization ability or explain why such experiments are unnecessary for the paper.  
3.	The evidence pages supported by the proposed method seem to be limited to five pages. When the relevant evidence pages for a question exceed five, performance may decrease. This characteristic may reduce the method's general performance.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730190452268

thread_type:
official_review

<thread_object>
creator: reviewer_ingb

<content>
summary:
This paper introduces the LoCAL framework, which aims to enhance the understanding of multi-page, visually-rich documents by large multimodal models (LMMs). The framework employs dual LMM adapters for evidence page retrieval and question answering, demonstrating state-of-the-art performance on public benchmarks. The proposed method involves using LMMs as multimodal retrievers to fetch relevant pages and answer user questions based on these pages, utilizing hidden states for efficient retrieval. The paper also introduces a new dataset, LoCAL-bench, comprising 226 documents and 471 question-answer pairs across nine domains. The results highlight the effectiveness of LoCAL.

strengths:
1.The paper introduces the LoCAL framework, which effectively broadens the capabilities of large multimodal models (LMMs) for understanding multi-page, visually-rich documents. This is a significant advancement in the field.

2. The implementation of dual LMM adapters for evidence page retrieval and question answering is a novel approach that enhances the efficiency and performance of the models.

3. The introduction of the LoCAL-bench dataset, which includes a diverse range of documents and question-answer pairs, provides a valuable resource for further research and development in this area.

weaknesses:
While the paper does mention the limitations of traditional methods using document parsers for retrieval-augmented generation, it focuses on introducing and evaluating the LoCAL framework without a direct comparison to RAG methods.

questions:
How to evaluate the quality of the LoCAL-bench dataset? Because the data is purely genereted from the GPT-4o model, although with human verificaition I still doubt its quality compared with other human-labeld datasets.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716042707

thread_type:
official_review

<thread_object>
creator: reviewer_yv7k

<content>
summary:
This paper introduces LoCAL (LoRA-Contextualizing Adaptation of Large multimodal models), a framework for extending LMMs to handle long, multi-page document understanding. The key insight is using LMMs themselves as efficient retrievers for relevant pages before performing question answering, rather than relying on traditional document parsers or trying to process entire documents at once. It implements efficient parameter sharing through dual LoRA adapters to build a dual-module architecture where a single LMM serves both as a retriever and question answerer.

strengths:
- The paper introduces an approach to multipage document understanding by combining LLMs with a retrieval mechanism tailored for multi-page, visually-rich documents. The use of dual LoRA adapters for separate retrieval and question-answering tasks is a creative adaptation that enhances the efficiency and modularity of the model.
- The paper provides an extensive set of experiments across multiple datasets, including SlideVQA, MMLongBench-Doc, DocVQA, and the newly proposed LoCAL-bench.

weaknesses:
The idea of using LMM for evidence page retrieval is interesting but not entirely novel. Previous work like ColBERT (Khattab & Zaharia, 2020) has already employed contextualized late interaction in document retrieval tasks. Moreover, using LORA to efficiently adapt LLMs for different tasks is also widely used in many scenarios. Therefore, the paper could better position its contribution by clearly delineating how LoCAL surpasses existing methods in LMM retrieval and adaptation for long documents.

questions:
- Based on the fine-tuning dataset, you trained the retrieval module using the original ColPali training data, supplemented with DocMatix-IR and PFLDocVQA. Have you noticed any performance improvements compared to the retrieval module trained only on the original ColPali training data on Table 1 benchmarks and also ViDoRe benchmark from ColPali?
- According to the ColPali paper, the authors use the BGE-M3 embedding model as the text-based baseline. Do you believe this model could significantly outperform the SBERT baseline, given that BGE-M3 is more advanced on existing benchmarks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721598407

thread_type:
official_review

<thread_object>
creator: reviewer_wtc2

<content>
summary:
It presents a framework named LoRA-Contextualizing Adaptation of Large multimodal models (LoCAL) to broaden the horizons of LMM for multi-page document understanding. 
LoCAL is implemented with two specific LMM adapters, one for evidence page retrieval and the other for question answering. 
Empirical results show state-of-the-art performance on public benchmarks, demonstrating the effectiveness of LoCAL.

strengths:
1. A novel framework named LoCAL to broaden the horizons of LMMs, where it uses intermediate LMMs hidden embedding for efficient question-based evidence page retrieval. 
2. It finetunes LMMs through dual LoRA adapters for evidence page retrieval and question answering.
3. It collects a visually-rich document QA dataset, LoCAL-bench.
4. It empirically demonstrates its effectiveness.

weaknesses:
1. This article needs to be compared with more methods, such as bge-large, NV-Embed-v2, SigLIP, ColPali.
2. Two LoRA with one model, actually there are still two models, not a real unified model.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732696125628

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response

comment:
We thank the reviewers for their positive feedback and valuable suggestions. We are pleased that they recognize the novelty and effectiveness of our framework. Below, we address common concerns regarding the proposed LoCAL Bench dataset and missing baselines. Specific questions raised by individual reviewers will be addressed separately. We will incorporate detailed revisions into the camera-ready version based on these responses.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary:**

The paper introduces LoCAL, a framework designed to enhance the ability of large multimodal models to understand long, visually rich documents. LoCAL addresses inefficiencies in traditional retrieval and direct processing methods by employing LMMs for evidence page retrieval and question answering, supported by LoRA adapters. The approach leverages hidden embeddings for question-based retrieval, showing superior performance over classical methods. The study also introduces the LoCAL-bench dataset, comprising multimodal documents from nine domains, and demonstrates state-of-the-art results on public benchmarks, highlighting its efficiency and effectiveness for long-document understanding.

**Strength:**
- Proposes a novel framework to enhance the ability of LMMs for understanding long, visually-rich documents.
- Implements dual LoRA adapters for evidence page retrieval and question answering.
- Experiments are thorough, comparing the approach with both LLM-based and non-LLM-based baselines.

**Weakness:**
- The paper needs to compare LoCAL with additional methods (e.g., bge-large, NV-Embed-v2, SigLIP, ColPali) and evaluate its generalization ability across downstream tasks
- LoCAL-Bench lacks characteristics of long documents, limiting its ability to fully evaluate LoCAL's performance for multi-page understanding.
- It focuses on introducing and evaluating the LoCAL framework without a direct comparison to RAG methods.