SV-RAG: LoRA-Contextualizing Adaptation of  MLLMs for Long Document Understanding
Authors: Jian Chen
Ruiyi Zhang
Yufan Zhou
Tong Yu
Franck Dernoncourt
Jiuxiang Gu
Ryan A. Rossi
Changyou Chen
Tong Sun
Venue: ICLR 2025 Poster (2025)
Keywords: Large Multimodal Models
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: SV-RAG enhances long-document understanding by adapting MLLMs for self-visual retrieval-augmented generation, optimizing both evidence retrieval and question answering with specialized LoRA adapters.

Abstract: Multimodal large language models (MLLMs) have recently shown great progress in text-rich image understanding, yet they still struggle with complex, multi-page visually-rich documents. Traditional methods using document parsers for retrieval-augmented generation suffer from performance and efficiency limitations, while directly presenting all pages to MLLMs leads to inefficiencies, especially with lengthy ones.  In this work, we present a novel framework named **S**elf-**V**isual **R**etrieval-**A**ugmented **G**eneration (SV-RAG), which can broaden horizons of *any* MLLM to support long-document understanding. We demonstrate that **MLLMs themselves can be an effective multimodal retriever** to fetch relevant pages and then answer user questions based on these pages. SV-RAG is implemented with two specific MLLM adapters, one for evidence page retrieval and the other for question answering. Empirical results show state-of-the-art performance on public benchmarks, demonstrating the effectiveness of SV-RAG.

Published as a conference paper at ICLR 2025
SV-RAG: LORA-CONTEXTUALIZNN ADAPTATION OF MLLMS FOR LONG DOCUMENT UNDERSTANDING
Jian Chen',, Ruiyi Zhang?, Yufan Zhou?, Tong Yu², Franck Dernoncourt Jiuxiang Gu², Ryan Rossi Changyou Chen', Tong Sun?
University at Buffalo', Adobe Research2 ruizhangeadobe.com
ABSTRACT
Multimodal large language models (MLLMs) have recently shown great progress in text-rich image understanding, yet they still struggle with complex, multi-page visually-rich documents. Traditional methods using document parsers for retrieval- augmented generation suffer from performance and efficiency limitations, while directly presenting all pages MLLMs leads to inefficiencies, especially with lengthy ones. In this work, we present novel framework named Visual Retrieval-Augmented Generation (SV-RAG) ), which can broaden horizons of any MLLM to support long-document understanding. We demonstrate that MLLMs themselves can be an effective multimodal retriever to fetch relevant pages and then answer user questions based on these pages. SV-RAG is implemented with two specific MLLMM adapters, one for evidence page retrieval and the other for question answering. Empirical results show ofate-of-the-att performance public benchmarks, demonstrating the effectiveness of SV-RAG.
INTRODUCTION
Documents serve as a critical medium for the preservation and dissemination of information, with millions produced annually. These documents are not limited to simple text; they encompass complex layouts and a variety of modalities such as text, tables, charts, and images. Visually-rich document understanding (VDU) is thus an essential and challenging area of research. Recently, Multimodal Large Language Models (MLLMs) has emerged, showcasing remarkable abilities to process and understand documents. These models span both proprietary and
domains, like GPT-4o OOenAA 2023). Gemini-1.5 (Team et al. (023). and Claude-3 among closed-soure models, and Interni M--0nM-M-44-4-2- XC2-44DDD (Dong al. (0244) Vn.erVVL-Chhl (Chen et 2023b). VLaNe-N (Liu 2024a). Mini-CPM (Hu et al. 2024). mPL.UGDDeOOl (Ye et al. 2023b), and TextMonkey (Luu al.
ctt al. 2024d) in open-source space. Their performance has been particularly notable in single-page
DU tasks demonstrated on datasets like VQA (Mathew et al. 2021), ChartQA (Masry et al.
2022) and Info (Mathew et al. 2022)..
In real-world applications, they often present documents that much longer, containing dozens
or hundreds of (Maes(Ma et al.] 2024d] Tanaka et al., 2023 Islam et al.] [003 Zhu et al.) 2021).
Addressing the understanding of such lengthy documents presents MLLMs with new challenges
(Ma et al. 2024d). One way is to utilize a classical document parser (Rausch et al.
2021 to extract
information and formulate prompt for LLM (Wang et al. 2023 Lamott al.! 2024 which is
difficult to recover the layout in prompts and suffers performance degeneration from the document parser. The other way is exploit the long context windows of large models, allowing them to take multiple pages at once. However, most of the input pages are not relevant to user requests, and efficiency will be compromised when the document contains hundreds of pages Ma al. (2024d); Islam al. 2023) or there is document collection (Tito et al. 2021).
In this work, we first retrieve evidence pages to obtain relevant information within vast and varied landscape of content. Unlike using a classical document parser, we propose using MLLMs as the information encoder, which have shown great generalization ability as they have been trained on huge text corpus. After obtaining the embedding of each page, we further utilize contextualized
"qqual contribution, work done when JC Adobe Research.

---

Published as a conference paper at ICLR 2025
late interaction for relevance scoring (Khattab & Zaharia 2020). This design shows significantly better efficiency and accuracy than using the classical document parser to extract information. Top-k pages then selected from hundreds of pages and provided to MLLMs answer user questions documents.
Question
Evidence Page
What follows content creation the flow chort?
BOOST TEAM WORKFLOW
MLLM
Simillerit--asso
Retrioal
22
MLLM LoRA-QA
Guneration
RFI ATIONSHIP BUILDING & PM
Answer
[LatA-Batriiee
Multi-page Ducument
Shared MILM
Figure I: Overview of the SV-RAG pipeline. The multi-page document and query are encoded by a customized MLLM (yellow). The most relevant page is retrieved through similarity-based matching, and a fine-tuned MLLM (blue) generates the final answer from the evidence.
Based on this design demonstrated in Figure I we introduce the SV-RAG framework for multi-page document understanding, which includes modules for evidence page retrieval and answer generation. Our contributions can be summarized as follows.
We propose novel framework named SV-RAG to broaden the horizons of MLLMs, where we use intermediate MLLMs hidden embedding for efficient question- based evidence page retrieval.
We finetune MILMS through dual LoRA adapters for evidence page retrieval and question answer- ing, respectively, enabling SV-RAG to be edge-friendly with great memory efficiency.
We collect a visually-rich document QA dataset,
comprising nine domains including magazine, flyer, newsletter, product manual, and presentations, etc. This dataset is built upon web-crawl documents, containing 226 documents and 471 question answer pairs.
We empirically show that SV-RAG, with only 4B parameters, achieves state-of-the performance on disR-Bench and four public benchmarks, rivaling Gemini- 1.5-pro on MMLongBench- Doc and demonstrating its effectiveness.
RELATED WORK
Visually-rich Document Understanding Visual Document Understanding (VDU) is the field focused on interpreting text-rich images such as tables (Zhong et al. 2019), charts (Masry et al. 2022), and webpage screenshots (Liu al. [024c] Tanaka et al./ 2021] These images are complex. featuring a mix of text and visual elements that convey abundant information (Gu ct al. 2024). To evaluate multimodal document understanding, tasks range from low-level recognition tasks, such as information extraction, to high-level cognitive tasks, such as visual question answering (Mathew et al. 2020). Models VDU are typically divided into two categories: OCR-dependent (Xu et al. 2020) and OCR-free, based on their reliance on Optical Character Recognition (OCR). OCR-dependent models are often trained to synchronize textual and visual data. For instance, UDoP (Tang et al. 2023) pre-trained to restore obscured textual and layout details using both image and partial text inputs. OCR-free approaches must include text recognition training. Dount (Kim et al. 2022) is an example of OCR-free model that focuses producing unbroken text sequences, disregarding structural details. In contrast, Pix2Struct (Lee et al.) 2023a). another OCR-free model, focuses on interpreting the structure by creating HTML DOM trees from webpage screenshots. However, this technique does not easily transfer to other image types. Our method focuses on the visual question-answeriggg task, specifically targeting questions over long documents consisting of multiple pages of multimodal information.
Multimodal Retrieval-Augmente Generation Augmenting language models with information from various knowledge sources has been found to boost their performance different NLP tasks. The Dense Passage Retriever (DPR) (Karpukhin et al. 20200) trains its retrieval mechanism with documents from within the same batch, using contrastive learning with negatively sampled examples, which enhances its capabilities in open-domain question answering. Document Screenshot Embedding (DSE) (Ma et al., (2024c) uses MLLMs as encoders for both document screenshots and queries, training through contrastive learning to achieve enhanced multimodal retrieval. Both REALM
2

---

Published as a conference paper at ICLR 2025
(Guu et al. 2020) and Aetrieval-Augmentte Generation (RAG) Gao et al. 2023b) consider the passages they retrieve as hidden variables and train the retrieval and generation components together, improving the efficiency of the eetreval-augmeee RAG, the Plug-and-play (Chen et al. 2024d) approach uses GradCAM (Selvaraju et al.) 2020) to fetch pertinent image segments corresponding given query. The MuRAG (Chen et al. 2022) model introduces a multimodal strieval-auggentee Transformer that utilizes an external multimodal memory for language generation enhancement. Unlike other approaches that retrieve information from various knowledge sources, SV-RAG focuses on retrieving relevant evidence pages from a given document. This helps MLLMs generate accurate and explainable answers based on the retrieved content. MM-GEM Ma et al. (2024a) trains an MLLM with hybrid loss for similarity computation and caption generation on natural images. contrast, our approach targets visually rich documents, using two LoRA modules to specialize in each task.
generation approach. Taking cues from textual
Multimodal Large Language Models While Large Language Models (LLMs) excel at text-only question answering (QA) (Dasigi et al. L Lee et al. 2023b), they cannot process other modalities. To enable multimodal tasks like Visual Question Answering (VQA), MLLMs transform images and videos into visual tokens that LLMs can understand. To train these MILMs, MiniGPT-4 (Zhu et al. 2023) leverages ChatGPT produce data compliant with high-quality instructions, while LLa VA (Liu et al. 2023b) prompts GPT-4 with image captions and bounding boxes. Chen et al.
2023a 2024a) have prompted OpenAI GPT-4V generate more than 1M pieces of quality data
to train MLLMs. LLaMA-Adapter (Zhang et al. 2023 Gao et al. 2023a) and mPLUG-Ow (Ye ct al. (223b) align text and image features with large-scale image-tex text pairs. InstructBLIP (Dai et al. approach. mPLUG-OW (Ye et al. 2023a () implements multi-task instruction fine-tuning with public document datasets.
2023) has restructured 13 vision-language tasks to fit an instruction-
Recent research (Liu et al. 2023a 2024a Bai et al.) 2023 Dong et al., 2024 Xu et al. 2024 Luo
ct al. 2024) improves visual encoders by increasing resolution, leading to significant advances in downstream applications but also raising memory costs, especially in multi-page tasks. TextMonkey
Liu et al. (024d) compresses visual tokens using a token resampler. Our method extends MLLMs to handle multi-page documents by retrieving relevant pages, reducing computation and distractions from long token sequences.
3 SV-RAG METHOD
Multi-page document understanding aims to answer questions related to long and complex documents containing both text and images from users. We denote a document of n-pages as a sequence of images, X = X2, }. Text token sequence of a question is denoted as {41 92........................ }n Traditional approaches that begin with a parsing step to extract content elements such as images. tables, and forms from documents, then generate answers based on these contents using LLMSs (Saad- Falcon et al., [2023] Wang et al., 2023). Here, we first consider using MILMS to handle this task and avoiding the heuristic document parsing process, where we directly convert each page into a single image. It is not desired, as most pages in a document are irrelevant to user questions and performing an evidence page retrieval can further enhance the efficiency.
We introduce SV-RAG, a method that efficiently leverages the capabilities of pre-trained MLLMs for long document question-answe answering (QA). SV-RAG can broaden the horizon of MLLMs to answer questions over long documents or document collections with hundreds of pages. This finding is based the fact that hidden states of MLLMs can be effective page representations for question- based retrieval, as shown in Section 5.6) This representation ability can be further enhanced with contrastive training using LoRA adapter, demonstrating surprising retrieval performance of MLLMs. Furthermore, we can finetune a LoRA- of QA to further enhance the performance of SV- RAG on specific domains. In summary, we first retrieve evidence pages to rank these images based on their relevance score to a given question q, then select the most relevant images, which are then fed into the MLLM to generate the answer. In this section, we introduce the SV-RAG architecture in retrieval training in Section 3.2 and dual-adapter designs in Section 3.3
Section 3.1
3.1 ARCHITECTURE
Figure presents an overview of our model architecture, which comprises two MLLM- based modules for the retrieval of evidence pages and question answers.
3

---

Published as a conference paper at ICLR 2025
TOP-3 RETE
Iregr
bo:: Iolen
Amsseer:
BANK-BASED BETRIEVAL
Thaiand Days
Next token prediction LM head
f>
col-proj
O pooo
LLM layers
AW.
f1
f
AW.
shared LLM layers
AW.
Doseritoo
the
What
130
What
SEQUENTIAL ENCODE INACE
Viston Eucoder
Vision Encoder
ENCODE GUERY
ANSWER GEMERATION
Figure 2: Model overview of SV-RAG. It contains two modules, which are finetuned using LoRA (Hu itt al. 2021), sharing the same pretrained multimodal LLM backbone. The retrieval module selects evidence pages for the other QA module, which provides responses to user questions.
Col-Retrieval Module Building on the approach introduced in ColPali (Faysse et al. 2024), we employ a modified MLLM for retrieval, comprising a vision encoder fo, a large language model (LLM) ff and a Col-projection layer fp- For an input image X, the vision encoder computes sequence of visual embeddings (X),, which are then concatenated with token embeddings y. derived from fixed text prompt: " "enecrre the image." This combined input is fed into the LLM. The projection layer fp then transforms the LLM's last hidden states into a low-dimensional feature Similarly, for an input question q. the question is first augmented into using a prompt template. Then, its token embedding Yg is processed without visual input as E, fy (y,)). Finally, late-interaction score $Li (Ey E.) is computed between the feature sequences, measuring the relevance of page image the question text. More details about scoring method is provided section 3.2
space, resulting in feature sequences that can be represented E,, (fo))
Question-Answering Module The QA module uses a classic LLaVA- -ike architecture (Liu et al. 2024b), utilizing a vision encoder fu to compute visual embeddings, which are combined with token embeddings and processed by an LLM ff- The LLM then generates text answers autoregressively through next-word prediction.
3.2 CONTEXTUALIZED LATE INTERACTION
We utilize the contextualized late interaction (Col) technique (Khattab & Zaharia 2020) to compute relevance scores for evidence retrieval. Unlike traditional single-vector encoders, such as CLIP (Radford et al. 2021), the Col technique introduces inter- inter-sqquence similarity metric called the late-interaction score, which captures more fine-grained question-image relevance. Formally, the late-interaction score between text feature sequence Ey = .......... Cy of length and visual feature sequence E,, {em..... evm. } of length m is defined as:
n max Cy: Gay jE(L...,m)
$LI SLI E.))
Σ
(1)
We use it as similarity score in contrastive learning to facilitate ranked retrieval. Specifically, we train our retrieval module to maximize the late-interaction score between a question and its corresponding evidence image, considering these as positive pairs. We then identify the most similar, but unassociated, image within the batch to form the hardest negative pair and minimize the score for this pair. Figure A.l shows a training pair example. The loss function is defined as:
L log(1 + exp(su (E, E,) (., E)))
(2)
The training process of the Col-retrieval module is summarized in Algorithm I
3.3 PARAMETER SHARING VIA DUAP-ED
To reduce memory usage, we optimize the model by sharing a single MLLM that includes both the vision encoder fu and the language model across both the retrieval and QA modules. To
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Col-retrieval training
Require:
Pre-trained MLLM {fe, Ji training batch of evidence image and question pairs
(X1,y1)) (........)))) 1: Initialize the Col-projection layer fp-
2: while not converged do
3: Get E: (x.. (1...bb
4: Get E
(1,
5:
Compute S., = E B.).
6:
Get negative image index for each y;: arg (S.)) Gradient update using loss function 2
7:
where = = Eg.
8: end while
accommodate the different tasks required by each module, insert sets of adapters into the Ji using the LoRA method (Hu et al. (02T). In the retrieval module, we use a set of adapters W, to create the retrieval-LLM. ff. For the QA module, a different set of adapters A W. is added to the fi. creating the QA-LLM, ff" In this way, we support both tasks using a single LLM and vision encoder, adding only 2% additional parameters.
VISR-BENCH
Visually-rich Document Selection About 4,000 PDF documents are crawled from the Web and contents of these documents are extracted via a document parser" We keep the document with figures and throw away text-only or scan documents. To select documents with specific types of figures, we build figure scheme that includes 19 figure types after reviewing different documents. We find some types of figures are not informative, such as logo and banner. We use the pretrained CLIP model ViT-L/14-333 (Radford et al. (02T) to perform figure classification on the extracted figures and keep out of 19 types of figures, including tables, maps, diagrams, infographics, data charts, workflows, and screenshots. After that, we also annotate the document types for all selected documents.
Question-Answer Collection Document parser returns all document elements in JSON format and the figures are saved separately as image files. We retrieve the JSON file for the document to obtain the contexts of each figure. Then combine the figures with their contexts and GPT--4o (API version 2024-02- 1--0-021-------- 15-preview) to generate QA pairs. For the GPT-4o prompts, we provide two demonstrations and ask GPT-4o generate a QA pair. In addition, we perform automatic verification using GPT-4o to ensure the quality of the generation. Specifically, we only provide the figure to GPT-4o and ask it with the generated question; if GPT-4o can answer correctly, we will keep that QA pair in the SV-RAG Bench. This heuristic filter ensures that the answers are from document figures and double-checks the correctness of generated answers.
Dataset Statistics VisR-Bench contains 226 documents and 471 human- question-answer pairs. Figure 3 shows
shows the distributions of the document types and the length distribution by document type. VisR- Bench has great diversity of documents compared to previous work (Tanaka et al. 2023 Islam et al.
2023 Ma et al. 2024d).
EXPERIMENTS
We assess the performance SV-RAG evidence page retrieval and visual question answering capabilities. We first evaluate the retrieval accuracy of the Col-retrieval module within SV-RAG and compare with several baselines on VQi (Tanaka et al.) 2023), MM-LongBench (Ma et al. 2024d), DUDE (Van Landeghem et al. 2023), DocVQA (Mathew et al. 2020 2021 and Visk-Bench. We then conduct experiments on question answering using SV-RAG and compare the results with other LMM baselines, inleuding single page and cross-page VQA. All experiments are implemented
Adohe Extract API: http:///eeeeeeeaaacoommmeeeeeeee

---

Published as conference paper at ICLR 2025
Newsletter
50
15.2%
Averags Number of Pages
Info Sheet
ttneeray
3.5%
Magazine
30
Recipe
Product Manual
Flyer
5A0
6.62
11.2%
2.66
Brochure
Instructions
Newsleter
Info Shoct Product Manual
Brochuro Irsttuctors
Flyor Redbe
Magazire linerary
Presentas
Figure 3: Distribution of document types (left) and average document lengths in each types (right).
with PyTorch and conducted on Nvidia A100 GPUs, The Col-retrieval modules are fine-tuned for epochs with batch size 32 and a learning rate of 5e-5, using the AdamW optimizer and LoRA adapters on all linear layers in the LLM. The LoRA rank is set to 32.
5.1 DATASETS
Finetuning Dataset We train our Col-retrieval modules using the original training data of ColPali (Faysse al. 2024). which includes 39,463, 10,074, 13,251, 10,000, and 45,940 question- pageee pairs filtered from DocVQA, InfoVQA (Mathew et al. 2022). TATDQA (Zhu et al. 2024), arXivQA eLi al. 2024), and synthetic data across various topics, including government reports, healthcare, artificial intelligence, and energy. We incorporated DocMatix-IR (Ma et al.) 2024b) and PFL- DocVQA Tito et al. 2023b), using GPT-4o to filter out duplicate images and unsuitable questions. The expanded dataset improved top-1 retrieval accuracy MMLongBench- - by ~1% without affecting other benchmarks. We fine-tuned our QA modules using the training split of the SIi dataset Tanaka al., 2023). The Slide VQA dataset contains 1,919 slides in the training set, 400 in the test set, and 300 in the development set, with each slide consisting of 20 pages. The training split includes 10,290 samples, each annotated with questions, answers, and corresponding evidence.
Evaluation Dataset We evaluated our method's performance on four public datasets VOA,
MMLongBench-Doc (Ma et al. 2024d), VQA (Mathew et 2021), and DUDE (Van Landeghem et al. 20oo)-along with our proposed VisR-Bench dataset. The evaluation was conducted both single-evidence (SP) and cross-evidence (MP) settings, where questions require information from either single page or multiple pages within a long document. For Doc we used 5,349 SP and 5, 5,187 MP QA pairs from the validation split. Similarly, we combined the test and dev splits of Slide VQA to form 2,995 SP and 763 MP QA pairs for evaluation. For DUDE, we evaluated 6,307 QA pairs from the validation split.
MMLonnBench-DDcc, which consists of 135 PDF documents averaging 50.4 pages (ranging from 9 to 468 pages), contains 1,081 QAs total. From these, we extracted 488 single-evidence QAs to assess the performance of MI..Ms designed for single-image tasks. Additionally, we report the results of our best persfrmeffn model across all categories in MMLongBench-DDe providing a comprchensive comparison against wfaee-ofthhe-a LMMs.
5.2 EVALUATION METRICS
We evaluate model's performance evidence retrieval and question-answering using several key metrics: Top-k Accuracy, Exact Match (EM) (Tanaka et al., (003),, Generalized Accuracy (G-Acc) (Ma et al. 2024d), Average Normalized Levenshtein Similarity (ANLS) (Biten et al. 2019), and Partial Normalized Levenshtein Similarity (PNLS) (Chen et al. 2024b). detailed explanation of
each metric be found in Appendix

---

Published as a conference paper at ICLR 2025
5.3 COMPARATIVE RETRIEVAL ACCURACY ANALYSIS
We evaluated the accuracy of the Col-retrieval module in SlideVQA, MMLongBench-Doo. SP- comparing it with the baseline methods including CLIP (ViT-L/14) (Radford et al. 202T), SigLip (so400m-par Saatmn-ptchhhaa papp-aaatthhhhhhee 14-384) Zhai et al. (2023). BM25 (Robertson et al. 2009), SBERT (Reimers Gurevych, 2019). BGE-M3
Voc, and
al. (202cc). BGF-large Xiao et al. (2023), and NV-Embed-v2 Lee et al. 2024). For encoder models, we used their text and image encoders to compute the cosine similarity between the feature of the question and the page. For text-based methods, the text content in the JIMLongBench-Dor and SV-RAG Bench datasets is extracted using a document parser to ensure higher accuracy. For SIi and SP-DocVQA, where only scanned images are available, the text is extracted using Paddle-OCR
VQi
MMLong
VisR-B
SP-Doc VQA
accuracy
topl
top5
topl
top5
topl
topS
top1
top5
Tex--based Methods
BM25
69.3
91.1
25.3
47.6
32.2
57.5
30.9
61.7
SBERT
73.0
91.0
44.7
70.2
38.8
72.1
47.4
74.0
BGE-M3
74,3
92.0
42.7
66.6
47.7
78.1
47.8
77.5
Bge-large
93.3
47.4
71.5
53.7
80.3
56.7
81.5
NV-Embed-v2
82.2
94.3
47.4
69.0
55.2
82.7
51.7
80.2
Encoder Models
CLIP
58.4
86.9
32.4
63.4
33.4
62.1
37.1
69.4
SigLip
66.2
90.1
44.9
69.4
53.2
81.3
39.3
71.9
Col-Retrieval Modules
Col-Paligemma 89.0
98.7
60.7
82.0
67.9
90.8
62.3
85.9
Vol-InternVL2
88.5
98.3
61.3
83.0
69.3
90.7
63.2
85.9
Col-Phi-3-vision
90.6
98.8
64.8
84.8
71.9
91.8
65.1
87.0
Table 1: Retrieval accuracy results on four datasets, where MMLong refers to MMLongBenchh-Doe, SV-RAG-B refers to
Bold font indicates the best model.
The results indicate that Col-retrieval outperforms all baselines, achieving more than 98% in top- retrieval accuracy on the VQi dataset, where each slide consists 20 pages. However, performance decreases on other datasets as the data become more complex and document lengths increase significantly.
5.4
MAIN RESULTS
We compared the performance our method with popular lightweight LMMs on document question answering tasks, using PaliGemma (Beyer al. 2024). Phi-3-v (Abdin et al. 2024), and InterVLL2- 4B (Chen et al. 2023b) as the backbone LMMs for both retrieval and QA modules, following the dual adapter design from Section 3.3 We fine-tuned the retrieval module using the 118,695 training question-page page pairs used in ColPali (Faysse et al., 2024). The QA module is fine-tuned using
SlideVQA's training split. We reported the original evaluation metrics used in prior works, including EM, G-Acc, and ANLS, and additionally reported PNLS, which better evaluates LLM-generated responses.
Table presents the comparison results. We first evaluate SV-RAG on single-evidence questions from SP-SlideVQA, MMMLongBenc--Doo and SP-DocVQA, where the required information is on single page. To demonstrate the iuestion-answerigg capabilities of LMMs, we include four "cheating" baselines where models are given the ground truth evidence page. Next, we test SV-RAG on cross- evidence questions from MP-SlideVOa. pages. We only test SV-RAG with Intern VL.2-BB backbone, since the other two LMM are pre-trained for single-page understanding. SV-RAG's performance is compared with classical encoder-only and encoder-decoder models, including BERT (Kenton & Toutanoval 2019), Longformer (Beltagy et al. 2020), Big Bird (Zaheer et al. 2020). T5 (Raffel et al. 2020)) Hi-VTS (Tito et al., (023a). and Mv3 (Huang et al. 2022). with results taken from the best settings in the original
and DUDE, where information spans multiple
7

---

Published as a conference paper at ICLR 2025
papers. InternVL2-8 and GPT-4o, processing all pages, serve the state-of-the of-tt--rrt baselines for open-source and proprietary multipage LMMs, respectively. We demonstrate how the limitations of the retrieval and QA modules can impact overall performance through challenging examples from the SlideV VAA dataset, as shown in Appendix C Additional comparisons with text-only baselines utilizing document parser provided in Appendix G.1
Table 2: Quantitative Results in Multi-Page QA: "#Param" refers to number of parameters. "Evidence" reports evidence setting: T (true evidence page), A (all pages), and Rk (top-k retrieved), Reported metrics include PNLS, Exact Match, Generalized Accuracy, and ANLS. indicates models with LoRA adapter on QA module. Results for all encoder/decoder models are taken from their respective papers, with "-" indicating missing not applicable results. Bold font indicates the best open-source model, excluding cheating baselines.
SP-S
MMLooggencc
SP-DocVQA
Method
#Param
Evidence
EM
PNLS
G-Ace
PNLS
ANLS
PNLS
Single SinglePPage Evidence
Cheating Baselines
PaliGemma
3B
T
37.30
0.63
23.9
0.38
0.65
0.79
Phi-3-v
4B
T
13.72
0.80
33.7
0.52
0.65
0.85
InterVLL2
4B
T
15.03
0.58
40.4
0.55
0.84
0.88
GPT-4o
T
30.59
0.84
56.8
0.62
Moltsinage MLLMs
InterVL22
8B
A
12.62
0.65
14.1
0.22
0.50
0.55
GPT-4o
A
27.28
0.81
54.5
0.57
0.69
SV-RAG Models (Proposed)
SV-RAG-PaliGemma
3B
R1
35,03
0.60
23.9
0.35
0.56
0.69
SV-RAG-PallGeemal
3B
R1
49.75
0.65
23.1
0.38
SV-RAG--Pii--visst
4B
R1
12.85
0.78
30.7
0.50
0.55
0.75
SV-RAAG-Phi-3-isiin
4B
RI
58.13
0.77
28.4
0.44
0.68
0.73
SV-RAG-IntemVL2
4B
R5
16.40
0.58
0.48
0.76
SV-RAG--mtemVLL22
4B
R5
45.07
0.77
34.0
0.49
0.71
0.75
Cross-Pase Page Evidence
MP-Slide
MP-DocVQA
DUDE
Method
#Param
Evidence
EM
PNLS
ANLS
PNLS
ANLS
PNLS
Racader/Decnder models
BERT-Large
334M
0.53
0.25
Longformer
148M
0.55
0.27
Big Bird
131M
0.58
0.26
T5-Base
223M
0.51
0.42
LayoutLMv3
0.55
0.20
Hi-VT5
316M
0.62
0.23
Mull-Image MLLMs
V1.
17.04
0.53
0.68
0.75
0.37
0.56
GPT-4o
A
16.09
0.73
0.67
0.79
0.54
0.70
SV-RAG Models (Pmyusedd)
SV-RAG--ntemWI.2
4B
R5
24.25
0.61
0.70
0.76
0.36
0.57
SV-RAG-IntemmLn
4B
R5
31.98
0.59
0.71
0.76
0.45
0.54
Retrieval vs Multipage We observe SV-RAG consistently outperforms Intern VL2-8B, across vari- ous settings. The primary issue with LMMs that long documents are transformed into excessively long visual token sequences, leading to significant memory burdens, reported later in section 5.5 In datasets like MMLong and DocVQA, some documents exceed hundreds of pages. causing out-of-memont errors, even servers with 8 x A A100 (80GB) GPUs. In such cases, we assigned a zero score in our experiments. In contrast, GPT-4o exhibits strong multi-page reasoning capabilities. However, the accuracy of the cheating baseline slightly surpasses that of using all pages, as providing only the evidence pages helps GPT-4o avoid distractions from irrelevant information in the longer context. Moreover, SV-RAG with Intern VL2-4B backbone perform slightly better than the one with Phi-3-vision backbone on MMLongBench- Doc and SP-DocVQA possibly due to the improvement in retrieval accuracy by using top-5 pages, which is crucial for longer documents.
8

---

Published as a conference paper at ICLR 2025
Impact of Fine-tuning We observe that SV-RAG QA modules with PaliGemma and Intern backbones show a significant increase in EM on the SIideVQA dataset, surpassing their cheating baselines after fine-tuning VQA. The model with the Phi-3-vision backbone shows notable improvements Exact Match (EM) scores without gains in PNLS, suggesting that fine-tuning primarily enhanced the model's attention and answer formatting. This could be because the pre- trained model was already optimized for these question types. Nevertheless, as shown in Figure D.I empirically find that fine-tuning still improves answering performance, However, we notice on MMLLongBench-De
performance drop for fine-tune sune DE-RAG-Phi-3-- fine-tuning can harm LLM generalization. A similar trend is seen with the Inter VL2-4B backbone on the DUDE dataset.
indicating that
Comparison with SOTA LMMs Finally, we present the complete results of SV-RAG-Intern V1.2- 4B on the MMLongBench- Doc dataset to highlight the advantages of our method. As shown in T our model, with only billion parameters, outperforms all open-source LMMs and achieves performance comparable proprietary models such as Claude-3 Opus and Demini-1.5-PPre
Evidence Source
Evidence Page MUL
Method
MParam
ACC
F1
LAY
CHA
TAB
FIG
UNA
Openssource Models
DeepSeek-VI. PeepSeek-VLLChat
6.5
5.2
7.6
5.2
7.0
12.8
5.4
Idefics2
8B
9.0
10.6
4.8
4.1
8.7
7.7
7.2
5.0
7.0
6.8
MiniCPM-llama3-Va.
8B
11.9
10.8
5.1
5.9
12.2
9.5
9.5
4.5
8.5
8.6
Intern M-XC2-4KH
9.9
14.3
6.3
13.0
12.6
7.6
9.6
10.3
9.8
mPLU'G-DocOwl
8.1B
8.2
8.4
2.0
3.4
9.9
7.4
6,4
6.2
6.9
6.3
Qwen-VT.en Chattt
9.6B
9.0
2.2
6.9
5.2
5.4
Monkey-Chan
9.8B
6.8
7.2
3.6
6.7
9.4
6.6
6.2
6.2
6.2
5.6
CoLVLL2--LMMAA-Cht
2.7
3.2
6.9
3.9
4.0
InemmVI-Chat-v15.
26B
14.0
16.2
7.1
10.1
16.6
14.9
12.2
17.5
14.6
13.0
EMU2-Chat
37B
6.1
9.7
2.6
3.8
7.7
5.7
6.1
16.5
8.3
5.5
SV-RAG Models (Proposed)
SV-RAG-IntemVL2 (RS)
4B
26.5
18.8
22.3
19.6
23.6
33.2
13.1
12.4
22.2
22.8
SV-RAG-IntemVL.2 (R5)
26.3
22.1
20.7
25.2
34.0
10.6
15.7
24.2
Proprietury Modeis
Claude-3 Opus
24.7
13.0
17.1
25.6
13.8
Gemini-1.5-p
21.0
17.6
6.9
14.5
15.2
21.1
11.1
69.2
28.2
20.6
GPT-4V
28.3
28.2
32.4
26.8
36.4
27.0
31.2
32.4
31.2
GPT-4o
46.3
46.0
45.3
50.0
44.1
54.5
41.5
20.2
42.8
44.9
Table Performance various models on MMMoongBenh--Doo Questions are categorized in two ways: (1) by evidence source type-text (TXT), layout (LAY), chart (CHA), table (TAB), and image (IMG); and (2) by evidence piges-single-page (SIN), cross-page (MUL), and unanswerable (UNA). Models using LoRA adapters fine-tuned on Slide for the QA module are marked with t Bold font indicates the best open-soneee open-source model. The results baseline models are adopted from the original MMLongBench-DD paper Ma al. (2024d)
5.5 EFFICIENCY OF DIFFERENT MODELS
To evaluate the efficiency of SV-RAG, we conducted experiments on the SlideVQA dataset, which has 20 pages per question with a resolution of 1024x768. We recorded peak GPU memory usage and time costs for retrieval and QA modules separately. The GPU memory is manually recorded using the nvidia-see command, which tends to report higher numbers than the actual memory required by the application due to overhead and memory management processes. We tested backbones including PaliGemma, Phi-3-v, and Intern VL2-4B, all equipped with LoRA adapters. Since PaliGemma and Phi-3-v are single-page models, we used top-1 retrieved image as input. Intern VL2-4B, however, supports multi-image input, allowing to test with the top-1, 5, and 12 retrieved images.
As shown in Table the QA module's ' memory consumption increases with the number of evidence pages, with 13 images (1024x768) exceeding the 80GB limit A100 GPU resulting in out-of- memory error. In contrast, the retrieval module maintains low memory usage, as SV-RAG processes pages independently. with costs equivalent to single-page reasoning. Although multi-evidence QA

---

Published as a conference paper at ICLR 2025
Retrieval
QA
SV-RAG-Backhone
Page
Time
Mem
Time
Mem
Paligemma
R1
2.3
9.2
1.0
12.4
Phi-3-vision
R1
4.1
11.6
0.9
12.9
Intern VL.-44
R1
14.2
1.4
14.6
Intern InternVL2-44 VL2-4B
R5
9.2
14.2
2.8
40.8
Intern InternVL2-44 VL2-4B
R12
9.2
14.2
4.1
76.4
Table 4: Time (s) cost and Peak GPU memory (GB) cost of SV-RAG models with different backbones.
requires more memory, SV-RAG remains efficient and compact, making it well-suited for answering questions from fewer evidence pages in essorcee-oonstrinee environments. This demonstrates SV-RAG's ability to balance performance and resource usage, ensuring scalability across diverse deployment scenarios. Additional results on the retrieval efficiency are presented in Table F.1
5.6 ABLATION
In our experiment, we use the hidden states from the last transformer layer (index 31) as the feature sequence. However, LLMs consist multiple transformer layers, each encoding different types of information. To assess the impact of layer selection, we conduct an ablation study on the hidden states used to compute the late interaction score [T. Given the high computational cost of training col-retrieval module across all layers, we instead evaluate top-1 accuracy the MMLongBenee dataset using hidden states from different layers the vision model with pre-trained weights.
Top-1 Retrieval Accuracy
60
Fine-tured
55
40
13 15
17
27
31
Index of hidden states layer
Figure 4: Top-1 retrieval accuracy MMLLngBench--Doc using different hidden states across all layers of Phi-3-vision.
Figure 4 shows that the hidden states of the 21th layer yield the highest accuracy. After fine- fine-tuning model with hidden states from this layer, we observed improved accuracy compared to using hidden states of the final layer. In particular, using hidden states from earlier layers can significantly reduce computational costs, enabling faster retrieval during inference.
CONCLUSIONS
In this paper, we propose SV-RAG, a lightweight MILMS for visually-rich document understanding. SV-RAG has unique design to facilitate multi-page document understanding using dual LoRA adapters. The research highlights that small open-source models are great at processing multipage documents and underscored the importance of efficient retrieval mechanisms in filtering irrelevant pages. Furthermore, we collect the bench dataset for document understanding, and empirical results benchmarks demonstrated the effectiveness of SV-RAG. We hope these findings provide valuable insights for optimizing lightweight MLLMs, aiming to improve accuracy and efficiency in visually-rich document understanding.
10

---

Published as a conference paper at ICLR 2025
7 LIMITATIONS
SV-RAG is the first MLLM that can perform visual-retrieval generation for document question answering using a single model. However, it still requires computational resources for training and inference, which may limit its practical applicability in resource-constrrines environments. SV-RAG should be mobile friendly, as it only requires a single base model. This base model can be Phi-3-Silica within MS operating systems or an Apple on-device model within Apple IOS 18. A routing mechanism in Apple Intelligence can better balance computational cost and performance. However, our experiments not performed on these real-world devices, which are necessary for pushing forward document intelligence.
8 ETHICS STATEMENT
The VisR-Bench dataset was curated with careful consideration of ethical and legal concerns. All documents are sourced from publicly available data with licenses explicitly permitting research use. To ensure data integrity and compliance, we provide links to the original sources instead of distributing the documents. Additionally, all QA pairs have been manually reviewed to exclude harmful content and personally identifiable information. The dataset does not expose sensitive user data, and experimental results are reported as aggregate statistics to prevent information leakage while ensuring reproducibility. These measures uphold ethical and legal standards while supporting responsible Al research.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: highly capable language model locally on your phone. arXiv preprint arXiv:2404.14211 14219, 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprint ar ar38v:230.. 12966, 2023.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05 /50, 2020.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726 2024.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawa- har, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4291-4301, 2019.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gptdv-synthesized data for a lite vision-language model. arXiv preprint arXiv: 11644, 2024a.
Jian Chen, Ruiyi Zhang, Yufan Zhou, Ryan Rossi, Jiuxiang Gu, and Changyou Chen. Mmr: Evaluat- ing reading ability of large multimodal models. arXiv preprini arXiv:2408. 14594, 2024b.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality. multi caltiigranularityy text embeddings through self-knowledge distillation, 2024c.
Jiaxing Chen, Yuxuan Liu, Dehu Li, Xiang An, Ziyong Feng, Yongle Zhao, and Yin Xie. Plug-and-play grounding of reasoning in multimodal large language models. arXiv preprint arXiv:2403.19322. 2024d.
Lin Chen al. Sharegpt4v: Improving large multi-modal models with better captions, 2023a.
11

---

Published as a conference paper at ICLR 2025
Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W Cohen. Murag: Multimodal retrieval-augmented generator for open question answering over images and text. arXiv preprint arXiv:2210.02928, 2022.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo. Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic lisual-linguisic tasks. arXiv preprins arXiv:2312. 14238, 2023b.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purposs. purpose vision-language models with instruction tuning, 2023.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-sscking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011, 2021.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Quyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Interoln-xcomposerz: Mastering free-form text-image composition and comprchension vision- -ainnll large model. arXiv preprint 1rXiv22401116400 2024.
Manuel Faysse, Hugues Sibille, Tony Wu, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Col- pali: Efficient document retrieval with vision language models. arXiv preprint arXiv:2407.01449, 2024.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. adaater v2: Parameter-efficient visual instruction model, 2023a.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Setrieval-augment generation for large language models: A survey. arXiv preprint arXiv:2312. 10997, 2023b.
Jiuxiang Gu, Xiangxi Shi, Jason Kuen, Lu Qi, Ruiyi Zhang, Anqi Liu, Ani Nenkova, and Tong Sun. ADOPD: large-scale document page decomposition dataset. The Twelfth International Conference on Learning Representations, 2024. URL
X1ptaXp0Ya
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06399 2024.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutimv3: Pre-training for document ai with unified text and image masking. In Proceeuliggs of the 30th ACM International Conference on Multimedia, pp. 4083-4091, 2022.
Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Fi- nancebench: new benchmark for financial question answering. arXiv preprint arXiv:2311.1199 2023.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu. Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04900 2020.
12

---

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Published as a conference paper at ICLR 2025
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, pp. 2. Minneapolis, Minnesota, 2019.
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pp. 39-48, 2020.
Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document under- standing transformer. Computer Vision ECCV 2022, pp. 498 517, 2022. ISSN 1611-3349, doi: doi
10.100779788-3-011119955 22. URL http:///o.....oooo/o////00/00110
29
Marcel Lamott, Yves-Noel Weweler, Adrian Ulges, Faisal Shafait, Dirk Krechel, and Darko Obradovic. Lapdoc: Layout-aware prompting for documents. arXiv preprint arXiv:2402.098AA 2024.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training iims as generalist embedding models. arXiv preprint 2rXiv:2405.17433 2024.
Kenton Lee, Mandar Joshi, Iulia Raluca Ture, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning, pp. 18893-18912. PMLR, 2023a.
Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-in Lee, and Moon- Lee. Qasa: advanced question answering on scientific articles. In International Conference on Machine Learning, pp.
PMLR, 2023b.
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng. Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vasion-language models. arXiv preprint 4rXiv:2403.00231, 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. nexta-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https://
Ilava-vl. www.vvl..gituuuii.ooe wtt.tha.ooooooog/2200000-1111110 30- 11a--aaaaaaae
neural information processing systems, 36, 2024b.
Junpeng Liu, Yifan Song. Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? arXiv preprint arXiv:2404.05955 2024c.
Yuliang Liu, Biao Yang. Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473 2024d.
Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: -ixtture-of-resolutie adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003, 2024.
Feipeng Ma. Hongwei Xue, Guangting Wang. Yizhou Zhou, Fengyun Rao, Shilin Yan, Yueyi Zhang, Siying Wu, Mike Zheng Shou, and Xiaoyan Sun. Multi-modal generative embedding model. arXiv preprint X405 19333, 2024a.
13

---

Published as a conference paper at ICLR 2025
Sheng-Chieh Xueguang Ma, Sheng- Chich Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multimodal retrieval via document screenshot embedding. arXiv:2406. 1/255, 2024b.
Xueguang Ma, Sheng-Chich Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multimodal retrieval via document screenshot embedding, arXiv preprint arXiv:2406. 11251, 2024c.
Yubo Ma, Yuhang Zang. Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations. arXiv preprint arXiv:2407.01523, 2024d.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqu: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: A dataset for vqa on document images, 2020.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
OpenAL. Gpt-4 technical report, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8766 8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Johannes Rausch, Octavio Martinez, Fabian Bissig, Ce Zhang, and Stefan Feuerriegel. Docparser: Hierarchical document structure parsing from renderings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1328-43334 2021.
Nils Reimers and Iryna Gurevych. Sentence- Sentence embeddings using siamese bert-networks.
In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. As- sociation for Computational Linguistics, 11 2019. URL https:/ts////xx http:s///aaxxvoog//bb/////////// 1908.10084
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4):333-38) 2009.
Jon Saad-Falcon, Joe Barrow, Alexa Siu, Ani Nenkova. Ryan A Rossi, and Franck Dernnncour. Pdftriage: question answering over long, structured documents. arXiv preprint a7X8v22309.08872 2023.
Peter H Sellers. The theory and computation of evolutionary distances: pattern recognition. Journal of algorithms, 1(4):359-373, 1980.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via gradient-based localiza- tion. International journal of computer vision, 128:336-359, 2020.
Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmre: Machine reading comprehension on document images. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 18878-13888, 2021.
Ryota Tanaka, Kyosuke Nishida. Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 13636-13645, 2023.
14

---

Published as a conference paper at ICLR 2025
Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. Unifying vision, text, and layout for universal document processing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19254-19264, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Document collection visual question answering. In Document Analysis and Recognition-ICAAR 2021: 16th International Conference, Lausanne, Switzerland, September 5-10, 2021, Proceedings, Part II 16. pp. 778-792. Springer, 2021.
Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Hierarchical multimodal transformers for multipage doevqa. Puttern Recognition, 144: 109834, 2023a.
Rubèn Tito, Khanh Nguyen, Marlon Tobaben, Raouf Kerkouche, Mohamed Ali Souibgui, Kangsoo Jung, Lei Kang, Ernest Valveny, Antti Honkela, Mario Fritz, and Dimosthenis Karatzas. Privacy- aware document visual question answering. arXiv preprint arXiv:2312. 10108, 2023b.
Jordy Van Landeghem, Rubèn Tito, Lukasz Borchmann, Michat Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Jurkiewicz, Mickaël Coustaty, Bertrand Anckaert, Ernest Valveny, et al. Docu- ment understanding dataset and evaluation (dude). In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9528-19540, 2023.
Wenjin Wang, Yunhao Li, Yixin Ou, and Yin Zhang. Layout and task aware instruction prompt for zero-shot document image question answering, 2023.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: Imm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403. /211//7000 1703, 2024.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. pp. 1192-1200, 2020.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:23/0.05126, 2023a.
Qinghao Ye et al. mplug-owl: Modularization empowers large language models with multimodality, 2023b.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Yang, al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 13:17283-172977 2020.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975-11986. 2023.
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16199, 2023.
Xu Zhong, Elaheh ShaficiBavani, and Antonio Jimeno Yepes. Image-based table recognition: data, model, and evaluation. arXiv preprint 1911. 2019.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023.
15

---

Published as a conference paper at ICLR 2025
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. arXiv preprint arXiv:2105.07624, 2021.
Fengbin Zhu, Chao Wang, Fuli Feng, Zifeng Ren, Moxin Li, and Tat-Seng Chua. Doe2SoarGraph: Discrete reasoning over visually-rich table-text documents via semantic-oriented hierarchical graphs. Nicoletta Calzolari, Min- Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.). Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation CORIGCCLINGGGGGOOO 2024), pp. 51 5119-513,, Torino, Italia, 2024, ELRA and ICCL. URL htttp:://ocoaathhoooooooor////// Lrec Trec-main..nnee 456
16


Reviews:
[{"review_id": "P1Zs5yrIJj", "reviewer": "Reviewer_WtC2", "summary": "It presents a framework named LoRA-Contextualizing Adaptation of Large multimodal models (LoCAL) to broaden the horizons of LMM for multi-page document understanding. \nLoCAL is implemented with two specific LMM adapters, one for evidence page retrieval and the other for question answering. \nEmpirical results show state-of-the-art performance on public benchmarks, demonstrating the effectiveness of LoCAL.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A novel framework named LoCAL to broaden the horizons of LMMs, where it uses intermediate LMMs hidden embedding for efficient question-based evidence page retrieval. \n2. It finetunes LMMs through dual LoRA adapters for evidence page retrieval and question answering.\n3. It collects a visually-rich document QA dataset, LoCAL-bench.\n4. It empirically demonstrates its effectiveness.", "weaknesses": "1. This article needs to be compared with more methods, such as bge-large, NV-Embed-v2, SigLIP, ColPali.\n2. Two LoRA with one model, actually there are still two models, not a real unified model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UDe0Zzk2ho", "reviewer": "Reviewer_yV7k", "summary": "This paper introduces LoCAL (LoRA-Contextualizing Adaptation of Large multimodal models), a framework for extending LMMs to handle long, multi-page document understanding. The key insight is using LMMs themselves as efficient retrievers for relevant pages before performing question answering, rather than relying on traditional document parsers or trying to process entire documents at once. It implements efficient parameter sharing through dual LoRA adapters to build a dual-module architecture where a single LMM serves both as a retriever and question answerer.", "questions": "- Based on the fine-tuning dataset, you trained the retrieval module using the original ColPali training data, supplemented with DocMatix-IR and PFLDocVQA. Have you noticed any performance improvements compared to the retrieval module trained only on the original ColPali training data on Table 1 benchmarks and also ViDoRe benchmark from ColPali?\n- According to the ColPali paper, the authors use the BGE-M3 embedding model as the text-based baseline. Do you believe this model could significantly outperform the SBERT baseline, given that BGE-M3 is more advanced on existing benchmarks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces an approach to multipage document understanding by combining LLMs with a retrieval mechanism tailored for multi-page, visually-rich documents. The use of dual LoRA adapters for separate retrieval and question-answering tasks is a creative adaptation that enhances the efficiency and modularity of the model.\n- The paper provides an extensive set of experiments across multiple datasets, including SlideVQA, MMLongBench-Doc, DocVQA, and the newly proposed LoCAL-bench.", "weaknesses": "The idea of using LMM for evidence page retrieval is interesting but not entirely novel. Previous work like ColBERT (Khattab & Zaharia, 2020) has already employed contextualized late interaction in document retrieval tasks. Moreover, using LORA to efficiently adapt LLMs for different tasks is also widely used in many scenarios. Therefore, the paper could better position its contribution by clearly delineating how LoCAL surpasses existing methods in LMM retrieval and adaptation for long documents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1AIwSjzaKv", "reviewer": "Reviewer_inGb", "summary": "This paper introduces the LoCAL framework, which aims to enhance the understanding of multi-page, visually-rich documents by large multimodal models (LMMs). The framework employs dual LMM adapters for evidence page retrieval and question answering, demonstrating state-of-the-art performance on public benchmarks. The proposed method involves using LMMs as multimodal retrievers to fetch relevant pages and answer user questions based on these pages, utilizing hidden states for efficient retrieval. The paper also introduces a new dataset, LoCAL-bench, comprising 226 documents and 471 question-answer pairs across nine domains. The results highlight the effectiveness of LoCAL.", "questions": "How to evaluate the quality of the LoCAL-bench dataset? Because the data is purely genereted from the GPT-4o model, although with human verificaition I still doubt its quality compared with other human-labeld datasets.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The paper introduces the LoCAL framework, which effectively broadens the capabilities of large multimodal models (LMMs) for understanding multi-page, visually-rich documents. This is a significant advancement in the field.\n\n2. The implementation of dual LMM adapters for evidence page retrieval and question answering is a novel approach that enhances the efficiency and performance of the models.\n\n3. The introduction of the LoCAL-bench dataset, which includes a diverse range of documents and question-answer pairs, provides a valuable resource for further research and development in this area.", "weaknesses": "While the paper does mention the limitations of traditional methods using document parsers for retrieval-augmented generation, it focuses on introducing and evaluating the LoCAL framework without a direct comparison to RAG methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IAduPk4V1J", "reviewer": "Reviewer_nKcG", "summary": "The paper aims to solve the problem of long document understanding. It proposes a method based on the LoRA technique that uses latent embeddings from LLMs to perform evidence retrieval and question answering at same time. To demonstrate the effectiveness of the method, the paper introduces a new dataset, LoCAL-Bench, and conducts experiments using multiple benchmarks.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well-motivated, and the experiments demonstrate that the proposed method effectively addresses the challenge of long document understanding. \n2.\tThe structure of the LoCAL method is well-designed and compatible with multiple existing LLM models. Furthermore, sharing LLM parameters fully leverages their linguistic capabilities and enhances memory efficiency, thereby extending applicability. The proposed solution in the article is simple, effective, and insightful.\n3.\tThe experiments are well-organized, and comparisons with multiple baselines, including LLM-based and non-LLM-based methods, are sound and the results convincing. \n4.\tThe paper is well-written, fluent, and highly readable.", "weaknesses": "1.\tThe LoCAL method primarily aims to solve the problem of multi-page long document understanding. However, the LoCAL-Bench dataset does not appear to feature the characteristics of long document length, which makes it hard to examine the real performance of the method. An ideal dataset should include a greater number of document pages than existing benchmarks. The authors should clarify whether the LoCAL-Bench dataset is necessary to demonstrate the effectiveness of the proposed method.\n2.\tAlthough the method achieves comparable performance to proprietary models such as Claude-3 Opus and Gemini-1.5-Pro, these proprietary models still exhibit strong abilities in other downstream tasks. However, the generalization ability of the method remains untested, which is crucial for LLM-based methods. Additionally, some work such as Textmonkey (Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473, 2024d.) also test their performance on multiple downstream tasks. Therefore, the author should conduct the experiments on the generalization ability or explain why such experiments are unnecessary for the paper.  \n3.\tThe evidence pages supported by the proposed method seem to be limited to five pages. When the relevant evidence pages for a question exceed five, performance may decrease. This characteristic may reduce the method's general performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)