VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents
Authors: Shi Yu
Chaoyue Tang
Bokai Xu
Junbo Cui
Junhao Ran
Yukun Yan
Zhenghao Liu
Shuo Wang
Xu Han
Zhiyuan Liu
Maosong Sun
Venue: ICLR 2025 Poster (2025)
Keywords: Retrieval-augmented Generation
Vision-language Models
Primary area: generative models

Abstract: Retrieval-augmented generation (RAG) is an effective technique that enables large language models (LLMs) to utilize external knowledge sources for generation.  However, current RAG systems are solely based on text, rendering it impossible to utilize vision information like layout and images that play crucial roles in real-world multi-modality documents.  In this paper, we introduce VisRAG, which tackles this issue by establishing a vision-language model (VLM)-based RAG pipeline.  In this pipeline, instead of first parsing the document to obtain text, the document is directly embedded using a VLM as an image and then retrieved to enhance the generation of a VLM. Compared to traditional text-based RAG, VisRAG maximizes the retention and utilization of the data information in the original documents, eliminating the information loss introduced during the parsing process. We collect both open-source and synthetic data to train the retriever in VisRAG and explore a variety of generation methods. Experiments demonstrate that VisRAG outperforms traditional RAG in both the retrieval and generation stages, achieving a 20–40% end-to-end performance gain over traditional text-based RAG pipeline. Further analysis reveals that VisRAG is efficient in utilizing training data and demonstrates strong generalization capability, positioning it as a promising solution for RAG on multi-modality documents. Our code and data are available at https://github.com/openbmb/visrag.

Published as a conference paper at ICLR 2025
VISRAG: VISION-BASED RETRIEVAL-AUGMENTER GENERATION ON MULTI-MODALITE DOCUMENTS
Shi Yu'r, Chaoyue Tang"*, Bokai Xu Junbo Junhao Ran Yukun Yan 't Zhenghao Liu', Shuo Wang', Xu Han', Zhiyuan Liu Maosong Sun
*Department of Computer Science and Technology. Tsinghua University
2 Modeelestt Inc. "Rice University "Northeastern University yus 2 @mails . ts tsinghua. cn
ABSTRACT
Retrieval-augmented generation (RAG) is an effective technique that enables large language models (LLMs) to utilize external knowledge sources for generation. However, current RAG systems are solely based on text, rendering it impossible utilize vision information like layout and images that play crucial roles in real- world multi-modality documents. In this paper, we introduce VisRAG, which tackles this issue by establishing a vision-language model (VLM)-based RAG pipeline. In this pipeline, instead of first parsing the document obtain text, the document is directly embedded using a VLM as an image and then retrieved enhance the generation of VLM. Compared traditional text-based RAG, VisRAG maximizes the retention and utilization the data information the original documents, eliminating the information loss introduced during the pars- ing process. We collect both open-source and synthetic data to train the retriever in VisRAG and explore a variety of generation methods. Experiments demonstrate that VisRAG outperforms traditional RAG in both the retrieval and generation stages, achieving 20-40% end-to-end performance gain over traditional text- based RAG pipeline. Further analysis reveals that VisRAG is efficient in utilizing training data and demonstrates strong generalization capability, positioning it as a promising solution for RAG multi-modality documents. Our code and data are available at http:::://// /
com/ hwtn///pnmmbb/vvce /witmmbb/virrgggg
INTRODUCTION
Trained on massive data, large language models (LLMs) have shown strong abilities in common NLP tasks using their parametric knowledge (Wei et al. 2022 Zhao et 2023 AAchiam et al. 2023). However, the issue of hallucination (Ji al.] 2023 Bang al.) 2023 and the challenge of updating the parametric knowledge limit their real-world application specific domains. Retrieval- augmented generation (RAG) alleviates this problem by supplying the LLM with information trieved from a custom outer knowledge base (Guu et al. 2020: Lewis et al. 2020 Yu et al. [003)). Open-source RAG frameworks like llamaindex (Liu 2022) have been developed facilitate the research and deployment of RAG.
Typical retrieval- augmented generation (RAG) pipelines are text-based, operating segmented texts as retrieval units (Yu et al. 2023 Asai et al. 2024 Yan et al., (2024), which we refer to as TextRAG. In real-world scenarios. knowledge often presented multi-modality documents such as textbooks and manuals, which may have texts and figures intersected together. To acquire texts from such data sources, a parsing stage required, which typically involves cascade of processes, including layout recognition, optical character recognition (OCR), and post-processing steps like text joining (Zhang al. [2024] Liu 2022). While effective most scenarios, the parsing process inevitably introduces errors, which can negatively impact the retrieval and generation phases. More- over, TextRAG utilizes only textual information, overlooking potential information present in other modalities like images. Although research has been conducted on image retrieval and multi-modal
Equual contribution. Corresponding authors.

---

Published as a conference paper at ICLR 2025
RAG, these approaches primarily focus predefined scenarios wherein images and descriptive texts are properly extracted and paired (Wei et al. 2023; Sharifymoghaddam et al. 2024] Zhou et all, 2024), differing from real-world scenarios where texts and images (including higures) are often interleaved within a single document page.
The recent development of vision-language models (VLMs) has introduced promising approach to understanding complex visual cues in images and documents (OpenBMB 2024b; WWan et al.) 2024). By integrating a language model with a vision encoder, VLMs demonstrate superior abil- ities in applications such as describing pictures (Alayrac et al. 2022), explaining figures (Bavishi al., ). and transcribing (printed and handwritten) text from document images (Laurençon et al. 2024). Given the robust capabilities of VLMs in capturing multi-modal information present in images, an intriguing question arises: can the basic language model in the retrieval and generation components of TextRAG be substituted with VLM, thus the parsing stage is bypassed and all the information of the document is preserved?
In this paper, we present Vision-based estrieval-augmenter Generation (VisRAG), to study the fea- sibility of building pure-vision RAG pipeline using VLMs. VisRAG is built with a VLM-based retriever VisRAG-Ret and generator VisRAG-Gen. Inherited the bi-encoder of text-based dense re- triever (Karpukhin et al., 2020), VisRAG-Ret maps the query and the document into an embedding space, but utilizing the document's image directly instead of relying on extracted textual content. The embedding is obtained by applying weighted mean pooling on the final hidden states of the in- put text vision tokens. After retrieving top-k document images, VisRAG processes these images to generate the answer. While it is straightforward to use a VLM that supports multi-image input for generation, for VLMs that can only accept one single image, we propose page concatenation and weighted selection techniques to enable the handling of multiple documents. Throughout the pro- cess, VisRAG preserves all information in its original visual format, thereby preventing the potential information loss or distortion that might occur in traditional RAG pipelines.
To evaluate VisRAG on real-world multi-modal doc- uments, we construct datasets from open-source vi- sual question answering (VQA) datasets and syn- thetic query-document pairs derived from web- crawled PDFs.
MiniCPM-V Generator
52.44
GPT-4o Generator
terms of retrieval, VisRAG- Ret outperforms state-of-the-art text- and vision- centric retrievers and achieves better results than solely relying on its constituent vision encoder or language model under identical training con- ditions. For generation, ViRRAG-Gen surpasses traditional text-based generators with open-source VLMs. With VLMs capable of handling mul- tiple images, VisRAG shows increasing perfor- mance gains with more retrieved documents, indi- cating the potential for multi-page reasoning. As depicted in Figure [ in a direct comparison of pipeline performances, VisRAG achieves a 40% rel- ative improvement over TextRAG using MiniCPM- V 2.6 (OpenBMB 2024b) as the generator and 20% relative improvement with GPT-4o (OpenAl 2024) the generator, attributed to the cascade ef- fect. Further analysis reveals that VisRAG possesses better training data efficiency and generalization ability than baseline models, and demonstrates bustness across both text-centric and vision-centre documents. VisRAG shows great promise in replacing TextRAG as the next-generation standard for RAG pipelines.
43.54
37.97
20
10
TextRAG
ViRRAG
Figure 1: TextRAG vs. VisRAG on final gen- eration accuracy. In TextRAG, parsed text serves as the basis for both retrieval and gen- cration processes, In contrast, VisRAG lever- ages the original document image directly by using VLM-based retriever and generator. Details can found Sec.
2 RELATED WORK
Retrieval-augmeeneed Generation (RAG). RAG enhances large language models (LLMs) by incorporating retrieved information from external knowledge bases, which assists in addressing knowledge-intensivee tasks (Guu al. 2020), reducing hallucinations (Semnani et al. 2023). and
2

---

Published as a conference paper at ICLR 2025
acquiring new knowledge (Vu et al. 2023). An RAG pipeline typically comprises text-based retriever that fetches relevant information from the knowledge base given the user query, and an LLM-based generator that reads the query along with the retrieved information to generate
).23). Prior research on RAG primarily focuses on: a) improving the retriever, which typically text encoder producing text embeddings, through generator feed-
swer (Shi et al. 2024b et et al.
back Yu et al. 2023: Shi et al.) 2024b): b) enhancing the generator via supervised fine-tuning (Lin al.
2024 Xu et al. 2024a), in-context pre-training (Shi et al. 2024a), or advanced prompting (Xu et al. 2024c);; and c) developing advanced RAG pipelines to handle long-form or multi-hop ques- tion answering (Jiang et al., 2023] Asai et al. 2024). However, research on RAG has predominantly targeted cleaned text corpora like Wikipedia from an academic standpoint, Building effective RAG
pipelines for real-world, multi-modal documents remains challenge.
Vision-language Models. Recent advancements in vision-language models (VLMs) have greatly improved fine-gra tne-grained multi-modal understanding. Since CLIP (Radford et al. 2021 pioneered contrastive visual-text alignment, models like Flamingo (Alayrac al.) (022), LLaVA (Liu al. 2023b), and BLIP (Li et al. 2022) have expanded LLMs to process visual inputs by connecting languages models with a CLiP-style vision encoder. Research has then shifted towards more ad- vanced multi-task and multi-stage pre-training paradigms, enabling models to generalize across a wide range of vision-language tasks (Liu et al., 2024aa Bai et al., 2023 Wang et al. 2023: Dai et al, 2023). This is followed by notable advancements in high-resolution visual understanding (Xu al.
2024b Bavishi et al.) 2023 Lin et al.
2022 Lee et al. 2023 allla2223... 2023) and OCR capabilities (Kim et al.
Hong al. [2024] Chen et al.] 2024b). Specifically, VLMs like the DocOwl series Ye et al. 2023a
Hu et al.) 2024b a). UReader Ye et al. 2023b), and TextMonkey (Liu et al. 2024b) are purpose-
built tackle OCR-free document understanding. More recently, breakthroughs have been made in multi-image understanding (Li et al. 2024a, Wang et al., 2024). Recent open-source VLMs like the MiniCPM-V (Yao al. 2024) and Qwen2- VL (Wang et al., (024) series combine the merits of recent techniques, achieving state-of-the-art performance. Those features of VLMs provide a foun- dation for our vision-based RAG pipeline, which requires multi-modal document understanding.
Multi-modality Retrieval and RAG. Multi-modal retrieval encompasses wide range of tasks, such as retrieving a matching image given the text (Han et al.] 2017), retrieving a text-image pair to answer a question (Chang et al. 2002)) and retrieving texts that answer the given query about provided image (Hu et al. 2023a Luo et al. (023). etc. |Wei et al. (2023) propose UnilR, a universal multi-modal retrieval model capable of addressing the aforementioned multiple tasks. The retrieved information is then employed for incorporating knowledge (Hu et al. 20236] [uo et al.) (202I) or in-context learning (Tan et al., 2024 Liu et al., 2023a), with the aim of generating answers or im- ages (Sharifymoghaddae
et al. 2024). Prior research mentioned above is conducted on academic datasets, where texts and images are meticulously extracted from raw data and paired (e.g., images with their captions). to make it feasible to do separate encoding of data in different modalities. This hinders their applicability in real-world RAG scenarios, as real-world multi- documents are of- ten presented in mixed modalities, and information may be distributed across various combinations of modalities. Concurrent works DSE (Ma et al. 2024) and ColPali (Faysse et al. 2024) address this issue by directly encoding the image of a document for retrieval. However, these studies focus on retrieval, they lack a comprehensive comparison of their approaches with text-based retrieval in both in-domain and out-of-domain settings, and do not conduct an end-to-end RAG evaluation.
METHODOLOGY
In this section, we first recap the typical RAG pipeline (Sec. [.1]. then present our VisRAG frame- work (Sec. 3.2) and the construction of our training and evaluation data (Sec. (33).
3.1 PRELIMINARY: RETRIEVAL- GENERATION
A typical retrieval-augmennte generation (RAG) pipeline consists of retriever and generator, both built on large language models (LLMs |
This pipeline operates on a knowledge corpus D,
In many cases, the retriever uses language models smaller than 1B parameters, which may not be consid- ered "large", but we the term I.I.M for simplicity.
3

---

Published as a conference paper at ICLR 2025
VIaRAG
What is the trend in sound frequency sensitivity from opex the cachlea?
No related information
Decreasing
INORI HL SKI HNC (100) zu 2000 ust 4000
TextRAG-Rut
PERCEIVING SOUND FREQUINCIES
TexIRAG- Gen
ViaRAG- Gan
ViaRAG-Rat
Comcopanda uoss cochics
Index
Page Text
Page
VaRRAG-let
Index
Parse
Image
Documenta
Figure 2: TextRAG (left) vs. VisRAG (right). Traditional text-based RAG (TextRAG) relies on parsed texts for retrieval and generation, losing visual information multi-modal documents. Our vision-based RAG (VisRAG) employs VLM-based retriever and generator to directly process the document page's image, thereby preserving all information the original page.
which processed into units for retrieval and generation, denoted as D = {di,.... dx where is the number of retrieval units. Given a text query q and the retrieval corpus D. the retriever functions as R (g. D) + Dr. taking and D as inputs and producing a candidate set DR C 'D. To enable efficient search, the units in the knowledge corpus D pre-encoded into embeddings. During RAG pipeline inference, approximate nearest neighbor (ANN) search is applied to retrieve DR. which serves the knowledge source for generation. The generation process a function G (g. DR) + where represents the answer and G denotes the LLM generator. This achieved by prompting the LLM with the query and the retrieved units DR to generate an answer.
be defined as
As shown in Figure (left), traditional RAG frameworks (TexIRAG) typically utilize text-based units for retrieval and generation. However, real-world scenarios, data often appear in complex, multi-modal documents, requiring an additional parsing step obtain text. In this paper, we propose use the page as the fundamental unit for retrieval and generation, which directly processed by vision language models (VLMs) as image without further processing during retrieval and generation. In subsequent sections, we use the terms "page" and "document" interchangeably.
3.2 VISRAG: VISION- RETRIEVAL-AUGMENTER GENERATION
In this section, present Aision-based Retrieval -uggeenne Generation (VisRAG), shown in Figure (right). In contrast to traditional RAG frameworks which use text segments for both re- trieval and generation, VisRAG leverages the image of the document to preserve all information.
3.2. RETRIEVAL
The first stage of VisRAG, VisRAG-Ret, aims retrieve set of pages from the corpus D given q. We follow the dual-encoder paradigm in text-based dense retrieval models (Karpukhin et al.) 2020) but employ VLM rather than LLM encode the query and page. Specifically, the query and page are encoded separately as text and image in the VLM, producing in a sequence hidden generative VLMs with causual attention, we adopt the position-weige weighted mean pooling over the last-layer VLM hidden states (Muennighoff
states. To derive the final embedding, and given that
2022), giving higher weights later tokens:
00 h
where h, is the i-th hidden state, S the sequence length, w, Σ is the i-th weight, and is the query page embedding. The similarity score is calculated by the cosine similarity of the query
4

---

Published as a conference paper at ICLR 2025
and page embedding. VisRAG-Ret is optimized using the InfoNCE loss:
exp(s(q, 1/>)
l(g.d*, D = - log exp(s(q, -La--- exp(s(q,
(2)
where d', D are positive document and the negative document set of q, respectively, s(g, d) the similarity score between and d, and 7 is the temperature.
3.2.2 GENERATION
The second stage of VisRAG, VisRAG-Gen, focuses on generating the answer according to the user query and retrieved pages using a VLM. We propose the following mechanisms to enable VisRAG- Gen to handle multiple retrieved pages in DR for generation. The prompts used for generation is presented in Appendix
Page Concatenation. A straightforward approach is to concatenate all pages in Dr into single image to accommodate most VLMs that are trained to accept single image. Formally,
+ VLM LLM--Sngle(g.. Concat((dld Dx))),
where VL.M-Single a VLM that accepts a single image with text prompt and Concat is the image concatenation operation. In this paper, we experiment with horizontal concatenation.
Weighted Selection. Another approach is to ask the VLM to generate an answer for every page from top-k; and select final one with the highest confidence (Lewis et al. 2020) Shi et al. 2024b). The final confidence is defined as the weighted generation probability of the answer:
P(alg, PR) P(alg, d) X(g.d).
(4)
where P(ald, q) is calculated as the reciprocal of the perplexity of generating the answer condi- tioned on the single document d, and A(d, q) is the normalized retrieval score:
(
X(g. d) En ex(gdd))
(5)
VLMs Accepting Multiple Images. Some recent VLMs like MiniCPM-V 2.6 (OpenBMB 2024b) and Qwen-VL 2 (Wang et al. (024) are designed and trained to accept multiple images input to perform cross-image reasoning. This capability may be useful for the generation as the required information could be located on a single page from the retrieved document set De for single-hop questions or spread across multiple pages for multi-hop questions. Formally, we have
+ VLM-Multi(g, {dld E DR
(6)
where VLM-Multi is the VLM that accepts multiple images with text prompt.
3.3 DATA CONSTRUCTION
To effectively build and evaluate RAG pipelines multi-modal documents, construct datasets using a combination of visual question answering (VQA) datasets and synthetic data. The statistics of our constructed dataset provided in Table
Data Sources. We collect question-document pairs from a series of VQA datasets, targeting dif-
ferent document types: MP-DocVQA (Tito al. 2023) for industrial documents, ArXivQA (Li et al.) 2024b). ChartQA (Masry et al.) 2022). Infographics
(Mathew et al. 2022). and
PlotQA Methani al. (2000) for various figure types, and Slide VQA (Tanaka et al. (023) for
presentation slides. All datasets feature questions that can be answered using a single document (page), except for VQA, which includes multi-hop questions requiring information from mul- tiple pages. We follow the original datasets' train-test splits, except for MP-DocVQA and Info- graphicsVQA, where the validation split serves as our evaluation set. Additionally, we enhance our training set by collecting openly available PDFs from online sources and generating queries using GPT-4o (OpenAl 2024), with details presented in Appendix A. We assemble the retrieval corpus by gathering the document associated with each query from the training and evaluation sets.
5

---

Published as a conference paper at ICLR 2025
Table I: Dataset statistics. We collect data from visual question answering (VQA) datasets for train- ing and evaluation and synthetic additional query-document pairs for training. We apply filtering on VQA datasets remove context-dependent queries that are not suitable for retrieval.
Train
Evaluation
Source
Document Type
Q-D Pairs
Q (% Preserved)
#
# Pos. D per Q
ArXivQA 2024h
Arxiv Figures
25,856
816 (8%)
8,066
1.00
ChartQA 2022)
Charts
4,224
63 (5%)
500
1.00
MP-Doc VQA (2023
Industrial Documents
10,624
591 (11%)
741
1.00
InfoVQA (2022
Infographics
17,664
718 (26%)
459
1.00
PlorQA 22020
Scientific Plots
56,192
863 (4%)
9,593
1.00
VIi (2023
Slide Decks
8,192
556 (25%)
1,284
1.26
Synthetic
Various
239,358
Query Filtering. Some queries extracted from VQA datasets are context-dependent, which lack specificity to certain entity. For instance, the response to "Where was the conference held?" varies based on the contextual document. Using such context-dependent queries in open retrieval tasks is ineffective because they lack strong document specificity. To address this, we implement an addi- tional filtering stage remove these conlext-dependent questions, where we prompt GPT-4o (Ope-) nAI 2024) with human- annotated in-context samples to generate the classification label. Table shows a substantial reduction context- dependen questions across evaluation sets. The details of filtering are presented in Appendix A.2
Evaluation Metrics. We report the retrieval and generation performance the evaluation sets of the datasets sourced from VQA datasets. For retrieval, we use MRR @ 10 and Recall@ 10 as the metrics. For generation, consistent with methods applied the source datasets, we report the answer accuracy, employing relaxed exact match metric which allows a 5% error margin for numeric responses (Masry et al. 2022 Methani al. 2020
4 EXPERIMENTAL METHODOLOGG
In this section, introduce our setup for experiments. Descriptions of the LLMs/VLMs used in our experiments can be found in Appendix
Document Parsing. To evaluate the performance of VisRAG against TextRAG, we introduce two text extraction methods. The first, "(OCR)", employs a pipeline that uses PPOCR (Du et al,, 2020) to detect text regions and then merges nearby boxes to reduce fragmentation. The second, "(Captioner)", is a model-based approach that directly extracts text from document images using MiniCPM-V 2.0 (OpenBMB 2024a Yao et al. 2024 fine-tuned on paired (document image, tracted text) data. More details are provided in Appendix
Retrieval Experiments. VisRAG-Ret is document embedding model built on MiniCPM-V 2.0, a vasion-language model that integrates SigLIP (Zhai et al. (223) as the vision encoder and MiniCPM (Hu et al. (2024c) as the language model. To ensure fair comparisons, periments into three settings: off-the-shelf,, out-of-domain, and in-domain, as depicted below, We report VisRAG- Ret'sss performance in both oft-oi- and in-domain settings.
organize ex-
Off-the-shelf: We directly evaluate popular text and image retrieval models extracted texts, including BM25 (OCR), a lexical model; bge-large-en-vl. (Xiao et al. (203) (OCR) and NV-Embed-v2 (Lee al. 2024) (OCR), state-of-the-ar text embedding models with sizes 335M and 7.85B, respectively: and SigLIP, a CLIP-style (Radford et al. 2021 vision model serving the encoder for MiniCPM-V series.
Out-of-domain: Out-of-domain models are trained solely on synthetic data and eval- uated the VQA datasets without in-domain supervision. These models include MiniCPM (OCR), MiniCPM (Captioner). and SigLIP. MiniCPM (OCR) and (Captioner) are MiniCPM-hasee text embedding models trained and evaluated extracted text.

---

Published as a conference paper at ICLR 2025
Table 2: Overall retrieval performance in MRR@ 10. The best retrieval performance in each group is marked in bold, and the second best performance is underlined. We train ColPali (Faysse et al. 2024) on our dataset. Corresponding Recall @ 10 performance can be found in Table 6
Model
# Para.
ArxivQA
ChariQa
DocVQA
InfoVQA
PlatQA
SlideVQA
Average
Off the shelf Models
BM25 (OCR)
43.65
61.47
75.21
66.94
57.28
86.78
65.23
bge-large 2023 (OCR)
335M
39.29
59.64
50.76
72.38
51.33
81.38
59.13 13
NV-Emhed 2024 (OCR)
7,85B
75.46
SigLIP 2023
883M
31,39
64,71
46.56
62,85
30.23
75.14
51.81
Out
Out-of-domain: domain:
Models Fine
Syniketic
MiniCPM (OCK)
2.72B
47.96
61.04
67.04
79.36
36.04
87,93
63.33
MiniCPM (Captioner)
2.72B
42.07
71.84
64.48
76.10
29.76 79.76
81.01
60 88
SigLIP 2023
883M
46.81
68.40
57.61
67.12
31.92
85.14
59,50
VisRAG-Kett
3,438
69.17
66.37
73.06
84.65
45.57
90.09
71.49
(c) In-domain: Models Fine-tuned
Synthetic and In-domain data
MiniCPM (OCR)
2,72B
58.43
77.74
72.54
83,45
64.78
91.74
74.78
MiniCPM (Captioner)
2.72B
56.15
74,06
67.57
81.22
55.43
84.27
69.78
SigLIP 2023
883M
59.16
81.34
64.60
74.59
61.32
89.08
71.68
ColPali 2024
2.92B
72.50
73.49
82,79
81.15
55.32
93.99
76.54
VisRAG-Ket
3.43B
75.11
75.37
86.37
62.14
In-domain: Models in this category are trained on the blend of the VQA training data and synthetic data. We evaluate the same set of models in the out-of-domain setting to show model performance when supervised labels are available. We also report the performance of ColPali (Faysse et al. (024) on our evaluation data. ColPali is page embedding model that encodes a screenshot of a page into multiple vectors. We train ColPali on our dataset using the official code and Hlper-parameteer provided in its paper.
Generation Experiments. To evaluate generation performance, we fix the retrieval model to VisRAG-Ret and report the performance various generation models and methods. For VisRAG- Gen, we compare the performance of the single-image VLMM MiniCPM-V V 2.0, which only accepts single image, against the multi-image VLM MiniCPM-V 2.6 (OpenBMB 2024b Yao et al. 2024) and GPT-4o (OpenAl 2024). MiniCPM-V 2.6 is an upgrade of MiniCPM-V 2.0. incorporating
Qwen2-7B YYang et al. 2024) as the language model and supporting multi-image input. We evaluate the performance of page concatenation and weighted selection on the single-image VLM. Addition- ally, we report the performance of text-based generation baselines, including MiniCPM (OCR) and GPT-4o (OCR), where only extracted texts are used for generation. For all experiments, we report results using the top-1, top-2, and top-3 retrieved documents, as well as an "Oracle" condition where the model is provided with only the positive document(s) to show the performance upper bound.
Implementation Details. VisRAG-Ret is fine-tuned using in-batch negatives (Karpukhin et al., 2020) for one epoch with batch size of 128 on 8 NVIDIA A A100 80GB GPUs. The temperature parameter in Equation 2 is set to 0.02. Baseline retrievers are fine-tuned with the same hyper- parameters, and textual baselines utilize extracted text data as document-side input. The generation part does not use any fine-tuning, we directly off off-the-shheff LLMs/VLM, for generation.
EVALUATION RESULTS
In this section, we first present the overall performance of VisRAG (Sec. D. followed by analyses of training data efficiency (Sec. 5.2) and performance on different subsets (Sec. 5.3
5.1 OVERALL PERFORMANCE
Retrieval Performance. In this experiment, we compare VisRAG-Ret with (a) off-the-shelf mod- els, and trained baselines in (b) out-of-domain setting where only leverage synthetic data, and in (c) in-domain setting where we leverage both in-domain and synthetic training data.
As shown in Table 2 (b), VisRAG-Ret, trained out-of-domain data, significantly outperforms both off-the-shelf models BM25 and bge-large, and achieves 95% of the performance of NV-Embed- v2, state-of-the-art text retrieval model with 7.85B parameters. Note that bge-large and NV.
7

---

Published as a conference paper at ICLR 2025
Table 3: Overall generation performance in accuracy (%). All models and methods utilize the same retriever, VisRAG-Ret. Performance relative to Oracle is colored in blue.
Model Methhod
Input
ArxivOA
ChariJA
DocVQA
IntoVQA
PlotA
SlideVQA
Average
(a) TexaRAG Gen: Text based
Generation
top
43.38 (96.2%)
Q2.7%)
31.47 (75.9%)
20.19 (92.9%))
(SL05)
(94888)
27.68 (57.8%8)
MiniCPM (OCR)
top-2
42. (91.5))
23.81 (68:215
33.67 (5)
20.19
14.14
30.40 (98 396)
27.39
top-3
(90.88%))
20.63 (59 18)
(76.5%)
(84-09) 019)
16.34 (94.05)
(94 253
26.71 (84 959)
Oracle
45.10
(100))
34.92 (1006)
41 16.46/11 (1006)
21.73 (30015)
17.38 (10066
30.94 (100953
31.92 (10096)
top-1
58.33
(0.00%
(64
(90.655)
13,90
(85.015)
42.99
GPT-40 (OCR)
top-2
59. 44 (96.8%)
47.62 (71 45%
56.51 (38.65)
47 07.08 195.19)
15.87 (77.8%)
51.08 (92.8%)
46.27 (56.8%)
top-3
61.76 (00.66))
44.44 (66.7%)
55.67 (87.3%)
49.58
14.72 (72.2%)
49.28 (89.51))
45.91 (85.75))
Oracio
(1000))
66.67 (100%)
63.79 (1006))
50.56 (1005)
20.39 (10096)
55.04 (1000))
52.97 (10015)
(b)
VisRAG-Gen: Single-inage VLM (MmiCPMM-
top-1
59.07 (96 (nli
34.92 (88.05)
39.42 (14 416)
29.53 (16
17.84
36.15 (91
36.16
Page Concatenation
top-2
57.35 (95 1%
19.05 (5%)
32.32 0%)
22. (2 993
15.41 (6688%)
33.45 (84-99)
29.
top-3
59.19
(98.2%)
22:22 (56.055)
24.87 (47)96)
20.33 (596166)
16.92 (73.466)
30,22 (76.7%6)
28. 96 (08.558)
Oracle
60.29
39.68
52.
23,06 (1002))
39. 19.39
(10995)
top-1
59.07
(98.0%)
34.92 (58.05)
39 39,42.0 (74.4%)
29.53 (56 5%9
17.84 (77.4%)
36.15 (57 .4%)
36. 16 (5. 3%)
Weighted Selection
top-2
60.29
(100.0%)
33.33 (84.0%)
39. 26 174
28.97 (84-9433
18:08 (78.415)
36.69 (88.7%)
36. 10 (85.0%)
top-3
60.78
(100a)5)
31.75 (20.01))
38.41 (72.598)
28. 69 (S4.1163
17.03 (33.915)
36.33 (87.8%5)
35.50 (83.298)
Oracle
60:29 (10099)
(1098
52.96 (1099))
34. 14.12 (10066
23,06
41.37 (10056
41.91 (100963
[c] VisRAG-Gen Multi-imge
VLM
top-1
66.30,91 (9.998) 9%
(69 89))
60.24 (72.45)
56.41 (98.99)
40.79 (65 196)
(84: (5)
53.12, 9%)
MiniCPM-V 2.6
top-2
66.79 (94.0%2
52.38 (76.7%)
67.17 750
(34.0))
38.35
(6.2
50.90 (88.25)
54.92 (30 95)
top-3
67.77
(19 198)
(88
(85.69)
38.93
1%6
(8799))
56.12 555)
Oracle
71.08 (1006)
68.25 (1009)
83.25 (1099)
63.65 (1009))
(1009))
57.73 (1009)
67.78 (1009)
top-1
64.71 (98.0%)
32.38
58, 38.8% (1)
(38.9)
20.74 (66
54,86 (8500%)
52:44 (31.4))
GPT-4o
top-2
63,36 (95.9%)
49.21 (1))
64.13 (80348)
66.85 (95.696)
20.16 (64.4%%)
58.45 (90.515)
53.69 (82.918)
top-3
62.01 (91.9%)
53.97 (M 115)
67.17 (s4
66,43 (95006))
19:35 M963
60.97 (94 4ts)
54, 34.98
Oracle
66.05 (100%)
68.25 (1009)
(1006)
71 (1005))
31.29 (1005))
64.57 (1009)
63.49
Embed-v2 are trained on millions of query-doc pairs (Xiao et al. 2023 Lee et al. 2024). which
10x more than our training data. Although bge-large outperforms BM25 benchmarks like MTEB (Muennighoff et al., (202)), it fails on our datasets, indicating text-based embedding models trained on clean text struggle with texts parsed from real-world documents.
When trained with the same data setup, as demonstrated in Table Table VisRAG-Ret outperforms text models MiniCPM (OCR) & (Captioner) and the vision model SigLIP by significant margin. The advantage is more pronounced in the out-of-domain setting, where VisRAG-Ret achieves 13% and 20% gains over MiniCPM (OCR) and SigLIP. respectively, compared 4% and 9% in the in-domain setting. This indicates that VisRAG-Ret has better generalization capability compared to text- and vision-centric models. Notably, despite utilizing the same VLM MiniCPM-V 2.0 for parsing, MiniCPM (Captioner) performs worse than VisRAG-Ret, indicating that directly encoding with VLMs works better than using VLMs for parsing. This can be attributed to the inevitable information loss when multi-modality information is transcribed into text.
Further analysis reveals that MiniCPM (OCR) and SigLIP perform differently across datasets: SigLIP excels in ArxivQA and CharlQA, while MiniCPM (OCR) significantly outperforms SigLIP in Doc and Infographics VQA. This may be due to the different focuses of the two models: MiniCPM focuses on text, while SigLIP focuses visual signals. VisRAG-Ret, built top of MiniCPM-V 2.0, with a SigLIP encoder and a MiniCPM language model, combines the merits of both and performs well across all datasets, capturing more holistic information from a document.
Compared to ColPali, a multi-vector document page embedding model, VisRAG-Ret not only main- tains superior performance but also achieves much better memory efficiency. ColPali represents page with 256KB of data distributed across 1030 128-dim vectors (Faysse et al. 2024), whereas VisRAG-Ret uses just 4.5KB in a single 2304-dimensional vector. This makes VisRAG-Ret more suitable for scaling to millions or billions of documents in real-world applications.
Generation Performance. In this experiment, apply series of text- and vision-based genera- tors and methods on top of the same retriever VisRAG-BB Ret to study their effectiveness in generating the answer given the query and retrieved documents. Table the performance of (a) text-based generation (TextRAG AG-Gen),
(b) generation using the VLM MiniCPM-V 2.0 which only accepts a single image as input, and (c) generation using VLMs which accept multiple images as input.
When models are provided with only the ground-true documents ("Oracle"), VisRAG-GG Gen models, which process the document image directly, significantly outperform TextRAG-Gen models, which
8

---

Published as a conference paper at ICLR 2025
Correct Genaration
178
51.1%
Correct Generation
367
Cornect Retrieval 552
76.9%
79.4%
570
All Que
52, 1%
Incernec Genenation
718
374
24.3%
Inporrect Generation
203
23.1%
Incorract Retrieval
20.8%
incomee Fatrinus
166
148
(a) TexIRAG with MiniCPM (OCR) as retriever (b) VisRAG VisRAG-Re Ret as retriever and and MiniCPM-V 2.6 (OCR) as the generator.
MiniCPM-V 2.6 as the generator.
Figure 3: Pipeline performance of (a) TextRAG and (b) VisRAG on Infographics We visualize the portion of queries that have the positive document retrieved at the top-1 position ("Correct Re- trieval"), ) and that are answered correctly given the top-1 retrieved document ("Correct Generation")
rely solely extracted text. For instance, MiniCPM-V 2.0 achieves 30% higher performance than MiniCPM (OCR) when using ground-truth documents. This underscores the importance visual clues in extracting answers from documents.
In practical scenarios where models receive the top-1 to 3 retrieved documents, which may in- clude noise, VisRAG-Gen consistently outperforms TextRAG-Gen within the same model series. Specifically, for MiniCPM-V V 2.0, capable of processing only a single image, the weighted selection approach demonstrates better performance than page concatenation when handling documents. However, neither method shows performance improvement as the number of retrieved documents increases, trend commonly observed in TextRAG pipelines (Zhu et al., 2024). con- trast, MiniCPM-V 2.6 and GPT-4o, both capable of processing multiple images input, exhibit a notable performance gain as the number retrieved documents increases, suggesting that only VLMs pre-trained on multi-image data can effectively reason over multiple retrieved pages.
retrieved
End-to-end Performance. In this experiment, we study the effectiveness of the VisRAG pipeline, by comparing with the TextRAG pipeline. We construct TextRAG using MiniCPM (OCR) and MiniCPM-V 2.6 (OCR) for retrieval and generation, respectively, and VisRAG using VisRAG-Ret for retrieval and MiniCPM-V 2.6 for generation. The performance on InfographicsVQA visually represented in Figure 3 Notebly, VisRAG achieves higher rate of accurately retrieving docu- ments than TextRAG, and demonstrates a significantly improved rate correct answer generation from accurately retrieved documents. The cumulative improvements in both retrieval and genera- tion phases result in an overall accuracy increment from 25% to 51%. Across the six evaluation datasets, VisRAG shows 40% relative accuracy increment average, illustrated in Figure The case study of VisRAG and TextRAG is pre- sented in Appendix
5.2 TRAINING DATA EFFICIENCY
Avenaee MRR.i1
In this experiment, study the training data efficiency of VisRAG-Ret by evaluating the per-
formance of VisRAG-Ret trained under different amounts of synthetic training data, i.c. in the
20
MiniCPM
out-of-domain setting. shown in Figure 4 to achieve the same performance bge-large (OCR), VisRAG-Ret requires training on only 20K exam- ples, whereas MiniCPM (OCR) needs about 75K examples. In later training stages, VisRAG-Ret still maintains 13% performance advantage MiniCPM (OCR). Although NV-Embed-v2 (OCR) slightly outperforms VisRAG-Ret trained
bge-large (OCR)
NV-Emhed-v2
0.
5.0e+04
1.0e+05
1.Se+05
2.0e+05
Train Q.D
Figure 4: Average retrieval performance of VisRAG-Ret MiniCPM (OCR) trained with different numbers training examples.

---

Published as a conference paper at ICLR 2025
Relative Perfoceanne of ViRRAG
Retrieval
Retrieval & Generation
Relative Performance of VidRACt (P)
TeaRAG (Baseline)
Dovumen: Length tokens)
100
100
A MMRRG8
80
A Accuracy (%)
80
Avg. Length
ArxivQA
10
40
4. 1/ o
40
20
20
Avg.
20
40
80
100%
20
40
80
100%
20
500
15a
-00
15-
400
Document Length (F
6 Accuaay (%)
10-
400
Document Lengts (8
A MRR@10
300
InfographicsVO0 VQA
-5
100
100
15-
20
20
40
60
80
100%
-15-
20
40
60
80
100%
Relative retrieval and generation performance of VisRAG, VisRAG (SigLIP), and Tex- IRAG on different subsets of queries. The X-axes represent the query subsets where the lengths of the positive documents fall within specific percentile ranges. For comparative analysis, set Tex- tRAG's performance to zero and show the performance differences of other models from TextRAG.
Figure 5:
240K synthetic dataset, is trained on millions of curated query-document pairs and has an 8B pa- rameter scale. This suggests that capturing multi-modal information is more effective and efficient than merely increasing training data and model parameters but relying solely on the text modality.
5.3 PERFORMANCE ON DIFFERENT DATA SUBSETS
In this experiment, we assess the retrieval and generation performance of VisRAG and TextRAG de- fined in Figure 3 well VisRAG (SigLIP), which replaces the retriever in VisRAG with SigLIP. In Figure []] we report their performance across different data subsets of ArxivQA and Infograph- iesVQA by categorizing queries based the lengths of their positive documents, measured by the number tokens of the extracted text. Documents with higher volume of extracted text may pri- oritize textual information
visual content. For each group, performance differences between VisRAG and TextRAG, as well as between VisRAG (SigLIP) and TextRAG, compare how each model performs relative to TextRAG. We observe that, in general, the relative performance of VisRAG and VisRAG (SigLIP) improves as the length of the relevant document decreases. This suggests that models with vision encoders can better understand docu- ments that emphasize visual information. However, VisRAG (SigLIP) consistently underperforms VisRAG across data subsets and, some cases, even performs worse than TextRAG. In contrast, VisRAG outperforms TextRAG most subsets, indicating that the underlying language model in VisRAG is crucial for better understanding the semantics conveyed through visual cues.
calculate and plot the average
6 CONCLUSION
In this paper, we propose VisRAG, a novel retrieval-augmeened generation (RAG) paradigm that uti- lizes laniunelanguage models (VLMs) to facilitate retrieval and generation within RAG pipeline, thereby eliminating the parsing stage required in traditional text-based RAG. Our empirical re- sults demonstrate that VisRAG consistently outperforms text- baxtd RAG retrieval and generation while maintaining a simpler pipeline. We hope that VisRAG will inspire future RAG development incorporate VLMMs for handling multi-modal documents.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by the Institute Guo Qiang at Tsinghua University. It is also partially sup- ported by the Natural Science Foundation of China under Grant No. 62206042.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Proceedings of NeurlPS, volume 35, pp. 23716-23736, 2022.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In Proceedings of ICI.R 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding. local- ization, text reading, and beyond, 2023.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt reasoning, hallucination, and interactivity. In Proceedings AACL/UCNLP 2023, pp- 675-718, 2023.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşıllar Introducing our multimodal models, 2023. BtRp:ttp:: h/wp::///wwwww.. adept. ai ai/bloq fuyu-8b.
Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. In Proceedings of CVPR, pp. 16495-16504, 2022.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangho Wu, Zhiyi Zhang, Zhi- hong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpr4v-synthesized data for a lite vision-language model. arXiv preprint 24V:2440..1664, 2024a.
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang. Jun Liu, Guirong Chen, Yupeng Huo. et al. Guicourse: From general vision language models to versatile gui agents. arXiv preprint arXiv:2406. 11317, 2024b.
Gordon V Cormack, Charles LA Clarke, and Stefan Buettcher. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of SIGIR, 2009.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuingg. In Proceedings of NeurIPS, 2023.
Yuning Du Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, and Haoshuang Wang. Pp-OCR: A Practical Ultra Lightweight OCR System. arXiv, 0e0/2009909994 2020.
Manuel Faysse, Hugues Sibille, Tony Wu, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Col- pali: Efficient document retrieval with vision language models. arXiv preprint arXiv:2407.01449, 2024.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Retrieval augmented language model pre-training. In Proceedings of ICML, pp. 3929-3938, 2020.
Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao and Larry S Davis. Automatic spatially-aware fashion concept discovery. Proceedings of ICCV, pp. 1463-1471, 2017. 2017.
11

---

Published as a conference paper at ICLR 2025
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong. Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of CVPR. pp. 14281-14290, 2024.
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Ji Zhang, Qin Jin, Fei Huang. and Jingren Zhou. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of EMNLP, pp. 3096-3120, 2024a.
Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan. Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docow/ High-resolution compressing for ocr-free multi-page document understanding. arXiv preprint arXiv:2409.03422 2024b.
Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming- Wei Chang. Open-domain visual entity recognition: Towards recogniz- ing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12065-12075, 2023a.
Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang. Yuxiang Huang. Xinrong Zhang, Zhen Leng Thai, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, da- hai li, Zhiyuan Liu, and Maosong Sun. Miniepm: Unveiling the Potential of Small Language Models with Scalable Training Strategies. In First Conference Language Modeling, volume abs/2404.06335, 2024c.
Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang. Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. Reveal: Reureeee visual-language pre-training with multi-source multimodal knowledge memory. In Proceedings of CVPR, pp. 23369-23379, 2023b.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Surv., (12):248:1 1--:::4::-4:38.. 2023.
Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang. Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of EMNLP, pp. 7969-7992, 2023.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Pro- ceedings of EMNLP, pp. 6769-6781, 2020.
Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam. Jinyoung Park. Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document un- derstanding transformer. In Proceedings of ECCV, pp. 498-517. Springer, 2022.
Hugo Laurençon, Léo Tronchon, Matthicu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint 2rXiv:2405.02246 2024.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Ny-Embed: Improved Techniques for Training LLMs as Generalist Embed- ding Models. arXiv, 0bb/2405.17422. 2024.
Kenton Lee, Mandar Joshi, Tulia Raluca Ture, Hexiang Hu, Fangyu Liu, Julian Martin Eisensch- los, Urvashi Khandelwal, Peter Shaw, Ming- Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In Proceedings of ICML, pp. 18893-18912. 2023.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Reureetal-augmented generation for knowledge-intenst NLP tasks. Proceed- ings of NewrlPS, 2020.
12

---

Published as a conference paper at ICLR 2025
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Alava-onevision: Easy visual task transfer. arXiv preprint Stt.:2408.0332066 2024a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In Proceedings of ICML, pp- 12888-12900, 2022.
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng. Lingpeng Kong, and Qi Liu. Mul- timodal ArXiv: Dataset for Improving Scientific Comprehension of Large Vision-Language Models. In Proceedings of ACL, pp. 14369-14387, 2024b.
Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Ro- driguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. Ra-dit: Retrieval-augmented dual instruction tuning. In Proceedings of ICLR, 2024.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint wxXiv:2311.0775..22 2023.
Bingshuai Liu, Chenyang Lyu, Zijun Min, Zhanyu Wang. Jinsong Su, and Longyue Wang. Retrieval- augmented multi-modal hhain-of-thought reasoning for large language models. arXiv preprint arXiv:2312.017/4 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Proceed- ings of NeurlPS, volume 36, pp. 34892-3499 34916, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of CVPR, pp. 26296-26306, 2024a.
Jerry Liu. LlamaIndex. 11 2022.
Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li. Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473 2024b.
Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral. Weakly-supervised visual-retriever- reader for knowledge-bard based question answering. In Proceedings of EMNLP, pp. 6417-6431, 2021.
Man Luo, Zhiyuan Fang, Tejas Gokhale, Yezhou Yang, and Chitta Baral. End-to-end knowledge retrieval with multi-modal queries. In Proceedings of ACL, pp. 8573-8589, 2023.
Xueguang Ma. Sheng- Chich Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multimodal retrieval via document screenshot embedding. arXiv preprint arXiv:2406. /251, 2024.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. Chartqa: A Bench- mark for Question Answering about Charts with Visual and Logical Reasoning. In Proceedings of ACL, pp. 2263-2279, 2022.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Emest Valveny, and C. Jawahar. Infographievga. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2582-2591 2022.
Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In The IEEE Winter Conference on Applications of Computer Vision (WACV), March 2020.
Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint 222.::2202.08904,, 2022.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embed- ding benchmark. In Proceedings of EACL. pp. 2014-2037, 2023.
13

---

Published as a conference paper at ICLR 2025
OpenAl. Hello, gpt-4o
openai, 2024.
URL http:///ogeaal.com//neee
hello-gpt-407
OpenBMB. openbmb/minicpm-v22, 2024a. URL htte:://ugghnggaaee.o///ogonnnn
MiniCPM-V-2
OpenBMB. opwnbnbbmminncmmm-2221.. 2024b. URL httpp:://mucchngfffa....////a//p
Mini MiniCPM-V-2-2
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of ICML. pp. 8748-8763, 2021.
Sina Semnani, Violet Z Yao, Heidi C Zhang, and Monica S Lam. Wikichat: A few-shot Ilm-based chatbot grounded with wikipedia. arXiv preprint 2tti::200.11299. 2023.
Sahel Sharifymoghaddam, Shivani Upadhyay, Wenhu Chen, and Jimmy Lin. Unirag: Universal retrieval augmentation for multi-modal large language models. arXiv preprint arXiv:2405.13311, 2024.
Weijia Shi. Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A Smith, Luke Zettlemoyer, Wen-tau Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document boundaries. In Proceedings of ICLR, 2024a.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zeuleeoyer, and Wen-tau Yih. Replug: Retrieval-augmented black-box language models. In Proceedings of NAACL, pp. 8364-8377, 2024b.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that read. In Proceedings CVPR, pp. 8317- 8326, 2019.
Cheng Tan, Jingxuan Wei, Linzhuang Sun, Zhangyang Gao, Siyuan Li, Bihui Yu, Ruifeng Guo, and Stan Z Li. Retrieval meets reasoming: Even high-school textbook knowledge benefits multimodal reasoning. arXiv preprint 240i:::400.20834 2024.
Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A Dataset for Document Visual Question Answering on Multiple Images. In Proceed- ings of AAAI, pp. 13636-13645, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre. Morgane Rivière, Mihir Sanjay Kale, Julictte Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 2024.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. Proceed- ings of NeurlPS Datasets and Benchmarks Track, 2021.
Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Hierarchical multimodal transformers for Multipage DocVQA. Pattern Recognition, 144:109834, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung. Denny Zhou, Quoe Le, et al. Freshllms: Refreshing large language models with search engine augmentation. arXiv preprint arXiv:2310.032/11 2023.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing rision-languee model's perception of the world at any resolution, 2024.
14

---

Published as a conference paper at ICLR 2025
Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogylm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03075 2023.
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang. Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. arXiv preprint 1rxiv:2311.17/:. 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. TMLR, 2022.
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian- Yun Nic. C-pack: Packaged resources to advance general chinese embedding. arXiv preprint arXiv:2309.07597, 2023.
Peng Xu, Wei Ping, Xianchao Wu, Zihan Liu, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa 2: Bridging the gap to proprietary Ilms long context and rag capabilities. arXiv preprint arXiv:2407. 14482, 2024a.
Ruyi Xu, Yuan Yao. Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: Imm perceiving any aspect ratio and high- resolution images. arXiv preprint arXiv:2403. 1703, 2024b.
Zhipeng Xu, Zhenghao Liu, Yibin Liu, Chenyan Xiong, Yukun Yan, Shuo Wang, Shi Yu, Zhiyuan Liu, and Ge Yu. Activerag: Revealing the treasures of knowledge via active learning. arXiv preprint aXXiv:2402. 2024c.
Shi-Qi Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hus Ling. Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15864 2024.
An Yang, Baosong Yang. Binyuan Hui. Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng. Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-V: A GPT-4v Level MLLM on Your Phone. arXiv, abs/2408. 008000001000 01800, 2024.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. mplug-docom docool: Modularized multimodal large language model for document understanding. arXiv preprini Stti:23007002499 2023a.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, al. Ureader: Universal ocr-free risually-situate language understanding with multimodal large language model. In Findings of EMNLP, pp. 2841-2858, 2023b.
Zichun Yu, Chenyan Xiong, Shi Yu, and Zhiyuan Liu. Augmentation-adues retriever improves generalization of language models as generic plug-in. In Proceedings of ACL, pp. 2421-2436, 2023.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer, Sigmoid Loss for Language P Proceedings of ICCV, pp. 11941-1195 19952, 2023.
Image Pre-Training. In
Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, et al. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv:2405. 19327, 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. survey of large language models. ArXiv preprint, 2023.
15

---

Published as a conference paper at ICLR 2025
Tianshuo Zhou, Sen Mei, Xinze Li, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, Yu Gu, and Ge Yu. MARVEL: unlocking the multi-modal capability of dense retrieval via visual module plugin. In 2024.
Proceedings of ACL, pp.
Kunlun Zhu, Yifan Luo, Dingling Xu, Ruobing Wang, Shi Yu, Shuo Wang, Yukun Yan, Zhenghao Liu, Xu Han, Zhiyuan Liu, et al. Rageval: Scenario specific rag evaluation dataset generation framework. arXiv preprint arXiv:2408. 01262, 2024.
16

---

Published as a conference paper at ICLR 2025
A
DATA CONSTRUCTION DETAILS
A.1 SYNTHETIC DATA
Table 4: Statistics of crawled documents. We prompt GPT-4o to generate queries on these docu-
ments.
Name
Source
Description
Pages
Textbooks
https://penssax.orgg
College-level textbooks including various subjects
10,000
ICML Papers
ICMI. 2023
ICML papers various topics
5,000
NeurIPS Papers
NeurIPS 2023
NeurlPS papers various topics
5,000
Manuallib
http://www.mannakib.comm
Manuals of various kinds of products
20,000
To augment the training dataset of VisRAG, we gather additional documents from the web and utilize GPT-4o to generate queries based these documents. The sources of the collected documents listed Table The prompt employed is shown Figure
Hello, have
rich document library. Assume you cursous but very ignorant human. You eften ask me questions (queries) to seek a precise document as a reference your quastion request.
Now, you have recelved another task: document image. rich document librery based on your query. Your task now is to imagine various different angles questions that might
(Larges)
provided
*+* Your goal is accurately find this document target potential reference document candidate through queries in a very rich document library. tex:, images,
The queationa
might need referenses implicit meanings in the document.
Naximum
query answer pairs
Ralow 15 your oucput format:
"answer": "*, "quary"
"answer": "1, "query"
"answer": "7, "quary"
(
document 1)
Figure 6: Prompt for GPT-4o to generate queries, where {{ document }} is the document page.
A.2 QUERY FILTERING
significant portion of queries in VQA datasets context-dependert and thus unsuitable for retrieval. To filter out such queries, we prompt GPT-4o (OpenAl 2024) using the instruction shown in Figure 7 which includes human-annotated samples from DocVo VQA. Although this filtering step reduces context-dependent queries, a small number may still remain. However, their presence is minimal and does not significantly impact the overall quality of our dataset.
As mentioned in Sec. 3.3
17

---

Published as a conference paper at ICLR 2025
QA heze, you IMC categories:
questions divided into
The categery tA: When you this question alene without given document, you sure to find a unique document in corpua to provide a unique answer.
The calegory 4B:
When you see this question alone without given document, you will document give determinis:ic because you will find miltiple candidate documents in corpus, which may lead to di ffarent answera this quest
find
this question,
Hare are some examplas:
mentioned
right
leftside margin?
Shat is the date meationed in the second table? +B
shat Shat ia the number at the bottom of the page, in bold? t8
PUF?
Sho presented the results on cabín air quality study commerciall airoraft? #A Shat is the To whom thia ta addreased? AR
of the corporation? 4p
many shat 4A
interviews were completed during April 10th through the 12th?
subject of document/eetter?
Sho sent the letter? $8 Heading the document?
Shat the slope menziened in the fisst table? +B
what la the date in the letter? AB letter?
Shat Which part of virginia is this letter sent from? #B
botherad by cigarette odars?
which cigarette would better if offered or thicker cigarette? $2
Oigarettes will be produced and submitted to 0/0 Fanel for what purpose? 4a. What is the heading of first table?
Nhat ls RIP-6 value foz ROOL KS? 42 hetero- atoms One variable that has implicitly not been controlled? #2 Nhich corporation's letterhead this? #3
poler compounds contain?
what la the contact what the date mentioned this letter? +B Another medel of the 33mm with ventilation will made Semiworks within how Hand many weeks? +A
name mentioned in letter?
made utilizing Shat is the source? 48
level which component?
Shat Shat is the subject? 4B
heading of document?
Shat is the S.D. mentioned in the DOSE-ug 0.0000 in the third table? 4B which base paper will costed in-house with various levels of patassium
phesphate and malenic acid in order to cptimize the system? 4A which test used evaluste
menthol How much parcent had not noticad any differerce in the ador of vsss? 4A Shat the cigarette coce of RIP-6 (W/O Filter) 21/4887 Shat is the meeting date? +3
that been shipped?
How many points are there in modifications to readout instrumen:ation? AA what in the subject. of this letter?
what is the index for Reteation of Franchise? #B
Shat Shat la the full *orm of POVC? #A
heading second table?
what mm Marlboro Menthol weze juoteviyvel amoked by the Richmond Panel? Shat sert commniccationIeetee is this? tB How many one-on-one interviews were completed during April 10th through the April 12ths
During the praceas of prototype production ion and ringtipping, some cigarettes were observed have holed which pepar? FA
How many distine: rechanisms appear play role in the breakup of smoke column into multi dimensional flowfield? 4&
where the conference held?
Nho is in ze this letter? F8
Under BOLD, primary production Blend #24- will completed by which date?
Query: If query
Determine in the query belongs to Category 4& or Category #B. Output only & er B.
Figure 7: Prompt for GPT-4o to classify queries, where {{ query }} is the query to be classified.
Label B denotes context-dependent queries.
18

---

Published as a conference paper at ICLR 2025
B DOCUMENT PARSING
In this paper, we experiment with two categories of document parsing strategies: pipeline- parsing and model-based parsing.
B.1 PIPELINE-BASHH PARSING
We consider the following document parsing pipelines:
Pytesseract. Pytesseract is Python wrapper for Google's Tesseract OCR engine, offering straightforward interface for text extraction from images. Unlike more complex methods, Pytesser- act requires minimal pre-processing. By invoking the image to..ttttssttt....o string function, OCR is per- formed in single step, directly returning the extracted text. Tesseract internally handles bounding boxes, confidence scores, and orientation correction.
PaddlePaddle OCR (PPOCR) (Du et al. 2020) is widely used for doc- ument text extraction, covering text detection, classification, and recognition. First, a text detection model identifies text regions and generates bounding boxes. These regions are then processed by classification model to correct orientation issues like rotation or flipping. Next, a recognition model extracts the textual content from the corrected bounding boxes, returning recognized text with fidence scores. Only results with confidence scores above 0.6 are retained, and the bounding box coordinates, along with the recognized text, stored for further processing. We apply the following strategies to obtain the final parsing result:
PPOCR-based Methods.
To enhance text coherence, this policy combines adjacent text boxes based on vertical proximity (within 15 pixels) and horizontal alignment (within 100 pixels), reducing text fragmentation. This iterative merging process consolidates eligible text boxes into unified bounding boxes with concatenated text. Finally, the text from the remaining bounding boxes is combined with line breaks to produce the final result.
Adjacent Merging:
Layout Preserving: This policy maintains the original document structure by ordering text boxes based on their spatial positions. Spaces and line breaks are dynamically inserted to reflect horizontal and vertical gaps between text regions. This approach ensures that the extracted text mirrors the original document layout, preserving its formatting in the final result.
We run the aforementioned pipelines data, and fine-tune a MiniCPM retriever to assess performance, The results are presented in Table Methods based PPOCR demonstrate significantly better performance compared to pytesseract, with adjacent merging and layout preserving yielding similar results. Consequently, we opt to use the adjacent merging policy for our "(OCR)" runs.
dataset to obtain text-based training and evaluation
Table 5: Overall retrieval performance of different document parsing pipelines.
ArxivQA
ChariQA
DoeVQA InfoVQA
PlotQA
SQideVQA
Average
In-domain: Models Fine-tuned on Synthetic and In-domain duta
MiniCPM (Pytesscract)
41.53
72.40
70.67
76.45
55.96
79.79
66.
MiniCPM (Adjacent Merging)
58.43
77.74
72.54
83,45
64.78
91.74
74.78
MiniCPM (Layout Preserving)
55.81
75.40
71.70
83.12
63.65
91.64
73.55
B.2 MODEL-BASHD PARSING
In addition to pipeline-based methods, also employ model-based parsing approach using MiniCPM-V 2.0 directly transcribe document images into text. This method referred to as "(Captioner)".
To train this model, we collect data from two sources: a) ALLaVA (Chen et al. 2024a) (image, cap- tion) pairs. and b) VQA documents with descriptions generated by GPT-4V. We use the prompt in Figure & to instruct GPT-4V to generate detailed descriptions of documents from Doc,QA, ChartQA, VQA, InfographicsVQA, TextVQA (Singh et al. 2019), and ArxivQA.
19

---

Published as a conference paper at ICLR 2025
Bazed on the Layout information, output the text in the image. Try not medify the text, but you nead to indicate the structure such es title, body text, subtitle, table,
Note:
there are charts or geaphs, they should described detail. if you Seel that there are more than 4000 words or mast of the zext in the imaga is unclear or most the text contents the image are not writter Engl then directly
<none>.
( document 1]
Figure 8: Prompt for GPT-4V to generate page description, where {{ document }} is the docu- ment page.
We train MiniCPM-V 2.0 with batch size of 2048 and learning rate of 5e-6 for epoch.
C MODELS USED IN THIS PAPER
MiniCPM (Hu et al. 2024c) a large language model (LLM) with 2.4 billion non-embedding pa- rameters, demonstrating capabilities comparable much larger models, such Llama2-7B (Tou- vron et al. 2023) and Gemma-7B (Team et al. 2024). In this paper, we employ MiniCPM to generator (Table
construct the baseline text-based retriever (Table
SigLIP Zhai et al. 2023 is CLIP-style multi-modal model designed to align text and vision representations. We utilize SigL.IP-400m, released by Hugging Face") which incorporates Flash Attention 2, increases maximum resolution to 980x980, and adopts the NaVIT strategy to allow (a) variable resolution images and (b) aspect ratio preserved images. this paper, SigLIP is used to develop the baseline vision-based retriever (Table 2
MiniCPM-V 2.0 (OpenBMB 2024a Yao et al. 2024) is a vision-language model (VLM) with
2.8 billion non-embedding parameters, built upon SigLIP-400Mm and MiniCPM. It can process single images up to 1.8 million pixels (e.g.. 1344x1344) at any aspect ratio. We use MiniCPM-V 2.0 to build VisRAG-Ret (Table and ViRRAG-Gen (Table b)), well as the document parsing model.
MiniCPM-V 2.6 (OpenBMB 2024b Yao et al. 2024) an upgrade of MiniCPM-V 2.0 and MiniCPM-Llama3- 2.5 (Yao al.) 2024). It is built upon SigLIP-400MM and Qwen2-7B (Yang et al., 2024) with a total 8.5B parameters, exihibiting significant performance improvement over MiniCPM-Llama3-V 2.5 (Yao al. 2024). Different from previous models, MiniCPM-V 2.6 can accept multiple images as the input and perform multi-modal in-context learning. It also demonstrates stronger OCR capabilities. We use MiniCPM-V 2.6 build VisRAG-Gen (Table and text-basee exxt-baed generation baseline MiniCPM-V 2.6 (OCR) (Figure Figure 5).
Note that, MiniCPM-LLlmm3- 2.5 Yao et al. 2024 is not used in this paper.
GPT-4o OOpenA 2024 is OpenAl's latest multi-modal model, capable of processing any com- bination text, audio, image, and video inputs and generating outputs text, audio, and image formats. We use GPT-4o to construct VisRAG-Gen (Table and to synthesize training data.
D RETRIEVAL. PERFORMANCE IN RECALL 10
Table presents the retrieval performance Recall@ 10.
E PROMPTS FOR GENERATION
We present the prompts of VisRAG-Gen and TextRAG-Gen Table
20

---

Published as a conference paper at ICLR 2025
Table 6: Overall retrieval performance in Recall @ 10.
Model
# Para.
ArxivQA
ChartQA
DocVQA
IntoVQA
PlatQA
SlddVQQA
Average
Off-the-shelf
Models
BM25 (OCR)
S4.29
79.37
86.80
82.59
76.01
91.44
78.45
bge Larglare 2023 (OCR)
335M
48.65
76.19
68.19
88.16
73.12
90.11
74.07
NV-Embedry 2024 (OCR)
7,85B
70.10
88.89
89.85
95.13
80.88
97.84
87.11
LigLIP 2023
883M
44.98
77.78
68.02
84,68
58.29
89.03
70.46
Out-of-domain:
Models Fine-tined Syurhetic Data
MiniCPM (OCR)
2,728
59.07
79.37
84.26
91.64
60.25
94.78
78.23
MiniCPM (Captioner)
2.72B
55.64
82.54
79.19
92.06
57.71
90.11
76.21
SigL.I 2023
883M
60.17
82.54
75.47
84.82
59.33
92.81
7.5855
VisRAG-Ket
3,43B
81.00
84.13
87.65
97.08
71.84
95.59
86.22
(0) In-domain: Models
Fine-taned
Synthetic and In-domain data
MiniCPM (OCR)
2,728
69.36
88.89
87.14
94.15
90.61
96.85
87.83
MiniCPM (Captioner)
2.72B
85.71
SigLIP 2023
883M
73.90
92.06
83,08
93.04
89.57
94.15
87.63
ColPali 2024
2.92B
82.72
88.89
94.75
94.43
80.30
97.21
89.72
VisRAG-Ket
3.43B
87.25
90.48
91.20
97.08
97.39
Table 7: Prompt templates for generation. "Others" refers to all VQA datasets except ArxivQA.
TextRAG
VisRAG
Hint:
{( parsed document(s) }}
{{ document(s) }}
Question: [[ query }}
Question: (query)
Options:
Options:
A. {{ Option
A. {f Option 1 }}
ArxivQA
Option
B. ({ Option
C. Option
C. Option []
{{ Option }}
D. 11 Option 4 }}
directly with letter of correct option
directly with the letter of the option
the first character.
the first character.
[mage:[( parsed document(s) }}
{f document(s) }
Others
Answer the question using single phrase.
Answer the question using single word phruse. Question: {{ query }}
Question: {{ query }}
Answer:
Answer:
F CASE STUDY
We show two cases in Table 8 and Table 9 In both instances, we compare VisRAG with TextRAG, maintaining the same setup as described in the "End-to-end Performance" paragraph in Sec. 5.1
In the first case from Voc,QA, the user queries about "Club letty," however, the term "Club Jetty" in the relevant document not successfully extracted due its decorative font. This leads to TextRAG failing to retrieve the document, while VisRAG successfully retrieves it.
In the second case from Infographics VQA, although both TextRAG and VisRAG successfully re- trieve the document, TextRAG generates an incorrect response due to the loss of layout information, making it unclear which number (53% or 49%) pertains to Europe. VisRAG effectively utilizes the layout information and generates the correct answer.
21

---

Published as a conference paper at ICLR 2025
Table 8: Case study from VOA. In this case. VisRAG successfully retrieves the ground-truth document, while TexIRAG fails, leading to VisRAG's correct generation and TextRAG's incorrect generation.
TextRAG
VISRAG
Query
On which day is Club Jetty closed?
EXPLORE KAUAT
Kresh
rY
to you
Smokers
EXPRESS VMWI
Express
Airlines
PAPERDACK
Retrieved
Top-1 Document
CLUB JETTY
ASIA JOE
X Incorrect
Correct
SMOKERS- EXPRESS- Expresse- Airlneee-- Yes do
rills fort
club. repolatiose No menters only: Winh litll Nit- lick servicet support
Sinkers
EXPLOREKAUAT- (We paistent Windsuriing SULWWWINDSRRING- EXPERIENCEISS NOW
OPEN- Ia to Windheff- (comitied intrection)- Spazia intoiiactorye Lzsoin Race- on fram mestere- Ien gulloe> WAlllIA-- MARINA- RESTAURANT On de kanls of he Wallas Rivert- you'- COFFEE & NUT ROASTING CENTER c. "HOME STYLE COOKING' fomms hakol suilol diope and 26 other camoase EASY (EAINING- EXCURSIONS= RENTALS-- Phone 245-9200 ur Kunn Sui ou. 7550-- The Market Plac--hop 39 dhe Caonnu Planations Waipolli, Kassis coffee tes spics hasts Complinetarr tasopntiionn (fom Walls ues Hoch- daner only.. Phone: 522 4311 NOW! lunch daily from Il am PAPERBACK: HUT Hi. name Sunny ..t- and okn one of the moste- unique resturants in the werld-- in Litte. Kran e- It's called the Cass Blanca and ue otter Kanal's only bree gommen dining service in weyer frenlly and cassol stmospher -- We'te opon every migh frem- 500010:30 for dinner with- Brarch on Sursbey- and lice entettemmnnt in our OASIS - lounge until the was snall hous. Oe You we spssidlizaes tas in Itlian and Franch cubine with las of freshe local seafood and Kauni's- only 245-9181
Smokense- Express Airines and carying snokense in aylo this No ancaming balees iomberr mnd 18je- M Comple necoqupere Frez desimiion ana comns ara sinacimme Inlligh phone arsice- Q Die- count enise pudaages foon Smokers Travll- A sulscrip- ins to "Lat's Purty" to the official Binkerse Sunkers Expresa is the Inincilde- of Willian Wilts ande- George "Mickey" Richardson, at couple of Coooa Beadh. Florda business- mes who lite tot smoke. They organized- the club, December of last years- The club headgrareeed e- at the Space Crest aipoote- nea Cocoa Beach and-- has made inngements in lasee- up 19 spenslly opippede-- and ceaily recordiioned DC 9 Some of the destitations Bey plat serve with bos stop service foo Spice Coast exec- uttre cippote- include Orlanoo Atlatn, Chicogo --- Dallas, Las Vegas, and Atlmmic City- (Uspress Trive Migazint)e s Remal car disoorecee 1I Smoken logres discontt homee- shopping poitz- (I Creal rimires and cacopeakecee for mom- hom only - V Fuzz Lotn aitiidde tarrcchp prsccagee- - W Discouni str Seight racuu X Diaasani oupors fir desimation on seca sostamats- - Y Specii party light Li Ve and Allanic City with evers Tih and c- 1ith fight free e- 2 The bea tninel. most stenties -o suff of caployestoeess co in the induary. With the help of consultart. thes PAA) Smokers Express isc beginning the Bl Cer- onccaion process e- Those are the ABC's of moveline-- greer fin newe- smoken art where membersip ship does have neal privllgga. e- The find 50000 membendigs cree-r chorter aiee tine. Membershig anchudes zumber of special pela which you will fidd intresting.- - Menherstip is resmoted-- -n 1o persons 18 years age-- or oreeee Tiko a look ste- what memeers will mecevv yo would like nore- information abont Smokens- Express Airtnoo you on cren witte Smaken Pepross Spite South Albanic Accmcc Coers Boash Fl. 3391 (407) 783-6124 Smnkere Expras Nuntensle- Manbers Cattifiaaie- Smaken Expraa
fresh
liie
Document Parsing Result
Fresh Feait Daquifs. Call us for reseivation and frce hoiel picope- from moi nome - know you'll lice Kasii and have thee line of your lifc at the Casa Blanes an Bee Beaselleree- Ganzse Hansiimm We luve the mot complete sdiection u. of paperbakk bools on the isind... Over 5,000 books in soock OPEN EARLY- CLOSE LATE: The Markei Place at Cocomut Plantaionn Walpolli, Kanac - 822 32161 CLUBIETTY Restaunnn and Caboerte Nastlioll Boy CANTONESE FOOD
Beyar. Chesant (reaerll of
a
the clib coste $25 amaulyy and
specialt of the bossee- COMPLESTE PLSTE MuNU-inchilng TRRA--KONTTRRRAAAAA pm Cleaa THRSDDYSS MUSSC
DINNER: 5-94445
Dance by-
7:30 pm - Afler dineer Danc:
DIS00 Couney
mik-r--Lltu secee- 243.40%... hoars
HALEKO ROAD 245-0181 SUGAR MILL SNACKS
ASLAJOE MUUMUUS SOUVENIRS HANDICRAFTS IMPORTS COCONUTC- PLANTATION MARKET
VIP. Loungs at fliht iniiaiin simprts Frze smoles in dightte: Fiee headphousse
beverage seviter talinge applications parsomelt praciaally evey pest operatons. These postionse- avlliatl mem oityooyyy-R... Rea
anluuss Smoker Express
Trascl-- Cardeo
inflishe movies
Ios Creame e Beny VELVET PAINTINS. T.SHIRTS- Markei Place Ai Cocomtt Plasations 484 Kihio Ebwy Wilpoali, Kapas, Karais-
PLACE 1- Frosh Pnite Drinkoe Coldl Drink
SSunwwicho Mazalania
Walle-
M.S. Sun. LLcinde 20Noveebee
people-Stecke Steooe- Burposs--
922-9981c
baiing happ- Inmalee-
Answer
Mondays X Incorrect
Tuesdays v Correct
22

---

Published as a conference paper at ICLR 2025
Table 9: Case study from InfographicsvOO. In this case, both VisRAG and TextRAG successfully retrieve the correct document: however, only VisRAG effectively leverages the layout information, enabling accurate generation. In contrast, TextRAG suffers from information loss of the layout, resulting incorrect responses.
TextRAG
V&RRG
Query
What percent account holders Europe using LinkedIn for linding joh?
Social media job seeking trends
Makaii
contone LinkedIn
anquuarty contr une gion thsight vaiee sniter you shnliys
What propurtinn of apouut hakees
93% %
30%
Retrieved Top-1 Document
dnnppenbrratee bur bad your
53
49%
40%
Facebook
Choggto pai't Seanrs, Pacilooa
1%
13
92%
Michael Pagge
Both Correct
Social mediat- job seeking trends Michael Page's annual global survey of financial services and banking+- employees was conducted in April 2014.more than 3,300 people participatede- Linkedln Linkedin's 's pupularity continues grow, though many job seekers don't think of it oft- their strategy.So hirers need to look to other sourcing chunnels tooc- What pro- portion of account holders use Linkedin for job seeking?+- 93 8 30% of respon- dents have amaccount-upe- 10% from last yeart- more womenc- than men saye- they don't 't haver- an accounte- 53%++ In Europee- 49%-- North Americat 40% In the UK+ + Facebooke- Despite last year's hype around Graph Search, Facech..acboookoo hasn' It made any progress with monetising:- its recruitment potential jjobseekess remain very negative about Facebook playing any parte= 13% said they'd be happye- see adverts 92% said they would not bee- happy be contacted bye- recruiter on Facebookt 1% Don't bank on social media Michael Page brings you broader range of talent, and jobse- www.michaelpage.co.. htwp://g..oo.aa/alaaaaaeeaa ausaaaarrrrr respondentse- (who are job seekers) said theye- would use look for jobs+- MichaelPagee- Financial Servicese- Specialists financial services recruitmente- www.mmclallage.oo.....
Document Parsing Result
Answer
49% Incorrect
53% Correct
23

---

Published as a conference paper at ICLR 2025
G
ADDITIONAL RETRIEVAL AND GENERATION RESULTS
Table 10: Additional retrieval performance in MRR @ 10.
Model
ArxivQA
CharlQA
DocVQA
InfoVQA
PlotQA
SlideVQA
Average
Out-of-domain: Models Fine-tuned
Synthetic Data
MiniCPM (OCR)
47.96
61.64
67.04
79.36
36.04
87.93
63.33
SigLIP 2023)
46.81
68.40
57.61
67.12
31.92
85.14
59.50
(OCRC + SigLIP (RRF)
54.07
72.33
65.46
75.32
38,98
88.06
65.70
(c) In do-miin; Models Tine-tueed on Synthetic and In-domain data
MiniCPM (OCR)
58.43
17.74
12.54
83.45
64.78
91.74
74.78
SigLIP (2023)
59.16
81.34
64.60
74.59
61.32
89.08
71.68
(OCR) + SigLIP (RRF)
64.19
85.39
71.75
80.88
66.09
92.94
76.87
Table 11: Additional generation performance in accuracy (%). All models and methods utilize the same retriever, VisRAG-Ret. Performance relative to Oracle is colored in blue.
Model/ Method
Input
ArxivQA
ChariQA
DocVQA
InfoVQA
PlatQA
SudeVQQ
Average
(b) VisRAG-Gen:
Single-image VLM (MiniCPM-V22 2.0)
lop-6
$9.19
22.
14.72
15.60 791
(M0/79)
25.41 (00)
Page Concatenation
top-10
56.74 (94 MS)
20.63 (52.0%)
10.32(19.50)
13.93 (4088)
17.15
22.84 (58.0r8)
23.60 (56 5%)
Oracle
60.29 (1005)
39.60 (1009))
$2.96
34.12 (100%)
23,06 (1009)
39.39 (100/83
41.58 (100%)
VisRAG-Geu Gen: Muli--iaae VLM
top-6
67.89 (95 998
17.14
70.SS (54.19)
51.25 (908
35.81
1923
$1.00
(90:99)
55.06 (81 899)
MiniCPM-V V 2.6
lop-10
64,95 (91.4%)
(8..7))
54.48 (65.49))
36.49 (57.398)
30.94 (9944499490
$1.80 (80.79)
49.30 (22.8%)
Oracle
71.08 (1005))
68.25 (10098)
83.25 (1005))
63.65 (10095)
62.69 (100%)
57.73 (1909))
67.78 (10055)
Top-1
66.30 (94.99)
$3.97 (73.5)
65.82 (73,55)
55.71 (8)
31.33 (04.0%)
55,38 (85.19)
58. (AU0PS)
Qwen2-VL
top-2
65.44 (91 598
52.38 (21 2963
70.90 (81.4%)
55.15 (85 598
47.05 (58 2563
58.99 (90.45))
58. 298)
top-3
67.03 (95 88)
$7.14 (78.3%)
73.60 (84.5%)
52.79 (81.9%)
44.96
(56. 193
58.63 (89.8%)
59.03 (81.05)
Oracle
69.98 (10ms)
73.02 (10095)
87.14 (1008))
64.48 (10095)
80.19 (1006)
(1006))
73.35 (100163
In this section, we present supplementary evaluation results for both retrieval and generation on our
dataset.
Table shoshows additional retrieval results obtained by applying reciprocal rank fusion (RRF) (Cor- mack et al. 2009) to combine the outputs of MiniCPM (OCR) and SigLIP. It is a straightforward method to integrate textual information extracted from the page with visual clues. The results indicate that fusing text and image modalities provides a meaningful performance boost over in- dividual modality baselines. However, this approach still falls short of the performance achieved by our VisRAG-Ret model (71.49 for out-of-domain, 77.91 for in-domain). This underscores the superior capability of VisRAG-Ren in understanding both modalities within a unified architecture
Table II provides additional generation results using top-6 and top-10 10 retrieved documents from VisRAG-Ret. For these experiments, we evaluate the performance of MiniCPM-V 2.0 using the page concatenation method and MiniCPM-V 2.6 with direct feeding. We also report the perfor- mance of another SOTA VLM, Qwen2-VL-7BB -nssrrcct (Wang et al., (024). The results indicate significant performance degradation when handling a larger number of retrieved pages, for both page concatenation (MiniCPM-V 2.0) and multi-page input (MiniCPM-V 2.6). MiniCPM-V 2.6 exhibits greater robustness increasing context compared MiniCPM-V 2.0. Open-source VLMs still face challenges in reasoning over multiple pages and extracting relevant information from noisy retrieved data. Results for Qwen2-Vi VL demonstrate stronger document understanding capabilities, outperforming MiniCPM-V 2.6 in these tasks.
H RETRIEVAL EFFICIENCY
In this experiment, we evaluate the retrieval efficiency of VisRAG-Ret and MiniCPM (OCR) by measuring two key components: offline document parsing and encoding latency, and online query encoding and search latency. Query and document encoding are conducted on an NVIDIA A100 40G GPU with batch size of 1, while document parsing is performed on single core of an Intel Xeon Platinum 8350C CPU. The reported latencies are averaged over the queries and documents from the PlotQA dataset. The results are summarized in Table [2]
As shown in the table, although VisRAG-Ret a VLM-based model, requires more time for document encoding compared to MiniCPM (OCR), it bypasses the time-consuming parsing stage required by
24

---

Published as a conference paper at ICLR 2025
Table 12: Retrieval efficiency (ms). We report offline latencies per document, including document parsing and encoding latencies, as well as online latencies per query, including query encoding and search latencies.
Offline Latency per Document
Online Lateney per Query
Parsing Encoding
Total
Encoding
Search
Total
MiniCPM (OCR) VisRAG-Ret
284
28
312
28
26
54
121
121
28
26 54
MiniCPM (OCR). This leads to a 58% reduction in total document processing time for VisRAG-Ret For online query processing, the latencies of VisRAG-Ret and MiniCPM (OCR) are nearly identical, as the queries consist solely textual inputs.
I RETRIEVAL PERFORMANCE ON TEXT RETRIEVAL BENCHMARKS
Table 13:
Retrieval performance on subsets of the text retrieval benchmark BEIR (Thakur et al.
2021 in NDCG@ 10. VisRAG-Ret performs retrieval on rendered document screenshots.
Model
SciFact
NFCorpus
Scidoes
MiniCPM (OCR)
61.04
14.12 27.02
13.01
VisRAG-Ret
62.47
16.25
To evaluate how VisRAG-Ret performs retrieval scenarios involving only textual data, we conduct an experiment using the BEIR (Thakur al. (022I) text retrieval benchmark. To evaluate VisRAG- Ret, we convert the document texts into rendered screenshots and apply VisRAG-Ret to this modified dataset. We use the Pillow[] library to convert text documents into screenshots, setting width of 800px, a font size of 24px, and the Deja VuSans font. The height of each screenshot varies depending on the document length, with margin of 20px and line spacing of 4px. For comparison, include MiniCPM (OCR) in the evaluation, utilizing raw textual data directly available in BEIR. Note that the term "OCR" in MiniCPM (OCR) is used solely for naming consistency.
As shown in Table 13 VisRAG-Ret, relying only on the rendered screenshots, significantly outper- forms MiniCPM (OCR) which uses textual information. This result highlights that VisRAG-Ret's pooling-base representation effectively captures textual details and is well-suited for text-heavy document retrieval.
hhttps///pthhon-plllowooy
25


Reviews:
[{"review_id": "uYgHYCZmsa", "reviewer": "Reviewer_A94Q", "summary": "This paper proposes VisRAG, a multimodal retrieval-augmented generation pipeline. Different from traditional RAG methods, which rely on text parsing to retrieve information from visual content, this paper utilizes a VLM-based retriever and generator to navigate information relevant to the query and generate responses. The VisRAG design consists of two main stages: (1) Retrieval: Given a query, the VLM retrieves a set of relevant images from the dataset leveraging the cosine similarity between the query embedding and image embeddings; (2) Generation: the VLM uses a combination of retrieved images and the query to produce responses. VisRAG is compared with a series of baselines and show advantages in both retrieval and generation capabilities.", "questions": "1. Significance of position-weighted mean pooling: How would the model perform with these designs? Would the proposed position-weighted mean pooling produce significantly better results?\n2. Effects of concatenated multiple images: Would this affect VLM performance, particularly when handling a larger number of retrieved images (e.g., more than five)?\n3. Efficiency of VLM-based retrieval: Would using the VLM significantly increase retrieval time? How would the number of retrieved images affect retrieval time?\n4. Clarity on accuracy measurement: How is accuracy measured? Is an additional LLM used to evaluate the textual responses?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. [Originality] The proposed method is promising. With the VLMs showing increasing capabilities in understanding images, encoding images with VLMs is a natural improvement from traditional text-based retrieval methods.\n2. [Quality] The paper presents relatively comprehensive experiments, demonstrating advantages in both retrieval and generation capabilities of the proposed methods.\n3. [Significance] The proposed method clearly outperforms various baselines.\n4. [Clarity] The paper is generally well-written and easy to follow.", "weaknesses": "Major concerns:\n1. In the paper, when retrieving relevant images, the authors adopt position-weighted mean pooling over the last-layer VLM hidden states. This choice seems heuristic-based. Although the paper points out that VLMs are with causal attention, potentially emphasizing weights on later tokens, after multiple VLM layers, we could expect information to propagate across most tokens. In fact, previous works often use the leading token or simply learn a linear layer from all tokens. How would the model perform with these designs? Would the proposed position-weighted mean pooling produce significantly better results?\n2. When using multiple images to generate responses, one method proposed in the paper is image concatenation, which is performed horizontally. This could change the resolution of the final image. Would this affect VLM performance, particularly when handling a larger number of retrieved images (e.g., more than five)?\n3. The paper does not report retrieval efficiency compared to text-based methods. Would using the VLM significantly increase retrieval time? How would the number of retrieved images affect retrieval time?\n\nMinor comments:\n1. When measuring the performance of generation, how is accuracy measured? To me, this is particularly unclear for synthetic data, where ground truth answers and queries are generated by GPT-4. Is an additional LLM used to evaluate the textual responses?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qPBhlntBBg", "reviewer": "Reviewer_gQ1V", "summary": "The paper introduces VisRAG to address the limitations of text-only RAG systems by incorporating vision-language models (VLMs). VisRAG enables the processing of documents as images. By training the VisRAG retriever with open-source and synthetic data and exploring various generation methods, experimental results show VisRAG's superiority in both retrieval and generation tasks. The authors also plan to make their code and data publicly available.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Improve text-only RAG by establishing a vision-language model (VLM)-based RAG pipeline.\n2. The experiment is thorough under the designed scenario.\n3. The authors have promised to make the data and code open source.", "weaknesses": "1. Lack of novelty: For retrieval, it uses the dual-encoder-style and just changes them to VLM embeddings. The position-weighted mean pooling is also off-the-shelf. For generation, they only try a few straightforward strategies.\n2. Lack of design for document understanding: There is a lack of design considerations specifically for document understanding. The proposed method is not tailored for document understanding scenarios. Alternatively, why not validate the method on non-document (common image) datasets?\n3. Confusion regarding the application: Why does a person always need the assistance of document images when asking general questions? For example, in Line 884, Why would a person directly ask a question like 'Where were the two magnesium containing papers made at?' without reference to any document? If he already has this document as a reference, why does he still need to do a retrieval process at first?\n4. Why only experiment with text-only or image-only features, not image+text(+layout)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bqrUn3C8Fk", "reviewer": "Reviewer_ZBwP", "summary": "The paper focus on the research of vision-language model (VLM)-based RAG\nPipeline and propose a novel method named VisRAG. It  broadens the external data processed by RAG from text to image. To achieve the goal, the authors introduce two vision-optimized modules: VisRAG-Ret and VisRAG-Gen. This VisRAG-Ret directly utilizes the images without extracted textual content and establish  the map between the query and the documents in the latent space. The VisRAG-Gen leverage the help of the existing LLMs and VLMs for generating the answers. The image processing technology is integrated to realize the efficient generation of multiple inputs. Moreover, a new dataset is built combining a vision question answering (VQA) dataset and synthetic data.", "questions": "1. In the comparison process, Colpali and Vis RAG-Ret should be fine-tuned based on the same data, rather than using the pre-trained model to compare directly. Please add more comparing results.\n2.The article needs to provide more comparison results with the current best models, such as GPT o1, Qwen and other leading models that can directly input image for VQA task.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "In this paper, the authors propose an image-based RAG method named VisRAG, and construct a new dataset to verify the relevant effects. This paper does a lot of comparative experiments to verify its innovation. A large number of state-of-the-art methods have been combined and proven to be effective.", "weaknesses": "The method in this paper is mainly based on the retrieval of image data, which is basically consistent with the text retrieval framework, and lacks innovation. In addition, similar to the comparative model Colpali's innovation, the VisRAG is fine-tuned with a new dataset and new VLM which is not sufficient to express its innovation in multimodality area.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fsxco5AM8D", "reviewer": "Reviewer_kCGC", "summary": "This paper proposes VisRAG, a model that leverages vision-language models (VLMs) for retrieval-augmented generation in multi-modal documents. The framework comprises two components: VisRAG-Ret for retrieval and VisRAG-Gen for generation, utilizing a data construction method that combines visual question answering (VQA) datasets with synthetic data. Experiments demonstrate that VisRAG outperforms traditional RAG in both retrieval and generation tasks, exhibiting improved training data efficiency and generalization capabilities. It achieves a significant relative increase in end-to-end accuracy, highlighting its potential to replace text-based RAG for multi-modal documents.", "questions": "See \"weaknesses\"", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.  The paper provides a clear and detailed description of the VisRAG framework., which makes it easy to understand.\n2. The experimental results on document-related VQA tasks are promising, suggesting that the proposed method could be a practical solution.", "weaknesses": "1. The rationale behind the proposed VisRAG-Ret is unclear. If no OCR information is included, how can the target image be retrieved, especially for answers that require precise text content? From my perspective, a pooling-based representation feature does not offer sufficient information.\n\n2. The experimental setup is not sufficiently convincing, indicating that the VQA task alone cannot fully demonstrate the method's effectiveness in the retrieval task. In other words, employing such a complex pipeline to address the VQA task is akin to using a sledgehammer to crack a nut.\n\n3. The methods discussed are insufficient. To the best of my knowledge, there are several document-specific MLLMs, such as mlug-docowl, ureader, and text monkey, among others. While these works may not be directly related to RAG, they should at least be mentioned and analysized.\n\n4. In my view, the comparison is unfair, as both Minicpm and Siglip can not only perform VQA tasks but also handle other tasks related to document image understanding and content perception.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)