ColPali: Efficient Document Retrieval with Vision Language Models
Authors: Manuel Faysse
Hugues Sibille
Tony Wu
Bilel Omrani
Gautier Viaud
CELINE HUDELOT
Pierre Colombo
Venue: ICLR 2025 Poster (2025)
Keywords: document embeddings
vision language models
late interaction
document retrieval
information retrieval
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Document Retrieval from page images using Vision Language Models and Late Interaction

Abstract: Documents are visually rich structures that convey information through text, but also figures, page layouts, tables, or even fonts. Since modern retrieval systems mainly rely on the textual information they extract from document pages to index documents -often through lengthy and brittle processes-, they struggle to exploit key visual cues efficiently. This limits their capabilities in many practical document retrieval applications such as Retrieval Augmented Generation (RAG).
To benchmark current systems on visually rich document retrieval, we introduce the Visual Document Retrieval Benchmark $\textit{ViDoRe}$, composed of various page-level retrieval tasks spanning multiple domains, languages, and practical settings. 
The inherent complexity and performance shortcomings of modern systems motivate a new concept; doing document retrieval by directly embedding the images of the document pages. We release $\textit{ColPali}$, a Vision Language Model trained to produce high-quality multi-vector embeddings from images of document pages. Combined with a late interaction matching mechanism, $\textit{ColPali}$ largely outperforms modern document retrieval pipelines while being drastically simpler, faster and end-to-end trainable. 
We release models, data, code and benchmarks under open licenses at https://hf.co/vidore.

Published as a conference paper at ICLR 2025
ColPali: EFFICIENT DOCUMENT RETRIEVAL WITH
VISION LANGUAGE MODELS
Manuel Faysse Hugues Sibille*1,4
Tony Wu
Bilel Omrani'
Gautier Viaud'
Céline Hudelot
Pierre Colombo
MMliin Technology
EEquallai $CentralcSupélce, Paris-Saclay
*ETH Zürich
manuel. faysse@centralesupplee...r
ABSTRACT
Documents are visually rich structures that convey information through text, but also figures, page layouts, tables, or even fonts. Since modern retrieval systems mainly rely on the textual information they extract from document pages index documents -often through lengthy and brittle processes-, they struggle to exploit key visual cues efficiently. This limits their capabilities in many practical docu- ment retrieval applications such as Retrieval Augmented Generation (RAG). To benchmark current systems on visually rich document retrieval, we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of various page-level retrieval tasks spanning multiple domains, languages, and practical settings. The inherent complexity and performance shortcomings of modern systems motivate new concept; doing document retrieval by directly embedding the images of the document pages. We release ColPuli, Vision Language Model trained produce high-quality multi-vector embeddings from images of document pages. Combined with a late interaction matching mechanism, ColPali largely outperforms modern document retrieval pipelines while being drastically simpler, faster and end-to-end trainable. We release models, data, code and benchmarks under open licenses https://hf. co/vidore
INTRODUCTION
Document Retrieval consists of matching user query relevant documents in a given corpus. is central to many widespread industrial applications, either as standalone ranking system (search engines) or as part of more complex information extraction or Retrieval Augmented Generation (RAG) pipelines.
Over recent years, pretrained language models have enabled large improvements in text embedding models. In practical industrial settings, however, the primary performance bottleneck for efficient document retrieval stems not from embedding model performance but from the prior data ingestion pipeline. Indexing standard PDF document involves several steps. First, PDF parsers Optical Character Recognition (OCR) systems are used to extract words from the pages. Document layout detection models can then be run segment paragraphs, titles, and other page objects such tables, figures, and headers. chunking strategy then defined to group text passages with some semantical coherence, and modern retrieval setups may even integrate a captioning step to describe visually rich elements in a natural language form, more suitable for embedding models. In our experiments (Table 2). typically find that optimizing the ingestion pipeline yields much better performance visually rich document retrieval than optimizing the text embedding model.
Contribution 1: ViDoRe. In this work, argue that document retrieval systems should not be evaluated solely on the capabilities of text embedding models (Bajaj et al. 2016 Thakur et al. 2021; Muennighoff et al., (022). but should also consider the context and visual elements of the documents to be retrieved. To this end, we create and openly release ViDoRe. a comprehensive benchmark to evaluate systems on page-level document retrieval with wide coverage of domains, visual elements, and languages. ViDoRe addresses practical document retrieval scenarios, where
"Equal Contribution

---

Published as a conference paper at ICLR 2025
Standard Retrieval sú 0.68 NDCG@S
offine
online
GID D3 223 dio
Simillarity score
Test
Text Node
Model
WT&?
7.22s page
OCR
Layout detection
22ms / query
ColPali (ours)
fd 0.81 NDCG@5
Similarity
offline
Vision LLM
onfine
Maxdiin
What
LLM
secodar
Maxtin
Jry
0.39s /age
30ms / query
Figure 1: ColPali simplifies document retrieval w.r.t. standard retrieval methods while achieving stronger performances with better latencies. Latencies and results are detailed in section 5 and subsection B.4
queries often necessitate both textual and visual understanding for accurate document matching. We highlight the shortcomings of current text-centric systems in these settings
Contribution 2: ColPali. We propose a novel concept and model architecture based on Vision
Language Models (VLMs) efficiently index documents purely from their visual features, allowing for subsequent fast query matching with late interaction mechanisms (Khattab & Zaharia 2020) Our method, ColPali, significantly outperforms all other retrieval systems on ViDoRe while being fast and end-to-end trainable. These results demonstrate the potential and the many benefits of this novel Retrieval Vision Space concept, which could significantly alter the way document retrieval approached in the industry moving forward. We release all resources at https://hht http//hf....vvvorre..e/
2 PROBLEM FORMULATION & RELATED WORK
Problem Setting, In our setting, retrieval system scores how relevant a document d from corpus D is with respect to a query 9. Computing the similarity score s(q. d) E for each of the D documents in the corpus creates ranking we can use to extract the most relevant documents. In
this work, we focus on page-level retrieval: given a query, is the correct document page retrieved by the system? ? For coherence with existing literature, we further use the term document to refer to individual pages, i.e. the atomic retrieved elements in our setting. As we focus on practical industrial retrieval applications (RAG, search engines) with potentially large corpora sizes, latency constraints are imposed on scoring systems. Most current retrieval systems can be decomposed into (1) an offline indexation phase in which document index is built and (2) an online querying phase in which query is matched to documents from the index and where low latency is vital to the user experience.
Under these industrial constraints, identify three main properties efficient document retrieval systems should exhibit: (RI) strong retrieval performance, as measured by standard retrieval metrics; (R2) fast online querying, measured through average latencies; (R3) high throughput corpus indexation, ie. the number of pages that can be embedded in a given timeframe.
'The ViDoRe benchmark leaderboard is hosted publicly at httpps///bugiingfaacccc/ssspssss
vidore- leaderbard encourage further developments.
2

---

Published as a conference paper at ICLR 2025
2.1 TEXTUAL RETRIEVAL METHODS
Document Retrieval in Text Space.
Statistical methods based on word frequency like TF-IDF (Sparek Jones 1972) and BM25 (Robertson) et al. T994) are still widely used due to their simplicity and efficiency. More recently, neural embedding models based on fine-tuned large language models display state-of-the-ar performance on variety of text embedding tasks and top the retrieval leaderboards (Muennighoff et al. 2022
Neural Retrievers. bi-encoder models (Reimers & Gurevych 2019 Karpukhin et al. 2020
Wang et al. 2022). documents are independently mapped offline to a dense vector space. Queries are embedded online and matched to documents through a fast cosine distance computation. slower, but slightly more performant alternative, cross-encoder systems (Wang et al., 22200 Cohere 2024) concatenate query and document as single input sequence and iteratively attribute matching scores to each possible combination. This enables full attention computation between query and document terms but comes at the cost of computational efficiency, as |D] encoding passes must be done online.
Multi-Vecton retrieval via late interaction. In the late interaction paradigm introduced by CoIBERT (Khattab & Zaharia 2020), an embedding is pre-computed and indexed per document token. At runtime, similarity can be computed with individual query token embeddings. The idea is benefit from the rich interaction between individual query and document terms while taking advantage of the offline computation and fast query matching enabled by bi-encoders. See (sccttioonn El for more details.
Retrieval Evaluation. Although benchmarks and leaderboards have been developed to evaluate text embedding models (Thakur al., 2021 Muennighofi et al. 2022). much of the performance improvements in industrial use cases of embedding models stem from the prior data ingestion pipeline. While documents often rely on visual elements to more efficiently convey information to human readers, text-only systems barely tap into these visual cues. Other work has also independently studied table or chart retrieval systems through repurposed Question Answering datasets (Zhang et al. 2019. Nowak et al. 2024) but only assessing specialized methods for each task.
To our knowledge, no benchmark evaluates document retrieval systems in practical settings; in an end-to-end manner, across several document rypes and topics, and by evaluating the use of both textual and visual document features.
2.2 INTEGRATING VISUAL FEATURES
Contrastive Vision Language Models. Mapping latent representations of textual content to corre- sponding representations of visual content has been done by aligning disjoint visual and text encoders through contrastive losses (Radford et al., 202T Zhai al. 2023). While some OCR capabilities exist in these models, the visual component is often not optimized for text understanding.
The Fine-grained Interactive Manguagg-Immgee Pre-training YYoo al. 2021) framework extends the late interaction mechanism to cross-modal Vision Language Models, relying on max similarity operations between text tokens and image patches.
Visually Rich Document Understanding. To beyond text, some dlccmment-focuseee models jointly encode text tokens alongside visual or document layout features (Appalaraju et al. 2021 Kim
et al. 2021 Huang et al. 2022; Tang et al. 2022). Large Language transformer Models (LLMs)
with strong reasoning capabilities have recently been combined with Vision Transformers (ViTs)
(Dosovitkkiy et al.] 2020) to create VLMs (Alayrac et al. 2022 Liu et al.) 2023 Bai et al.) 2023
Laurençon et al. 2024b) where image patch vectors from contrastively trained ViT models (Zhai et al.
2023) are fed as input embeddings to the LLM and concatenated with the text-token embeddings.
PaliGemma. The PaliGemma-3B model (Beyer et al. 2024) extends concepts from Pali3 (Chen et al. (2023), and projects SigLIP-SO400m//1e Alladuumohssnn et al. (023) patch embeddings into Gemma-28 2B's text vector space (Gemma Team et al. 2024). Along with its reasonable size w.r.. other performant VLMs, an interesting property of PaliGemma's text model is that it is fine-tuned with full-block attention on the prefix (instruction text and image tokens). See Appendix E for more details
VLMs display enhanced capabilities in Visual Question Answering. captioning, and document understanding (Yue al. 2023), but are not optimized for retrieval tasks.
3

---

Published as a conference paper at ICLR 2025
3 THE ViDoRe BENCHMARK
Existing benchmarks for contrastive vision-language models primarily evaluate retrieval for natural images (Lin et al. 2014; Borchmann et al. [202 Thapliyal et al.) 2022). On the other hand, textual retrieval benchmarks (Muennighoft et al. 2022) are evaluated at at textual passage level and not tailored for document retrieval tasks. We fill the gap with ViDoRe, a comprehensive benchmark for document retrieval using visual features.
3.1 BENCHMARK DESIGN
ViDoRe is designed to comprchensively evaluate retrieval systems on their capacity to match queries to relevant documents the page level. This benchmark encompasses multiple orthogonal subtasks, with focuses on various modalities text, figures, infographics, tables; thematic domains medical, business, scientific, administrative; or languages English, French. Tasks also span varying levels of complexity, in order to capture signals from both weaker and stronger systems. As many systems require large amounts of time to index pages (captioning baannngg---eddde-e approaches can take dozens of seconds per page for instance), we limit the number of candidate documents for each retrieval task in order to evaluate even complex systems in a reasonable timeframe without sacrificing quality. For trainable retrieval systems, we provide reference training set that can be used to facilitate comparisons.
Dataset
Language
Queries
# Documents
Description
Academic Tasks
VQA
English
500
500
Scanned documents from UCSF Industry Infographics scrapped from the web High-quality financial reports
V
English
500
500
TAT-DQA
English
1600
1600
arXiVQA
English
500
500
Scientific Figures from arXiv Tables scrapped from the web
TabFQuAD
French
210
210
Practical Tasks
Energy
English
100
1000
Documents about energy Administrative documents
Govemment
English
100
1000
Healthcare
English
100
1000
Medical documents
AI
English
100
1000
Scientific documents related to AI Environmental reports
Shift Project
French
100
1000
Table 1: ViDoRe comprehensively evaluates multimodal retrieval methods.
Academic Tasks. We repurpose widely used visual question- -antronnnnnweringgg benchmarks for retrieval tasks: for each page-question-answer triplet, we use the question as the query, and the associated page as the gold document (Table T). These academic datasets either focus on single specific modalities (Mathew et al.) 2020. 2021 Li et al. 2024) or target more varied visually rich documents (Zhu et al., 2022). Morcover, consider TabFQuAD, a human-labeled dataset tables extracted from French industrial PDF documents released with this work. Details can be found in lubsecctionn A.I
Practical tasks. We construct topic-specific retrieval benchmarks spanning multiple domains to go beyond repurposed QA datasets and evaluate retrieval in more realistic industrial situations (e.g. RAG). To achieve this, collect publicly accessible PDF documents and generate queries pertaining to document pages using Claude-3 Sonnet, a high-quality proprietary vision-language model (Anthropic) 2024). In total, we collect 1,000 document pages per topic, which we associate with 100 queries extensively filtered for quality and relevance by human annotators. The corpus topics are intentionally specific maximize syntactic proximity between documents, creating more challenging retrieval tasks and covering an array of orthogonal domains Table 1
Evaluation Metrics. We evaluate performance on our benchmark (Requirement R1) using standard metrics from the retrieval literature (nDCG, Recall @ K. MRR). We report nDCG@ 5 values as the main performance metric in this work and release the complete sets of results along with the model
Answers are generated alongside queries to (1) ground queries and improve their quality and (2) provide resources to foster future work.
httpp:://uggingg: co/vidore
4

---

Published as a conference paper at ICLR 2025
To validate compliance with practical industrial requirements (section 2 we also consider query latencies (R2) and indexing throughputs (R3).
3.2 ASSESSING CURRENT SYSTEMS
Unstructured. We evaluate retrieval systems representative of those found in standard industrial RAG pipelines. As is common practice, we rely on the Unstructured off-the-shelf tool in the highest resolution settings to construct high-quality text chunks from PDF documents. Unstructured orchestrates the document parsing pipeline, relying on deep learning vision models to detect titles and document layouts (Ge et al. 2021). OCR engines (Smith, 2007) to extract text in non-native PDFs, specialized methods models to detect and reconstruct tables, and implements chunking strategy (by-title) that leverages the detected document structure to preserve section boundaries when concatenating texts. As is common practice, in our simplest Unstructured configuration (lext-only), only textual elements are kept and figures, images, and tables are considered noisy information and filtered out.
Unstructured + X. While Unstructured is strong baseline by itself, we further augment Unstructured output by integrating the visual elements. In OCR), tables, charts, and im- ages are run through an OCR engine, processed by Unstructured, and chunked independently. In (+ Captioning), we set up fully-fledged captioning strategy (Zhao et (023). in which we feed visual elements to a strong proprietary Vision Language Model (Claude-3 Sonnet (Anthropic) (024)) obtain highly detailed textual descriptions of the elements. Both strategies aim to integrate visual elements in the retrieval pipeline but incur significant latency and resource costs (subsection 5.2).
Embedding Model. To embed textual chunks, we evaluate Okapi BM25, the de facto standard sparse statistical retrieval method, and the dense encoder of BGE-M3 (Chen et al. (202)), multilingual neural method with SOTA performance in its size category. Chunks are embedded and scored independently, and page-level scores obtained by max- max-pooling over the page's chunk scores
Contrastive VLMs. We also evaluate the strongest available vision-language embedding models; Jina CLIP (Koukounas et al. 2024), Nomic Embed Vision (Nomic 2024). and SigLIP- MIgLIP-S0400m// (Alabdulmohsin et al. 2023).
Results. From performance perspective, best results obtained by combining the Unstructured parser with visual information, either from captioning strategies or by running OCR on the visual elements (Talle 2p. Little difference is between BM25 and BGE-M3 embeddings highlighting the visual information bottleneck. Contrastive VLMs lag behind. Beyond retrieval performance (R) ), the indexing latencies (R2) reported in Figure 2 illustrate that PDF parsing pipelines can be very lengthy, especially when incorporating OCR or captioning strategies. Querying latencies at runtime (R3) are very good for all evaluated systems (< 22 ms on NVIDIA L4) due to fast query encoding and cosine similarity matching.
PDF Parser (7.22s)
Layout Detection
Siglip (0.12s)
OCR Captioning
ColPali (0.39s)
Page Encoding
3
4
6
7
Latency (s)
Figure 2: Offline document indexing with ColPuli is much simpler and faster compared to standard retrieval methods. The PDF Parser results are obtained following the Unstructured settings with BGE-M3 detailed in |subsection 3.2) All indexing speeds are averaged per-page latencies. More details in subsection B.4
www. www.uunsttuuuue.dcoo
We empirically validated the max-pone mox-ppoling strategy over sub-page chunks to be more effective than concate- nating all page chunks before embedding pagewise.
5

---

Published as a conference paper at ICLR 2025
4
LATE INTERACTION BASED VISION RETRIEVAL
4.1 ARCHITECTURE
Vision-Language Models. Encouraged by their strong document understanding capabilities, we propose adapting recent VLMs for retrieval. The key concept is to leverage the alignment between output embeddings of text and image tokens acquired during multi-modal tine-tuning. To this extent, we introduce ColPali, Paligemma-3BB extension that is capable of generating ColBERT-style multi- vector representations of text and images (Figure T). PaliGemma-3B small size, the many released checkpoints fine-tuned for different image resolutions and tasks, and the promising performances on various document understanding benchmarks. We add projection layer to map each of the language model's output token embeddings (whether from text or image tokens) to a vector space of reduced dimension D 128 as used in the ColBERT paper (Khattab & Zaharia 2020) keep lightweight -ag-od-embedding representations.
a strong candidate due to its
Given query q and document d, we denote as Eg E RNNDD and Ea E IR NaxD their respective multi-vector representation in the common embedding space ED, where N₂ and Na are respectively the number of vectors the query and in the document page embeddings. The late interaction operator, LI (g. d), is the sum over all query vectors Eq (j), its maximum dot product (-I-) with each of the Na document embedding vectors Ed(1:Na).
Late Interaction.
LI dq
Σ IEILA, JEllL,Nall
max
(Eg
Ea 6))
(I)
Contrastive Loss. The Late Interaction operation is fully differentiable, enabling backpropagation. Let a batch (gs, d& [e]]_,6]] composed of query-page pairs, where for all & E [11, bl]. the document page de is the document corresponding query qu. Following Khattab & Zaharia (2020), we define our in-batch contrastive loss L as the softmaxed cross-entropy of the positive scores 5 LI (gk, de)
w.r.t. to the maximal in-batch negative scores 5 = max Li7k
LI (gx: di 16
6
exp (st)
C log
exp (st) exp (85)
=
Σ log (1 + exp (o% -s))
(2)
4.2 MODEL TRAINING
Dataset. Our training dataset of 118,695 query-page pairs is comprised of train sets of openly available academic datasets (63%) and synthetic dataset made up of pages from web-crawled PDF documents and augmented with VLM-generated (Claude-3 Sonnet) pseudo-questions (37%). Dataset split details are given in subsection A.3] Our training set is fully English by design, enabling us to study zero-shot generalization to non-English language:] We explicitly verify no multi-page PDF document is used both ViDoRe and in the train set to prevent evaluation contamination. A validation set is created with 2% of the samples to tune hyperparameters. We openly release the training datase for reproducibility and to encourage further research.
Parameters. All models trained for I epoch on the train set. Unless specified otherwise, we train models boloat 16 format, use low-rank adapters (LoRA, Hu et al. (202T) with a = 32 and r = 32 on the transformer layers from the language model, as well the final randomly initialized projection paged_adamw layer, and use a paged_as adamw 8bit optimizer. We train on an GPU setup with data parallelism, a learning rate of 5e 5 with linear decay with 2.5% warmup steps, and a batch size of 32.
Query Augmentation. As in Khattab & Zaharia (2020), we append 5 <unused0> tokens to the query tokens to serve soft, differentiable query expansion or re-weighting mechanism.
WWe reformulate the loss to leverage the numerically stable softplus function where softplus(x) =
log (1 + exp (z:)
Multilingual data present the pretraining corpus of the language model (Gemma-2B) and potentially occurs during PaliGemma-3B's multimodal training.
/https://hugginn http://hwg.n.eeee.eooaaaatttttta co//aatasett/viioorr/c/ollllll train set

---

Published as a conference paper at ICLR 2025
5 RESULTS
ArxivQ DucQ
InfoQ
TabF
TATQ
Shift
A1
Energy
Gov.
Health.
Avg.
Unstructured uxii-att
BM25
34.1
44,0
59.6
90.4
78.3
78.8
82.6
BGGE-M
28.4.57
36.1
68.5
88 4_50
76.81
77.7
84.6110
Unstructured
BM25
31.6
36.8
62.9
46.5
62.7
64,3
92.8
85,9
83,9
87.2
65.5
BGE M3
31.4,02
25.7.m
60. pp
70.8
50.5
p3
90.2,
83.6
84,9
mm
1996
Unstructured Cipriietng
BM25
40.1
38.4
70,0
35.4
61.5
60.9
88.0
84.7
82.7
89.2
65.1
-GG-M3
35.7.44
32.9154
71,9tL9
69.1
1887
43.8Lm
73.1
88.8-0a
83.3
80,4
91.3
67.0
Contrastive VLMs
Jina-CLIP
25.4
11.9
35.5
20.2
3.3
3.8
15.2
19.7
21.4
20.8
17.7
Nomic-vision
17.1
10.7
30.1
16.3
12.9
10.9
11.4
15.7
SigLIP (taaio
43.2
30.3
64.1
58.1
26.2
18.7
62.5
65.7
66.1
79.1
51.4
Ours
SigLIP (Vaadio
43.2
30.3
64.1
58.1
26.2
18.7
62.5
65.7
66.1
79.1
$1.4
BiSigLIP (dfne-iniig)
58.5-153
32.9r
70.5
62.7:46
30.5
26.5
74.3-188
73.7
74.2
82.3
58.611
BiPali (ILMM)
56.5..3
30.0.a9
67.4
76.9
33.4
43.7
71.2.41
61.9
73.8
73.6,
88.8
ColPali (+Izas beac
79.1-326
54.4
81.8
83.9
65.8
73.2
96.2-250
91.0
92.7
94.4
81.3
Table 2: Comprehensive evaluation of baseline models and our proposed method on ViDoRe. Results are presented using @ metrics, and illustrate the impact of different components. Text-only metrics are not computed for benchmarks with only visual elements.
5.1 PERFORMANCE (R1)
We show performance is achieved iteratively through the combination of three factors; (1) carefully crafted task-specific dataset, (2) pairing a pretrained LLM to a vision model to better leverage text semantics from the image, and (3) using multi-vector embeddings rather than a single vector representation to better capture the vast amount of visual information present in document.
Fine-tuning Vision Model on document retrieval oriented dataset: BiSigLIP. SigLi is strong vision- lision-language bi-encoder producing single vector embeddings, and pretrained on billions of image-text pairs from the English split of WebLI (Chen et al., 2023). Further tnne-tunig the textual component of this model on our document-oriented dataset (BiSigLIP) yields clear improvements across the board, particularly on figure retrieval (ArxivQA) and table retrieval tasks (TabFQuAD).
Feeding image patches to a LLM: BiPali. In the PaliGemma model architecture, HigLIP-generatee patch embeddings are fed to a text language model and we can obtain LLM contextualized output patch embeddings 10 This technique aligns the image token representations with the text token embeddings in the LLM's embeddings space, and augments the vision model embeddings with the language model's 's text understanding capabilities. We average pool these representations to obtain single dense vector, effectively creating a PaliGemma bi-encoder model (BiPali). After fine-tuning on the training dataset,
obtain model that performs slightly worse in English than the tuned BiSigLIP variant [1] However, we see notable improvements in French tasks, indicating that BiPali's LLM (Gemma 2B) helps multilingual text understanding. This particularly notable as our training dataset does not contain non-English samples.
Leveraging Vector Embeddings through Late Interaction: ColPali. One benefit of inputting image patch embeddings through language model is that they are natively mapped to a latent space similar the textual input (query). This enables leveraging the ColBERT strategy to construct one
hhttps://hugging Face co/ google/ /oggoogl///iglpppp s0480m- patch1 4- 384
Note that the SigLIP model used in PaliGemma slightly differs in terms of number patches 1024 patches for PaliGemma's vision encoder, and 729 for the standalone SipLIP model.
"This can be explained by the fact that contrary to SigLIP, the original PaliGemma is not trained on contrastive matching tasks, but rather next token prediction. Our contrastive fine-tuning phase on 119K images to transform PaliGemma into bi-encoder is 5 orders of magnitude smaller than SigL.IP's original contrastive training.
7

---

Published as a conference paper at ICLR 2025
embedding per image patch token, and at inference compute all interactions between text tokens and image patches, resulting in a step-change improvement in performance compared to BiPali. Results in Table shohow that our ColPali model also largely outperforms the strong baselines based on Unstructured and captioning, as well as all evaluated text-image embedding models. The difference is particularly stark the more visually complex benchmark tasks, such as Infographic VQA, ArxivQA, and TabFQuAD, respectively representing infographics, figures, and tables. However, text-centric documents are also better retrieved by the ColPuli models across all evaluated domains and languages, making our approach the overall perf-perg document-retrieval model.
Negative Results. For extensiveness, we also train ColSigLIP, late interaction variant of the BiSigLIP model but obtain abysmal performances. We attribute this to the large gups w.r.t. SigLIP's pre-training, in which only pooled latent representation used in the contrastive loss, which does not optimize the representations of individual patch and token embeddings. Similarly, we train a BiSigLIIP PatiGemma variant, in which we retrieve the image representations from the SigI.IP model that has been further updated by PaliGemma fine-tuning, and use the text representations from PaliGemma's 's text model. After fine-tuning on our dataset, performance is severely inferior to SigLIPy anailla Vanaila which simply encodes with SigLIP's original text and vision components. This indi- cates a logical misalignment between SigLIP embeddings, and Gemma embeddings after PaliGemma training. We detail these results in subsection C.T
5.2 LATENCIES & MEMORY FOOTPRINT
Online Querying. (R2) Logically, querying latencies differ between ColPali and BGE-M3 embedding model. For BGE, encoding takes about 22 ms for 15 tokens, while encoding query with ColPali's language model takes about 30 m 2 For smaller corpus sizes, computing the late interaction operation induces marginally small overheads (= 1 ms per 1000 pages in the corpus), and the cosine similarity computation between bi-encoder vectors is even faster. Optimized late interaction engines (Santhanam et al. 2022 Lee al. 2023) enable to easily scale corpus sizes to millions of documents with reduced latency degradations.
Offiine Indexing. (R3) Standard retrieval methods using bi-encoders represent each chunk as single vector embedding, which is easy to store and fast to compute. However, processing a PDF to get the different chunks is the most timecconsuiing part (layout detection, OCR, chunking), and using captioning to handle multimodal data will only exacerbate this already lengthy process. On the other hand, ColPali directly encodes pages from their image representation. Although the model is larger than standard retrieval encoders, skipping the preprocessing allows large speedups at indexing 13 FFigure 2). As pages are embedded end-to-end in single forward pass, the VRAM usage depends exclusively on the sequence length (number of patches per image) which is fixed as well, enabling efficient batching strategies fully leverage hardware acceleration. ColPali also benefits from most LLM efficiency improvements introduced in the ecosystem such as Flash Attention (Dao) 2023).
Storage Footprint. Our method requires storing a vector per image patch, along with extra text tokens "Describe the image" concatenated to image patches. We project each PaliGemma vector to a lower dimensional space (D 128) to maximize efficiency, leading to a memory footprint of 257.5 KB per page (subsection B.3). Importantly, the memory footprint of the naive ColBERT indexing strategy can be drastically improved through compression and clustering mechanisms (Santhanam)
et al. 2022 Clavié et al. 2024).
Token pooling. Token pooling (Clavié et al. 2024) a CRUDE--cmpliant method (document hatpu//ddceetioo--fnniyy. that aims amountto reduce the amount of multi-vector embeddings. For ColPali, many image patches share redundant information, e.g. white background patches. By pooling these patches together,
reduce the amount of embeddings while retaining most information. Retrieval performance with hierarchical mean token pooling on image embeddings is shown in Figure (left). With a pool factor of the total number of vectors is reduced by 66.7% while 97 97.8% of the original performance is maintained. We note that the Shift sttaset-commpsed of the most text-dense documents- iss a clear outlier, showcasing more information dense documents contain less redundant patches and may be prone to worse performance degradation with such pooling techniques.
"Computed for a batch size of (online), and averaged over 1000 queries. Sccssucsniiion B.4 13 Meeasurss NVIDIA 1.4 GPU, averaged on 100 pages, with batch size of 4 pages for ColPali and 8 text. chunks for Bi-Encoders. On average, a page divided into 2.1 chunks. See Scesuussection H.4
8

---

Published as a conference paper at ICLR 2025
2019 Average Hourly Generation by Fuel Type
Relative NDCG@5 (%)
100
98
96
Dalsset
Shift
94
92
Energy
Tabf TabFQuad Quad
90
DocVQA
88
ArcivGA
Average
86
7
Pool Factor
Query: 'Which hour of the day had the highest overall eletricity generation in 2019?"
Figure 3: (Left: Token Pooling) Relative performance degradation when reducing the number of stored embeddings per document. (Right: Interpretability) For each term in a user query, ColPali identifies the most relevant document image patches (highlighted zones) and computes query-to- page matching score.
5.3 INTERPRETABILITV
By superimposing the late interaction heatmap on top of the original image, we can visualize the most salient image patches with respect to each term of the query, yielding interpretable insights into model focus zones. As epitomized in in/Figure 3) (right), we observe ColPali exhibits strong OCR capabilities as both the words "hourly" and "hours" " present a high similarity score with the query token <_hour>, We also note particular focus on other non-trivial image features such the x-axis representing hours being salient. Other visualization examples are shown in Appendix
ABLATION STUDY
We run various ablations to better understand the mechanisms at play. By default, result deltas reported below refer to nDCG@5 values averaged over all ViDoRe tasks. Detailed results in
Tradeoffs between model size and the number of image patches. We train a variant of PaliGemma with half the number of image patches (512). While we observe a clear performance degradation with respects to to the 1024-patch ColPali model (-24.8 nDCG@5) 5), memory usage is much lower. As an alternative to PaliGemma. architecture and based on a Mistral- (Jiang et al. 2023) language backbone and a SigLIP vision encoder paired with perceiver resampler. The most notable differences with PaliGemma lie in the size of the language model (2B and 7B resp.) and the number of image patches (between 512 and 2048 for PaliGemma, and 64 post-resampling for Idefics? Our results suggest better language models enable more efficient representations of image embeddings Colldefics2 with 64 patches largely outperforms out ColPali with 512 patches 1+20.1 nDCG@5) 5). However Colldefies2 (64) remains less accurate than ColPali (1024) (-4.7 nDCG@5) while being about twice as slow in terms of training and inference latency. These results suggest there are tradeoffs between performance (R1), latencies during online querying (R2) and offline indexation phases (R3), and index memory size.
train Idefics2-8B (Laurençon et al., 2024b), VLM with similar
Unfreezing the vision component. We train ColPali variant by also backpropagating through and updating the vision encoder and the projection layer. This leads to a slight performance degradation (-0.7 nDCG@5). These conclusions may change with larger scales of training data.
Impact of "query augmentation" tokens. In CoIBERT, special tokens are concatenated to the input query to serve as soft query augmentation buffers. Training without these tokens, we observe no significant performance difference in the English benchmarks. However, performance the French
tasks seems to improve (+9.8 nDCG@G on Shift, +6.3 nDCG@5 on TabFQuAD, Table 7
Impact of the Pairwise CE loss. Training with an in-batch negative contrastive loss, instead of the pairwise CE loss that only considers the hardest negative sample, leads to slight performance degradation (-1.6 nDCG@5) the aggregated benchmark.
Wiith the option of adding 4 sub-image crops of 64 tokens each the sequence, for total of 320 tokens

---

Published as a conference paper at ICLR 2025
Adapting models to new tasks. Contrary to more complex multi-step retrieval pipelines, ColPali can be trained end-to-end, directly optimizing the downstream retrieval task which greatly facilitates fine-tuning to boost performance on specialized domains, multilingual retrieval, specifie visual elements the model struggles with. To demonstrate, we add 1552 samples representing French tables and associated queries to the training set. This represents the only French data in the training set, with all other examples being kept unchanged. We see clear nDCG @ 5 improvements (+2.6) and even starker Recall I gains (+5) on the TabFQuAD benchmark, with no performance degradation on the rest of the benchmark tasks (+0.4 nDCG@5 overall).
Better VLMs lead to better visual retrievers. As improved VLMs are released, it interesting to observe if improved performances on generative tasks translate once these models are adapted for image retrieval tasks through ColPali training strategies. We train the recently released Qwen2-VI. VI. 2B (Wang et al. 2024b), a SOTA 2 billion parameter generative VI.,, with the same data and training strategy, obtaining ColQwen2- VI.. To approximately match ColPali's memory requirements, we limit the number of image patches to 768, slightly less than ColPali's 1024 patches. We observe clear performance improvements of +5.3 nDCG@5 values over ColPali showcasing clear performance correlations between generative benchmarks performance and retrieving metries.
Out-of-domain generalization. Some of the datasets in the ViDoRe benchmark have train sets, which have integrated within the ColPali train set (eg. academic tasks). This is standard in embedding models [Wang et al.) 2024a Lee et al., 2024), and while ColPali also exhibits strong performance tasks in which this is not the case (French data is never seen by the model during training for instance), it remains interesting to evaluate model performance when training is done on a fully disjoint data distribution. We train a ColPali variant solely using the recent DocMatix dataset (Laurençon et al, 2024a). a large scale, synthetically annotated visual document question answering dataset, which we subsample to obtain comparably-sizee train set. Results on ViDoRe show the performance drop is minor (-2.2 nDCG@5), still outperforming the closest baseline method by over 12 points. These results showcase ColPali generalizes well outside of its training distribution, and demonstrate that our results are not unreasonably boosted with respect to baselines (BGE-M3) that cannot be fine-tuned on the same data 15
7 CONCLUSIONS
In this work, we introduced the Visual Document Retrieval Benchmark (ViDoRe), which evaluates document retrieval systems in realistic settings involving visually complex documents. We demon- strated that current retrieval pipelines and contrastive vision-language models struggle to efficiently exploit visual information embedded in documents, leading to suboptimal performance. To address this, we presented ColPali, novel retrieval method that leverages Vision- Languag Models to create high-quality, multi-vector embeddings purely from visual document features. ColPali largely outperforms the best existing document retrieval methods while enabling faster corpus indexing times and maintaining low querying latencies, thus circumventing many pain points of modern document retrieval applications. We hope to drive industrial adoption, and to encourage future work by publicly releasing the ViDoRe benchmark, the data, the codebase, and all models and baselines from our work.
Future Work. Beyond performance improvements that could be obtained through better data, backbone models or training strategies, our vision at term is to combine visual retricval systems and visually grounded query answering to create end-to-end RAG systems that purely function from image features. This idea is supported by concurrent work (Ma et al.) (2024) showcasing the strong promises of VLMs for visual QA, and may eventually become a new industrial standard for document processing. In this line of work, reliability is key, and confidence estimation techniques for Information Retrieval methods could become central implement abstention mechanisms (Gisserot, Boukhlef et al. 2024), and are particularly interesting given the information rich multi-vecte scoring mechanisms of late interaction systems. Expanding benchmarking efforts cover more languages, modalities, and tasks is also a crucial future research direction (Jiang et al. 2024).
1o train with data resembling the one BGE-M3 models would see at inference time would require running complex extraction pipelines for the more than 100K documents in the training set, notably relying on external proprictary captioning models which is both too costly and lengthy. This is not needed to train vision-based models.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
For transparency, reproducibility and to foster future work, release our training data, model checkpoints (adapters), entire codebase, and complete evaluation benchmark under MIT licenses as detailed in the main paper. We also host public ViDoRe leaderboard to foster concurrent work in the space. The supplementary material further details training configurations for our models (also specified in HuggingFace model repositories), and dives into the process used to generate synthetic data, how latency computations are performed, as well as provides further detailed evaluation results.
ACKNOWLEDGEMENTS
This work is partially supported by Illuin Technology. and by a grant from ANRT France. This work was performed using HPC resources from the CINES ADASTRA through Grant 2024-AD01 1015443 and from IDRIS with grant 101157224R... We extend our warm thanks Jonathan Dong, Caio Corro, Victor Pellegrain and Ender Konukoglu for their valuable feedback on the paper.
REFERENCES
Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting ViT in Shape: Scaling Laws for Compute-Optiimal Model Design. 2023, doi: (0.48550/ARXIV220055
13035. URL http:////aaai..oorbb/2223.. 13035
Publisher: arXiv Version Number: 5.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Jain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford. Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei. Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a Visual Language Model for Few-Shot Learning. 2022. doi: 00.48550/ARXI:222444 14198. URL httpps///axxi org/abs/ aaaaabs/22222 14198
Publisher: arXiv Version Number: 2.
Anthropic. The Claude 3 Model Family; Opus, Sonnet, Haiku, 2024. httpa//owooooooeeeooooooo/aaaaaa 887777777
URL
8857627/
Model_Card_Claudee pdf
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. DocFormer: End-to-End Transformer for Document Understanding, 2021. http:::stt:///r/t wwwpaaiiiv.orgg/aaeeee wwwp:/rg//////////// / Version Number: 2.
2106. 11539
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. 2023. doi: 10.48550/ARXIV.23308 12966. URL
12966
Publisher: arXiv Version Number:
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song. Alina Stoica, Saurabh Tiwary, and Tong Wang. MS MARCO: A Human Generated MAchine Reading COmpre- hension Dataset, 2016. URL https:///rxxii /tttps////arii.oor/////11111 09268 Version Number: 3.
Lucas Beyer, Andreas Steiner, André Susano Pinto. Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. Paligemma: versatile 3b vlm for transfer, 2024. URL https://arxiv.orrt
abs/2407.60726

---

Published as a conference paper at ICLR 2025
Burton H. Bloom. Space/time trade-offs in hash coding with allowable errors. Communn ACM, 13(7): 422 4224 July 1970. ISSN 0001-0782. doi: 10.1145/362686.3622600 URL http:: org/
1e. 1e11111166666.... 362692
Place: New York, NY, USA Publisher: Association for Computing
Machinery.
Lukasz Borchmann, Michat Pietruszka, Tomasz Stanislawek, Dawid Jurkiewiez. Michal Turski, Karolina Szyndler, and Filip Graliński. DUE: End-to-End Document Understanding Benchmark. In Thirty-fisth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. BGE M3-Embedding: Multi-Granularity Text Embeddings Through Self-Knowledge Multi-Lingual,, UtLhttts:://arx/ wwwp:iiv.ogg////////e/ee 03216 Version Number: 3. Distillation, 2024.
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic, Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers, Xiaohua Zhai. and Radu Soricut. PaLI-3 Vision Language Models: Smaller, Faster, Stronger, 2023. URL https://arxx: org/ htg/abs//o 2210 09199 Version Number: 2.
Benjamin Clavié, Antoine Chaffin, and Griffin Adams. Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling, 2024. URL /ttp:://s///r org/ abs/2409. 1 4683 Version Number:
Cohere. Introducing Rerank 3: A New Foundation Model for Efficient Enterprise Search & Retrieval, April 2024. URL htttt::::/tthhee comboe///ooo/ooobo//r/rrar
Tri Dao. Flashattention-22: Faster attention with better parallelism and work partitioning, 2023, URL
http:://arxiv.orr/ab////3 08691
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision Transformers Need Registers. 2023. doi: 00.48550/ARXIV.20 16588. URL httpps://arriv.orr/aaa///////
16588 Publisher: [object Object] Version Number: 2.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018. URL https://////rrir.rrrr /bs//
1810.04805
Version Number: 2.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. 2020. doi:
11929. WRLLhhtps:://aat /tttps://arivv. .or..iioorr///ooooo wtw.://ggab//....ooo 11929 11929 Publisher: arXiv Version Number: 2.
Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. YOLOX: Exceeding YOLO Series in 2021, 2021. URL httpp::///axxiv.or////ba//
08430 Version Number: 2.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito. David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan. George Tucker, George-Christian Christian Muraru, Grigory Rozhdestvenskly. Henryk Michalewski. Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Caan-saptsste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comaneseu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinherg, Wojeiech Stokowiec, Yu-hui Chen, Zafarali Ahmed,
12

---

Published as a conference paper at ICLR 2025
Zhitao Gong. Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open Models Based on Gemini Research and Technology, 2024. URL https://arxxi.orr//bb////
08295
Version Number: 4.
Hippolyte Gisserot- Manuel Faysse, Emmanuel Malherbe, Céline Hudelot, and Pierre Colombo. Towards trustworthy reranking: A simple yet effective abstention mechanism, 2024.
URL https://aaxxvv.rgg//b///////////
12997
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation Large Language Models. 2021. doi: 10.48550/ Publisher: arXiv Version
URXIV.2106.09682 URL httpp:://arkie org abs/2106. 09685
Number: 2.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking. 2022. doi: 10.4850/ARXXV.224.0002 URL hhtpps://arkiie wttp://.rrrabb///////a// 08387 Publisher: arXiv Version Number: 3.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Aman Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Místral 7B. 2023. doi: 10.48550/ARXIV.2311 06825. URL /ttpss org abs//31/ .68255
Publisher: arXiv Version Number: 1.
Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. Vhm2vec: Training vision-languag models for massive multimodal embedding tasks, 2024. URL https
//arxiv.org/abs//20 05162
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering, 2020.
URL www.aaiii.iogaa///////ao 04986
Version Number: 3.
Omar Khattab and Matei Zaharia. ColBERT: Efficient and Effective Passage Search via Con- textualized Late Interaction over BERT. 2020, doi: 10448550ARXII.2000.220
URL
https://arxiv. http:///axxxv..ogabbbb//// 2004./220.1111 12832
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. OCR-free Document Un- derstanding Transformer, 2021. /ttp:://s/// atgg///b///1..... 15664 Version Number: 5.
Andreas Koukounas, Georgios Mastrapas, Michael Günther, Bo Wang, Scott Martens, Isabelle Mohr, Saba Sturua, Mohammad Kalim Akram, Joan Fontanals Martínez, Saahil Ognawala, Susana Guzman, Maximilian Werk, Nan Wang, and Han Xiao. Jina CLIP: Your CLIP Model Is Also Your Text Retriever, 2024. WtLLhhtss://arxtt https/////aaxxx.ooo/a//oo/ wttp:o////bbb///gg// 20204, Version Number: I.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understand- ing linion-language models: insights and future directions., 2024a.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- language models?, May 2024b. URL http:/ttpp///xii.orrrt org absp///.../2.... 02246 2XXi:24455002266
[cs].
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Ny-embed: Improved techniques for training Ilms generalist embedding models, 2024. URL httpss///aaxxi htt.:/ogg/////////eeo/oo 22050 17428
Jinhyuk Lee, Zhuyun Dai, Sai Meher Karthik Duddu, Tao Lei, Iftekhar Naim, Ming-Wei Chang, and Vincent Y. Zhao. Rethinking the Role of Token Retrieval in Multi-Vector Retrieval, 2023. URL
01982 Version Number: 3.
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vasion-language models, 2024.
13

---

Published as a conference paper at ICLR 2025
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects in Context, 2014. URL httthtt///axxii.or///aaass 1405. 0312 Version Number: 3.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. 2023. doi: 0088550/ARXV.2334000050 Version Number: 1.
URL
08488 Publisher: arXiv
Lucas Beyer*. Andreas Steiner*, André Susano Pinto*, Alexander Kolesnikov*, Xiao Wang*, Xiaohua Zhai*, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Jeremiah Harmsen, Daniel Keysers, Neil Houlsby, Xi Chen, Emanuele Bugliarello, Thomas Unterthiner, Keran Rong, Matthias Minderer, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Julian Eisenschlos, Manoj Kumar, Matko Bošnjak, Matthias Bauer, Fangyu Liu, Adam Gryener. Alexey Gritsenko, Paul Voigtlaender, Pinclopi Papalampidi, Olivier Henaff, Skanda Koppula, Xi Xiong, Radu Soricut, Model release contributors and general support. Tris Warkentin, Kat Black, Luiz Gustavo Martins, Glenn Cameron, Raj Gundluru, Manvinder Singh, Meg Risdal, Nilay Chauhan, Nate Keating, Nesh Devanathan, Elisa Bandy, Joe Fernandez, Antonia Paterson, Jenny Brennan, Tom Eccles, Pankil Botadra, Ben Bariach, Lav Rai, Minwoo Park, Dustin Luong, Daniel Vlasic, Bo Wu, Wenming Ye, Divyashree Sreepathihalli, Kiranbir Sodhia, Alek Andreev, Armand Joulin, Surya Bhupatiraju, Minh Giang, Joelle Barral, and Zoubin Ghahramani. PaliGemma, 2024. URL https://www/ wew.ae..gg...oo.oo.ooo..... cww. m/ 23393
Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li. Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, and Aixin Sun. Mmlongbench-doe: Benchmarking long-context document understanding with visualizations, 2024. http:::ttt:/rttt http://aaxxww.og/bbbbbe orgg/bs///00/.0// .1....155 523
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Doc A Dataset for VQA on Document Images. 2020. doi: 10.46550/ARXXV.2007..0000 2007 00398
URL httpps///axxi.... htap::://axxivoor///al/ org/abs/
Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V Jawahar. Infographic VQA, 2021. URL htt.:ttss//aarii..orr//////2//./ 12756 Version Number: 2.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive Text Embed- ding Benchmark, 2022.
07316 Version Number: 3.
Nomic. Nomic Embed Vision: Expanding The Nomic Latent Space, June 2024. URL https:
//blog. http://aa/pppssss//o/mmeeeeeaaee
Averi Nowak, Francesco Piccinno, and Yasemin Altun. Multimodal chart retrieval: A comparison of text, table and image based approaches. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp- Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 307
10.18653/w1/2024.naaea. mac.020000...ac-oooo..007077 URL htttp:://aclanthooooo..o../
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. 2021. doi: 10.48550/ ARXIV.2103.00022 URL httppp/://arxiw.oggbb///////
Publisher: arXiv Version
Number: 1.
Nils Reimers and Iryna Gurevych. Sentence- Sentence Embeddings using Siamese BERT- Networks, 2019, URL lnttps://arxiv.orr//aa//
10084 Version Number: 1.
Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Baanoock-eealiieu and Mike Gatford. Okapi at TREC-3. In Donna K. Harman (ed.), Proceedings of The Third Text REtrieval Conference, TREC 1994, Gaithersburg. Maryland, USA. November 2-4, 1994, volume 500-225 of NIST Special Publication, pp. 126. National Institute of Standards and Technology (NIST), 1994, URL http://trec.. nist. gov/ http://tt....ooleleeeeeee//////s ps.gz
14

---

Published as a conference paper at ICLR 2025
Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. PLAID: An Efficient Engine for Late Interaction Retrieval, 2022. URL httpp::////:: wtwg/b///////..aa 09707 Version Number: 1.
R. Smith. An Overview of the Tesseract OCR Engine. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007) Vol pp. 629-633, Curitiba, Parana, Brazil, September 2007. IEEE. ISBN 978-0-7695-28222 doi: 10. 10.1109000
URL
http://ieeexploree wtt.:eeeeeeeeeeeeeoeereeeoooree 1
ISSN: 1520-5363.
Karen Sparek Jones. A
OF TERM SPECIFICITY AND ITS APPLICATION IN RETRIEVAL. Journal of Documentation, 28(1):11-21, January 1972. ISSN 0022-0418. doi: 10.1 108/cb02656..
BRL/htpps:/0 //t::///wwww emerald. wtt.cco////ii/e cwwm/innsiggtt/ootttt//o///
10.1100/eb22226/ful//mm
Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. Unifying Vision, Text, and Layout for Universal Document Processing, 2022. URLLhttps://arae .rgg at////b///22/ 02623 Version Number: 3.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych, BEIR: A Heterogenous Benchmark for Zero- shot Evaluation of Information Retrieval Models, 2021. URL
https://arxiv. wttp://wwww//bb//////ee/ae 08663 Version Number: 4.
Ashish Thapliyal, Jordi Pont-Tuset, Xi Chen, and Radu Soricut. Crossmodal--6600: Massively Multilingual Multimodal Evaluation Dataset, 2022. URL http:://araiv.orr/aa///////// 12522 Version Number: 2.
L Liang Wang, Nan Yang. Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text Embeddings by Weakly-Supervised Contrastive Pre-training, 2022. URL Version Number:
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models, 2024a. URL https://arxxv.oor//bbaa 00368
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024b. URL https://aa/ org/ abs//4/9. 12191
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou MiniLM: Deep Self-Attention Distillation for Task Compression TreiTrained Transformers, April 2020. URL httpp://axxiv wwwpaaawwww...rrab///bb///////// 10957 10957 arXiv:2002.10955 [cs].
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe NIu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained Interactive Language-Image Pre-Traine Tre-Training, 2021. URL /https://arxiv http://xwwvoog/bbbb/// wttp/////./......//////// 07783
Version Number: 1.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen. MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI, 2023, URL ttttps://axxi..og//bbssss
16502
Version Number:
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer, Sigmoid Loss for Language Image Pre- Training. 2023. doi: 10.48550/ARXI1.23031.1 15343. htt//stss///axxiii org/ / 2303. 15343
Publisher: [object Object] Version Number: 4.
Li Zhang, Shuo Zhang, and Krisztian Balog. Table2vec: Neural word and entity embeddings for table population and retrieval. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development Information Retrieval, SIGIR '19, ACM, July 2019. doi: 10.1 141//33111443331333
http://p doi. www.ddddoooooog////// org/10. /wt../../g///./11/331111111 0101111111113333...... 3331333
15

---

Published as a conference paper at ICLR 2025
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding. Xiaobao Guo, Minzhi Li, Xingxuan Li, and Shafiq Joty. Retrieving Multimodal Informa- tion for Augmented Generation: A Survey, 2023. URL BRLLhttps: /tttts://arxiv..om http:///...rrgaa///33.333/ Version Number: 3.
abs/2303. 10868
Fengbin Zhu, Wengiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua. Towards Complex Document Understanding By Discrete Reasoning. 2022. doi: 10. 00.4850//RXX..2207. 11871. URL http::htpt://tttse http://aaxxvv.oograb/////// wrgg/bs//220..... 11871 Publisher: arXiv Version Number: 3.
A BENCHMARK DATASETS
A.I ACADEMIC DATASETS
VOA (Mathew et al. 2020 includes collected images from the UCSF Industry Documents
Library. Questions and answers
manually annotated.
InfoVQA (Mathew et al.
2021) includes infographics collected from the Internet using the search query "infogruphics". Questions and answers were manually annotated.
TAT-DQA (Zhu et al., 2022 is a large-scale Document VQA dataset that was constructed from publicly available real-world financial reports. It focuses rich tabular and textual content requiring numerical reasoning. Questions and answers were manually annotated by human experts in finance.
arXivQA (Li al. 2024) VQA dataset based figures extracted from arXiv publications. The questions were generated synthetically using GPT-4 Vision.
TabFQuAD (Table French Question Answering Dataset) is designed evaluate TableQA models in realistic industry settings. We create additional queries to augment the existing human-ann aanaan-dd ones using the same method described in subsection A.2
A.2 PRACTICAL DATASETS
Methodology. Creating a relevant retrieval dataset close to real use cases is a major challenge as the dataset needs be both sufficiently large for effective fine-tuning and sufficiently diverse to cover broad range of modalities (full text, tables, charts, ...), domains (industry, healthcare, ...). and query-document interactions (extractive questions, open-ended questions, ...). Our approach to building this dataset involves several steps: (1) we use web crawler to collect publicly available documents on various themes and sources, (2) we convert these PDFs into a series images, one per page, and (3) we generate queries related to each image using VLM.
Web-Crawler. We implemented a web crawler to efficiently collect large volumes of documents related to given topic. The crawler is seeded with user-defined query (e.g. "artificial intelligence"") and then uses GPT-3.5 Turbo to brainstorm related topics and subtopics. This query augmentation strategy aims at both broadening and deepening the search. GPT-3.5 Turbo is further used to generate diverse search queries from each subtopic. This query set is then consumed by pool of parallel workers whose job is fetch the associated most relevant documents. We use SerpAP along with a filetype filter (PDF documents only) to programmatically scrape Google Search rankings. Each file hashed and stored in Bloom filter (Bloom) 1970) shared among workers avoid duplicate documents in the final corpus. Unique scraped files are downloaded, and inserted into SQLite database along with additional metadata.
Datamix. Using the web crawler, we collected approximately 100 documents for each of the following four seeds: "energy", "government reports", "healthcare industry"" and "artificial intelligence" " These seeds were meticulously hand-picked to align with real-use cases for retrieval models and visually rich pages. We also removed all documents containing any private information.
Query Generation. To increase the efficiency of our query generation scheme and to limit API calls, we generate at most questions per image. From all the documents collected, we randomly sample 10,000 images per theme and call Claude-3 Sonnet with the following prompt:
httpsss//seeppp..ccom
16

---

Published as a conference paper at ICLR 2025
are an assistant specialized in Multimodal RAG tasks.
The task is the following: given an image from pdf page, you will have to generate questions that can be asked by user to retrieve information from large documentary corpus.
The question should be relevant
page, and should not too specific or too general. The question should be about the subject of the page, and the answer needs to be found in the page.
Remember that the question is asked by user get some information from large documentary corpus that contains multimodal data. Generate a question that could be asked by user without knowing the existence and the content of the corpus.
Generate as well the page. the format question.
the question, which should be found in the answer should list of words answering
Generate at most THREE pairs of questions and answers per page in dictionary with the following format, answer ONLY this dictionary NOTHING ELSE:
"questions": [
"question": "XXXXXX",
"answer": ["YYYYYY"]
}.
{
"question": "XXXXXX",
"answer": ["YYYYYY"]
},
( "question": "XXXXXx",
"answer": ["YYYYYY"]
}.
]
where XXXXXX is the question and ['YYYYYY'] is the corresponding list of answers that could be long needed.
Note: If there are no questions to ask about the page, return an empty list. Focus making relevant questions concerning the page. Here is the page:
Human Validation. We manually validate every single synthetically created query ViDoRe to ensure quality, query relevance, and consistency with the benchmark objective of evaluating retrieval in practical industrial settings. During this step, we randomly assign document-pain queries to volunteer annotators and instruct them to filter out queries that do not fit the above-listed criteria. We also instruct annotators to flag any documents they deem to contain PII information or content not suited for academic benchmark. No flag our prior PDF collection strategy. 100 queries per topic are collected in this manner. Annotators colleagues and collaborators of the authors who volunteered help. Each annotator spent approximately 3 hours filtering the larger query set down to 100 high-quality queries per topic.
raised during the entirety of the process, validating
A.3 TRAINING DATASET
The statistics of the train set are given the following table. The creation the train set follows the same methodology in subsection A.2, We made that PDF document cannot have pages in both the training set and the test set to prevent data leakage and that there are no duplicate documents in each split.
17

---

Published as a conference paper at ICLR 2025
Dataset Split
Split Size
Language
Domain
VQA
39,463
English
Scanned documents from UCSF Industry
VQA
10,074
English
Infographies scrapped from the web
TATDQA
13,251
English
High-quality financial reports
arXivQA
10,000
English
Scientific Scientific Figures from arXiv
Scrapped PDFs
45,940
English
Varied PDFs from 3885 distinct URL domains
TOTAL
118,695
English-only
Mixed
Table 3: Details on the different splits in the dataset used train ColPali.
B IMPLEMENTATION DETAILS
B.I CODEBASE
The codebase is written in PyTorch and leverages HuggingFace tooling for model implementations and trainers
B.2 HYPERPARAMETERS
Hyperparameters are tuned a validation split composed of 2% of the training dataset. We find bi-encoder methods to be more sensible to learning rate variations than late interaction- models and achieve the best performance for all models with a learning rate of 5e 5. We experiment with LoRA rank and values and do not notice particular improvements past batch sizes are kept small due to long sequence lengths that complicate scaling past 4. We simulate larger batch sizes with multi-GPU training and train with total batch size 6 32 with accumulation, for epoch on our training set.
= 32. Per-device
B.3 EMBEDDING SIZE
Minimizing storage footprint be essential industrial retrieval systems if databases contain millions of documents. With this criterion in view, we have compared the embedding sizes of the models in our study. As shown in Table 4 ColPali's embedding size is an order of magnitude larger than BM25 and two orders of magnitude larger than BGE-M3. However, in practical scenarios, embeddings by centroid cluster, or quantizing embeddings to binary representa- can reduce storage costs by two orders of magnitude (Santhanam et al. 2022) with minimal
pooling multi-
tions 20 performance hits, and make storage costs competitive with other systems.
Model
Embedding size (KB)
BGE-M3
8.60
BM25 (dense emb.)
3.00
BM25 (sparse emb.)
1.56 0.51
ColPali (float16)
257.5
Table Comparison of the embedding sizes for the DocVQA test set from ViDoRe w.r.t. different retrieval models. The mean + std size is given for the sparse embeddings. In general multiple vectors (2-5) per page are used for BGE-M3 and BM25.
B.4 LATENCY COMPUTATIONS
To ensure comparison fairness, the latencies of different retrieval systems shown Figure are measured the same g2-standard-8 GCP VM with a NVIDIA L4 GPU. Document pages are embedded using the highest settings of Unstructured with captioning (see sucessubtttioon 3.2). SigLIP and ColPali are both loaded with bfloat 16 parameter dtypes. The reported times in Table 5 are the
1 nnttps:/ppytorch.orr
https: https://huggingface.c
https://blog.wespa. httph///bgggeegaaaaaaaeenaaaaaaa towp:////oaaaaooooooooooooooolo-
18

---

Published as a conference paper at ICLR 2025
average per-page latencies for each indexing operation on 1000 randomly chosen documents across all splits the ViDoRe benchmark test set. A batch size of was used for the BGE-MS model used with Unstructured, and a batch size of was used for SigLIP and ColPali.
Latency (s)
Indexing operation
Unstructured
SigLIP
ColPali
Layout detection
0.81
NA
NA
OCR
2.67
NA
NA
Captioning
3.71
NA
Page encoding
0.03
0.12
0.39
Total
7.22
0.12
0.39
Table 5: Page-level latencies for document indexing using various retrieval systems. SigL.I and ColPali are much faster than Unstructured because they don't require the layout detection, OCR, and captioning operations.
B.5 CAPTIONING
Examples of captions generated for visually rich document chunks with Claude-3 Sonnet are shown in Figure 5 and Tigure 4. The prompt used for generating the description is the following:
You assistant specialized in document analysis. Given table figure, you have provide detailed summary the content in maximum 3000 characters. Your summary should qualitative and not quantitative. Here is the table/figure to analyze: (image}. Answer ONLY with the caption of the table/figure.
35,000
Nucheer
Bio
Geothermal
Solarr
Wind
Hydro
Natural Gau
Other
Imports
30,000
(MN))
25,000
10,000
5,000
Figure 4: Example from the "Energy" test set.
Caption: The image depicts the hourly energy generation profile, illustrating the contributions of various energy sources over 24 hours. The data is presented as stacked bar chart, with the x-axis representing the hours of the day from to 2, and the y-axis showing the average hourly generation in MW. The bars are segmented into different colors, each representing distinct energy source: nuclear, bio, geothermal, solar, wind, hydro, natural gas, and other imports. The chart provides insights into the temporal variations in energy generation across different sources, highlighting the interplay between baseload and intermittent sources throughout the day.
19

---

Published as a conference paper at ICLR 2025
Eyslen Reaard
Couvyy Servre
Pogect View
Oedare
Elsancia Stuthenk
Adver
Parcwoor Departoan
Project Sudgets
18
Scheckint 10eagg
Eneovted coriracs
Precwa
Selector Comni Rod aa Ecoanees
10
akkgooskel U9A US
Negolaleon
NoSiae
Invescsiaappocciinn sepmonti ('APPT)
Aparowed
Rocacaa Haruutio (9F1
Photogophh
AA---lits
19
(Opeslling rorceb
Mainkemnnp acfecsla
daria
16A
Weeey nwel minute
Moxtll pogrest Fopc 05
AllONIOO GAAA liga
Figure 5: Example from the "Government Reports" test set.
Caption: The image shows a table titled "System of Record" which outlines the different types of documents or records maintained across various systems or departments within an organization related project management and construction. The rows list documents like project plans, budgets, schedules, contracts, purchase orders, invoices, change requests, bid submissions, drawings, manuals, meeting minutes, and reports. The columns indicate the system or department responsible for maintaining each record, such as County Servers, Project View, OnBase, CGI Advantage Financial System, and Purchasing Department. The table uses "W" and "T" markers to denote which system department serves the primary source (writer) storage location (trailer) for each rype of document.
20

---

Published as a conference paper at ICLR 2025
C ADDITIONAL RESULTS
C.1 OTHER METRICS
ArxivQ DocQ
InfoQ
Tab
TATQ
Shift
A1
Energy
Gov.
Health.
Avg.
Unstructured tau-eely
BM25
26.6
34.6
45.0
86.0
70.0
68.0
74,0
BGE-M3
22.8
26.
51.0
81.0,50
72.0paa
67.0
77.0
Unstructured
BM25
26.7
28.9
$4.0
30.4
50.0
$2.0
86.0
77.0
74.0
80.0
55,9
BGE-M3
28.1
22.9
S3.8,pa
55.7-225
38.61114
56.0
82.0
79.0
76.0
83.0
57.5
Unstructured CCaptening
BM25
35.5
30,2
61.5
24.3
47.0
79.0
76.0
75.0
81.0
55.9
BGF M3
29.3,6m
26.0ga
62,1
106
58.6-14
30,61144
55.0
80.0+1
78.0ps
69.0
83.0
57.2
Contrastive VLMs
Jina-CLIP
19.4
13.0
17.0
Nomic-vision
10.4
6.7
22.1
9.6
1.6
0.0
9.0
9.0
7.0
13.0
8.8
Sigl LIP Coeeta
34.2
21.3
51.8
46.1
17.9
13.0
30.0
51.0
47.0
65.0
39.7
Ours
SigL.I
34.2
21.3
51.8
46.1
13.0
80.0
51.0
47.0
65.0
39.7
BiSigLIP (sinotaang))
49.2-150
23.8125
59.0
52. 160
20.7
16.0
62.0
61.0
m00
55.0
72.0-10
47.
BiPali (+11
46.4.
20.0
54.6
63.2-m
20.4
34.0
59.0
45.0)
57.0
56.0. as
45.61418
ColPuli
72.4
45.6
74.6
75.4
53.1
55.0
93.0
85.0
85.0
88.0
72.7
Table 6: Comprehensive evaluation of baseline models and our proposed method on ViDoRe. Results are presented using Recall@) metrics. Text-only metrics are not computed for benchmarks with only visual elements.
C.2 MODEL VARIANTS
ArxivQ DocQ InfoQ
TabF
TATQ
Shift
Al
Energy Gov. Health.
Avg.
ColSigLIP (PaliGemma)
5.1
2.5
1.0
3.4
3.4
2.3
3.2
BiSigLIP (PaliGemma)
18.5
14.6
33.4
39.5
16.1
27.6
32.6
35.7
ColSigLIP (Original)
2.3
1.8
2.6
4.1
1.4
ColPali (No Q.A. Tokens)
80.4
53.2
82.4
77.4
65.7
63.4
97.0
89.9
93.6
92.4
79.6
ColPali (Docmatix)
71.3
48.0
80.0
83.9
59.1
73.8
95.7 93.8
92.5
93.1
79.1
ColPali (224)
71.0
37.4
62.3
65.7
28.6
20.4
65.7 66.8
73.9
73.0
56.5
ColPali (Vision Trained)
78.8
53.9
81.3
81.7
64.4
70.6
95.3 91.7
93.5
94.7
80.6
ColPali (No Pairwise)
79.0
53.0
82.1
85.3
63.2
66.2
94.9 88.9
92.7
92.1
79.7
ColPali (+TabFQuAD training)
77.6
54.7
82.6
86.5
65.4
73.9
94.8
92.4
94.8
Colldefies2 (64)
73.6
48.0
82.4
81.6
63.0
57.2
95.5
86.9
91.2
ColQwen2 (768)
86.4
56.2
89.8
88.7
85.7
94.8
97.3
ColPali (Reference: 448)
79.1
54.4
81.8
83.9
65.8
73.2
96.2
91.0
92.7
94.4
81.3
Table 7: Benchmark scores for the "negative results" and various ablations on ViDoRe; ColPali for reference. Results are presented using nDCG@5 metrics. Text-only metrics are not computed for benchmarks with only visual elements.
21

---

Published as a conference paper at ICLR 2025
La déalin depuis les années 1970.
découverts
La tall moyemne augmenté liprement au des aenéss 2000 de concert noureau gyde de decuvertts aux champo olthhrr, Copendart, des chiffes ort sepris rapidoment dedin repogrant sendance géndrale Depuls 2007:
II. Historique de production
La production dhyyrrocabuures liquides depuis début
1990, (11 Mb].. Catte production trowvst depuis 1077 sur stooas donwron 150 milkons de baris an production @Tydhocartuures Kazakhstan rasee pringpalmmett oitule
represente
barls
sharrd profandes (shal), lancament production chemp giaand de Kashagan
Production terre mer d' d'hydrocarbures liquides
(1965-2019)
Land
100
33775
2540
23955
2005
2013
Query: "Quelle partie de la production pétrolière du Kazakhstan provient de champs mer ?"
Figure 6: Similarity of the image patches w.r.t. the underlined token in the user query. This example from the Shift test set.
D MORE SIMILARITY MAPS
In Figure 6) ColPali assigns high similarity all patches with the word "Kazakhstan" when given the token Kazzkksstan. Moreover, our model seems to exhibit world knowledge capabilities as the patch around the word "ashagan"-an offshore oil field in Kazakhstan shows high similarity score.
It is also interesting to highlight that both this similarity map and the one displayed in Figure 3 (right) showcase a few white patches with high similarity scores. This behavior might first seem surprising the white patches should not carry a meaningful signal from the original images. We believe the vectors associated with these patches share a similar role with the ViT registers (Darcet et al. 22023. ie. these patches were repurposed for internal computations and stored the global information from the whole image.
E MODEL GLOSSARY
SIGLIP
SigLIP (Sigmoid Loss for Language Image Pre-Training. builds upon CLIP (Contrastive Language- Image Pretraining)-4 foundational model that aligns images and text by maximizing the similarity between correct image-text pairs while minimizing it for incorrect ones, leveraging contrastive loss (Zhai al. 2023). Unlike CLIP [Radford et al. 2021), which applies the softmax function to the logits, LigLIP uses the sigmoid activation function. This innovation eliminates the need for global view all pairwise similarities between images and texts within a batch, enabling more flexible batch size scaling (up to 1M items per batch, with effective optimal batch size of 32k). This approach allows IPP to achieve state-of-the-art performance zero-shot image classification tasks.
PALIGEMMA
PaliGemma is a 3B-parameter dision-language model. integrates the SigLIP vision encoder with a Gemma-2B language decoder, connected via multimodal linear projection layer (Lucas Beyer* et al.
22

---

Published as a conference paper at ICLR 2025
2024). The model processes images by segmenting them into fixed number of Vision Transformer Dosovitskiy et al. 2020) tokens, which are prepended to an optional text prompt.
A distinguishing feature of PaliGemma is its operation as a Prefix-Language Model (Prefix-LM), This design ensures full attention between image tokens and the user-provided input (prefix) while generating outputs auto-regressively (suffix). This architecture allows image tokens to access the task-specific query during processing, facilitating more effective task-dependent reasoning.
PaliGemma trained in four stages: unimodal pretraining with existing components, extended multimodal pretraining, short high-resolution pretraining, and task-specific fine-tuning.
COLBERT
ColBERT (Contextualized Late Interaction over BERT) is retrieval model designed to balance speed and effectiveness in information retrieval tasks (Khattab & Zaharia 2020). Traditional retrieval models are typically categorized based on their type of interaction: either processing queries and documents independently for efficiency (bi-encoders) jointly to capture rich contextual relationships (cross- encoders). ColBERT combines the advantages of both approaches through a novel late interaction mechanism.
Queries and documents are encoded separately using BERT, enabling offline pre-computation of document representations for scalability. Instead of pooling embeddings into single vector, ColBERT retains token-level embeddings and employs a MaxSim operator to compute fine- ane-grained similarity scores. For each query token, the model determines the maximum similarity with document tokens, summing these scores to compute relevance.
This architecture preserves the contextual richness of deep language models while significantly improving computational efficiency. By delaying the interaction step, ColBERT supports vector similarity indexing, facilitating end-to-end retrieval from large collections without prohibitive costs. Empirical evaluations on passage search datasets demonstrate that ColBERT achieves competitive effectiveness compared to existing BERT-based models (Devlin et al.) 2018). while executing queries orders of magnitude faster and with drastically reduced computational requirements.
23

---

Published as a conference paper at ICLR 2025
F EXAMPLES FROM THE ViDoRe BENCHMARK
Energy
Query: What types of accounts or Query: What is the estimated to- Query: What the projected products allow investors to defer tal savings for a PV system in peak electricity demand in Cali- paying taxes?
Durham under the net metering fornia for the year 2030? (flat rate) billing option the system's useful life 25 years?
invesing
2010 copactiies
CCST
Artificial Intelligence
Query: What common Query: What did the robot moni- Query: What is the key approach outcome areas targeted by TAlI tor determine when to activate used in the PDP architecture? for different age groups?
or deactivate the blower motor and blinker?
24

---

Published as a conference paper at ICLR 2025
Healthcare Industry
Query: What does the AVPU mula for the ferroclectric material are involved in public financing scale stand for in assessing the Lead Zirconium Titanate (PZT)? for healthcare in the US?
Query: What is the chemical for- Query: What government entities
level of consciousness of seri- ously child?
Perrosdecttion
UCLA
Government Reports
Query: What some mandates Query: What is the strategy of Query: What is the trust signal for the EPA under the Pollution KPMG Hazem Hassan? Prevention Act?
score for the consumer industry best-in-class archetype?
APPENNEUE CURRENT PNLLUTIS PROVENITIO
Who we are?
25
15
Erfarmmaaryy
25

---

Published as a conference paper at ICLR 2025
Shift
Query: Selon le graphique, Query: Quelle partie de la pro- Query: Quels sont les pays quelle est capacité d'import la consommation réelle de carbu- rants SAF (biocarburants durables pour I' l'aviiaion) prévues 2050
duction pétrolière du Kazakhstan ayant plus grande part des provient de champs en mer? ?
découvertes cumulées de pétrole brut en 2020 (en milliers de barils, hors découvertes cumulées) ?
Hebl
Ereas time
26


Reviews:
[{"review_id": "SfpR4nJ9Ec", "reviewer": "Reviewer_kCQZ", "summary": "The paper presents two contributions in the context of document retrieval: (1) it presents ViDoRe, a new benchmark used to evaluate document retrieval algorithms with an emphasis on documents containing visual information, which is not covered by most of the existing benchmarks; and (2) it introduces ColPali, a model architecture that uses Vision-Language Models to efficiently index the documents in an efficient manner, and performing retrieval at query time with a competitive cost.\n\nViDoRe is composed from different existing academic datasets (mainly Vision Question Answering benchmarks), plus topic-specific publicly-accessible PDFs collected by the authors. ColPali is built based on PaliGemma-3B and the ColBERT strategy to generate a set of vision-text tokens from PDFs (during indexing) and text queries (during retrieval).\n\nThe results reported in the paper show that standard methods of document retrieval are either very expensive for indexing (e.g. solutions based on the off-the-self \"Unstructured\" tool, augmented with image captioning or OCR) or provide lower quality (e.g. approaches based on contrastive Vision-Language encoders). The proposed", "questions": "See questions / concerns implied in the weaknesses section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed benchmark, ViDoRe, covers an important gap for evaluating existing document retrieval systems.\n- The method presented in the paper significantly improves existing approaches, even those that require some fine-tuning (see table 2).\n- The paper includes a section with extensive ablation studies that justify some of the decisions made during the design of the ColPali method.\n- Hyperparameter tuning was not directly done on the ViDoRe benchmark, but on a 2% split from their training set (however, this training set poses some problems, see the weaknesses bellow).", "weaknesses": "- The paper builds ColPali iteratively starting from a SigLIP model, and using previously existing recipes. However, each of the steps is not very well detailed in the paper itself, which may make the paper hard to truly understand for readers not familiar with SigLIP, ColBERT, or PaliGemma. I would suggest expanding the details on section 5.1.\n- For sytems that require any sort of tuning, the authors have used a dataset made of rouhgly 120k query-page pairs, 63% of which come from the same distribution used in the academic ViDoRe benchmark. The authors made sure that no query or page in the evaluation set is part of the training data, however the training data is very \"in-distribution\" of the evaluation data, which may bias both the hyperparameter tuning and the quality of the methods on ViDoRe. I would strongly suggest that the authors (also) present results, for the different methods that require some sort of tuning, using only the synthetic training data (including hyperparameter search). This way, we can measure if the gap between the proposed method and the Unstructured approaches is mainly due to the architecture or due to the (in-domain) fine-tuning  and hyperparameter selection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jpHRpWdnwk", "reviewer": "Reviewer_nYTk", "summary": "This paper introduces a novel document retrieval method called ColPali, which uses Vision Language Models (VLMs) to generate high-quality multi-vector embeddings directly from images of document pages. The method aims to address the performance bottlenecks of current text-centric retrieval systems when dealing with visually rich documents. The paper also introduces a new benchmarking framework, ViDoRe, to evaluate system performance in visually rich document retrieval.", "questions": "No additional questions.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The ViDoRe benchmark covers multiple domains and languages, providing a comprehensive framework to evaluate the capabilities of document retrieval systems.\n\n2. ColPali presents an innovative concept that significantly improves performance through retrieval in the vision space. Experimental results demonstrate that ColPali significantly outperforms existing methods in several visually complex tasks and offers fast indexing and querying capabilities.\n\n3. The paper mentions that all resources, including models, data, and code, are released under open licenses, which will promote further research and application within the community.", "weaknesses": "1. Although the ViDoRe benchmark covers multiple domains, the generality and adaptability of ColPali in broader application scenarios need further validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Ue0BsmShf", "reviewer": "Reviewer_rAjC", "summary": "In this paper, the authors explore a different approach to visually-rich document retrieval. To better assess current systems for this task, they construct a benchmark, called ViDoRe, which consists of various page-level retrieval tasks spanning multiple domains, languages, and practical settings. Inspired by ColBERT, the authors propose to perform document retrieval by directly embedding the images of the document pages. The proposed model, termed ColPali, is a Vision Language Model with a late interaction matching mechanism. The proposed model is evaluated on the ViDoRe benchmark and compared with existing pipelines for document retrieval.", "questions": "The authors are encouraged to further explain the novelty of the proposed ColPali and give more quantitative results regarding the latencies of different document retrieval model/systems.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work investigates a very important problem: retrieving information from visually-rich documents. Visually-rich documents are quite common in practice and usually with complex layouts as well as informative elements such as figures and tables, which cannot be well handled by existing pipelines for document retrieval.\n2. Different from previous pipelines, the authors accomplish visually-rich document retrieval by directly embedding the images of the document pages with VLMs and late interaction mechanisms. Experiments demonstrate the effectiveness and efficiency of the proposed ColPali model.\n3. The proposed ViDoRe benchmark is a good contribution to the community as it can be used to evaluate systems on page-level document retrieval with a wide range of domains and applications, in the setting of simultaneous textual and visual understanding.\n4. Overall, the paper is well-written. The main idea and key technical details are clearly presented.", "weaknesses": "1. The proposed pipeline is mainly inspired by ColBERT and PaliGemma. The author should give more explanations and analyses to prove the novelty of it.\n2. The authors specially emphasize the importance of retrieval efficiency in industrial applications. However, the quantitative results regarding the latencies of different document retrieval model/systems are not detailed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iZFeuZTuIR", "reviewer": "Reviewer_7YUV", "summary": "The paper introduces ColPali, a document retrieval model that leverages LVLMs to create high-quality contextualized embeddings from images of document pages. This approach allows for efficient and fast query matching, leading to improved performance in document retrieval tasks. The paper's main contributions are: 1. The ViDoRe Benchmark: a comprehensive benchmark for evaluating document retrieval systems across various domains, visual elements, and languages. 2. The ColPali Model: a model that indexes documents based on their visual features and uses a late interaction mechanism for query matching, outperforming existing retrieval systems in terms of speed and accuracy.", "questions": "**Page-Level Retrieval Capability**: Does the ColPali support page-level retrieval within a PDF, such as those in DUDE and MPdocvqa? Both MPdocvqa and DUDE also provide the gold document of a PDF.\n\n**Text Retrieval Functionality**: Is ColPali capable of performing conventional text-based retrieval, or is its functionality limited to document images and visual features?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Performance Improvements**: ColPali demonstrates superior performance on the ViDoRe benchmark compared to modern document retrieval pipelines and is significantly faster, making it suitable for practical applications.\n\n**End-to-End Trainability**: The ColPali model is end-to-end trainable, simplifying the optimization process for the retrieval task.\n\n**Open-Source Release**: The authors release all project artifacts, including models and code, to encourage further development and research in the field.\n\n**Multilingual Capabilities**: ColPali shows the ability to generalize to non-English languages, expanding its potential use cases globally.", "weaknesses": "**Benchmark Samples**: The manuscript should include examples from the ViDoRe benchmark that showcase its diversity in modalities, thematic domains, and language types. This will provide a more comprehensive understanding of the benchmark's scope and applicability.\n\n**Retrieval Candidate Space Clarification**: The manuscript does not clearly define the retrieval candidate space for the ViDoRe benchmark. Clarifying this aspect is essential for understanding the benchmark's parameters and the retrieval process.\n\n**ColPali Model Details**: The manuscript lacks essential details about the ColPali model. For instance, the definitions of N_q and N_d\n  are not provided, nor is it explained how the document representation is generated. It is unclear whether the representation is derived from features corresponding to the [BOS] or [EOS] tokens, or if it is an average of the entire sequence features. These details are vital for replicating the study and understanding the model's workings.\n\n**Broader Evaluation**: The evaluation is currently limited to the ViDoRe benchmark, with no experiments conducted on publicly available datasets such as those used for the retrieval task in VLM2Vec [1] and page prediction in MPdocVQA [2]. Expanding the evaluation to include these datasets would strengthen the experimental results and increase their convincing.\n\n\n[1] Jiang Z, Meng R, Yang X, et al. VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks[J]. arXiv preprint arXiv:2410.05160, 2024.\n\n[2] Tito R, Karatzas D, Valveny E. Hierarchical multimodal transformers for multipage docvqa[J]. Pattern Recognition, 2023, 144: 109834.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)