ColPali: Efficient Document Retrieval with Vision Language Models

Authors: Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, CELINE HUDELOT, Pierre Colombo
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: document embeddings, vision language models, late interaction, document retrieval, information retrieval

Abstract:
Documents are visually rich structures that convey information through text, but also figures, page layouts, tables, or even fonts. Since modern retrieval systems mainly rely on the textual information they extract from document pages to index documents -often through lengthy and brittle processes-, they struggle to exploit key visual cues efficiently. This limits their capabilities in many practical document retrieval applications such as Retrieval Augmented Generation (RAG).
To benchmark current systems on visually rich document retrieval, we introduce the Visual Document Retrieval Benchmark $\textit{ViDoRe}$, composed of various page-level retrieval tasks spanning multiple domains, languages, and practical settings. 
The inherent complexity and performance shortcomings of modern systems motivate a new concept; doing document retrieval by directly embedding the images of the document pages. We release $\textit{ColPali}$, a Vision Language Model trained to produce high-quality multi-vector embeddings from images of document pages. Combined with a late interaction matching mechanism, $\textit{ColPali}$ largely outperforms modern document retrieval pipelines while being drastically simpler, faster and end-to-end trainable. 
We release models, data, code and benchmarks under open licenses at https://hf.co/vidore.

Decision: accept (poster)
Reviewer scores: 5, 8, 5, 3

Reviews and discussion:
<thread>
creation date:
1730174138934

thread_type:
official_review

<thread_object>
creator: reviewer_7yuv

<content>
summary:
The paper introduces ColPali, a document retrieval model that leverages LVLMs to create high-quality contextualized embeddings from images of document pages. This approach allows for efficient and fast query matching, leading to improved performance in document retrieval tasks. The paper's main contributions are: 1. The ViDoRe Benchmark: a comprehensive benchmark for evaluating document retrieval systems across various domains, visual elements, and languages. 2. The ColPali Model: a model that indexes documents based on their visual features and uses a late interaction mechanism for query matching, outperforming existing retrieval systems in terms of speed and accuracy.

strengths:
**Performance Improvements**: ColPali demonstrates superior performance on the ViDoRe benchmark compared to modern document retrieval pipelines and is significantly faster, making it suitable for practical applications.

**End-to-End Trainability**: The ColPali model is end-to-end trainable, simplifying the optimization process for the retrieval task.

**Open-Source Release**: The authors release all project artifacts, including models and code, to encourage further development and research in the field.

**Multilingual Capabilities**: ColPali shows the ability to generalize to non-English languages, expanding its potential use cases globally.

weaknesses:
**Benchmark Samples**: The manuscript should include examples from the ViDoRe benchmark that showcase its diversity in modalities, thematic domains, and language types. This will provide a more comprehensive understanding of the benchmark's scope and applicability.

**Retrieval Candidate Space Clarification**: The manuscript does not clearly define the retrieval candidate space for the ViDoRe benchmark. Clarifying this aspect is essential for understanding the benchmark's parameters and the retrieval process.

**ColPali Model Details**: The manuscript lacks essential details about the ColPali model. For instance, the definitions of N_q and N_d
  are not provided, nor is it explained how the document representation is generated. It is unclear whether the representation is derived from features corresponding to the [BOS] or [EOS] tokens, or if it is an average of the entire sequence features. These details are vital for replicating the study and understanding the model's workings.

**Broader Evaluation**: The evaluation is currently limited to the ViDoRe benchmark, with no experiments conducted on publicly available datasets such as those used for the retrieval task in VLM2Vec [1] and page prediction in MPdocVQA [2]. Expanding the evaluation to include these datasets would strengthen the experimental results and increase their convincing.


[1] Jiang Z, Meng R, Yang X, et al. VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks[J]. arXiv preprint arXiv:2410.05160, 2024.

[2] Tito R, Karatzas D, Valveny E. Hierarchical multimodal transformers for multipage docvqa[J]. Pattern Recognition, 2023, 144: 109834.

questions:
**Page-Level Retrieval Capability**: Does the ColPali support page-level retrieval within a PDF, such as those in DUDE and MPdocvqa? Both MPdocvqa and DUDE also provide the gold document of a PDF.

**Text Retrieval Functionality**: Is ColPali capable of performing conventional text-based retrieval, or is its functionality limited to document images and visual features?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712949644

thread_type:
official_review

<thread_object>
creator: reviewer_rajc

<content>
summary:
In this paper, the authors explore a different approach to visually-rich document retrieval. To better assess current systems for this task, they construct a benchmark, called ViDoRe, which consists of various page-level retrieval tasks spanning multiple domains, languages, and practical settings. Inspired by ColBERT, the authors propose to perform document retrieval by directly embedding the images of the document pages. The proposed model, termed ColPali, is a Vision Language Model with a late interaction matching mechanism. The proposed model is evaluated on the ViDoRe benchmark and compared with existing pipelines for document retrieval.

strengths:
1. This work investigates a very important problem: retrieving information from visually-rich documents. Visually-rich documents are quite common in practice and usually with complex layouts as well as informative elements such as figures and tables, which cannot be well handled by existing pipelines for document retrieval.
2. Different from previous pipelines, the authors accomplish visually-rich document retrieval by directly embedding the images of the document pages with VLMs and late interaction mechanisms. Experiments demonstrate the effectiveness and efficiency of the proposed ColPali model.
3. The proposed ViDoRe benchmark is a good contribution to the community as it can be used to evaluate systems on page-level document retrieval with a wide range of domains and applications, in the setting of simultaneous textual and visual understanding.
4. Overall, the paper is well-written. The main idea and key technical details are clearly presented.

weaknesses:
1. The proposed pipeline is mainly inspired by ColBERT and PaliGemma. The author should give more explanations and analyses to prove the novelty of it.
2. The authors specially emphasize the importance of retrieval efficiency in industrial applications. However, the quantitative results regarding the latencies of different document retrieval model/systems are not detailed in the paper.

questions:
The authors are encouraged to further explain the novelty of the proposed ColPali and give more quantitative results regarding the latencies of different document retrieval model/systems.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715544542

thread_type:
official_review

<thread_object>
creator: reviewer_nytk

<content>
summary:
This paper introduces a novel document retrieval method called ColPali, which uses Vision Language Models (VLMs) to generate high-quality multi-vector embeddings directly from images of document pages. The method aims to address the performance bottlenecks of current text-centric retrieval systems when dealing with visually rich documents. The paper also introduces a new benchmarking framework, ViDoRe, to evaluate system performance in visually rich document retrieval.

strengths:
1. The ViDoRe benchmark covers multiple domains and languages, providing a comprehensive framework to evaluate the capabilities of document retrieval systems.

2. ColPali presents an innovative concept that significantly improves performance through retrieval in the vision space. Experimental results demonstrate that ColPali significantly outperforms existing methods in several visually complex tasks and offers fast indexing and querying capabilities.

3. The paper mentions that all resources, including models, data, and code, are released under open licenses, which will promote further research and application within the community.

weaknesses:
1. Although the ViDoRe benchmark covers multiple domains, the generality and adaptability of ColPali in broader application scenarios need further validation.

questions:
No additional questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730891103860

thread_type:
official_review

<thread_object>
creator: reviewer_kcqz

<content>
summary:
The paper presents two contributions in the context of document retrieval: (1) it presents ViDoRe, a new benchmark used to evaluate document retrieval algorithms with an emphasis on documents containing visual information, which is not covered by most of the existing benchmarks; and (2) it introduces ColPali, a model architecture that uses Vision-Language Models to efficiently index the documents in an efficient manner, and performing retrieval at query time with a competitive cost.

ViDoRe is composed from different existing academic datasets (mainly Vision Question Answering benchmarks), plus topic-specific publicly-accessible PDFs collected by the authors. ColPali is built based on PaliGemma-3B and the ColBERT strategy to generate a set of vision-text tokens from PDFs (during indexing) and text queries (during retrieval).

The results reported in the paper show that standard methods of document retrieval are either very expensive for indexing (e.g. solutions based on the off-the-self "Unstructured" tool, augmented with image captioning or OCR) or provide lower quality (e.g. approaches based on contrastive Vision-Language encoders). The proposed

strengths:
- The proposed benchmark, ViDoRe, covers an important gap for evaluating existing document retrieval systems.
- The method presented in the paper significantly improves existing approaches, even those that require some fine-tuning (see table 2).
- The paper includes a section with extensive ablation studies that justify some of the decisions made during the design of the ColPali method.
- Hyperparameter tuning was not directly done on the ViDoRe benchmark, but on a 2% split from their training set (however, this training set poses some problems, see the weaknesses bellow).

weaknesses:
- The paper builds ColPali iteratively starting from a SigLIP model, and using previously existing recipes. However, each of the steps is not very well detailed in the paper itself, which may make the paper hard to truly understand for readers not familiar with SigLIP, ColBERT, or PaliGemma. I would suggest expanding the details on section 5.1.
- For sytems that require any sort of tuning, the authors have used a dataset made of rouhgly 120k query-page pairs, 63% of which come from the same distribution used in the academic ViDoRe benchmark. The authors made sure that no query or page in the evaluation set is part of the training data, however the training data is very "in-distribution" of the evaluation data, which may bias both the hyperparameter tuning and the quality of the methods on ViDoRe. I would strongly suggest that the authors (also) present results, for the different methods that require some sort of tuning, using only the synthetic training data (including hyperparameter search). This way, we can measure if the gap between the proposed method and the Unstructured approaches is mainly due to the architecture or due to the (in-domain) fine-tuning  and hyperparameter selection.

questions:
See questions / concerns implied in the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733238532094

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Review Process

comment:
We briefly summarize the reviews and rebuttals to facilitate the AC’s assessment of our work.

**Strenghts.** All reviewers agree that our paper - which first introduces the concept of producing document embeddings from page images using Vision Language Models - tackles a “*very important problem*”, and the reviews highlight both the efficiency and performance gains of our proposed method ColPali, the value of our benchmark which “*fills an important gap*”, and appreciate our thorough resource release.

**Interrogations.** Reviewers kCQZ and nYTk had questions regarding the generalization of our method outside the training data distribution, which we thoroughly answered, notably pointing to an experiment in the paper designed to evaluate this. Reviewer kCQZ was satisfied by the response and expressed that they were convinced our work was “*very likely generalizable to OOD data*”.

**Improvements.** Reviewer rAjC acknowledges the paper is "*well written*" and "*clearly presented*", and we have further updated the manuscript to mainly:

-  Include additional background information on Vision Language Models and add minor clarifications to our construction process (kCQZ, rAjC)
- Move some latency details from the appendix to the main paper (rAjC)
- Add further dataset examples to the appendix, and mention related work released after the ICLR deadline (7YUV)

We have done our best to address all comments in detailed individual rebuttals, hopefully strengthening our work.  We thank all reviewers once again for their constructive comments and overall appreciation of our contribution.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper has two main contributions toward document retrieval: the construction of ViDoRe benchmark and the VLM-based model called ColPali, both of which move away from the text-centric document retrieval paradigm. The reviewers find the benchmark significant and the experimental results convincing. Overall, the authors address concerns raised by the reviewers during the discussion period (see below). Both the concept introduced in this paper and the released models, data, code and benchmarks would be beneficial to the community.