VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality
  Documents
Authors: name: Shi Yu; name: Chaoyue Tang; name: Bokai Xu; name: Junbo Cui; name: Junhao Ran; name: Yukun Yan; name: Zhenghao Liu; name: Shuo Wang; name: Xu Han; name: Zhiyuan Liu; name: Maosong Sun
Published: 2024-10-14 15:04:18

Abstract: Retrieval-augmented generation (RAG) is an effective technique that enables
large language models (LLMs) to utilize external knowledge sources for
generation. However, current RAG systems are solely based on text, rendering it
impossible to utilize vision information like layout and images that play
crucial roles in real-world multi-modality documents. In this paper, we
introduce VisRAG, which tackles this issue by establishing a vision-language
model (VLM)-based RAG pipeline. In this pipeline, instead of first parsing the
document to obtain text, the document is directly embedded using a VLM as an
image and then retrieved to enhance the generation of a VLM. Compared to
traditional text-based RAG, VisRAG maximizes the retention and utilization of
the data information in the original documents, eliminating the information
loss introduced during the parsing process. We collect both open-source and
synthetic data to train the retriever in VisRAG and explore a variety of
generation methods. Experiments demonstrate that VisRAG outperforms traditional
RAG in both the retrieval and generation stages, achieving a 25--39\%
end-to-end performance gain over traditional text-based RAG pipeline. Further
analysis reveals that VisRAG is effective in utilizing training data and
demonstrates strong generalization capability, positioning it as a promising
solution for RAG on multi-modality documents. Our code and data are available
at https://github.com/openbmb/visrag .