Are We on the Right Way for Assessing Document Retrieval-Augmented
  Generation?
Authors: name: Wenxuan Shen, name: Mingjia Wang, name: Yaochen Wang, name: Dongping Chen, name: Junjie Yang, name: Yao Wan, name: Weiwei Lin

Retrieval-Augmented Generation (RAG) systems using Multimodal Large Language
Models (MLLMs) show great promise for complex document understanding, yet their
development is critically hampered by inadequate evaluation. Current benchmarks
often focus on specific part of document RAG system and use synthetic data with
incomplete ground truth and evidence labels, therefore failing to reflect
real-world bottlenecks and challenges. To overcome these limitations, we
introduce Double-Bench: a new large-scale, multilingual, and multimodal
evaluation system that is able to produce fine-grained assessment to each
component within document RAG systems. It comprises 3,276 documents (72,880
pages) and 5,168 single- and multi-hop queries across 6 languages and 4
document types with streamlined dynamic update support for potential data
contamination issues. Queries are grounded in exhaustively scanned evidence
pages and verified by human experts to ensure maximum quality and completeness.
Our comprehensive experiments across 9 state-of-the-art embedding models, 4
MLLMs and 4 end-to-end document RAG frameworks demonstrate the gap between text
and visual embedding models is narrowing, highlighting the need in building
stronger document retrieval models. Our findings also reveal the
over-confidence dilemma within current document RAG frameworks that tend to
provide answer even without evidence support. We hope our fully open-source
Double-Bench provide a rigorous foundation for future research in advanced
document RAG systems. We plan to retrieve timely corpus and release new
benchmarks on an annual basis.