ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic
  Iterative Reasoning Agents
Authors: name: Qiuchen Wang; name: Ruixue Ding; name: Zehui Chen; name: Weiqi Wu; name: Shihang Wang; name: Pengjun Xie; name: Feng Zhao
Published: 2025-02-25 09:26:12

Abstract: Understanding information from visually rich documents remains a significant
challenge for traditional Retrieval-Augmented Generation (RAG) methods.
Existing benchmarks predominantly focus on image-based question answering (QA),
overlooking the fundamental challenges of efficient retrieval, comprehension,
and reasoning within dense visual documents. To bridge this gap, we introduce
ViDoSeek, a novel dataset designed to evaluate RAG performance on visually rich
documents requiring complex reasoning. Based on it, we identify key limitations
in current RAG approaches: (i) purely visual retrieval methods struggle to
effectively integrate both textual and visual features, and (ii) previous
approaches often allocate insufficient reasoning tokens, limiting their
effectiveness. To address these challenges, we propose ViDoRAG, a novel
multi-agent RAG framework tailored for complex reasoning across visual
documents. ViDoRAG employs a Gaussian Mixture Model (GMM)-based hybrid strategy
to effectively handle multi-modal retrieval. To further elicit the model's
reasoning capabilities, we introduce an iterative agent workflow incorporating
exploration, summarization, and reflection, providing a framework for
investigating test-time scaling in RAG domains. Extensive experiments on
ViDoSeek validate the effectiveness and generalization of our approach.
Notably, ViDoRAG outperforms existing methods by over 10% on the competitive
ViDoSeek benchmark.