OmniDocBench: Benchmarking Diverse PDF Document Parsing with
  Comprehensive Annotations
Authors: name: Linke Ouyang, name: Yuan Qu, name: Hongbin Zhou, name: Jiawei Zhu, name: Rui Zhang, name: Qunshu Lin, name: Bin Wang, name: Zhiyuan Zhao, name: Man Jiang, name: Xiaomeng Zhao, name: Jin Shi, name: Fan Wu, name: Pei Chu, name: Minghao Liu, name: Zhenxiang Li, name: Chao Xu, name: Bo Zhang, name: Botian Shi, name: Zhongying Tu, name: Conghui He

Document content extraction is crucial in computer vision, especially for
meeting the high-quality data needs of large language models (LLMs) and
retrieval-augmented generation (RAG) technologies. However, current document
parsing methods suffer from significant limitations in terms of diversity and
comprehensive evaluation. To address these challenges, we introduce
OmniDocBench, a novel multi-source benchmark designed to advance automated
document content extraction. OmniDocBench includes a meticulously curated and
annotated high-quality evaluation dataset comprising nine diverse document
types, such as academic papers, textbooks, slides, among others. Our benchmark
provides a flexible and comprehensive evaluation framework with 19 layout
category labels and 14 attribute labels, enabling multi-level assessments
across entire datasets, individual modules, or specific data types. Using
OmniDocBench, we perform an exhaustive comparative analysis of existing modular
pipelines and multimodal end-to-end methods, highlighting their limitations in
handling document diversity and ensuring fair evaluation. OmniDocBench
establishes a robust, diverse, and fair evaluation standard for the document
content extraction field, offering crucial insights for future advancements and
fostering the development of document parsing technologies. The codes and
dataset is available in https://github.com/opendatalab/OmniDocBench.