READoc: A Unified Benchmark for Realistic Document Structured Extraction
Authors: Zichao Li, Aizier Abulaiti, Yaojie Lu, Xuanang Chen, Jia Zheng, Hongyu Lin, Xianpei Han, Le Sun
Accepted at: acl25

READOC: A Unified Benchmark for Realistic Document Structured Extraction

Zichao Li1,2,*, Aizier Abulaiti1,2,*, Yaojie Lu1, Xuanang Chen1,†, Jia Zheng1,†, Hongyu Lin1, Xianpei Han1, Shanshan Jiang3, Bin Dong3, Le Sun1
1Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences
2University of Chinese Academy of Sciences
3Ricoh Software Research Center Beijing Co., Ltd
{lizichao2022,aizier2022,luyaojie,chenxuanang,zhengjia}@iscas.ac.cn {hongyu,xianpei,sunle}@iscas.ac.cn {shanshan.jiang,bin.dong}@cn.ricoh.com

Abstract

Document Structured Extraction (DSE) aims to extract structured content from raw documents. Despite the emergence of numerous DSE systems, their unified evaluation remains inadequate, significantly hindering the field’s advancement. This problem is largely attributed to existing benchmark paradigms, which exhibit fragmented and localized characteristics. To offer a thorough evaluation of DSE systems, we introduce a novel benchmark named READOC, which defines DSE as a realistic task of converting unstructured PDFs into semantically rich Markdown. The READOC dataset is derived from 3,576 diverse and real-world documents from arXiv, GitHub, and Zenodo. In addition, we develop a DSE Evaluation Suite comprising Standardization, Segmentation and Scoring modules, to conduct a unified evaluation of state-of-the-art DSE approaches. By evaluating a range of pipeline tools, expert visual models, and general Vision-Language Models, we identify the gap between current work and the unified, realistic DSE objective for the first time. We aspire that READOC will catalyze future research in DSE, fostering more comprehensive and practical solutions.

1 Introduction

The wealth of knowledge preserved in documents is immeasurable. Document Structured Extraction (DSE), which involves converting raw documents into machine-readable structured text (Tkaczyk et al., 2015; Zhong et al., 2019; Shen et al., 2022; Lo et al., 2023), is increasingly crucial in real-world scenarios. It facilitates building extensive knowledge bases (Wang et al., 2020), constructing high-quality corpora (Jain et al., 2020), and plays a pivotal role in Retrieval-Augmented Generation (RAG) (Gao et al., 2023) applications for Large Language Models (LLMs) (Achiam et al., 2023).

Recent progress in Document AI (Appalaraju et al., 2021; Huang et al., 2022; Ye et al., 2023; Hu et al., 2024) has led to the creation of numerous DSE systems (Breezedeus, 2022; Paruchuri and Lampa, 2023; Blecher et al., 2023). However, the absence of unified evaluation in real-world scenarios has left uncertainty about their performance levels and hindered their further development. This issue is largely due to the limitations of prevailing benchmark paradigms, which exhibit fragmented and unrealistic characteristics. Firstly, as depicted in Figure 1, existing benchmarks typically fragment DSE into distinct subtasks, including document layout analysis (Zhong et al., 2019), optical character recognition (Karatzas et al., 2015), table-of-contents extraction (Hu et al., 2022), reading order detection (Wang et al., 2021), table recognition (Smock et al., 2022) and formula conversion (Deng et al., 2017). Due to their narrow focus, diverse data sources, and inconsistent formats, existing benchmarks lack a unified framework to com-

![A comparison between fragmented and localized DSE task views and the READOC benchmark paradigm.](page_1012_384_484_384.png)

Figure 1: A comparison between fragmented and localized DSE task views and the READOC benchmark paradigm.
<table>
  <tr>
    <th rowspan="2">Benchmark</th>
    <th colspan="6">Coverage</th>
    <th colspan="2">Task Paradigm</th>
  </tr>
  <tr>
    <th>Layout Analysis</th>
    <th>Character Recognition</th>
    <th>Table Recognition</th>
    <th>Formula Conversion</th>
    <th>ToC. Extraction</th>
    <th>Order Detection</th>
    <th>INPUT</th>
    <th>OUTPUT</th>
  </tr>
  <tr>
    <td>PubLayNet</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>Page image</td>
    <td>Layout blocks</td>
  </tr>
  <tr>
    <td>DocBank</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>Page image & text</td>
    <td>Layout blocks</td>
  </tr>
  <tr>
    <td>Robust Reading</td>
    <td>✗</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>Text image</td>
    <td>Plain text</td>
  </tr>
  <tr>
    <td>PubTabNet</td>
    <td>✗</td>
    <td>✗</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>Table image</td>
    <td>HTML table</td>
  </tr>
  <tr>
    <td>TabLeX</td>
    <td>✗</td>
    <td>✗</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>Table image</td>
    <td>\( \LaTeX \) table</td>
  </tr>
  <tr>
    <td>Im2Latex-100K</td>
    <td>✗</td>
    <td>✗</td>
    <td>✓</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>Formula image</td>
    <td>\( \LaTeX \) formula</td>
  </tr>
  <tr>
    <td>ReadingBank</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✓</td>
    <td>✓</td>
    <td>Unsorted tokens</td>
    <td>Sorted tokens</td>
  </tr>
  <tr>
    <td>HRDoc</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✓</td>
    <td>✓</td>
    <td>Doc images & text</td>
    <td>Logical tree</td>
  </tr>
  <tr>
    <th>READoc</th>
    <th>✓</th>
    <th>✓</th>
    <th>✓</th>
    <th>✓</th>
    <th>✓</th>
    <th>✓</th>
    <th>Realistic PDF doc</th>
    <th>Markdown text</th>
  </tr>
</table>

Table 1: A comparison between READoc and existing DSE benchmarks, including PubLayNet (Zhong et al., 2019), DocBank (Li et al., 2020), Robust Reading (Karatzas et al., 2015), PubTabNet (Zhong et al., 2020), TabLeX (Desai et al., 2021), Im2Latex-100K (Deng et al., 2017), ReadingBank (Wang et al., 2021), and HRDoc (Ma et al., 2023).

prehensively evaluate DSE systems. Additionally, current research often unrealistically targets localized regions, such as layout blocks or tables within a single page. This approach overlooks the complexity of real-world documents, which typically span multiple pages with hierarchical headings and require long-range dependencies to construct a global structure. Benchmarks focusing on isolated pages or blocks fail to provide realistic evaluations.

To address these issues, we introduce READoc, a unified benchmark designed to quantify the gap between existing work and the goal of REAlistic Document Structured Extraction. READoc formulates DSE as an end-to-end task, converting multi-page PDFs into structured Markdown. We automatically construct 3,576 PDF-Markdown pairs from arXiv, GitHub, and Zenodo, covering diverse types, years, and topics to reflect real-world complexity. Additionally, we develop a DSE evaluation S3uite with three modules: Standardization, Segmentation, and Scoring, enabling unified evaluation of diverse DSE systems, including pipeline tools (Paruchuri and Lampa, 2023), expert models (Blecher et al., 2023), and Vision-Language Models (Achiam et al., 2023).

Our contributions are three-fold: 1) READoc is the first benchmark to frame DSE as a PDF-to-Markdown paradigm, which is realistic, end-to-end, and incorporates diverse data. 2) An evaluation S3uite is proposed to support the unified assessment of various DSE systems and to quantify multiple capabilities required for DSE. 3) We present the gap between current research and realistic DSE, emphasizing the importance of exploring new modeling paradigms. The code and data are publicly available at https://github.com/icip-cas/READoc.

2 Related Work

2.1 Task Views and Benchmarks

DSE is a crucial task, yet existing benchmarks focus on discrete subtasks: document layout analysis (Zhong et al., 2019; Li et al., 2020) identifies layout blocks; optical character recognition (Karatzas et al., 2015) extracts text from images; table structure recognition (Zhong et al., 2020) transforms tables into structured formats; mathematical formula conversion (Deng et al., 2017) converts formulas into semantic formats; table-of-contents (ToC) extraction (Ma et al., 2023) constructs hierarchical heading trees; reading order detection (Wang et al., 2021) sorts page elements by reading order. We summarize relevant benchmarks in Table 1. However, their heterogeneity complicates unified DSE evaluation.

Recent research conceptualizes DSE as a single-page image-to-markup task (Blecher et al., 2023; Lee et al., 2023), and targeted benchmarks such as OmniDocBench (Ouyang et al., 2024) have emerged. Although OmniDocBench demonstrates commendable richness in evaluation and diversity of data, its single-page approach falls short in handling multi-page or lengthy documents often encountered in real-world scenarios.

2.2 Methods for Document Structured Extraction

Due to the intricacies of textual, graphical, and layout information within documents (Xu et al., 2020),
a universally accepted method for DSE has yet to emerge. A common simplistic strategy involves leveraging external parsing engines (PyMuPDF, 2024) to extract text and metadata from digital-born PDFs. With the rise of deep learning techniques, Numerous systems (Li et al., 2022; Paruchuri and Lampa, 2023; Contributors, 2024) have integrated a series of submodels into a pipeline, with each submodel dedicated to a specific subtask of DSE. Recent advancements have shifted towards end-to-end DSE methodologies, with some works leveraging Transformer (Vaswani et al., 2017) expert models to convert document page images directly into structured formats such as HTML (Lee et al., 2023) or Markdown (Blecher et al., 2023).

Recently, large Vision-Language Models have garnered widespread attention (Achiam et al., 2023; Liu et al., 2024c), with document understanding (Feng et al., 2023; Hu et al., 2024) being a key focus of their capabilities. Efforts to enhance VLMs, document understanding capability have focused on methods like tailored training tasks (Ye et al., 2023) and resolution adaptation (Li et al., 2024), leading to notable improvements. VLMs obtain impressive results on various DSE subtasks, such as OCR (Liu et al., 2024d), table recognition (Zhao et al., 2024), and formula conversion (Xia et al., 2024). Additionally, some research has explored converting page images into structured text using VLMs (Lv et al., 2023; Wei et al., 2024a; Liu et al., 2024a). However, the lack of a unified benchmark leaves uncertainty about the gap between VLMs’ current capabilities and realistic DSE needs.

3 Task Definition

We establish a realistic task paradigm for end-to-end DSE, using raw PDF documents as input due to their prevalence and unstructured nature, which poses challenges with dispersed content and multimodal information. On the other hand, we employ Markdown as the output format, leveraging its lightweight markup to represent structural elements like headings and lists. We adopt a variant of Markdown (Blecher et al., 2023) that supports LaTeX syntax for tables and formulas. Markdown, as the target format, can be chunked, indexed as flat text, or directly ingested by LLMs.

In summary, READOC uniformly defines DSE as a task that takes a complete PDF file as input and generates structured text in Markdown format, which is well-defined, practical, and challenging for DSE systems. Examples of task inputs and outputs are provided in Appendix A.

4 Benchmark Construction

READOC is a unified DSE benchmark derived from real-world documents. We select heterogeneous documents from arXiv preprints¹, GitHub READMEs² and Zenodo’s Open Research Repository³, which are then automatically processed to construct PDF-Markdown pairs. READOC consists of 3,576 documents: 1,009 in the READOC-arXiv subset, 1,224 in the READOC-GitHub subset, and 1,343 in the READOC-Zenodo subset. Each subset offers unique challenges: READOC-arXiv features complex academic structures such as formulas and tables, with diverse multi-column layout templates. However, its heading styles are simple, often following easily recognizable patterns like "1.1 Introduction." In contrast, READOC-GitHub includes only basic elements like paragraphs and headings, and presents a uniform single-column layout style. However, building its ToC structure is challenging due to varied and often unmarked heading styles. READOC-Zenodo contains longer documents (many exceeding 30 pages) and diverse types, such as posters, reports, theses, and books, challenging layout analysis. Additionally, its 27 languages further increase text processing and semantic understanding complexity. Each subset exhibits significant diversity in types, topics, eras, and so on, establishing READOC as a robust benchmark. We describe more construction details in Appendix B.

4.1 Document Collection and Processing

READOC-arXiv. For the arXiv preprints, the collection process involves using type keywords such as “Conference” and “Journal” to select papers and filtering for English language. Preprints without LaTeX files or with unclear main LaTeX files are excluded. The selected documents are first converted from LaTeX to HTML using LaTeXML⁴, followed by a conversion from HTML to Markdown using a modified version of the Nougat (Blecher et al., 2023) process. Only documents that complete this process without any errors and maintain correct table syntax after conversion are included.

¹https://arxiv.org/
²https://github.com/
³https://zenodo.org/
⁴https://github.com/brucemiller/LaTeXML
Figure 2: Visualization of data distribution in READOC. (a) Document disciplines of READOC-arXiv. (b) Document topics of READOC-GitHub. (c) Language distribution of READOC-Zenodo.

<table>
  <tr>
    <th>Statistics</th>
    <th>arXiv</th>
    <th>GitHub</th>
    <th>Zenodo</th>
  </tr>
  <tr>
    <td>Documents</td>
    <td>1,009</td>
    <td>1,224</td>
    <td>1,343</td>
  </tr>
  <tr>
    <td>Avg. Pages</td>
    <td>11.67</td>
    <td>6.54</td>
    <td>14.93</td>
  </tr>
  <tr>
    <td>Avg. Depth</td>
    <td>3.10</td>
    <td>3.11</td>
    <td>2.66</td>
  </tr>
  <tr>
    <td>Avg. Length</td>
    <td>10,209.50</td>
    <td>1,978.10</td>
    <td>8,255.85</td>
  </tr>
  <tr>
    <td>Year Span</td>
    <td>1996 - 2024</td>
    <td>2008 - 2024</td>
    <td>2014 - 2024</td>
  </tr>
  <tr>
    <td>Types / Disciplines</td>
    <td>6 / 8</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Topics</td>
    <td>-</td>
    <td>2,805</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Language count</td>
    <td>1</td>
    <td>1</td>
    <td>27</td>
  </tr>
</table>

Table 2: The data statistics of READOC.

READDoc-GitHub. The GitHub README files are originally in Markdown format, and we collect and filter them based on specific criteria: they must have obtained than 500 stars, be written in English, exclude HTML syntax, etc. To maintain the simplicity of this subset, we exclude files that contain tables and formulas. After initial preprocessing, the Markdown files are converted to PDFs using Pandoc\footnote{https://github.com/jgm/pandoc} as the conversion engine and Eisvogel\footnote{https://github.com/Wandmalfarbe/pandoc-latex-template} as the template. Only documents that successfully complete this entire workflow without execution errors or warnings are retained.

READDoc-Zenodo. For the Zenodo dataset, we collect 910 DOCX and 433 HTML files from Zenodo. DOCX files are converted to Markdown using Microsoft’s Markitdown (microsoft, 2024) tool, then to PDF via python-docx\footnote{https://pypi.org/project/python-docx/}. HTML files are converted to Markdown using Pandoc\footnote{https://github.com/jgm/pandoc}, then to PDF via Chromium. We exclude conversion failures and overly long/short files, retaining only structurally rich documents.

4.2 Dataset Statistics

We present the basic statistics of READOC in Table 2, showcase the diversity of READOC in Figure 2, and provide additional statistics in Appendix B. Overall, our benchmark is divided into three subsets. READOC-arXiv consists of 1,009 documents, with an average of 11.67 pages, 10,209.50 tokens, and 3.10 heading levels. These documents cover a timeline from 1996 to 2024, ensuring ample diversity across 6 types and 8 disciplines. READOC-GitHub comprises 1,224 documents, with an average of 6.54 pages, 1,978.11 tokens, and 3.11 heading levels. These documents are sourced from projects spanning the years 2008 to 2024, encompassing a rich tapestry of 2,805 topics. READOC-Zenodo contains 1,343 documents, with an average of 14.93 pages, 8,255.85 tokens, and 2.66 heading levels. These documents span from 2014 to 2024 and include 27 languages, e.g., English and French.

5 Evaluation S$^3$uite

Considering the potential confusion that various models may have with our Markdown syntax and the multifaceted nature of the capabilities required for DSE tasks, we propose an Evaluation S$^3$uite, consisting of three sequential modules: Standardization, Segmentation, and Scoring, as shown in Figure 3. The S$^3$uite ensures that READOC can automatically perform a unified evaluation of the end-to-end DSE task, yielding reliable and effective assessment results. More implementation details are in Appendix C.

5.1 Standardization

The first module of the S$^3$uite standardizes the output Markdown text to align with the ground truth Markdown format. This alignment is essential for mitigating the impact on evaluation accuracy caused by variations in the format and syntax
<table>
  <tr>
    <th>Standardization</th>
    <th>Segmentation</th>
    <th>Scoring</th>
  </tr>
  <tr>
    <td>
      <b>Constraining Coefficient</b><br>
      ===<br>
      Coefficient<br>
      ...<br>
      \begin{equation}<br>
      D_e=\text{extrm}(V)^{-\delta}\langle\delta\rangle<br>
      \end{equation><br>
      (see equation)<br>
      the diffusion index \( \langle\delta\rangle \) is 1/3.<br>
      index $|\delta|_S$ is 1/3.<br>
      ![figure_1.jpg](figure_1.jpg)<br>
      name[acc]<br>
      [x:1]---[1]<br>
      [x|88.7]<br>
      [y|67.8]
    </td>
    <td>
      <b># Constraining Coefficient</b><br>
      #D_e=\text{extrm}(GeV)^{-\langle\delta\rangle}\\
      the diffusion index \( \langle\delta\rangle \) is 1/3.<br>
      \begin{table}
        \begin{tabular}{c r}
          X & 88.7 \\
          Y & 67.8
        \end{tabular}
      \end{table}
    </td>
    <td>
      <b>Semantic Unit Evaluation</b><br>
      Heading Detection - Concat EDS - Tree EDS<br>
      Table Recognition - Concat EDS - Tree EDS<br>
      Reading Order Detection - Block-level KTDS - Token-level KTDS<br>
      <b>Formula Conversion</b><br>
      Formula Conversion - Embedded EDS - Isolated EDS<br>
      <b>Text Extraction</b><br>
      Text Extraction - Concat EDS - Vocab FT<br>
    </td>
  </tr>
  <tr>
    <td colspan="2">
      <b>Headings</b><br>
      <b>Formulas</b><br>
      <b>Tables</b><br>
      <b>Plain Text</b>
    </td>
    <td></td>
  </tr>
</table>

Figure 3: The three modules of the READOC Evaluation S^3uite: Standardization, Segmentation and Scoring.

of texts generated by different DSE systems. The standardization includes aligning the boundaries of formulas, such as $$, \begin{equation}, and \end{equation}; unifying different Markdown heading styles; aligning Markdown tables with LATEX table formats for evaluation consistency; removing images and eliminating the link syntax, etc. This module ensures that the focus remains on the core DSE capabilities, rather than being clouded by formatting disunity.

5.2 Segmentation

The second module of the S^3uite divides the standardized Markdown text into distinct semantic units. To facilitate READOC in highlighting various specialized DSE capabilities of the models within a single document, we divide both the output and the ground truth Markdown text into four units: Headings of different levels, Formulas in both embedded and isolated forms, Tables, and residual Plain Text encompassing basic text and simple formatting such as bold, italic, and lists.

5.3 Scoring

The scoring module comprises two submodules:

The Semantic Unit Evaluation submodule implicitly assesses the models’ layout analysis ability of identifying semantic units and explicitly measures four specialized capabilities: 1) Text Extraction refers to extracting plain text through PDF bytecode parsing or visual methods (e.g., OCR tools, VLMs). We measure edit distance similarity (EDS) after concatenating plain text and the F1 score of the plain text vocabulary. 2) Heading Detection involves detecting headings and constructing a hierarchical ToC tree. We measure EDS after concatenating all headings and construct ToC trees to calculate tree edit distance similarity (TEDS). 3) Formula Conversion involves transforming mathematical formulas into LATEX format. We measure EDS by concatenating all embedded formulas, as well as all isolated formulas. 4) Table Recognition involves identifying tables’ structures. We evaluate EDS after concatenating all tables. Then, we convert tables into structural trees, use a maximum bipartite matching algorithm to find the optimal mapping between tables and calculate TEDS for the matched tables.

The Reading Order Detection submodule determines whether the model extracts document elements in the correct order. We first segment each document into blocks based on semantic unit boundaries and line breaks, create two ordered lists from these blocks, and calculate Kendall’s Tau Distance Similarity (KTDS) between the lists. Additionally, we divide each document into sequential tokens, construct lists based on the positions of co-occurring tokens at their first appearance, and compute KTDS between the token-level lists.

6 Experiments

6.1 Compared Methods

Baselines. We employ three simple and widely applicable methods as baselines. The first is PyMuPDF4LLM (PyMuPDF, 2024), a PDF bytecode parsing engine that converts digital-born PDFs into Markdown using embedded metadata. The second is Tesseract (Smith, 2007), an OCR tool for text extraction and basic page segmentation. The third is MarkItDown (microsoft, 2024), a versatile tool designed for converting various file types into Markdown format.

Pipeline Tools. We evaluate three tools that support PDF-to-Markdown functionality, which integrate complex engineering with PDF parsing engines and advanced deep learning submodels. The pipeline tools we evaluate include the following. Marker (Paruchuri and Lampa, 2023), MinerU
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="6">Semantic Unit Evaluation</th>
    <th rowspan="2">Reading Order</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>Text<br>Concat Vocab</th>
    <th>Heading<br>Concat Tree</th>
    <th>Formula<br>Embed Isolate</th>
    <th>Table<br>Concat Tree</th>
    <th></th>
    <th>Block Token</th>
  </tr>
  <tr>
    <th colspan="9">Baselines</th>
  </tr>
  <tr>
    <td>PyMuPDF4LLM</td>
    <td>66.66</td>
    <td>74.27</td>
    <td><b>27.86</b></td>
    <td>20.77</td>
    <td>0.07</td>
    <td><b>0.02</b></td>
    <td>23.27</td>
    <td><b>15.83</b></td>
    <td>87.70</td>
    <td>89.09</td>
    <td><b>40.55</b></td>
  </tr>
  <tr>
    <td>Tesseract OCR</td>
    <td><b>78.85</b></td>
    <td>76.51</td>
    <td>1.26</td>
    <td>0.30</td>
    <td><b>0.12</b></td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>96.70</td>
    <td><b>97.59</b></td>
    <td>35.13</td>
  </tr>
  <tr>
    <td>MarkItDown</td>
    <td>73.88</td>
    <td><b>79.64</b></td>
    <td>2.90</td>
    <td>0.93</td>
    <td>0.10</td>
    <td>0.00</td>
    <td>0.34</td>
    <td>0.08</td>
    <td><b>97.13</b></td>
    <td>97.08</td>
    <td>35.21</td>
  </tr>
  <tr>
    <th colspan="9">Pipeline Tools</th>
  </tr>
  <tr>
    <td>MinerU</td>
    <td><b>88.32</b></td>
    <td><b>91.22</b></td>
    <td>67.06</td>
    <td><b>41.97</b></td>
    <td><b>62.77</b></td>
    <td><b>70.76</b></td>
    <td>59.34</td>
    <td>52.85</td>
    <td><b>98.52</b></td>
    <td><b>97.90</b></td>
    <td><b>73.07</b></td>
  </tr>
  <tr>
    <td>Pix2Text</td>
    <td>85.85</td>
    <td>83.72</td>
    <td>63.23</td>
    <td>34.53</td>
    <td>43.18</td>
    <td>37.45</td>
    <td>54.08</td>
    <td>47.35</td>
    <td>97.68</td>
    <td>96.78</td>
    <td>64.39</td>
  </tr>
  <tr>
    <td>Marker</td>
    <td>79.11</td>
    <td>82.71</td>
    <td>63.60</td>
    <td>39.39</td>
    <td>3.47</td>
    <td>48.74</td>
    <td><b>64.61</b></td>
    <td><b>72.36</b></td>
    <td>98.04</td>
    <td>97.74</td>
    <td>64.98</td>
  </tr>
  <tr>
    <td>Docling</td>
    <td>79.73</td>
    <td>85.39</td>
    <td><b>68.74</b></td>
    <td>38.33</td>
    <td>0.23</td>
    <td>0.0</td>
    <td>54.09</td>
    <td>66.56</td>
    <td>98.05</td>
    <td>97.18</td>
    <td>58.83</td>
  </tr>
  <tr>
    <th colspan="9">Expert Visual Models</th>
  </tr>
  <tr>
    <td>Nougat-small</td>
    <td>87.35</td>
    <td>92.00</td>
    <td>86.40</td>
    <td>87.88</td>
    <td><b>76.52</b></td>
    <td>79.39</td>
    <td><b>55.63</b></td>
    <td><b>52.35</b></td>
    <td>97.97</td>
    <td>98.36</td>
    <td>81.38</td>
  </tr>
  <tr>
    <td>Nougat-base</td>
    <td><b>88.03</b></td>
    <td><b>92.29</b></td>
    <td><b>86.60</b></td>
    <td><b>88.50</b></td>
    <td>76.19</td>
    <td><b>79.47</b></td>
    <td>54.40</td>
    <td>52.30</td>
    <td><b>97.98</b></td>
    <td><b>98.41</b></td>
    <td><b>81.42</b></td>
  </tr>
  <tr>
    <td>GOT-OCR 2.0</td>
    <td>84.47</td>
    <td>86.24</td>
    <td>66.69</td>
    <td>57.68</td>
    <td>53.48</td>
    <td>56.23</td>
    <td>50.40</td>
    <td>34.50</td>
    <td>97.73</td>
    <td>97.50</td>
    <td>68.49</td>
  </tr>
  <tr>
    <th colspan="9">Vision-Language Models</th>
  </tr>
  <tr>
    <td>DeepSeek-VL-7B-Chat</td>
    <td>31.89</td>
    <td>39.96</td>
    <td>23.66</td>
    <td>12.53</td>
    <td>17.01</td>
    <td>16.94</td>
    <td>22.96</td>
    <td>16.47</td>
    <td>88.76</td>
    <td>66.75</td>
    <td>33.69</td>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-V2.5</td>
    <td>58.91</td>
    <td>70.87</td>
    <td>26.33</td>
    <td>7.68</td>
    <td>16.70</td>
    <td>17.90</td>
    <td>27.89</td>
    <td>24.91</td>
    <td>95.26</td>
    <td>93.02</td>
    <td>43.95</td>
  </tr>
  <tr>
    <td>LLaVa-1.6-Vicuna-13B</td>
    <td>27.51</td>
    <td>37.09</td>
    <td>8.92</td>
    <td>6.27</td>
    <td>17.80</td>
    <td>11.68</td>
    <td>23.78</td>
    <td>16.23</td>
    <td>76.63</td>
    <td>51.68</td>
    <td>27.76</td>
  </tr>
  <tr>
    <td>InternVL-Chat-V1.5</td>
    <td>53.06</td>
    <td>68.44</td>
    <td>25.03</td>
    <td>13.57</td>
    <td>33.13</td>
    <td>24.37</td>
    <td>40.44</td>
    <td>34.35</td>
    <td>94.61</td>
    <td>91.31</td>
    <td>47.83</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td><b>79.44</b></td>
    <td><b>84.37</b></td>
    <td><b>31.77</b></td>
    <td><b>18.65</b></td>
    <td><b>42.23</b></td>
    <td><b>41.67</b></td>
    <td><b>47.81</b></td>
    <td><b>39.85</b></td>
    <td><b>97.69</b></td>
    <td><b>96.35</b></td>
    <td><b>57.98</b></td>
  </tr>
</table>

Table 3: Evaluation of various Document Structured Extraction systems on READOC-arXiv.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="6">Semantic Unit Evaluation</th>
    <th rowspan="2">Reading Order</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>Text<br>Concat Vocab</th>
    <th>Heading<br>Concat Tree</th>
    <th>Formula<br>Embed Isolate</th>
    <th>Table<br>Concat Tree</th>
    <th></th>
    <th>Block Token</th>
  </tr>
  <tr>
    <td>MinerU</td>
    <td>57.28</td>
    <td>59.95</td>
    <td>30.73</td>
    <td>22.83</td>
    <td>38.75</td>
    <td>26.82</td>
    <td>65.46</td>
    <td>66.87</td>
    <td>46.08</td>
  </tr>
  <tr>
    <td>Marker</td>
    <td>59.34</td>
    <td>61.68</td>
    <td>30.28</td>
    <td>18.29</td>
    <td>40.68</td>
    <td>29.10</td>
    <td>57.77</td>
    <td>66.40</td>
    <td>46.44</td>
  </tr>
  <tr>
    <td>Nougat-base</td>
    <td>57.54</td>
    <td>66.87</td>
    <td><b>35.99</b></td>
    <td><b>26.98</b></td>
    <td>13.99</td>
    <td>11.55</td>
    <td>93.56</td>
    <td>93.01</td>
    <td>49.94</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td><b>64.16</b></td>
    <td><b>71.76</b></td>
    <td>25.07</td>
    <td>15.4</td>
    <td><b>45.75</b></td>
    <td><b>31.88</b></td>
    <td><b>95.23</b></td>
    <td><b>95.2</b></td>
    <td><b>55.56</b></td>
  </tr>
</table>

Table 5: Evaluation of three representative Document Structured Extraction systems on READOC-Zenodo.

(Contributors, 2024), Pix2Text (Breezedeus, 2022) and Docling (Auer et al., 2024).

Expert Visual Models. We evaluate Nougat-small and Nougat-base (Blecher et al., 2023), specialized transformer models trained on arXiv academic documents under the single-page image-to-Markdown paradigm, with parameter sizes of 250M and 350M, respectively. Additionally, we evaluate the GOT-OCR 2.0 model (Wei et al., 2024b), a unified, end-to-end OCR system with 580M parameters.

Vision-Language Models. We evaluate VLMS using the same single-page image-to-Markdown paradigm as expert models. For efficiency, we select open-source models with fewer than 30 billion parameters and lightweight proprietary models. Only VLMS with basic instruction-following and preliminary Markdown understanding are retained. The retained VLMS include: the open-source models DeepSeek-VL-7B-Chat (Lu et al., 2024), MiniCPM-Llama3-V2.5 (Hu et al., 2023), LLaVa-1.6-Vicuna-13B (Liu et al., 2024b), InternVL-Chat-V1.5 (Chen et al., 2024), and the proprietary model GPT-4o-mini (Achiam et al., 2023). More implementation details are included in Appendix D.

6.2 Experimental Results

Results for READOC-arXiv, READOC-GitHub and READOC-Zenodo are presented in Table 3, 4 and 5, respectively. We draw insights from both DSE system categories and specialized capabilities.

From the point of DSE system categories, we observe that: 1) Pipeline tools are often plagued by complex engineering challenges. Docling falls short in recognizing embedded and isolated formulas, while Maker struggles with detecting embedded formulas, issues that are overlooked within the complex pipeline designs. In contrast, MinerU performs significantly better than other pipeline
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="4">Semantic Unit Evaluation</th>
    <th colspan="2">Reading Order</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Text<br>Concat</th>
    <th>Vocab</th>
    <th>Heading<br>Concat</th>
    <th>Tree</th>
    <th>Block</th>
    <th>Token</th>
  </tr>
  <tr>
    <th colspan="8">Baselines</th>
  </tr>
  <tr>
    <td>PyMuPDF4LLM</td>
    <td>85.21</td>
    <td>77.27</td>
    <td><b>12.13</b></td>
    <td><b>11.05</b></td>
    <td>98.61</td>
    <td>98.43</td>
    <td><b>63.78</b></td>
  </tr>
  <tr>
    <td>Tesseract OCR</td>
    <td>82.06</td>
    <td>82.32</td>
    <td>6.65</td>
    <td>4.25</td>
    <td>98.48</td>
    <td>99.01</td>
    <td>62.13</td>
  </tr>
  <tr>
    <td>MarkItDown</td>
    <td>85.23</td>
    <td>85.69</td>
    <td>2.07</td>
    <td>0.64</td>
    <td><b>99.4</b></td>
    <td><b>99.42</b></td>
    <td>62.07</td>
  </tr>
  <tr>
    <th colspan="8">Pipeline Tools</th>
  </tr>
  <tr>
    <td>MinerU</td>
    <td>84.46</td>
    <td>84.78</td>
    <td>67.24</td>
    <td><b>47.15</b></td>
    <td><b>99.51</b></td>
    <td><b>99.18</b></td>
    <td>80.39</td>
  </tr>
  <tr>
    <td>Pix2Text</td>
    <td>78.99</td>
    <td>78.51</td>
    <td>60.53</td>
    <td>39.42</td>
    <td>97.94</td>
    <td>97.38</td>
    <td>75.46</td>
  </tr>
  <tr>
    <td>Marker</td>
    <td><b>89.50</b></td>
    <td><b>88.11</b></td>
    <td><b>72.81</b></td>
    <td>37.51</td>
    <td>99.03</td>
    <td>99.13</td>
    <td><b>81.02</b></td>
  </tr>
  <tr>
    <td>Dooling</td>
    <td>44.53</td>
    <td>50.73</td>
    <td>40.17</td>
    <td>22.8</td>
    <td>70.77</td>
    <td>70.19</td>
    <td>49.87</td>
  </tr>
  <tr>
    <th colspan="8">Expert Visual Models</th>
  </tr>
  <tr>
    <td>Nougat-small</td>
    <td>76.04</td>
    <td>77.11</td>
    <td><b>62.81</b></td>
    <td>38.73</td>
    <td>98.33</td>
    <td>96.12</td>
    <td>74.86</td>
  </tr>
  <tr>
    <td>Nougat-base</td>
    <td>75.26</td>
    <td>76.79</td>
    <td>62.39</td>
    <td>37.01</td>
    <td>97.65</td>
    <td>95.62</td>
    <td>74.12</td>
  </tr>
  <tr>
    <td>GOT-OCR 2.0</td>
    <td><b>78.18</b></td>
    <td><b>84.53</b></td>
    <td>62.08</td>
    <td><b>50.06</b></td>
    <td><b>98.40</b></td>
    <td><b>98.60</b></td>
    <td><b>78.64</b></td>
  </tr>
  <tr>
    <th colspan="8">Vision-Language Models</th>
  </tr>
  <tr>
    <td>DeepSeek-VL-7B-Chat</td>
    <td>34.85</td>
    <td>41.97</td>
    <td>38.69</td>
    <td>20.15</td>
    <td>96.47</td>
    <td>84.55</td>
    <td>52.78</td>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-V2.5</td>
    <td>71.61</td>
    <td>78.88</td>
    <td>39.66</td>
    <td>22.99</td>
    <td>97.91</td>
    <td>97.97</td>
    <td>68.17</td>
  </tr>
  <tr>
    <td>LLaVa-1.6-Vicuna-13B</td>
    <td>37.03</td>
    <td>52.88</td>
    <td>27.49</td>
    <td>16.13</td>
    <td>95.65</td>
    <td>90.45</td>
    <td>53.27</td>
  </tr>
  <tr>
    <td>InternVL-Chat-V1.5</td>
    <td>72.59</td>
    <td>78.27</td>
    <td>56.85</td>
    <td>32.70</td>
    <td>98.39</td>
    <td>97.69</td>
    <td>72.75</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td><b>85.06</b></td>
    <td><b>89.41</b></td>
    <td><b>62.65</b></td>
    <td><b>42.04</b></td>
    <td><b>98.81</b></td>
    <td><b>99.05</b></td>
    <td><b>79.50</b></td>
  </tr>
</table>

Table 4: Evaluation of various DSE systems on READOC-GitHub.

![Line graphs showing Method Performance vs Document Depth and Method Performance vs Document Length](page_1012_120_382_382.png)

Figure 4: Relationship between DSE systems’ performance and the depth or length of documents in READOC-GitHub.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th rowspan="2">Modeling Paradigm</th>
    <th colspan="4">Semantic Unit Evaluation</th>
    <th colspan="2">Reading Order</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>Text<br>Concat</th>
    <th>Vocab</th>
    <th>Heading<br>Concat</th>
    <th>Formula<br>Embed Isolate Concat Tree</th>
    <th>Table<br>Concat</th>
    <th>Order<br>Block Token</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>Single Page</td>
    <td><b>77.95</b></td>
    <td><b>83.10</b></td>
    <td>40.63</td>
    <td>23.82</td>
    <td>39.03</td>
    <td>54.05</td>
    <td>44.70</td>
    <td>97.79</td>
    <td>96.61</td>
    <td>59.82</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>Multiple Pages</td>
    <td>70.28</td>
    <td>78.66</td>
    <td><b>62.71</b></td>
    <td><b>51.87</b></td>
    <td>33.22</td>
    <td>33.62</td>
    <td>43.50</td>
    <td>34.09</td>
    <td><b>98.87</b></td>
    <td><b>96.31</b></td>
    <td><b>60.31</b></td>
  </tr>
</table>

Table 6: Comparison of page-level modeling paradigms, for documents within 5 pages of READOC-arXiv.

models in both embedded and isolated formula recognition. Besides, Pix2Text encounters program crashes and fails to process certain PDFs (3 files on READOC-arXiv and 4 on READOC-GitHub), posing a significant usability issue. 2) Expert models struggle with generalization and scalability issues. Nougat performs well on READOC-arXiv but declines markedly on READOC-GitHub (i.e., from 81.42 to 74.12) and READOC-Zenodo (e.g., from 81.42 to 49.94), which features simpler layouts and fewer semantic units, indicating poor transfer learning ability. Moreover, scaling up from Nougat-small to Nougat-base does not boost performance (+0.04 on READOC-arXiv, -0.74 on READOC-GitHub). Additionally, while GOT-OCR 2.0 scores lower than Nougat-small and Nougat-base on READOC-arXiv, it outperforms them on the READOC-GitHub subset. 3) VLMs generally underperform in complex academic documents. The best-performing open-source model, InternVL-Chat-V1.5, scores 47.83 on READOC-arXiv, while the proprietary model GPT-4o-mini scores 57.98, both of which are lower than the pipeline tools. On READOC-Zenodo, GPT-4o-mini achieves an average score of 55.56, suggesting its potential in handling multilingual and multi-format documents.

From the perspective of specialized capabilities, we observe that: 1) Building hierarchical ToC trees from a global perspective remains a significant challenge, as existing systems predominantly focus on single-page images. Pipeline tools lack modules to assess heading depth, leading to substantial drops in Tree EDS compared to Concat EDS. Expert models can exhibit strong ToC construction for specific documents, which is more a superficial imitation rather than a semantic understanding of the logical structure. On READOC-arXiv, Nougat-base scores 88.50 in TEDS, while on READOC-GitHub, it drops to 37.01. 2) Understanding localized structured data such as tables and formulas is relatively difficult. VLMs perform poorly on these tasks. Even the expert model Nougat-base, trained on arXiv documents,
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">Semantic Unit Evaluation (Avg.)</th>
  </tr>
  <tr>
    <th>Single-col.</th>
    <th>Multi-col.</th>
    <th>Drop ↓</th>
  </tr>
  <tr>
    <td>MinerU</td>
    <td>52.75</td>
    <td>48.34</td>
    <td>4.41</td>
  </tr>
  <tr>
    <td>Pix2Text</td>
    <td><b>56.93</b></td>
    <td><b>55.34</b></td>
    <td><b>1.59</b></td>
  </tr>
  <tr>
    <td>Marker</td>
    <td>56.70</td>
    <td>53.39</td>
    <td>3.31</td>
  </tr>
  <tr>
    <td>Nougat-small</td>
    <td><b>80.51</b></td>
    <td>74.09</td>
    <td>6.42</td>
  </tr>
  <tr>
    <td>Nougat-base</td>
    <td>80.48</td>
    <td><b>74.16</b></td>
    <td><b>6.32</b></td>
  </tr>
  <tr>
    <td>InternVL-Chat-V1.5</td>
    <td>39.96</td>
    <td>32.83</td>
    <td>7.13</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td><b>49.23</b></td>
    <td><b>47.08</b></td>
    <td><b>2.15</b></td>
  </tr>
</table>

Table 7: Relationship between DSE systems’ semantic unit evaluation and the layout of documents in READOC-arXiv.

<table>
  <tr>
    <th>Methods</th>
    <th>Modeling Paradigm</th>
    <th>Heading<br>Concat</th>
    <th>Tree</th>
  </tr>
  <tr>
    <td>Pix2Text</td>
    <td>Single Page</td>
    <td>63.38</td>
    <td>38.89</td>
  </tr>
  <tr>
    <td>Nougat-small</td>
    <td>Single Page</td>
    <td>64.87</td>
    <td>39.53</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>Single Page<br>Multiple Pages</td>
    <td>69.78<br><b>83.71</b></td>
    <td>45.12<br><b>68.11</b></td>
  </tr>
</table>

Table 8: Comparison of page-level modeling paradigms, for documents within 5 pages of READOC-GitHub.

has shown only modest performance, with average metrics of 65.56 on these two tasks. 3) Reading Order Detection is a relatively easy capability to acquire. The baseline tool Tesseract, which uses heuristics for page segmentation, scores 96.70 and 98.48 in block-level KTDS on READOC-arXiv and READOC-GitHub, respectively.

6.3 Fine-grained Analysis

Impact of Document Length and Depth. Figure 4 displays the results of three representative DSE systems on READOC-GitHub. Pipeline tools and expert models exhibit similar performance trends, remaining stable with variations in document length but declining sharply as document depth increases. In contrast, VLMs demonstrate stability with changes in document depth, while their performance decreases as document length increases. This indicates that different DSE systems exhibit distinct shortcomings in realistic scenarios, which have not been previously revealed.

Impact of Document Layout. Using pdfplumber (Singer-Vine and The pdfplumber contributors, 2024) and heuristic rules, we classify documents in READOC-arXiv into single- and multi-column categories, representing different layout complexities. Table 7 illustrates the average semantic unit evaluation scores across the two document types. All systems show performance degradation on complex multi-column documents, highlighting that our semantic unit evaluation implicitly measures the layout analysis capability. Among the systems, GPT-4o-mini exhibits the best layout analysis capability, while InternVL-Chat-V1.5 shows the most significant performance decline, reflecting substantial differences in performance levels among VLMs.

Exploration of the Multi-Page Paradigm. Previous researches focus on processing single pages and perform poorly in constructing global ToC trees. To explore how DSE systems might utilize global information, we investigate a paradigm where multiple pages are processed simultaneously. Specifically, we employ GPT-4o-mini to receive all page images of the document at once and convert them into Markdown text. We conduct experiments on documents with up to 5 pages, as shown in Tables 6 and 8. While this method significantly enhances global ToC construction compared to the single-page paradigm, processing multiple pages simultaneously reduces local fine-grained capabilities, such as table and formula conversion, indicating that DSE modeling for multi-page documents still needs further development.

Considerations of Efficiency. DSE focuses on practical efficiency, which may involve real-time RAG calls or large-scale corpus construction. We sample 50 documents from READOC-arXiv and measure the throughput of DSE systems, as shown in Table 9. Despite significant advancements in GPU memory and computational power, VLMs remains considerably lower efficiency compared to pipeline tools and expert Transformer models, indicating the need for future improvements in not only performance but also efficiency.

<table>
  <tr>
    <th>Methods</th>
    <th>Time Cost (s) per Document</th>
    <th>NVIDIA GPU Devices</th>
  </tr>
  <tr>
    <td>Marker</td>
    <td>23.86</td>
    <td></td>
  </tr>
  <tr>
    <td>MinerU</td>
    <td>30.96</td>
    <td>1× Titan RTX (24GB)</td>
  </tr>
  <tr>
    <td>Nougat-small</td>
    <td>51.34</td>
    <td></td>
  </tr>
  <tr>
    <td>Nougat-base</td>
    <td>101.07</td>
    <td></td>
  </tr>
  <tr>
    <td>Pix2Text</td>
    <td>188.10</td>
    <td></td>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-V2.5</td>
    <td>392.37</td>
    <td>1× A100 (80GB)</td>
  </tr>
  <tr>
    <td>InternVL-Chat-V1.5</td>
    <td>1,182.02</td>
    <td>2× A100 (80GB)</td>
  </tr>
</table>

Table 9: Comparison of Efficiency of DSE systems.

6.4 Case Study

We compare results from four representative systems in Figure 5. Key observations include: 1)
Figure 5: A case study from READoc-GitHub. More cases are in Appendix D.

GPT-4o-mini misclassifies heading levels, revealing the limitations of single-page paradigms that fail to globally perceive the document’s logical structure. 2) InternVL-Chat-V1.5 exhibits hallucinations and inaccurate text extraction, illustrating the differences in DSE capabilities between open-source and proprietary VLMs. 3) Nougat-small fabricates content completely unrelated to the original document, reflecting the poor generalization ability of expert models. 4) Pix2Text triggers an execution error, demonstrating the complexity in developing pipeline tools.

7 Conclusion

This paper proposes READOC, a novel benchmark that frames document structured extraction as a realistic, end-to-end task, i.e., transforming unstructured PDFs into semantically rich Markdown text. Based on an evaluation S^3uite, We conduct a unified evaluation of state-of-the-art approaches, including pipeline tools, expert models and general VLMs. Our experimental results reveal critical gaps in current researches when applied to realistic scenarios and underscore the importance of exploring new research paradigms.

Limitations

Our work has two main limitations: 1) First, there exists some noise in the PDF-Markdown pairs generated through the automated framework. Although we have implemented various filtering and post-processing methods to minimize this impact, it remains difficult to eliminate completely. In fact, this is a common challenge in current document processing benchmark fields, and we will continue to explore more efficient and accurate processing paradigms in the future. 2) Second, although we introduce a Standardization module in our DSE evaluation S^3uite to unify the outputs from different models to the ground truth format, there are still some scenarios we cannot fully account for. We plan to introduce more comprehensive format unification modules in future work.

Ethics Statement

All the data, tools and model weights we use come from publicly available sources. We only use them for evaluation purposes in document structured extraction. When using these resources for this study, we strictly adhere to their licensing agreements.

Acknowledgment

We sincerely thank the reviewers for their insightful comments and valuable suggestions. This work was supported by Beijing Municipal Science and Technology Project (Nos. Z231100010323002), Beijing Natural Science Foundation (L243006), the Natural Science Foundation of China (No. 62306303, 62476265) and the Basic Research Program of ISCAS (Grant No. ISCAS-JCZD-202401).

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shayamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Srikan Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. 2021. Docformer: End-to-end transformer for document understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 993–1003.
Christoph Auer, Maksym Lysak, Ahmed Nassar, Michele Dolfi, Nikolaos Livathinos, Panos Vagennas, Cesar Berrospi Ramis, Matteo Omenetti, Fabian Lindlbauer, Kasper Dinkla, Lokesh Mishra, Yusik Kim, Shubham Gupta, Rafael Teixeira de Lima, Valery Weber, Lucas Morin, Ingmar Meijer, Viktor Kuropiatnyk, and Peter W. J. Staar. 2024. Docling technical report. Preprint, arXiv:2408.09869.

Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic. 2023. Nougat: Neural optical understanding for academic documents. arXiv preprint arXiv:2308.13418.

Breezedeus. 2022. Pix2text (p2t). https://github.com/breezedeus/pix2text. Accessed: 2024-07-18.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821.

MinerU Contributors. 2024. Mineru: A one-stop, open-source, high-quality data extraction tool. https://github.com/opendatalab/MinerU/. Accessed: 2025-04-14.

Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M Rush. 2017. Image-to-markup generation with coarse-to-fine attention. In International Conference on Machine Learning, pages 980–989. PMLR.

Harsh Desai, Pratik Kayal, and Mayank Singh. 2021. Tablex: a benchmark dataset for structure and content information extraction from scientific tables. In Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part II 16, pages 554–569. Springer.

Hao Feng, Qi Liu, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang. 2023. Docpacle: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding. arXiv preprint arXiv:2311.11810.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.

Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. 2024. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403.12895.

Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. Large multilingual models pivot zero-shot multi-modal learning across languages. arXiv preprint arXiv:2308.12038.

Pengfei Hu, Zhenrong Zhang, Jianshu Zhang, Jun Du, and Jiajia Wu. 2022. Multimodal tree decoder for table of contents extraction in document images. In 2022 26th international conference on pattern recognition (ICPR), pages 1756–1762. IEEE.

Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutlmv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4083–4091.

Sarthak Jain, Madeleine van Zuylten, Hannaneh Hajishirzi, and Iz Beltagy. 2020. Scirex: A challenge dataset for document-level information extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7506–7516.

Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. 2015. Icdar 2015 competition on robust reading. In 2015 13th international conference on document analysis and recognition (ICDAR), pages 1156–1160. IEEE.

Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenchslo, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning, pages 18893–18912. PMLR.

Chenxia Li, Ruoyu Guo, Jun Zhou, Mengtao An, Yuning Du, Lingfeng Zhu, Yi Liu, Xiaoguang Hu, and Dianhai Yu. 2022. Pp-structurev2: A stronger document analysis system. arXiv preprint arXiv:2210.05391.

Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujin Li, and Ming Zhou. 2020. Docbank: A benchmark dataset for document layout analysis. In Proceedings of the 28th International Conference on Computational Linguistics, pages 949–960.

Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. 2024. Monkey: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26763–26773.

Chenglong Liu, Haoran Wei, Jinyue Chen, Lingyu Kong, Zheng Ge, Zining Zhu, Liang Zhao, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. 2024a. Focus anywhere for fine-grained multi-page document understanding. arXiv preprint arXiv:2405.14295.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024b. Llavanext: Improved reasoning, ocr, and world knowledge.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024c. Visual instruction tuning. Advances in neural information processing systems, 36.

Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. 2024d. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473.

Kyle Lo, Zejiang Shen, Benjamin Newman, Joseph Z Chang, Russell Authur, Erin Bransom, Stefan Can- dra, Yoganand Chandrasekhar, Regan Huff, Bailey Kuehl, et al. 2023. Papermage: A unified toolkit for processing, representing, and manipulating visually-rich scientific documents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 495–507.

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaocheng Sun, et al. 2024. Deepsee-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525.

Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wen-hui Wang, Li Dong, Weiyao Luo, et al. 2023. Kosmos-2.5: A multimodal literate model. arXiv preprint arXiv:2309.11419.

Jiefeng Ma, Jun Du, Pengfei Hu, Zhenrong Zhang, Jian-shu Zhang, Huihui Zhu, and Cong Liu. 2023. Hrdoc: Dataset and baseline method toward hierarchical reconstruction of document structures. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1870–1877.

microsoft. 2024. Markitdown. https://github.com/microsoft/markitdown. Accessed: 2025-01-20.

Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, et al. 2024. Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive annotations. arXiv preprint arXiv:2412.07626.

Vik Paruchuri and Samuel Lampa. 2023. Marker: Convert pdf to markdown quickly with high accuracy. https://github.com/VikParuchuri/marker. Accessed: 2025-04-14.

PyMuPDF. 2024. Pymupdf4llm. https://github.com/pymupdf/RAG. Accessed: 2024-07-26.

Zejiang Shen, Kyle Lo, Lucy Lu Wang, Bailey Kuehl, Daniel S Weld, and Doug Downey. 2022. Vila: Improving structured content extraction from scientific pdfs using visual layout groups. Transactions of the Association for Computational Linguistics, 10:376–392.

Jeremy Singer-Vine and The pdflumber contributors. 2024. pdflumber. Accessed: 2024-08-05.

Ray Smith. 2007. An overview of the tesseract ocr engine. In ICDAR ’07: Proceedings of the Ninth International Conference on Document Analysis and Recognition, pages 629–633, Washington, DC, USA. IEEE Computer Society.

Brandon Smock, Rohith Pesala, and Robin Abraham. 2022. Pubtables-1m: Towards comprehensive table extraction from unstructured documents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4634–4642.

Dominika Tkaczyk, Pawel Szostek, Mateusz Fedoryszak, Piotr Jan Dendek, and Łukasz Bolikowski. 2015. Cermine: automatic extraction of structured metadata from scientific literature. International Journal on Document Analysis and Recognition (IJ-DAR), 18:317–335.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Kinney, et al. 2020. Cord-19: The covid-19 open research dataset. In Annual Meeting of the Association for Computational Linguistics.

Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, and Furu Wei. 2021. Layoutreader: Pre-training of text and layout for reading order detection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4735–4744.

Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, En Yu, Jianjian Sun, Chunrui Han, and Xianguy Zhang. 2024a. Small language model meets with reinforced vision vocabulary. arXiv preprint arXiv:2401.12503.

Haoran Wei, Chenglong Liu, Jinyue Chen, Jia Wang, Lingyu Kong, Yanming Xu, Zheng Ge, Liang Zhao, Jianjian Sun, Yuang Peng, Chunrui Han, and Xianguy Zhang. 2024b. General ocr theory: Towards ocr-2.0 via a unified end-to-end model. Preprint, arXiv:2409.01704.

Renqiu Xia, Song Mao, Xiangchao Yan, Hongbin Zhou, Bo Zhang, Haoyang Peng, Jiahao Pi, Daocheng Fu, Wenjie Wu, Hancheng Ye, et al. 2024. Docgenome: An open large-scale scientific document benchmark for training and testing multi-modal large language models. arXiv preprint arXiv:2406.11633.

Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1192–1200.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. 2023. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2841–2858.

Weichao Zhao, Hao Feng, Qi Liu, Jingqun Tang, Shu Wei, Binghong Wu, Lei Liao, Yongjie Ye, Hao Liu, Houqiang Li, et al. 2024. Tabpedia: Towards comprehensive visual table understanding with concept synergy. arXiv preprint arXiv:2406.01326.

Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes. 2020. Image-based table recognition: data, model, and evaluation. In European conference on computer vision, pages 564–580. Springer.

Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. 2019. Publaynet: largest dataset ever for document layout analysis. In 2019 International conference on document analysis and recognition (ICDAR), pages 1015–1022. IEEE.

A Task Example

This section provides an example of input and output for the Document Structured Extraction (DSE) task as defined by READOC. As illustrated in Figure 6, DSE systems are required to process a multi-page, real-world PDF document as input and produce a structured Markdown text as output.

B Details of Dataset

B.1 Construction Details

Type Keywords of READOC-arXiv. As we aim to improve data diversity in READOC across types, disciplines, topics, and so on, we use custom keywords to capture the document category types in READOC-arXiv, as illustrated in Table 10. We treat the comments of each arXiv preprint as retrieval targets, determining whether they contain the keywords to map them to specific document types. The document type distribution of READOC-arXiv is illustrated in Figure 10.

Modification of Nougat process. As described in the main body of the paper, we modify Nougat’s source code (Blecher et al., 2023) to convert arXiv documents in HTML format into Markdown text. Specifically, our modifications include: 1) improved support for converting the \tableofcontents command; 2) recognition and conversion of nested lists; 3) support for identifying and converting sub-tables and subfigures; and 4) removal of line breaks within headings in the Markdown format.

<table>
  <tr>
    <th>Document Type</th>
    <th>Keywords</th>
  </tr>
  <tr>
    <td>Workshop</td>
    <td><i>Workshop</i></td>
  </tr>
  <tr>
    <td>Conference</td>
    <td><i>Conference</i></td>
  </tr>
  <tr>
    <td>Journal</td>
    <td><i>Journal</i></td>
  </tr>
  <tr>
    <td>Dissertation</td>
    <td><i>Thesis, Dissertation</i></td>
  </tr>
  <tr>
    <td>Guide</td>
    <td><i>Handbook, Manual, Guide, Tutorial, Technical Note</i></td>
  </tr>
  <tr>
    <td>Others</td>
    <td>-</td>
  </tr>
</table>

Table 10: Document type keywords of READOC-arXiv.

B.2 Additional Data Statistics

In addition to the diversity in disciplines, topics and languages, we provide supplementary statistical information here. This includes the distribution of the READOC dataset by year, as shown in Figure 7, and the distribution by page count, depth, and length, as illustrated in Figure 8 and Figure 9.

C Details of Evaluation S$^3$uite

C.1 Standardization Details

We standardized the Markdown text output from various tools and models according to a set of specific rules:

• Alignment of different formula boundaries. For isolated formulas, we standardize the starting boundary by converting $$, \begin{equation}, \begin{gather}, \begin{multline}, \begin{equation}, to \(, with a similar format for the ending boundary. For embedded formulas, we convert the starting boundary $ to \(, with a similar format for the ending boundary.

• Heading formatting. We standardize document headings by converting them to text lines beginning with consecutive “#” symbols, where the number of “#” indicates the heading level.

• Removal of certain elements. External URLs are removed from the link format and image references are also removed.

• Table formatting. We standardize tables by converting Markdown-compliant tables to LaTeX format. The reason for using LaTeX as
the standardized table format is that, compared to native Markdown, it can represent more complex features such as multi-row and multi-column layouts.

C.2 Segmentation Details
The segmentation module primarily based on a series of regular expressions written in Python, as illustrated in Figure 11.

C.3 Scoring Details
This section mainly discusses the metric calculations involved in the scoring module. Specifically, we focus on the following three similarity measures:

1) Edit Distance Similarity (EDS): The edit distance \( ED(A, B) \) is defined as the minimum number of single-character edits (insertions, deletions, or substitutions) required to change string \( A \) into string \( B \). The edit distance similarity is calculated using the formula:

\[
EDS(A, B) = 1 - \frac{ED(A, B)}{\max(|A|, |B|)}
\]

where \( |A| \) and \( |B| \) are the lengths of strings \( A \) and \( B \), respectively.

2) Tree Edit Distance Similarity (TEDS): The tree edit distance \( TED(T_1, T_2) \) is the minimum number of operations needed to transform one tree \( T_1 \) into another tree \( T_2 \). The operations typically include insertion, deletion, and relabeling of nodes. The tree edit distance similarity is computed as follows:

\[
TEDS(T_1, T_2) = 1 - \frac{TED(T_1, T_2)}{\max(|T_1|, |T_2|)}
\]

where \( |T_1| \) and \( |T_2| \) represent the sizes of trees \( T_1 \) and \( T_2 \), respectively.

3) Kendall’s Tau Distance Similarity (KTDS): The Kendall’s Tau distance measures the ordinal association between two rankings. The number of discordant pairs \( K_d \) is defined as the number of pairs where \( x_i \) and \( x_j \) are in different orders. The Kendall’s Tau distance similarity is given by the formula:

\[
KTDS(X, Y) = 1 - \frac{2 \cdot K_d}{n(n-1)}
\]

where \( n \) is the total number of items.
Figure 7: Visualization of year distribution in READOC. (a) Document publication years of READOC-arXiv. (b) Document publication years of READOC-GitHub.

<table>
  <tr>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>top_p</td>
    <td>0.8</td>
  </tr>
  <tr>
    <td>top_k</td>
    <td>100</td>
  </tr>
  <tr>
    <td>temperature</td>
    <td>0.7</td>
  </tr>
  <tr>
    <td>do_sample</td>
    <td>True</td>
  </tr>
  <tr>
    <td>repetition_penalty</td>
    <td>1.05</td>
  </tr>
</table>

Table 11: Generation parameters of Vision-Language Models.

D  Details of Experiments

D.1  Implementation of VLMs
This section primarily supplements the evaluation process of Vision-Language Models. Specifically, we adopt a one-page image-to-Markdown approach, concatenating the Markdown text generated for each page. The prompts we use are illustrated in Figure 12, and we uniformly apply the generation parameters for all VLMs as shown in Table 11.

D.2  Additional Case Study
We provide two additional case analyses, as shown in Figures 13 and 14. It is evident that none of the DSE systems demonstrated satisfactory performance, as they exhibited various types of errors, including Missing Content, Misidentifying Headings, Inaccurate Text Extraction, Confused Reading Order, Hallucinations, and Repetitions.
Figure 8: Visualization of year distribution in READOC-arXiv. (a) Document page counts of READOC-arXiv. (b) Document depth (heading levels) of READOC-arXiv. (b) Document Length (token counts) of READOC-arXiv.

Figure 9: Visualization of year distribution in READOC-GitHub. (a) Document page counts of READOC-GitHub. (b) Document depth (heading levels) of READOC-GitHub. (b) Document Length (token counts) of READOC-GitHub.

<table>
  <tr>
    <th>Headings</th>
    <td>re.compile(r'^([#]{1,6}) +(.)', re.MULTILINE)</td>
  </tr>
  <tr>
    <th>Embedded Formulas</th>
    <td>re.compile(r"\\((.*?)(?<!\\))\\")</td>
  </tr>
  <tr>
    <th>Isolated Formulas</th>
    <td>re.compile(r"\\((.+?)(?<!\\))", re.S)</td>
  </tr>
  <tr>
    <th>Tables</th>
    <td>re.compile(r"\n\begin{table}\{(?:\n\begin{table}\}|*?\}\end{table}\}", re.S)</td>
  </tr>
  <tr>
    <th>Plain Text</th>
    <td>(The remaining part is treated as plain text)</td>
  </tr>
</table>

Figure 10: Document types of READOC-arXiv.

Figure 11: Segmentation module’s regular expressions.
Prompt for READoc-arXiv
This image displays a document page, convert the page content into Markdown format. Use continuous # to denote headings at each level. Display tables following markdown or latex format. Use $ or \( \) to surround inline math, and use $$ or \( \) to surround isolated math block. Don't explain, directly output the Markdown-format content.

Prompt for READoc-GitHub
This image displays a document page, convert the page content into Markdown format. Use continuous # to denote headings at each level. If it's a blank page, don't output anything. Otherwise, directly output the Markdown-format content without explanation.

Figure 12: Prompts of Vision-Language Models.
READDoc Ground Truth
starting configuration Beckett’s text explicitly notes that every possible non-empty subset of characters appear on stage together at least once but admits that these subsets do not appear a uniform number of times[1].

If each subset of appears precisely once, then the series of subsets – represented by the aircraft binary word “called for” by each code in Fisher & Gray lists of all sets of bit words are equivalent to Hamilton paths and cycles in the \(\mathcal{U}_3\)-memory graph[3]. We feel this note provides the necessary background to this work in a more detailed account of history and reharsh search see [8].

# 2 Preliminaries

The existence of a Beckett-Gray code is equivalent to being able to realize every subset of an \(n\)-set exactly once as successive states of a queue. The queue entries are the bit positions of a cyclic code. In Table 1 we give a few examples of such a Beckett-Gray code and the corresponding queue states.

<table>
  <tr>
    <th>subset</th>
    <th>queue</th>
    <th>bit pos</th>
  </tr>
  <tr>
    <td>{}</td>
    <td>()))))-))()</td>
    <td>1 1 1 3 3 4 5 7 7 7 9 9</td>
  </tr>
  <tr>
    <td>{a}</td>
    <td>()))-))-)))</td>
    <td>1 1 3 5 7 7 9 9</td>
  </tr>
  <tr>
    <td>{b}</td>
    <td>(-))-))-)))</td>
    <td>1 3 5 7 7 9 9</td>
  </tr>
  <tr>
    <td>{c}</td>
    <td>(--))-))-)))</td>
    <td>1 3 5 7 9 9</td>
  </tr>
  <tr>
    <td>{ab}</td>
    <td>(()-))-))-))()</td>
    <td>1 1 3 5 7 7 9 9</td>
  </tr>
  <tr>
    <td>{bc}</td>
    <td>(()-))-))-))-</td>
    <td>1 3 5 7 7 9 9</td>
  </tr>
  <tr>
    <td>{ac}</td>
    <td>(--))-))-))-)</td>
    <td>1 3 5 7 7 9 9</td>
  </tr>
  <tr>
    <td>{abc}</td>
    <td>(--))-))-))-))-</td>
    <td>1 3 5 7 9 9</td>
  </tr>
</table>

Table 1: A 3-bit non-cyclic binary Beckett-Gray code.

Two Gray codes are called _isomorphic_, if one can be obtained from either by permuting bit positions, adding a fixed \(\forall\SI{}{\bit}\) bit word to all words and/or reversing the list of words. Permuting bit positions preserves the Beckett-Gray code.

Figure 13: A case study from READOC arXiv.

READDoc Ground Truth
- https://github.com/smqcur/jasonruns
- https://github.com/ContextLogic/eventmaster
- https://isg.github.com... (Python/pandas backend)

## Query API

Example timeseries request
```javascript
{
  "panelId": 1,
  "range": {
    "from": "2016-10-31T06:33:44.866Z",
    "to": "2016-10-31T12:33:44.866Z",
    "raw": false
  },
  "target": [
    { "target": "upper_50", "refId": "A", "type": "timeserie" },
    { "target": "upper_75", "refId": "B", "type": "timeserie" }
  ]
}
```

Example timeseries response
```javascript
...
{
  "error": null,
  "dataSources": [
    {
      "target": "upper_75", // The field being queried for 'datatransfers'
      "dataPoints": [
        [1457241509741000000L, 446779702.9], // Metric value .. milliseconds
        [1457241729741000000L, 446779702.9]
      ]
    }
  ]
 ...

Reviewer paper summaries: This paper proposes a benchmark for pdf-to-markdown evaluation, together with a comprehensive evaluation method considering multiple aspects. 
Results on general VLMs, expert visual models and pipeline tools are reported, and several valuable conclusions are made.
The weaknesses are: (1) there are no new methods proposed; (2) there are certain room to improve this evaluation pipeline, which negatively affect my evaluation of this paper.

This paper proposed a novel benchmark named READoc and evaluation method S3uite for end-to-end document structured extraction, and testified the performances of various models on this benchmark.
Summary of strengths: 1. The data resources include arxiv papers and github readme documents, whose diversity is clearly illustrated by statistics, and the necessity of diversity is clearly illustrated in the paper. 
2. The proposed evaluation suite is reasonable. 
3. This paper is well-written and contains rich experiments.

1.The motivation to address real-world document structure extraction is practical.
2.The proposed S3uite is a useful metric for evaluating document structure extraction.
3.The paper is well-written and easy to understand.
Summary of weaknesses: 1. I am not fully satisfied with the pdf-to-markdown paradigm. 

The native grammar of Markdown lacks expressiveness of the complex structures studied in this paper, e.g. tables, and I believe that is why the authors choose to standardize the expression scheme based on LaTeX. So why not just use the pdf-to-LaTeX paradigm? Based on that paradigm, the dataset could be directly built by arxiv data resources, and can be expanded to the extraction of other complex structures, e.g. tikz charts. 

The authors have mentioned that this benchmark is the first one with the pdf-to-markdown paradigm. However, as for me, I think this benchmark should still be compared with those of the pdf-to-LaTeX paradigm. 

2. The data amount of the proposed dataset (READoc) is too small, which should have been better. 

The total amount of 2,000 is not enough. The available arxiv papers with tex code released is far more than that. I wonder why the authors only choose a very small subset of these resources. (This also applies to github readme documents. )

Another issue is that the current amount of READoc is not able to be used in fine-tuning, considering that fine-tuning the DSE model is rather data-consuming. 
It should be better to propose a larger dataset, fine-tuning current methods and report their results. 

3. The evaluation metrics could be improved. 

This paper focuses on modeling the complex structures within documents. However, there are only text-centric metrics in the evaluation (edit distance, bleu, meteor). 
Complex structures are expressed by (LaTeX) code, so it should be considered that (1) whether the generated (LaTeX) code could be compiled, and (2) whether the render effect of the generated code is the same as the original pdf document. 
It is suggested to introduce metrics to model the compiling effects, like [1]. 


References:
[1] Cdm: A reliable metric for fair and accurate formula recognition evaluation.

1.The constructed dataset still differs from real-world scenarios, as datasets like arXiv and GitHub are overly simplistic. It would be better to reference datasets like M6Doc and DocLayNet to create data that is more representative of real-life document structures.
2.The dataset was constructed by scraping data from arXiv, while Nougat used a large amount of arXiv data for training. This raises concerns about whether there is an overlap between the benchmark data and the training data, potentially leading to data leakage.
3.There is a lack of testing on more specialized models. The performance of dedicated parsing models, such as KOSMOS-2.5 or Vary, has not been evaluated. Including such models in the experiments would provide a more comprehensive comparison and strengthen the paper’s claims.
4.The benchmark data construction is very similar to that of NOUGAT, making the benchmark less novel and potentially unfair to other models. 
5.The reading order metric is high, but it is unclear whether the construction process took into account whether it aligns with human reading order in real-world scenarios. Ensuring the dataset reflects realistic reading patterns would enhance its applicability to practical situations.
Comments, suggestions and typos: See `Summary Of Weaknesses`.

The author needs to construct a benchmark that is more representative of real-world scenarios and conduct experiments on a broader range of models. This would better highlight the various challenges faced in realistic document structure extraction.
Overall assessment scores: 2.5

2.5
Metareview score: 3
Metareview: This paper introduces READoc, a benchmark dataset for evaluating the conversion of multi-page PDF documents into structured markdown. The dataset is composed of 2,233 diverse documents, including arXiv preprints and GitHub READMEs. The paper also proposes a multi-aspect evaluation suite that standardises the markdown format, segments the content into four units (headings, formulas, tables and text) and computes scores using edit distances. Experiments are conducted on READoc to compare the effectiveness of multiple methods ranging from simple OCR tools to VLMs.

Reviewer agree about the strengths of this work: the paper is well written, it proposes a standardised benchmark dataset that is valuable for evaluating document structure extraction quality, and the proposed evaluation framework is sound. The main concerns raised by the reviewers relate to the choice of the PDF-to-markdown paradigm (as opposed to PDF-to-LaTeX or DocumentQA as a proxy for assessing LLMs’ ability to process documents) and the limited diversity and simplicity of the benchmark, which consists only of arXiv papers and GitHub READMEs.

Overall, I believe this paper has merits and provides a valuable benchmark for document structured extraction. However, I feel that the concerns raised by the reviewers are significant, and that addressing them would further strengthen the work.
Summary of reasons to publish: - The paper is clear and well written
- It introduces a standardised benchmark for multi-page PDF-to-markdown evaluation,
- It proposes a reasonable multi-aspect evaluation suite and report extensive experiments