DocGenome: A Large Benchmark for Multi-Modal Language Models in Real-World Academic Document Understanding

Authors: Renqiu Xia, Song Mao, Xiangchao Yan, Hongbin Zhou, Bo Zhang, Haoyang Peng, Jiahao Pi, Daocheng Fu, Wenjie Wu, Hancheng Ye, Shiyang Feng, Mingsheng Li, Bin Wang, Chao Xu, Conghui He, Pinlong Cai, Min Dou, Botian Shi, Sheng Zhou, Yongwei Wang, Bin Wang, Junchi Yan, Fei Wu, Yu Qiao
Primary area: datasets and benchmarks
Keywords: scientific document structuring, document understanding, chart table and equation understanding

Abstract:
Scientific documents record research findings and valuable human knowledge, comprising a vast corpus of high-quality data. Thus, leveraging multi-modality data extracted from these documents and assessing large models' abilities to handle scientific document-oriented tasks is meaningful. Despite promising advancements, large models still perform poorly on multi-page scientific document extraction and understanding tasks, and their capacity to process within-document data formats such as charts and equations remains under-explored. To address these issues, we present DocGenome, a structured document dataset constructed by annotating 500K scientific documents from 153 disciplines in the arXiv open-access community, using our custom auto-labeling pipeline. DocGenome features four characteristics: 1) Completeness: It is the first dataset to structure data from all modalities including 13 layout attributes along with their LaTeX source codes. 2) Logicality: It provides 6 logical relationships between different entities within each scientific document. 3) Diversity: It covers various document-oriented tasks, including document classification, visual grounding, document layout detection, document transformation, open-ended single-page QA and multi-page QA. 4) Correctness: It undergoes rigorous quality control checks conducted by a specialized team. We conduct extensive experiments to demonstrate the advantages of DocGenome and objectively evaluate the performance of current large models on our benchmark.

Decision: reject
Reviewer scores: 6, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730610671372

thread_type:
official_review

<thread_object>
creator: reviewer_i9vg

<content>
summary:
This work presents a large-scale multimodal academic document understanding dataset. It includes training and high-quality test sets, along with 7 benchmarking tasks proposed. The manuscript provides a clear description of the data collection and quality control processes. A series of large multi-modal models are benchmarked on the proposed test set, and training experiments are conducted to verify the effectiveness of the proposed training set.

strengths:
1. The proposed large-scale multimodal academic document understanding dataset makes a solid contribution to the related research community, especially given the scarcity of such datasets.

2. The dataset collection and quality control are carefully designed, executed, and documented. The provided anonymous GitHub repository includes detailed documentation and links for downloading the dataset, supporting reproducibility and accessibility.

3. A training set is provided along with the test set, and training experiments are conducted to verify the effectiveness of the training set.

weaknesses:
1. The QA data creation process depends heavily on GPT-4. As the QA pairs are generated by GPT-4, it can introduce biases regarding the type and difficulty of the questions. Further examinations into the potential biases would therefore be beneficial.

2. The evaluation metrics used for different tasks could be improved. Specifically, edit distance or BLEU may not accurately evaluate Equation-to-LaTeX and Table-to-LaTeX tasks, as these metrics do not account for the semantic equivalence of different LaTeX expressions. Additional evaluation could also be performed to verify the grammatical correctness of generated LaTeX expressions. Moreover, Open-ended QA tasks are evaluated using GPT-4 to compare reference and generated answers. While this is likely a reasonable approach, human evaluation to verify the reliability of GPT-4’s judgment would be beneficial.

questions:
1. The manuscript discusses that the GPT-4-generated QA pairs are verified and updated by human annotators. What is the acceptance rate of the original QA pairs, or alternatively, what is the editing rate? Given that GPT-4 achieves around 60%-70% accuracy on QA tasks in Table 3, this suggests that a substantial portion of the QA pairs were likely updated.

2. Why are large multimodal models (e.g., GPT-4V, QWen-VL) not benchmarked on the Equation-to-LaTeX and Table-to-LaTeX tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660057502

thread_type:
official_review

<thread_object>
creator: reviewer_4uws

<content>
summary:
The main contribution of this paper is a dataset called DocGenome that
is curated mostly automatically.

1.  The main component of paper is a dataset 500k samples of structured
    representation of academic papers. The author developed a pipeline
    to automatically parse the latex source.

2.  The authors also curate a test set of 7 tasks (vision-only,
    language-only, and multi-modal) to evaluate the performance of
    models on the dataset.

3.  The author also verified that the training data could be helpful and
    lead to better performance on downstream tasks via training on
    DocGenome data.

strengths:
1.  The paper is nicely written, and there authors have conducted a lot
    of experiments evaluating different aspects of this dataset.

2.  The dataset, as well as the document conversion pipeline, could be a
    helpful resource for the community.

weaknesses:
1.  Overall this paper feels like a resubmission from venues like the
    NeurIPS dataset and benchmark track. The methodology contribution is
    relatively weak: the method of creating the dataset (i.e., parsing
    latex sources) is not novel (e.g., a lot of earlier work like GROTOAP, PubLayNet, DocBank, all use similar approaches ),
    neither is the framing of the tasks (all these tasks have been
    studied before to some extent). I am inclined to a borderline reject
    unless the authors can give a strong statement in terms of the
    novelty of the dataset or the methodology.

2.  While I find there could be one potentially novel aspect of the
    dataset---the "13 categories of component units and 6 types of
    logical relationships between them", it is not clear how the authors
    actively use this component in the experiments. (The empirical
    studies mostly focus on using individual components but not the
    relationships.) I'd encourage the authors to provide more
    insights/experiments on this.

3.  There's another limitation of the datasets: since it is curated via
    automatically parsing latex sources, the variance and the structure
    could be limited, and the trained models might not be able to
    transfer to other types of papers (which is also mentioned by the
    authors in line 458.)

questions:
1.  The author mentioned that "Each QA pair is reviewed by three
    reviewers for cross-verification." (line 310) It would be great if
    the authors can provide more details in terms of the inter-annotator
    agreement or additional details. "Finally, the two
    manually-evaluated results, along with the automatically-evaluated
    result are cross-verified with the original text to ensure accuracy
    and consistency" (line 317) I am not sure how this process works,
    can you provide more details?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691882340

thread_type:
official_review

<thread_object>
creator: reviewer_tmkl

<content>
summary:
Long  document understanding is an interesting and complex problem. The paper provides a document processing pipeline and a large and diverse collection of documents for academic document understanding. The paper also evaluates the performance of large models on multiple tasks using the benchmark dataset.

strengths:
As shown in Table 1 of the paper, the document collection is superior to existing collections on multiple aspects.

weaknesses:
1. While datasets and benchmarking are important to drive research, the paper does not contain research contributions.
2. The documents are based on latex formatted documents which is a subclass of all documents.

questions:
1. Is the collection representative of scientific documents? In other words, does latex formatting constrain documents in some way? For example, latex denotes non-existing references by ??. If a document is created using other formatting software, such mistakes are not flagged.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730870764636

thread_type:
official_review

<thread_object>
creator: reviewer_c3jl

<content>
summary:
The paper proposed a new dataset for academic document understanding. By processing 500k documents from arXiv (using their source LaTeX files) the authors create a new large dataset which covers diverse disciplines, preserves data in all modalities from papers, and covers 7 document-oriented tasks. 
The paper describes its automatic processing and labeling pipeline, along with the two metrics used for quality assurance. The authors then describe how the dataset is split into train and test subsets; where the data is divided into tiers based on the previously mentioned metrics and 1004 papers are sampled from the top tier. 
The sampled papers are them used to create QA pairs about their content (both single-page and multi-page questions) using GPT-4V, which are them validated by professional faculty members.
The paper also present the benchmarking of different LLMs on its test set, along with the usage of its training data to effectively create new models that outperform selected baselines.

strengths:
The paper presents a new large dataset of multi-modality academic documents for document-understanding tasks. This is a welcome and useful target as most existing datasets are limited in scope, diversity, and especially do not preserve all modalities of data in the source papers, losing important information.

The process to create the dataset is well presented and pragmatic and both the toolset and processed data can be valuable in themselves or for further collections and annotations for new tasks. 

While somewhat small when compared to the overall collected data, the annotated test set with questions is shown to be already a useful benchmark for recent large multi-modality models.

weaknesses:
Unfortunately, the dataset also presents somewhat limited novelty and impact. 

In its current described form, one of its main contributions is the created test set with annotated QAs. However, these seem to have limited coverage of non text modality QA and the annotated relationships are limited to layout level, not focusing on critical data/information relationships. As a dataset targeting multi-modality models, it is critical that these are emphasized.

This is especially the for charts/plots and tables, which are rich in information and relationships. I missed seeing some analysis on the modalities and specific discussions on issues and how the dataset addresses them.

questions:
With so many different disciplines covered in the data, how exactly were the faculty members selected to review the annotated dataset?

Will the data and codebase be released under a permissive license? With the lack of explicit table-QA or image-QA/chart-QA annotations, for example, it would be critical that the data can be used for re-annotation and extensions.

In line 161 you say \ref commands are removed, but these seem essential in relationship extraction. Also, Table 2 implies they are indeed used.

If GPT-4V was used to create all questions, is there a reason its performance in Table 3 is not so high and even worse than GPT-4o?

How many QA pairs were actually created and used? My understanding is that 4 QA pairs are created per sample, but only 3k total QA pairs were kept? Did I miss some details on how this was filtered?

Please provide more details on the sampled data for the OOD experiments. This data should also be released for reproducibility.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733186073416

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
ICLR Reminder of Reviewer 4Uws

comment:
Dear Reviewer 4Uws,

We have thoroughly addressed your concerns and look forward to your feedback, which will greatly assist us in making any necessary revisions or clarifications.

Thank you for your time and consideration.

Best regards,
Authors of paper 6842
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces DocGenome, a dataset curated through automated methods consisting of a large number of samples representing the structured content of academic papers. To create this dataset, it describes a pipeline developed to automatically parse LaTeX source files. To evaluate the dataset, the paper includes a test suite of seven tasks. These tasks cover vision-only, language-only, and multi-modal challenges, providing a framework to assess model performance on DocGenome.The paper also demonstrates that training on DocGenome data can improve model performance on downstream tasks, showing its utility for research in document understanding and related fields.  

Reviewers generally found the paper well-written and the released dataset valuable for the research community. They appreciated the quality control measures implemented during dataset creation and acknowledged the usefulness of the training set.  

However, a significant critique focused on the paper's methodological and technical contributions, as well as the dataset's novelty, which reviewers considered relatively limited (4Uws, c3jL, TmKL). Some reviewers also expressed concerns regarding limited coverage of non text modalities (c3jL).  Concerns were also raised about the automated construction process of the dataset, particularly its reliance on GPT-4 (i9VG, 4Uws). Additional issues included a lack of experiments addressing novel aspects of the dataset, potential biases in its creation, the emphasis on LaTeX formatting, and requests for further details—all of which were largely resolved during discussions.  

Overall, while the resource offers value to the community, its technical contributions and novelty appear limited. The authors' response summarized the weaknesses raised by reviewers as a separate comment, but notably omitted addressing these major critiques.