Graph-based Document Structure Analysis
Authors: Yufan Chen
Ruiping Liu
Junwei Zheng
Di Wen
Kunyu Peng
Jiaming Zhang
Rainer Stiefelhagen
Venue: ICLR 2025 Poster (2025)
Keywords: Document Structure Analysis
Document Layout Analysis
Document Relational Graph
Primary area: datasets and benchmarks

Abstract: When reading a document, glancing at the spatial layout of a document is an initial step to understand it roughly. Traditional document layout analysis (DLA) methods, however, offer only a superficial parsing of documents, focusing on basic instance detection and often failing to capture the nuanced spatial and logical relationships between instances. These limitations hinder DLA-based models from achieving a gradually deeper comprehension akin to human reading. In this work, we propose a novel graph-based Document Structure Analysis (gDSA) task. This task requires that model not only detects document elements but also generates spatial and logical relations in form of a graph structure, allowing to understand documents in a holistic and intuitive manner. For this new task, we construct a relation graph-based document structure analysis dataset(GraphDoc) with 80K document images and 4.13M relation annotations, enabling training models to complete multiple tasks like reading order, hierarchical structures analysis, and complex inter-element relationship inference. Furthermore, a document relation graph generator (DRGG) is proposed to address the gDSA task, which achieves performance with 57.6% at $mAP_g$@$0.5$ for a strong benchmark baseline on this novel task and dataset. We hope this graphical representation of document structure can mark an innovative advancement in document structure analysis and understanding. The new dataset and code will be made publicly available.

Published as a conference paper at ICLR 2025
GRAPH-BASER DOCUMENT STRUCTURE ANALYSIS
Yufan Chen, Ruiping Liu, Junwei Zheng,
Di Wen,
Kunyu Peng,
Jiaming Zhang!
Rainer Stiefelhagen
CV:HCI Lab, Karlsruhe Institute of Technology
(firstname Lastname a kit.
edu
100%
0800
ND -D0G0
#50205
2575
005
Stute Heads
DLA
072
Stale Heade
65
(a) Document graph structure of layouts with different relations.
Performance of proposed method.
Figure I: GraphDoc Dataset Overview. Figure Ia illustrates the threefold considerations, including (i) the inclusion of spatial and logical relations, (ii) support for multiple relations between layouts pairs, (iii) and the integration of non-textual elements. Figure demonstrates state-of-the-ar performance of our proposed method, showcasing mAP results for the Document Layout Analysis (DLA) task, as well as mR, and m.AP, results for the graph-based Document Structure Analysis (gDSA) task on the GraphDoc dataset.
ABSTRACT
When reading a document, glancing at the spatial layout of document an ini- tial step to understand roughly. Traditional document layout analysis (DLA) methods, however, offer only superficial parsing of documents, focusing on ba- sic instance detection and often failing to capture the nuanced spatial and logical relations between instances. These limitations hinder DLA-based models from achieving a gradually deeper comprehension akin to human reading. In this work, we propose novel graph-based Document Structure Analysis (gDSA) task. This task requires that model not only detects document elements but also generates spatial and logical relations in form of a graph structure, allowing to understand documents in holistic and intuitive manner. For this new task, construct relation graph-based document structure analysis dataset (GraphDoc) with 80K document images and 4.13M relation annotations, enabling training models complete multiple tasks like reading order, hierarchical structures analysis, and complex inter-element relation inference. Furthermore, document relation graph generator (DRGG) is proposed to address the gDSA task, which achieves perfor- mance with 57.6% % at mAP, @0.5 for strong benchmark baseline this novel task and dataset. We hope this graphical representation of document structure can mark an innovative advancement in document structure analysis and understand- ing. The new dataset and code will be made publicly available at GraphDoc.
INTRODUCTION
Understanding the structural layout of documents is fundamental aspect of document analysis and comprchension. Traditional Document Layout Analysis (DLA) methods primarily focus on detecting and classifying basic document elements, e.g., text blocks, images, and tables. While these
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Table 1: Modern Document Structure Analysis Datasets. V. V. L. O, H and G stand for Vi- sual, Textual, Layout, Order, Hierarchy and Graph modality. DLA, ROP, HSA and GSA stand for Document Layout Analysis, Reading Order Prediction, Hierarcheial Structure Analysis and Graph Structure Analysis. NTI stands for Non-Textual Instance.
Dataset
Year
Instunce Level
Modulity
L
II
Amaag
Object Calegories
Object Instances
Relation Caleguries
Relations
NTI
DLA ROP
Tasks
USA GSA
Format
FUNSD
2019
Semaniik: Emilyy
199
4
7411
Seansed
ReadinglSank
Word
500K
98.18M
DocX
XFUND
2022
Semanna: Emity
1393
4
0.3GM
Seansed
Form NLU
2023
Semantic Enity
357
7
0.03M
PDD
HRDee
X025
Line
56K
14
1.79M
PDF
Comp HiRDoc
2024
Line
42K
14
0.91M
PDD
Net
2019
hrngraph
S40K
3.31M
PDF
DocLayNet
2022
Furgraph
11
1.00
PDF
GraphDoc
2024
Paragraph
SOK
11
1.10M
4.13M
PDF
methods have achieved significant success in document analysis at superficial level, they often overlook capturing the intricate spatial and logical relations that exist between different document components. This limitation hampers the ability of models to achieve a deeper, more human-like understanding of document structures. Recent advancements in deep learning and computer vision have led to the development of models that can process multimodal information, integrating visual, textual, and layout information (Gu et al., 2021; Huang et al., 2022). However, these models still lack the capability to comprehend the complex relations inherent in document layouts, particularly the spatial and logical relations that define the document structure. As shown in Table 1, existing datasets such as PubLayNet (Zhong al., 2019) and DocLayNet (Pfitzmann et al., 2022) provide annotations for basic layout elements but do not include detailed relational information inside.
To address these challenges, we propose a novel task called graph-based Document Structure Anal- ysis (gDSA), which aims to not only detect document elements but also generate spatial and logical relations the form of graph structure. This approach allows for a more holistic and intuitive un- derstanding of documents, akin to how humans perceive and interpret complex layouts. For this task, we introduce the GraphDoc dataset, a large-scale relation graph-bases bbseed document structure analysis dataset comprising 80,000 document images and over 4 million relation annotations. GraphDoe in- cludes annotations for both spatial relations (Up, Down, Left, Right) and logical relations (Parent, Child, Sequence, Reference) between document components, e.g., text, tuble, and picture. This rich relational information enables models to perform multiple tasks like reading order prediction, hier- archical structure analysis, and complex inter-clement relationship inference. To tackle the gDSA task, we propose the Document Relation Graph Generator (DRGG), an end-to-end architecture designed to generate relational graphs from document layouts. DRGG combines object detection with relation prediction, capturing both spatial and logical relations between document elements. Our experiments demonstrate that DRGG achieves mean Average Precision of 57.6 57.6% a relation confidence threshold of 0.5 (mAP, y @0.5), setting a strong baseline for this novel task and dataset.
In summary, our contributions are as follows:
We introduce GraphDoc, a graph-based document structure analysis dataset that provides detailed annotations of both spatial and logical relations between document components.
We provide a comprehensive analysis of graph- based Document Structure Analysis (gDSA) paradigms and demonstrate that our DRGG model effectively addresses the gDSA task.
We conduct extensive experiments on the GraphDoc dataset and upstream DLA tasks, prov- ing the effectiveness of the gDSA approach for document layout analysis.
2 RELATED WORK
Document Layout Analysis. To analyze the document layout is a fundamental task of the document understanding. Recent advancements in deep learning (Schreiber et al., 2017; Prasad et al., 2020) treat Document Layout Analysis (DLA) traditional visual object detection or segmentation chal- lenge, employing convolutional neural networks (CNNs) to address this task. Drawing inspiration from BETT (Bao et al., 2022), compared to the CNN-based methods, DiT (Li et al., 2022) trains document image transformer specifically for DLA, achieving promising results, albeit overlooking the textual information within documents. Beyond the single modality, Dni (Gu et al., 2021) and
2

---

Published as a conference paper at ICLR 2025
LayoutLMy3 (Huang et al., 2022) integrate text, vision, and layout modalities within unified archi- tecture. Not only methods and architectures, but also benchmark datasets have achieved promising evolution. While PubLayNet (Zhong al., 2019) and DocLayNet (Pfitzmann et al., 2022) have only two modalities, ie., visual and layout, FUNSD (Jaume et al., 2019), XFUNSD (Xu et al., 2022), ReadingBank (Wang et al., 2021) and Form-NLU (Ding et al., 2023a) have textual, visual, layout and order modalities. It is regrettable that the aforementioned datasets, despite considering other modalities, are designed solely for textual information without non-textual information considera- tion. HRDoc (Ma et al., 2023) and its improved version, the Comp-HRDoc dataset (Wang et al., 2024), both take into account multimodal processing of both textual and non-textual information. Additionally, they introduce a hierarchical structure as new modality for document analysis. How- ever, all publicly available datasets do not consider the graphical structure of document, which is crucial for both spatial and logical structure analysis of documents. In this work, we propose Graph- Doc dataset, which contains six modalities, ie.. textual visual, layout, order, hierarchy and graph, targeting complex Document Structure Analysis (DSA) tasks.
Graphical Representation and Generation. To construct a graph-based structured representation is foundational step toward higher-level visual understanding. Graph-based representation Scene Graph Generation (SGG) is versatile tool for various vision-language tasks, such as image caption- ing (Gao et al., 2018; Yang et al., 2019b), visual question answering (Li et al., 2019; Zhang al., 2019), content-based image retrieval (Johnson et al., 2015; Schuster et al., 2015), image genera- tion (Johnson et al., 2018; Mittal al., 2019), and referring expression comprehension (Yang al., 2019a). On the other hand, in the field of natural language processing knowledge graph generation also well-explored. Instead of building the entire global graph structures, some methods (Li al., 2016; Yao et al., 2019; Malaviya et al., 2020) look into simpler problem of graph completion. AI- ternatively, other works (Roberts al., 2020; Jiang et al., 2020; Shin al., 2020; Li & Liang, 2021) propose to query the pre-trained models to extract the learned factual and commonsense knowledge. CycleGT (Guo et al., 2020) is an unsupervised approach for both text-to-grapl and graph-to-text generation. In this method, the graph generation process utilizes pre-existing entity extractor, followed by a classifier for relations. Inspired by the graph generation from computer vision and natural language processing, we propose a graph-based task for document analysis called graph- based Document Structure Analysis A. gDSA refers to the task of mapping document images into a comprehensive structural graph that contains the understanding of document structure.
Document Relation Extraction. Document relation extraction is a crucial task in understanding the complex interactions within documents by identifying relations between document elements. ReadingBank (Wang et al., 2021) is designed for the task of reading order detection, which aims to capture the sequence of words as naturally understood by human readers. FUNSD (Jaume al., 2019), Form-NLU (Ding et al., 2023a) and XFUND (Xu et al., 2022) focuses on extracting relations in semi-structured documents, particularly text-only forms. Addresses the challenges scanned documents by identifying key-value pairs and relations between textual elements. PDF-VQA (Ding et al., 2023b) extends document relation extraction to multimodal documents by incorporating vi- sual question answering techniques. This dataset requires the identification of relations between document elements within PDFs. HRDoc (Ma et al., 2023) constructs a dataset for document re- construction but overlooks the spatial structure and the interaction between textual and non-textual elements. Our proposed GraphDoc dataset includes both spatial and logical relations between tex- tual and non-textual elements, resulting in comprehensive analysis of document structure.
3 METHODS
3.1 GRAPHDOC DATASET
In this section, we introduce the GraphDoc dataset, specifically developed for document layout and structure analysis. Additionally, we define the corresponding tasks and describe the annotation pipeline employed for constructing such datasets.
3.1.1 TASK DEFINITION
The goals of the GraphDoc Dataset can be represented into two tasks: Document Layout Analysis (DLA) and graph-based Document Structure Analysis (gDSA). We detail definitions respectively.
3

---

Published as a conference paper at ICLR 2025
Document Layout Analysis (DLA). This task focuses extracting layout information with labeled bounding box, representing layout elements within the document. For the DLA task, the setup is similar to that of the DocLayNet (Pfitzmann et al., 2022) dataset, with the layout element size being at the paragraph level except Table and Picture. The labels are categorized into 11 distinct classes: Caption, Footnote, Formula, List-item, Page-footer, Page-header, Picture, Section-header, Table, Text, and Title. DLA task can be represented by the following objective function:
LDAA Cabox (b;, b,) + Lat(Ci, 2.)
(1)
where b, and b, are the ground truth and predicted bounding boxes for the i-th layout element, re- spectively, and c; and é are the corresponding class labels. The loss function LDLA thus encapsulates both the bounding box regression loss L and the classification loss Less
Graph-based Document Structure Analysis (gDSA), gDSA aims to extract the relational graph among layout elements within the document, which could be formed G (V, E). For gDSA, nodes V correspond to the layout elements, edges E represent the relations between these layout elements, e.g., reference. The objective for gDSA could be expressed as:
Document
Image Text Anelysis
Document
mmae Analysi
Document Text Analysi
CIDSA= = Σ (Las(Us, 0.) + Lra( fiy))
(2)
(eIDS)EE
Here, vi and 6g are the ground truth and predicted labels for the layout element i, and rig and Pig represent the ground truth and predicted relations between the layout elements and j. The classification loss Las for the nodes ensures that the lay- out elements are accurately identified, while the relation loss fue for the edges captures the accuracy of the predicted relations within the document's s structure. Additionally, the specific functions of all the above-mentioned losses depend on the requirements of the model and the task.
Figure 2: Overview Dataset's Task, which illustrates both DLA and gDSA tasks of GraphDoc are based on image analysis.
GraphDoc
Two sub-tasks can derive further from the gDSA task: Reading Order Prediction (ROP) and Hierar- chical Structure Analysis (HSA). The ROP task involves determining the correct sequence in which the layout elements should be arranged. The HSA task focuses on identifying the hicrarchical rela- tions among the layout elements and establishing structural organization within the document. In addition to the tasks deseribed above, the gDSA task further leverages reference relations to establish connections between textual and non-textual layout elements within the document. This integration ensures that these two types of layout element are not analyzed in isolation, but rather as inter- connected components. As shown in Figure 2, the gDSA tasks in the GraphDoc dataset achieves a novel and comprehensive visual analysis of document task, paving the way for novel document visual content analysis of modern complex documents.
3.1.2 DATASET COLLECTION
Our GraphDoc Dataset is primarily derived from the DoeLayNet (Pfitzmann et al., 2022) dataset, which contains over 80,000 document page images spanning a diverse array of content types, in- cluding financial reports, user manuals, scientific papers, and legal regulations. We leveraged the existing detailed annotations and the PDF files offered through DocLayNet Dataset, to create new annotations that focus specifically uments. Additionally, in accordance with the License CDLA 1.0, users are permitted to modify and redistribute enhanced versions of datasets based on the DocLayNet dataset. Due page limitations of DocLayNet, we will only consider relations within the same page and not those across pages.
the relations between various layout elements within the doc-
3.1.3 DOCUMENT RELATIONAL GRAPHS
For visually rich documents, the spatial layout and relations between various layout elements carry significant meaning. These relations include hierarchical relations between section headers and text, sequential relations between text blocks, and references to tables or figures. Understanding these structural and relational details aids in better extraction of document information and in gaining deeper comprehension of the document as a whole. Moreover, graphs themselves are an effective modality for enhancing the performance of scene understanding tasks.
4

---

Published as a conference paper at ICLR 2025
Logical Relationshp
(4) Sequence
(c)
Figure 3: Logical Relationship in GraphDoc Dataset. There are instinct types of relations. The relational graph effectively filters out extrancous connections that might appear in other types of diagrams, providing a clearer representation of the actual relationships.
Consequently, in our GraphDoc dataset, we have defined two types of relational graphs. The first type is the spatial relational graph, which primarily categorizes spatial relations into four types: up, down, left, and right. In scientific literature, the spatial structure is typically more standardized, often formatted as either two-column single-column documents in Manhattan-Layout, which refers to a grid-like layout where content is arranged in straight, non-overlapping rectangular regions. Thus, these four spatial relations can effectively cover most of the spatial relations between layout elements within scientific documents.
The second type is the logical relation graph, which is independent layout position and focuses on capturing the relations between layout elements from logical structure perspective. In this logical relation graph, we categorize all relations between document layout elements into four types of relations: parent, child, sequence, and reference. All logical relations are illustrated in Figure for better understanding. The detailed definitions of relation are as follows:
Parent: Indicates the parent part of parent-child. For example, section header can be the parent of the subsection header, as in Fig. 3(b).
Child: Represents the child part of a parent-child. For instance, paragraphs that belong to section are considered children of that section header, as in Fig. 3(c).
Sequence: Denotes the sequential order of layout elements. For example, the natural read- ing order of paragraphs in a section the steps in procedure, in Fig. 3(d).
Reference: Captures citation or references. For example, figure or table being cited within the text or references to external documents, as in Fig. 3(e).
3.1.4 DATASET ANNOTATION PIPELINE
In order to create high-quality annotations for the GraphDoc-Dataset, we invested significant effort in enhancing the relational annotations while maintaining the foundational document layout anno- tations (DLA) from the original DocLayNet (Pfitzmann al., 2022). One of the primary challenges we encountered was the complexity of accurately capturing and annotating the intricate relations between document components, particularly for tasks involving spatial and logical structures. For these challenges, we designed heuristic rule-based relation annotation system. This system is based on the DLA task annotations and the provided PDF files from the DocLayNet dataset. The steps for relation annotating with rule-based system are as follows:
Content Extraction: We apply the Tesseract OCR and PDF parser to extract the text content contained within the bounding boxes of all categories except for Table and Picture.
Spatial relation Extraction: To extract spatial relations in the four directions, we heed Do- cLayNet annotation rules, which ensure that there is no overlap between bounding boxes. This allows us to determine spatial relations by scanning pixel by pixel along the x-axis and y-axis for spatial relations up, down, left, and right. We record only the nearest adjacent bounding box in each direction to avoid redundant definitions.
Basic Reading Order: We designed an algorithm to detect Manhattan or non-Manhattan layouts according the spatial relation among all annotations. Additionally, we employ
hatps///esserracttoorggiibbiii

---

Published as a conference paper at ICLR 2025
the Recursive X-Y Cut algorithm (Ha et al., 1995) roughly establish basic reading order based on the general left-to-right, top-to-bottom reading rule.
categorized into four groups based their roles: (1) elements with direct structural relations; (2) non-textual content within the logical structure; (3) elements lacking direct associations; and (4) references. We establish an internal tree structure for the first two groups based on the text annotation, category, and basic reading order. Within the non-textual content group, Caption designated as the child of the corresponding Table and Picture, to provide textual representations.
Hierarchical Structure: Annotations
Using the extracted hierarchical structure, we establish parent and child relations within each group. child nodes under the same parent are sequentially or- dered via sequence relations based on basic reading order. We match annotation texts to construct reference relation. The reference relations among Table and Picture are estab- lished, excluding Caption. However, references within Caption to others are maintained.
Relation Completion:
In summary, we developed a rule-based relation annotation system that efficiently constructs instance-level relational graph annotations, aligned with document elements bounding box and cat- egory annotations for the gDSA task. Moreover, the most of the results have been manually verified and refined. Our annotation system captures the inherent spatial and logical relations of document layouts, resulting a robust foundation for training and evaluating models on complex DSA tasks.
3.1.5 DATASET STATISTICS
In total, the GraphDoc dataset extends DocLayNet (Pfitzmann et al., 2022) by enriching with detailed relational annotations while maintaining consistency in instance categories and bounding boxes. comprises 80, 000 single-page document images, each selected from an individual doc- ument. resulting in 1.10 million instances across 11 categories: Caption, Footnote, Formula, List- item, Page-footer, Page-header, Picture, the relational data into eight categories defined in Sec. 3.1.3, yielding 4.13 million relation pairs. Spatial relations constitute 64.06% of these pairs, while logical relations make up the remaining 36.94%. It shows that spatial relations dominate the dataset, reflecting the structured nature of document layouts, where components such as Section- Puge-footer, and Text are frequently positioned in spatial proximity, Logical relations, although comprising smaller portion, play critical role in linking elements, e.g., Table and Picture to the corresponding Text.
Table, Text, and Title. We have expanded
The detailed distribution of these relation pairs is illustrated in Figure 4, which pro- vides comprehensive overview of the re- lational statistics within the dataset. The left side of Figure 4 presents an aggregate view of the total relation flow between dif- ferent object categories, disregarding the specific types of relations (c.g., spatial or logical). This visualization highlights how various document elements, such as Text, Picture, and Section-header, interact within the dataset. The intensity of rela- tion flow between categories such as Text and Picture underscores the typical struc- ture of documents, where these elements frequently co-occur or are positioned in proximity to one another.
Figure 4: Relation statistics on the GraphDoc dataset. The chord diagram on the left illustrates the distribution of rela- tionships among various layouts. The heatmap the right visualizes the intensity of relations based layouts (deeper color means higher intensity). Below the heatmap, detailed image presents the case of Reference relations for Picture.
On the right-hand side of Figure 4, the figure delves deeper into a specific relation type reference. The top section presents a heatmap that captures the frequency and distribution of reference rela- tions between different object categories. This heatmap highlights that category Table and Picture have significantly intensive interactions observed with other document layout elements, e.g., Text and List-item. The lower section provides concrete examples of these reference relations, illustrat- ing the detailed reference situation of Picture in real-world document context. Together, these visualizations offer a holistic view of both the overall relational patterns and the specific behaviors of reference relations, providing deeper insights into the structural complexity of document layouts.

---

Published as a conference paper at ICLR 2025
3.2 DOCUMENT RELATION GRAPH GENERATOR
In this section, we introduce the Document Relation Graph Generator (DRGG), an architecture designed generate instance-level relational graphs. DRGG provides end-to-end solution to construct graphs that capture both spatial and logical relations between document layout elements. By leveraging visual features, DRGG aims detect and analyze the structure of document layout elements accurately.
Decoder
As depicted Figure 5, the proposed model based on Encoder- Decoder architecture with backbone for feature extraction. The backbone extracts low-level features from the document image, which are refined through the Encoder- framework. These re- fined features processed through two main heads: the object detection head, responsible for document layout analysis task, and the relation head (DRGG), which predicts relations. DRGG is designed as a aang-and-pay component, enabling seamless inte- gration with existing models without requiring any modification. DRGG consists of two parts: relation feature extractor and relation feature aggregation.
Object Heads
tolin-ggrggett
Relation Heads
Figure 5: Proposed Document Relation Graph Generator (DRGG) for Document Layout Anal- ysis and Document Structure Analysis. The key of our model illustrated in the Relation Hcad, which is responsible for predicting relations tween layout elements. The remaining parts are the standard encoder-decoder architecture used for object detection.
Relation Feature Extractor. The object queries (X") and object feature representations (X') cal- culated at each decoder layer 1 are fed into inde- pendent relation feature extractors in DRGG respec- tively. These are then processed separately through two independent pooling layers (P) and Multi-Layer Perceptrons (MLP,,) in extractors follows:
D' MLPP
DJ (IPP)((2(X)))
where X E R.W N. Xdeetee Di.2 E Nddga Pooling aggregates information across channels, reduc- ing redundancy and improving robustness. The extracted one-dimensional relational features then through upsampling layer (U) and further refined through MIP layers (MLP,.. then concate- nated with the original object features form unified representation of the relational feature (D). The two representations are subsequently expanded into two dimensions along different axes and concatenated derive the final relational features:
Fd Concat (a (MLP) (MML, + x') @ Idmaaa (U; + x!)² @ 1dmaa); (4) where Fl NRN NRNxxxx2daas x2danes This approach captures both direct relations, e.g., spatial proximity, and indirect relations, e.g., reference, between elements.
Relational Feature Aggregation. The extracted relation features from each decoder layer are com- bined using a weighted aggregation method to form a unified representation of the relations between all object queries. This unified representation is subsequently incorporated into the relation predictor (MLP,) g) to generate the relational graph prediction:
G = MLP,
Σ
a 00Fl pel
where E NxN* is number of relation category. (1) are learnable weights for token aggrega- tion. This query-based mechanism ensures that the final document relation graph, which represents combination of image features, spatial layout, and semantic relations, would also be able to improve the accuracy of both document layout analysis and relational prediction.
The output of DRGG is a well-structured graph where nodes represent document elements, and edges represent the relations between these elements. By combining DLA result from detection head, DRGG ensures a more detailed and accurate representation for document structure analysis. More details about the DRGG architecture are presented in the supplementary Sec. C.
3.3 EVALUATION METRICS FOR GDSA
In traditional Scene Graph Generation (SGG) evaluations, metrics such as Mean Recall@ and Pair-Recall k assess the top-k subject-predicate-ohjeee triplets ranked by predicted confidence
7

---

Published as a conference paper at ICLR 2025
scores (Lorenz et al., 2024). However documents often contain variable number of relations, and limiting the evaluation to a fixed top-k can result in important relations being overlooked they are not among the top predictions. Furthermore, there is significant class imbalance in the relations within documents: spatial relations are prevalent, whereas logical relations such as reference are relatively rare. This imbalance poses challenges for evaluation metries that rely on top-k filtering. Threshold-based filtering. in contrast, allows for the inclusion of all relations that exceed a certain threshold, regardless of their frequency or ranking. This approach ensures that rare but critical re- lations are adequately considered during evaluation. Moreover, unlike in traditional SGG, where typically only one relation exists between subject-object pairs, layout elements in the gDSA task can have multiple coexisting relations (e.g., spatial and logical relations), both of which are essen- tial for understanding the document structure. Therefore, the proposed evaluation metrics, mR, and mAP, g. should be capable of measuring the performance in both aspects: detecting layout elements and identifying multiple relations between them, including less frequent but significant relations.
To address these challenges, first perform an exact matching of predicted instances to ground- truth instances based on both bounding box overlap and object category correspondence. Once this mapping is established, we evaluate the predicted relations within this matched set. Similar to the Intersection over Union (IoU) threshold used in object detection, we introduce a relation confidence threshold Ta. All relations with confidence scores exceeding this threshold considered positive relation predictions. The remaining settings align with standard SGG evaluation metrics. This method ensures that the relation evaluation depends on both the performance of document layout analysis and the relation predictions. By explicitly considering the impact of bounding box detection and label prediction on the quality of relation predictions, our evaluation provides comprehensive assessment of the gDSA task. The detailed algorithmic process is presented in Algorithm 1.
Algorithm Relation Graph Evaluation Metrics mR, @TR and mAP, @ @T for gDSA Task
Input: Predicted instances lour, ground truth instances Igt, predicted relations R, ground truth relations Rgt. IoU threshold TroU, relation score threshold TR
Output: Mcan Recall mR, @ @TR, Mean Average Precision mAP, @TR
I: Step I: Instance Matching
2: Initialize mapping M [ar] = null for each x
Jgt
3: for all i E Toue do
4: Find x < arg max (abelt) (o
5: if o null and IoU(2, TroU and (M[x) = null (o, > IoU(2, M(x))
then
6:
M[x] <-
7: <- inverse mapping of
8: Step 2: Relation Evaluation
9: G + ((Z,,,, To) E Rgt
Fa, To E Ioct
10: Xr + (i.,,,,oo. 8p) E R Sp TR and L(t.), + null}
ARR@TT= + fin(XT, G)
> Calculate mean recall threshold Tr
12: mAp, @Tn - SmAr (XT, G)
> Calculate mean average precision at threshold Tn
13: return mR, Tn. mAp, g @TR
4
EXPERIMENTS
4.1 COMPARED METHODS
To evaluate the effectiveness of our proposed DRGG framework on the GraphDoc dataset, we con- ducted experiments comparing it with several state-of-the-an methods in document layout analysis (DLA) and graphical structure analysis (GSA), including DETR (Carion et al., 2020), Deformable DETR (Zhu et al., 2021), DINO (Zhang et al., 2022), and RoDLA (Chen et al., 2024). These meth- ods represent broad range of approaches in object detection and relation extraction. We further explore the impact of various backbone architectures, including InternImage (Wang et al., 2023), ResNet (He et al., 2016), ResNeXt (Xie et al., 2017), and Swin Transformer (Liu et al., 2021), across these models. This allows us to understand the influence of different combination of feature extraction backbones and detector the overall performance of the models.
4.2 IMPLEMENTATION DETAILS
For a fair comparison, we train and evaluate all methods in the MMDetection (Chen et al., 2019) framework. All experiments were conducted using the GraphDoc dataset for both training and
8

---

Published as a conference paper at ICLR 2025
validation. To evaluate the performance of our proposed end-to-end model, we jointly trained and evaluated the DLA and gDSA tasks without separating them. For the object detector component, we employed the model's original configuration. More details are in Appendix F.
4.3 EVALUATION METRICS
To assess the performance of the models on the DLA and gDSA tasks, we employ a set of evaluation metries tailored to capture both the layout elements' detection accuracy and the correctness of the predicted relations. For the DLA task, we utilize the mean Average Precision (mAP) at multiple Intersections over Union (IoU) thresholds.i.e. mAP@50:5:95. This metric computes the average precision across IoU thresholds ranging from 0.50 to 0.95 in increments of 0.05. It accounts for both the localization accuracy of the bounding boxes and the classification accuracy of the layout element categories. In the gDSA task, we report mR, at confidence thresholds of 0.5 and mAP, y at confidence thresholds of 0.5, 0.75, and 0.95. By employing these metrics, we ensure a comprchensive evaluation of both the detection of document layout elements and their complex relational structures, reflecting the real-world challenges of document structure analysis tasks.
4.4 RESULTS
In this section, we evaluate our proposed DRGG with several models on the GraphDoc dataset to benchmark DLA and gDSA tasks. More detailed results of DRGG design are in Appendix E.
Document Layout Analysis. Table 2 presents the results of the DLA task, where we report the mean Average Precision (mAP@50:5:95) for different combinations of backbones and object detec- tors. Our proposed DRGG framework, integrated with the InternImage backbone and the RoDLA detector, achieves mAP of 81.5%, surpassing all other combinations, including the original setup without DRGG. This result highlights the effectiveness of integrating a powerful backbone with detector specifically optimized for document layout analysis. Among the other detectors evaluated, DINO achieves mAp of 79.5% with the InternImage backbone, showing competitive performance. Deformable DETR and DETR obtain lower mAP scores of 73.4% and 68. 2%, respectively, indi- cating challenges in capturing complex document layouts with these models. When analyzing the impact of different backbone networks using the RoDLA in combination with DRGG, the InternIm- age backbone consistently outperforms others. Specifically, Internlmage achieves mAP of 81.5%, compared to 77.9% with ResNeXt, 73.7% 7% with Swin Transformer, and 71.0% with ResNet. These results suggest that the advanced feature extraction capabilities of Internlmage are crucial for accu- rately detecting and classifying diverse layout elements in complex documents.
Table DLA and gDSA Task Results with DRGG on GraphDoc Dataset. mAP@ 50:5:95 mAP@550:5:95 the mean Average Precision(mAA) computed at loU thresholds ranging from 0.50 to 0.95 in incre- ments of 0.05 in DLA Task. mR, @0.5 denotes the mean Recall(mR) in gDSA Task for relation confidence threshold 0.5. mAP, g 0.5. mAP, @0.75, and mAP, @ 0.95 denote the mean Average Precision gDSA Task for relation confidence threshold 0.5, 0.75, and 0.95, respectively.
denotes
DLA
gDSA
Backbone
Detector
Relation Head
mAP@50:5:95
mR,005 00.5
mAP, 80.5
00.75
MAP, e0.95
Interamaage
RoDLA
80.5
DETR
68.2
7.1
13.5
Deformable DETR
73.4
11.5
25.4
11.8
8.5
Intermmaage
DRGG (Ours)
DINO
79.5
19.2
25.2
18.7
14.5
RoDLA
81.5
30.7
$7.6
56.3
46.5
ResNet
71.0
13.8
45.8
17.6
13.3
ResNeXt
77.9
16.9
40.3
18.4
13.6
ReDLA
DRGG (Ours)
Swin
73.7
11.4
26.1
13.5
7.9
Internlmage
81.5
30.7
57.6
56.3
46.5
Graph based Document Structure Analysis. For the gDSA task, evaluate the models using mean Recall (mR. @0.5) and mean Average Precision at different relation confidence thresholds (mAP, @0.5, mAP, @0.75, and mAP, @0.95). As shown in Table 2, the combination of InternIm- age, RoDLA and DRGG achieves superior performance across all metrics. Specifically, it attains a mean recall of 30.7% and the highest mean average precision scores of 57.6 % at 0.5 threshold,

---

Published as a conference paper at ICLR 2025
56.3% at 0.75, and 46.5% 0.95. Comparatively, other models exhibit significantly lower perfor- mance on the gDSA task. DINO, despite performing well on the DLA task, achieves a mean recall of 19.2% and mean average precision of 25.2% at 0.5 threshold. Deformable DETR and DETR perform even worse, with mean recalls of 11.5% and 7.1%, respectively. These results emphasize the difficulty of accurately predicting relational structures in documents and demonstrate the ef- fectiveness of our proposed DRGG framework in addressing this challenge. Examining different backbones with the RoDLA and DRGG further highlights the importance of the backbone network in gDSA performance. The InternImage backbone consistently yields the best results, with signifi- cant margins over ResNeXt, Swin Transformer, and ResNet. This suggests that capturing complex relational information in documents requires not only specialized detectors but also powerful feature extraction capabilities provided by advanced backbone networks.
Relation prediction analysis per category To gain deeper insights into the model's performance on different types of relations, we present per-category relation detection results Table 3. Our DRGG model with Internimage and RoDLA achieves the highest Average Precision (AP, @0.5) across almost all relation categories. For spatial relations, left and right, the model achieves near- perfect scores of 99.0%, indicating exceptional ability to capture spatial positioning between layout elements. In up and down relations, it attains impressive scores of 49.0% each, outperforming other models by substantial margins. In logical relations, parent and child, the model achieves scores of 45.5% for both, demonstrating effectiveness in identifying hierarchical structures within documents. For the sequence relation, critical for understanding reading order, the model attains an AP of 56. 46,, significantly higher than other configurations. The reference relation remains challenging, with the highest AP being 18.8% achieved by ResNeXt with RoDLA. Our model achieves an AP of 16.8% in this category. The lower performance in reference relations suggests that further work is needed to improve the detection of less frequent and more complex relations, possibly by incorporating textual content understanding or additional context.
Table 3: Per-category relation detection results with DRGG model on the GraphDoc dataset, evalu- ated with AP at relation confidence threshold of 0.5 (AP, @0.5).
Backbone
Detector
Relation Head
Up
Down
Left
Right
Parent
Child
Sequence
Reference
DETR
32.4
29.7
8.9
22.8
18.8
27.7
Deformable DETR
16.8
19.8
99.0
11.9
12.9
12.9
20.8
Internimage
DRGG (Ours)
DINO
37.1
38.3
18.8
18.8
11.9
15.8
53.5
7.6
RoDLA
49.0
49.0
99.0
99.0
45.5
45.5
56.4
16.8
ResNet
15.1
17.2
27.7
27.7
6.9
4.0
17.8
16.8
ResNeXt
23.6
24.6
99.1
99.1
11.9
11.9
33.7
18.8
RoDLA
DRGG (Ours)
Swin
18.8
19.8
33.7
99.0
3.9
3.8
23.5
5.6
InternImage
49.0
49.0
99.0
45.5
45.5
56.4
16.8
5 CONCLUTION
In this paper, we introduced the GraphDoc dataset and proposed novel graph-based document structure analysis (gDSA) task. By capturing spatial and logical relations among document layouts, significantly enhanced the understanding of document structures beyond traditional layout anal- ysis methods. Furthermore, we developed the DRGG, an end-to-end architecture that effectively generated relational graphs reflecting the complex interplay of document layouts. As auxiliary module, DRGG leveraged both spatial and logical relations to improve document structure analysis tasks. We conducted extensive experiments, and the results demonstrated that DRGG achieved su- perior performance on the gDSA task, attaining an mR, @0.5 of 30.7% and mAP, @0.5, 0. 0575, and 0.95 scores of 57.6%, 56.3%, and 46.5%, respectively. This performance enhanced the effectiveness of combining document layout analysis with relation prediction to capture document structures.
Limitations. Our model structure focused only on visual modality input without multi-mode modality input consideration, which may have influenced the performance of complex document structure analysis. Future work should explore this integration enhance the model's performance on rela- tional graph prediction. Additionally, our dataset and approach were primarily designed for single- page documents, and extending them to effectively include multi-page documents posed a challenge that remained unaddressed. We acknowledged these limitations and believed that addressing them would be essential for making significant strides toward achieving a duuaa--lik understanding of documents, paving the way for intelligent document processing systems.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
In this section, we outline the efforts made to ensure the reproducibility of our work. All essen- tial details necessary for reproducing our dataset, model, evaluation metrics, and results can be found in the main paper and the appendix. The data annotation process, including how prepared the relation annotations, detailed in Section 3.1.4 and Appendix A.1. The model architecture and implementation specifics, including hyperparameters and training configurations, thoroughly Section 3.2 and 4.2 and detailed in Appendix C and F Lastly, the calculations for the evaluation metrics, including all necessary references to ensure exact reproduction, are documented in Section 3.3, 4.3 and Appendix B.
described
ACKNOWLEDGMENTS
This work was supported in part by Helmholtz Association of German Research Centers, in part by the Ministry of Science, Research and the Arts of Baden-Württembenr Württemberg (MWK) through the Coop- erative Graduate School Accessibility through Al-based Assistive Technology (KATE) under Grant BW6-03, and in part by Karlsruhe House of Young Scientists (KHYS). This work was partially performed on the HoreKa supercomputer funded by the MWK and by the Federal Ministry of Edu- cation and Research, partially the HAICORE @ KIT partition supported by the Helmholtz Asso- ciation Initiative and Networking Fund, and partially on bwForCluster Helix supported by the state of Baden- Württemberg through bWHPC and the German Research Foundation (DFG) through grant INST 35/1597-1 FUGG.
REFERENCES
Hangbo Bao, Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image trans-
formers. In International Conference Learning Representations, 2022. URL https: / //openreview... net./ http//formmmddp//e dtt.ummidddp--22255eee
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part pp. 213-229 229, 2020.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu. Zheng Zhang, Dazhi Cheng. Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li. Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang. Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: arXiv:/906.07/5:
2019.
Yufan Chen, Jiaming Zhang, Kunyu Peng, Junwei Zheng, Ruiping Liu, Philip Torr, and Rainer Stiefelhagen. Rodla: Benchmarking the robustness of document layout analysis models. In CVPR, 2024.
Yihao Ding, Siqu Long, Jiabin Huang. Kaixuan Ren, Xingxiang Luo, Hyunsuk Chung, and Soyeon Caren Han. Form-nlu: Dataset for the form natural language understanding. In Pro- ceedings of the 46th International ACM SIGIR Conference on Research and Development In- formation Retrieval, pp. 2807-2816, 2023a.
Yihao Ding, Siwen Luo, Hyunsuk Chung, and Soyeon Caren Han. Pdf-vqa: A new dataset for real-world vqa on pdf documents. In Gianmarco De Francisci Morales, Claudia Perlich, Natali Ruchansky, Nicolas Kourtellis, Elena Baralis, and Francesco Bonchi (eds.), Machine Learning and Knowledge Discovery in Databases: Applied Data Science and Demo Track, pp. 585-601, Cham, 2023b. Springer Nature Switzerland. ISBN 978-3-031-43427-3.
Lizhao Gao, Bo Wang, and Wenmin Wang. Image captioning with scene graph based semantic concepts. In Proceedings of the 2018 10th international conference on machine learning and computing, pp. 225-229, 2018
11

---

Published as a conference paper at ICLR 2025
Jiuxiang Gu, Jason Kuen, Vlad Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun. Unidoc: Unified pretraining framework for document understanding. Advances in Neural Information Processing Systems, 34:39-50, 2021.
Qipeng Guo, Zhijing Jin, Xipeng Qiu, Weinan Zhang. David Wipf, and Zheng Zhang. CycleGT: Unsupervised graph-to-texi and text-to-graph generation via cycle training. In Thiago Castro Fer- reira, Claire Gardent, Nikolai llinkh., Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina (cds.), Proceedings of the 3rd International Workshop Natural Language Generation from the Semantic Web (WebNLG+), pp. 77-88, Dublin, Ireland (Virtual), 12 2020. Association for Computational Linguistics. URL https: / aaaaacaaa www.llothhogg.....m www.looo..ooooooooooooo wwwgg//0000000 1ebnlg-1.23
Jackyu Ha, R.M. Haralick, and L.T. Phillips. Recursive x-y cut using bounding boxes of connected components. In Proceedings of 3rd International Conference on Document Analysis and Recog- nition, volume 2, pp. 952 955 vol.2, 1995. doi: 10. 109/ICDA.. 19T.IDARR.995.000000 19955602059.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. Proceedings of the 30th ACM International Conference on Multimedia, 2022.
Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. Funsd: A dataset for form under- standing in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pp. 1-6. IEEE, 2019.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 3668-3678, 2015.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1219-1228, 2018
Harold W. Kuhn. The hungarian method for the assignment problem. In Michael Jünger, Thomas M.
Liebling. Denis Naddef, George L. Nemhauser, William R. Pulleyblank, Gerhard Reinelt, Gio- vanni Rinaldi, and Laurence A. Wolsey (eds.), 50 Years of Integer Programming 1958-2008 doi: 10.1007/ 0070000777888 3-
From the Early Years to the State-of-the-Arr. pp. 29 47. Springer, 2010.
978-3-540-68279-01 2. URL https://///// http://ooooooo/////////
Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei. Dit: Self-supervised pre-training for document image transformer. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 3530-3539, 2022.
Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation-aware graph attention network for vi- sual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10313-10322, 2019.
Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel. Commonsensee knowledge base completion. In Proceedings of the 54th Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pp.
2016.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1582-45977 Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ 1/202 1 long.oo...33. 353. URL https: //acl aaccaanthlllo ogy org/ 2021 .acl- -ong...o........
12

---

Published as a conference paper at ICLR 2025
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021.
Julian Lorenz, Robin Schön, Katja Ludwig, and Rainer Lienhart. A review and efficient implemen-
tation of scene graph generation metrics, 2024.
Jiefeng Ma, Jun Du, Pengfei Hu, Zhenrong Zhang, Jianshu Zhang, Huihui Zhu, and Cong Liu. Hrdoe: Dataset and baseline method toward hierarchical reconstruction of document struc- tures. Proceedings of the AAAI Conference on Artificial Intelligence, 1772)1187011877, Jun.
2023. doi: 10. N..1669aaii.v3775 v3712.25277. URL https: petps://ojss aaai org/ iinexx wnnd..ppp///
AAAI/article///aeaaaaaaaaaaaaaae http//iiww///22....2 25277.
Chaitanya Malaviya, Chandra Bhagavatula, Antoine Bosselut, and Yejin Choi. Commonsense knowledge base completion with structural and semantic context. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 2925-2933, 2020.
Gaurav Mittal, Shubham Agrawal, Anuva Agarwal, Sushant Mehta, and Tanya Marwah. Interactive image generation using scene graphs. arXiv preprint arXiv: 1905003433, 2019.
Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S. Nassar, and Peter Staar. Doclaynet: A large ouman-annotall dataset for eecument-layoou segmentation. In Proceedings of the 28th
ACM SIGKDD Conference on Knowledge Discovery and Data Mining. KDD '22, pp. 3743-3751, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781 17881333338500 doi: 10. 10.1145/3534678..355 353399443 URL hwtptttpp///cc // /ooi. www.oo.../.............. 145 353 .133833533333
Devashish Prasad, Ayan Gadpal, Kshitij Kapadni, Manish Visave, and Kavita Sultanpure. Cas- cadetabnet: An approach for end to end table detection and structure recognition from image- based documents. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 572-573, 2020.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference Empirical Methods in Natural Language Process- ing (EMNLP), pp. 5418-5411 5426,, Online, November 2020. Association for Computational Linguis- tics. doi: 10. 10.18663//1/2222emmmplle emnlp-mainn in. 437.
URL https: / aacl antha ogy. org/2020.
Sebastian Schreiber, Stefan Agne, Ivo Wolf, Andreas Dengel, and Sheraz Ahmed, Deepdesrt: Deep learning for detection and structure recognition of tables in document images. In 20/7 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pp. 1162- 1167. IEEE, 2017.
Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D Manning. Gen- erating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the fourth workshop on vision and language. pp. 70-80, 2015.
Taylor Shin, Yasaman Razeghi, Robert
Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In Empirical Methods in Natural Language Processing (EMNLP). 2020.
Jiawei Wang, Kai Hu, Zhuoyao Zhong. Lei Sun, and Qiang Huo. Detect-ordee-conssrueet A tree construction based approach for hierarchical document structure analysis. Pattern Recognition, 156: 10836, 2024. ISSN 0031-3203. doi: hutps://doi.org//1.000666 patco 2024.110836. URL https:/ /wwww/wwww.///m/////////o htt.://ccccccreett.o.cccceeeee/e SOC31320324005879
W. Wang, J. Dai, Z. Chen, Z. Huang. Z. Li, X. Zhu, X. Hu, T. Lu and Y. Qiao. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14408-14413 Los Alamitos, CA, USA, jun 2023. IEEE Computer Society. doi:
X. Wang.
URL https:/ /tt///di.. doi. wtt.laieeeccccotttcooeee..oor.r. 10. 10.1109/cvPR5220 2023. 01355.
00.11999CVPR3222222220000
13

---

Published as a conference paper at ICLR 2025
Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, and Furu Wei. Layoutreader: Pre-training of text and layout for reading order detection, 2021.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Computer Vision and Pattern Recognition, 2017.
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang. Yijuan Lu, Dinei Florencio, Cha Zhang, and Furu Wei. XFUND: A benchmark dataset for multilingual visually rich form understanding. In Find- ings of the Association for Computational Linguisties: ACL 2022, pp. 3214-322 4224, Dublin, Ire- land, May 2022. Association for Computational Linguistics. doi: 10. 10.18653/1/2022 253. URL https: / //a//aaanntttt www.nnooooogyoo.oooooooooooooooe org/ 022...2..22 findings ac1.233.
Sibei Yang, Guanbin Li, and Yizhou Yu. Cross-modal relationship inference for grounding refer- ring expressions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4145-4154, 2019a.
Xu Yang. Kaihua Tang, Hanwang Zhang, and Jianfei Cai. Auto-encoding scene graphs for image captioning. Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 10685-10694 2019b.
Liang Yao, Chengsheng Mao, and Yuan Luo. Kg-bert: Bert for knowledge graph completion. arXiv preprint IrXiv:I909.000
2019.
Cheng Zhang, Wei-Lun Chao, and Dong Xuan. An empirical study on leveraging scene graphs for visual question answering. arXiv preprint arXiv 12007///113,,.. 2019.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection, 2022.
Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for docu- ment layout analysis. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1015-1022. IEEE, Sep. 2019. doi: 111100//DAR.2019.0000
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https::/openrevie..
www.ooommddggnnoooooo 6ke.
14

---

Published as a conference paper at ICLR 2025
A DETAILS OF GRAPHDOC DATASET
A.1 RULE- BALEE RELATION ANNOTATION SYSTEM
In this subsection, we provide an in-depth explanation of the methodologies employed in rule- based relation extraction system. This detailed account covers the technical aspects of each step, which were briefly outlined in the main text.
Content Extraction: We extract textual content from all bounding boxes except those labeled as Table and Picture by combining Optical Character Recognition (OCR) and direct text extraction from PDF files. Initially, we utilize pdfplumber to extract text and positional information directly from PDFs, enabling accurate mapping of text snippets their corresponding bounding boxes. For regions where direct extraction is ineffective- as scanned documents or encrypted PDFs-we apply Tesseract OCR configured with appropriate language settings. By selectively employing OCR only when necessary, we enhance both the efficiency and accuracy of the content extraction process. Integrating both methods ensures comprehensive and reliable retrieval of textual informa- tion across various document types and qualities.
To determine spatial relations in the four cardinal directions, we leverage the non-overlapping property of bounding boxes ensured by the DocLayNet annotation rules. For each bounding box, we calculate its center point and identify the nearest neighboring bounding box in each direction by checking for horizontal and vertical overlaps. If two bounding boxes overlap horizontally, we consider them for left or right relations; if they overlap vertically, we consider them for up down relations. We compute edge distances only when the bounding boxes do not overlap in the respective direction, ensuring accurate neighbor identification. Recording only the nearest neighbor in each direction maintains simplicity and avoids redundancy. This approach efficiently constructs spatial map of document elements, which is crucial for understanding the layout and for subsequent processes like determining the reading order and building hierarchical structures.
Spatial Relation Extraction:
We establish a basic reading order that mirrors natural human reading pat- terns. First, we analyze the document layout to determine if it follows a Manhattan (grid-like) or non-Manhattan structure by assessing alignment consistency and spacing uniformity. We then ap- ply the Recursive X-Y Cut algorithm (Ha et al., 1995) to segment the page hierarchically based on whitespace gaps. This algorithm recursively divides the page into smaller regions, creating a tree structure where leaf nodes correspond to individual bounding boxes. We traverse this tree in depth- first manner, ordering the content from left to right and top to bottom, adjusted for the document's s language and layout specifics. For multi-column layouts, we modify the traversal process content column by column, respecting the intended flow. This method provides a logical reading sequence that aligns with human expectations and supports tasks like text extraction and summarization.
Basic Reading Order:
Hierarchical Structure: We organize the document elements into a hierarchical structure that re- flects their logical relations. Annotations are grouped into four categories:
Elements with direct structural relations (Section-Headdr, Text, Formula, List-Item);
Non-textual content within the logical structure (Table, Picture, Caption);
Elements lacking direct associations (Page-Header, Page-Footer, Titley;
References only (Footnotes)
For the first group, we construct the hierarchy by linking each Section-Header to the subsequent content elements (Texi, Formula, List-Item) that belong that section, based on the established reading order. Subsections are nested under their respective higher-level sections, creating a tree structure that mirrors the document's 's outline. For the second group, we associate each Caption with its corresponding Table or Picture based on their proximity in the document. The combined Table/Pictur and Caption units are then placed into the hierarchy at positions determined by the reading order, linking them to the relevant sections or subsections. This hierarchical arrangement effectively captures the logical structure of the document, facilitating tasks such as information retrieval and semantic analysis by reflecting the inherent relations among the document elements.
umber "nttps://github. com/ jttp://svieeeeeeeeeee f "https://tesseractt .or.gi www.rrggggbiiiiioooooo
15

---

Published as a conference paper at ICLR 2025
Building the hierarchical structure, we establish Parent, Child, Sequence, and Reference relations among the elements. Child nodes under the same parent are connected via sequence relations that reflect the established reading order, with attributes indicating their positional sequence. Reference relations are identified by scanning the text for markers such as citations and footnote indicators, linking them to corresponding elements:
Relation Completion:
Footnotes: Superscript numbers symbols in the text are linked to Footoote elements.
Tables and Figures: Mentions, for example, 'see Table I' are linked to the respective Table or Picture elements.
We exclude Caption elements from being directly referenced to avoid redundancy but maintain references within captions to other elements. Consistency and integrity checks are performed to ensure all relations are correctly established, resolving any conflicts based on predefined rules.
While documents from various domains may have unique characteristics, adopting consistent and general rule of relations allows for a unified approach to structure analysis. To address domain- specific nuances and ensure accuracy, we incorporate human verification, which helps adapt our method to diverse document domains while maintaining relation type definition principles. The extensive human verification and refinement cover approximately 58.5% of the dataset. We reviewed 4,852 pages of Government Tenders, 12,000 pages of Financial Reports. 6,469 pages of Patents, and 8,000 pages from other domains. The refinement rates for relation labels varied across domains: approximately 23% for Financial Reports, 8% for Scientific Articles, 26% for Government Tenders, and 17% for Patents. Based on our comprehensive cross- valisavaii evaluation, we believe that our dataset and benehmark can
dataset is high-quality for the proposed gDSA task. We hope provide an innovative advancement in DSA and document an understanding research field.
A.2 DETAILED STATISTICS OF GRAPHDOC DATASET
In this section, we provide detailed statistics on the GraphDoc dataset. Building upon Do- cLayNet (Pfitzmann al., 2022), GraphDoc extends it with rich relational annotations while main- taining coherence in instance categories and bounding boxes for a comprehensive analysis of doc- ument structures. As shown in Figure 6i, spatial relations constitute significant portion of the relational data, representing more than half of all annotated relations. Of the remaining logical relations, parent and child and sequence relations dominate, while reference relations form a com- paratively smaller subset. This distribution highlights the relation dataset appears to be imbalanced, which could easily lead to long-tail problems during model training.
Spatial Relations in the dataset are dominated by four types: down, up, left, and right, each repre- senting the relative positioning of document components. Spatial relations are essential in document structure analysis because they provide contextual information beyond the raw bounding boxes of document layout elements. Simply knowing the positions of elements is insufficient for under- standing the document's relational structure, especially when real-world perturbations occur, e.g., document image rotation and translation. By defining four fundamental spatial relation types, we aim capture how document elements interact within document fundamentally, facilitating more robust and generalized understanding across different domains. As represented in Figures 6a and 6b, document elements Section-header and Text commonly follow a vertical arrangement, po- sitioned above Text, reflecting a conventional reading order. This vertical structuring is consistent across most document types and contributes an intuitive user experience when processing doc- ument layouts. In addition, as illustrated in Figures 6c and 6d, left and right relations account for another significant portion of spatial proximity relations. Understanding these left-right positional relations is critical when reconstructing the visual layout during document parsing tasks, as they often indicate the intended grouping of related elements.
Logical Relations are essential for understanding both the hierarchical and contextual connections between document layouts. These include parent, child, sequence, and reference relations, each contributing to the logical structure within documents. Parent and child relations define the hierar- chical structure document elements.As observed in Figures 6e and 6f, logical relations provide a clearer horizon compared to spatial relations. Captions are primarily the children of Picture and Ta- ble, while rection-heade often serves as the parent of Text, Formula, and List-item. These relations fundamental defining the document's logical structure, they guide the flow of information
16

---

Published as a conference paper at ICLR 2025
and the progression from one element to another. Additionally, sequence relations important for capturing the order which document components should be read or interpreted. Figure 6g indi- cates that sequence relations mainly occur among Text, List-item, and Formula categories. Figure 6h demonstrates that reference relations, while limited in number, are critical for linking different parts of the document. These relations typically appear among List-item, Text, Table, and Picture ele- ments, forming cross-references that provide additional context or clarification. While reference relations constitute smaller fraction of the overall relational data, their significance cannot be over- looked, as they are key to understanding interdependencies between document elements.
(a) The distribution of up relation based layouts interaction
(b) The distribution of down rela- tion based on layouts interaction
(c) The distribution of left relation based layouts interaction
(d) The distribution of right relation based layouts interaction
(e) The distribution of parent rela- tion based on layouts interaction
(l) The distribution of child relation based layouts interaction
(09000
90006
430000
AL3IE ENISO
44:04
30000
(g) The distribution of sequence re- lation based layouts interaction
(i) The number of relations accord- ing to relation type. Figure 6: The overview of relation distribution GraphDoc Dataset.
(h) The distribution of reference re-
lation based on layouts interaction
B EVALUATION METRICS
This section details the evaluation metrics for assessing model performance on the document layout analysis (DLA) and graph-based document structure analysis (gDSA) tasks. Specifically, discuss the Mean Average Precision (mAP) for the DLA task, and the Mean Recall (mR) and Mean Average Precision for relations (mAP,) , in the gDSA task.
Mean Average Precision for DLA (mAP). For the DLA task, we employ the mAP over multiple Intersection over Union (IoU) thresholds, denoted as mAP@ (50:5:95). To compute the mAP, we first
17

---

Published as a conference paper at ICLR 2025
calculate the Average Precision (AP) for each class each IoU threshold 0.50, 0.53, 0.95 by integrating the area under the precision-recal curve. Then, we average the APs over all classes and loU thresholds:
MAP
Σ IET
E C
APd(0)
(6)
T
where is the set of IoU thresholds and C is the number of classes. A prediction is considered correct if the predicted class matches the ground truth and the IoU exceeds threshold t.
Mean Recall for gDSA (mR,). In the gDSA task, we employ Mean Recall (mR) to evaluate the model'ss ability to detect relations, especially given multiple coexisting relations and class imbalance. To compute the mR, we first match predicted instances to ground truth based on class labels and Intersection over Union (loU) with threshold commonly set 0.5. Next, extract relations triplets. We then apply a relation
from the matched instances, defined as subject-object-predd confidence threshold TR and consider only relations with confidence scores above Tg. For each relation category r, the recall is computed as:
TPr
Recallr
TPr + FNr
where TPr is the number of true positives and FNr is the number of false negatives for relation r. The Mean Recall then calculated by averaging the recalls over all relation categories:
mR
Recall,,
(8)
R
where the total number relation categories.
Mean Average Precision for gDSA (mAPg). To further comprehensively assess model perfor- mance document relational graph prediction, we use the Mean Average Precision for gDSA (mAP,). We begin by performing instance matching and relation extraction described in the computation of mR. We then evaluate the relations at confidence thresholds TR E {0.5, 0.75, 0.95). For each relation category, compute precision and recall, and calculate the Average Precision (AP) by integrating the precision-recal. curve. The mAP, g is then obtained by averaging the APs over all relation categories:
R Σ AP, R
MAP
(9)
where AP, is the Average Precision for relation category and R is the total number of relation categories. This metric balances precision and recall, rewarding models that predict correct relations with high confidence.
Since elements can have multiple relations, we treat relation prediction multi-label classification problem for each pair of instances. By evaluating performance per relation category and averaging. we ensure that rare but important relations are appropriately weighted, effectively addressing class imbalance. Additionally, relation evaluation depends on correctly detected instances, linking the quality of relation prediction to the performance task and m.y and mAP, for the gDSA task, we provide a comprehensive evaluation framework that addresses the challenges of document structure analysis, including multiple relations and class imbalance. This approach encourages the development of models capable of effectively interpreting complex document structures.
the DLA task. By employing mAP for the DLA
18

---

Published as a conference paper at ICLR 2025
Decoder
Object Heads
Backbone
Encoder
layouts
image
Relation Heads
Relation
Weighted Token Aggregation
Predictors
Rei ation Feature Extractor
MLP
Sample
Up
MLP
Relations
Concat
Sample
MLP
Up
MLP
Object Queries
The overall architecture and the work flow of the proposed DRGG model. Given an image of document as imput, the backbone will extract the feature from the document image and forward to the Encoder- Decoder architecture. The output of Decoder will be forwarded to the object heads and the relation heads for the prediction document layouts relations.
Figure 7:
C DRGG
In this subsection, we provide detailed structural analysis of the Document Relation Graph Gener- ator (DRGG) and detailed structural illustration as shown in Figure 7.
C.1 ANALYSIS OF WEIGHTED TOKEN AGGREGATION STRATEGY
The Weighted Token Aggregation strategy in DRGG a crucial mechanism that fine-tunes the im- portance of relational features extracted from different decoder layers, resulting in more accurate and refined predictions. In the DETR framework, object queries at various layers capture feature information at different scales and abstraction levels, which leads to inherent variations in the cor- responding relational features. These differences are key to understanding how document elements relate each other. Different types of relations in documents require attention distinct aspects of the layout. For instance, reference relations requires deeper focus on the content within the document elements. On the other hand, spatial relations demand more emphasis on the geomet- ric properties and boundaries of the document elements. This nuanced understanding of relational features what enables DRGG employ a single relation head to effectively capture and classify multiple types of relations simultaneously. By adjusting the contribution of relational information from different decoder layers, DRGG can adapt to the varying scopes demands each type of relation, ensuring comprehensive and precise representation of document structure.
C.2 RELATION PREDICTOR WITH AUXILIARY RELATION HEAD
To enhance the stability and accuracy of DRGG's relational predictions, we introduce an auxiliary relation prediction head. This auxiliary relation head focuses solely determining whether rela- tion exists between two document elements, without classifying the type of relation. By decoupling the existence of a relation from its categorization, the auxiliary relation head acts stabilizer, ensuring that false positives are minimized during inference.
During training, both the main relation predictor and the auxiliary relation head trained simul- tancously using Binary Cross Entropy (BCE) loss. At test time, the predictions from the auxiliary relation head combined with the main relation predictor's output by multiplying their respec-
19

---

Published as a conference paper at ICLR 2025
tive results. This multiplicative correction reduces uncertainty and enhances the robustness of the relational predictions.
Let the output from the main relation predictor, responsible for classifying specific relations, be denoted as Gredd E R xxxx, where N represents the number of document elements and is the number of relation categories. Similarly, let the auxiliary relation head output, which predicts the existence any relation between elements, be denoted as Aprdd E RNNN, where each entry in Apred represents a binary prediction (relation exists not) for a pair of document elements.
During inference, the final relational prediction Grinal is computed by multiplying the two outputs element-wise:
Grinll = Greed A
(10)
pred
where @ denotes the element-wise product, and Sk represents the auxiliary relation head's pre- pred
dictions expanded along the third dimension to match the number relation categories k. This operation ensures that only relations that are confidently predicted exist by the auxiliary relation head are retained in the final output.
C.3 LOSS FUNCTION WITH HUNGARIAN MATCHING
For training, the loss computation in DRGG leverages the results of the Hungarian matching al- gorithm (Kuhn, 2010) from the object detection head in the final decoder layer. This algorithm ensures instance-level matching between predicted document elements and the ground truth ele- ments, providing one-to-one mapping between predictions and annotations. Once this matching established, the predicted relation graph can be filtered and adjusted according to the matched pairs, which is critical for accurately training the relation predictor.
The Hungarian matching algorithm aims to minimize the total matching cost by finding the optimal permutation that maps the set of predicted elements {PI, .......PP to the ground truth elements T {11, 12, .....,,,,, J. The cost function is defined as:
Cost(a) = Σ C(ps, to(1),
(II)
where C(pi,tori to()) is the loss between the predicted element pt and its matched ground truth element (o()) The optimal matching is obtained by minimizing this cost:
o² = arg min N
C(p., ta());
(12)
where G is the set of all possible permutations of N elements. This matching is critical for aligning predicted relations with the ground truth during training, ensuring that predictions are corrected for each element's 's actual match.
The loss function for both the relation predictor and the auxiliary relation head is based on Binary Cross Entropy (BCE), computed independently for each of the predictions. Specifically, let Ggt E Nx denote the ground truth relational graph, and let Agt E N denote the ground truth existence of relations (i.e., whether a relation exists between pairs of elements). The total loss Cecal is the sum of the losses for objects heads and relation predictor and the auxiliary relation head:
Listal Lets
Libuxx XLrel + o Lrelan
(13)
where À is a hyperparameter that controls the weight of the prediction head loss and is another hyperparameter that controls the weight of the auxiliary relation head loss.
The relation prediction loss fae is defined as:
20

---

Published as a conference paper at ICLR 2025
Cliskk) log (ped)) + +11 - log(1 ())
Lel
(14)
where Gli.k.) and ((,.A) denote the ground truth and predicted probabilities for the k-th relation category between elements and j.
Similarly, the auxiliary relation existence loss Laux given by:
N Σ
(A6 log Ald + Alt log(1 - - A
(15)
By incorporating both losses, DRGG is trained to accurately predict both the existence and the type relations between document elements. The auxiliary relation head plays crucial role in stabilizing the predictions, while the Hungarian matching ensures precise, instance-level alignment between predictions and ground truth, thus improving the overall quality of the relational graph.
D ADDITIONAL RESULTS OF DRGG
In this section, provide detailed supplementary results from our additional DRGG experiments to offer deeper insights into the gDSA task and the structural design of DRGG.
Results on Different Document Domains of GraphDoc Dataset.
To comprehensively evaluate the performance DRGG, we conducted experiments across multiple document domains separately in GraphDoc dataset, reflecting diverse layouts and structural com- plexities. These experiments aim to demonstrate the adaptability of our method to varying document types. The detailed results of six different document domains (i.e., Financial Reports, Scientific Ar- ticles, Laws and Regulations, Government Tenders, Manuals, and Patents) are presented in Table and Table 5 below. We used Internlmage as the backbone, RoDLA as the detector, and DRGG for relationship extraction. The tables below summarize the performance in terms of mRg and mAPg under relation confidence thresholds of 0.5, 0.75, and 0.95 under the loU threshold of 0.5.
Table 4: mR, Results on different document domains of GraphDoc Dataset.
Relation Confidence Thresholds
Financial Reports
Sclentific Articles
and Regulations
Government Tenders
Manuals
Patents
0.5
15.0
46.3
38.7
40.6
40.6
22.7
0.75
12.3
42.0
36.5
38.7
35.6
20.5
Table 5: mAP, Results on different document domains of GraphDoc dataset.
Relation Conlidence Thresholds
Reports
Scientille Articles
Repulations
Government Tenders
Manuals
0.5
$2.6
$4.5
63.2
55,9
46.8
31.8
0.75
50.9
$2.9
58.7
51.4
44.4
30.7
0.95
20.2
47.5
$4.6
48.1
32.5
29.3
The results demonstrate clear domain-specific trends. Laws and Regulations achieve the highest mAp, @0.5 with 63.2, benefiting from their structured and consistent layouts, while Patents per- form worst, with mR, @0.95 at 17.5, due to their dense and complex layouts. Both mR, and mAP, decline the relation confidence threshold increases, reflecting the challenges of capturing pre- cise relationships under stricter criteria. These findings highlight the varying complexities across domains and the need for robustness in handling diverse document structures.
Results on Spatial and Logical Relations of GraphDoc Dataset.
To investigate the impact of different relationship types, analyzed DRGG's performance doc- uments containing only spatial relations compared to those containing both spatial and logical rela- tions. We used Internlmage the backbone, RoDLA the detector, and DRGG for relationship
21

---

Published as a conference paper at ICLR 2025
prediction. We used Internlmage as the backbone, RoDLA the detector, and DRGG for relation- ship extraction. m and mAPg metrics were computed under relation confidence thresholds 0.5, 0.75, and 0.95 with IoU threshold of 0.5, as shown in Table 6.
Table Results for relation prediction performance under different relation types.
Spatial Relation
Logical Relation
mR, @0.5
mR, 0.075
mR, @0095
mAP, @0..5
mAP, @0075
mAP, .0.95
32.1
27.7
22.1
49.5
49.5
41.3
26.7
23.9
20.1
57.5
56.2
37.6
The results show that capturing spatial and logical relations is challenging, as indicated by the lower metrics. Spatial relations alone achieve an mR, @0.5 of 32.1 and a mAp, @0.5 of 49.5. When logical relations are included, mR, @0.5 drops to 26.7, while mAP, @0.5 slightly improves to 57.5. Nevertheless, performance declines significantly stricter thresholds, i.e. mR, @0.95 and mAg @0.95.
E ABLATION STUDY RESULT OF DRGG
In this section, we present the ablation study of the DRGG design to validate the effectiveness of the DRGG model. The analysis evaluates four key aspects: the impact of using DRGG as relational graph prediction head, the effectiveness of the relation feature extractor module, the influence of loU thresholds, and effect of relation confidence thresholds on different relation types.
Ablation of DRGG Model. Table 7 highlight the effectiveness of integrating the DRGG relation prediction head into the document layout analysis task. InternImage combined with DINO sees an improvement from 80.5 to 81.5, the highest among all configurations, illustrating the harmony between the DRGG head and advanced backbones. This improvement mark the DRGG module's utility in capturing complex document structures, it effectively augments the detector's ability to model relationships between document elements. These findings validate the design of DRGG and its critical role in advancing the accuracy and reliability of document structure analysis.
Table 7: Ablation study of DRGG model impact for DLA Task
DLA
Backbone
Detector
Relation Head
N0:5:50:5:95
InternImage
DINO
ResNet
74.3
ResNeXt
RoDLA
77.7
InternImage
80.5
InternImage
DINO
ResNet
71.0
DRGG
ResNeXi
RoDLA
77.9
InternImage
81.5
Ablation of Relation Feature Extractor. Table 8 illustrates the importance of the relation feature extractor in the DRGG model. When paired with InternImage and RoDLA, the feature extractor significantly outperforms linear layer replacement across all metries. For DLA, it achieves higher mAP result of 81.5. In gDSA, the extractor shows clear advantages in mR, and mAP,-
Table 8: Ablation study of relation feature extractor module in DRGG model compared with single linear layer instead of relation feature extractor module in DRGG model
DLA
gDSA
Backbone
Detector
Relation Head
mAP 50:5:95
mR, 600.5
mAP, 600.5
mAP, 650.75
mAP, 690.55
DRGG
81.5
30.7
57.6
56.3
46.5
InternImage
RoDLA
linear layer DRGG
79.9
25.8
52.9
42.3
30.5
22

---

Published as a conference paper at ICLR 2025
Ablation of IoU Thresholds. We understand the importance of evaluating model performance under high IoU thresholds to assess alignment between predicted and actual bounding boxes. To evaluate the impact of high IoU thresholds model performance, Internimage as the backbone, RoDLA as the detector, and DRGG for relationship extraction. The results of Table 9 below present y and mAP, values under IoU thresholds of 0.5, 0.75, and 0.95:
conducted experiments using
Table 9: Impact of IoU thresholds mR, and mAP,, g.
IoU Threshold
@ @0.
mR, @0.75
mR, @0.95
mAP, @0.5
mAP, @0.75
mAP, @0.95
0.5
30.7
28.2
24.5
57.6
56.3
46.5
0.75
28.8
26.5
23.0
56.7
54.8
36.8
0.95
22.1
20.7
18.4
55.5
54.3
36.5
As shown in the results, at the highest IoU threshold of 0.95, the model achieves 18.4 mRg @0.95 and 36.5 mAP, @0.95, demonstrating the significant challenges in capturing precise alignments, particularly in complex or densely packed layouts where bounding box prediction errors have greater impact. While lower IoU thresholds allow the model to achieve higher recall and precision, stricter thresholds demand lie-grained alignment, which may not always be feasible due to the inherent limitations of bounding box prediction accuracy. These findings emphasize the need to balance strict alignment metrics with practical utility based on specific application requirements. Higher loU thresholds, while providing stricter metries, may not fully capture the model's 'ss overall effectiveness in scenarios where moderate overlap suffices.
Ablation of Relation Confidence Thresholds among Relation Categories. Table 10 and Table 11 shows the influence of different relationship confidence thresholds in the context of imbalanced sample sizes among relation categories. We used Internlmage the backbone, RoDLA the detector, and DRGG for relationship extraction. mRg @0.5, mRg @0.75, and mRg @0.95 denote the mean Recall in the gDSA Task for relation confidence threshold 0.5, 0.75, and 0.95 under loU threshold 0.5, respectively. mAPg @0.5, mAPg @0.75, and mAPg @0.95 denote the mean Average Precision in the gDSA Task for relation confidence threshold 0.5, 0.75, and 0.95 under IoU threshold 0.5, respectively.:
Table 10: mR, Results different relation confidence thresholds.
Confidence Threshold
Up
Down
Left
Right
Parent
Child
Sequence
Reference
0.5
41.7
50.0
71.4
71.4
12.5
25.0
0.0
0.0
0.75
41.7
33.3
42.9
57.1
12.5
12.5
0.0
0.0
0.95
8.3
8.3
28.6
28.6
12.5
0.0
0.0
0.0
Table 11: mAP, Results at different relation confidence thresholds.
Confidence Threshold
Up
Down
Left
Right
Parent
Child
Sequence
Reference
0.5
49.0
49.0
99.0
99.0
45.5
45.5
56.4
16.8
0.75
47.4
45.1
99.0
99.0
45.5
45.5
51.2
16.8
0.95
40.4
40.4
49.5
49.5
37.6
36.6
46.5
0.0
From the experiment result, we could find that, spatial relations, i.e., Left, Right, Up. and Down achieve consistently higher mR and mAP values compared to logical relations, i.e., Parent, Child, Sequence, and Reference, reflecting their prevalence in the dataset and larger training sample sizes. As the confidence threshold increases, both mR and mAP values decline across all relation types, with logical relations showing the steepest drop; for instance, Reference achieves 16.8 mAP at a 0.5 threshold but drops to 0.0 at 0.95, highlighting the challenges of capturing infrequent ambiguous relationships. A confidence threshold of 0.5 strikes a balance between precision and recall, but addressing dataset imbalance through weighted training could further enhance performance.
23

---

Published as a conference paper at ICLR 2025
Sociion
Donr Chidd
Rasert
Down
Talle
Tasle
Tabde
Down
Dowe
MARC
Pager tarder
Pagu header
Dowe
Qhiy
Model Prediction Figure 8: Qualitative Results for DRGG prediction, compared with ground truth on GraphDoc Dataset.
Ground Truth
F IMPLEMENTATION DETAILS
Hardware Setup. In this work, all experiments were conducted on a computing cluster node equipped with four Nvidia A100 GPUs, each with 40 GB of memory. Each node would also with 300 GB of CPU memory.
Training Settings. We implemented our method using Py PoTorc v1.10 and trained the model with the AdamW optimizer using a batch size of 4. The initial learning rate was set to 1 x 10-1, with weight decay of 5 x 10-3. The AdamW hyperparameters, betas and epsilon, were configured to (0.9, 0.999) and 1 x 10- respectively. To enhance the model's robustness and accuracy, we employed a multi-scale training strategy. Specifically, the shorter side of each input image was randomly resized to one of the following lengths: 480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800, while ensuring that the longer side did not exceed 1333 pixels. This approach helps the model generalize better to varying document sizes and layouts, reflecting the diverse nature of real-world document data.
G QUALITATIVE RESULTS OF DRGG
In this section, we present several qualitative results predicted by DRGG on GraphDoc validation dataset, alongside their corresponding ground truth annotations for comparison.
As illustrated in Figure 8, errors in relation prediction arise primarily from two sources. First the ambiguity in densely populated layouts, where elements, e.g., captions and figures, lack clear align- ment. Secondly, misclassification and inaccurate bounding boxes, from the DLA stage, propagate errors to the relation prediction process. Despite these challenges, DRGG demonstrates promising capabilities in capturing key spatial and logical relationships, such as parent-child links between Picture and Caption. Nonetheless, the DRGG performance hindered in DIA accuracy, seen in cases of misclassified tables leading to missing relationships. To address these issues, we suggest incorporating multimodal embeddings that combine visual and textual features, improving the DLA backbone for enhanced detection performance, and integrating post-processing methods to refine predictions using contextual cues. Additionally, extending DRGG multi-page relational under- standing will enhance its applicability for comprehensive document structure analysis and relation predictions.
24


Reviews:
[{"review_id": "J7SCsn0YX8", "reviewer": "Reviewer_jDzA", "summary": "This paper introduces a dataset which is built on top of the DocLayNet dataset to provide detailed annotation of both spatial and logical relation annotations for understanding the structure of single-page documents. As well as, a new framework, Document Relation Graph Generator, is proposed as a baseline for conducting layout and structure understanding in an end-to-end manner.", "questions": "1. Why do you think spatial relation should be considered as one of the relation types of this dataset which might be directly acquired by leveraging the predicted bbox coordinated?\n2. Could you provide more statistics details before and after human checking?\n3. Do you think the rules might be updated based on different document domains?\n4. For the relation prediction dataset, is there any reason you choose single-page datasets? It is more natural to use multipage-scenarios to establish the entire document's logical structure.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The research problem is interesting and novel which is the less explored in this area.\n2. The dataset generation and the evaluation metrics look reasonable.\n3. Some analyses are conducted to show the insight of the datasets such as the correlations and co-occurrence of different relation pairs.", "weaknesses": "1. Some key concepts are not clearly defined: e.g. why spatial relations are important, why only four spatial relation types are defined, Is proper all documents from different domains to adopt the same rule to define the relations between semantic entities.\n2. Lack of analysis about rule-based generated relation and human refining details which is essential for demonstrating the high quality of the dataset.\n3. It would be clearer if the detailed categorisation of parent, child, and reference relation types could be provided with examples (visualised would be much better) to enable the reader to understand definitions and tasks clearly.\n4. The description of Document Relation Graph Generator is not clearly described to me. The overall workflow with the corresponding aim is relatively hard to understand. I have gone through it several times but still hard to understand including input to the framework, why the encoder and encoder structure, and how the decoder works during training and inferencing. \n5. The experimental section is underexplored. Analysis of different document domains, fine-grained relation type analysis (section->subsection, section->paragraph). \n6. Similar to quantitative analysis, more qualitative analyses are expected. \nI really like the idea of the paper, but more in-depth analysis is essential to maximising the contributions of the author desired.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XOErLNs5Pg", "reviewer": "Reviewer_HRLr", "summary": "This paper introduces the **GraphDoc dataset**, a new dataset for document layout and structure analysis derived from the DocLayNet dataset by adding relation annotations. The authors propose a new task called **graph-based Document Structure Analysis (gDSA)** that involves not only detecting document elements, but also generating **spatial and logical relations** in the form of a graph. To address this new task, the authors also propose a model called the **Document Relation Graph Generator (DRGG)** that can generate relation graphs from document layouts, achieving 57.6% mAPg@0.5 on the GraphDoc dataset.  The authors hope that this graphical representation of document structure will be an innovative advancement in document structure analysis and understanding.", "questions": "Line 284: \"most of the results have been manually verified and refined\". What percentage of results have been manually verified and what percentage of results have been manually refined?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a large new dataset (80,000 single-page document images ; 1.10 million instances across 11 categories: Caption, Footnote, Formula, Listitem, Page-footer, Page-header, Picture, Section-header, Table, Text, and Title.) that extends DocLayNet Pfitzmann et al. (2022), and the relative location of the document elements is key to document understanding", "weaknesses": "This paper doesn't really have any major weaknesses. it is mostly a dataset paper and the dataset is strong. The authors also introduce a new model, which is rather straightforward but still interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XbqRNB2qKh", "reviewer": "Reviewer_zRCK", "summary": "The paper defines a graph-based Document Structure Analysis task (gDSA) and builds a new Document Structure Analysis dataset, GraphDoc, based on existing datasets. By constructing a document layout graph structure that includes spatial and logical relationships, it enhances the depth of document structure understanding. Additionally, the paper presents a new end-to-end Document Relation Extraction model, DRGG, designed to extract features from the document layout and generate a relation graph that includes spatial and logical relationships among document elements.", "questions": "1. A performance bottleneck has been observed in detecting reference relationships in the experiments. Given the importance of reference relationships in document structure understanding, could you supplement with an analysis of the error cases in reference relationship detection, discussing the reasons for these errors and suggesting improvements.\n2. It is suggested that the author can elaborate on the utility of the proposed dataset, evaluation tasks, and evaluation metrics in understanding complex document interactions. Is it possible to provide specific examples of how the GraphDoc dataset and DRGG model can be applied in real-world document understanding tasks? Highlighting of these advantages, such as Reading Order Prediction and Document Hierarchical Structure Analysis tasks mentioned in the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The author compares the proposed DRGG framework with several existing document layout analysis and graph structure analysis methods, demonstrating through experiments that this model exhibits outstanding performance on the gDSA task, particularly in handling complex document structures, which is significant for achieving deeper document understanding.", "weaknesses": "1. Most document instances exhibit multiple relationships. The paper should provide additional experiments to evaluate the precision and recall of DRGG in capturing instances that have both spatial and logical relationships, compared to those with only one type of relationship.\n2. In the dataset, the number of samples in the position relationship category is significantly higher than that in the logical relationship category. It would be better to consider adding experiments to demonstrate how to select the relationship confidence threshold to ensure evaluation accuracy in the context of imbalanced sample sizes among relationship categories.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WqtlmPsFLt", "reviewer": "Reviewer_G241", "summary": "In this paper, the authors introduced an approach to analyzing document structures using graph theory, which offers a unique perspective on understanding the complex relationships and patterns within documents. The primary contribution of this paper lies in its methodology that transforms textual content into graphical representations, enabling researchers and practitioners to visualize and analyze the hierarchical and relational aspects of documents more effectively.", "questions": "1. A higher IoU score suggests a better alignment between the predicted and actual bounding boxes.What is the IoU score when it falls between 0.7 - 0.95  and  0.8 - 0.95?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The authors propose a method that represents documents as graphs, where nodes could represent sentences or paragraphs and edges could capture relationships between these elements based on semantic similarity or other linguistic features. This original perspective offers a fresh way of understanding the hierarchical organization within texts.\n\n2. The authors provide a clear and systematic approach to converting textual documents into graph structures, followed by an analysis that leverages graph algorithms. They also discuss the advantages of their method over traditional text-based approaches, such as improved handling of complex document structures and enhanced interpretability through visual representations. \n\n3.  The clarity of the paper is good. The authors have structured the content logically, starting with an introduction to graph theory and its relevance to document analysis, followed by a detailed explanation of their method, implementation details, and results.", "weaknesses": "1. The paper could benefit from a more explicit discussion on potential weaknesses or limitations of the proposed method. This includes considerations such as scalability with large datasets, robustness to noise, and applicability across different types of documents.  \n\n2. There are lack of comparison with MLLM based methods. The results of GPT4 should be reported for readers to compare.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)