<figure><img src="image_1.png" /></figure>

HAL
open science

# Hierarchical Document Parsing via Large Margin Feature Matching and Heuristics

Duong Anh Kiet

<figure><img src="image_2.png" /></figure>

## To cite this version:

Duong Anh Kiet. Hierarchical Document Parsing via Large Margin Feature Matching and Heuristics. DocUI@AAAI-25, 2025, Philadelphia, United States. 10.48550/arXiv.2502.07442. [hal-05040156]

HAL Id: hal-05040156

https://hal.science/hal-05040156v1

Submitted on 18 Apr 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Hierarchical Document Parsing via Large Margin Feature Matching and Heuristics

Duong Anh Kiet

L3i Laboratory, La Rochelle University
17042 La Rochelle Cedex 1 - France
anh.duong@univ-lr.fr

## Abstract

We present our solution to the AAAI-25 VRD-IU challenge, achieving first place in the competition. Our approach integrates large margin loss for improved feature discrimination and employs heuristic rules to refine hierarchical relationships. By combining a deep learning-based matching strategy with greedy algorithms, we achieve a significant boost in accuracy while maintaining computational efficiency. Our method attains an accuracy of 0.98904 on the private leader-board, demonstrating its effectiveness in document structure parsing.

**Code** — https://github.com/ffyyytt/VRUID-AAAI-DAKiet

## Models —

https://www.kaggle.com/models/fdfyaytkt/vruid-aaai-dakiet/PyTorch/figure

## Introduction

Document structure parsing is essential for visually rich document understanding, enabling applications such as information retrieval and document summarization (Xu et al. 2020). Unlike traditional text-based document processing, mining reports and other complex documents contain heterogeneous semantic entities, requiring hierarchical parsing to infer parent-child relationships (Li et al. 2020).

The AAAI-25 Visually-Rich Document (VRD-IU) Leader-board task involves predicting parent-child relationships among detected entities in mining reports. This problem is challenging due to scanned and inconsistently structured documents, necessitating inference from spatial and contextual cues. Performance is evaluated using accuracy.

In this paper, we present our solution, which ranked first in the competition. We combines large margin loss for improved feature discrimination with heuristic greedy algorithms for efficient hierarchical assignment. Method details our approach, Experiments presents experimental results, and Conclusion section concludes with insights and future directions.

## Related Work

Documents inherently exhibit a hierarchical structure, where parent-child relationships are fundamental across various

Copyright © 2025, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

formats, such as figure-caption and section-paragraph associations in reports (Ding, Lee, and Han 2024). Accurately modeling these relationships is crucial for document structure parsing. Prior works such as DocStruct (Wang et al. 2020), SERA (Zhang et al. 2021), and KVPFormer (Hu et al. 2023) have significantly contributed to this task by leveraging deep learning models for entity linking. However, these methods rely on conventional distance metrics, which may struggle to differentiate entities with highly similar feature representations (Duong and Gomez-Krämer 2025b).

Recent advancements in matching take inspiration from CLIP (Radford et al. 2021), instead of cross-modal learning between text and images, document structure parsing deals with entities of the same modality. However, we also need feature representations for child $fc$ and for parent $fp$. In terms of feature separation, large margin loss functions such as LMCot (Duong, Nguyen, and Truong 2022), Arc-Face (Deng et al. 2019), and NormFace (Wang et al. 2017) improve feature extraction by enforcing larger inter-class separability while maintaining intra-class compactness.

Beyond deep learning-based approaches, heuristic rules and greedy algorithms have proven effective in competitive settings (Duong and Gomez-Krämer 2025a). In the next section, we detail our integration of large margin loss for matching and the heuristic rules applied in the competition.

## Method

In this section we introduce our loss function with improvements from large margin loss. And then the heuristic rules we observed to increase the performance of the model.

### Margin loss for matching

Starting with the loss function in CLIP (Radford et al. 2021):

$$L_{\text{CLIP}} = -\frac{1}{N_c} \sum_{i=1}^{N} \log \frac{e^{f_{c_i} \times f_{p_{y_i}^T}}}{\sum_{j=1}^{N_p} e^{f_{c_i} \times f_{p_j^T}}}, \quad (1)$$

where $N_c$ and $N_p$ denote the number of child entities requiring a parent and the number of potential parent entities, respectively. The feature representations of child and parent entities are denoted as $fc_i$ and $fp_i$, where $i$ is the index of the entity. The term $fp_i$ corresponds to the feature of the actual parent of the $i$-th child entity, where $y_i$ indicates its
parent's index. In other words, the numerator captures the similarity between a child and its correct parent, while the denominator sums over all potential parent entities.

Then we can reformulate the equation 1 as:

$$L_{\text{CLIP}} = - \frac{1}{N_c} \sum_{i=1}^{N} \log \frac{e^{s \|f_{c_i}\| \|f_{p_{y_i}}^T\| \cos(\theta_{i,y_i})}}{\sum_{j=1}^{N_p} e^{s \|f_{c_j}\| \|f_{p_j}^T\| \cos(\theta_{i,j})}}, \quad (2)$$

where $\theta_{i,j}$ is the angle between the child feature vector $f_{c_i}$ and the parent feature vector $f_{p_j}$. A scaling factor $s$ is then introduced to restore the magnitude after normalization.

Following NormFace (Wang et al. 2017), we normalize the feature vectors so that $\|f_{c_i}\| = \|f_{p_j}\| = 1$ for all $i, j$. Building upon ArcFace (Deng et al. 2019), we further introduce a margin parameter $m$, which strengthens the relationship between parent-child pairs while increasing the separation between unrelated entities. By this way, the loss function is rewritten as:

$$L_{\text{matching}} = - \frac{1}{N_c} \sum_{i=1}^{N} \log \frac{e^{s \times \cos(\theta_{i,y_i} + m)}}{e^{s \times \cos(\theta_{i,y_i} + m)} + \sum_{j=1, j \neq i}^{N_p} e^{s \times \cos(\theta_{i,j})}}, \quad (3)$$

Thus, we effectively incorporate advancements in large margin cosine loss into feature matching learning. With this loss function, the feature extraction process is optimized such that, even with the margin parameter $m$, features of related parent-child entities remain closer in the learned space compared to unrelated ones. This encourages the model to form more discriminative representations, enhancing the accuracy of hierarchical relationship inference.

## Greedy algorithms

In addition to the loss function introduced earlier, we identified several structural patterns that improve both the accuracy and computational efficiency of our approach. These heuristics allow for a more efficient assignment of parent-child relationships, reducing the need for exhaustive pairwise comparisons. First, entities belonging to specific categories never have a parent, including: abstract, appendix_list, cross, figure, form_title, list_of_figures, list_of_tables, other, references, report_title, section, summary, table, table_of_contents, and title.

Second, a strong hierarchical relationship exists among the categories: section, subsection, subsection, subsubsection, and paragraph. These entities follow a strict sequential structure, where each entity is assigned as a child to the nearest preceding entity in the order.

Third, specific entities exhibit predefined parent-child dependencies:

* `table caption` is always a child of `table`.

* `figure caption` is always a child of `figure`.

* `form` is always a child of `summary`, `abstract`, `section`, `subsection`, `subsubsection`, or `subsubsubsection`.

* `list` is a child of `paragraph`, `section`, `subsection`, `subsubsection`, or `subsubsubsection`.

* `form body` is a child of `form title`, `summary`, `abstract`, `section`, `subsection`, `subsubsection`, or `subsubsubsection`.

By enforcing these structured dependencies, we further improve the accuracy of entity relationship prediction while maintaining computational efficiency.

## Experiments

In this section, we present the competition dataset, our implementation, and the archived results of the competition.

### Dataset

The dataset contains 571 training, 165 validation, and 81 test documents. Labels are provided only for the training set, while the validation and test sets are used for the public and private leaderboards. Each document contains multiple objects, each with a bounding box in the scanned document and optional text content. The task is to predict the parent object for each object, evaluated using accuracy.

### Implementation

We use a pre-trained BART-large (Lewis 2019) for text feature extraction and a pre-trained SwinV2-Tiny (Liu et al. 2022) for image features. The extracted features are concatenated with additional information such as page number and bounding box position. Two separate encoders, each with a 512-dimensional output, are used to extract features for parent and child entities.

### Results

Due to time constraints in the competition, we were unable to conduct extensive experiments on alternative methods or additional datasets. Table 1 presents the main results obtained on the competition dataset, evaluated on the validation set and test set.

<table><caption>Table 1: Performance comparison of our methods.</caption><thead><tr><th rowspan="2">Method</th><th colspan="2">Accuracy</th></tr><tr><th>val</th><th>test</th></tr></thead><tbody><tr><td>loss only</td><td>0.79674</td><td>0.85824</td></tr><tr><td>loss+greedy</td><td>0.97369</td><td>0.98904</td></tr></tbody></table>

## Conclusion

In this work, we proposed a novel approach for document structure parsing, combining large margin loss with heuristic-based greedy algorithms. Our method enhances feature discrimination while efficiently capturing hierarchical relationships. The results demonstrate the effectiveness of integrating deep learning with rule-based refinements, achieving state-of-the-art performance in the AAAI-25 VRD-IU challenge. Future work includes exploring more diverse datasets and refining heuristic rules for better generalization.
References

Deng, J.; Guo, J.; Xue, N.; and Zafeiriou, S. 2019. Arcface: Additive angular margin loss for deep face recognition. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, 4690–4699.

Ding, Y.; Lee, J.; and Han, S. C. 2024. Deep Learning based Visually Rich Document Content Understanding: A Survey. *arXiv preprint arXiv:2408.01287*.

Duong, A.-K.; and Gomez-Krämer, P. 2025a. Addressing Out-of-Label Hazard Detection in Dashcam Videos: Insights from the COOOL Challenge. *arXiv preprint arXiv:2501.16037*.

Duong, A.-K.; and Gomez-Krämer, P. 2025b. Scalable Framework for Classifying AI-Generated Content Across Modalities. *arXiv preprint arXiv:2502.00375*.

Duong, A.-K.; Nguyen, H.-L.; and Truong, T.-T. 2022. Large margin cotangent loss for deep similarity learning. In *2022 International Conference on Advanced Computing and Analytics (ACOMPA)*, 40–47. IEEE.

Hu, K.; Wu, Z.; Zhong, Z.; Lin, W.; Sun, L.; and Huo, Q. 2023. A question-answering approach to key value pair extraction from form-like document images. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 37, 12899–12906.

Lewis, M. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. *arXiv preprint arXiv:1910.13461*.

Li, M.; Xu, Y.; Cui, L.; Huang, S.; Wei, F.; Li, Z.; and Zhou, M. 2020. DocBank: A benchmark dataset for document layout analysis. *arXiv preprint arXiv:2006.01038*.

Liu, Z.; Hu, H.; Lin, Y.; Yao, Z.; Xie, Z.; Wei, Y.; Ning, J.; Cao, Y.; Zhang, Z.; Dong, L.; et al. 2022. Swin transformer v2: Scaling up capacity and resolution. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, 12009–12019.

Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In *International conference on machine learning*, 8748–8763. PMLR.

Wang, F.; Xiang, X.; Cheng, J.; and Yuille, A. L. 2017. Normface: L2 hypersphere embedding for face verification. In *Proceedings of the 25th ACM international conference on Multimedia*, 1041–1049.

Wang, Z.; Zhan, M.; Liu, X.; and Liang, D. 2020. Docstruct: A multimodal method to extract hierarchy structure in document for general form understanding. *arXiv preprint arXiv:2010.11685*.

Xu, Y.; Li, M.; Cui, L.; Huang, S.; Wei, F.; and Zhou, M. 2020. Layoutlm: Pre-training of text and layout for document image understanding. In *Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining*, 1192–1200.

Zhang, Y.; Zhang, B.; Wang, R.; Cao, J.; Li, C.; and Bao, Z. 2021. Entity relation extraction as dependency parsing in visually rich documents. *arXiv preprint arXiv:2110.09915*.