Venue: NAACL 2024

# Re\({}^{2}\): Region-Aware Relation Extraction from Visually Rich Documents

Pritika Ramu\({}^{\diamond\dagger}\)  Sijia Wang\({}^{\spadesuit}\) Lalla Mouatadid\({}^{\clubsuit}\) Joy Rimchala\({}^{\clubsuit}\) Liftu Huang\({}^{\clubsuit}\)

\({}^{\diamond}\) Adobe Research   Virginia Tech    Intuit AI Research

pramu@adobe.com  {sijiawang,lifuh}@vt.edu

{lalla_mouatadid,joy_rimchala}@intuit.com

Work done while interning at Virginia Tech

###### Abstract

Current research in form understanding predominantly relies on large pre-trained language models, necessitating extensive data for pretraining. However, the importance of layout structure (i.e., the spatial relationship between the entity blocks in the visually rich document) to relation extraction has been overlooked. In this paper, we propose **RE**gion-Aware **R**elation **E**xtraction (**RE\({}^{2}\)**) that leverages region-level spatial structure among the entity blocks to improve their relation prediction. We design an edge-aware graph attention network to learn the interaction between entities while considering their spatial relationship defined by their region-level representations. We also introduce a constraint objective to regularize the model towards consistency with the inherent constraints of the relation extraction task. To support the research on relation extraction from visually rich documents and demonstrate the generalizability of **RE\({}^{2}\)**, we build a new benchmark dataset, DiverseForm, that covers a wide range of domains. Extensive experiments on DiverseForm and several public benchmark datasets demonstrate significant superiority and transferability of **RE\({}^{2}\)** across various domains and languages, with up to 18.88% absolute F-score gain over all high-performing baselines1.

Footnote 1: Code and dataset available at [https://github.com/VT-NLP/Form-Document-IE](https://github.com/VT-NLP/Form-Document-IE)

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 8731–8747

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

Visually Rich Documents (VRDs) encompass various types such as _invoices_, _questionnaire forms_, _financial forms_, _legal documents_, and so on. These documents possess valuable layout information that aids in comprehending their content. Recent research Liu et al. (2019); Jaume et al. (2019); Yu et al. (2020) has focused on extracting key information, such as entities and relations, from VRDs by leveraging their layout structures and Optical Character Recognition (OCR) results2. Figure 1 shows an example where entity recognition aims to identify blocks of text in certain categories, such as \(Question(Q)\), \(Answer(A)\), and \(Header(H)\). Relation extraction further predicts the links among the entities, especially \(Q\)-\(A\) links indicating that the \(A\) block is the corresponding answer to the \(Q\) block.

Footnote 2: Optical Character Recognition will recognize a set of bounding boxes and their corresponding text from VRDs where each bounding box can represent a single word or a cohesive group of words, both semantically and spatially.

Extracting key information, especially relations in VRDs is a challenging task. Though similar to traditional extraction tasks in text-only Natural Language Processing (NLP) Grishman (1997); Chen et al. (2022), inferring relations in VRDs poses additional challenges. They require not only understanding the semantic meaning of entities but also taking into account the layout information, e.g., the spatial structures among the entity blocks in original VRDs. Previous studies mainly focused on combining the text and layout with language model pre-training Lu et al. (2019); Su et al. (2020); Chen et al. (2020); Powalski et al. (2021); Xu et al. (2022a); Wang et al. (2022b); Huang et al. (2022) or encoding the local layout information by constructing super-tokens Qian et al. (2019); Liu et al. (2019); Yu

Figure 1: Example of entity and relation extraction from a visually rich document. The colored boxes represent three categories of semantic entities and the arrows represent relations between them.



et al., 2017; Lee et al., 2022, 2023). However, the layout of the VRDs, especially the relative spatial relationship among the entity blocks, is still yet to be effectively explored for relation extraction.

To this end, we propose **RE**gion-Aware **R**elation **E**xtraction (\({\bf RE}^{2}\)) that leverages region-level spatial structures among the entities to reason about their relations3. Specifically, given the question and answer entities from each VRD, we define three categories of region-level representations for each entity block, through which we further characterize the relative spatial relationship between each pair of question and answer entities. We then employ a layout-aware pre-trained language model (i.e., LayoutXLM Xu et al. (2022)) to encode the entities and an Edge-aware Graph Attention Network (eGAT) to further learn the interaction between the question and answer entities in a bipartite graph while considering their spatial relationship. To ensure each answer is linked to at most one question, we design a constraint-based learning objective to guide the learning process, in combination with the relation classification objective.

Footnote 3: This work mainly focuses on extracting _Q-A_ relation given the gold _Question_ and _Answer_ entities.

To validate the effectiveness of \({\bf RE}^{2}\), we conduct extensive experiments on various benchmark datasets for a wide range of languages and domains. We evaluate \({\bf RE}^{2}\) on two public datasets FUNSD Jaume et al. (2019) and XFUND Xu et al. (2022), under supervised, multitask transfer, and zero-shot cross-lingual transfer settings. We also create a new benchmark dataset \({\bf DiverseForm}\) that covers diverse domains, such as Veterans Affairs, visa applications, tax documents, air transport and so on, and evaluate \({\bf RE}^{2}\) for cross-domain transfer. Experimental results show that \({\bf RE}^{2}\) outperforms the previous state-of-the-art approaches with a large margin on (almost) all languages and domains across all settings. Our ablation studies also verify the significant benefit of the region-level spatial structures of entity blocks for relation extraction. The contributions of this work are summarized as follows:

* We are the first to propose the region-level entity representations and utilize them to characterize the spatial structure among the entity blocks, which have been proven to be significantly beneficial to relation extraction from visually rich documents.
* We develop a new framework \({\bf RE}^{2}\) that leverages the spatial structures among the question and answer entities with an effective eGAT network and regularizes model predictions with a novel constraint objective. \({\bf RE}^{2}\) demonstrates superior performance across (almost) all languages and domains under supervised, cross-lingual, and cross-domain transfer settings.
* We contribute \({\bf DiverseForm}\), a new benchmark dataset that covers a wide range of domains to support the research on information extraction from visually rich documents.

## 2 Related Work

Recent research on visually rich document information extraction shows that incorporating 2D positional embedding and layout coordinates into the pre-trained language models improves VRD understanding Xu et al. (2020, 2022); Huang et al. (2022); Powalski et al. (2021). Wang et al. (2022) models the spatial relationship of fine and coarse-grained visual elements based on Intersection over Union (IoU) and focuses only on named entity recognition task. Incorporating relative spatial positions of entities is essential for relation extraction task. Luo et al. (2023) incorporates the relative spatial relation between entities on a fine-grained level and serves as a task for model pre-training. To deal with the variation of relation definitions, DocRel Li et al. (2022) proposes a contrastive learning framework that utilizes the coherence of existing relations in diverse enhanced positive views to generate relation representations. Zhang et al. (2021) further explores entity relation extraction as dependency parsing, incorporating minimum vertical and horizontal distances between the entities as layout heuristics. Compared with all these studies, our approach is the first to propose and incorporate multi-granular spatial structures among the entities, which have been shown to significantly improve relation extraction from VRDs.

Graph Attention Networks (GAT) Velickovic et al. (2018) have proven to be efficient for learning on graph-structured data Zhang et al. (2022). This is exemplified by the work GraphDoc Zhang et al. (2022), a multimodal graph attention-based model that simultaneously utilizes text, layout, and image information for visually rich document understanding. Though several studies Liu et al. (2019); Lee et al. (2022, 2023) have explored GNNs for entity extraction from VRDs, we are the first to design edge-aware GAT to improve relation extrac 

tion from VRDs, which presents additional challenges, encompassing spatial analysis to determine entity layout on the page and semantics between entities for identifying relations. GNNs have also been applied to relation extraction from textual documents [23; 19]. However, these methods cannot be directly adapted to relation extraction from VRDs due to the fundamental differences in document formats, structures, and the key challenges encountered in relation extraction: text-only documents primarily rely on linguistic cues and phrases for relation extraction, whereas VRDs necessitate consideration of both semantics and spatial context. Given that, we innovatively incorporate a multigranular layout heuristic into an edge-aware graph attention network, placing greater emphasis on capturing more fine-grained layout structures.

## 3 Approach

Given a visually rich document \(D\), a set of question entities \(Q=\{q_{1},q_{2},...,q_{m}\}\) and answers \(A=\{a_{1},a_{2},...,a_{n}\}\), we aim to identify all the connected pairs \((q,a)\) where \(q\in Q\) and \(a\in A\), indicating that \(a\) is the corresponding answer of \(q\). Each \(q_{i}\) or \(a_{j}\) can be denoted as \(\{[w_{0},w_{1},\cdots,w_{t}],(x_{0},y_{0},x_{1},y_{1})\}\), where \([w_{0},w_{1},\cdots,w_{t}]\) is the sequence of words denoting the entity span and \((x_{0},y_{0},x_{1},y_{1})\) is the coordinates for the entity bounding box. Figure 2 illustrates our **RE**2 framework that aims to leverage region-level spatial structures among the question and answer blocks to detect their association.

Footnote 2: We use bold symbols to denote vectors.

### Entity Representation

We first learn the encoding of question and answer entities based on LayoutXLM [45], a layout-aware transformer-based model that has been extended to support multilingualism by pretraining on multilingual VRD datasets.

Given a set of question entities \(Q=\{q_{1},q_{2},...,q_{m}\}\) and answers \(A=\{a_{1},a_{2},...,a_{n}\}\) from document \(D\), we obtain the entity embeddings \(\mathbf{Q}=\{\mathbf{q_{1}},\mathbf{q_{2}},...,\mathbf{q_{m}}\}\), \(\mathbf{A}=\{\mathbf{a_{1}},\mathbf{a_{2}},...,\mathbf{a_{n}}\}\), \(\mathbf{q_{i}},\mathbf{a_{i}}\in\mathbb{R}^{1\times F}\), where \(\mathrm{F}\) is the entity feature dimension3. For entities with multiple tokens, we use the embedding of their first token as their representations6.

Footnote 3: Preliminary experiments showed use of first subtoken performed better than average embedding of all subtokens.

### Region-Aware Graph Construction

Based on the spatial structures of the input VRD, we define three distinct categories of regions (i.e., bounding box) for each entity: (1) an **entity-level bounding box** that refers to the bounding box encompassing the entire entity span and is obtained by merging the bounding boxes of all the words in a span obtained by OCR [28; 45]; (2) a **paragraph-level bounding box** that is defined as a visually distinct section for the paragraph where the entity occurs within a document and corresponds to the clustering of words that are located within a dense region. The paragraph-level bounding boxes are extracted by an existing tool,

Figure 2: Overview of the **RE**gion-level **Relation Extraction (**RE**2) framework. A bipartite graph of Question and Answer entities is constructed. In the eGAT layer, the representation of each entity is updated based on the attention scores of its first-order neighbors.

 

EasyOCR6, which takes the maximum horizontal and vertical distances between adjacent word-level bounding boxes as hyperparameters to merge them into paragraph-level bounding boxes. Other OCR systems include Tesseract [14], Microsoft OCR and other open source OCR systems provided by OpenCV7. Paragraph level bounding boxes can be obtained by clustering word level bounding boxes obtained from any of the OCR systems.; and (3) a **tabular-based bounding box** if the entity occurs in a tabular structure demarcated by lines. We define a tabular-based bounding box as the coordinates of a table cell. Note that each entity can only appear in either a paragraph or a table, so other than its entity-level bounding box, we always assign either a paragraph-level or tabular-based bounding box for each entity, instead of both. Our preliminary results show that a tabular-based bounding box is vital because tabular structures are usually not well-captured by existing OCR tools. Illustrations of the three types of regions are shown in Figure 3. The pseudocode for extracting paragraph/tabular regions is present in Appendix F.

Footnote 6: [https://www.jaided.ai/easyocr/](https://www.jaided.ai/easyocr/)

Footnote 7: [https://opencv.org](https://opencv.org)

To characterize the links between the question and answer entities, we further propose to construct a complete bipartite graph, \(G=(Q,A,E)\), for each visually rich document, where the question entities \(Q=\{q_{1},q_{2},...,q_{m}\}\) and answers \(A=\{a_{1},a_{2},...,a_{n}\}\) are the nodes, and for each pair of \(q_{i}\) and \(a_{j}\), there is an edge \(e_{ij}\in E\) connecting them. Each entity is represented by the encoding learned from LayoutXLM as detailed in Section 3.1, and each edge is represented by a one-hot encoding vector based on the spatial relationship between the three categories of bounding boxes of the question and answer:

\[\bar{\mathbf{e}}_{ij}=[\text{I},\text{E}^{1}_{lr},\text{E}^{1}_{tb},\text{E}^ {0}_{lr},\text{E}^{0}_{tb},\text{R}_{lr},\text{R}_{tb}],\]

where each term is an indicator variable: I indicates whether the two entities are within the same paragraph/tabular region. If so, I \(=1\), otherwise, I \(=0\). When the two entities are from the same paragraph/tabular region, E\({}^{1}_{lr}\) and E\({}^{1}_{tb}\) further indicate the left-right (lr) and top-bottom (tb) spatial relationship of their entity-level bounding boxes. For example, E\({}^{1}_{lr}=1\) indicates that the entity-level bounding boxes of the two entities have a left-right spatial relation, otherwise, E\({}^{1}_{lr}=0\). When the two entities are not from the same paragraph/tabular region, E\({}^{0}_{lr}\) and E\({}^{0}_{tb}\) indicate the left-right and top-bottom spatial relationship of their entity-level bounding boxes, while R\({}_{lr}\) and R\({}_{tb}\) indicate the left-right and top-bottom spatial relationship of their paragraph/tabular level bounding boxes. Note that when the two entities are from the same paragraph/tabular region, the indicators of E\({}^{0}_{lr}\), E\({}^{0}_{tb}\), R\({}_{lr}\), R\({}_{tb}\) will be all zero. A top-bottom relationship is defined based on the relative positions of the x co-ordinates (\([x_{0},y_{0},x_{1},y_{1}]\) for \(q_{i}\) and [\(x_{2},y_{2},x_{3},y_{3}\)] for \(a_{j}\)). Specifically, a top-bottom relationship exists when either \(x_{0}\leq x_{2}\leq x_{1}\), or \(x_{0}\leq x_{3}\leq x_{1}\), or \(x_{2}\leq x_{0}\leq x_{3}\), or \(x_{2}\leq x_{1}\leq x_{3}\). Similarly,

Figure 3: Entity level bounding box (for question and answer entities) are shown in blue, paragraph-level bounding box in red and tabular-based bounding box in green.

 

we define a left-right relationship based on the relative positions of the y coordinates, employing a similar logic. The intuition to determine the spatial relationship is to detect whether there is a vertical/horizontal overlap between region \(q_{i}\) and \(a_{j}\).

To obtain a dense representation of each edge, we pass each one-hot encoding vector \(\bar{\mathbf{e}}_{ij}\) to a feed-forward network, and the resulting vector \(\mathbf{e}_{ij}=\mathrm{FFN}(\bar{\mathbf{e}}_{ij})\) is assigned as the edge weight between \(q_{i}\) and \(a_{j}\), where \(\mathbf{e}_{ij}\in\mathbb{R}^{1\times\mathrm{F}/2}\).

### Edge-aware Graph Attention Network

We further propose an edge-aware graph attention network (eGAT) , extended from the graph attention network (GAT) Velickovic et al. (2018) by incorporating the edge weights inferred by spatial information to learn the interaction between the question and answer nodes. In our experiments, eGAT consists of 2 encoding layers, while each layer updates the node embeddings based on the first-order neighbors with masked self-attention.

Specifically, given the node embeddings at layer \(l\), \(\mathbf{Q}^{l}=\{\mathbf{q}^{I}_{1},\mathbf{q}^{I}_{2},...,\mathbf{q}^{I}_{m}\}\), \(\mathbf{A}=\{\mathbf{a}^{I}_{1},\mathbf{a}^{I}_{2},...,\mathbf{a}^{I}_{n}\}\), we first compute the attention weight between \(q_{i}\) and \(a_{j}\) as follows

\[\begin{split}&\operatorname*{att}(\mathbf{W}^{l}\mathbf{q}^{l}_{i},\mathbf{ W}^{l}\mathbf{a}^{l}_{j})=\mathbf{W}^{\top}_{att}(\mathbf{W}^{l}\mathbf{q}^{l}_{i}||\mathbf{W}^{ l}\mathbf{a}^{l}_{j})\\ &\mathbf{c}(q^{l}_{i},a^{l}_{j})=\text{LeakyReLu}\Big{(}\! \operatorname*{att}\!\left(\operatorname*{att}\!\left(\mathbf{W}^{l}\mathbf{q}^{l}_{ i},\mathbf{W}^{l}\mathbf{a}^{l}_{j}\right)\right)\\ &\mathbf{\alpha}(q^{l}_{i},a^{l}_{j})=\underset{j}{\text{softmax}} \Big{(}\sum(\mathbf{e}^{l}_{ij}\cdot\mathbf{c}(\mathbf{q}^{l}_{i},a^{l}_{j}))\Big{)}\end{split}\]

where \(\cdot\) denotes scalar multiplication. \(\mathbf{W}^{l}\in\mathbb{R}^{F^{\prime}\times F}\) is a parameter matrix for shared linear transformation for \(\mathbf{q}^{l}_{i}\) and \(\mathbf{a}^{l}_{j}\). \(\mathbf{W}_{att}\in\mathbb{R}^{2F^{\prime}}\) is a weight vector for the attention mechanism. \(||\) denotes the catenation operation. \(\mathbf{e}^{l}_{ij}=\mathrm{FFN}^{l}(\mathbf{e}^{l-1}_{ij})\) where \(\mathbf{e}^{0}_{ij}\) is the initial dense representation \(\mathbf{e}_{ij}\) of each edge.

The resulting edge-aware normalized attention scores are then used to update the hidden representations of the question and answer nodes, respectively, with residual connection:

\[\bar{\mathbf{q}}^{l+1}_{i} =\mathbf{q}_{i}+\sum_{j\in\mathcal{N}_{i}}\mathbf{\alpha}(q^{l}_{i},a^{l} _{j})\mathbf{W}\mathbf{a}^{l}_{j}\] \[\bar{\mathbf{a}}^{l+1}_{j} =\mathbf{a}_{j}+\sum_{i\in\mathcal{M}_{j}}\mathbf{\alpha}(a^{l}_{j},q^{l} _{i})\mathbf{W}\mathbf{q}^{l}_{i}\]

where \(\mathcal{N}_{i}\) and \(\mathcal{M}_{j}\) denotes the first order neighbors of \(q_{i}\) and \(a_{j}\) respectively.

For each layer of eGAT, we apply multi-head attention Vaswani et al. (2017), where each attention head performs operations independently, and the mean of all attention heads is taken for aggregation. The updated representation of question node \(q_{i}\) and answer \(a_{j}\) is computed as follows:

\[\mathbf{q}^{l+1}_{i}=\sigma\bigg{(}\frac{1}{K}\sum_{k=1}^{K}\big{(}\mathbf{q}_{i}+\sum _{j\in\mathcal{N}_{i}}\mathbf{\alpha}(q^{l}_{i},a^{l}_{j})^{k}\mathbf{W}^{k}\mathbf{a}^{l}_ {j}\big{)}\bigg{)}\]

\[\mathbf{a}^{l+1}_{j}=\sigma\bigg{(}\frac{1}{K}\sum_{k=1}^{K}\big{(}\mathbf{a}_{j}+\sum _{i\in\mathcal{M}_{j}}\mathbf{\alpha}(a^{l}_{j},q^{l}_{i})^{k}\mathbf{W}^{k}\mathbf{q}^{l}_ {i}\big{)}\bigg{)}\]

where \(K\) is the number of independent attention heads and \(\mathbf{W}^{k}\) denotes the weight matrix for the \(k^{th}\) attention head. \(\sigma(\cdot)\) denotes a non-linear function (ELU is used for experiments). \(\mathbf{q}^{l+1}_{i}\) and \(\mathbf{a}^{l+1}_{j}\) are then used as input node embeddings for layer \(l+1\).

### Relation Extraction

Binary Relation PredictionWe predict a binary label for each question \(q_{i}\) and answer \(a_{j}\) pair, indicating their correspondence. The representations of \(q_{i}\) and \(a_{j}\) include LayoutXLM embedding, final node embedding from eGAT, edge representation of the pair, and an entity type representation (question or answer) learned by an embedding layer. The entity type embedding is crucial for determining the relation direction. The resulting \(q_{i}\) and \(a_{j}\) representations undergo two feed-forward networks and a biaffine classifier Dozat and Manning (2017) to obtain a score \(s_{i,j}\) for determining the association between the pair.

\[\mathbf{q}^{{}^{\prime}}_{i}=\mathrm{FFN}_{\mathrm{a}}(\mathbf{q}_{i}\parallel\mathbf{q}^{ L}_{i}\parallel\mathbf{e}^{L}_{ij}\parallel\mathbf{h}_{q})\]

\[\mathbf{a}^{{}^{\prime}}_{j}=\mathrm{FFN}_{\mathrm{a}}(\mathbf{a}_{j}\parallel\mathbf{a}^{ L}_{j}\parallel\mathbf{e}^{L}_{ij}\parallel\mathbf{h}_{a})\]

\[s_{ij}=\mathbf{q}^{{}^{\prime}}_{i}\mathbf{U}\mathbf{a}^{{}^{\prime}}_{j}+\mathbf{V}\big{(}\mathbf{ q}^{{}^{\prime}}_{i}\circ\mathbf{a}^{{}^{\prime}}_{j}\big{)}+\mathbf{b}\]

where \(\mathbf{h}_{q}\) and \(\mathbf{h}_{a}\) are the type embeddings of question and answer entities. Note that \(\mathbf{h}_{q}\) and \(\mathbf{h}_{a}\) remain the same across all questions and answers, respectively. \(\mathbf{U},\mathbf{V}\) and \(\mathbf{b}\) are trainable parameters. During training, the loss is computed following the cross-entropy loss

\[\mathcal{L}_{b}=-\sum y\cdot\log(p_{ij}).\]

where \(y\in\{0,1\}\) is the target binary label and \(p_{ij}=\textrm{softmax}(s_{ij})\), indicating the probability of a relation between \(q_{i}\) and \(a_{j}\) 

Constraint LossOur preliminary study shows that without any constraint, the model tends to predict multiple questions to be associated with one answer, which is against the definition of relation extraction for VRDs, where each answer is linked to at most one question. To address this issue, we incorporate the constraint into the learning process in the form of a constraint loss. Previous work [10, 29] demonstrated that declarative logical constraints can be converted into differentiable functions, and help regularize the model towards consistency with the logical constraints. We design a declarative logical constraint that holds true for relation extraction task from VRDs as follows, \(\forall{a_{j}}\in A,~{}\forall q_{i}\in Q\),

\[\text{rel}(q_{i},a_{j})\rightarrow\bigwedge_{q_{k}\in\mathbf{Q}\setminus\{q_{ k}\}}\neg\text{ rel}~{}(q_{k},a_{j}).\]

This means, for any \(a_{j}\in A\), if there exists one relation link between \(a_{j}\) and any particular \(q_{i}\) among all questions, there cannot be another relation link for this answer \(a_{j}\). We further define the following constraint loss derived from the logical constraints:

\[\mathcal{L}_{c}=y\cdot\Big{|}\log(p_{ij})-\frac{1}{|Q|-1}\sum_{ \begin{subarray}{c}k=0\\ k\neq i\end{subarray}}^{|Q|}\log\left(1-p_{kj}\right)\Big{|}\]

where \(Q\) denotes the whole set of questions in the document.

Overall Learning ObjectiveThe overall learning objective is a weighted combination of the binary cross entropy loss and the constraint loss:

\[\mathcal{L}=\beta\mathcal{L}_{b}+\delta\mathcal{L}_{c}\]

where \(\beta\) and \(\delta\) are hyperparameters.

## 4 Experiment Settings

### Datasets

The primary challenge in relation extraction from visually rich documents is the diverse layouts in form-like documents across domains and languages. However, **RE\({}^{2}\)** addresses this by introducing domain and language-independent region-level spatial structures. To validate its effectiveness, we conduct experiments on diverse benchmark datasets spanning multiple languages and domains.

FunsdThe FUNSD dataset [35] is derived from the RVL-CDIP dataset [8], featuring scanned document images with OCR ground truth. It includes bounding boxes and annotations for four entity types: _Question_, _Answer_, _Header_, and _Other_. The dataset emphasizes relational links, particularly focusing on Question-Answer links. We follow the data split and experimental settings of prior studies [27, 26], utilizing 149 documents for training and 50 for evaluation, and report the best performance on the evaluation set.

XfundXFUND [27] is a diverse multilingual dataset with visually rich documents in seven languages: Portuguese, Chinese, Spanish, French, Japanese, Italian, and German. Featuring 1,393 fully annotated forms, each language has 149 forms for training and 50 for testing, providing ground truth OCR, entity, and relation annotations. Notably, XFWND shares document format similarities with the FUNSD dataset.

DiverseFormTo best demonstrate the performance of domain transfer of **RE\({}^{2}\)**, we further cre

\begin{table}
\begin{tabular}{l|r|r|r|r|r|r|r|r|r} \hline Model & EN & ZH & JA & ES & FR & IT & DE & PT & Avg. \\ \hline XLM-RoBERTBaSE [20] & 26.59 & 51.05 & 58.00 & 52.95 & 49.65 & 53.05 & 50.41 & 39.82 & 47.69 \\ InfoXLMBase [27] & 29.20 & 52.14 & 60.00 & 55.16 & 49.13 & 52.81 & 52.62 & 41.70 & 49.10 \\ LayoutXLMBASE [27] & 54.83 & 70.73 & 69.63 & 68.96 & 63.53 & 64.15 & 65.51 & 57.18 & 64.32 \\ LiLT[InfoXMLBASE [27] & 62.76 & 72.97 & 70.37 & 71.95 & 69.65 & 70.43 & 65.58 & 58.74 & 67.81 \\ \hline
**RE\({}^{2}\)** (Our Approach) & **71.76** & **79.60** & **75.36** & **75.59** & **76.38** & **77.45** & **75.86** & **59.76** & **73.98** \\ \hline \end{tabular}
\end{table}
Table 1: Language-specific fine-tuning results (F1%) on FUNSD(EN) and XFWND.

\begin{table}
\begin{tabular}{l|r|r|r|r|r|r|r|r|r} \hline Model & EN & ZH & JA & ES & FR & IT & DE & PT & Avg. \\ \hline XLM-RoBERTBaSE [20] & 26.59 & 51.05 & 58.00 & 52.95 & 49.65 & 53.05 & 50.41 & 39.82 & 47.69 \\ InfoXLMBase [27] & 29.20 & 52.14 & 60.00 & 55.16 & 49.13 & 52.81 & 52.62 & 41.70 & 49.10 \\ LayoutXLMBase [27] & 54.83 & 70.73 & 69.63 & 68.96 & 63.53 & 64.15 & 65.51 & 57.18 & 64.32 \\ LiLT[InfoXMLBase [27] & 62.76 & 72.97 & 70.37 & 71.95 & 69.65 & 70.43 & 65.58 & 58.74 & 67.81 \\ \hline
**RE\({}^{2}\)** (Our Approach) & **71.76** & **79.60** & **75.36** & **75.59** & **76.38** & **77.45** & **75.86** & **59.76** & **73.98** \\ \hline \end{tabular}
\end{table}
Table 2: Zero-shot cross-lingual results (F1%) (trained on EN (FUNSD) and tested on other languages) 

ate a new dataset, **DiverseForm**, by curating government forms from Aggarwal et al. (2020) and Sarkar et al. (2020). These forms encompass a wide range of question types, including checkboxes, tables, multiple-choice questions (MCQs), and fill-in-the-blank fields. The domains of the forms cover various areas such as Veterans Affairs, visa applications, tax documents, air transport, legal forms, vehicle-related forms from the Department of Motor Vehicles (DMV), and miscellaneous forms from different government agencies. These forms are of single page and were originally empty and they are designed to collect confidential information such as health data and tax details. To populate the forms, we employed two annotators who used synthetic data generated by The One Generator8 for fields such as names, addresses, and other necessary information. This approach ensures the privacy and security of individuals' personal information while providing a realistic representation of the data typically found in these government forms. We then hire another annotator to label the _Question_ and _Answer_ entities as well as their relations for these documents using the annotation tool UBIAI9, which also offers its customized OCR model for extracting text from uploaded images. However, due to the serialized top-left to bottom-right text extraction approach of the OCR, the spans of entities are sometimes fragmented in complex layout forms. During the annotation process, these fragmented spans are identified and merged to achieve the correct serialization of spans. After labeling the entities and relations for these documents, we further hire three annotators to validate the annotations. All the annotators are senior undergraduate students majoring in Computer Science and are paid a rate of \(\$15/hour\). We name the final annotated dataset as **DiverseForm**, which comprises a total of 150 training documents and 50 testing documents. Details of **DiverseForm** annotation and statistics is in Appendix B.

Footnote 8: [https://thoengenerator.com/](https://thoengenerator.com/)

Footnote 9: [https://ubiai.tools/](https://ubiai.tools/)

## 5 Experiment Setting and Hyperparameters

The NVIDIA A40 GPU was utilized for all fine-tuning tasks. Paragraph-level regions are created using EasyOCR through horizontal merging of text boxes when their distance is within 2, and vertical merging is performed when the distance is within 1, with the paragraph flag set to True. The model undergoes end-to-end training, incorporating finetuning of the LayoutXLM base model. The eGAT layers and relation extraction head are trained from scratch, employing 2 eGAT layers for all experiments. The training process consists of 5000 steps with a batch size of 4, a learning rate of 5e-5, and a warm-up ratio of 0.1. Cross-entropy loss is weighted at 1, and constraint loss is weighted at 0.02.

### Inference Details

During the inference phase, the input comprises head entities, tail entities, bounding boxes (acquired from OCR), and the document image. This input undergoes a modeling process similar to the training phase, wherein additional processing is applied to derive entity-level, paragraph-level, and tabular-level bounding boxes. Subsequently, these bounding boxes are normalized to establish a relative spatial representation of entities, facilitating relation extraction tasks.

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline \hline Model & **DiverseForm** & FUNSD \(\rightarrow\)**DiverseForm** & **DiverseForm**\(\rightarrow\)**FUNSD** \\ \hline LayoutXLMBASE & 69.72 & 37.33 & 32.58 \\ LiLT[InfoXML]BASE & 64.15 & 41.56 & 30.26 \\
**RE\({}^{2}\)** & **70.87** & **41.78** & **50.32** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Supervised results on **DiverseForm** and cross-domain transfer results between **DiverseForm** and **FUNSD**. (F1%)

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c|c|c} \hline \hline Model & EN & ZH & JA & ES & FR & IT & DE & PT & Avg. \\ \hline XLM-RoBERTBABSE & 36.38 & 67.97 & 68.29 & 68.28 & 67.27 & 69.37 & 68.87 & 60.82 & 63.41 \\ InfoXLMBASE & 36.99 & 64.93 & 64.73 & 68.28 & 68.31 & 66.90 & 63.84 & 57.63 & 61.45 \\ LayoutXLMBASE & 66.71 & 82.41 & 81.42 & 81.04 & 82.21 & 83.10 & 78.54 & 70.44 & 78.23 \\ LiLT[InfoxLM]BASE & 74.07 & 84.71 & **83.45** & **83.35** & 84.66 & **84.58** & 78.78 & **76.43** & 81.25 \\ \hline
**RE\({}^{2}\)** (Our Approach) & **74.11** & **88.25** & 82.27 & 83.23 & **86.83** & 84.02 & **81.89** & 71.04 & **81.46** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Multitask fine-tuning performance (F1%) on FUNSD(EN) and XFUND.

 



### Experiment Results

**Language-specific fine-tuning** results are presented in Table 1, where each model is fine-tuned on language X and tested on language X. The experimental findings show that the proposed model outperforms all the baselines across all evaluated languages. To evaluate the **cross-lingual zero-shot transfer** capability, the model is fine-tuned on the FUNSD dataset in English, followed by testing on multiple languages. The experimental results, as shown in Table 2, demonstrate the superiority of our model over the baseline approach in terms of zero-shot performance. This outcome provides compelling evidence that the incorporated region-level spatial structures and constraints for relation extraction exhibit effective transferability across different languages. We also conduct a significance test for both our approach and the best-performing baseline (i.e., LiLT[InfoXLM]BASE Wang et al. (2022)) under the settings of langauge-specific fine-tuning and cross-lingual zero-shot transfer. As shown in Table 7 in Appendix C, our approach significantly outperforms the baseline under both settings.

Table 3 displays the results of **multitask finetuning**, where the model is trained on all language training sets and tested on each individual language. The superior performance showcases the model's successful learning of layout invariance across languages. By capturing shared layout characteristics, the model demonstrates improved generalization, enhancing performance across diverse linguistic contexts. This emphasizes the importance of incorporating layout information in cross-lingual settings and underscores the model's adaptability and knowledge transfer for effective document processing across various languages.

Note that \({\bf RE}^{2}\) shows less competitive performance on Portuguese (PT) due to more complex layout structures. Portuguese forms exhibit a combination of mixed tables and paragraph structures, making it challenging to determine the appropriate usage for paragraph-level regions or tabular regions. An example is shown in Appendix D.

We also assess the generalization of \({\bf RE}^{2}\) and two high-performing baselines based on **DiverseForm** and FUNSD, which cover two sets of distinct domains. We conduct experiments under the settings of both domain-specific fine-tuning and cross-domain transfer where the models are trained on one dataset and tested on the other. As shown in Table 4, \({\bf RE}^{2}\) significantly outperforms the two strong baselines when fine-tuned on **DiverseForm** and tested on **DiverseForm** or FUNSD. The improvement of \({\bf RE}^{2}\) when it's trained on FUNSD and tested on **DiverseForm** is marginal, probably due to the greater diversity and complexity in document layout of **DiverseForm** compared to FUNSD.

### Ablation Study

Effect of Node and Edge Embeddings from eGATThe node and edge embeddings from eGAT are concatenated with the entity representations before being passed to the biaffine classifier. A series of ablation studies are conducted to assess the individual contributions of the layout information. The results of these studies are presented in Table 5. Figure 6 in Appendix E provides visual evidence that solely relying on the updated node embeddings from eGAT fails to adequately capture the layout heuristics and results in the omission of numerous relations. Conversely, employing only the updated edge embeddings without considering the node embeddings leads to an over-prediction of relations with limited regard for the semantic relevance of the entities involved. Optimal performance is achieved through the joint utilization of both node and edge embeddings, indicating the importance of integrating both sources of information to effectively capture the region-level spatial structures and consider the semantic context of the

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c|c} \hline Model & EN & ZH & JA & ES & FR & IT & DE & PT & Avg. \\ \hline \({\bf RE}^{2}\) & **71.76** & **79.60** & **75.36** & **75.59** & **76.38** & **77.45** & **75.86** & **59.76** & **73.98** \\ \hline - node embedding & 70.19 & 78.93 & 75.00 & 74.60 & 76.00 & 76.82 & 73.20 & 57.29 & 72.75 \\ - edge embedding & 57.42 & 69.37 & 67.93 & 72.01 & 73.73 & 69.67 & 63.48 & 55.61 & 66.15 \\ \hline - constraint loss & 68.52 & 77.77 & 74.49 & 74.78 & 75.20 & 75.66 & 73.61 & 57.48 & 72.19 \\ \hline - entity level regions & 44.69 & 76.89 & 66.71 & 73.11 & 62.44 & 70.63 & 62.10 & 44.30 & 62.61 \\ - paragraph/tabular regions & 71.57 & 79.5 & 74.17 & 72.05 & 74.98 & 76.79 & 74.55 & 57.49 & 72.64 \\ \hline \end{tabular}
\end{table}
Table 5: Ablation study results (F1%) on eGAT (node and edge embeddings), constraint loss, paragraph/tabular regions and entity level regions.



relations.

Effect of Constraint LossThe constraint loss has been modeled to encourage each answer entity to be linked to at most one question. Table 5 shows that incorporating the constraint loss significantly improves the F1 score of \({\bf RE}^{2}\), especially precision. The detailed experimental results are evidenced in Appendix G.

Effect of Region InformationWe also investigate the impact of each category of regions on characterizing the spatial relationship among the entities and further affecting the performance of \({\bf RE}^{2}\). As shown in Table 5, the inclusion of each category of region information significantly improves the performance of \({\bf RE}^{2}\). The absence of entity-level regions resulted in a substantial decrease in performance, underscoring the vital role of pairwise entity layout information, i.e., whether the question and answer entities are arranged vertically (top-bottom) or horizontally (left-right). Figure 7 in Appendix E shows an example to compare the relation predictions with and without paragraph/tabular regions, indicating that incorporating paragraph/tabular regions helps prevent the model from predicting relations across semantically different regions. The result of this ablation study proves the effectiveness of the multi-granular region information.

## 6 Conclusion

In this work, we propose a novel entity relation extraction model, \({\bf RE}^{2}\), that incorporates layout heuristics and constraints that are generalizable across different languages. Experimental results on 8 different languages and our proposed dataset DiverseForm show the effectiveness of our proposed method under four settings (language-specific, cross-lingual zero-shot, multi-lingual fine-tuning, and cross-domain transfer).

### Limitations

In this work, we found the incorporation of layout heuristics to be compelling and we are excited by how leveraging region information improves performance drastically. One of the limitations of our model is its reliance on a relatively limited set of heuristics and features. For instance, we have not yet incorporated visual information and template-based knowledge, which could potentially improve the accuracy and robustness of the relation extraction task. Additionally, the current model employs an exhaustive inference approach, considering all possible relations during prediction. While this ensures comprehensive coverage, it also results in longer inference times for each relation type. These limitations indicate avenues for further improvement, such as exploring additional heuristics and incorporating more efficient inference strategies, to enhance the performance and efficiency of our model.

### Ethical Considerations

The forms in the DiverseForm dataset are synthetically constructed and should not be mistaken for real forms. The values within these forms are populated through random generation, adhering to patterns that reflect typical data; however, these entries are not genuine. By employing synthetic data, we ensure that the model is trained on data closely resembling real-world scenarios without compromising the privacy and security of actual individuals. This approach is in line with ethical guidelines that prioritize data protection and privacy rights, making it a responsible choice for developing models that handle sensitive information. The proposed model is designed to enhance understanding of various document layouts, including checkboxes, tables, and fill-in-the-blank fields--areas often overlooked in previous studies. Its potential misuse is dependent on unauthorized access to genuine information.

#### Acknowledgments

This research is partially supported by a research award from Intuit AI Research and award No. 2238940 from the Faculty Early Career Development Program (CAREER) of the National Science Foundation (NSF). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

## References

* Aggarwal et al. [2020] Milan Aggarwal, Mausoom Sarkar, Hiresh Gupta, and Balaji Krishnamurthy. 2020. Multi-modal association based grouping for form structure extraction.

 

In _The IEEE Winter Conference on Applications of Computer Vision_, pages 2075-2084.
* Chen et al. (2022) Muhao Chen, Lifu Huang, Manling Li, Ben Zhou, Heng Ji, and Dan Roth. 2022. New frontiers of information extraction. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorials_.
* Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In _ECCV_.
* Chi et al. (2021) Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3576-3588, Online. Association for Computational Linguistics.
* Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Dozat and Manning (2017) Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing.
* Grishman (1997) Ralph Grishman. 1997. Information extraction: Techniques and challenges. In _Information Extraction A Multidisciplinary Approach to an Emerging Information Technology: International Summer School, SCIE-97 Frascati, Italy, July 14-18, 1997_, pages 10-27. Springer.
* Guo et al. (2019) Zhijiang Guo, Yan Zhang, and Wei Lu. 2019. Attention guided graph convolutional networks for relation extraction. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 241-251, Florence, Italy. Association for Computational Linguistics.
* Harley et al. (2015) Adam W. Harley, Alex Ufkes, and Konstantinos G. Derpanis. 2015. Evaluation of deep convolutional nets for document image classification and retrieval.
* Huang et al. (2022) Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutlmv3: Pre-training for document ai with unified text and image masking. In _Proceedings of the 30th ACM International Conference on Multimedia_, MM '22, page 4083-4091, New York, NY, USA. Association for Computing Machinery.
* Jaume et al. (2019) Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. 2019. Funsd: A dataset for form understanding in noisy scanned documents.
* Kay (2007) Anthony Kay. 2007. Tessercat: an open-source optical character recognition engine. _Linux J._, 2007(159):2.
* Lee et al. (2022) Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nana Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, and Tomas Pfister. 2022. Format: Structural encoding beyond sequential modeling in form document information extraction.
* Lee et al. (2023) Chen-Yu Lee, Chun-Liang Li, Hao Zhang, Timothy Dozat, Vincent Perot, Guolong Su, Xiang Zhang, Kihyuk Sohn, Nikolai Glushnev, Renshen Wang, Joshua Ainslie, Shangbang Long, Siyang Qin, Yasuhisa Fujii, Nan Hua, and Tomas Pfister. 2023. Formnetv2: Multimodal graph contrastive learning for form document information extraction.
* Li et al. (2019) Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. 2019. A logic-driven framework for consistency of neural models. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3924-3935, Hong Kong, China. Association for Computational Linguistics.
* Li et al. (2022) Xin Li, Yan Zheng, Yiqing Hu, Haoyu Cao, Yunfei Wu, Deqiang Jiang, Yinsong Liu, and Bo Ren. 2022. Relational representation learning in visually-rich documents.
* Liu et al. (2019) Xiaojing Liu, Feiyu Gao, Qiong Zhang, and Huasha Zhao. 2019. Graph convolution for multimodal information extraction from visually rich documents. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers)_, pages 32-39. Association for Computational Linguistics.
* Lu et al. (2019)Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. _ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks_. Curran Associates Inc., Red Hook, NY, USA.
* Luo et al. (2023) Chuwei Luo, Changxu Cheng, Qi Zheng, and Cong Yao. 2023. Geolayoutlm: Geometric pre-training for visual information extraction.
* Powalski et al. (2021) Rafal Powalski, Lukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michal Pietruszka, and Gabriela Palka. 2021. Going full-tilt boogie on document understanding with text-image-layout transformer. In _Document Analysis and Recognition-ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5-10, 2021, Proceedings, Part II 16_, pages 732-747. Springer.
* Qian et al. (2019) Yujie Qian, Enrico Santus, Zhijing Jin, Jiang Guo, and Regina Barzilay. 2019. GraphLE: A graph-based 

framework for information extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 751-761. External Links: Link Cited by: SS4.
* M. Sarkar, M. Aggarwal, A. Jain, H. Gupta, and B. Krishnamurthy (2020)Document structure extraction using prior based high resolution hierarchical semantic segmentation. In European Conference on Computer Vision, pp. 649-666. Cited by: SS3.
* W. Su, X. Zhu, Y. Cao, B. Li, Lewei Lu, F. Wei, and J. Dai (2020)VL-bert: pre-training of generic visual-linguistic representations. In International Conference on Learning Representations, Cited by: SS4.
* A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017)Attention is all you need. Cited by: SS4.
* P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio (2018)Graph attention networks. External Links: 1803.0221 Cited by: SS4.
* H. Wang, M. Chen, H. Zhang, and D. Roth (2020)Joint constrained learning for event-event relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 696-706. External Links: Link, Document Cited by: SS4.
* J. Wang, L. Jin, and K. Ding (2022a)Lilt: a simple yet effective language-independent layout transformer for structured document understanding. External Links: 2205.02215 Cited by: SS4.
* W. Wang, Z. Huang, B. Luo, Q. Chen, Q. Peng, Y. Pan, W. Yin, S. Feng, Y. Sun, D. Yu, and Y. Zhang (2022b)Ernie-mmlayout: multi-grained multimodal transformer for document understanding. External Links: 2206.02215 Cited by: SS4.
* Y. Xu, Y. Xu, T. Lv, E. Cui, F. Wei, G. Wang, Y. Lu, D. Florencio, C. Zhang, W. Che, M. Zhang, and L. Zhou (2022c)Layoutlmv2: multi-modal pre-training for visually-rich document understanding. External Links: 2206.02215 Cited by: SS4.
* Y. Xu, M. Li, L. Cui, S. Huang, F. Wei, and M. Zhou (2020)Layoutlm: pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery; Data Mining, KDD '20, New York, NY, USA, pp. 1192-1200. External Links: ISBN 978-1-4503-3213-X, Link, Document Cited by: SS4.
* Y. Xu, T. Lv, L. Cui, G. Wang, Y. Lu, D. Florenceio, C. Zhang, and F. Wei (2022d)XFUND: a benchmark dataset for multilingual visually rich form understanding. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, pp. 3214-3224. External Links: Link, Document Cited by: SS4.
* W. Yu, N. Lu, X. Qi, P. Gong, and R. Xiao (2021)Pick: processing key information extraction from documents using improved graph learning-convolutional networks. In 2020 25th International Conference on Pattern Recognition (ICPR), Los Alamitos, CA, USA, pp. 4363-4370. External Links: Link, Document Cited by: SS4.
* W. Yu, N. Lu, X. Qi, P. Gong, and R. Xiao (2020)PICK: processing key information extraction from documents using improved graph learning-convolutional networks. arXiv preprint arXiv:2004.07464. Cited by: SS4.
* S. Zhang, Q. Ning, and L. Huang (2022a)Extracting temporal event relation with syntax-guided graph transformer. In Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, United States, pp. 379-390. External Links: Link, Document Cited by: SS4.
* Y. Zhang, B. Zhang, R. Wang, J. Cao, C. Li, and Z. Bao (2021)Entity relation extraction as dependency parsing in visually rich documents. External Links: 2106.02215 Cited by: SS4.
* Y. Zhang, P. Qi, and C. D. Manning (2018)Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 2205-2215. External Links: Link, Document Cited by: SS4.
* Z. Zhang, J. Ma, J. Du, L. Wang, and J. Zhang (2022b)Multimodal pre-training based on graph attention network for document understanding. External Links: 2206.02215 Cited by: SS4.
* H. Zhu, Y. Lin, Z. Liu, J. Fu, T. Chua, and M. Sun (2019)Graph neural networks with generated parameters for relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 1331-1339. External Links: Link, Document Cited by: SS4.

## Appendix A Data Preprocessing

To accurately determine the layout heuristics, it is important to get the bounding box of the entire entity span. If token-level bounding boxes are provided, the boxes can be merged to obtain a span-level box. All the paragraph/tabular regions are detected and their bounding boxes are obtained. We identify the region an entity belongs to by checking the entity's Intersection over Union (IoU) with the regions and assign the region with the maximum IoU 



## Appendix B Annotation Details and Statistics of DiverseForm

The guidelines of annotating entities are as follows:

* Question: A word, set of words, or sentence worded or expressed so as to elicit information from the person filling the form.
* Questions are annotated even if they haven't been answered
* Questions and sub-questions are labeled as the same type of entity.
* Header: A word, set of words, or sentences worded or expressed so as give context or encapsulate a set of questions.
* Annotate headers even if their questions haven't been answered
* Headers do not have answers directly attached to them
* Answer: A word, set of words, or sentence written in response to a question.
* Responses in the form of checkbox options count as answers.
* In multiple choice type questions, all the options are annotated as answers (following FUNSD and XFUND)

The guidelines of annotating relations are as follows:

* Question-Answer: A link exists between a question entity and an answering entity when the answer is a response to a particular question.
* When multiple answers exist for a question, there are multiple Question-Answer links from the same question entity.
* Answers to a sub-question should only be linked to the sub-question and not the parent question.
* Question-Question: A link exists between a question entity and another question entity if one question is a sub-question of another question or one question is conditioned on the answer of another question.
* For example, "If yes, ..." type of question has a Question-Question link with the parent question.
* A question that is split into multiple fine-grained questions has a Question-Question link between them. For example, "Address" can have further questions such as "Apt. No", "Street Name", "City", "State", "Zip Code".
* Header-Question: A link exists between a header entity and a question entity if the questions are present under the section or subsection that is characterized by the header.
* If multiple questions exist under a header, there are multiple Header-Question links from the same header entity.
* Often confused with Question-Question links and can be differentiated based on layout structure, font style, and other visual aspects of the questions from the form.

The guidelines of annotation of tables are as follows: We mainly deal with one dimensional tables. For the case that each cell in the table is related to both row and column questions, there will be a Question-Question link between the questions extracted from the row and column, indicating that one question is a sub-question of another question or one question is conditioned on the answer of another question. This is part of the annotation guidelines for FUNSD and our own dataset. Based on these annotation rules, the constraint of one answer having one question still holds.

Figure 4 shows the distribution of domains in **D1-verseForm**. Miscellaneous consists of forms for voter registration, agriculture, scholarship, immigration, property tax, etc. Veteran's Affairs encompasses varying forms ranging from child support payments to retirement funds. There is rich layout variation within each domain shown in the chart. The number of entities and relations of each type in **DiverseForm** are tabulated in Table 6.

## Appendix C Significance Test

Table 7 shows the significance test results for both our approach and the best performing baseline (i.e., LiLT[InfoXLM]BASE Wang et al. (2022a)) under the settings of language-specific fine-tuning and cross-lingual zero-shot transfer. The results for all experiments reported were averaged across 3 runs



## Appendix D Case Study

Figure 5 visualizes paragraph-level regions, tabular regions, and predictions for a Portuguese form in FUNSD. It shows that paragraph-level regions are suitable for the top portion of the form, while tabular regions specifically pertain to the bottom table. In this particular form, the decision was made to adopt paragraph-level regions, resulting in the exclusion of the tabular layout despite its ability to convey more information. We acknowledge that there are instances where our proposed approach may struggle to accurately distinguish between paragraph-level and tabular regions, leading to a performance decrease.

## Appendix E Visualizations of Ablation Results

Figure 6 shows the visualization of predictions of the ablation study of node and edge embeddings. Figure 7 shows the visualization of predictions of the ablation study of incorporating paragraph/tabular regions.

## Appendix F Pseudocode

The following pseudocode extracts the tabular and paragraph-level regions from a VRD.

## Appendix G Ablation Results of Constraint Loss

The constraint loss has been modeled to encourage each answer entity to be linked to at most one question. Table 8 shows that incorporating the constraint loss significantly improves the F1 score of \({\bf RE}^{2}\), especially precision.



```
1:Apply horizontal kernel to the image
2:Apply vertical kernel to the image
3:Find horizontal lines
4:Find vertical lines
5:returnCombined horizontal and vertical lines
```

**Algorithm 1** IdentifyHorizontalAndVerticalLines(image)

```
1:TabularBoxList = []
2:Find contours in lines
3:for each contour do
4: Compute the bounding box
5: Append the box to TabularBoxList
6:endfor
7:returnTabularBoxList
```

**Algorithm 2** FindBoundingBoxes(lines)

Figure 5: Visualization of paragraph-level regions (a), tabular regions (b) and predictions (c) for a Portuguese form in XFUND.

 

```
1:FinalBoxList = []
2:for each box in boundingBoxes do
3:if the box contains any text and has no intersection with existing boxes in FinalBoxList then
4: Append the box to FinalBoxList
5:endif
6:endfor
7:return FinalBoxList
```

**Algorithm 5** CheckAllTextPresent(FinalBoxList, text)

Figure 6: Visualization of predictions of the ablation study of node and edge embeddings, where red lines denote the question span, green lines denote the answer span, and blue lines denote the question answer relation predictions.

 

```
1:missingText = []
2:if the text is not present inside the bounding boxes of any of the FinalBoxList then
3: Append text to missingText
4:endif if
5:return missingText
```

**Algorithm 6** GetMissingText(FinalBoxList, text)

Figure 7: Visualization of predictions of the ablation study of incorporating paragraph/tabular regions.

 



\begin{table}
\begin{tabular}{l|c|c|c|c|c|c} \hline \hline  & \multicolumn{3}{c|}{**RE\({}^{2}\)**} & \multicolumn{3}{c|}{Baseline} & \\ \hline Setting & Mean & SD & Mean & SD & P-value \\ \hline Language-Specific Fine-Tuning & 73.98 & 6.15 & 67.81 & 4.98 & 0.0447 \\ Zero-Shot Cross-Lingual & 62.08 & 8.53 & 49.30 & 6.55 & 0.0047 \\ \hline \end{tabular}
\end{table}
Table 7: Significance Test Results

\begin{table}
\begin{tabular}{r c|c|c|c|c|c|c|c|c|c|c|c} \hline \hline \multirow{2}{*}{Model} & \multicolumn{3}{c|}{EN} & \multicolumn{3}{c|}{ZH} & \multicolumn{3}{c|}{JA} & \multicolumn{3}{c}{ES} \\ \cline{2-13}  & P & R & F1 & P & R & F1 & P & R & F1 & P & R & F1 \\ \hline   **RE\({}^{2}\)** & 69.71 & 73.74 & 71.67 & 76.80 & 82.62 & 79.60 & 70.16 & 81.39 & 75.36 & 70.26 & 81.78 & 75.59 \\ \hline
**RE\({}^{2}\)**- constraint loss & 58.76 & 82.16 & 68.52 & 74.77 & 81.01 & 77.77 & 69.13 & 80.75 & 74.49 & 69.41 & 81.05 & 74.78 \\ \hline Model & \multicolumn{3}{c|}{FR} & \multicolumn{3}{c|}{IT} & \multicolumn{3}{c|}{DE} & \multicolumn{3}{c}{PT} \\ \cline{2-13}  & P & R & F1 & P & R & F1 & P & R & F1 & P & R & F1 \\ \hline
**RE\({}^{2}\)** & 70.05 & 83.97 & 76.38 & 74.34 & 80.83 & 77.45 & 73.01 & 78.94 & 75.86 & 48.06 & 78.99 & 59.76 \\ \hline
**RE\({}^{2}\)**-constraint loss & 71.54 & 80.57 & 75.79 & 72.32 & 79.32 & 75.66 & 71.74 & 75.59 & 73.61 & 46.98 & 74.02 & 57.48 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Precision, Recall and F1 score of ablation study of Constraint Loss on **RE\({}^{2}\)**