Hypergraph based Understanding for Document Semantic Entity Recognition (2024)

Abstract: Semantic entity recognition is an important task in the field of visually-rich document understanding. It distinguishes the semantic types of text by analyzing the position relationship between text nodes and the relation between text content. The existing document understanding models mainly focus on entity categories while ignoring the extraction of entity boundaries. We build a novel hypergraph attention document semantic entity recognition framework, HGA, which uses hypergraph attention to focus on entity boundaries and entity categories at the same time. It can conduct a more detailed analysis of the document text representation analyzed by the upstream model and achieves a better performance of semantic information. We apply this method on the basis of GraphLayoutLM to construct a new semantic entity recognition model HGALayoutLM. Our experiment results on FUNSD, CORD, XFUND and SROIE show that our method can effectively improve the performance of semantic entity recognition tasks based on the original model. The results of HGALayoutLM on FUNSD and XFUND reach the new state-ofthe-art results.

heading: 1 Introduction
text: With the development of information technology, documents have become a main information carrier nowadays, which contains kinds of information type, such as text, table and image. Manual recognition of these documents often requires plenty of manpower. OCR tools can only help us to identify the text, layout and other simple information in the document. To further understand
*Corresponding author. †Equal contribution. This work was supported by the National Natural Science Foundation of China (No. 62306216, No. 72074171, No. 72374161), the Natural Science Foundation of Hubei Province of China (No. 2023AFB816), the Fundamental Research Funds for the Central Universities (No. 2042023kf0133).
documents, Visually-rich Document Understanding (VRDU) (Xu et al., 2020b) is proposed to make use of visual, textual and other information for more in-depth analysis.
Semantic Entity Recognition (SER) is an important task in the field of VRDU. Its purpose is to extract and classify the text with special semantic information in documents. Different from text sequences in traditional natural language processing tasks, the information in documents is not onedimensional, single-modal and continuous, but twodimensional, multimodal and discrete. It is necessary to analyze not only text information, but also other modal information such as layout and vision in the document. Figure 1 shows the difference between the traditional named entity recognition (NER) task on a single modal text and the semantic entity recognition task on a document. Firstly, the text form of a single modal text task is a fixed text sequence, while the discrete text in a document is composed of text nodes in different locations. Secondly, the named entity recognition task of a single modal text only needs to consider the semantic relationship between the tokens in the text sequence. However, the semantic entity recognition task on the document needs to consider not only the semantic relationship between nodes, but also the position relationship between nodes. Finally, the span range of entity tags of NER task is flexible, while the range of task tags of semantic entity recognition task on document is affected by nodes. Texts of the same node in the document share the same label in most cases.
With the development of pre-training technology, document pre-training model has become popular. LayoutLM (Xu et al., 2020b) is the first multi-modal pre-trained model to associate text with layout and vision, achieving leading results on multiple downstream document understanding tasks including semantic entity recognition. Subsequently, more multi-mode pretraining models, such
2950
as LayoutLMv2 (Xu et al., 2020a), BROS (Hong et al., 2022), ERNIE-Layout (Peng et al., 2022) and LayoutLMv3 (Huang et al., 2022) have been proposed successively. By integrating text, layout and visual information, they realize the understanding and information extraction of documents. So far, GraphLayoutLM (Li et al., 2023a) and GeoLayoutLM (Luo et al., 2023) have the best performance in semantic entity recognition tasks. GraphLayoutLM achieves the best F1 score of 94.39 and 93.56 on the FUNSD (Jaume et al., 2019) and XFUND (Xu et al., 2021) datasets. GeoLayoutLM achieves the best F1 score of 97.97 on the CORD (Park et al., 2019) datasets. However, these existing methods focus on the upstream document understanding part and pay little attention to the downstream task. GeoLayoutLM has studied the novel relational extraction head and achieves great improvement in the relational extraction task. But it has not done more research on the semantic entity recognition task. We study the problem of ignoring the downstream head and classification method in the semantic entity recognition task in the existing document intelligence work and propose a novel improvement scheme.
Traditional Semantic Entity Recognition. The traditional document semantic entity recognition task process is shown in (a) of the Figure 2. In document understanding process, text nodes are spliced into text sequences and become text token sequences of documents after tokenization. These text nodes will be transformed to the highdimensional feature representations after the analysis of the document understanding model. To extract semantic information from document token features, linear layer or multilayer perceptron (MLP) will be used to convert high-dimensional features into label probabilities and the training objective is cross entropy loss. Although this method can distinguish the node categories in the document, it ignores the characteristics of the document structure and it is difficult to make the classification layer pay attention to the node span.
Hypergraph Semantic Entity Recognition. Inspired by Global Pointer (Su et al., 2022), we use the idea of hypergraph to extract the semantic information of documents and propose a Hypergraph Attention(HGA) strategy for document semantic entity recognition. (b) of the Figure 2 shows us the process of hypergraph semantic recognition. Different from the traditional classification method, the semantic entity recognition idea of HGA regard the document token features as graph nodes. The target entity is the set of nodes with the same hyperedge and the hyperedge type represents the entity label type. The process of hypergraph extraction is to establish hyperedges between token feature nodes. Besides, we use the span hyperedge encoding to add the span information of text nodes. Through the hypergraph and span position, the head can better focus on the entity boundary information and establish the relationship between the document discrete text span and the entity boundary.
Our main contributions are as follows:
• We construct a novel hypergraph attention document semantic entity recognition method, HGA. It transforms the traditional token sequence classification problem into a hypergraph construction process. By establishing different types of hyperedges between text nodes, the head can extract semantic entities.
• We propose a novel span hyperedge position encoding and balanced hyperedge loss. Span hyperedge position encoding makes the head focus more on the same text span prompt during hyperedge construction. Balanced hyperedge loss can help to solve the problem of matrix sparsity caused by too many hyperedge types in some scenarios.
• We construct a novel document semantic entity recognition model HGALayoutLM based on the HGA method. Our code will be available at https://github.com/LineKite/HGALayoutLM. The experiment results show that the model has good performance in the scene with few types of labels. HGALayoutLM has obtained the best results on the FUNSD, SROIE and XFUND datasets.

heading: 2 Related Work
text: In recent years, self-supervised pre-training technology has become the mainstream trend in the fields of natural language processing (NLP) and
computer vision (CV). BERT (Devlin et al., 2018) is a classic pre-training model that has shown great effectiveness in various tasks such as question answering, natural language generation and text classification. Masked Language Modeling (MLM) is a significant pre-training task proposed by BERT that enables models to learn textual representations by predicting the raw vocabulary ids of randomly masked word markers based on context. Since then, a series of mask language models such as RoBERTa (Liu et al., 2019), ALBERT (Lan et al., 2019) and XLNet (Yang et al., 2019) have been proposed successively. These models achieve good results on natural language understanding tasks.
However, the single modal language model (Lan et al., 2019; Liu et al., 2019; Peng et al., 2023; Li et al., 2023b) can not understand documents with complex formats and diverse types well. To fully understand the content of complex documents, LayoutLM (Xu et al., 2020b) adds layout and document information on the basis of BERT to supplement the document format missing from plain text. Following LayoutLM, BROS (Hong et al., 2022), LayoutLMv2 (Xu et al., 2020a), XYLayoutLM (Gu et al., 2022), ERNIE-Layout (Peng et al., 2022), LayoutLMv3 (Huang et al., 2022) and other multimodal pre-training document understanding models have been proposed successively and constantly make breakthroughs in various tasks in the field of document understanding. These models understand the document through the fusion of text, layout and vision information. Since document nodes are suitable to be represented by graph structures, some works begin to apply graph structures to document understanding models, such as ERNIE-
mmLayout (Wang et al., 2022), ROPE (Lee et al., 2021), FormNet (Lee et al., 2022) and GraphLayoutLM (Li et al., 2023a).
The latest GraphLayoutLM and GeoLayoutLM (Luo et al., 2023) are both built on the basis of LayoutLMv3. They have achieved the most excellent results in several tasks of document information extraction. Inspired by some graph based works (Kipf and Welling, 2016; Velickovic et al., 2017; Wang et al., 2023), GraphLayoutLM models the document structure based on the hierarchical and positional layout of the document and represents the document layout modeling with a graph structure. To integrate graph structure information into the process of document understanding, GraphLayoutLM proposes graph reordering and graph masking strategies, adding graph information into the document understanding model in the form of sequence and self-attention mask. GeoLayoutLM implements geometric pre-training to enrich and enhance feature representation through three specially designed geometry-related pre-training tasks. In addition, GeoLayoutLM uses a novel relation head in the fine-tuning phase and obtains a big improvement over LayoutLMv3 in the relation extraction task. At present, little attention is paid to the effects of downstream task heads on the performance of various types of tasks. GeoLayoutLM proposes a novel relational head, but there is still a lack of research on the downstream task of semantic entity recognition in the field of document understanding. Most of the current models use a linear layer and cross-entropy to predict BIO label probabilities when dealing with semantic entity recognition tasks, such as LayoutLM, BROS, Lay-
outLMv2, etc. LayoutLMv3 and its derived models utilize a linear layer in the few label case and employ MLP when number of label types is large. These approaches are fundamentally the same. Differently, UDop (Tang et al., 2023) is a new unified document intelligent framework, which adopts encoder-decoder structure. In addition, with the development of large language models (LLMs), some works (Hong et al., 2023; Fujitake, 2024; Luo et al., 2024) have begun to apply large model technology to document intelligence. However, the decoder will cost a large computational cost. Taking inspiration from Global Pointer (Su et al., 2022), we design a simple hypergraph head that incorporates document span information to achieve better SER task performance.

heading: 3 Methodology

heading: 3.1 Overview
text: The process of semantic entity recognition based on Hypergraph Attention is shown in Figure 3. Different from traditional semantic entity recognition methods, HGA focuses on extracting special entities. Instead of using BIO annotation method as the label annotation strategy for model input, we regard each special label (such as header, question and answer) as a label type. The entity without special meaning represented by the Other label in
the BIO annotation will not be labeled as a hyperedge type in the hypergraph construction process. HGA regards token features as unit nodes and the process of establishing hyperedges between tokens can realize the extraction of special entities. It is worth noting that the node referred to here correspond to each token of token sequence. Text nodes, as mentioned earlier, are discrete pieces of text at different locations in the document. A text node corresponds to one or more token feature nodes. The process of hyperedge extraction can realize the extraction of special semantic entites and classification of different entity labels. Entities that do not have the meaning of a special label (that is, the entities of the Other label in the BIO annotation) will not have the connection of any special hyperedge.
To assist the construction of hyperedges, we use the span of each text node to generate the span position corresponding to the feature sequence. Then we use the span position encoding to add span information to the hypergraph construction process. In this way, the model can divide the hyperedge according to the text node span, so as to achieve more accurate extraction of the special entity range. In the stage of semantic entity extraction, we use multi-label classification to determine whether a node is connected by a hyperedge. Since there may be more than one type of hyperedges satisfying the join condition. To ensure the uniqueness of
the entity type, we select the hyperedge with the maximum probability to establish the connection based on multi-label classification result.

heading: 3.2 Hypergraph Attention Head
text: We use the multi-head self-attention matrices to represent the hypergraph. Consider a hypergraph with L number of nodes and N class of hyperedges. We use a multihead attention score of shape N×L×L as the representation of this hypergraph. Hyperedge classes are represented by different heads of multi-head attention. The attention matrix corresponding to each head represents the distribution of a type hyperedge.
In the hypergraph, each token corresponds to a node. Assume the document token sequence is x = {x1, x2, ..., xn}. After understanding the document model, we convert the input token sequence into a high-dimensional feature representation sequence of the tokens:
h = {h1, h2, ..., hn} = Model({x1, x2, ..., xn}), (1) where h ∈ RL×H is the high-dimensional feature representation sequence of the token and Model(·) is the document understanding model. L indicates the token sequence length, which also represents the number of token nodes. H is the feature dimension size. Based on h, we can obtain the query vector q and the key vector k:
q = {qα : Wq,αh+ bq,α}, k = {kα : Wk,αh+ bk,α},
(2)
where α ∈ ZD is one head in multi-head attention, which can be regarded as a type in D kinds of hyperedges. With multi-head query vector and key vector, hypergraphs can be represented by a selfattention score calculated by q and k:
s = qTk = {sα(i, j) : qTi,αkj,α, i ∈ ZL, j ∈ ZL}. (3)
sα(i, j) is the attention score at the α type hyperedge span with [i, j]. qi,α and kj,α are the start and end of the span with [i, j] in the α type hyperedge matrix. In this way, we implement hypergraph extraction of semantic entities.

heading: 3.3 Span Position Encoding
text: As we mentioned in Introduction, tokens of the same text node normally share the same semantic label in the process of semantic entity recognition of documents. We hope that the head can consider
this span boundary prompt during entity extraction. Therefore, we construct the span position of the token sequence based on the text nodes and incorporate span information into the heads through position encoding. As shown in Figure 3, token feature sequence h{h1, h2, ..., hn} and text node sequence N = {N0, N1, ..., Nm} has a surjective relation. We define this relational mapping as:
f(hi) = Nj , hi ∈ h,Nj ∈ N. (4) Based on this relation mapping, we construct the span position. For the same text node Nj , all token feature nodes that have a mapping relationship with the same text node Nj share the same position:
pi = Position(f(hi))
= Position(Nj) = j, hi ∈ h,Nj ∈ N, (5)
where pi is the span position of token feature hi, Position is the index of Nj . In this way, we can obtain the span position sequence p = {p1, p2, ..., pn}. On the basis of p, we use rotary position coding (Su et al., 2021) to generate position encoding R, which satisfies RTi Rj = Rj−i. Then the calulation of multi-head hypergraph score will be adjust to the following form:
sα(i, j) = (Ripi,α)T (Rjkj,α) = pTi,αRTi Rjkj,α = pTi,αRj−ikj,α.
(6)
Because the start is always before the end when the span of token sequence is extracted. Span extraction nodes should not appear in the lower triangular region of the hypergraph attention score. For the purpose of making the hyperedge construction more reasonable, we add mtril to the hypergraph matrix and the final hypergraph score format is as follow:
sα(i, j) = p T i,αRj−ikj,α +mtril(i, j). (7)

heading: 3.4 Balanced Hyperedge Loss
text: In the process of loss calculation, we collect positive samples Pα and negative samples Nα respectively for each type of hyperedge α . The positive sample indicates that there is a α type hyperedge span with [i, j] in α type hypergraph, while the reverse is a negative sample. The formats of Pα and Nα are as follows:
Pα = {sα(i, j)|lα(i, j) = 1}, Nα = {sα(i, j)|lα(i, j) = 0},
(8)
where l is the hypergraph label matrix corresponding to s. With the sets of positive and negative samples,we can get the positive sample loss Lp and the negative sample loss Ln:
Lp = log  1 + ∑
(i,j)∈Pα e−sα(i,j)
  ,
Ln = log  1 + ∑
(i,j)∈Nα esα(i,j)
  .
(9)
Different from Global Pointer (Su et al., 2022), we gain the final loss with a balance factor b ∈ [0, 1) to avoid the matrix sparsity caused by too many label types. The final training loss of hypergraph attention score can be expressed in the following form:
L = (1 + b)Lp + (1− b)Ln. (10)

heading: 3.5 HGALayoutLM
text: To verify the performance of the HGA method, we apply HGA to the latest GraphLayoutLM to build a novel semantic entity recognition model, HGALayoutLM. We use GraphLayoutLM as the base model for feature encoding. According to its input requirements, we input four multi-modal document information of the document: text, layout, visual and graph to obtain the feature sequence of the text tokens. Before input, we sort the sequence of text tokens using the layout graph according to the reordering strategy of GraphLayoutLM. On the basis of this graph structure-prompted document understanding model, we use the hypergraph attention layer as the head for document semantic entity recognition. The feature sequence of the token and the generated span position are used as the head input. The HGA method is used to help the model extract and classify semantic entities according to the text node span prompts.

heading: 4 Experiment

heading: 4.1 Experimental Setup
text: Model Settings. The model settings are consistent with those of GraphLayoutLM. The text sequence length is 512 and the document image is resized to 3× 224× 224 dimensions. The image is cut into 196 patches in the size of 16× 16. Transformer self-attention layer scaling factor α is set to 32. For HGALayoutLMBASE, the hidden layer dimensions, the number of encoder self-attention
layers, the number of self-attention heads and intermediate dimensions for feed-forward networks are set to 768,12,12 and 3072, respectively. The head number of graph mask layer is 6. The hidden layer dimension, encoder self-attention layer number, self-attention head number and intermediate dimensions for feed-forward networks of HGALayoutLMLARGE are set to 1024,24,16 and 4096, respectively. The head number of graph mask layer is 8. The hidden size of hypergraph attention layer in both base and large model is set to 64. To ensure the fairness of the experiment, we convert the results of hypergraph extraction into the format of BIO annotations for comparison.
Datasets. We select four commonly used document information extraction datasets. Three of these datasets are in English, including FUNSD, CORD and SROIE. The other is the Chinese dataset, XFUND. The current XFUND task semantic entity recognition task of comparative experiment results is less and there is almost no LARGE version experiment results. We only choose the BASE version of the model for our experiments. Detailed dataset information and finetuning hyperparameters settings can be viewed in Tables 1 and 2, respectively.
Baselines. We choose the classical natural language processing model BERT (Devlin et al., 2018)
as the single modal document understanding comparison model and select several classical multimodal document understanding models, such as LayoutLM (Xu et al., 2020b), BROS (Hong et al., 2022), LayoutLMv2 (Xu et al., 2020a) and LayoutXLM (Xu et al., 2021). We also include the latest works in document understanding for comparison, such as ERNIE-Layout (Peng et al., 2022), LayoutLMv3 (Huang et al., 2022), GeoLayoutLM (Luo et al., 2023), GraphLayoutLM (Li et al., 2023a) and UDop (Tang et al., 2023). It is
worth noting that according to the code design of LayoutLMv3 and GraphLayoutLM, different heads are selected under different conditions of the number of label types. Specifically, the model uses linear layer as the classification head when there are less than 10 types of labels (e.g. FUNSD, SROIE, XFUND). On the contrary, when the number of labels is greater than or equal to 10 (e.g. CORD), the MLP is selected as the classification head.

heading: 4.2 Main Results
text: The English datasets experiment results are shown in Table 3. The BASE version of HGALayoutLM using hypergraph attention layer as the head has achieved the best results on FUNSD and SROIE datasets (94.32 on FUNSD and 99.53 on SROIE), even when compared to the LARGE versions of
models. Compared with GraphLayoutLMBASE using linear classification, HGALayoutLM achieves improvements of 0.89, 0.39 and 0.54 on FUNSD, CORD and SROIE datasets, respectively. The LARGE version of HGALayoutLM has achieved F1 scores of 95.31 and 99.61 on FUNSD and SROIE respectively, further updating the best performance on these datasets. Compared with GraphLayoutLM in the LARGE version, HGALayoutLM has F1 score 1.15 and 0.19 higher on FUNSD and SROIE datasets, respectively. This demonstrates the effectiveness of HGA on the task of less labels.
However, we can find that the performance of HGA is not outstanding on the CORD dataset. We think this is because the CORD dataset has a large number of label categories. The number of labels
in CORD is an amazing 30, compared with the 3 or 4 label categories in other datasets. Since in the process of constructing the hypergraph, different types of hyperedges are built separately. Plenty of label categories will make the effective span nodes of hypergraph matrix sparse, which is not conducive to semantic entity recognition. However, by comparing GraphLayoutLM, we can find that HGA head can still improve the performance.
The experiment results of Chinese dataset, XFUND, are shown in Table 4. We can find that our HGALayoutLM has achieved the state of the art in XFUND (Precision 92.79, Recall 95.70 and F1 94.22). This further verifies the effectiveness of HGA head.

heading: 4.3 Ablation Study
text: To verify the effectiveness of our Span Position Encoding. We conduct ablation study on FUNSD. We can see from Figure 4 that the entity extraction effect without position encoding (w/o pos) is much worse than that with position encoding. In addition, we also compare the performance of our span position encoding (w/ span pos) with that of traditional position encoding (w/ pos). We can find that the performance of our span position encoding is obviously better than that of traditional position encoding.This demonstrates the effectiveness of our span position encoding with span prompt.
In order to prove that Balanced Hyperedge Loss can solve the problem of sparse hyperedge matrix caused by too many entity types. We conduct experiment statistics on different value of balance factor on CORD dataset with plenty of entity types and present the results in Figure 5. We can see that the performance of the unbalanced model (b = 0) is not ideal, even worse than the performance of the MLP head. However, proper balance factor allow the model to pay more attention to the hyperedge entities and achieve better results. For example, the performance when b is 0.4 exceeds the performance when the MLP is used as the head.

heading: 4.4 Anaysis of Different Head
text: To analyze the effects of different head, we adopt GraphLayoutLMBASE and HGALayoutLMBASE as the base model to conduct comparative experiments on three different heads: linear layer, MLP and HGA. The experiments are carried out on FUNSD, CORD, SROIE and XFUND datasets.
The experiment results are shown in Table 5. As the simplest network structure, the linear layer has the worst classification effect. The MLP increases the number of linear layers on top of the linear layer. It also joins activation layers and dropout layers to linear layers. The more complex network structure makes MLP slightly better than the semantic entity recognition of a single linear layer on most datasets. As our proposed hypergraph attention method, HGA performs significantly better than the other two classifiers,which shows the effectiveness of HGA, which demonstrates the superior performance of HGA.
To test the complexity of HGA, we compare HGALayoutLM with the model with traditional heads. The PyTorch-OpCounter tool is used to calculate the time and space complexity. The number of entity types is set to 3. As we can see from Table 6, HGA does not bring a large cost of time and space calculation and HGA is even less costly than MLP in terms of time and space computation. This indicates that our performance improvement is not due to the increase in the number of parameters.

heading: 4.5 Comparison with Large Language Model
text: We conduct a comparative analysis of our HGALayoutLM with the latest document multimodal large language model, LayoutLLM (Luo et al., 2024), to analyze the advantages and disadvantages of the models. LayoutLLM, which uses LayoutLMv3 as encoder and Llama as decoder, has so far achieved the state of the art results on several document intelligence tasks. We show the comparison results in Table 7. We can see that our HGALayoutLM slightly underperforms compared to the professionally fine-tuned document large language model.
https://github.com/Lyken17/pytorch-OpCounter
However, under the premise of similar performance to large language models, our model parameters and computational consumption are much lower than the existing large language models. This fully demonstrates the advantage of our method.

heading: 5 Conclusion
text: In this work, we propose a semantic entity recognition method (HGA) based on hypergraph attention. This method extracts semantic information from documents by establishing different hyperedges between feature nodes. On the basis of the hypergraph, we design span position encoding and balanced hyperedge loss to enhance the entity extraction capability of the hypergraph attention head. We use the HGA method to build a novel semantic entity recognition model HGALayoutLM based on GraphLayoutLM. This model has good performance in SER tasks. Experiments show that our method achieves the state of art on semantic entity recognition tasks on the FUNSD and XFUND datasets.

heading: 6 Limitation
text: The HGA method can achieve good performance on semantic entity recognition tasks, but there is still a lot of work for us to improve. On the one hand, when there are more types of semantic entities, the cost of improvement from HGA becomes higher. The number of superedge matrices increases because of more semantic entity categories. This not only leads to sparse label matrices, but also to more model parameters. How to solve the matrix sparsity and parameter growth caused by the number of label types is the future work we need to study. On the other hand, since our proposed head is currently targeted at semantic entity recognition tasks in the document domain. In the future, we will explore more general head to adapt to diverse document task types.

heading: Future Work
text: How to solve the matrix sparsity and parameter growth caused by the number of label types is the future work we need to study. On the other hand, since our proposed head is currently targeted at semantic entity recognition tasks in the document domain. In the future, we will explore more general head to adapt to diverse document task types