UNER: A Unified Prediction Head for Named Entity Recognition in Visually-rich Documents

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 09:36:12

Abstract: The recognition of named entities in visually-rich documents (VrD-NER) plays a critical role in various real-world scenarios and applications. 
However, the research in VrD-NER faces three major challenges: complex document layouts, incorrect reading orders, and unsuitable task formulations. 
To address these challenges, we propose a query-aware entity extraction head, namely UNER, to collaborate with existing multi-modal document transformers to develop more robust VrD-NER models. 
The UNER head considers the VrD-NER task as a combination of sequence labeling and reading order prediction, effectively addressing the issues of discontinuous entities in documents. 
Experimental evaluations on diverse datasets demonstrate the effectiveness of UNER  in improving entity extraction performance. 
Moreover, the UNER head enables a supervised pre-training stage on various VrD-NER datasets to enhance the document transformer backbones and exhibits substantial knowledge transfer from the pre-training stage to the fine-tuning stage. 
By incorporating universal layout understanding, a pre-trained UNER-based model demonstrates significant advantages in few-shot and cross-linguistic scenarios and exhibits zero-shot entity extraction abilities.

Main paper: 
 related work: 
 Pre-trained Document Transformers: Recent research in pretraining techniques in NLP [2, 4, 39] and CV [1, 12] have demonstrated the significant potential in building multi-modal transformer for document understanding. Inspired by BERT [4] , LayoutLM [30] improved the masked language modeling task to build a multimodal document transformer that can effectively combine layout and textual information to understand and process documents with complex layouts. Following this idea, LayoutLMv2 [33] and Lay-outLMv3 [6] focus on incorporating visual information and designing various cross-modal alignment pre-training tasks, and significantly enhance the model's performance in various document understanding tasks. Meanwhile, some studies have realized the importance of layout information and focus on improving text-layout interactions, such as StructuralLM [9] , LiLT [26] , LayoutMask [24] , and GeoLayoutLM [16] . While the above models are pre-trained with self-supervised tasks, UDOP [23] proposed a supervised pretraining stage by unifying multiple VrDs tasks in a generative framework to fully exploit the correlation among different tasks.The UDOP model and universal NER studies in NLP [15, 34] have inspired us to propose a unified model for VrD-NER.Reading-order-aware Methods: Understanding the reading order of documents remains a key challenge in VrD-NER and previous studies tried to address this issue from different perspectives. Lay-outReader [29] proposed a sequence-to-sequence approach and a reading order dataset for order prediction. ERNIE-Layout [18] designed a reading order prediction task during pre-training and leveraged a serialization module to rearrange the order of input tokens. However, these methods require extra annotated data or introduce computation complexity. Inspired by discontinuous NER studies [10, 28] , Zhang et al. introduced TPP, a prediction head to classify the document tokens and predict their correct reading orders in an integrated manner. These studies have inspired us to For better illustration, we use a document with a reading order issue as input. Given the input document, the UNER-based model receives the entity names ("header", "question", and "'answer") as queries and conducts token classification and order prediction in its two submodules, TOP and QTC. Here we use numbers to denote the ground-truth labels ("1" for positive and blanks for negative) and colored backgrounds to denote the binary classification predictions (gray for positive and white for negative). Ultimately, we combine the predictions for decoding and obtain the full entity spans. Incorrect predictions are denoted by the red color or a cross.reformulate the VrD-NER task and include a token order prediction task. 
methodology: 
 The UNER head is an entity prediction head that works in collaboration with existing document transformers, which takes the token embeddings of the transformer backbone as input. Given an input document with 𝐿 tokens, a pre-trained document transformer is first utilized to obtain the token embeddings matrix E ∈ R 𝐿×ℎ , where ℎ is the dimension of the embeddings. The 𝑖-th entry of E, denoted as e 𝑖 ∈ R ℎ , represents the multi-modal embedding vector of the 𝑖-th token. These token embeddings are then inputted into the UNER head, which consists of two sub-modules: query-aware token classification and token order prediction. The pipeline of a UNER-based VrD-NER model is illustrated in Figure 2 .Assuming that we need to extract 𝐶 types of entities from the document. when using a sequence-labeling-based (SL) head with the BIO-tagging scheme, the entity extraction task is treated as a (2𝐶 + 1)-way classification problem for each token embedding e 𝑖 :EQUATIONThe function F 𝑠𝑙 maps the token embedding e 𝑖 to a BIO-tagging label. However, this approach has a limitation in that it can only handle a fixed number of pre-defined entity types.To address this issue, we propose the Query-aware Token Classification (QTC) module. This module first uses a query encoder to encode the entity names or descriptions as classification queries. It then leverages these queries as semantic clues to perform binary classification for all tokens.To handle 𝐶 entity types, we start by encoding each entity name using a query encoder. This encoder transforms each entity name into a ℎ-dimensional embedding, resulting in a query embedding matrix Q ∈ R 𝐶 ×ℎ . In this matrix, each row corresponds to an entity type, and the 𝑗-th entry of Q, denoted as q 𝑗 ∈ R ℎ , represents the embedding of the 𝑗-th query.Then we combine the token embedding matrix and query embedding matrix to generate the query-aware token embedding tensor:E ⊕ Q → S ∈ R 𝐿×𝐶 ×ℎ, where ⊕ stands for the broadcastable addition operation. We use s 𝑖,𝑗 to denote (𝑖, 𝑗)-th entry of S, which is a query-aware token embedding for the 𝑖-th token and 𝑗-th query and satisfies: s 𝑖,𝑗 = e 𝑖 + q 𝑗 . For each s 𝑖,𝑗 , we will conduct a binary classification to determine if the 𝑖-th token belongs to the 𝑗-th entity type:EQUATIONWe will conduct 𝐿 × 𝐶 binary classifications for all token-query pairs and the total classification loss is calculated as :EQUATIONHere CE(•, •) is the cross-entropy loss function and ŝ𝑖,𝑗 ∈ {0, 1} is the ground-truth label. In QTC, the shape of the query-aware token embedding tensor is dynamically determined by the number of input queries. This flexibility allows us to input any number and type of queries into the module, thereby avoiding the limitation of pre-defined entity types in the SL head.Moreover, as we use independent binary classifications, each token can be possibly classified into multiple categories. Such a design makes the UNER head capable of extracting nested and overlapped entities. While these entities are not fully represented in current VrD-NER benchmarks, they are common in real-world applications, as users may have different extraction goals within the same document and the same token may belong to multiple entity types. For instance, in Figure-1 case (1), a UNER-based model can simultaneously extract the full address (denoted in green), the city name ("Batang Kal"), and the state name ("Selangor") by using "address", "city", and "state" as input queries. In this case, these entities are overlapped and cannot be extracted with an exclusive 𝐶-way classification head.Furthermore, the QTC module utilizes entity names as semantic clues for classification, which enables it to extract unseen entity types through semantic knowledge transfer.After token classification, it is necessary to arrange the tokens in order to form correct entity spans. In the SL head, the BIOtagging scheme labels the "Beginning" and "Inside" information, which helps in token arrangement. However, it fails to handle discontinuous entities or tokens that are incorrectly ordered in VrDs. To overcome this limitation, we introduce a Token Order Prediction (TOP) module that predicts the token orders in VrDs.The TOP module treats the token order prediction task as an edge prediction task on a token graph. For a document with 𝐿 tokens, we represent its token order using a directed graph with 𝐿 vertices. This graph is represented by a binary matrix G ∈ {0, 1} 𝐿×𝐿 . If the 𝑘-th token is the successor of the 𝑖-th token, the (𝑖, 𝑘)-th entry of G, which represents the directed edge from 𝑖 to 𝑘, is set to 1. Otherwise, it is 0. By formulating the task in this way, the token order prediction becomes equivalent to predicting the edges on the graph.During training, TOP performs binary classification for all directed edges in the graph:EQUATIONThe objective of this task is to optimize the following loss function:EQUATIONwhere ĝ𝑖,𝑘 ∈ {0, 1} represents the ground-truth label for each edge in the graph.It is important to note that ĝ𝑖,𝑘 is derived from the entity annotations in the VrD-NER dataset, so it only covers the tokens that are part of entities. The order of the non-entity tokens is unknown during training, and therefore, the corresponding losses in Equation-5 should be ignored. Fortunately, after training with annotated documents, the knowledge of reading order learned from the entity-related tokens can be transferred to non-entity tokens and tokens in other types of documents. Such knowledge transfer is discussed in detail in Section-4.4.The total training loss of the UNER head can be represented as follows:EQUATIONwhere 𝜆 is a hyper-parameter that controls the balance between the two losses. As shown in Figure -2 , the ground truth labels in L 𝑞𝑡𝑐 and L 𝑞𝑡𝑐 are predominantly zeros, with only a small portion being ones. This label imbalance issue poses a challenge for loss optimization, so we employ the ZLPR loss [21] to enhance the training of the two losses and improve optimization efficiency.During inference, we utilize the predictions of the QTC and TOP modules to construct full entity spans using the following steps:(1) For each entity type (e.g., the 𝑗-th type), we identify all tokens that meet the condition F 𝑞𝑡𝑐 (s𝑖, 𝑗) > 0, along with their valid token orders satisfying F 𝑡𝑜𝑝 (e 𝑖 , e 𝑘 ) > 0.(2) Based on the token orders, we identify the tokens without any predecessor tokens and use them as the starting tokens to create multiple token series. (3) For each token series, we select the best subsequent token for the last token by choosing the token with the highest token order classification score and then adding it to the current token series. (4) Repeat step (3) until the last token does not have any valid subsequent tokens, and use the resulting token series to form entity spans.It should be noted that the UNER head enables multiple entity names to be queried for each document during both training and inference. When provided with a document and 𝐶 entity names, we only need to compute the document transformer and TOP module once and reuse their outputs to collaborate with the QTC module for 𝐶 iterations. This feature significantly alleviates the computational burden for recomputing the backbones when handling dense entity extraction tasks and enables a more flexible model deployment. 
experiments 4.1 datasets: 
 In our experiments, we utilize 7 commonly used VrD-NER benchmarks to compare the performance of different methods. To evaluate the effectiveness of supervised pre-training in VrD-NER, we incorporate two additional datasets for pre-training, which encompass a wide range of document types and entity types. Detailed information of these datasets can be found in Table-1. Fine-tuning: To demonstrate the effectiveness of our method, we perform fine-tuning on 7 popular VrD-NER benchmarks. Notably, when comparing with other prediction heads, we use the reannotated versions of FUNSD [8] and CORD [17] , namely FUNSD-r and CORD-r datasets [37] . The original datasets are not suitable for evaluating the ability to extract discontinuous entities, as the annotations in the original datasets are manually adjusted and do not reflect real-world scenarios. We use the re-annotated versions to better assess the entity extraction performance of different prediction heads. To provide solid experiments, we use another five 2 : The F1 scores (%) of VrD-NER task on FUNSD-r and CORD-r datasets. We compare UNER with other prediction heads using two backbones, LayoutLMv3 and LayoutMask. The best results are denoted in boldface. benchmarks when comparing UNER-based models with other baselines. These benchmarks, including a cross-linguistic dataset in 7 languages, cover a wide range of document types, entity types, and languages. Supervised Pre-training: During supervised pre-training, we expect the dataset to have a better diversity, so we choose two large datasets, SVRD and DocILE [20] for pre-training. The SVRD dataset originates from the ICDAR 2023 Competition 1 and consists of densely annotated key-value linkings in various types of documents in real-world scenarios. By using the key entities as a description of the entity type, we transform these annotations into entity extraction labels, resulting in a VrD-NER dataset with hundreds of entity types. Since SVRD documents are primarily in Chinese, we utilize the DocILE dataset to enhance language diversity. This dataset comprises 5k documents with 55 entity types in English.In our experiments, we compare different prediction heads using the base version of LayoutLMv3 and LayoutMask as transformer backbones, following the backbone setting in TPP. The query encoder in the QTC module is a 4-layer transformer with cross-attention to the transformer backbone. It is initialized with the first four 1 https://rrc.cvc.uab.es/?ch=21&com=introduction layers of Q-Former in BLIP-2 [11] with 38M parameters. We employ the tokenizer and the 1D-position embedding layer from the transformer backbone to process queries during query encoding. As each query can contain multiple tokens, we use the embedding of the first token as the query embedding. To determine the optimal value of the hyper-parameter 𝜆, we conduct ablation studies. In our experiments, we set 𝜆 = 0.1 and 𝜆 = 0.5 for models with or without supervised pre-training, respectively. The batch size is set to be 16 for all fine-tuning datasets. For few-shot experiments, we repeat them five times and report the average scores. During the supervised pre-training stage, both the backbone and the UNER head are trained on the pre-training datasets for 20 epochs before fine-tuning.Comparison with Prediction Heads: We first conducted experiments to compare the performance of the UNER head with other prediction heads, namely the SL and TPP heads. Following the setting in TPP, we combined these heads with different backbones and evaluated their performance on two VrD-NER datasets, namely FUNSD-r and CORD-r. We use the entity-level F1 score as the evaluation metric. The experiment results are presented in Table-2, which demonstrate that the UNER head outperformed both the SL and TPP heads on both datasets, regardless of the backbone used. These results have demonstrated its superiority as a prediction head for the VrD-NER task.It is important to highlight that the FUNSD-r and CORD-r datasets have discontinuous entities due to complex layouts and incorrectly ordered tokens. This significantly impacts the performance of the SL head, as it is unable to handle such entities. Additionally, the SL head is incapable of learning and conveying token order knowledge effectively, which limits its usefulness as a VrD-NER head.While the TPP head is capable of handling discontinuous entities, it faces optimization issues that negatively affect its performance. This is because the TPP head attempts to jointly optimize token classification and token order prediction in a unified manner. For each entity, it predicts one token graph to represent entity-specific token paths, resulting in the need to predict a 𝐿 × 𝐿 × 𝐶 binary tensor and 𝐿 2 𝐶 classification sub-tasks. Such entity-specific token Figure 3 : The few-shot performance on the CORD-r dataset when using different percentages of training samples (from 5% to 100%). We use LayoutMask as the backbone and compare its performance with different prediction heads or supervise pre-training conditions. ( 1): The entity-level F1 scores in VrD-NER when using TPP, UNER, and UNER with supervised pretraining ("UNER+SP"). ( 2)&( 3): For the UNER-based method, we also report the performance of its submodules, the token-level classification accuracy in QTC, and the token order classification accuracy in TOP. As we do not have the complete reading order annotations for all the tokens in the documents, in TOP we only calculate the accuracy for entity-related tokens.graphs in TPP also influence the transfer of token order knowledge among different entity types.On the other hand, the UNER head has two separate optimization goals and reduces the number of binary classifications sub-tasks to (𝐿 2 +𝐿𝐶). Additionally, UNER explicitly learns and represents token order knowledge in one unified token graph which is independent of the entity types, making it suited for transferring universal token order knowledge. This design makes the UNER head easier to optimize and more efficient with limited data.To further highlight this advantage, we compared the TPP and UNER head on the CORD-r dataset in a few-shot setting, where the model is fine-tuned using only a subset of the training samples. The results of this comparison are summarized in Figure -3 . The findings from these experiments highlight that UNER consistently outperforms TPP across all training percentages. It is noteworthy that the performance advantage of UNER becomes more pronounced as the percentage of training data decreases. For instance, when utilizing only 5% of the data, UNER achieves a notable improvement of +21.24% compared to TPP. This demonstrates the data efficiency of UNER in handling limited training data. Comparison with VrD-NER Baselines: The above results have proved the advantage of UNER compared to other prediction heads. In this section, in order to show that the UNER head can help in building excellent VrD-NER models, we further extended our comparison of two UNER-based models against other state-of-theart VrD-NER models on 5 datasets: SROIE, EPHOIE, WildReceipt, SIBR, and XFUND.The results of these comparisons are presented in Table-3. Note that XFUND is a cross-linguistic dataset and comprises seven subdatasets, so we report the averaged F1 scores across them. The UNER head demonstrates highly competitive results on all datasets with both backbones, and it achieves the highest F1 score on three out of the five datasets overall, which is a significant achievement considering the number and diversity of the compared benchmarks. Notably, "LayoutLMv3+UNER" outperforms other methods by a significant margin (+2.93%) on the WildReceipt dataset. These results further validate the effectiveness and compatibility of our UNER head to work with document transformer backbones. Comparison in Multi-task Setting: In order to demonstrate the potential of UNER in unified entity extraction, we conducted a multi-task experiment using four datasets: SROIE, EPHOIE, Wil-dReceipt, and SIBR. The main objective was to illustrate that the UNER head can effectively handle multiple datasets with one model. We accomplished this by fine-tuning the UNER-based model on the combined training samples from these four datasets and evaluating the model's performance separately. The results are presented in Table-3 as "+Multi-task". The UNER-based model in the multi-task setting achieved competitive results compared to other methods, highlighting its potential as a unified VrD-NER head.However, we also observed that the F1 scores of the UNER-based models in the multi-task setting were slightly lower compared to the scores in a single-task setting. We believe that this is due to the difficulty in balancing the number of training samples for different datasets. As our focus is not on searching for the best performance with a particular group of datasets, we simply combined these datasets for training without making careful adjustments in order to showcase the vanilla capability of UNER for unified VrD-NER.The UNER head has a significant advantage as it can be utilized for supervised pre-training, which further enhances the self-supervised As the performance of entity extraction in UNER relies on the accuracy of its sub-modules, we also calculate the accuracy scores of token classification and token order prediction in the QTC and TOP modules to gain a more comprehensive understanding. The results are listed in Figure -3 .Our observations indicate that supervised pre-training has significant benefits in few-shot scenarios. Specifically, the "UNER+SP" model achieves remarkably high performance even with a small portion of training samples, outperforming the "UNER" model by a large margin on all tasks and configurations. Notably, the f1 score of "UNER+SP" in "VrD-NER" is surprisingly high with extremely limited data (74.62% with only 5% training samples). Furthermore, "UNER+SP" with 5% training samples has a higher accuracy score in "TOP" than "UNER" with 25% samples (98.01% compared to 97.39%). These advantages highlight that the UNER-based model benefits significantly from the document knowledge learned during pretraining on large datasets. This knowledge facilitates the model's adaptation to the other downstream VrD-NER dataset, particularly when the available data is limited.Comparison in Zero-shot Setting: In order to further explore the effectiveness of knowledge transfer through supervised pretraining, we conducted pre-training of the UNER-based model using various combinations of datasets and assessed their zero-shot performance on other VrD-NER benchmarks. The pre-trained models exhibited the ability to extract entities in a zero-shot setting on the SROIE and EPHOIE datasets, and the results are summarized in Table-5. In addition to overall VrD-NER performance, we also provide zero-shot accuracy scores for the QTC and TOP sub-modules for a more comprehensive analysis. Our analysis indicates that the zero-shot entity extraction ability is largely attributed to the inclusion of the SVRD dataset, which significantly improves the QTC for both SROIE (15.14%) and EPHOIE (57.38%). Furthermore, incorporating the DocILE dataset further enhances the VrD-NER abilities, particularly in SROIE where the F1 score increased from 0.07% to 21.90%.In order to gain a better understanding of the entity extraction ability of the pre-trained model, we tested the best model (pretrained on both SVRD and DocILE) with various documents and queries, and presented some prediction results in Figure -4 . Our observations indicate that the model is effective in extracting entities with explicit trigger words. Using these trigger words as queries consistently yields satisfactory results, regardless of the layout alignment. However, the model exhibits limitations when handling entities without explicit trigger words, unnatural queries, or queries with varying meanings. For example, the entity names in the CORDr dataset are specifically designed, such as "menu.sub.price" and "total.cashprice". Similarly, the entity type "Header" is used in multiple datasets, but its meaning can differ across documents, leading to inconsistent predictions. These findings underscore the potential for using diverse pre-training datasets to develop a unified VrD-NER model with stronger zero-shot abilities, while also identifying challenges in transferring document understanding knowledge. 
conclusion: 
 This paper presents UNER, a query-aware entity extraction head for unified named entity extraction to address the challenges in VrD-NER. Experimental results across various datasets demonstrate the superior performance of UNER compared to existing methods. Moreover, UNER exhibits knowledge transferability when supervised pre-training is applied and contributes to building robust backbones for VrD-NER tasks.  
: 
 These issues contribute to the complexity of understanding the document and result in discontinuous entities when the document is arranged in common reading order (e.g., top-tobottom and left-to-right).and then adapted to different document understanding tasks. These models have the capability to jointly encode the text, layout, and visual information of documents, enabling the generation of multimodal representations that prove beneficial to downstream tasks.Despite the above advancements made in improving document representations, the VrD-NER task still faces significant challenges due to the inherent characteristics of document data. It suffers from two common issues in real-world documents, complex document layouts and reading orders issues (see illustration in Figure -1 ). Previous studies solve these issues from two perspectives: (1) Introducing layout-related or spatial-aware pre-training tasks to enhance the ability of the models to understand the layout of documents [16, 24] ;(2) Proposing datasets with annotations or pre-processing modules for reading order prediction to facilitate accurate identification of the order of token in documents [18, 29] .However, the current task formulation in VrD-NER hinders the progress made to solve the above issues. Many document transformers approach the VrD-NER task as a sequence labeling problem and utilize a sequence labeling (SL) head with the BIO-tagging scheme [19] for task-specific fine-tuning. Nevertheless, the SL head is specifically designed for plain text in flat NER problems, thus cannot fully utilize multi-modal representations to learn reading order knowledge or represent discontinuous entities in documents. This significantly limits the application of these backbones and makes them fall short in handling the incorrect token order arrangement in real-world documents and discontinuous entities arising from complex layouts, leading to suboptimal performance in entity extraction and necessitating additional token serialization tools. To overcome this limitation, researchers have explored alternative task formulations. Zhang et al. [37] recognized the importance of reading order prediction in VrD-NER and proposed a token-path prediction head (TPP) to combine token order prediction and token classification in an integrated manner. However, the TPP head brings extra optimization targets and faces optimization issues. To obtain optimal performance, it requires substantial training samples for each entity type and performs poorly in low-resource settings.Despite this, the SL head and the TPP head have a major drawback in that they are not effectively utilized for supervised pretraining, as they are designed for a fixed number of entity types. Recent studies have recognized the power of supervised pre-training in VrD-NER [23, 35] . Tang et al. [23] proposed to unify pre-training and multi-domain downstream tasks in a generative scheme and leveraged 11 supervised datasets for pre-training, which exhibits that adding supervised pre-training can further improve the model performance.Based on the above observations, we recognize the significant benefits of developing a better prediction head for VrD-NER. To this end, we propose UNER, a Unified prediction head for Named Entity Recognition. The UNER head is a query-aware entity extraction head that collaborates with existing document transformers to build better VrD-NER models. A UNER-based model can appropriately represent discontinuous entities and predict entities in correct reading orders. By harnessing the power of the supervised pre-training, it can effectively comprehend complex layouts and understand new documents with limited data.Specifically, UNER models the VrD-NER task as a combination of sequence labeling task and reading order prediction, which is realized by two modules: (1) A query-aware token classification module that leverages the entity names (e.g., "address", "flight" and "units price" in Figure -1 ) as extraction clues for token-level classification; (2) A token order prediction module that predicts the pairwise token orders in the entities. By incorporating the predictions from the two modules, the UNER head is capable of learning the reading order knowledge from entity annotations and correctly predicting discontinuous entities in VrDs.Additionally, since the UNER head can adaptively use any entity names as input queries, we can use a single UNER-based model to train on different NER datasets without the need for parameter reinitialization, thus enabling a supervised pre-training stage before fine-tuning. By leveraging supervised pre-training on external annotated documents with various layouts and entity types, we can incorporate universal layout understanding knowledge into the model and further enhance its performance on downstream VrD-NER tasks.To assess the performance of UNER, we conducted experiments using 7 datasets from various domains and languages. The experimental results consistently demonstrated that UNER achieved significant improvements on all datasets, thereby showcasing its effectiveness as a prediction head for VrD-NER. Additionally, when applying a supervised pre-training stage, the UNER head exhibited notable advantages in cross-linguistic and few-shot settings, as well as demonstrating zero-shot VrD-NER capability. This discovery highlights the document knowledge transferability of UNER.The contributions of our paper can be summarized as follows:(  
 
 Appendix:  


Decision: The paper presents a novel approach, UNER, a query-aware entity extraction head designed to enhance Named Entity Recognition for Visually-rich Documents (VrD-NER). By combining sequence labeling with reading order prediction, UNER effectively handles complex document layouts and discontinuous entities. The method is shown to generalize well, performing robustly in zero-shot and cross-lingual settings. Reviewers praised the comprehensive experimental analysis and the method's novelty in addressing challenging issues in VrD-NER. However, some concerns were noted, including the need for more detailed explanations of certain experimental methods and the modest improvements observed in results. Despite these minor issues, the paper's contributions are significant, demonstrating clear advancements in VrD-NER and justifying acceptance.

Reviews: review: paper_summary: paper_summary: The paper addresses challenges in Named Entity Recognition for Visually-rich Documents (VrD-NER) by proposing UNER, a query-aware entity extraction head. UNER works with existing document transformers to improve VrD-NER models by combining sequence labeling and reading order prediction. This approach effectively handles complex layouts and discontinuous entities. Additionally, UNER supports supervised pre-training, enhancing model performance on downstream tasks.
strength_weakness: strength_weakness: - The proposed UNER is novel, addressing the issues of complex document layouts and reading order, and is orthogonal to previous methods.
- The authors conducted thorough experimental analysis to demonstrate the effectiveness of UNER.
- UNER shows excellent generalization, performing well on unseen languages.
limitations: limitations: - The authors need to briefly introduce the methods used in the experimental section within the related work chapter.
- In Table 3, there are many missing values. The authors should explain why these values were not reported.
- The authors should provide an explanation for why the ESP model performs better than UNER in some multilingual domains.
paper_summary: paper_summary: This work aims to deal with the visually-rich name entity recognition problem and tackle some critical issues, such as the diverse layout information and incorrect reading order. To solve these problems, they propose a Unified prediction head, a query-aware entity extraction head that collaborates with both the token representation and read order information. They conduct experiments under various domains, language, and settings.
strength_weakness: strength_weakness: 1. The topic is quite interesting and significant for the research field. The author tackled quite important issues in this area and conducted adequate experiments to prove the effectiveness of their methods.
2. This method achieves better performance in the zero-shot learning setting and shows good potential in language transferability.
limitations: limitations: 1. Some things are not clear, maybe I am not familiar with the field.  For example, the entity type in the given example in Figure 2, what is the 'Answer' label mean?
paper_summary: paper_summary: The paper introduces UNER, a unified prediction head for Named Entity Recognition in visually-rich documents. UNER models VrD-NER as a combination of sequence labeling and reading order prediction, addressing complex layouts and discontinuous entities. It enables supervised pre-training on various datasets, incorporating universal layout understanding into document transformer backbones, leading to improved performance and cross-lingual capabilities.
strength_weakness: strength_weakness: 1. UNER addresses the common issues of complex layouts and incorrect reading orders in VrD-NER, enabling the representation of discontinuous entities and the prediction of entities in correct reading orders.
2. The incorporation of a supervised pre-training stage using UNER enhances the document transformer backbones with universal layout understanding, leading to improved performance in downstream VrD-NER tasks.
limitations: limitations: 1. The experimental results seem to show only modest improvements, and the paper does not demonstrate significant advantages of the proposed method over existing approaches.
2. The query-aware extraction approach may require specifying entity types multiple times, potentially impacting performance to some extent.
3. It would be interesting to analyze whether visual-related NER tasks can be effectively handled by current multi-modal language models, such as GPT-4V, and how these models perform on such tasks compared to UNER-based models.
paper_summary: paper_summary: The recognition of named entities in visually-rich documents (VrD-NER) is essential but challenging due to complex layouts, incorrect reading orders, and unsuitable task formulations. To tackle these challenges, the proposed query-aware entity extraction head, UNER, collaborates with multi-modal document transformers to improve VrD-NER models. UNER addresses issues of discontinuous entities by combining sequence labeling and reading order prediction, demonstrating effectiveness through experiments on diverse datasets. Additionally, pre-trained UNER models show significant advantages in few-shot, cross-linguistic, and zero-shot entity extraction scenarios.
strength_weakness: strength_weakness: 1. The experiments were comprehensive.
2. Experimental results across various datasets demonstrate the superior performance of UNER compared to existing methods.
limitations: limitations: 1. The authors did not provide source code for reproducibility.
2. How should the term (2C+1)-way in line 283 be understood? Please provide a more detailed explanation. 
3. It would be better to use italics for mathematical symbols in the manuscript, such as the matrix \textbf{E} in line 270. 
4. Please explain the Multi-task experiment mentioned in Table 3. 
5. In the experiments, documents are used as model input; however, from the sample shown in the figure, it can be seen that the documents are in image format. How is this handled? 
6. During fine-tuning, did the authors fine-tune using all the data from the seven datasets together? 
7.  There are significant gaps between paragraphs in the article that need to be adjusted appropriately.