PPTSER: A Plug-and-Play Tag-guided Method for Few-shot Semantic Entity Recognition on Visually-rich Documents (2024)

Abstract: Visually-rich document information extraction (VIE) is a vital aspect of document understanding, wherein Semantic Entity Recognition (SER) plays a significant role. However, few-shot SER on visually-rich documents remains relatively unexplored despite its considerable potential for practical applications. To address this issue, we propose a simple yet effective Plug-and-Play Tagguided method for few-shot Semantic Entity Recognition (PPTSER) on visually-rich documents. PPTSER is built upon off-the-shelf multi-modal pre-trained models. It leverages the semantics of the tags to guide the SER task, reformulating SER into entity typing and span detection, handling both tasks simultaneously via cross-attention. Experimental results illustrate that PPTSER outperforms existing fine-tuning and few-shot methods, especially in low-data regimes. With full training data, PPTSER achieves comparable or superior performance to fine-tuning baseline. For instance, on the FUNSD benchmark, our method improves the performance of LayoutLMv3base in 1-shot, 3-shot and 5-shot scenarios by 15.61%, 2.13%, and 2.01%, respectively. Overall, PPTSER demonstrates promising generalizability, effectiveness, and plug-and-play nature for few-shot SER on visually-rich documents. The codes will be available at https://github.com/whlscut/PPTSER.

heading: 1 Introduction
text: Information extraction from visually-rich documents (VIE) is a process that concentrates on extracting pertinent information from various sources such as scanned images, documents, and PDF files. It effectively leverages layout and visual cues to decode the content enclosed within these documents (Xu et al., 2020). As an important part of VIE, Semantic Entity Recognition (SER) aims to extract entity spans from the visually-rich document. SER
has been hailed as a significant advancement in the realm of document intelligence, and has found widespread applications in numerous sectors.
The advent of multi-modal pre-trained models (Xu et al., 2020; Li et al., 2021c; Gu et al., 2021; Huang et al., 2022b; Yu et al., 2023) has ushered in a rapid evolution in SER methodologies. These models, pre-trained on a large corpus of scanned documents in a self-supervised manner, have significantly enhanced the comprehension ability of SER. Despite the remarkable achievements of the multi-modal pre-trained models, they often rely on extensive data for fine-tuning. However, acquiring a large volume of well-annotated SER data poses significant challenges such as: (1) Acquiring such data necessitates substantial financial resources and time. Annotators are required to label a multitude of OCR detection boxes in the document, adhering to meticulously designed guidelines. Identification of content within a box and accurately assigning labels to them are also tedious tasks. (2) The availability of data is often restricted due to privacy concerns. In scenarios involving sensitive information, such as invoices and insurance documents, data accessibility is severely limited due to the confidential nature of this information.
Despite the scarce research (Cheng et al., 2020; Yao et al., 2021; Wang and Shang, 2022) on fewshot Semantic Entity Recognition for visually-rich documents (few-shot SER), results have shown limitations in terms of generality and performance, and were limited to the specific application scenario. This paper, inspired by the comprehension capabilities of pre-trained models and the selective focus nature of the attention mechanism, introduces a novel approach called PPTSER, a Plug-and-Play Tag-guided method for few-shot Semantic Entity Recognition on visually-rich documents. The underlying principle of PPTSER consists of three main components: (1) Semantic Understanding and Alignment: Words related to SER tags are
10522
used as a prompt and are concatenated with the document’s text tokens. This combined input is then fed into a multi-modal pre-trained model. The motivation behind this is that the pre-trained model is expected to understand the semantics of both the document tokens and the tag-related prompt, thereby bringing the hidden states of the tokens and tag-related words for a specific entity type closer together. (2) Decoupling of SER task: SER task is segmented into Entity Classification and Entity Boundary Detection. This division aims to facilitate the resolution of boundary determination among adjacent entities of the same category within visually-rich documents. (3) Efficient Usage of Multi-head Attention: The attention weight obtained from the last attention block between the tag-related prompt and document tokens is directly used as the probability of tokens belonging to different tags. This mechanism, with different heads detecting various spans, is ideal for the SER task with numerous entity spans. By fully exploiting the weighted focus nature of the attention mechanism, the model eliminates the value transform layer, feed-forward layer in the last attention block, and omits a separate classifier layer compared to traditional fine-tuning methods (as depicted in Figure 1), As a result, the total parameter is reduced.
Extensive experiments are conducted to show the PPTSER’s effectiveness on commonly-used SER benchmarks, covering multiple languages, in fewshot to the full training set settings, and using different mainstream multi-modal pre-trained models.
The main contributions of this paper can be summarized as follows:
• We have demonstrated that the semantics of labels can effectively guide the SER task and have proposed a plug-and-play method ideal for few-shot SER on visually-rich documents. To the best of our knowledge, we are the first
to propose a pluggable method that has shown effectiveness on various pre-trained models and languages.
• By innovatively leveraging the multi-head attention mechanism embedded in the pretrained model, our method successfully extracts dense entities on visually-rich documents without adding any additional parameters.
• Experimental results show the superiority of our method over the traditional fine-tuning approaches in both few-shot and full-training-set scenarios. Moreover, PPTSER outperforms existing few-shot SER methods by significant margin, thereby underscoring its overall efficacy.

heading: 2 Method

heading: 2.1 Task Formulation
text: SER is usually formulated as a sequence labeling task. For given tokens from the document x = [xi], i = 1, 2, ..., n, SER aims to assign a label yi ∈ C for each token xi, where C is the SER label space. Subsequently, entity spans would be analyzed from the labeled tokens according to a specific scheme, such as BIO (Ramshaw and Marcus, 1995) and IO (Tjong Kim Sang and De Meulder, 2003).
In this paper, we primarily focus on the InLabel-Space setting for few-shot SER. Specifically, the pre-trained model is firstly fine-tuned on a small number of M annotated documents with label space C and then directly evaluated on the test set with the same label space C. This task presents a significant challenge as the model needs to learn the SER task with only limited training samples.
It is notable that in the context of few-shot SER, the few-shot setting of N-way K-shot indicates that
each of the N categories has K documents containing entities of that category as the support set, as visually-rich documents are annotated at document level. Moreover, a document often contains entity spans of distinct types, causing potential overlaps between the support sets for different entity types across N categories. Consequently, the overall number of annotated documents M < N ×K.

heading: 2.2 PPTSER
text: The fundamental concept and flow chart of PPTSER is shown in Figure 2. The method begins with the construction of a prompt based on SER tags. This prompt is then concatenated with the document tokens and jointly encoded using a unified pre-trained model. Within the transformer architecture of our model, attention weights between document tokens and the tag-related prompt are computed in hierarchical attention blocks. We use the attention weight between the tag-related prompt and document tokens, which can be considered as a form of cross-attention, obtained from the last attention block as the probability distribution of tokens belonging to different SER entity types.

heading: 2.2.1 Tag-related Prompt Construction and Target Generation
text: For an SER task with the label space C, we need to construct tag-related words c̃i for each ci ∈ C, and then the tag-related prompt C̃ = {c̃i}, i =
1, 2, ...m is built. In PPTSER, we simply use the tag names as the tag-related words.
To enable PPTSER to accurately identify the boundaries of entity spans, we employ BIO tagging scheme in our method. However, when dealing with an SER task involving entity types E = {ei|e0 = Other}, i = 0, 1, 2, ...,m (where Other represents the entities that are not of interest), the label space would be C = {e0, Bei , Iei}, and the prompt would be C̃ = {e0, beginning of ei, inner of ei}, where i = 1, 2, ...,m. In such a scenario, the prompt C̃ becomes not only semantically redundant but also excessively long, potentially impeding the effective semantic learning of the document tokens.
Thus, we reframe the SER task with a BIO tagging scheme into two separate tasks: entity typing and span detection. Entity typing involves assigning an entity type for each document token, while span detection aims to identify whether tokens are at the beginning or interior of an entity span.
To further clarify, consider an SER task using BIO tagging scheme with a predefined entity type set E = {ei|e0 = Other}, i = 0, 1, 2, ...,m. For entity typing, the label space and the tagrelated prompt would be Cent. = {cent.i |cent.i = ei} and C̃ent. = {c̃ent.i |c̃ent.i = cent.i },where i = 0, 1, 2, ...,m; And for span detection, the label space and the prompt would be Cdet. = {cdet.1 , cdet.2 } and C̃det. = {c̃det.1 , c̃det.2 }, where
Cdet. = C̃det. = {beginning, inner}; Then the full label space and prompt would be C = Cent. ∪ Cdet. = {cent.i , cdet.j } and C̃ = C̃ent. ∪ C̃det. = {c̃ent.i , c̃det.j }, where i = 0, 1, 2, ...,m; j = 1, 2. For a token with an entity type of ei(i ̸= 0) located at the beginning/inner of an entity span, the corresponding labels would be ci for entity typing and beginning/inner for span detection. However, for the token with an entity type of Other, the specific location of it within an entity span is irrelevant, and the loss for span detection is ignored here. Consequently, we can formulate the entity typing target yent. = [yent.i ] and the span detection target ydet. = [ydet.i ], where i = 1, 2, ..., n.
It is worth emphasizing that our PPTSER framework handles entity typing and span detection simultaneously. And prompts for them C̃ent. and C̃det. are encoded in parallel, allowing them to benefit from each other during the learning process.

heading: 2.2.2 Cross-attention within the Pre-trained Model
text: Once the tag-related prompt C̃ is constructed, it is concatenated with the document tokens x = [xi], i = 1, 2, ..., n, forming a boosted input x′ = x ⊕
C̃ = [xi, c̃ent.j , c̃det.k ], i = 1, 2, ..., n; j = 0, 1, 2, ...,m; k = 1, 2. Then, x′ is used as the Text Embedding encoded in the pre-trained model. Notably, when C̃ involves other kinds of embedding, such as positional or visual embeddings, they are set to 0, since C̃ are hypothetical tokens not found in the document. Let’s denote the hidden states from the second last block as Ht−1:
Ht−1 = [ht−1i , h̃ t−1 j , h̃ t−1 k ] (1)
where ht−1i , h̃ t−1 j , h̃ t−1 k are the hidden states for x, C̃ent., C̃det., correspondingly. Then, Ht−1 is partitioned into multiple segments Ht−1i along the channel dimension, where queries Qti and keys K t i of the i th attention head
are transformed as follows:
Qti = (W t i )qH t−1 i (2)
Kti = (W t i )kH t−1 i (3)
where (W ti )q and (W t i )k are learnable weights embedded in the last attention block. And the selfattention weight of distinct heads is computed as below:
(W ti )att. = Q t i(K t i ) T (4)
where (W ti )att. is a matrix with the shape of (n+ m + 3) × (n + m + 3). From this matrix, we extract a sub-matrix (W ti ) ′ att. that takes the prompt as queries and the document tokens as keys, which possesses the shape of (m+3)×n. (W ti ) ′ att. can be viewed as a form of cross-attention within the selfattention, which depicts the relationship between the tag-related prompt and document tokens.
We hypothesize that distinct heads of the attention mechanism enable the prompt to focus on distinct entity spans, which is suitable for the entityrich scenario in visually-rich documents. We select the maximum weight across heads to get a summary relationship between the prompt and tokens:
(W t) ′ att. = max
i∈{1,2,...,l} (W ti )
′ att. (5)
Further, (W t) ′ att. is partitioned into two components, namely (W t)ent.att. and (W t)det.att. as shown in Figure 3(a). These components use the hidden states of C̃ent. and C̃det. as queries, and possess the shape of (m+ 1)× n and 2× n, correspondingly. (W t)ent.att. and (W
t)det.att. represent the probability distribution for document tokens belonging to distinct tags. The losses are then calculated as follows:
Lent. = − 1 n
n∑
i=1
exp(went.pi )∑m j=0 exp(w ent. ji )
(6)
Ldet. = − 1 n
n∑
i=1
exp(wdet.qi )∑2 j=1 exp(w det. ji )
(7)
where Lent. and Ldet. are the losses for entity typing and span detection, went.ij and w det. ij are elements in (W t)ent.att. and (W t)det.att. , and y ent. i = cent.p , y det. i = c det. q . And the total loss is formulated as follows:
Loss = Lent. + αLdet. (8)
Here, α is the ratio factor to balance the losses, and we set α = 0.1 for models with segment-level positional embeddings and α = 1.5 for models with word-level positional embeddings.

heading: 2.2.3 Decoding during the Inference Stage
text: The inference stage is shown in Figure 3(b). We first apply the argmax operation on (W t)ent.att. and (W t)det.att. along distinct prompt words to get the predicted tag with the highest probability:
ŷent.i = argmax j∈{0,1,2...,m} went.ji (9)
ŷdet.i = argmax j∈{1,2} wdet.ji (10)
Then the prediction with BIO tagging scheme ŷ = [ŷi], i = 1, 2, ..., n is formulated as follows:
ŷi =    Bŷent.i , ŷ ent. i ̸= Other, ŷdet.i = beginning Iŷent.i , ŷ ent. i ̸= Other, ŷdet.i = inner
Other , ŷent.i = Other (11)
Finally, the entity spans are analyzed from ŷ using the BIO tagging scheme. Notably, spans not conform to BIO scheme, especially those starting with a token predicted as ŷdet.i = inner, are labeled as Other. This operation, aimed at enhancing predicting accuracy, is applied in both PPTSER and methods we compared to for a fair comparison.
To provide a more vivid demonstration of our method, let’s suppose an example from the FUNSD dataset, with the document content "... CASE TYPE: Asbestos ... 82504862", where "..." indicates omitted parts. Here, "CASE TYPE:" belongs to the entity type of question, "Asbestos" to the entity type of answer, and "82504862" to other. Assuming the tokenizer splits the document into "CASE", "TYPE:", "Asbestos", and "82504862", then x1 = "CASE", x2 = "TYPE:", x3 = "Asbestos", x4 = "82504862" and their labels for entity typing and span detection would be yent. = [question, question, answer, other] and ydet. = [beginning, inner, beginning,−1] in Figure 3.
Subsequently, "CASE", "TYPE:", "Asbestos", and "82504862" as document tokens are concatenated with the full tag-related prompt, and form the boosted input x′ = "CASE TYPE: Asbestos 82504862 other question answer header beginning inner". Then, x′ is input into the multi-modal pretrained model as the Text Embedding to obtain the multi-head attention weight and the aggregated attention weight from the last block, as shown in Figure 2.
During the training stage, as shown in Figure 3(a), the aggregated attention weight is split into attention weights between "other question answer header" and document tokens, as well as "beginning inner" and document tokens, which are then used to calculate the losses for entity typing and span detection, respectively, culminating in a combined total loss. As for the inference stage shown in Figure 3(b), we select the document tokens with the highest probability for "other question answer header" and "beginning inner" as ŷent. and ŷdet., then combine them to get the predictions under BIO tagging scheme ŷ following the procedure described previously. And the entity spans are finally analyzed from ŷ, forming the output of { question: "CASE TYPE:" } and { answer: "Asbestos" }.

heading: 3 Experiments

heading: 3.1 Experimental Settings
text: Benchmarks. We conducted experiments on several widely used SER benchmarks, including FUNSD (Jaume et al., 2019), CORD (Park et al., 2019) and XFUND (Xu et al., 2022). FUNSD targets form understanding with 199 scanned documents related to market reports, commercials, and more. CORD, centered on receipt understanding, features both coarse (e.g., menu, total) and finegrained (e.g., menu.unitprice, menu.price) annotations. This benchmark provides an official split of training, validation and test sets, and we strictly follow the procedure by selecting the model weight that achieved the best performance on the validation set for testing on the test set. XFUND focuses on document understanding covering multiple languages. In this article, our primary focus is on the Chinese subset of XFUND, denoted as XFUND-zh.
Few-shot Settings. PPTSER was evaluated on 1-shot, 3-shot, 5-shot, 7-shot and the full training set scenarios. With no official few-shot divisions in benchmarks mentioned above, we established our own following the process in Appendix A. We
selected as few samples as possible while meeting the few-shot setting, which aligns with the realworld application. Due to the inherent instability of few-shot experiments, we randomly generated 5 different divisions for every scenario and tested each with 2 diverse random seeds. Hence, our experiment result is the average of 10 runs, ensuring the reliability and credibility of our findings.

heading: 3.2 Comparisons with Existing Fine-Tuning Methods
text: Setup. The foundation for our method is built upon several widely used multi-modal pre-trained models, incorporating different combinations of modalities as input. This includes BROS (Hong et al., 2022) and LiLT (Wang et al., 2022a) with textual and layout input, and LayoutLMv2 (Xu et al., 2021a) and LayoutLMv3 (Huang et al., 2022b) with textual, layout and image input. Since BROS only supports English, we only tested it on FUNSD and CORD. For testing on XFUND-zh, we used LayoutXLM (Xu et al., 2021b), which is the multilingual version of LayoutLMv2. In our experiments, we utilized base-size pre-trained models.
Results. Table 1 showcases the performance of PPTSER against traditional fine-tuning methods. The results clearly demonstrate that our PPTSER outperforms traditional fine-tuning methods across all tested scenarios and benchmarks. This underscores the superior performance of PPTSER in diverse language contexts with various base models.
Overall, both PPTSER and the fine-tuning method demonstrate improved performance with
increased training data. However, our PPTSER consistently outperforms previous fine-tuning methods in all few-shot settings, especially with exceptionally scarce data. In the 1-shot scenario on FUNSD, where only a single annotated document is available, PPTSER achieves gains of +6.31% with BROS, +3.04% with LiLT, +3.95% with LayoutLMv2 and the highest gain of +15.62% with LayoutLMv3, emphasizing its effectiveness in datascarce situations. Notably, even when trained with the full training data, our PPTSER still achieves comparable performance to the fine-tuning method, and even outperforms it in certain scenarios. For example, we observe a gain of +0.86% on FUNSD with LayoutLMv3. This full data setting is often neglected in other few-shot research, further underscoring the superiority of our approach when dealing with varying amounts of available data.
Our findings demonstrate that PPTSER is highly adaptive to different amounts of training data with distinct base models, making it an effective method for addressing the SER problem.

heading: 3.3 Comparisons with Existing Few-shot Methods
text: Setup. We selected the PPTSER models that performed better under different modality settings, denoted as PPTSERLiLT and PPTSERLMv3, and compared them with previous few-shot methods. For a comprehensive comparison, we re-implemented LASER (Wang and Shang, 2022) on our few-shot divisions. However, it can only handle the coarselevel typing for CORD (CORD-Lv1) and is limited
to the English language. Since research on fewshot SER is rather limited, we selected two other few-shot NER methods for comparison. Specifically, We chose COPNER (Huang et al., 2022a) and EntLM (Ma et al., 2022b) due to their similar In-Label-Space setting with ours. Considering COPNER can also be used as a pluggable method, we also integrated it with LayoutLMv3 and LiLT, denoted as COPNERLiLT and COPNERLMv3.
Results. The overall experimental results are presented in Table 2. The results clearly show that PPTSER outshines existing few-shot NER and SER methods by a large margin. Interestingly, COPNER shows some degree of pluggability with various multi-modal pre-trained models, but PPTSER still outperforms it across all settings and benchmarks.
In summary, our PPTSER surpasses existing few-
shot NER and few-shot SER methods on various visually-rich documents, showcasing its effectiveness in handling few-shot SER challenge.

heading: 4 Ablation Study
text: We have conducted extensive analyses of our PPTSER to ensure its effectiveness and rationality. For convenience, experiments are conducted on the CORD dataset using PPTSER building upon LiLT.
Origin of Attention Weights. To pinpoint the source of superiority in PPTSER, we explored whether it stems from our meticulous design or the decreased over-fitting achieved through parameter reduction. We extracted attention weights from various blocks, including the default 12th block and shallower 7th ∼ 11th block. And the experimental results shown in Figure 4 reveal that extracting attention weights from the last block is more effective than from other blocks, which has greatly assured the effectiveness of our design.
Effectiveness of Decoupling Strategies. Table 3 also shows comparisons with different frameworks of prompts. In this context, default setting refers to our design to decouple the SER task into entity typing and span detection then processing them concurrently, while plain BIO prompt refers to the direct usage of the aforementioned C̃ = {e0, beginning of ei, inner of ei} as the prompt, without decoupling. The result shows that our decoupling avoids disrupting the language modeling of document tokens and performs better.
Prompt Engineering. We evaluated PPTSER using diverse prompt types. Beyond the default setting that uses tag names as prompts, we explored an unrelated words setting by replacing the whole prompts with irrelevant words like apple and orange. In the random embeddings setting, we replaced the whole prompt’s text embedding with random tensors. As Table 3 indicates, the default setting yields the highest score, suggesting the pretrained model does grasp the prompt’s semantics, and tag semantics can direct the SER task. This indicates the careful selection and design of prompts can markedly influence model performance.
Aggregation Strategies of Attention Weights. Table 3 compares how various strategies to aggregate attention weights across different heads affect performance. While default setting and mean refer to obtain the maximum and average value across attention weighs of distinct heads, single head uses just a single head of the attention weights to generate the final probability. The results indicate that the max operation outperforms others, which aligns with our hypothesis that different attention heads focus on entities with different semantics.
Parameter Efficiency. The parameter comparisons of our PPTSER methods and traditional finetuning are presented in Table 4. As the parameters might vary across diverse models and benchmarks, we offer a concise breakdown of the results from the methods we have tested. The results illustrate that our PPTSER has fewer parameters in comparison to traditional fine-tuning methods. For a more detailed analysis, please refer to Appendix E.

heading: 5 Related Works
text: SER on Visually-rich Documents. Although some early works of SER relied on heuristic algorithms (Simon et al., 1997; Schuster et al., 2013), the majority of research focused on neural networkbased methods. Some of them leveraged textual features (Chiu and Nichols, 2016), visual features (Guo et al., 2019), or combined them with layout features (Yu et al., 2021; Wang et al., 2021a) to address this issue, but the emergence of multi-modal pre-trained models has revolutionized SER. These models are jointly pre-trained on a large-scale unlabeled document dataset with textual, layout, and even visual cues, so they have the potential to better understand a structured document. LayoutLM (Xu et al., 2020) was the first to combine textual and OCR positional features at the pre-training stage. Later, LayoutLMv2 (Xu et al., 2021a) and LayoutLMv3 (Huang et al., 2022b) further integrated visual features into the pre-training process with different architectures. Moreover, Wang et al. (2022a) advanced the model architecture with a language-agnostic layout transformer in their work, LiLT. Alongside the advancements in model structures, other works (Appalaraju et al., 2021; Li et al., 2021b,a; Hong et al., 2022; Luo et al., 2023) have focused on the diverse pre-training tasks to facilitate the fusion of diverse modalities at pre-training stage. While these advancements have improved SER capabilities to some extent, their few-shot learning abilities still require further examination.
Few-shot SER on Visually-rich Documents. Unlike SER, few-shot SER is not fully explored
yet. Cheng et al. (2020) proposed to utilize graphmatching techniques (Zanfir and Sminchisescu, 2018), representing documents as graphs with nodes as OCR-scanned boxes. For an unseen document, the type of entities was determined by comparing the relationships in the graph of unseen document with those in the graphs of support documents. Yao et al. (2021) also adopted a graphmatching approach to address this issue, but the entity type was determined based on the relationships in different forms with more complex solvers. Taking a different way, Wang and Shang (2022) introduced a novel labeling scheme for SER. They reshaped SER as a generative task, and used LayoutReader (Wang et al., 2021b) for SER label generation. Although these studies preliminary explored few-shot SER, they lacked generality and pluggability, and their performances in various scenes require further exploration and improvement.
Few-shot NER in Plain Texts. While few-shot SER on visually-rich documents has seen limited exploration, there has been extensive research on few-shot Named Entity Recognition (NER) in plain texts (Wang et al., 2022b; Das et al., 2022; Ma et al., 2022a; Cheng et al., 2023). However, only few of these studies have considered the scenario where only limited data in the target domain is available. Huang et al. (2022a) proposed using the NER tag as a prompt and employing contrastive learning to address this issue. On the other hand, Ma et al. (2022b) reformulated few-shot NER as a Language Modeling task and used the pre-trained Masked Language Model head to predict a word related to the entity type for each text token. However, since these methods are designed to address the NER problem with sparse entities in plain texts, they might not be suitable for entity-rich scenarios in visually-rich documents. Additionally, they do not emphasize the issue of detecting entity boundaries, without which adjacent entities of the same type might be erroneously merged into one. The comparison with two representative few-shot NER methods also show the effectiveness of our method.

heading: 6 Conclusion
text: In this paper, we present PPTSER, an innovative and efficient strategy for few-shot entity recognition on visually-rich documents using a plug-andplay, tag-guided approach. PPTSER redefines the SER task as a dual-function operation of entity typing and span detection, and utilizes the atten-
tion weight between document tokens and prompts related to SER tags as the target probability distributions. Our findings show that PPTSER is both effective and versatile in various data situations, from few-shot to full data scenarios. In the future, we plan to further investigate the capabilities of PPTSER across a range of VIE tasks like Entity Linking. In addition, we aim to explore PPTSER’s potential in other few-shot scenarios, particularly those outside of the In-Label-Space setting. It is our hope that our work will spark further research and advancements in the realm of few-shot SER.
Limitations
Due to space constraints, our exploration of the few-shot SER setting is primarily limited to the In-Label-Space. Future research is essential to investigate the potential applications of our PPTSER in other few-shot settings and its adaptability to additional VIE tasks.
Ethical Considerations
Our proposed PPTSER is a purely methodological innovation, which inherently avoids direct negative social impacts. By leveraging the self-attention mechanism within multi-modal pre-trained models without adding extra modules, it does not introduce additional ethical risks beyond those already present in the existing multi-modal pre-trained models.

heading: Acknowledgement
text: This research is supported in part by National Natural Science Foundation of China (Grant No.: 62441604, 61936003).

heading: A Few-shot Divisions Generation
text: To cater to the real-world application scenarios, we have organized our few-shot divisions from the full training set with Algorithm 1. Our goal was to randomly select the minimum number of documents that satisfy the N-way K-shot requirement of each of the N categories has K documents containing entities of that category as the support set.
It is worth noticing that in the context of fewshot SER on visually-rich documents, the few-shot setting of N-way K-shot signifies that each of the N categories has K documents containing entities of that category as the support set, instead of there are K entity spans for each of N entity types as the support set for the setting of few-shot NER on plain texts.
B Implementation Details
B.1 Implementation Details of PPTSER
We used one NVIDIA 3090 to fine-tune our model with AdamW optimizer. The learning rate is 5e−5 with a warm up ratio of 0.1, and we fine-tuned the model for 2000 iterations with a batch size of 8 by default. Besides the default augmentation strategies for images adopted in LayoutLMv2 and LayoutLMv3, we did not employ any additional augmentation strategies.
B.2 Modification on few-shot NER method for visually-rich documents
In Section 3.3, we mentioned that we adapted two methods originally used for few-shot NER on plain text for few-shot SER on visually-rich documents. We will briefly introduce these modifications.
COPNER (Huang et al., 2022a). The COPNER method employs contrastive learning, feeding both entity label semantics and sentences into a plain text pre-trained language model. This approach uses the hidden state output of the pretrained model to calculate a contrastive loss between sentence tokens and label semantics, then determining the entity type of tokens. However, the original COPNER could only determine if a token belonged to an entity category, without recognizing boundaries between entities. Therefore, we also improved it with the entity typing and span detection framework introduced in our paper. That is, while determining the entity type of tokens, we also input the tokens beginning and inner into the pre-trained model to detect the entity boundary.
The model’s output hidden state is then used to calculate a contrastive loss between sentence tokens and these beginning and inner tokens.
Besides, we retained this core process but replaced the original language model pre-trained on pure text with a multi-modal pre-trained model. Experiments show that our use of multi-head crossattention methods is more suitable for SER tasks on visually-rich documents, especially in Chinese contexts.
EntLM (Ma et al., 2022b). EntLM treats NER as a task of Language Modeling. For testing a few-shot NER dataset on plain text, it first selects a related word for each entity type. Then, using the pre-trained Masked Language Modeling head of BERT, it predicts the probability distribution of each sentence token over these related words, thereby determining the probability distribution of tokens across different entity types.
The selection of related words relies on the distant data obtained from BOND (Liang et al., 2020), which uses BERT and the corpora from Wikipedia to create rough annotations for the NER test set. However, in the realm of visually-rich documents, such rough annotated data is not provided by BOND, and due to the relative abstract expression of SER tags from natural language expressions and the difference between structured documents and natural language expressions, it’s not feasible to obtain rough annotations using corpora from Wikipedia with BERT. Therefore, we directly use the ground truth annotations from the SER test set as distant data to find related words associated with entity types in the SER dataset. Although the experimental results on EntLM might be artificially high due to some exposure to the entity distribution in the test set, our proposed method significantly outperforms others that only accept text modality inputs, including EntLM.
In summary, methods for few-shot NER on plain text may not necessarily transition well to the task of few-shot SER on visually-rich documents. The notable performance of our proposed method in few-shot SER on visually-rich documents further highlights the innovation and contribution of our research.
Algorithm 1: Few-shot Divisions Generation Input: Novel Dataset with the label space C = {c1, c2, ..., cN}, full training set Dfull Output: N-way K-shot few-shot training set Dtrain
1 Dtrain = {} 2 Number of documents that contain entities of ci in Dfull: Q = {c1 : 0, c2 : 0, ..., cN : 0} 3 Document set that contain entities of ci in Dfull: R = {c1 : {}, c2 : {}, ..., cN : {}} 4 for doci in Dfull do 5 for cj in C do 6 if doci contain entities of cj then 7 Q[cj ] += 1 8 R[cj ].append(doci)
9 end 10 end 11 end 12 Q′ = sorted(Q, key = lambda x : x[1]) = {c′1 : n1, c′2 : n2, ..., c′N : nN} (n1 ≤ n2 ≤ ... ≤
nN ) 13 Number of documents that contain entities of ci in Dtrain: S = {c1 : 0, c2 : 0, ..., cN : 0} 14 for c′i in keys of Q′ do 15 for S[c′i] < K do 16 if R[c′i] is empty then 17 break 18 end 19 Randomly select a document doccandidate from R[c′i] 20 R[c′j ].pop(doccandidate) 21 if doccandidate /∈ Dtrain then 22 Dtrain.append(doccandidate) 23 for cj ∈ C do 24 if doccandidate contain entities of cj then 25 S[cj ] += 1 26 end

heading: 27 end
text: 28 else 29 continue 30 end 31 end 32 end

heading: C Further Analysis of Experimental Results
text: This section presents further analysis and additional performance metrics obtained from the main experiments.
C.1 Further Analysis of Comparisons with Existing Fine-Tuning Methods
Table 5a and Table 5b present the precision and recall of PPTSER compared to the traditional fine-tuning method. The results demonstrate that PPTSER consistently outperforms the traditional fine-tuning method in most cases, leading to improved overall performance in terms of F1 scores.
Additionally, we observed from Table 1 and Ta-
ble 5 that the improvement of PPTSER on the CORD dataset is generally less pronounced compared to its performance on other benchmarks. This prompted further investigation on our part.
As mentioned in Section 2.1, our N-way K-shot setting implies that each of the N categories has K documents containing entities of that category as the support set. Given that a single document in the CORD dataset cannot encompass entities from all categories, we selected more than K documents under the K-shot setup in the previous experiments. Consequently, we reselected 1 to 5 document samples as the training set for CORD and conducted additional experiments with LiLT, which showed the most significant improvement, and LayoutLMv2, which showed the least. The results, as illustrated
in Table 6, led to several conclusions:
• Sample Size: The K-shot division on CORD often includes more than K samples. For example, in the 1-shot experiment, our training set averaged 7.6 samples. In contrast, on FUNSD, a single sample typically encompasses entities of all 4 types, resulting in only 1 sample in the training set for the 1-shot scenario. Our findings indicate that as the sample size increases, the performance gap between our method and conventional fine-tuning diminishes, yet our method retains its advantage. Therefore, under the same K-shot setting, CORD involves more samples than other datasets, which leads to a less pronounced improvement.
• Label Complexity: We believe that the labels in CORD are more complex and abstract, making it harder for the model to grasp their semantic meanings compared to those
in FUNSD. For instance, entity categories in FUNSD include header, question, and answer, whereas in CORD, they involve more abstract types like menu.num and total.creditcardprice. The experimental results indicate that when the sample size is extremely small, the improvement on CORD from our method is limited. However, this improvement increases rapidly with the sample size, suggesting our method can more accurately capture the relationship between document tokens and tagrelated prompts with relatively more samples.
• Pre-trained Models: Different pre-trained models have varying degrees of understanding of labels. This could explain why some pre-trained models show weaker improvement on CORD. Our supplementary experiments reveal that the extent of improvement of our method is consistently better with the LiLT than the LayoutLMv2, suggesting that LiLT better understands the semantic information implied by the labels.
C.2 Further Analysis of Comparisons with Existing Few-shot Methods
We also present a detailed comparison of PPTSER with other few-shot methods, including precision and recall metrics in Table 7a and Table 7b. Similar to the F1 score, models enhanced with PPTSER usually demonstrate superior performance compared to both few-shot NER and few-shot SER methods.
CORD
7th block 8th block 9th block 10th block 11th block 12th block
1-shot 68.27 69.63 68.34 68.93 74.27 75.57 3-shot 80.09 81.16 80.81 81.51 82.89 83.83 5-shot 84.29 85.25 85.00 84.84 86.41 87.06 7-shot 84.58 85.86 85.48 85.97 86.79 87.73 Full Data 95.63 95.59 95.48 96.12 95.86 96.04
∆ -3.47 -2.55 -3.03 -2.57 -0.80 -
(a) F1 score (%) of PPTSER on CORD benchmark when obtaining the attention weight from different blocks.
CORD
7th block 8th block 9th block 10th block 11th block 12th block
1-shot 67.36 69.02 67.86 68.53 73.91 75.35 3-shot 79.76 81.19 80.78 81.46 82.76 83.85 5-shot 80.98 85.38 85.01 84.78 86.22 87.03 7-shot 84.41 85.92 85.52 85.97 86.68 87.74 Full Data 95.70 95.64 95.55 96.16 95.87 96.06
∆ -4.36 -2.58 -3.06 -2.63 -0.92 -
(b) Precision (%) of PPTSER on CORD benchmark when obtaining the attention weight from different blocks.
CORD
C.3 Further Analysis of Attention weights obtained from different blocks
We additionally provide the numerical metrics of distinct settings to obtain the attention weight from different blocks. Table 8 illustrates the experimental results, indicating that obtaining attention weights from the last block yields the best performance of F1 score, precision, and recall. Although the reduction of parameters alleviates over-fitting to some extent, since some shallower blocks outperform certain deeper ones in the 1-shot scenario, our default setting to obtain the attention weight from the last block significantly outperforms the alternative settings of obtaining the attention weight from shallower blocks. This finding strongly reinforces the effectiveness of our design.
C.4 Further Analysis of Different Designs on PPTSER
Besides, we also offer the precision and recall of our PPTSER under different designs in Table 9a and Table 9b. The outcomes clearly indicate that our PPTSER, characterized by its meticulous design, outperforms other designs across all metrics evaluated, including the F1 score, Precision, and Recall. This superiority not only showcases the robustness of our design but also significantly substantiates the efficacy of our PPTSER framework in tackling the few-shot SER problem on visually-rich documents.

heading: D Comparisons with Multi-modal Large Language Model
text: To further explore the superiority of our approach, we conducted few-shot SER experiments with current mainstream Multi-modal Large Language Models. We selected GPT-4Vision (Achiam et al., 2024) and LLaVA-1.6 (Liu et al., 2023). Since GPT-4Vision is not open-source, we employed InContext Learning (Brown et al., 2020) for few-shot learning. For the open-source LLaVA-1.6, we directly fine-tuned it with a small number of samples. The experiments were conducted on the FUNSD benchmark. Due to the dialog length limitation of GPT-4Vision, we only tested its performance in 1-shot and 3-shot scenarios. As shown in Table 10, our results surpass those of the two Multi-modal Large Language Models, further demonstrating the effectiveness of our proposed method on dedicated models.
E In-depth Parameter Analysis of PPTSER over Traditional Fine-tuning
We provide a further analysis of the parameter counts in this section. As shown in Table 4, our PPTSER maintains consistent parameters across different benchmarks with the same pre-trained model. This is attributed to the fact that PPTSER does not necessitate the use of an extra crossattention module in previous works (Dai et al., 2023) that employ cross-attention, and does not require an additional classifier layer, unlike the traditional fine-tuning method. Besides, the parameter variance arises when employing the traditional finetuning method with the same pre-trained models on different benchmarks, owing to variations in the number of entity types present in those benchmarks. Furthermore, as PPTSER omits the value transform layer and the feed-forward layer in the final attention block, we achieve a reduction in the parameter count of the pre-trained model it is based on. Additionally, the extent of parameter reduction varies among different pre-trained models due to disparities in their architectural designs, resulting in slice differences in the eliminations of the modules.

heading: Future Work
text: In the future, we plan to further investigate the capabilities of PPTSER across a range of VIE tasks like Entity Linking. In addition, we aim to explore PPTSER’s potential in other few-shot scenarios, particularly those outside of the In-Label-Space setting. It is our hope that our work will spark further research and advancements in the realm of few-shot SER. Limitations
Due to space constraints, our exploration of the few-shot SER setting is primarily limited to the In-Label-Space. Future research is essential to investigate the potential applications of our PPTSER in other few-shot settings and its adaptability to additional VIE tasks. Ethical Considerations
Our proposed PPTSER is a purely methodological innovation, which inherently avoids direct negative social impacts. By leveraging the self-attention mechanism within multi-modal pre-trained models without adding extra modules, it does not introduce additional ethical risks beyond those already present in the existing multi-modal pre-trained models