On Task-personalized Multimodal Few-shot Learning for Visually-rich Document Entity Retrieval
Authors: Jiayi Chen
Hanjun Dai
Bo Dai
Aidong Zhang
Wei Wei
Venue: EMNLP 2023 Findings (2023)
Keywords: document understanding
multiple modalities
entity retrieval
few shots
meta learning
out of distribution
TL;DR: New document types come out at a constant pace and each of them have a unique set of entity types leave us a unique challenge: a large amount of documents contain unseen and imbalanced entity types that occur only a couple of times.

Abstract: Visually-rich document entity retrieval (VDER), which extracts key information (e.g. date, address) from document images like invoices and receipts, has become an important topic in industrial NLP applications. The emergence of new document types at a constant pace, each with its unique entity types, presents a unique challenge: many documents contain unseen entity types that occur only a couple of times. Addressing this challenge requires models to have the ability of learning entities in a few-shot manner.  However, prior works for Few-shot VDER mainly address the problem at the document level with a predefined global entity space, which doesn't account for the entity-level few-shot scenario: target entity types are locally personalized by each task and entity occurrences vary significantly among documents. To address this unexplored scenario, this paper studies a novel entity-level few-shot VDER task. The challenges lie in the uniqueness of the label space for each task and the increased complexity of out-of-distribution (OOD) contents. To tackle this novel task, we present a task-aware meta-learning based framework, with a central focus on achieving effective task personalization that distinguishes between in-task and out-of-task distribution. Specifically, we adopt a hierarchical decoder (HC) and employ contrastive learning (ContrastProtoNet) to achieve this goal. Furthermore, we introduce a new dataset, FewVEX, to boost future research in the field of entity-level few-shot VDER. Experimental results demonstrate our approaches significantly improve the robustness of popular meta-learning baselines.

On Task-personalized Multimodal Few-shot Learning for Visually-rich Document Entity Retrieval
Jiayi Chen',, Hanjun Dai', Bo Dai2.4, Aidong Zhang', Wei Wei3.40
University of Virginia, Georgia Institute of Technology, 'Aceetture "Google, Mountain View, USA aidon) (oaogg))vviiiiiaia @virginia.edu, [hadai,bodai @google.com, wet.hweee@aceettreeee
Abstract
structured and organized information. As sub- problem of VrDU, the goal of Visually-rich Doc- ument Entity Retrieval (VDER) is to extract key information (e.g.. date, address, signatures) from the document images such as invoices and receipts with complementary multimodal information (Xu et al., 2021: Garncarek et al., 2021; Lee et al., 2022). In real-world VDER systems, new docu- ment types continuously emerge constant pace, each with its unique entity spaces (i.e., the set of entity categories that we are going to extract from the document). This poses a substantial challenge: a large amount of documents lack sufficient annota- tions for their umique entity types, which is referred to as few-shot entities. To tackle this, Few-shot Visually-rich Document Entity Retrieval (FVDER) has become crucial research topic.
Visually-rich document entity retrieval (VDER), which extracts key information (e.g- date, address) from document images like invoices and receipts, has become an important topic in industrial NI.P applications. The emergence of new document types at constant pace, each with its unique entity types, presents unique challenge: many documents contain unseen entity types that occur only couple of times. Addressing this challenge requires models to have the ability of learning entities in a few-shot manner. However, prior works for Few-shot VDER mainly address the problem at the document level with predefined global entity space, which doesn't account for the entity-lvvel few-shot scenario: target entity types are locally personalized by each task and entity occurrences vary significantly among documents. To address this unexplored scenario, this paper studies a novel entity-level few-shot VDER task. The challenges lie in the uniqueness of the label space for each task and the increased complexity of out-of-distribution (OOD) contents. To tackle this novel task, we present a task-aware meta- leattlllannnn based framework, with central focus achieving effective lask personalization that distinguishes between in- task and out-of-task distribution. Specifically, we adopt hierarchical decoder (HC) and employ contrastive learning (Contras(ProtoNet) achieve this goal. Furthermore, we introduce a new dataset, FewVEX, to boost future research in the field of entity-level few-shot VDER. Experimental results demonstrate our approaches significantly improve the robustness of popular meta-learning baselines.
Despite the importance of FVDER, there has been limited amount of prior works in this area. Recent efforts have leveraged pre-trained language models (Wang and Shang, 2022) or prompt mecha- nisms (Wang et al., 2023b) to obtain transferable knowledge from a source domain and apply it to a target domain, where a small number of document images are labeled for fine-tuning. These prior works address the few-shot problem in granularity of document level, assuming a globally predefined entity space and balanced entity occurrences across documents. However, certain real-worle scenar- ios, the few-shot challenge can also manifest at the entity level-i.e., the number of entity occurrences in labeled documents are limited, assuming the situations where entity classes might be locally spe- cialized by each user (task) and their occurrences maintain significant imbalance across documents. In such scenarios, prior methods struggle to (1) ef- ficiently address the model personalization on each task-specific label space and (2) effectively han- dle the increased complexity out- distribbbit contents, with few-shot entity annotations.
1 Introduction
Visually-rich Document Understanding (VrDU) aims to analyze scanned documents composed of
Work done when Jiayi Chen interned at Google. "Corresponding author
To provide a complementary research perspec-

---

Method
Instance Granularity
Unseen Entities?
Entity Space
Entity Occurrence
(Wang and Shang, 2022)
document level
yes
globalized
balanced
(Wang et al., 2023b)
document lovel
no
globalized
balanced
Ours
entity level
yes personalized
imbalanced
Table I: Comparison on task formulations and application scenarios.
tive alongside the existing document-level work, in this paper, we initiate the investigation for the unex- plored entity-level few-shot VDER. To begin with, we formulate an N-way soft-K -hot VDER task setting together with distribution of individual tasks, which simulates the application scenario of such task, that is, the user or annotator of each few- shot task is only interested in N personalized entity types and the number of labelled entity occurrences in a task is within a flexible range determined by K' shots, Table summarizes the differences in ap- plication scenarios between prior works and ours.
sample across the training documents in a way that guarantees to return soft-balanced few-shot anno- tations. Training documents are constructed in a way such that they cooperatively contain certain number of entity occurrences per information type.
Our contributions are summarized as follows. (1) To the best of our knowledge, this paper is the first attempt studying the Few-shot VDER prob- lem at the entity level, providing a complemen- tary research perspective in addition to the existing document-level works. (2) We propose a meta- learning based framework for solving the newly introduced task. While vanilla meta-learning ap- proaches have limitations this task, we propose several task-aware meta-Icarners to enhance task personalization by dealing with out-of-task distri- bution. (3) Experiment results on FewVEX demon- strate our proposed approaches significantly im- prove the performance of baseline methods.
Then, to tackle the limitations of prior methods on this new task, adopt meta- veta-learning based framework build upon pretrained language models, along with several proposed techniques for achiev- ing task personalization and handling out-of-task distribution contents. With the help of the meta- learning paradigm, (1) the learning experiences on some example tasks could be effectively utilized and (2) the domain gap between the pre-trained model and novel FVDER tasks is largely reduced, promoting quicker and more effective fine-tuning on future novel entity types. Yet we found pop- ular meta-learning algorithms (Finn et al., 2017: Snell et al., 2017; Chen et al., 2021) are still not ro- bust to the entity-level N-way soft- Khoho VDER tasks. The difficulty is that the background context that does not belong to the task pesk-personllzedd entity types occupies most of the predictive efforts, and also, such noisy contextual information varies lot across tasks and documents. To address this, we propose task-aware meta-learning techniques (ContrastProtoNet, ANIL+HC, etc.) to allow the meta-Learners to be aware of those multi-mode con- textual out-of-task distribution and achieve fast adaptation to the task-personalized entity types.
Entity-level Few-shot VDER Setting
General VDER. A document image is processed through Optical Character Recognition (OCR) (Chaudhuri et al., 2017) to form sequence of tokens X [x1, x2, X xL], where L is the quence length and each token x is composed of multiple modalities x [x| x x1 .... such as the token id (v), the Id position (p) of the to- ken in the sequence, the bounding box (b) rep- resenting the token's relative 2d position, scale in the image, and so on. The goal to predict Y [V1, .2........................ WL]. which assigns each token X a label yI to indicate either the token is one of enti- ties in a set of predefined entity types or does not belong to any entity (denoted as O class).
(v)
(b)
N-way Soft- -hot VDER in Entity Level. In real-world Few-shot VDER systems with label scarcity, individual users often show their personal interests in a small number of new entity types. Such user-dependent task personalization gives rise a novel entity-level task formulation of Few- shot VDER. Formally, an entity-level N-way soft- K-shot VDER task T = {S, E) consists of a train (support) set S containing M, documents, a test (query) set Q containing M,, documents, and a
Furthermore, we present new dataset, named FewVEX, which comprises thousands of entity- level N-way Soft-KSshot VDER tasks. We also in- troduce an automatic dataset generation algorithm. XDR, designed to facilitate future improvement of Few VEX, such as the expansion of the number of document types and entity types. Specifically, we set an upper bound for entity occurrences and

---

Meta-Traie Tasks
out-of-task entities background
paddlnggs
Collection
Ti
Task-specific Entity Types
Training Documents
entity type
entity type
entity type
T2
Support (Training) Set
Task
Doc
Generator
T
Base Types
Doo 5
Tr...
tokans
X1, X2,
Xas
Y1, Y2, YAu
T
Query (Validation) Set
Entity Types Raw Dataset
Tj
Doo
Task Generator
Doc
Novel Types
Doe
Collection of Testing Documents
T
Doc
Meta-Test Tasks
X", X
X*Ma
Y"1, Y'2
Y'Ms
Figure 1: Proposed task formulation and problem setting. Different colors represent different entity types. The pie chart split on the left indicates that the targel classes in testing tasks not seen in training tasks. On the right area, we show an example 3-way soft-2-shot task. In this example, = 2.
target class set containing N target entity types
one of & (lask-personalizer entity types) or (back- ground or out-of-task entity types), based on the few labeled entity occurrences for those & support set S, such that the model achieves high performance on the query set Q.
S [(X1, Yi), ................................ (XM, M, YM,)) Q [X], X _
X
(1)
E {el, 62 ,,,,..,,..Il eN
where X, [xj1. Xy2,..., xjr. is the sequence of multimodal token features of document j. Y; = [y,, U)2,-..,yyII, УリΣ] is the sequence of token labels corresponding to Xj. and cg denotes the c-th entity type in T. "N-way" refers to the N unique entity types the user is interested in, reflecting task person- alization. It is important to highlight that within S and Q documents, there may exist entities that fall outside the N target classes (e' & &). These entity types come from the distt--iittiiiution in contrast to what the task T aims to train on, which do not attract user interest, remain unlabeled, and thus are treated as the background O class. "Soft-K "shot" refers that, among the M, labelled documents in S, the total number of occurrences of each entity type E & is within range K pK, where p is the softening hyperparameter. An entity occur- rence is defined document with the same entity type as labels. We do not impose a strict constraint on the exact count K sicne the entity-level personalization scenario implies that the frequency of entity occurrences may vary dramatically from one document to the other, which makes it difficult to set strict limit. For instance, an entity type may occur more fre- quent in some documents and less so in others. The right area of Figure| shows an example N-way soft- -hot VDER task. The goal of task T is to obtain a model that assigns each token as either
Distribution over FVDER Tasks. Based on the above formulation for a single FVDER task, we further formulate a task distribution P(T) over FVDER tasks. Assume there is large pool of en- tity types C corresponding to the domain of P(T). For any task Tj = {S,, Ei P(T). its target entity types come from the class pool & C.
Global Objective. The global objective is to train meta-learner for P(T) such that any task Tú P(T) can take advantage of it and then quickly obtain a good task-personalized model. Follow- ing (Finn et al., 2017; Chen al., 2021), train the meta-learner, we simulate a meta-level dataset consisting of example FVDER task instances from P(T). Figure shows an overview of the dataset simulation of P(T). Specifically, a leeanee is trained from the experiences of solving a set of meta-training tasks Dirn Peta over set of base classes Class C C. where each training task is from the base classes & Crase. The experiences are given in the form of the ground truth labels of query sets. That is, the query sets of training tasks are treated as valida- tion sets, Q {(}), Y Me for VT: E Dtrn To evaluate the performance of the meta-learner on solving FVDER tasks with novel entity types Cnovel = C\ Chases we will individually train a set of meta-testing tasks Deest meta = (Ti,, Tr.., Taa
contiguous subsequence in the
(T,, T2...

---

in-tusk-distributiion (ITD) entities (Iubeled)
ouf-of-aaak-disttibutoo (OTD) entities / background (laheled)
in-task-distribution (ITD) entities (unlabeled)
out-o--iak--distibbttoo (OTD) entities / background (unlaheld)
P(T)
MetaePaarameterss
sampting
docamant
Tio
Sa
loken
A
class
(traii)
position
E-1
clas-2
position
Preds
6ocumer
Qi (best)
other features
(attention mechanism in each document)
Task
Multimodal Transformer
Task-dependent Embedding Space
Token Labelling
Figure The proposed task-aware meta-learning framework. framework applicable to both the metric-based method (aiming to learn σ) and gradient-based method (aiming learn (ó, b)))
where each testing task Er C. Cnovel. The query sets of meta-testing tasks are unlabelled testing data, that is, Q - (X5] M VT," E Direst meta-
is computed as huy - - (X:ji (xa) In practice, before meta-training, the multimodal Transformer is pretrained on the ITTCDIIP dataset (Harley et al., 2015). Details can be found in Appendix C.I.
3 Methodology
3.2 Task-dependent Embedding Space
We propose a meta-learning (i.e., learning-to-leamm) framework to solve the entity-level few-shot VDER tasks. Different from the recent advancements based on pre-traiie pre-training prompts (Wang and Shang, 2022; Wang al., 2023b), meta-learning helps to significantly promote quick adaptation and improve model personalization on task-specific entity types. The proposed framework consists three com- ponents: (1) a multimodal encoder (Section 3. 1) that encodes the document images within a task into a task-dependeut embedding space (Section 3.2); (2) a token labelling function (Section 3.3); and (3) meta-learner built upon the encoder- decoder model, where propose two task- personalized meta-leaming methods (Section 3.4). Figure shows an overview the framework.
Through the multimodal encoder, each task Ti is en- coded to pask-dependen embedding space. As il- lustrated in Figure 2, on the task-dependent embed- ding space, there are all the token embeddings in the task: H; (hall / [Z], (X,, Yj) S; Qi). There are several properties on the task's embed- ding space: (1) First, in addition to in-task distri- bution (ITD) entities from the target classes, there exists a large portion (nearly 90% as observed in our dataset FewVEX) of out-of-task distribution (OTD) entities background, which serve the context for ITD entities but dominate the task's embedding space. (2) Second, the OTD entities fol- lows multi-mode distribution poTn that consists of several unimodal distributions, each of which represents an outlier entity type aside from ITD. (3) Finally, it is not guaranteed that each unimodal component of poTo is observable in the train set S,-in many cases, an OTD entity type could occur in the query documents but is absent in the support documents. To sum up. the OTD distribution in a N N-way K-shot FVDER task is complex, dominates the entire task, and may vary between documents.
3.1 Multimodal Encoder
We consider an encoder network represented by a parameterized function fene with parameters Φ. The encoder aims to capture the cross-modal se- mantic relationships between tokens in a document image. To achieve this, we employ BERT-base Transformer (Kenton and Toutanova, 2019) with additional positional embedding layer for the 2d position of each input token, through which the complex spatial structure of the input document can be incorporated and then interacted with the textual contents via attention mechanisms. The embed- ding of token / in the document image j of task T
3.3 Token Labelling
On the basis of the task-dependent embedding space, the token labelling or decoding process can either leverage parameterized decoder f that acts as the classification head, or rely on non-

---

parametric methods, like nearest neighbors.
Meta Contrastive Loss. During meta-training, we encourage the N ITD entity types to be distin- guished from each other as well as far away from any unimodal component of OTD. To achieve this, we adopt the idea from supervised contrastive learn-
3.4 Task-aware Meta Learners
We consider two main categories of the meta- learning approaches: the gradient-based and the metric-based meta-leaming, on each of which we propose our own methods. We specifically pay attention to two properties when solving the entity- level N-way K-shot FVDER tasks: 1) Few-shot out-of-task distribution detection, which aims to distinguish the ITD (i.e., the target N entity types) against the OTD (i.e.. background or any outlier entity type). 2) Few-shot token labelling for in- task distribution tokens, which assigns each ITD token to one of the N in-task entity types.
ing (Khosla et al., 2020) to compute a meta con- trastive loss (MCON) from each task, which will be further used to compute meta-gradients for updat- ing the meta-parameters Φ. Intuitively, our meta- objective is that the query tokens from the ITD type-e should be pushed away from any OTD to- kens and other types of ITD tokens within the same task, and should be pulled towards the prototype Mi, of support tokens and the other query tokens belonging to the same entity type. Formally, let TY ((,1))z [L] (X.YY)) E Qi Vig E & denote collection of ITD validation tokens. The meta contrastive loss computed from is is
3.4.1 Task-aware ContrastProtoNet
We first focus on metric-based meta meeaaleerning (Snell et al., 2017; Oreshkin et al., 2018). The goal is to learn meta-parameters & for the encoder network, generally shared by all tasks T; P(T), such that, on each task's specific embedding space, the dis- tances between token points in S; and Q; are mea- sured by some metrics, c.g., Euclidean distances.
MCCON
Σ
Σ
(sa(v)
(T
VCAT VCATGN)
(2)
avgi(v) = log exp(b.ziv)
ΣucAGiA exp(h.
ProtoNet with or without Estimated OTD. One of the most popular and effective metric-based meta-learning methods is the Prototypical Network (ProtoNet) (Snell al., 2017). For each FVDER task T, = (S,, Qi,&) the prototype for each en- tity type e E E can be computed as the mean embedding of the tokens from S₁ belonging to that entity type, that is, Mi,e- = 1/14 Eiinee higt where Itm is collection of the token indices for the type-c tokens in the support set. For the out- of-task distribution (OTD), one may consider to estimate its mean embedding as an extra O-type prototype: p4; 1/ Eg,neerg hig--
For each anchor, i.e., the ITD validation token in document j. we let A (j,1) - { (r, )) E raal \\ ((i,))). Viy - -trma U {Meele E & Vist e} denote a collection of the positive embed- dings/prototypp for the anchor and let A(i,1) [hammff (r, m) JALL (1) U {MLedees, con- tain all the ITDD/TDD embeddings and prototypes (ALL = (Gi ()]l [L], (X,, Yj) S, Q()) in Te
Unsupervised OTD Detector. During the test- ing time for novel entity types, we adopt the non- parametric token-level nearest neighbor classifier, which assigns Xiji the same label as the support token that is nearest in the task's embedding space:
Challenges. A problem of the vanilla methods is that there no specific mechanism distinguishing the ITD entities against the OTD entities, which are weakly-supervised and partially observed from a multi-mode distribution poTD The prototype ze, is a biased estimation of the mean of poTo and the covariance of poTo can be larger than any of the ITD classes. In consequence, the task-specific ITD classes may not be clearly distinguished from the OTD classes on the task-dependent embedding space and most of tokens will be misclassified.
G = argmax, Dirm where (r.))E I
h high hirm: (3)
where I = {(r,, m)lm E (X,, E S.). The ITD or OTD entity tokens in Qi should be closer to the corresponding ITD or OTD tokens in S; that belong to the same entity type. However, since the embedding space dependent on the sup- port set is not sufficiently rich, the network may be blind to properties of the out-of-task distribution poTd that turn out be necessary for accurate en- tity retrieval. To tackle this, we exploit an unsuper- vised -iit--i-ittibutionn detector (Ren et al., 2021) operating on the task-dependent embedding space,
Regarding the above challenges, we propose a task-award aasee method that adopts two techniques to boost the performance.

---

in assistance with the classifier. Specifically, we de- fine an OTD detector: Vigs O rhijt) 2 R;; oth- erwise, figt = o2 where R2 is the task-dependent uncertainty threshold and r(h,j) is defined as the OTD score of each token computed as its minimum Mahalanobis distance among the N ITD classes: r(hiat) minlee, (hiza Mi,e) 07 0. Mr,e). Here, Sh, Eo.nerr (batt- - (ha - pi,e) is the covariance matrix for entity type e computed from the type-e tokens in the support set m) The higher OTD score indicates the more likely the token belongs to the background.
against OTD ones, and 2) entity classifier fent so that ITD tokens are classified one of the N en- tity types of the task. Specifically, suppose pon and p! denotes the OTD and ITD of the task Ti, respectively. The probability that the token h:j is from OTD is denoted as P(yy = 0) fo (hi), which is used as the OTD score to weight the en- tity prediction. The probability that the token is the entity type-e is computed as P(thi - e|xa, E P.TT) = (1 P(yig = ff (arggggnhhh (hzzte-
4 Few Dataset
There is no existing benchmark specifically de- signed for task-personalized Entity-level N-way Sshot VDER. To facilitate future research on this problem, we create a new dataset, FewVEX.
3.4.2
Computation-efficien Gradient-based Meta-learning with OTD Detection
For gradient-based meta learning, the goal is to learn the meta-parameter 0 = [db } globally shared over the task distribution P(T). which can be fast fine-tuned for any given individual task T,.
FewVEX is built from two source datasets: FUNSD (Jaume et al., 2019) con- tains images of forms annotated by the bounding boxes of 3 types of entities; CORD (Park et al., 2019) contains scanned receipts annotated by 6 su- perclasses which are divided into 30 fine-grained subclasses. From them, we collect 1199 document images (D) annotated by 26 entity types (C).
Source Collection':
Computation--fficient Meta Optimization. Al- though MAML (Finn et al., 2017) is the most widely adopted approach, the fact that it needs to differentiate through the fine-tuning optimization process makes it a bad candidate for Transformer- based encoder-decoder model, where we need to save a large number of high-order gradients for the encoder. Instead, we consider two alternatives which require less computing resources and more efficient. ANIL (Raghu et al., 2019) employs the same bilevel optimization framework MAML but the encoder is not fine-tuned during the inner loop. The features from the encoder are reused in different tasks, to enable the rapid fine tuning of the decoder. Reptile (Nichol et al., 2018) is a first-order gradient based approach that avoids the high-order meta-gradients. To further boost training efficiency, we exploit Federated Learning (Tian et al., 2022; Chen and Zhang, 2022a) for meta-optimization of Transformer.
Meta-learning Tasks: We use D and C. to construct VEX, represented by Dmeta {Dima meta Dist such that the testing tasks Dist focus on novel classes that are unseen in Dirn during meta-training, To create this, we split C into two separate sets = Crase U Crovel Crase N Cnovel = 0, where Chase is used for meta- veeta-training and Crowel for meta-testing.
Following the defini- tion in (1), each individual entity-level N-way soft- -hhot VDER task T (SS E) in either Dirn meta or Dist meta can be generated through the fol- lowing steps. (1) Pess-perso Class sam- pling. The task's target classes E are generated by randomly sampling N entity types from either Close (for the training task) or Cnovel (for the testing task). (2) Document sampling. Given the N tar- get classes, we then collect document images that satisfies the N N-way, soft K-shor entity occurrences (as in Appendix 1). (3) Annotation Conversion. A task only focuses on its specific N rarely-pressee entity types. The entities in the original annotated documents, whose class do not belong to E, are replaced with the background class?
Single Task Generation:
Task-aware Hierarchical Classifier (HC). A vanilla classifier can achieve high performance in the label-sufficient VDER. However, it turns out to be not robust in few-shot FVDER tasks because of the existence of the complicated out-of-task ehtities-the models usually either get overconfi- dent on the N IID entity types fail to distinguish target entities from the OTD background. For this reason, we incorporate OTD detection into the de- coder and propose hierarchical classifier, which has two classifiers &o = [11,, ) 1) binary clas- sifier phin so that all ITD tokens are classified
I Due to page limit, details are moved to Appendix B. Due to page limit, details can he found in Appendix B.2.2.

---

Meta Training (from Couse)
Meta Testing (from Crovel)
Range
Datasets
Domains
Entity Types
Tasks
Domains
Entity Types
Tasks
of N
VexV)
CORD
18
3000
CORD
128
[].5]
FewVEX(M
CORD+FUNSD
20
3000
CORD+FUNSD
256
[1. 6]
Table 2: Statistics of two variants of Few VEX. From each dataset, we can test different N-way K K'shot settings.
Proposed Datasets: We construct two variants of FewVEX, Ve)(EX(SS) focuses on single-domain receipt understanding, where Crase and Cnovel are split from the 23 entity types CORD. FewVEX(M)) focuses on a combination of receipt and form domains, where Cluss contains 18 classes from CORD and 2 from FUNSD; Gnovel contains the other 5 classes in CORD and 1 in FUNSD. The statistics of Few is summarized in Table 2.
well the trained meta-learners can distinguish in- task distribution (ITD) from out-of-task distribu- tion (OTD) for any novel given task, we plot ROC curves and calculate AUROC (Xiao et al., 2020) using the ITD scores over meta-testing tasks. A ran- dom guessing detector outputs an AUROC of 0.5. A higher AUROC indicates better TS performance.
5.1 Main Results
Future Extension: While CORD and FUNSD currently serve the source datasets for FewVEX VEX, we anticipate that future enhancements such ex- panding the number of entity types ((C)) and diver- sifying documents ((D)) will lead to better version of Few VEX. We introduce Crose-document Re- jection (XDR) sampling to facilitate this improve- ment. XDR samples the train/test documents of each task in way such that cooperatively ensures a specific range of entity occurrences per class, which mimics real-worke user annotation behaviors motivated by class-balanced requirements. In the future, with access to open-souree VDER datasets containing a wide array of classes and documents, XDR will enable the automated generation of nu- merous distinct task simulations. The pseudocode of XDR is shown in Algorithm 1 in the appendix.
Table reports the results Few VewVE.(S). Under the same N and K setups, traditional meta-learning methods fail balance the precision and recall performances: ANIL and Reptile using vanilla de- coders achieved high precision but tended to per- form low recall; the vanilla Prototypical Networks tended to be opposite: low precision but high recall. In contrast, ANIL+HC, Reptile+HC and Contrast- ProtoNet, achieved better precision-recall balance and thus higher FI scores and TS, proving that de- tecting and alleviating the influence of out-of-task distribution can improve task personalization and accuracy. Such phenomenon also illustrated Figure 3 and Figure in Appendix D.2, where we plot ROC curves and tSNE visualizations of token embeddings after task adaptation. Comparing our methods against baselines, we observe an elevation in the curves and more distinct boundaries between OTD and ITD and between ITD classes.
5 Experiments
The reasons are as follows. First, ANIL and Reptile treat the dominant OTD instances as an extra class as well. The problem turns out the im- balanced classification in meta-learning, one of the challenges in few-shot VDER tasks. By using an OTD detector, ANIL+HC and Reptile+HC can faster adapt to the task-specific boundary between OTD and ITD. Overall, this potentially increase the recall and task specificity score and the overall F1 score. Second, for the vanilla metric-based meth- ods, where OTD instances are treated one extra class, the ITD testing instances tend to be close to ITD class centers so that we have high recall. However, OTD instances dominate the task. It is possible that some OTD testing instances are closer to ITD centers than the OTD class center (the aver- age center of multiple OTD classes) so that most
Setups: We compare the proposed framework with aforementioned meta-learning baselines on Few VEX. Data generation and methods are imple- mented using JAX and Tensorflow. All experiments ran on 32 TPU devices. We use the Adam opti- For gradient
mizer to update the meta-parameter based methods, we use vanilla SGD for the inner- loop optimization and fix 15 SGD updates with constant learning rate of 0.015. Setup details and hyperparameters are available in Appendix C.4.
Evaluation Metrics: We consider two types of
quantitative metries. (1) Overall Performance: following (Xu al., 2020), we use the precision (P), recall (R), and micro F1-score over meta- testing tasks to measure the accuracy of entity re- trieval. (2) Task Specificity (TS): to evaluate how

---

4-way 1-shot
4-way 4-shot
5-way 2-shot
Methods
Overall
TS
Overall
TS
Overall
TS
P
F1
AUROC
AUROC
R
F1
AUROC
ProtoNet
0.02
0.10
0.03
NA
0.02
0.09
0.03
N/A
0.02
0.09
0.03
N/A
ProtoNe++EOD
0.13
0.47
0.21
N/A
0.11
0.58
0.23
N/A
0.11
0.35
0.17
N/A
ContrasdProoNNet
0.54
0.47
0.59
0.61
0.59
0.89
0.41
0.62
Reptile
0.48
0.10
0.15
0.58
0.62
0.44
0.51
0.67
0.39
0.09
0.14
0.59
ANIL
0.39
0.19
0.25
0.56
0.54
0.44
0.87
0.35
0.13
0.19
Reptile+IIC
0.35
0.20
0.63
0.63
0.65
0.98
0.34
0.12
0.18
0.65
ANIL+HC
0.40
0.58
0.50
0.95
0.47
0.59
0.51
0.98
0.38
0.56
0.46
0.92
Table 3: Performance on 4-way 1-shot, 4-way 4-shot, and 5-way 2-shot settings of FewVEX(S).
ITD v.s. OTD Entity Types
it hard identify in-task entities. Further, try to understand the disentangled structure of classes from the clusters. the right column in Figure 3, we zoom into the four ITD classes, where purple, red, blue, and green points denotes the task-specific four entity types, respectively. We observe that "menu (sub_uniprice)" (violet) is far away from the other three classes, while the other three classes slightly entangled. Such class structure repre- sents the relationships between these entity types, which is explainable: the red and blue classes be- long to the same superclass sub_total; ; the green and red are both etc information.
5.3 Multi-domain Few-shot VDER
Figure 3: (SNE visualization of the learned embedding space for randomly-selected meta-testing task, com- paring (a) vanilla ProtoNet and (b) ContrastProtoNet methods, under the 4-way 4-shot setting of Few VEX(S).
Table 4 reports the 4-way 2-shot results the mixed-domain Few VEX(M), which combines re- ceipts with forms for few-shot learning. The re- sults slightly underperforn those under the single- domain setting. reason could be that the structure of forms is different from that of receipts and it is challenging to find the good meta- perameeasss for both domains. Moreover, the number of classes in the form domain is much smaller than that in the receipt domain. Such imbalanced class combina- tion would push the meta- eettaparameters adapt the relative prominent domain.
of them are misclassified as one of ITD classes, i.e., low precision. opposite, ContrastProtoNet does not make any assumption on the OTD distribu- tion; instead, we enforce OTD to be far away from ITD classes and classify via token-level similarities while considering prohabilistic uncertainty.
5.2 Class Structure Disentanglement
Methods
F1
AUROC
We examine the explanability and disentanglement of the learned representations (generated by the meta-parameters of encoder). Figure 3 shows tSNE visualizations of the learned embedding space of a selected task. Overall, by comparing Figure to Table 3, the higher performance appears to be consistent with more disentangled clusters. More- over, from the first column containing ITD (red) tokens and OTD (blue) tokens, we observe that the blue points dominate the embedding space and comprises multiple clusters, which demonstrates the out-of-tar task distribution multimodal, making
ProtoNet
0.02
0.10
0.03
N/A
ProtoNet+EOD
0.18
0.46
0.26
N/A
ContrastProtoNet
0.54
0.46
0.50
0.85
Reptile
0.45
0.17
0.25
0.57
ANIL
0.39
0.26
0.56
Reptile+HI
0.42
0.30
0.88
ANIL+HC
0.44
0.56
0.49
0.97
Table 4: Performance on 4-way 2-shot FexM)
Related Works
Research related to Visually-rich Documents (VD) have emerged as significant topics in NLP. Here,

---

we briefly review the prior research of (1) models for general VD understanding; (2) the particular Entity Retrieval (ER) task for VD and existing Few- shot VDER methods; (3) the methodology-lene level related works in general few-shot learning".
General Few-shot Learning. Few-shot Learn- ing (FSL) has been studied in various AI/ML do- mains (Song et al., 2023). In CV or NLP do- mains, there are two FSL tasks closely related to Few-shot VDER: (1) Few-shot object detection or segmentation (Köhler et al., 2023; Antonelli et al., 2022) aims at localizing objects in visual data, where each object can be treated as en- tity in VDER; and, (2) Few-shot Named Entity Recognition (NER) aims at labelling tokens within contextual text sequence (Li et al., 2022; Huang et al., 2021). While few-shot NER and object de- tection algorithms can provide inspirations for few- shot VDER, the challenges we face and methodol- ogy details are relatively different. Beyond them, Multimodal Few-shot Learning (MFSL) utilizes complementary information from multiple modali- ties to improve a unimodal FSL (Chen and Zhang, 2021; Lin et al., 2023). The scope of this pa- per falls within the field of MFSL. While exist- ing FSL/MFSL approaches can be categorized into meta-learning approaches (Snell et al., 2017; Finn et al., 2017) and non-meta LLM pretraining -nddraining-and- and- fine-tuning approaches (Brown et al., 2020), we em- ploy the benefits from both LLM prior knowledge and the meta-learning for task-personalized fine- tuning. Furthermore, to enhance task specificity performance of few-shot VDER, we employ Few- shot Out-of-disse dffdiistiitttinn (OOD) Detection, which itself is a recently emerged task (Le et al., 2021).
General VD Understanding, Pretrained LLMs for general VD understanding have shown strong performance in general understanding of visually- rich multimodal documents, and therefore, can serve as pretrained prior for Few-shot VDER. There are many LLM candidates our framework can use as the pretrained encoder, such as Lay- outLM (Xu et al., 2020), which extends the stan- dard BERT (Kenton and Toutanova, 2019), and the recent Lay (Huang et al., 2022) and DocGraphLM (Wang et al., 2023a), which show im- provements by using advanced cross-modal align- ment or local-global position embeddings. In this paper, we use the basic BERT model for experi- ments since our focus how to improve the post fine-tuning on few-shot downstream tasks, without a restrict on the specification of LLM type. Ex- tending this research to other pretrained Document Understanding LLMs could be one of future works.
Few-shot VD Entity Retrieval. The particular Entity Retrieval (ER) tasks for VD have been stud- ied for many years using Deep Neural Networls, Graph Neural Networks, or traditional models (Zhang et al., 2020; Shi et al., 2023), or empowered by the contextual prior knowledge provided by VD- understanding LLMs (Xu et al., 2021; Lee et al., 2022; Hong et al., 2022). VDER in the few-shot scenarios pose unique challenges such as achiev- ing task personalization with limited annotation, yet has garnered comparatively less attention in prior research. Recent advancements in Few-shot VDER predominantly rely on pretrained LLMs and prompt design, followed by fine tuning on a small number of VD documents (Wang et al., Ib; Wang and Shang, 2022). Despite their success, this paper explores a complementary research perspec- tive. While previous works address the situation where the entity label space is fixed over tasks and entity occurrences do not shift lot, we tackle different application situation-every few-shot task is user-sp,, focusing on a small subspace of interested entity types (entity-level task personal- ization), and entity occurrences vary significantly between tasks and documents.
7 Conclusions
In this paper, we studied the multimodal few-shot lcarning problem for VDER. We started by propos- ing a new formulation of the FVDER problem to be an entity-level, NNway soft- learning under the framework of meta learning as well as a new dataset, Few which is designed to reflect the practical problems. To solve the new task, ploited both metric-based and gradient-based meta- learning paradigms, along with a new technique we proposed to enhance task personalization via out- -istk-iisttiuution awareness. The experiments showed that the proposed methods achieve major improvements over the baselines for FVDER. For future works, our approaches might be im- proved in the following directions: (1) A more bust algorithm that distinguishes between the OTD and ITD. (2) An advanced decoding process consid- ering graphical structures or implicit correlations between entity instance within each task. (3) Ex- ploring the causal role of pretrained models.
AAn extended version of Related Works is in Appendix A.

---

Acknowledgements
Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1yya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
We would like to express sincere appreciation to all those who contributed to this research. Special thanks to all the reviewers for their constructive feedback and comments, which greatly improved the quality of this paper.
Manuel Carbonell, Pau Riba, Mauricio Villegas, Ali- cia Fornés, and Josep Lladós. 2021. Named entity recognition and relation extraction with graph neu- ral networks in semi structured documents. In 2020 25th International Conference on Pattern Recogni- tion (ICPR), pages 9622-9627. IEEE.
Limitations
There exists few limitations to this work. Firstly, the derived dataset is based on the current open source ones for document understanding, which are small in their size and has very limited amount of classes. A dedicated dataset that is built specif- ically for the purpose of studying few-shot learn- ing for document entity retrieval is needed. Sec- ondly, the scope of our current studies is limited to non-overlapping entities. The performance of the models under nested and entities with overlapping ground truth is yet to be examined.
Arindam Chaudhuri, Krupa Mandaviya, Pratixa Badelia, and Soumya K Ghosh, 2017. Optical character recog- nition systems. In Optical Charucter Recognition Systems for Different Languages with Soft Comput- ing, pages 9 9-41. Springer.
Jiayi Chen and Aidong Zhang. 2021. Hetmaml: Task- heterogeneous moodel-agnostie meta- meta-learning for few- shot learning across modalities. In Proceedings of the 30th ACM International Conference Informa- tion & Knowledge Management, CIKM '21, page 191-200, New York, NY, USA, Association for Com- puting Machinery.
Ethics Statements
The dataset created in this paper was derived from public datasets (i.e., FUNSD, CORD) which are publicly available for academic research. No data collection was made during the process of mak- ing this work. The FUNSD and CORD datasets themselves are a collection of receipts and forms collected and released by third party paper which has been widely used in the field of visually rich document entity retrieval research and is not ex- pected to contain any ethnics issues to the best of our knowledge.
Jiayi Chen and Aidong Zhang. 2022a. Fedmsplit: forrelation-adaptii federated multi-task learning across multimodal split networks, In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 87-96.
Jiayi Chen and Aidong Zhang. 2022b. Topological transduction for hybrid few-shot learning. In Pro- ceedings of the ACM Web Conference 2022, WWW 22,, page 3134-3142, New York, NY, USA. Associa- tion for Computing Machinery.
Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. 2021. Meta-baseline: Exploring simple meta-learning for few-shot learning. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 9062-9071.
References
Simone Antonelli, Danilo Avola, Luigi Cinque. Do- nato Crisostomi, Gian Luca Foresti, Fabio Galasso, Marco Raoul Marini, Alessio Mecca, and Daniele Pannone. 2022. Few-shot object detection: survey, ACM Computing Surveys (CSUR), 14(11))
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on ma- chine learning, pages 126--135. PMLR.
Rinon Gal, Shai Ardazi, and Roy Shilkrot. 2020. Car- dinal graph convolution framework for document information extraction. In Proceedings of the ACM Symposium Document Engineering 2020, DocFing '20, New York, NY, USA. Association for Computing Machinery.
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xic, and R Manmatha. 2021. Docformer: End-to-end transformer for document understanding. Proceedings of the IEEE/CVF international con- ference on computer vision, pages 993-1003.
Tom Brown Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal. Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack
Lukasz Garcarek, Rafat Powalski, Tomasz Stanistawek, Bartosz Topolski. Piotr Halama, Michal Turski, and Filip Graliński. 2021. Lambert: layout-aware language modeling for information ex- traction. In International Conference on Document Analysis and Recognition, pages 532-547. Springer.

---

Jiuxiang Gu. Jason Kuen, Vlad Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun. 2021. Unidoc: Unified pretrain- ing framework for document understanding. In Ad- vances in Neural Information Processing Systems, volume 34, pages 39-50. Curran Associates, Inc.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. 2020. Su- pervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pages 18661-18673. Curran Associates, Inc.
Adam W. Harley, Alex Ufkes, and Konstantinos G. Der- panis. 2015. Evaluation of deep convolutional nets for document image classification and retrieval. In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pages 991-995.
Gregory Koch, Richard Zemel, and Ruslan Salakhut- dinov. 2015. Siamese neural networks for one- one-shot image recognition. In ICML deep learning workshop. volume 2. Lille.
Mona Köhler, Markus Eisenbach, and Horst-Michacl Gross. 2023, Few-shot object detection: A com- prehensive survey. IEEE Transactions on Newral Networks and Learning Systems, pages 1-21.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
Duong Le, Khoi Duc Nguyen, Khoi Nguyen, Quoc- Huy Tran, Rang Nguyen, and Binh-Son Hua. 2021. Poodle: Improving few-shot learning via penalizing of-tiistibtttion samples. In Advances in Neural Information Processing Systems, volume 34, pages 23942-23955. Curran Associates, Inc.
Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang. Dachyun Nam, and Sungrae Park. 2022. Bros: A pre-trained language model focusing on text and layout for better key information extraction from documents. In Proceedings the AAAI Conference on Artificial Intelligence, volume 36, pages 10767- 10775.
Chen-Yu Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Ren- shen Wang. Yasuhisa Fujii, and Tomas Pfister. 2022, FormNet: Structural encoding beyond sequential modeling in form document information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 3735-3754, Dublin, Ireland. Association for Computational Linguisties.
Jiaxin Huang, Chunyuan Li, Krishan Subudhi, Damien Jose, Shobana Balakrishnan, Weizhu Chen, Baolin Peng, Jianfeng Gao, and Jiawei Han. 2021. Few- shot named entity recognition: An empirical baseline study. In Proceedings of the 202] Conference on Empirical Methods in Natural Language Process- ing, pages 10408-10423, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jing Li, Billy Chiu, Shanshan Feng, and Hao Wang- 2022. Few-shot named entity recognition via meta- learning. IEEE Transactions on Knowledge and Data Engineering, (449))4445--2555
Yupan Huang, Tengchao Lx, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutimv3; Pre- Pre-training for doc- ument ai with unified text and image masking. In Proceedings of the 30th ACM International Confer- ence on Multimedia, pages 4083-400 4091.
Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu I Liu. 2021. Selfdoe: Self-supervised doc- ument representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5652-5660.
Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Di- mosthenis Karatzas, Shijian Lu, and C. V. Jawahar. 2019. lcdar2019 competition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1516-1520.
Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, and Deva Ramanan. 2023. Multimodality helps uni- modality: Cross-modal few-shot learning with mul- timodal models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recog- nition, pages 19325-19337.
Guillaume Jaume, Hazim Kemal Ekenel, and Jean- Philippe Thiran. 2019. Funsd: A datasct for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2. pages 1-6.
Xiaojing Liu, Feiyu Gao, Qiong Zhang, and Huasha Zhao. 2019. Graph convolution for multimodal in- formation extraction from visually rich documents. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 2 (Industry Papers), pages 32-39.
Taewon Jeong and Heeyoung Kim, 2020. Ood-maml: detec-
Meta-learning for few-shot tion and classification. Advances in Neural Informa- tion Processing Systems, 33:3907-3916.
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of NAACL-HI.T, pages 4171 -116.

---

Tingting Ma, Huiqiang Jiang. Qianhui Wu, Tiejun Zhao, and Chin- Yew Lin. 2022. Decomposed meta-
Ilya Sutskever. 2021. Zero-shot text-to-image gen- eration. In Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8821-8831. PMLR.
learning for few-shot named entity recognition. In Findings of the Association for Computational Lin- guistics: ACL 2022, pages 1584-159,, Dublin, Ire- land. Association for Computational Linguistics.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. 2021. A simple fix mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.0902
Yifei Ming, Hang Yin, and Yixuan Li. 2022. On the impact of spurious correlation for detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10051-
10059.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals. Razvan Pascanu, Simon Osindero, and Raia Hadsell. 2019. Meta-learning with latent embedding optimization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Hiroki Nakayama. 2018. seqeval: python framework for sequence labeling evaluation. Software available from https://github. com/chakki-workksssse
Alex Nichol, Joshua Achiam, and John Schulman. 2018. On first-order meta-learning algorithms. arXiv preprint
Zejiang Shen, Ruochen Zhang, Melissa Dell, Benjamin Charles Germain Lee, Jacob Carlson, and Weining Li. 2021. Layoutparser: unified toolkit for deep learn- ing based document image analysis. In Document Analysis and Recognition-ICDAN 2021: 16th Inter- national Conference, Lausanne, Switzerland, Septem- ber 5-10, 2021, Proceedings, Part 1 16, pages 131- 146. Springer.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. 2018. Tadam: Task dependent adaptive met- ric for improved few-sho learning. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Frederik Pahde. Mihai Puscas, Tassilo Klein, and Moin Nabi. 2021. Multimodal prototypical networks for few-shot learning. In Proceedings of the IEFF/CVF Winter Conference Applications of Computer Vi- sion, pages 2644-2653.
Dengliang Shi, Siliang Liu, Jintao Du, and Huijia Zhu. 2023. Layoutgen: A lightweight architecture for visually rich document understanding. In Document Analysis and Recognition ICDAR 2023, pages 149- 165, Cham. Springer Nature Switzerland.
Inho Park and Sungho Kim. 2020. Performance indi- cator survey for object detection. In 2020 20th In- ternational Conference on Control, Automation and Systems (ICCAS), pages 284-288.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017 Prototypical networks for few-shot learning. In Ad- vances Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jacheung Surh, Minjoon Sco, and Hwalsuk Lee. 2019. Cord: consolidated receipt dataset for post-oer parsing. In Workshop on Document Intelligence at NeurlPS 2019.
Yisheng Song. Ting Wang, Puyu Cai, Subrota K. Mon- dal, and Jyoti Prakash Sahoo. 2023. A comprehen- sive survey of few-shot learning: Evolution, applica- tions, challenges, and opportunities. ACM Comput. Surv, 55(13s).
Krishna Pillutla, Sham M. Kakade, and Zaid Harchaoui. 2022. Robust aggregation for federated learning. IEEE Transactions on Signal Processing, 70:1142- 1154.
Bo Sun, Banghuai Li, Shengcai Cai, Ye Yuan, and Chi Zhang. 2021. Fsce: Few-shot object detection via contrastive proposal encoding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7352-7362.
Rafat Powalski, Lukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michal Pietruszka, and Gabriela Palka. 2021. Going full-tilt boogie on document un- derstanding with text- text-image-laye layout transformer. In Document Analysis and RecognitionICDDAR 2021: 16th International Conference, Lausanne, Switzer- land, September 5-10, 2021, Proceedings, Part 11 16. pages 732- 747. Springer.
Yuanyishu Tian, Yao Wan, Lingjuan Lyu, Dezhong Yao, Hai Jin, and Lichao Sun. 2022. Fedbert: When fed- erated learning meets pre-training. ACM Transac- rions on Intelligent Systems and Technology (TIST), 13(4):1-26.
Yi Tu, Ya Guo, Huan Chen, and Jinyang Tang. 2023. Layoutmask: Enhance text-layout interaction in multi-modes pre-training for document understand- ing. arXiv preprint arXiv:2305.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. 2019, Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference on Learning Representa- tions.
I Laurens van der Maaten and Geoffrey E. Hinton. 2008 Visualizing data using t-sne. Journal of Machine Learning Research, 9-2579-2603
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and

---

Gaël Varoquaux, Lars Buitinck, Gilles Louppe, Olivier Grisel, Fabian Pedregosa, and Andreas Mueller. 2015. Scikit-learn: Machine learning without learning the machinery. GenMobile Mob. Compur. Commun., 19:29-33.
Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, and Furu Wei. 2021b. Layoutreader: Pre- training of text and layout for reading order detection. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4735
Oriol Vinyals, Charles Blundell, Timothy P. Lilli- crap, Koray Kavukeuoglu, and Daan Wierstra. 2016. Matching networks for one shot learning. In Neural Information Processing Systems.
Zhisheng Xiao, Qing Yan, and Yali Amit. 2020. Like- lihood regret: An out-of-distributioo detection score for variational auto-encoder. Advances neural in- formation processing systems, 33:20685- 20696..
Dongsheng Wang, Zhiqiang Ma, Arminch Nourbakhsh, Kang Gu, and Sameena Shah. 2023a. Docgraphim: Documental graph language model for information extraction. Proceedings of the 46th International ACM SIGIR Conference on Research and Develop- meni in Information Retrieval.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang. Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021. LavoutLMv2: Multi- modal pre-training for visually-rich document understanding. In Proceed- ings of the 59th Anmual Meeting of the Association for Computational Linguisties and the 1 1ith International Joint Conference on Natural Language Processing ( (Voiume I: Long Papers), ), pages 2579-2591, Online. Association for Computational Linguisties.
Jiapeng Wang. Lianwen Jin, and Kai Ding. 2022a. Lilt: A simple yet effective language-independent layout transformer for structured document understanding. In Annual Meeting of the Association for Computa- tional Linguistics, pages 7747-7757.
Yiheng Xu, Minghao Li, Lei Cui. Shaohan Huang, Furu Wei, and Ming Zhou. 2020. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD Interna- tional Conference on Knowledge Discovery & Data Mining, pages 1192-1200.
Xin Wang, Thomas Huang, Joseph Gonzalez, Trevor Darrell, and Fisher Yu. 2020. Frustratingly simple few-shot object detection. In International Con- ference on Machine Learning, pages 9919-9928. PMLRR
Yaging Wang, Subhabrata Mukherjee, Haoda Chu, Yuancheng Tu, Ming Wu, Jing Gao, and Ahmed Has- san Awadallah. 2021a. Meta self- self-training for few- shot neural sequence labeling. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1737-1747.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. 2018. Bayesian model-agnostic meta-learning. In Ad- vances in Neural Information Processing Systems, pages 7332-7342.
Lijun Yu, Jin Miao, Xiaoyu Sun, Jiayi Chen, Alexan- der G. Hauptmann, Hanjun Dai, and Wei Wei. 2023. Documentnet: Bridging the data gap in document pre-training. arXiv preprint 270.072300.08997.
Ze Wang, Yipin Zhou, Rui Wang. Tsung-Yio Yu Lin, Ashish Shah, and Ser Nam Lim. 2022b. Few-shot fast- adaptive anomaly detection. Advances in Neural Information Processing Systems, 35:4957-4977.
Peng Zhang, Yunlu Xu, Zhanzhan Cheng, Shiliang Pu, Jing Lu, Liang Qiao, Yi Niu, and Fei Wu. 2020. Trie: end-to- text reading and information extraction for document understanding. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1413-1422.
Zifeng Wang, Zizhao Zhang, Jacob Devlin, Chen-Yu Lee, Guolong Su, Hao Zhang, Jennifer Dy, Vincent Perot, and Tomas Pfister. 2023b. QueryForm: A sim- ple zero-shot form entity query framework. In Find- ings of the Association for Computational Linguis- tics: ACL 2023, pages 4146-413 44199, Toronto, Canada. Association for Computational Linguistics.
Martin Zinkevich, Markus Weimer, Lihong Li, and Alex Smola. 2010. Parallelized stochastic gradient descent. In Advances in Neural Information Processing Sys- tems, volume 23, Curran Associates, Inc.
Zilong Wang and Jingbo Shang. 2022. Towards few- shot entity recognition in document images: A lahel- framework. In Find- ings of the Association for Computational Linguis- ties: ACL 2022, pages 4174-4186, Dublin, Ireland. Association for Computational Linguistics.
aware

---

Appendix
Entity Retrieval in Visually-rich Documents. Visually-rich Document Entity Retrieval (VDER) aims at detecting bounding boxes for specific types of key information within scanned or digitally- born documents, which has garnered significant attention from researchers. While there are techni- cal differences between Data-sufficien VDER and Few-shot VDER, the former offers foundational solutions and serves as a valuable baseline frame- work. Therefore, reviewing existing Data sata-suffi VDER techniques is worthwhile. At early years, Deep Neural Networks (c.g., RNNs, CNNs) have been widely employed in addressing VDER tasks (Huang et al., 2019; Zhang et al.. 2020). Later, Graph Neural Networks (GNNs) (Liu et al., 2019; Gal et al., 2020; Carbonell et al., 2021; Shi et al., 2023) have gained substantial attention for their ef- fectiveness in tackling the structural layout informa- tion. Recent works empower LLMs of general VD Understanding to incorporate additional contextual prior knowledge and fine tume on VDER tasks (Xu et al., 2021; Garncarek et al., 2021; Lee et al., 2022; Hong et al., 2022). Beyond these research, this paper focuses on the Few-shot VDER cases with limited annotation, which pose unique challenges in achieving task personalization with scarce data, addressing shol imbalance, and handling task com- plexity due to tit-of-task-distriiinn
A Related Works
We review related works corresponding to the Few- shot VDER tasks in Section A.I and then review methooology--evel related works in Section A.2.
A.1 Visually-rich Document Related Works
Visually-rich Documents (VD) are a vital category of multimodal data in the field of Document AI, typically consisting of texts, images, and layout structure of contents. Research and industrial ap- plications pertaining to VD have emerged as sig- nificant topics in NLP over the past decade. Here we review the prior research works for VD-related tasks, including (1) LLMS for general VD under- standing; (2) the particular Entity Retrieval (ER) task in VD and its prior work; and (3) existing works considering VDER in shot scenarios.
Visually-rich Document Understanding LLMs. Large Language Models (LLMs) have shown strong performance in general understanding of visually-rich multimodal documents, and therefore. can serve as pretrained prior for Few-shot VDER. For this reason, here we review several LLM can- didates we can use. LLMs for text- text-image-lun image-layour document understanding have emerged since Lay- outLM (Xu et al., 2020), which extends the stan- dard BERT (Kenton and Toutanova, 2019) by ad- ditional layout information obtained from OCR (Chaudhuri et al., 2017) preprocessing. After this, SelfDoc (Li et al., 2021), UDoc (Gu al., 2021), LayoutLMv2 (Xu et al., 2021), TILT (Powal- ski et al., 2021), DocFormer (Appalaraju et al., 2021), LiLT (Wang et al., 2022a), and Layout (Huang et al., 2022), show improvements by using cross-modal alignment or modern feature encoders for the image modality (c.g., ResNets (He et al., 2016) and dVAE (Ramesh et al., 2021)). Very re- cent works, Uniformer (Yu al., 2023), Layout- Mask (Tu et al., 2023), and DocGraphLMM (Wang et al., 2023a), employ token-level strategics like local text-image alignment, local position embed- dings, and graph representation, to improve the modeling. In this paper, we use the basic BERT model for experiments since our focus is how to im- prove the post fine-tuning on fhw-shot downstream tasks, without restrict on the specification of L.M type. Extending this research other pretrained Document Understanding LLMs could be one of future works.
entities.
Few-shot VDER. There has been rare discus- sions on VDER in few-shot scenarios. Recent works primarily focus on pretraining LLMs and prompt design such that they can then be fine tuned on small number of VD documents (Wang et al., 2021b; Wang and Shang, 2022; Xu et al., 2021; Huang et al., 2022). Despite their success, our paper explores a complementary research perspec- tive: (1) While previous works emphasize first- stage LLM pretraining, our work focuses on the second-stage few-shot adaptation algorithm. (2) We tackle a different application situation from the previous works. Previous works address specific few-shot situation where the entity label space is fixed and entity occurrences do not vary a lot from one document to another. In contrast, our research tackles a different situation, where the entity label spaces and entity occurrences vary significantly be- tween tasks and documents, enabling entity-level task personalization (i.e., each personal few-shot task is only interested in a small subset of entity types). Both situations can happen in the real world. This paper addresses the second unexplored one.

---

A.2 General Few-shot Learning
algorithms (Sun et al., 2021) can provide inspi- rations for Few-shot VDER, there are still gaps between Few-shot VDER and these fields: the lack of Few-shot VDER datasets in the form of object detection or segmentation tasks; and, the scales of entity objects are often much smaller than the out- cuttoo-diitribution background objects.
Next, we review the methodology-level related works from the other domains that are closely re- lated to, but beyond, VDER. First, we briefly re- view of general Multimodal Few-shot Learning al- gorithms. Then, we review literature in the fields of CV and NLP that address non- NOn- but closely related tasks, including: (1) vision-only Few-shot Object Detection and Segmentation in the CV do- main; (2) text-only Few-shot Named Entity Recog- nition in the NLP domain; and (3) the general Few- shot -uttof-distributtin Detection.
Few-shot Sequence Labeling. This paper adopts the few-shot sequence labeling paradigm as intro- duced by (Wang et al., 2021a). Many other NLP tasks have also embraced this paradigm, including Few-shot Named Entity Recognition (NER) (Li et al., 2022). Few-shot NER that focus on limited entity occurrences was initially introduced in Few- NERD (Huang et al., 2021), and later, (Ma et al., 2022) proposed a meta-learning approach ad- dress this task. While Few-shot NER tasks are text- only, devoid of visual and layout modalities, and typically involve short texts, Few-shot VDER at the entity level presents a greater challenge, where the difficulty lies in effectively integrating layout struc- ture and visual information and achieving task per- sonalization from outtof-distribution background.
Multimodal Few-shot Learning. Few-shot Learning (FSL.) has been studied in various AI/ML domains, such as CV, NLP, healthcare, etc (Song et al., 2023). Multimodal Few-shot Learning (MFSL) jointly utilizes complementary information from multiple modalities to improve a uni-modal task (Pahde et al., 2021; Chen and Zhang, 2021; Lin et al., 2023). The scope of this paper falls within the domain of MFSL, with a spe- cific emphasis on multimodal documents. Existing MFSL work falls into two categories: non-meta learning methods and meta-lcarning approaches. The former typically involves a two-stage training process LLM pretraining and fine-tuning or prompt learning (Wang et al., 2023b). On the other hand, meta-Icarning approaches formulate task-level distribution, and then, learn adak-idee metric functions (Snell et al., 2017; Oreshkin et al., 2018; Koch et al., 2015; Vinyals et al., 2016) or employ bilevel optimization to learn titaa-paaameerss for fast task-adaptive fine tuming (Finn et al., 2017; Yoon al., 2018; Rusu et al., 2019; Chen and Zhang, 2022b). The proposed framework benefits from both LLMs and the meta-learning paradigm by being build upon the LLMS and then using meta-learning for task-adaptive fine-tuning.
Few-shot Out-of-Distribution Detection. Ma- chine learning models, when being deployed in open-world scenarios, have shown to erroneously produce high posterior probability for out-of- distribution (OOD) data. This gives rise to OOD de- tection that identifies unknown OOD inputs so that the algorithm take safety precautions (Ming et al., 2022). Recently, motivated by real-world ap- plications, OOD detection in the few-shot settings increasingly attracts attentions (Le et al., 2021; Jeong and Kim, 2020; Wang et al., 2022b), which faces new challenges such as a lack of training data required for distinguishing OOD from tasc-ipecific class distribution. In this paper, the proposed frame- work for Few-shot VDER employs few-shot OOD detection to improve performance: to prevent the prediction of background context as one of task- personalized entities, we encourage task-aware fine tuning to exclude statistically informative yet spu- rious features in the support set.
Few-shot Object Detection and Segmentation. Few-shot object detection and Few-shot Segmenta- tion are CV tasks that aim at recognizing and local- izing novel objects semantics in an image with only a few training examples (Wang et al., 2020; Köhler et al., 2023; Antonelli et al., 2022). The output of entity retrieval from document images consists of bounding boxes for entities, allowing it to be formulated as an object detection or segmenta- tion problem in the CV domain, where each object is treated as an entity (Shen et al., 2021). However, while fho-shot object detection and segmentation
B FewVEX Dataset
Since there is no dataset specifically designed for the Few-shot VDER task defined in Section 2, we construct a new dataset, Few VEX, to benchmark and evaluate Few-shot VDER tasks.

---

B.1 Collection of Entity Types and Documents
B.2.1 Entity Type Split
First, we collect the entity types C associated with the task distribution P(T) and a set of document images D annotated by these entity types.
To ensure that testing tasks in Dinc meta focus novel classes that are unseen during meta-training D meta. we should split the total entity types C into two separate sets = Chase CraselCroree Cnovel = 0 such that Chase is used for meta- veeta-trannig and Cnovel for meta-testing.
We consider two source datasets that are widely used in normal large-scale document understanding tasks such entity recognition, parsing, and infor- mation extraction. The first one is the Form Under- standing in Noisy Scanned Documents (FUNDS) dataset (Jaume et al., 2019) comprises 199 real, fully annotated, scanned forms, with a total of three types of entities (i.e.. questions, answers, heads). The second one is the Consolidated Receipt Dataset for post-OCR parsing (CORD) dataset (Park et al., 2019). CORD consists of 1000 receipt images of texts and contains superclasses (menu, void menu, subtotal, void total, total, and etc) which are divided into 30 fine-grained subclasses. For different entity types, the total numbers of entity occurrences over the CORD images are highly imbalanced, ranging from 1 occurrence of entity "void menu (nm)" to 997 occurrences of "menu (price)".
Specifically, we use split ratio Y to control the number of novel classes and randomly choose >|C| entity types from C as Crovel. Then, Clase C' \ Cnovel- Note that for the cases that some entity types occurs in less number of documents than the others, we set threshold U and any entity type that occurs in less than U documents are forced to be one of the novel classes.
B.2.2 Single N-way K-shot Task Simulation Each individual task T = {S, & in either Dinn meta or Diet meta can be generated by the follow- ing steps (summarized in Algorithm 1).
Personalized Class Sampling. The target classes of task & is generated by randomly sampling N entity types from either Coase (for the training task) or Crovel (for the testing task).
From the two datasets, we obtain combined source dataset denoted D, which contains 1199 unique document images with original annotations on 33 classes. However, we observe that some fine-grained classes in CORD occurs in less than mar;(Mai + Mgs) images, the maximum num- ber of documents within individual tasks. This will result in large amount of repetitive usage of the same documents within one task and between different tasks. Therefore, we further sort the 33 classes by the number of unique document images where they occur and then discard three entity types that occurs in low frequency.
Document Sampling. Given the N target classes, we then collect document images that satisfies the few-shot setting defined in Section 2. However, one problem of document sampling from the orig- inal corpus is the inefficiency. It is because, for each task, only a small number of documents that contain the corresponding classes can be the candi- date documents of the task. For example, if each document contains only small number of entity types, the majority of documents would be rejected. To improve sampling efficiency, one strategy is to count entities in each document in advance and, for each entity type, all the candidate documents that contain this type are temporally stored in a new dataset. We only look at the task-specific candidate datasets Dē = (DDlVe E E). where De {X, ())(XXY) E Dife E YJ. We pro- posed drossed Rejection sampling in Al- gorithm 1, which randomly sample M, documents such that the total number of entity instances is satisfied-that is, ~ pK shots per entity type. Likewise, we sample M, documents for Q, such that there are K₂ ~ pK, shots per entity type. We keep track table to record the current count of occurrences of each type of entity types in the task.
To sum up, we finally have a total of |C] =30 entity types and (DD = 1199 unique document im- ages annotated by these entity types. The pie chart
(on the left) in Figure/ illustrates the number of occurrences of the final entity types.
B.2 Collection of Training and Testing Tasks
We simulate distribution of tasks P(T) in Vex We create a meta-learning dataset D Dmnta- {Dim metar Dist ), consisting of meta- training set Dirn = [Ti, Tr...Trmn. contain- ing Ttrn training tasks and meta-testing set Diest = (Tr', To ..., Tia containing Tist testing tasks. Each task instance follows the N-way K- shot FVDER task setting that pays attention to N personalized entity types.

---

Label Conversion. In the few-shot setting, the majority region of an document does not follow the in-task distribution (ITD) of &. These regions' tokens are treated as either background or the other types of entities from the out-of-task distribution (OTD), whose original labels should be arbitrarily converted into 0 label. In addition, we map the original labels of ITD tokens to relative labels. For example, if we use I/O schema, the relative labels should range from label id 0 label id (N - 1).
derstanding under N-way K-shot setting, The train- ing and testing classes are both from CORD. The goal is to learn domain-invariant meta-parameters. Few VEX(M) focuses on learning domain -agnoiic meta- perta-arrssterss from a combination of receipt and form understanding, Receipt and form documents may appear in the same task.
C Experimental Setups
C.1 LLM-based Multimodal Encoder
Algorithm 1 cross-documeet Rejection (XDR) Sampling for Few-shot VDER Task Simulation
We pre-train the multimodal Transformer on the IIT-CDIP dataset (Harley et al., 2015). It should be noting that this paper does not focus on the pre- training technique. In fact, framework does not require well pre-trained encoder, since the meta-learning will further meta-tune the pre-traee pre-trained encoder to capture the domain knowledge of P(T). Thus, we stop the pre-training until an 81.5% token classification accuracy.
1: Require: N, K, Kg. Craser Cnovet, D.
2: Randomly sample N entity types from either Coase
Cnovel and ohtain E.
3: Initialize: S 0, Q =0
4: Initialize: D (D) Ve E) from D.
5: Initialize: N integers train_courat[e]
for Ve E E, c] for Ve &.
6: Initialize: N integers
7: Document sampling for
8: while minlee train_countl
K do
9: Find the least frequent entity type in the current task,
i.c., ē argmin tegtrann__cnnit
C.2 Training Parallelism
10:
Sample document Yj) from D
We employ the episodic training pipeline to learn the meta-parameters from training tasks (i.e.. episodes). At each meta- veta-training step, a total of r episodes are trained and then validated to obtain the meta- geeta-gradeets used for updating meta- pettmmaaaeeerss. Both meta-training and meta-testing were run in a multi multi-process manner. Each of our experiments was run on a total of 4 machines and on each ma- chine there are 8 local TPU devices. Since the parameter size of the Transformer-based encoder is large. we use the 8 devices of each machine to train one single episode in parallel. That is saying, at each tratt-traiiing step, a total of 4 tasks are used to compute the meta -raddents.ss
11: Add (X,, Y,) to
12:
for & do
13:
Remove the selected document from candidate dataset D + D \ {(X, Y)}
14:
Update (rrain_countle]11 if Y contains entity type if [trai_ccuun((e] > pk then
15:
16:
Mask [(rair__covvdee] -KK) instances of type-c by setting token labels to -1
17: end if
18: end for
19: end while
20: // Document sampling for Q
21: while min test_ _ott_ooo_]eee____ < K,, do
22: Find the least frequent entity type in the current task,
i.c., é argmin test_ccoonre
23:
Sample document (X,, Y2) from D
24:
Add (X,.Y) to
25: for e E do
26:
Remove the selected document from candidate dataset D D 1H(X.Y. Y)}
Both the support (train) and query (test) docu- ments in one task are divided and assigned to 8 devices. The prototypes, the nearest neighbors of data points, or the adapted parameters trained on the local support set, are computed on each local device. For validation on the query set, however, should consider, the scope of the entire task over different local devices. Therefore, we employ Fed- erated Learning techniques (Zinkevich et al., 2010; Pillutla et al., 2022; Chen and Zhang, 2022a; Tian et al., 2022) operating on multiple devices for a dis- tributed within- adaptation, where we collect the locally adapted parameters (at each inner-loop step) or the prototypes from the 8 devices of a sin- gle episode and average their parameters. Specifi- cally, for training parallelism of each episode/task.
27:
Update [eest_cmntje]]] i contains entity type test__oont e]] > pK, then
28:
29:
Mask cest__cumttee]] pK,) instances of type-c by setting token labels to
30: end if
31: end for
32: end while
33: Label conversion for v(X,, SUQ
34: return: T (8, Q.8)
B.3 Dataset Variants
We fix the testing shot Ky=4. We propose two variants of meta-dataset, each of which pay at- tention to different challenges in few-shot learn- ing. The statistics is summarized in Table VEX(S) focuses on single-domain receipt un-

---

there are 4 steps: (1) on each device, we first adapt model based on the partial support documents lo- cated on the device; (2) then, we collect the adapted knowledge from each of the 8 local devices and ag- gregate them; (3) on each device, we apply the collected adapted knowledge to the partial query documents; (4) the validation loss on the query sub- set each devices are collected and we take an average of them.
we intend to incorporate these results into the revised version of our paper.
C.4 Hyperparameters
We summarize the hyperparameters in Table
Hyperparameters
Value
P
0.6
20
Kg
4
C.3 Baselines
Table 5: Hyperparameters.
There are mainly two families of approaches for Few-shot VDER. (1) Meta-learning based Approaches. Our proposed strategies can im- prove both metric-based and gradient-based meta- learning methods. To validate our arguments, we compare ContrastProtoNet with its metric-based meta-learning baseline ProtoNet (Snell et al., 2017). We compare ANIL+HC with its gradient- gradient-based meta-learning baseline ANIL (Raghu al., 2019),
D Evaluation Methods
D.1 Quantitative Metrics
We consider two types of quantitative metrics.
Overall Performance. Following (Park and Kim, 2020; Xu et al., 2020), we use the precision (P), re- call (R) and micro F1-score over meta-testing tasks. We use the I/O tagging schema and the "seqeval" (Nakayama, 2018) tool to compute the P/R/F1.
etc. Extending our work to SOTA meta-learning methods could be one of our future works. (2) Non-Meta-learning based Approaches. We did not present a comparison with existing Non-Meta- learning based Few-shot VDER techniques (Wang and Shang, 2022; Wang et al., 2023b) due to the following reasons:
Task Specificity (TS). In the proposed frame- work, solve diitt--distiibuttion (OOD) detec- tion as subtask improve task personalization and avoid spurious features. We calculate ITD score for each data point representing how likely it belongs to the task-specific distribution. To eval- uate how well the learned meta-learners can dis- tinguish in-task distribution (ITD) from the out- of-task distribution (OTD), we calculate AUROC (Xiao et al., 2020) using the ITD scores over all test episodes. higher AUROC value indicate better TS performance, and random guessing detector corresponds to an AUROC of 50%. We use the 'skleam.metrien (Varoquaux et al., 2015) tool to compute the AUROC and plot ROC curves.
Existing Non-Meta-learning based Few- shot VDER techniques primarily address document-level scenarios ("Entity occur- rences do NOT vary from one document to another"). In contrast, our paper focuses on the entity level (" ("Entity occurrences vary from one document to another"). is thus not fair to compare methods which were designed un- der dissimilar problem settings.
We have conducted comparative experiments by applying (Wang and Shang, 2022) to our BERT-based LLM, a Non-Meta-learning based Few-shot multimodal NER technique, to specific problem setting. With the same fine-tuning steps (T=15) and learning rate, the Fl results of (Wang and Shang, 2022) on the 4-way 4-shot setting is only 0.115, while the Fl results of all gradient- based meta-lcarning methods (including our proposed method) is over 0.5 and that of all metric-based meth- ods (including our proposed method) is over 0.23. However, it may be not fair to compare with (Wang and Shang, 2022) since our paper studies a different setup. Despite this nuance,
D.2 Visualization
To visualize the TS, we plot the ROC curves of all the meta-testing tasks, where each curve represent one task. Another visualization for TS is to show how ITD and OOD are distinguished against each other. We randomly select a testing task and ex- ploit tSNE (van der Maaten and Hinton, 2008) to visualize the learned embeddings of all the tokens in the task, where ITD tokens are denoted as red points and OTD tokens are blue points.
Furthermore, we use tSNE to visualize the learned embeddings of only the ITD token in- stances in the task, where different colors represent different entity types.

---

DULIC
(a) token embeddings of a training task
TOAL MOLL DE
(b) token embeddings of a testing task (N=4 novel classes)
Figure Learned class distribution training task and a testing task 4-way 4-shot setting. The meta- parameters are trained using ContrastProtoNet Few VEX(S) Solid points represent train (support) tokens, cross points represent val/test (query) tokens, and the triangle points represent prototypes.
DX PENUIT
(a) ANIL (4-way 1-shot)
(b) ANIL HC (4-way 1-shot)
(c) ANIL + HC (4-way 4-shot)
Figure S: Visualization under way 4hotot and 4-way 1-shot settings of VeX(S) for ANIL and ANIL+HC.

---

(a) ProtoNet (4-way 4-shot)
(c) Reptile (4-way 4-shot)
(b) ContrastProtoNet (4-way 4-shot)
Reptile HC (4-way 4-shot)
Figure Visualization and ROC curves different methods on the 4-way 4-shot setting of FewVEX(S), For each method, the left subfigure is the (SNE visualization of the learned embeddings of in-task distribution (ITD) entities a randomly chosen meta-testing task, where different colors indicate different entity types); the middle subfigure shows the tSNE visualization of the learned embeddings of all tokens in the same meta-testing task, where the ITD entities are represented as red points and the out-of-task distribution (OTD) entities or background are represented as blue points; the right subfigure shows the ROC curves of meta- meta-testing testing tasks, where each colored line corresponds task, representing how ITD is distinguished from OTD based the model's output logits.
E Additional Results
We present additional visualization results in Fig- ure 6, Figure 4, and Figure 5.


Reviews:
[{"review_id": "lt40V0ZWCS", "reviewer": "Reviewer_KbGr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### ***Weakness 1:  While the proposed approach is evaluated on the FewVEX dataset, more extensive experimentation on other publicly available datasets could further validate the effectiveness and generalizability of the proposed methods.***\n\nThank you for your valuable suggestions! The notion of comparing with other public datasets holds immense potential in substantiating the efficacy and applicability of our proposed methods. However, based on our investigation, there is currently **no public available dataset** specifically designed for entity-level few-shot document entity retrieval tasks, since we are the first studying such real-world tasks.\n\nThe FewVex is proposed by our paper, by deriving from existing document understanding datasets (CORD and FUNDS). Our choice to focus on CORD and FUNDS is due to their significant diversity of entity types, which renders them ideal candidates for constructing FewVex.   Conversely, other document understanding datasets were not selected due to their lack of substantial entity type diversity, making them less suitable for the construction of FewVex.  **The proposed FewVEX will be the first public dataset** for entity-level few-shot document entity retrieval. We will release the code and our datasets upon the paper's acceptance. We hope FewVex will emerge as a valuable data source, aiding future researchers within this domain. \n\nFurthermore, we present **Algorithm 1 (available in the Appendix on page 12)**, an **automated dataset construction** algorithm capable of generating novel few-shot datasets with task-specific entity types from existing document understanding datasets with global entity types. The code for this algorithm will be made publicly available, and we anticipate that if, in the future, document understanding datasets encompassing a wider array of entity types become available, our algorithm will facilitate the creation of an even more comprehensive collection of publicly accessible datasets within this context.\n\n### ***Weakness 2: The paper could provide more detailed explanations or examples of the entity-level imbalanced few-shot VDER scenarios and how they differ from the document-level balanced ones. This would help readers better understand the specific challenges posed by entity-level imbalance.***\n\nWe present an illustrative example to compare the scenarios of entity-level imbalanced few-shot VDER and how they contrast with document-level balanced few-shot VDER scenarios:\n- “Document-level balanced few-shot VDER scenarios'”. For instance, in three document images, X1, X2, and X3, and three entity types, E1, E2, and E3, this situation assumes that E1 appears once in X1, E2 appears once in X1, and E3 also appears once in X1. Similarly, E1 appears once in X2, E2 appears once in X2, and E3 also appears once in X2. We use a Table below to summarize this example. This scenario is limited to specific document types whose layout structures and  entity types are predefined, such as Tax forms, where the “custom name” and “custom address' ' occur in all forms around the same position. \n\n   |                                   |  #Occurrence in X1   |   #Occurrence in X2     |   #Occurrence in X3   |\n   |-------------------------|:----:|:---------:|:------:|\n   |        Entity Type  E1                     |    1  |   1        |     1  |\n   |        Entity Type  E2                     |    1  |   1        |     1  |\n   |        Entity Type  E3                     |    1  |   1        |     1  |\n\n- “Entity-level imbalanced few-shot VDER scenarios”.  In this case, entity occurrences differ significantly between documents. For instance, E1 occurs 3 times in X1, while E2 appears once and E3 is absent. In contrast, E1 might not be present in X2, E2 might occur 3 times in document X2, E3 appears 2 times in X2.... This situation is often encountered in scenarios with substantial shifts in document layout or content, as seen in datasets like the receipt dataset (e.g., CORD). \n\n   |                                   |  #Occurrence in X1   |   #Occurrence in X2     |   #Occurrence in X3   |\n   |-------------------------|:----:|:---------:|:------:|\n   |        Entity Type  E1                     |    3  |   1        |     0  |\n   |        Entity Type  E2                     |    0  |   3        |     2  |\n   |        Entity Type  E3                     |    1  |   0        |     3  |\n\n\n### ***Weakness 3:  The paper could consider discussing the potential biases or limitations in the FewVEX dataset and how they may impact the generalizability of the results. Addressing any potential dataset limitations or biases would enhance the credibility of the experimental findings.***\n\nThank you for the valuable suggestion regarding the discussion of limitations in the FewVEX dataset. We identify two distinct limitations:\nThis is a very good suggestion discussing the limitations in the FewVEX dataset. We think there are two Limitations: (1) An inherent limitation pertains to the *number of entity types* in the FewVEX dataset. The current scope of classes  in the FewVEX may not be extensive. However, we have limited options when choosing document understanding datasets to generate FewVEX, and we decided to use CORD and FUNDS due to their relatively larger number of classes rather than other available document understanding datasets.  Despite such restriction, we introduce Algorithm 1, which provides a solution that automatically generates a few-shot dataset, and if, in the future, document understanding datasets encompassing a wider array of entity types become available, our  Algorithm 1 can help quickly derive few-shot datasets with more extensive classes and will improve the generalization of the proposed system. (2) *Limited Number of Domains*: Another limitation arises from the inclusion of only two domains–receipts and forms—within the FewVEX dataset. The inherent challenge posed by the limited domain diversity is that the available domains may not be sufficient to comprehensively assess the capacity of methods to handle domain shifts. Recognizing this concern, we are committed to expanding the dataset to incorporate additional domains in our future endeavors.\n\n### ***Weakness 4:  The paper could provide more specific details on the implementation and experimental setup to ensure reproducibility of the results.***\n\nWe implemented the baselines and proposed method using Jax and Tensorflow and conducted all experiments across 8 TPU devices. Each experiment was iterated 5 times with distinct random seeds, and the average results were recorded. In terms of model configuration, our approach involves a BERT-based encoder with an output dimension of 768. Subsequently, for gradient-based methods, a 2-layer MLP functions as the decoder. \nFor a more comprehensive understanding of our training specifics, we offer additional training-related experimental setup in Appendix C. To enhance reproducibility, we will release our code, thus facilitating the replication of our experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses a significant problem in the field of industrial NLP applications and provides a novel approach for visually-rich document entity retrieval. \n\n2. The incorporation of asymmetric across-document rejection (ADR) sampling helps to handle the imbalance in entity occurrences in a more realistic manner. \n\n3. The utilization of meta-learning techniques in out-of-task-distribution document contexts enhances the generalization ability of the models and improves upon popular meta-learning baselines. \n\n4. The introduction of the FewVEX dataset not only facilitates the evaluation of the proposed approaches but also sets a benchmark for future research in this area.", "reasons_to_reject": "1. While the proposed approach is evaluated on the FewVEX dataset, more extensive experimentation on other publicly available datasets could further validate the effectiveness and generalizability of the proposed methods. \n\n2. The paper could provide more detailed explanations or examples of the entity-level imbalanced few-shot VDER scenarios and how they differ from the document-level imbalance. This would help readers better understand the specific challenges posed by entity-level imbalance.\n\n3. Additionally, the paper could provide more specific details on the implementation and experimental setup to ensure reproducibility of the results.\n\n4. The paper could consider discussing the potential biases or limitations in the FewVEX dataset and how they may impact the generalizability of the results. Addressing any potential dataset limitations or biases would enhance the credibility of the experimental findings.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses the limitations of existing methods by focusing on the under-explored entity-level imbalanced few-shot VDER scenarios. The proposed across-document rejection (ADR) sampling algorithm efficiently handles the imbalanced nature of data by identifying and rejecting irrelevant instances. The use of meta-learning-based approaches, such as HC and ContrastProtoNet, in out-of-task-distribution document contexts enables robust few-shot entity training. The introduction of the FewVEX dataset serves as a valuable resource for evaluating the proposed approaches.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6sacdO2dRw", "reviewer": "Reviewer_BXH7", "summary": "", "questions": "Please provide more clarity with respect to my Point 1 of previous section .", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### ***Confusion 1 about Task Definition: A claim by paper is \"Line 071- Entity occurrences vary dramatically from one document to another\", and the confusion is that “Is it not true that in any NER case in the real world this happens?***\n\n**Response**: \n\nAccording to our observations, two distinct situations can exist in real-world document image entity retrieval cases:\n\n- **Situation 1: “entity occurrences do NOT vary from one document to another”**. For instance, in three document images, X1, X2, and X3, and three entity types, E1, E2, and E3, this situation assumes that E1 appears once in X1, E2 appears once in X1, and E3 also appears once in X1. Similarly, E1 appears once in X2, E2 appears once in X2, and E3 also appears once in X2. We use a Table below to summarize this example. This scenario is limited to specific document types whose layout structures and  entity types are predefined, such as Tax forms, where the “custom name” and “custom address” occur in all forms around the same position. \n\n   |                                   |  #Occurrence in X1   |   #Occurrence in X2     |   #Occurrence in X3   |\n   |-------------------------|:----:|:---------:|:------:|\n   |        Entity Type  E1                     |    1  |   1        |     1  |\n   |        Entity Type  E2                     |    1  |   1        |     1  |\n   |        Entity Type  E3                     |    1  |   1        |     1  |\n\n- ***Situation 2: “entity occurrences vary dramatically from one document to another”***. In this case, entity occurrences differ significantly between documents. For instance, E1 occurs 3 times in X1, while E2 appears once and E3 is absent. In contrast, E1 might not be present in X2, E2 might occur 3 times in document X2, E3 appears 2 times in X2.... This situation is often encountered in scenarios with substantial shifts in document layout or content, as seen in datasets like the receipt dataset (e.g., CORD). \n\n   |                                   |  #Occurrence in X1   |   #Occurrence in X2     |   #Occurrence in X3   |\n   |-------------------------|:----:|:---------:|:------:|\n   |        Entity Type  E1                     |    3  |   1        |     0  |\n   |        Entity Type  E2                     |    0  |   3        |     2  |\n   |        Entity Type  E3                     |    1  |   0        |     3  |\n\nTo clarify, our claim is that ***“Both situations can happen in the real world. While the first situation has been addressed in prior research [1,2], the second situation has never received research attention.”*** As such, our paper proposes a solution for the second situation.\n\nFurthermore, despite our primary focus on the second situation and our experiments within this context, it's important to note that our method is also applicable to the first situation without necessitating modifications.\n\n[1] Zilong Wang and Jingbo Shang. 2022. Towards few-shot entity recognition in document images: A label-aware sequence-to-sequence framework. arXiv preprint arXiv:2204.05819.\n\n[2] Zifeng Wang, Zizhao Zhang, Jacob Devlin, Chen-Yu Lee, Guolong Su, Hao Zhang, Jennifer Dy, Vincent Perot, and Tomas Pfister. 2022. Queryform: A simple zero-shot form entity query framework. arXiv preprint arXiv:2211.07730.\n\n### ***Confusion 2 about Task Definition: a claim by paper is \"$N$ way means $S$&$Q$ do not contain any entity types other than $\\mathcal{E}$\", and the confusion is that “Again is that not true in any NER task ? I am finding it hard to differentiate these claims of uniqueness from a 'Few shot multimodal NER' setting.***\n\n**Response**: \n\nThanks for your question! We want to clarify that we are not claiming that the $S$ & $Q$ documents do not contain any entity types other than the N types in $\\mathcal{E}$.  Our claim is that ***“$S$ & $Q$ documents do not contain any LABELED entity types other than the N types in E. Other entity types aside from $\\mathcal{E}$ can indeed exist within $S$ and $Q$ documents, but they remain unlabeled and are treated as background or out-of-distribution information.”***\n\nTo be more clear, let's provide an illustrative example of the content within an N-way task $\\mathcal{T}=(S,Q,\\mathcal{E})$. \n- Around 5% tokens in $S$ & $Q$ documents are associated with either LABELED or UNLABELED entity types from the set $\\mathcal{E}$, comprising the entities that align with the **in-task distribution (ITD)**.\n- Roughly 20% tokens in $S$ & $Q$ documents belong to UNLABELED entity types **other than $\\mathcal{E}$**, which are the **out-of-task-distribution** entities.\n- Approximately 75% tokens in $S$ & $Q$ document belong to UNLABELED background information, also **not part of $\\mathcal{E}$** and from out-of-task distribution. \nIt's important to note that for these UNLABELED tokens, we have no knowledge about whether they represent background, in-task distribution, or out-of-task distribution entities. More detailed clarification regarding the task information is offered in lines 333-349.  We intend to enhance the clarity of this explanation in the upcoming revised version of the task definition.\n\nFurthermore, it is true that some real-world applications can directly lead to our claim: ***“$S$ & $Q$ documents do not contain any LABELED entity types other than the N types in E. Other entity types aside from $\\mathcal{E}$ can indeed exist within $S$ and $Q$ documents, but they remain unlabeled and are treated as background or out-of-distribution information.”***  To illustrate, consider a scenario where a document encompasses five entity types: E1, E2, E3, E4, and E5. Suppose there is a user of the system, who is only interested in 3 of these entities (i.e., E1, E3, E4) and provides few-shot labels for them. The user is **not interested in E2 and E5 and does not label them, but E2 and E5 still exist in the training and testing documents and are left as the background context to the user**. Such a scenario is practical in the realm of Few-shot VNER systems, and our claim aligns with the personalization of entity types for individual tasks/users.\n\n\n### ***Weakness 2: Comparisons are needed against Few shot multimodal NER techniques , not just meta learning based ones. particularly if the previous point is valid.***\n\n**Response**: \n\nIn our paper, we do not present a comparison with existing Non-Meta-learning based Few-shot multimodal NER techniques due to the following reasons:  \n- Existing Non-Meta-learning based Few-shot multimodal NER techniques primarily address document-level scenarios (\"Entity occurrences do NOT vary from one document to another\"). In contrast, our paper focuses on the entity level (\"Entity occurrences vary from one document to another\").  It is thus not fair to compare methods which were designed under dissimilar problem settings.\n- We have conducted comparative experiments by applying [1], a Non-Meta-learning based Few-shot multimodal NER technique, to our specific problem setting. With the same fine-tuning steps (T=15) and learning rate, the F1 results of [1] on the 4-way 4-shot setting is only **0.115**, while the F1 results of all gradient-based meta-learning methods (including our proposed method) is **over 0.5** and that of all metric-based methods (including our proposed method) is **over 0.23**. However, it may be not fair to compare with [1] since our paper studies a different setup. Despite this nuance, we intend to incorporate these results into the revised version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Paper deals with the multimodal few shot entity recognition problem which is very real and practical problem\n2. Sound experiments have been done among the meta learning approaches proposed and other meta learning approaches .\n3. Results are good improvement over the methods compared.", "reasons_to_reject": "1. Task definition needs better clarity. The paper claims to solve a niche problem which has no dataset but is not clear what the niche is.\n Some of the claims made by the paper about the 'uniqueneness' of the problem are :\nClaim by paper : Line 071- Entity occurences vary  dramatically from one document to another \nConfusion : Is it not true that in any NER case in real word this happens ?\nClaim 2 by paper  : N way means S&Q do not contain any entity types other than E(epsilon)  \nConfusion : Again is that not true in any NER task ?\nSimilar claims are repeated in Line 183-200 again with the same set of confusions. I am finding it hard to differentiate these claims of uniqueness from a 'Few shot multimodal NER' setting.\n2.Comparisons are neeed against Few shot multiomdal NER techniques , not just meta learning based ones. particularly if the previous point is valid.", "questions_to_authors": "Please provide more clarity with respect to my Point 1 of previous section .", "paper_topic_and_main_contributions": "The paper proposes to solve the problem of entity extraction from visually rich document for unseen entity types. The paper proposes a new dataset for this problem . The solution provided by the paper for this problem consists of three parts 1) multimodal encoder that encodes the input document into a 'task-dependent embedding space' 2) A decoder that does the token labelling 3) a meta learner that learns parameters for the encoder and decoder such that labelling can be done on the out-of-task distribution as well. They propose a few techniques for the meta learning part.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "L0E8tcr12P", "reviewer": "Reviewer_1N1p", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### ***Weakness 1: the derived dataset is based on the current open-source ones for document understanding, which are small in their size and have a very limited amount of classes. A dedicated dataset that is built specifically for the purpose of studying few-shot learning for document entity retrieval is needed and it would benefit from a larger number of entities.***\n\n***Response:*** \nWe appreciate the comments from the reviewer! We understand the importance of a larger-scale dataset for demonstrating the effectiveness of our method more comprehensively. While acknowledging its significance, we opted to use the derived FewVEX dataset for experiments for the following reasons:\n- ***Sufficiency of Current Dataset to Our Objective***: As we are mostly studying the occurrence shift of entities across documents (as demonstrated in the two examples we shared with other reviews), the most important aspect of the dataset is to formulate the entity-document distributions that can authentically simulate real-world entity-level few-shot situations, where each task needs only a few labeled classes and documents. From that perspective, a dataset built from the existing document understanding ones should be enough to generate a multitude of distinct entity-level few-shot simulations/tasks to effectively evaluate the few-shot learning methods.\n- ***Algorithmic Potential for Future Enhancement:*** While the current dataset is enough to validate our algorithms, we concur that further improvement on increasing the number of entity types and documents will better benefit this research community. In our paper, we made this future enhancement possible by proposing the **Algorithm 1 (in Appendix B on Page 12)**, which facilitate the auto-generation of a large amount of distinct task simulations from open-source document understanding ones with a large amount of classes. We currently select CORD and FUNSD as the input source of Algorithm 1 due to their relatively larger number of classes rather than other available document understanding datasets. If, in the future, document understanding datasets encompassing a wider array of entity types become available, our Algorithm 1 can help quickly derive few-shot datasets with a more extensive number of classes and documents. Using the method proposed in the paper helps to reduce the amount of time used to generate data for few-shot learning cases.\n- ***Established Practice in Few-Shot Learning:*** Deriving few-shot datasets from other non-few-shot sources is well-established in the image few-shot learning domain. Notably, MiniImageNet for few-shot classification emerged from the ImageNet dataset [1].\n\n\n\n[1] Vinyals, Oriol, Blundell, Charles, Lillicrap, Tim, Wierstra, Daan, et al. Matching networks for one shot learning. In Neural Information Processing Systems (NIPS), 2016\n\n### ***Weakness 2: the paper does not provide a comprehensive analysis of the computational complexity of the proposed methods, which could be a concern for practical applications.***\n\n***Response:*** We appreciate the comments from the reviewer. We present the computational complexity analysis for both the baseline methods and our proposed approach as follows.\n\n|                                  | Total Parameters | Outer-loop Parameters | Inner-loop Parameters | Inner-loop Space Complexity | Inference Time Complexity | 4-way 4-shot F1 Performance (T=15) |\n|-------------------------|:------------------:|:--------------------------:|:---------------------------|:---------------------:|:-------------------------------:|:-------------------------------------------:|\n|     Fine Tuning  [2]      |    11.1M             |   0                                |                  11.1M         |            O(N)            |          O(T)            | 0.115 |\n|         Reptile         |    11.1M   |  11.1M  |  11.1M | O(N)  |  O(T) | 0.51 |\n|    **Reptile+HC (ours)**  |    11.2M   |  11.2M  |  11.2M | O(N+2)|  O(T) |**0.64**|\n|                  ANIL                    |    11.1M   |  11.1M  |   102k | O(N)  |  O(T) | 0.50 |\n|     **ANIL+HC (ours)**           |    11.2M   |  11.2M  |   200k | O(N+2)|  O(T) |**0.51**|\n|       Vanilla ProtoNet              |    11.0M   |  11.0M  |    0   | O(DL) |  O(1) | 0.03 |\n|       ProtoNet + EOD              |    11.0M   |  11.0M  |    0   | O(DNK)|  O(1) | 0.23 |\n| **ContrastProtoNet (ours)**  |    11.0M   |  11.0M  |    0   | O(DL) |  O(1) |**0.60**|\n\n\nIn the above table, N represents the number of classes in a task, and K is the number of labeled entity occurrences per class. L denotes the number of tokens in a document, set as L=320. D signifies the encoded parameter dimension, with D=768. T corresponds to the number of fine-tuning steps in the inner loop. The number of steps has been preset for all gradient-based methods to observe their convergence speed. Our method achieves superior results with the same time complexity and a marginal (0.01%) increase in parameter size. The primary memory cost (11M) is attributed to the BERT-like encoder.\n[2] Zilong Wang and Jingbo Shang. 2022. Towards few-shot entity recognition in document images: A label-aware sequence-to-sequence framework. arXiv preprint arXiv:2204.05819.\n\n### ***Others: missing reference on Federated Learning on line 485.***\nThank you for bringing the missing reference to our attention. The federated learning technique we employed is FedAvg [2]. We will correct this in the revised version.\n\n[2] McMahan, Brendan, et al. \"Communication-efficient learning of deep networks from decentralized data.\" Artificial intelligence and statistics. PMLR, 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the paper presents a novel approach to the challenging problem of Few-shot Visually-rich Document Entity Retrieval (FVDER) and provides significant improvements over the existing baselines\n- the proposed methods achieve major improvements over the baselines for FVDER\n- the paper also identifies potential directions for further improvement, which can inspire future research in this area\n- the proposed dataset that the authors introduce can help the community benchmark new approaches on the proposed task", "reasons_to_reject": "- the derived dataset is based on the current open-source ones for document understanding, which are small in their size and have a very limited amount of classes.  A dedicated dataset that is built specifically for the purpose of studying few-shot learning for document entity retrieval is needed and it would benefit from a larger number of entities\n- the paper does not provide a comprehensive analysis of the computational complexity of the proposed methods, which could be a concern for practical applications.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes an approach for the extraction of entities from visually rich documents. \n\nThe main contributions of this paper are: \n1. Proposing a new formulation of the Few-shot Visually-rich Document Entity Retrieval (FVDER) problem as an entity-level, N-way soft-K-shot learning under the framework of meta-learning.\n2. Introducing a new dataset that is designed to reflect practical problems in FVDER.\n3. Exploiting a wide range of approaches, including metrics-based and gradient-based meta-learning methods, along with a few new techniques for this new setting.\n4. Achieving major improvements over the baselines for FVDER.\n5. Identifying potential directions for further improvement, including a better algorithm that distinguishes between the OTD and ITD and a formulation that considers the correlations between entity instances within each meta-learning task. ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- missing reference on Federated Learning on line 485"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission studies a very real and practical problem of multimodal few shot entity recognition and handles the imbalance issue at entity-level with a newly proposed method. The soundness of the proposed method is validated by the original experiment results and the additional results provided during the rebuttal.
Decision: Accept-Findings