Few-Shot Document-Level Event Argument Extraction
Year: 2023

Abstract:
Event argument extraction (EAE) has been well studied at the sentence level but under-explored at the document level. In this paper, we study to capture event arguments that actually spread across sentences in documents. Prior works usually assume full access to rich document supervision, ignoring the fact that the available argument annotation is usually limited. To fill this gap, we present FewDocAE, a Few-Shot Document-Level Event Argument Extraction benchmark, based on the existing documentlevel event extraction dataset. We first define the new problem and reconstruct the corpus by a novel N -Way-D-Doc sampling instead of the traditional N -Way-K-Shot strategy. Then we adjust the current document-level neural models into the few-shot setting to provide baseline results under inand cross-domain settings. Since the argument extraction depends on the context from multiple sentences and the learning process is limited to very few examples, we find this novel task to be very challenging with substantively low performance. Considering FewDocAE is closely related to practical use under low-resource regimes, we hope this benchmark encourages more research in this direction. Our data and codes will be available online1.

heading: 1 Introduction
text: Event argument extraction (EAE), a sub-task of event extraction, is a fundamental task for many downstream NLP applications in the IE community. For example, events and arguments play an important role in the knowledge base population from unstructured data (Ge et al., 2018; Li et al., 2021). And the real world public affairs management relies on recognizing the event arguments from daily news and social media (Yuan et al., 2018; Ritter et al., 2012). Although tremendous progress has been made under the supervised setting, current
1https://github.com/Xianjun-Yang/FewDocAE
neural models typically rely on large-scale humanannotated data, which is not reliable considering the huge amounts of novel events and arguments emerging in many fields every day. Few-shot learning (FSL) (Fei-Fei et al., 2006) is proposed to tackle such limitations to make machine learning models more applicable given limited annotated examples and has been used a lot in the IE area (Han et al., 2018; Ding et al., 2021; Lai et al., 2021).
Previous research (Yang et al., 2019; Tong et al., 2020) mainly focuses on sentence-level event extraction, such as the popular ACE2005 (Doddington et al., 2004) dataset. In recent years, researchers have begun to realize that the complete event and arguments actually spread in a full document or paragraph (Li et al., 2021; Ebner et al., 2020; Li, 2022). And the focus starts to turn into documentlevel event extraction motivated by the newly proposed large-scale document-level corpora, namely the WikiEvents (Li et al., 2021), RAMS (Ebner et al., 2020) and the recent largest corpus DocEE (Li, 2022). Following these datasets, many novel methods for solving such new challenges brought by the longer context have also been investigated and witness significant progress (Du and Cardie, 2020; Li et al., 2021; Xu et al., 2022).
However, the traditional supervised learning methods heavily rely on large-scale annotated training data, but we are witnessing new events every day due to the rapid emergence of new affairs. Thus it is not durable to greedily make large collections of newly appeared events for real-life applications. Therefore, more attention has been paid to few-shot event extraction. But the current research has only considered the few-shot EAE on a single sentence (Lai et al., 2021, 2020; Deng et al., 2020), ignoring the big gap between realistic scenarios. Therefore, we aim to pave a new way for few-shot EAE at the document level towards urgent data scarcity problems on the complete documents.
The recently released large-scale document-level
8029
EE datasets include RAMS (Ebner et al., 2020) and DocEE (Tong et al., 2022), and their statistics is shown in Table 1. RAMS and DocEE contain 139 and 59 event types, 65 and 158 arguments types, with a total collection of 9, 124 and 21, 450 documents, respectively. While other datasets such as WikiEvents (Li et al., 2021) and MUC-4 (Grishman and Sundheim, 1996) only contain extremely limited types and documents, thus not suitable for our settings. Therefore, we formulate our FewDocAE based on the largest DocEE dataset. Different from FSL for single sentences by traditional N -way-KShot sampling, a novel N -Way-D-Doc sampling strategy is proposed for our document-level task, as can be seen from the example in Fig. 1. Besides, previous few-learning problems often fall into the pitfall of robust and fair evaluation, and we follow the FLEX (Bragg et al., 2021) (Fewshot Language Evaluation across(X) many transfer types Principles) to design our settings to avoid such weaknesses. Moreover, prototypical networks (ProtoNet) (Snell et al., 2017) have been proven to be very powerful for solving few-shot problems by representing each category as a prototype in both Vision (Pan et al., 2019; Dong and Xing, 2018) and NLP (Sun et al., 2019; Gao et al., 2019) domains. We combine ProtoNet and a pre-trained language encoder for establishing baselines on our new task and provide a comprehensive analysis.
The key contributions of this work include:
• We are the first to introduce few-shot document-level event argument extraction,
greatly extending supervised document-level EAE to few-shot scenarios.
• We reconstruct a realistic FewDocAE dataset along with a new few-shot sampling algorithm, N -Way-D-Doc sampling.
• We conduct comprehensive experiments to provide benchmark results and find the tasks are extremely challenging and worth further investigation.

heading: 2 Related Work

heading: 2.1 Document-Level Information Extraction
text: In the IE community, previous work mainly focuses on sentence-level tasks. For example, the commonly-used relation extraction benchmark TACRED (Zhang et al., 2017), the ACE05 2 and KBP2017 3 event extraction datasets, the CoNLL2003 (Sang and De Meulder, 2003) and OntoNotes 5.0 (Pradhan et al., 2013) named entity recognition corpora, all focus on single sentence-level semantics. Since information extraction often involves document-level reasoning, recently there have been great efforts to contribute document-level benchmarks. For instance, the DocRED (Yao et al., 2019) is the largest dataset to extend relation extraction to the document level. The recent RAMS (Ebner et al., 2020) and DocEE (Tong et al., 2022) corpora focus on multi-sentence event extraction. Although
2https://catalog.ldc.upenn.edu/LDC2006T06 3https://tac.nist.gov/2017/KBP/
tremendous progress has been made in the information extraction area on sentence-level tasks, the emerging document-level datasets raise new challenges. The difficulty mainly comes from the long context semantic representation brought by multiple sentences and the extremely unbalanced label distribution.

heading: 2.2 Document-Level Event Argument Extraction
text: Event extraction can be classified into triggerword and no-trigger word based extraction, including event detection and event arguments extraction. Many approaches and datasets (Petroni et al., 2018; Hürriyetoğlu et al., 2021; Giorgi et al., 2021; Zavarella et al., 2022) across diverse domains have been proposed for document-level argument extraction to go beyond single-sentence inference. For example, (Ebner et al., 2020) build RAMS to include cross-sentence argument annotations but still limits the arguments around the event in a 5- sentence window. An end-to-end generative transformer (Du et al., 2021) regards argument extraction as a template-filling task. Besides, (Li et al., 2021) formulates the task as conditional generation following event templates, and contributes to the WIKIEVENTS dataset, which consists of only 246 documents with less than one-fourth of annotated cross-sentence arguments. Very recently, (Du et al., 2022) introduce a new global neural generationbased framework by constructing a document memory store to record the contextual event information for improving capability. The largest documentlevel event extraction dataset is the DocEE (Tong et al., 2022), which consists of 27, 000+ events, 180, 000+ arguments over 27, 485 Wikipedia articles. In this paper, we use the DocEE as base set for our task.

heading: 2.3 Few-Shot Argument Extraction
text: Few-shot learning for information extraction is proposed to tackle such circumstances when only limited instances are annotated. There have been growing interests under few-shot settings for named entity recognition (Ding et al., 2021; Das et al., 2022), and relation extraction (Han et al., 2018; Popovic and Färber, 2022) under single-sentence and document-level scenarios. There has also been research (Deng et al., 2020; Lai et al., 2021; Feng et al., 2020; Lai et al., 2020) for few-shot event extraction within single-sentence. However, to the best of our knowledge, there is no research about
document-level few-shot event argument extraction until the submission date.
To fill this gap, this work focuses on few-shot learning for document-level argument extraction. Instead of building a new dataset from scratch, we aim at leveraging the existing supervised dataset for reconstructing the instances by a novel N -WayD-Doc sampling strategy, inspired by similar work (Sabo et al., 2021; Popovic and Färber, 2022).

heading: 3 Task Formulation

heading: 3.1 Argument Extraction Definition
text: The event argument extraction usually depends on the first-stage detected events, but we assume gold event labels to reduce the task complexity in our work. This is opposite to joint extraction where the task is to jointly extract all events and their associated arguments all at once (Sha et al., 2018; Yang and Mitchell, 2016). Since the DocEE dataset (Tong et al., 2022) follows the main event extraction (Hamborg et al., 2018) setting where no trigger words exist and the article title t and the article a itself together determine the event type, we follow their setting and assume the event type e is given, then aim at extracting all related arguments with types Re.
Formally, given a document D = {w1, ..., w|D|} and its corresponding event type e, where |D| is the total number of words, the event argument extraction aims to detect the boundaries and types for all possible continuous spans {wstart, wend} in the document D according to event argument types Re.

heading: 3.2 Document-Level Few-Shot Argument Extraction
text: Following previous work about sentence-level fewshot event detection (Deng et al., 2020), we define the document-level few-shot argument extraction as the following. Given the event instance e, its associated argument types set Re, the support set S and the query set Q, the few-shot task T is defined as:
S = {..., Rsi , ...}, Rsi = (Dsi , {..., (bsi , tsi ), ...)}
Q = {..., Rqi , ...}, R q i = (D q i , {..., (b q i , t q i ), ...})
T = {S,Q} where (bi, ti) represents the i-th event argument boundaries and type in document Di in the support
s and query q set. Rs/qi is the set of all the annotated arguments in Di and S/Q is the combinations of Rs/qi from different documents. Following the episode training for few-shot learning, a task T is one episode aiming to predict all the instances in Q given S. The few-shot learning is usually formalized as an N -Way-K-Shot problem, which means that there are N possible argument types and K supporting instances for each argument type for every task T . Note that the argument types set Rtrain and Rtest are disjoint.
In practice, given all the support documents in S, we want to extract all the arguments for documents in Q. Since it is not guaranteed each support instance Dsi contains only one argument, the K in the traditional N -Way-K-Shot setting is no longer guaranteed to be an integer. Previous research (Yang and Katiyar, 2020) tries to use greedy sampling to guarantee the strict K shots requirements for sentence-level few-shot NER task, but this is not applicable due to the sparse density of arguments in the document as also been observed by (Ding et al., 2021). And they loose the K shots requirement to K∼2K shots. However, this is still not realistic under a document-level setting since the arguments spread become even sparser
and K∼2K shots are still not guaranteed. It is notable that (Popovic and Färber, 2022) tackle this problem for few-shot document-level relation extraction by D-Doc setting where both N and K are variables between documents and individual episodes. We argue that variable N is not suitable for deploying applications and will make models complicated, so we design a novel N -Way-D-Doc sampling, where N and D are both fixed, resulting in variable K shots.

heading: 3.3 Domain Split
text: To make our task closer to realistic applications, we consider three settings for investigating the difficulty of tasks and model performance. In the Indomain circumstance, we manually choose event and event argument types from the same coarsegrained label sets to ensure that they share the same domain knowledge. To explore the meta-learning ability with varying amounts of training bases, we further set In domain (small) and In domain (base) with a small and medium numbers of event types contained in the training set. The authors (Tong et al., 2022) provide Cross domain scenario, where the training and test labels are entirely disjoint, sharing no mutual domain information. We follow their splits for our domain adaptation task.

heading: 4 Constructing Few-Shot Episodes
text: In this section, we talk about the details regarding the dataset conversion and how we obtain episodes for training. For our FewDocAE task, the key components for constructing a realistic few-shot dataset are made of two parts: avoiding data leakage and constructing effective support and query pairs. Generally, to avoid leaking new event arguments information in the training phase we replace all other labels of arguments contained in the validation/test sets with O. In this way, it is guaranteed that they have disjoint argument types sets, and this is also close to the realistic scenarios. To provide each episode with a support and query set, we greedy sample instances to make sure they satisfy our N - Way-D-Doc choice. Also, they support and query
Algorithm 1 N-Way-D-Doc Sampling for FewShot Episode Generation
Require: Dataset X , Label set Y , N , D Ensure: 1: S ← ∅; // Init the support set 2: CountN← 0; //Init the count of entity types; 3: CountD← 0; //Init the count of documents; 4: while CountN ̸= N or CountD ̸= D do 5: Randomly sample (x,y) ∈ X ,Y ; 6: Compute CountN and CountD; 7: if CountN > N or CountD > D then 8: Continue; 9: else 10: S = S⋃(x,y) ; 11: Update CountN,CountD ; 12: end if 13: end while
instances should always come from different documents.

heading: 4.1 Choosing Datasets
text: We initially consider the two largest documentlevel event extraction datasets for our tasks: RAMS (Ebner et al., 2020) is annotated in a 5-sentence window around each event trigger and contains 9, 124 annotated events from news based on an ontology of 139 event types and 65 roles. In addition, DocEE includes 21, 450 document-level events with 109, 395 arguments, making it the largest document-level event extraction dataset. Since we aim at building a document-level benchmark, we finally exclude RAMS for its 5-sentence limits and narrow argument types. Eventually, we choose DocEE for our FewDocAE task from their original release 4. Besides, in order to make sure there are enough examples in each episode for support and query, we exclude all events and argument types that have lower than 2 annotated examples in the train/validation/test set.

heading: 4.2 Determining Event Arguments Types
text: In the released DocEE corpus, there are 31 hard news event types and 28 soft news event types with their corresponding arguments. Their original paper follows no-trigger words design (Nguyen et al., 2016; Zheng et al., 2019) and assumes one main event per document. For arguments, there are
4https://github.com/tongmeihan1995/DocEE
358 event argument types belonging to those 59 event types. The argument annotation is done on the whole document resulting in some documents could contain up to 40 sentences and 7k words.
Since different events could still contain the same argument types, we could mask the overlapped arguments in two ways. The first one masks all arguments in the training set with O if they also appear in the val and test sets as used in (Ding et al., 2021), while in the second strategy, we can mask all the arguments in the val and test sets if they are shared by the training set. The intuition is that the former one has the risk that the model is trained with O types, but is forced to predict their true labels which are not O during testing. We believe the latter setting reduces the difficulty by making sure that the new arguments appearing during the prediction stage are not labeled with O during training, and use this strategy for all our experiments.
For the In domain (small) setting, we manually choose 14 and 15 event types for constructing the validation and test episodes, while leaving all other 32 event types for training purposes. This results in disjoint event sets for train/val/test, but they share some domain knowledge. Since the number of event types used for training is only about two times that of testing, we believe this setting is more challenging. The intuition is that for few-shot learning, we aim to get a good feature extractor during training using massive base data so that those learned features could benefit the model by predicting novel instances. We use this setting for exploring the few-shot learning limitation when the training base is small. For the In domain (base) setting, we use a larger training base with 49 event types and adopt the remaining 10 event types for validation and testing. Again, our choice of event splitting guarantees they share in domain event types. In contrast to previous situation, we now aim at investigating the full capability of few-shot learning given enough training base.
For Cross domain, we follow the original event splits in DocEE where the authors choose the natural disasters events as the target domain, including Floods, Droughts, Earthquakes, Insect Disaster, Famine, Tsunamis, Mudslides, Hurricanes, Fire, and Volcano Eruption, and leave the remaining 49 event types as source domains.
Besides, there are six arguments, including Date, Causes, Areas affected, Location,
Casualties, and Losses that occur frequently in all the splits. To fully leverage the capability of meta-learning during the training phase, we leave those arguments in the training set only and mask them as O in the val and test sets. The full statistics of our domain split result is shown in Table 2.

heading: 4.3 Sampling Strategy
text: The traditional N -Way-K-Shot sampling for fewshot learning fails when applied to our documentlevel settings. The reason is that one document with the sparse spread of event arguments could contain one or many arguments. Thus greedy sampling K shots instance can not be guaranteed. Soft sampling methods like N -Way-K∼2K Shots in (Ding et al., 2021) still do not work for our long documents since K∼2K Shots are still hard to be satisfied. Note that (Popovic and Färber, 2022) adopts D-doc sampling for document-level relation extraction. However, their approach can not guarantee a fixed number of N classes. For example, their 1-Doc results in 2.18-Way-2.36-Shot instances on average. We argue that variable N is not ideal for our task.
We introduce a new sampling strategy, N -WayD-Doc sampling as shown in Algorithm 1, to guarantee a fixed number of N classes within D documents. In our approach, we first pick N event argument types, then randomly sample D documents and keep all the arguments within N types in the D documents unchanged while discarding other argument types. This process is not finished until there are exactly N classes within D documents. Besides, few-shot learning is notoriously famous for lacking challenging-yet-realistic testing setups and failing to employ careful experimental design (Bragg et al., 2021). To make our benchmark more robust, we sample episodes by making sure they evenly come from combinations of different documents, events, and argument types.

heading: 4.4 Sampling Results
text: We sample around 30k∼50k episodes for the training and around 3k episodes for validation and testing sets under 3-Way-2-Doc and 6-Way-2-Doc settings. For the 3-Way-1-Doc setting, since not all documents have at least three unique arguments, we sample about 30k for training and 1.5k for Val and test. We also show the argument type characteristics of our various N -Way-D-Doc sampling results in Table 3. As we can see, the average sampled number of arguments for both settings
is close. For the 3-Way-1-Doc settings, there are around 4.47/3 = 1.5 arguments per type, while only around 1.14 and 1.10 arguments per type for other settings, demonstrating the diversity of our splitted domains.

heading: 4.5 N-Way-D-Doc Choice
text: Due to the unique nature of few-shot documentlevel extraction, we only test limited combinations of N-way and D-doc for two reasons. First, there are only limited documents that simultaneously contain the same N argument types, thus we can not find enough samples when extending to more than 6-way. Second, we only keep 1/2-doc for the two reasons: on the one hand, extending to more documents requires doing sequence labelling on much longer documents that consumes much more GPU memory that we struggle to handle even by a 40G GPU. Similar memory issue has also been reported by (Sabo et al., 2021) when handling fewshot learning for relation extraction. On the other hand, adding more documents will also increase the number of NOTA argument types, which will be detrimental for extracting useful real arguments since the vast majority would be None types. Considering the current settings already result in relatively low performance, we do not aim to further increase the challenge.

heading: 5 Models
text: Previous work on document-level EE using BERT_Seq (Du and Cardie, 2020; Tong et al., 2022) demonstrate the success of using a pretrained BERT model to sequentially label words in the article. And the superior performance of the long document transformer (e.g. Longformer (Beltagy et al., 2020)) has also been proven to improve the argument extraction task (Tong et al., 2022). We thus follow their baseline settings and use BERT or Longformer as document encoders for our task. In order to adapt to our few-shot setting, inspired by the successful applications of the prototypical network (ProtoNet) (Snell et al., 2017) for meta-learning, we assume there exists one prototypical representation for each argument type. Then we implement the models by extending ProtoNet to language encoder with token-level similarity.

heading: 5.1 Document Encoder
text: We adopt Longformer (led-base-163845) and BERT (bert-base-uncased6) as our encoder for all the experiments. In order to handle long sequences, we split all inputs with a chunk length of 1024 and 512, respectively.
Formally, suppose the document D = {w1, ..., w|D|} where wi represents the ith token and |D| is the maximum length. By feeding the tokens into the document encoder, we get the contextualized token representation:
[h1, ..., h|D|] = Encoder([w1, ..., w|D|])

heading: 5.2 Prototypical Networks (ProtoNet)
text: The ProtoNet approach (Snell et al., 2017) assumes there exists one prototypical representation for each argument class and learns a metric space where categorization is performed by labeling each query term with the value calculated from the distance between prototype representations that are closest to it. In practice, we use the average representation of all tokens in each argument to represent the contextualized representation of that argument type. Formally, for support set Sri containing all the arguments of type ri, following previous step ht is the representation of token t. By calculating all the prototypes,
ri =
∑ t∈Sri ht
|Sri |
we get the argument representation set R = {r1, ..., rN , rN+1} where ri represent the ith argument prototype and rN+1 represents the O type. Then for the query token hq, the target label is assigned as
label = argmin i (d(ri, hq))
. We use the L2 distance as the distance metric. Besides, we also add a Baseline model based on ProtoNet-LongFormer without any finetuning. 5https://huggingface.co/allenai/led-base-16384 6https://huggingface.co/bert-base-uncased
The reason is that training on many episodes is very costly as will be shown below. And we are interested on the original performance and how much benefits the finetuning could bring in.

heading: 5.3 Nearest Neighbor Tagger (NNShot)
text: NNShot (Yang and Katiyar, 2020) is a simple but strong system based on token-level nearest neighbor classification for the few-shot sequence tagger task. It first obtain contextual representations for all tokens in their respective documents. Then it assigns the token q a tag ri corresponding to the most similar token in the support set:
label = argmin i (arg min i′∈Sri
(d(r′i, hq)))
where Sri represents the set of support tokens with ri tags.
However, simply extending the sentence-level NNShot to a long document is not plausible. For example, for two documents both with 4, 096 tokens and each token has an embedding dimension of 768, the token-level similarity computation requires more than 50GB GPU memory. To make token similarity calculation of long inputs possible, we use another linear layer to reduce the 768 dimensional representation into 32 dimension and use the L1 distance as the distance metric for computational efficiency.

heading: 5.4 ProtoNet-MNAV
text: Here we adjust the Multiple NOTA(None-Of-theAbove) Vectors(MNAV) proposed by (Sabo et al., 2021) for few-shot relation extraction to our FewDocAE since they both face the same issue that the majority labels belong to NOTA. Instead of initializing the NONA vectors by randomly as by (Sabo et al., 2021) or from sampled support sets as in (Popovic and Färber, 2022) and then gradually update them, we adopt a K-means MNAV strategy. Specifically, we perform a K-means clustering for all NOTA representations where K is set to be a hyperparameter. Then for ProtoNet-based models, we still determine the label type by calculating their token similarity. And we attribute all token nearest to the K NOTA vectors as NOTA type. This
way we ideally reduce the risk of only representing many NOTA token by one vector as also pointed by (Allen et al., 2019), since there might be multiple NOTA Prototypes. We always use LED as the encoder for ProtoNet-MNAV method.

heading: 6 Experiments and Results
text: We show the experimental results and analysis in this section. More experimental details and configurations can be found in Appendix A. In general, one experiment costs around 15 hours and in total there are more than 100 runs, which also limits more advanced models that we can choose.
In Domain For the In domain setting where the training and test examples are sampled from the same domain, the performance is exhibited in Table 4 and Table 5 for the In domain (small) and In domain (base) results, respectively. As we can see, the baseline without finetuning can not guarantee good performance, and the ProtoNetLongformer models consistently outperform the ProtoNet-BERT by a large margin under three N - Way-D-Doc settings. On the one hand, this gap can be explained by the superior encoding ability of long documents by LongFormer. On the other hand, the results convince our motivation by extending to document-level argument extraction as a large portion of arguments can only be extracted across sentences, as also confirmed under supervised condition by (Tong et al., 2022). Besides, we also observe a significant performance drop when moving from In domain (base) to In domain (small) under most results, which clearly manifests the benefits of using training base with broader event types. This validates the intuition that more diverse training base can help train a better argument feature extractor. Also, we can see that as we increase the Ways of N and Docs of D, the overall results continue increasing, demonstrating instances with more ways and more Docs are easier to be predicted. However, ProtoNet-MNAV does not bring performance gain as expected, possibly due to the more unclear decision boundary as illustrated in Appendix C.2. In general, the current models remain relatively low results, demonstrating the challenges of our FewDocAE task in document-level. We additionally show some case study in Appendix C.
Cross Domain The overall arguments extraction results are shown in Table 6 for the Cross domain
setting. Compared with its In domain (base) counterpart, the performance degradation is witnessed under both ProtoNet-Longformer and ProtoNetBERT models in all settings. This is expected since we split the cross domain to avoid the in domain knowledge and such domain adaptation is more challenging.
We also test the performance using NNShotLongformer in Table 7 under Cross domain. To be compatible with computation memory limits brought by NNShot token-level similarity calculation, we do not take whole documents and split the documents within chunks of 1024 tokens. For the 1-Doc setting, an average F1 score of 5.9%, and for the 3/6-Way-2-Doc settings, F1 of 3.73% and 4.59% points are observed using NNShot model. The performance decreases a lot by NNShot, which we attribute to the likely underrepresented representation by dimension reduced operation. However, due to the memory limits, we leave more investigation for further research. Besides, how to efficiently adapt NNShot to long documents is also an open challenge.
Overall Analysis We attribute the observed results to three main reasons: First, the documentlevel argument extraction involves reasoning over a much longer context compared to the sentencelevel. Actually, the average number of sentences for DocEE is 30.71 as pointed out in Table 1, which dramatically increases the difficulty of effective encoding over long documents. Even though Longformer can capture attention over long sentences, the ability is still limited. On the other hand, document-level argument extraction faces the new challenge of extremely unbalanced label distribution with more than 95% of label O. Compared to its sentence-level counterparts, where the label unbalance already degrades the performance, the document-level sparse distribution of arguments further exacerbates the unbalanced distribution. The majority of O labels make it difficult for the model to learn a good representation of arguments among the representation space. Besides, to make the few-shot learning close to a realistic setting, we follow the long-tail arguments distribution of the original DocEE dataset. This extremely unbalanced setting is a good testbed for validating the model ability due to its similar distribution of many real-world few-shot problems, as also pointed by (Sabo et al., 2021). As the results suggest, the long-tail distribution makes it hard for models to

heading: In domain (base) Cross domain
text: uniformly focus on all labels. Third, due to the high GPU memory and computation requirements brought by long documents, we only aim at providing benchmark baselines results. More advanced methods might help, but we leave it for future work.
Error Analysis Finally, we report the false positive (FP) and false negative (FN) scores in Table 8 for two domain using ProtoNet-Longformer. The overall FP results demonstrate that most O are correctly predicted. However, considering the large number of O labels compared with real arguments, even a small portion of FP still leads to a large performance drop of final results. As for real arguments prediction, the main misclassification errors come from assigning one argument type to another type considering FN is low.

heading: 7 Conclusion
text: In order to handle new emerging event arguments with limited annotations and adapt it to the realworld document-level scenario, we propose FewDocAE benchmark to advance the research of few-
shot learning for document-level event argument extraction. We conduct comprehensive experiments by extending previous models into our task under in-domain and cross-domain. Our experiments confirm the necessity of moving to document level by showing that current models still witnesses suboptimal performance. We also demonstrate the benefits of using a more diverse training base to learn a good argument feature extractor. The current results show that FewDocAE is challenging due to the long document and limited examples, as well as the intrinsic charisma of few-shot learning. The relatively low extraction score illustrates the difficulty of this novel task, in the meanwhile it also provides new chances for advancing this field. In summary, we hope FewDocAE shed new light on a more realistic but challenging setting for event argument extraction. In the future, we hope to investigate more advanced methods for solving this problem.

heading: 8 Limitations
text: As we mentioned, we only focus on event arguments with the assumption that event type is already provided. However, this is not always true for many applications in real life scenarios. But it would be out of the scope of this work to combine them together, so we leave it for future work. Besides, considering the long input of document-level extraction, the computing memory consumption significantly increase to tens of times compared with its sentence-level counterpart. We only consider the 1/2-Doc cases, although in reality more docs are possible. We believe finding a solution for decreasing the memory requirements would be of great impact for future research in this direction.

heading: 9 Acknowledgments
text: Xianjun Yang was supported by the UC Santa Barbara NSF Quantum Foundry funded via the Q-AMASEi program under NSF award DMR1906325.

heading: A Experiments
text: A.1 Evaluation Metrics Since we treat this event arguments extraction as a sequence labeling task, we employ IO notation, where all tokens within an argument type are labeled as I-type while all other tokens are labeled as O. Besides, we report all the performance on non O types. The prediction is only considered as correct when all tokens within that argument are correctly classified. We use macro precision, recall, and F1 score to measure the performance.
A.2 Experimental Configuration For all transformer-based models, we employ the released model from HuggingFace7 and set the learning rate to [1e− 4, 1e− 5, 5e− 5], of which 1e− 5 is the best parameter except for 3w1d. For 3w1d tasks, we use a learning rate of 1e− 6, otherwise, it will not converge. We try the different batch size of [1, 2, 4, 6], of which 4 and 6 does not lead to converging, 2 achieves the best performance. We use AdamW (Loshchilov and Hutter, 2019) as the optimizer and gradient clipping of 1.0. We use the NLTK8 package for sentence tokenization. All models can be put into four NVIDIA A60 GPUs with an RAM of 48GB each. The training procedure takes around 15 hours and 10 hours for 60k iterations to complete for 2-Doc and 1-Doc settings, respectively. We report the mean and variance for all experiments under two random seeds, except only one run for ProtoNet-MNAV. The validation is done by every 4k interaction on the training episodes, and we use the best checkpoint from the validation results for testing. The number of query instances during testing is always set to 1. For ProtoNet-MNAV, we tried the hyperparameter K ranging from 2 to 6 and did not observe obvious difference. We report the results based on K=6.

heading: B Memory and Computation Issues
text: Due to the length distribution of this documentlevel task, we set at least 1,024 tokens as chunk
7https://huggingface.co/models 8https://www.nltk.org/
length for input to LED-based models. However, a 2-Doc setting results in 4-doc documents coming from both the query and support sets. Considering the argument-extraction is conducted at every token level, the similarity score calculation imposes a severe memory issues for adapting more complicated methodologies. We believe that this is a big challenge for few-shot document-level tasks, which is not only a issue for small language models (Sabo et al., 2021) but also true for large models like GPT-3 (Brown et al., 2020) and leave more exploration for future work.

heading: C Case-study
text: C.1 Predictions vs. True
Here in Figure 2 and Figure 3 we show two example case study of how our predictions differ from the true labels. The different color corresponds to different event argument types. And we also highlight whether the predictions are accurate or not. All examples are drawn from the test set in in-domain(samll) setting and the predictions are always the ProtoNet-LongFormer model. As we can see, the majority of predictions are wrong with only a few exceptions in Fig. 2. Besides, in Fig. 3 we additionally show the false positive results which also accounts a large protion for the final performance. In general, the current model can not well handle the document-level predictions under few-shot setting, and the prototypical representation for different labels still struggle with token classification.
C.2 Visualizations
Here we plot the two dimensional t-SNE9 projection of the prototypical embeddings in Fig. 5 and Fig. 4. Ideally 7 or 12 clusters as expected for ProtoNet-LongFormer and ProtoNet-MNAV models. But the results show that the clusters are not always well seperated from each other, which might explains the reason why we still get low performance. Actually, we can also clearly see some clusters indeed include many same argument types, like class 0, 1 and 6 in Fig. 4, but these clusters still spread across the many different locations. For example, we can clearly see 3 big clusters for 6 and 5 big clusters for 1, which indicts that multiple prototypes exist for different argument types. However, when we extend to multiple NONA vectors
9https://scikit-learn.org/stable/index.html
by ProtoNet-MNAV model, we even see a little performance drop as mentioned in Section 6. Ideally multile NONA vectors can better represent more diverse NONA class but as we can observe from Fig. 5 that the multiple NONA vectors(0 to 5) actually make the overall cluster boundary more obscure. The resulting clusters become even more difficult to be classified.
In general, the current model succeed performing classification for part of the arguments but still fails generating well represented representations for some difficult cases and thus leads to suboptimal resutls.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
3 A2. Did you discuss any potential risks of your work? 7

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: abstrct and 1

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: Grammarly
B 3 Did you use or create scientific artifacts? 3
3 B1. Did you cite the creators of artifacts you used? 3
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 3

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 3

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3
C 3 Did you run computational experiments?

heading: Appendix B

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix B

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: , 2020) and leave more exploration for future work