Few-shot Event Detection: An Empirical Study and a Unified View
Year: 2023

Abstract:
Few-shot event detection (ED) has been widely studied, while this brings noticeable discrepancies, e.g., various motivations, tasks, and experimental settings, that hinder the understanding of models for future progress. This paper presents a thorough empirical study, a unified view of ED models, and a better unified baseline. For fair evaluation, we compare 12 representative methods on three datasets, which are roughly grouped into prompt-based and prototype-based models for detailed analysis. Experiments consistently demonstrate that prompt-based methods, including ChatGPT, still significantly trail prototype-based methods in terms of overall performance. To investigate their superior performance, we break down their design elements along several dimensions and build a unified framework on prototype-based methods. Under such unified view, each prototype-method can be viewed a combination of different modules from these design elements. We further combine all advantageous modules and propose a simple yet effective baseline, which outperforms existing methods by a large margin (e.g., 2.7% F1 gains under low-resource setting). 1

heading: 1 Introduction
text: Event Detection (ED) is the task of identifying event triggers and types in texts. For example, given “Cash-strapped Vivendi wants to sell Universal Studios”, it is to classify the word “sell” into a TransferOwnership event. ED is a fundamental step in various tasks such as successive event-centric information extraction (Huang et al., 2022; Ma et al., 2022b; Chen et al., 2022), knowledge systems (Li et al., 2020; Wen et al., 2021), story generation (Li et al., 2022a), etc. However, the annotation of event instances is costly and labor-consuming, which mo-
†Corresponding Author. 1Our code will be publicly available at https://github.com/mayubo2333/fewshot_ED.
tivates the research on improving ED with limited labeled samples, i.e., the few-shot ED task.
Extensive studies have been carried out on fewshot ED. Nevertheless, there are noticeable discrepancies among existing methods from three aspects. (1) Motivation (Figure 1): Some methods focus on model’s generalization ability that learns to classify with only a few samples (Li et al., 2022b). Some other methods improve the transferability, by introducing additional data, that adapts a well-trained model on the preexisting schema to a new schema using a few samples (Lu et al., 2021). There are also methods considering both abilities (Liu et al., 2020; Hsu et al., 2022). (2) Task setting: Even focusing on the same ability, methods might adopt different task settings for training and evaluation. For example, there are at least three settings for transferability: episode learning (EL, Deng et al. 2020; Cong et al. 2021), class-transfer (CT, Hsu et al. 2022) and task-transfer (TT, Lyu et al. 2021; Lu et al. 2022). (3) Experimental Setting: Even focusing on the same task setting, their experiments may vary in different sample sources (e.g., a subset of datasets, annotation guidelines, or external corpus) and sample numbers (shot-number or sampleratio). Table 1 provides a detailed comparison of representative methods.
In this paper, we argue the importance of a unified setting for a better understanding of few-shot ED. First, based on exhaustive background investigation on ED and similar tasks (e.g., NER), we con-
11211
duct an empirical study of twelve SOTA methods under two practical settings: low-resource setting for generalization ability and class-transfer setting for transferability. We roughly classify the existing methods into two groups: prototype-based models to learn event-type representations and proximity measurement for prediction and prompt-based models that convert ED into a familiar task of Pretrained Language Models (PLMs).
The second contribution is a unified view of prototype-based methods to investigate their superior performance. Instead of picking up the best-performing method as in conventional empirical studies, we take one step further. We break down the design elements along several dimensions, e.g., the source of prototypes, the aggregation form of prototypes, etc. From this perspective, five prototype-based methods on which we conduct experiment are instances of distinct modules from these elements. And third, through analyzing each effective design element, we propose a simple yet effective unified baseline that combines all advantageous elements of existing methods. Experiments validate an average 2.7% F1 gains under low-resource setting and the best performance under class-transfer setting. Our analysis also provides many valuable insights for future research.

heading: 2 Preliminary
text: Event detection (ED) is usually formulated as either a span classification task or a sequence labeling task, depending on whether candidate event spans are provided as inputs. We brief the sequence labeling paradigm here because the two paradigms can be easily converted to each other.
Given a dataset D annotated with schema E (the set of event types) and a sentence X = [x1, ..., xN ]
T ∈ D, where xi is the i-th word and N the length of this sentence, ED aims to assign a label yi ∈ (E ∪ {N.A.}) for each xi in X . Here N.A. refers to either none events or events beyond pre-defined types E. We say that word xi triggering an event yi if yi ∈ E.

heading: 2.1 Few-shot ED task settings
text: We categorize few-shot ED settings to four cases: low-resource (LR), class-transfer (CT), episode learning (EL) and task-transfer (TT). Lowresource setting assesses the generalization ability of few-shot ED methods, while the other three settings are for transferability. We adopt LR and CT in our empirical study towards practical scenarios. More details can be found in Appendix A.1. Low-resource setting assumes access to a dataset D = (Dtrain,Ddev,Dtest) annotated with a label
set E, where |Ddev| ≤ |Dtrain| ≪ |Dtest|. It assesses the generalization ability of models by (1) utilizing only few samples during training, and (2) evaluating on the real and rich test dataset. Class-transfer setting assumes access to a source dataset D(S) with a preexisting schema E(S) and a target dataset D(T ) with a new schema E(T ). Note that D(S) and D(T ), E(S) and E(T ) contain disjoint sentences and event types, respectively. D(S) contains abundant samples, while D(T ) is the lowresource setting dataset described above. Models under this setting are expected to be pre-trained on D(S) then further trained and evaluated on D(T ).

heading: 2.2 Category of existing methods
text: We roughly group existing few-shot ED methods into two classes: prompt-based methods and prototype-based methods. More details are introduced in Appendix A.2. Prompt-based methods leverage the rich language knowledge in PLMs by converting downstream tasks to the task with which PLMs are more familiar. Such format conversion narrows the gap between pre-training and downstream tasks and benefits knowledge induction in PLMs with limited annotations. Specifically, few-shot ED can be converted to machine reading comprehension (MRC, Du and Cardie 2020; Liu et al. 2020; Feng et al. 2020), natural language inference (NLI, Lyu et al. 2021), conditional generation (CG, Paolini et al. 2021; Lu et al. 2021, 2022; Hsu et al. 2022), and the cloze task (Li et al., 2022b). We give examples of these prompts in Table 6. Prototype-based methods predict an event type for each word/span mention by measuring its representation proximity to prototypes. Here we define prototypes in a generalized format — it is an embedding that represents some event type. For example, Prototypical Network (ProtoNet, Snell et al. 2017) and its variants (Lai et al., 2020a,b; Deng et al., 2020, 2021; Cong et al., 2021; Lai et al., 2021) construct prototypes via a subset of sample mentions. In addition to event mentions, a line of work leverage related knowledge to learn or enhance prototypes’ representation, including AMR graphs (Huang et al., 2018), event-event relations (Deng et al., 2021), definitions (Shen et al., 2021) and FrameNet (Zhao et al., 2022). Zhang et al. (2022b) recently introduce contrastive learning (Hadsell et al., 2006) in few-shot ED task. Such method also determines the event by measuring the
distances with other samples and aggregates these distances to evaluate an overall distance to each event type. Therefore we view it as a generalized format of prototype-based methods as well.
For comprehensiveness, we also include competitive methods from similar tasks, i.e., Named Entity Recognition and Slot Tagging, which are highly adaptable to ED. Such expansion enriches the categorization and enables us to build a unified view in Section 3. For instance, some methods (Hou et al., 2020; Ma et al., 2022a) leverage label semantics to enhance or directly construct the prototypes. Others (Das et al., 2022) leverage contrastive learning for better prototype representations.

heading: 3 A Prototype-based Unified View
text: Due to the superior performance (Sections 5 and 6), we zoom into prototype-based methods to provide a unified view towards a better understanding. We observe that they share lots of similar components. As shown in Table 2 and Figure 2, we decompose prototype-based methods into 5 design elements: prototype source, transfer function, distance function, aggregation form, and CRF module. This unified view enables us to compare choices in each design element directly. By aggregating the
effective choices, we end with a Unified Baseline. Formally, given an event mention x, prototypebased methods predict the likelihood p(y|x) from logits(y|x) for each y ∈ (E ∪ {N.A.})
p(y|x) = Softmaxy∼(E∪{N.A.})logits(y|x)
The general framework is as follows. Denote the PLM’s output representation of event mention x and data cy in prototype source Cy as hx and hcy respectively, where h ∈ Rm and m is the dimension of PLM’s hidden space. The first step is to convert hx and hcy to appropriate representations via a transfer function f(·). Then the methods maintain either a single or multiple prototypes cy’s for each event type, determined by the adopted aggregation form. Third, the distance between f(hx) and f(hcy) (single prototype) or f(hcy)’s (multiple prototypes) is computed via a distance function d(·, ·) to learn the proximity scores, i.e., logits(y|x). Finally, an optional CRF module is used to adjust logits(y|x) for x in the same sentence to model their label dependencies. For inference, we adopt nearest neighbor classification by assigning the sample with nearest event type in ∪y∈(E∪{N.A.})Cy , i.e.,
ŷx = argmin y∈(E∪{N.A.}) min cy∈Cy d(f(hx), f(hcy))
Next, we detail the five design elements: Prototype source Cy (purple circles in Figure 2, same below) indicates a set about the source of data / information for constructing the prototypes. There are mainly two types of sources: (1) event mentions (purple circle without words): ProtoNet and its variants in Figure 2(b),(c),(d) additionally split a support set Sy from training data as prototype source, while contrastive learning methods in Figure 2(a) view every annotated mention as the source (except the query one). (2) Label semantics (purple ellipses with words): Sometimes, the label name ly is utilized as the source to enhance or directly construct the prototypes. For example, FSLS in Figure 2(e) views the text representation of type names as prototypes, while L-TapNet-CDT in Figure 2(c) utilizes both the above kinds of prototype sources. Transfer function f : Rm → Rn (yellow modules) transfers PLM outputs into the distance space for prototype proximity measurement. Widely used transfer functions include normalization in Figure 2(b), down-projection in Figure 2(c), reparameterization in Figure 2(a), or an identity function. Distance function d : Rn × Rn → R+ (green modules) measures the distance of two transferred representations within the same embedded space. Common distance functions are euclidean distance in Figure 2(d) and negative cosine similarity in Figure 2(b),(c),(e). Aggregation form (blue modules) describes how to compute logits(y|x) based on a single or multiple prototype sources. Aggregation may happen at three levels. (1) feature-level: ProtoNet and its variants in Figure 2(b),(c),(d) aims to construct a single prototype hc̄y for each event type y by merging various features, which ease the calculation logits(y|x) = −d(f(hx), f(hc̄y)). (2) score-level: CONTAINER in Figure 2(a) views each data as a prototype (they have multiple prototypes for each type y) and computes the distance d(f(hx), f(hcy)) for each cy ∈ Cy. These distances are then merged to obtain logits(y|x). (3) loss-level: Such form has multiple parallel branches b for each mention x. Each branch has its own logits(b)(y|x) and is optimized with different loss components during training. Thus it could be viewed as a multi-task learning format. See unified baseline in Figure 2(f).
CRF module (orange modules) adjusts predictions within the same sentence by explicitly considering the label dependencies between sequential inputs. The vanilla CRF (Lafferty et al., 2001) and its variants in Figure 2(a),(b),(c) post additional constraints into few-shot learning.

heading: 4 Experimental setup

heading: 4.1 Few-shot datasets and Evaluation
text: Dataset source. We utilize ACE05 (Doddington et al., 2004), MAVEN (Wang et al., 2020) and ERE (Song et al., 2015) to construct few-shot ED datasets in this empirical study. Detailed statistics about these three datasets are in Appendix B.1. Low-resource setting. We adopt K-shot sampling strategy to construct few-shot datasets for the lowresource setting, i.e., sampling Ktrain and Kdev samples per event type to construct the train and dev sets, respectively.2 We set three (Ktrain,Kdev) in our evaluation: (2, 1), (5, 2) and (10, 2). We follow Yang and Katiyar (2020) taking a greedy sampling algorithm to approximately select K samples for each event type. See Appendix B.2 for details and the statistics of the sampled few-shot datasets. We inherit the original test set as Dtest. Class-transfer setting. The few-shot datasets are curated in two sub-steps: (1) Dividing both event types and sentences in the original dataset into two disjoint parts, named source dataset and target dataset pool, respectively. (2) Sampling few-shot samples from the target dataset pool to construct target dataset. The same sampling algorithm as in low-resource setting is used. Then we have the source dataset and the sampled target dataset. See Appendix B.2 for details and the statistics of the sampled few-shot datasets. Evaluation Metric We use micro-F1 score as the evaluation metric. To reduce the random fluctuation, the reported values of each setting are the averaged score and sample standard deviation, of results w.r.t 10 sampled few-shot datasets.

heading: 4.2 Evaluated methods
text: We evaluate 12 representative methods, including vanilla fine-tuning, in-context learning, 5 prompt-
2Recent systematic research on few-shot NLP tasks (Perez et al., 2021) is of opposition to introducing an additional dev set for few-shot learning. We agree with their opinion but choose to keep a very small dev set mainly for feasibility consideration. Given the number of experiments in our empirical study, it is infeasible to conduct cross-validation on every single train set for hyperparameter search.
based and 5 prototype-based methods. These methods are detailed in Appendix B.3. Fine-tuning To validate the effectiveness of fewshot methods, we fine-tune a supervised classifier for comparison as a trivial baseline. In-context learning To validate few-shot ED tasks still not well-solved in the era of Large Language Models (LLMs), we design such baseline instructing LLMs to detect event triggers by the means of in-context learning (ICL). Prompt-based (1) EEQA (QA-based, Du and Cardie 2020), (2) EETE (NLI-based, Lyu et al. 2021), (3) PTE (cloze task, Schick and Schütze 2021), (4) UIE (generation, Lu et al. 2022) and (5) DEGREE (generation, Hsu et al. 2022). Prototype-based (1) ProtoNet (Snell et al., 2017), (2) L-TapNet-CDT (Hou et al., 2020), (3) PACRF (Cong et al., 2021), (4) CONTAINER (Das et al., 2022) and (5) FSLS (Ma et al., 2022a). See Table 2 and Figure 2 for more details.

heading: 4.3 Implementation details
text: We unify PLMs in each method as much as possible for a fair comparison in our empirical study. Specifically, we use RoBERTa-base (Liu et al., 2019) for all prototype-based methods and three non-generation prompt-based methods. However, we keep the method’s original PLM for two prompt-based methods with generation prompt, UIE (T5-base, Raffel et al. 2020) and DEGREE (BART-large, Lewis et al. 2020). We observe their performance collapses with smaller PLMs. Regarding ICL method, we use ChatGPT (gpt-3.5-turbo-0301) as the language model. See more details in Appendix B.4.

heading: 5 Results: Low-resource Learning

heading: 5.1 Overall comparison
text: We first overview the results of the 12 methods under the low-resource setting in Table 3. Fine-tuning. Despite its simpleness, fine-tuning achieves acceptable performance. In particular, it is even comparable to the strongest existing methods on MAVEN dataset, only being 1.1% and 0.5% less under 5-shot and 10-shot settings. One possible reason that fine-tuning is good on MAVEN is that MAVEN has 168 event types, much larger than others. When the absolute number of samples is relatively large, PLMs might capture implicit interactions among different event types, even though the samples per event type are limited. When the
sample number is scarce, however, fine-tuning is much poorer than existing competitive methods (see ACE05). Thus, we validate the necessity and progress of existing few-shot methods. In-context learning. We find the performance of ICL-based methods lags far behind that of tuningrequired methods, though the backbone of ICL approach (ChatGPT) is much larger than other PLMs (<1B). A series of recent work (Ma et al., 2023; Gao et al., 2023; Zhan et al., 2023) observe the similar results as ours 3. Thus we validate few-shot ED tasks could not be solved smoothly by cutting-edge LLMs and deserves further exploration. Prompt-based methods. Prompt-based methods deliver much poorer results than expected, even compared to fine-tuning, especially when the sample number is extremely scarce. It shows designing effective prompts for ED tasks with very limited annotations is still challenging or even impossible. We speculate it is due to the natural gap between ED tasks and pre-training tasks in PLMs.
Among prompt-based methods, PTE and DEGREE achieve relatively robust performance under all settings. DEGREE is advantageous when the sample size is small, but it cannot well handle a dataset with many event types like MAVEN. When sample sizes are relatively large, EEQA shows competitive performance as well.
3We refer readers to Ma et al. (2023) for a more detailed discussion on why ICL approaches stumble across few-shot ED tasks.

heading: 5.2 Prototype-based methods
text: Since prototype-based methods have overall better results, we zoom into the design elements to search for effective choices based on the unified view. Transfer function, Distance function, and CRF. We compare combinations of transfer and distance functions and four variants of CRF modules in Appendices C.1 and C.2. We make two findings: (1) A scaled coefficient in the distance function achieves better performance with the normalization transfer function. (2) There is no significant difference between models with or without CRF modules. Based on these findings, we observe a significant improvement in five existing methods by simply substituting their d and f for more appropriate choices, see Figure 3 and Appendix C.1. We would use these new transfer and distance functions in further analysis and discussion.
Prototype Source. We explore whether label semantic and event mentions are complementary pro-
totype sources, i.e., whether utilizing both achieves better performance than either one. We choose ProtoNet and FSLS as base models which contain only a single kind of prototype source (mentions or labels). Then we combine the two models using three aggregating forms mentioned in Section 3 and show their results in Figure 4. Observe that: (1) leveraging label semantics and mentions as prototype sources simultaneously improve the performance under almost all settings, and (2) merging the two kinds of sources at loss level is the best choice among three aggregation alternatives.
Contrastive or Prototypical Learning. Next, we investigate the effectiveness of contrastive learning (CL, see CONTAINER) and prototypical learning (PL, see ProtoNet and its variants) for event mentions. We compare three label-enhanced (since we have validated the benefits of label semantics) methods aggregating event mentions with different approaches. (1) Ll-ProtoNet: the strongest method utilizing PL in last part. (2) Ll-CONTAINER: the method utilizing in-batch CL as CONTAINER does. (3) Ll-MoCo: the method utilizing CL with MoCo setting (He et al., 2020). The in-batch CL and MoCo CL are detailed in Appendix C.4.
Figure 5 suggests CL-based methods outperform Ll-ProtoNet. There are two possible reasons: (1) CL has higher sample efficiency since every two samples interact during training. PL, however, further splits samples into support and query set during training; samples within the same set are not interacted with each other. (2) CL adopts score-level aggregation while PL adopts feature-level aggregation. We find the former also slightly outperforms the latter in Figure 4. We also observe that MoCo CL usually has a better performance than in-batch CL when there exists complicated event types (see MAVEN), or when the sample number is relatively large (see ACE 10-shot). We provide a more detailed explanation in Appendix C.4.

heading: 5.3 The unified baseline
text: Here is a summary of the findings: (1) Scaled euclidean or cosine similarity as distance measure with normalized transfer benefits existing methods. (2) CRF modules show no improvement in performance. (3) Label semantic and event mentions are complementary prototype sources, and aggregating them at loss-level is the best choice. (4) As for the branch of event mentions, CL is more advantageous than PL for few-shot ED tasks. (5) MoCo CL performs better when there are a good number of sentences, otherwise in-batch CL is better.
Based on these findings, we develop a simple but effective unified baseline as follows. We utilize both label semantic and event mentions as prototype sources and aggregate two types of sources at loss-level. Specifically, we assign two branches with their own losses for label semantic and event mentions respectively. Both two branches adopt scaled cosine similarity dτ (u, v) = −u T v τ as distance measure and normalization f(h) = h/∥h∥2 as transfer function. We do not add CRF modules.
For label semantic branch, we follow FSLS and set the embeddings of event name as prototypes. Here hx and hey represent the PLM representation of event mention x and label name ey, respectively.
ey = Event_name(y)
logits(l)(y|x) = −dτ (f(hx), f(hey)) For event mention branch, we adopt CL which aggregates prototype sources (event mentions) at score-level. If the total sentence number in train set is smaller than 128, we take in-batch CL (CONTAINER) strategy as below:
logits(m)(y|x) = ∑
x′∈Sy(x)
−d(f(hx), f(hx′)) |Sy(x)|
Sy(x) = {x′|(x′, y′) ∈ D, y′ = y, x′ ̸= x} is the set of all other mentions with the same label.
If the total sentence number in train set is larger than 128, we instead take MoCo CL maintaining a queue for Sy(x) and a momentum encoder.
We then calculate the losses of these two branches and merge them for joint optimization:
p(l/m)(y|x) = Softmaxy[logits(l/m)(y|x)] L(l/m)(y|x) = − ∑
(x,y)
ylog(p(l/m)(y|x))
L = L(l) + L(m)
The diagram of the unified baseline is illustrated in Figure 2(f) and its performance is shown in Table 3. Clearly, unified baseline outperforms all existing methods significantly, 2.7% F1 gains on average, under all low-resource settings.

heading: 6 Results: Class-transfer Learning
text: In this section, we evaluate existing methods and the unified baseline under class-transfer setting. Here we do not consider in-context learning because previous expetiments show it still lags far from both prompt- and prototype-based methods.

heading: 6.1 Prompt-based methods
text: We first focus on 4 existing prompt-based methods and explore whether they could smoothly transfer event knowledge from a preexisting (source) schema to a new (target) schema. We show results in Figure 6 and Appendix D.1. The findings are summarized as follows. (1) The transfer of knowledge from source event types to target event types facilitates the model prediction under most scenarios. It verifies that an appropriate prompt usually benefits inducing the knowledge learned in PLMs. (2) However, such improvement gradually fades with the increase of sample number from either source or target schema. For example, the 5- shot v.s 10-shot performance for PTE and UIE are highly comparable. We speculate these prompts act more like a catalyst: they mainly teach model how to induce knowledge from PLMs themselves rather than learn new knowledge from samples. Thus the performance is at a standstill once the sample number exceeds some threshold. (3) Overall, the performance of prompt-based methods remains inferior to prototype-based methods in class-transfer setting (see black lines in Figure 6). Since similar results are observed in low-resource settings as well, we conclude that prototype-based methods are better few-shot ED task solver.

heading: 6.2 Prototype-based methods
text: We further explore the transfer ability of existing prototype-based methods and unified baseline4. Thanks to the unified view, we conduct a more thorough experiment that enumerates all possible combinations of models used in the source and target domain, to assess if the generalization ability affects transferability. That is, the parameters in PLMs will be shared from source to target model. We show results in Figure 7 and Appendix D.2.

heading: 1. Is transfer learning effective for prototype-based
text: methods? It depends on the dataset (compare the first row with other rows in each column). For ACE05 and MAVEN datasets, the overall answer is yes. Contrary to our expectation, transfer learning affects most target models on ERE dataset negatively, especially for 2- and 5-shot settings.

heading: 2. Do prototype-based methods perform better than
text: simple fine-tuning? It depends on whether finetuning the source or target model. When fine-tuning a source model (row 2), it sometimes achieves comparable even better performance than the prototypebased methods (last 4 rows). When fine-tuning a target model (column 1), however, the performance drops significantly. Thus, we speculate that powerful prototype-based methods are more necessary in target domain than source domain.
4Transfer and distance functions in all methods are substituted to appropriate ones and CRF modules are removed.
3. Is the choice of prototype-based methods important? Yes. When we select inappropriate prototypebased methods, they could achieve worse performance than simple fine-tuning and sometimes even worse than models without class transfer. For example, CONTAINER and L-TapNet are inappropriate source model for ACE05 dataset.

heading: 4. Do the same source and target models benefit the
text: event-related knowledge transfer? No. The figures show the best model combinations often deviate from the diagonals. It indicates that different source and target models sometimes achieve better results.

heading: 5. Is there a source-target combination performing
text: well on all settings? Strictly speaking, the answer is No. Nevertheless, we find that adopting FSLS as the source model and our unified baseline as the target model is more likely to achieve competitive (best or second best) performance among all alternatives. It indicates that (1) the quality of different combinations show kinds of tendency though no consistent conclusion could be drawn. (2) a model with moderate inductive bias (like FSLS) might be better for the source dataset with abundant samples. Then our unified baseline could play a role during the target stage with limited samples.

heading: 7 Conclusion
text: We have conducted a comprehensive empirical study comparing 12 representative methods under unified low-resource and class-transfer settings. For systematic analysis, we proposed a unified framework of promising prototype-based methods. Based on it, we presented a simple and effective baseline that outperforms all existing methods significantly under low-resource setting, and is an ideal choice as the target model under class-transfer setting. In the future, we aim to explore how to leverage unlabeled corpus for fewshot ED tasks, such as data augmentation, weaklysupervised learning, and self-training.

heading: Acknowlegement
text: This study is supported under the RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative, the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant, as well as cash and in-kind contribution from the industry partner(s).

heading: Limitations
text: We compare 12 representative methods, present a unified view on existing prototype-based methods, and propose a competitive unified baseline by combining the advantageous modules of these methods. We test all methods, including the unified baseline, on three commonly-used English datasets using various experimental settings and achieve consistent results. However we acknowledge the potential disproportionality of our experiments in terms of language, domain, schema type and data scarcity extent. Therefore, for future work, we aim to conduct our empirical studies on more diverse event-detection (ED) datasets.
We are fortunate to witness the rapid development of Large Language Models (LLMs Brown et al. 2020b; Ouyang et al. 2022; Chung et al. 2022) in recent times. In our work, we set incontext learning as a baseline and evaluate the performance of LLMs on few-shot ED tasks. We find current LLMs still face challenges in dealing with Information Extraction (IE) tasks that require structured outputs (Qin et al., 2023; Josifoski et al., 2023). However, we acknowledge the ICL approach adopted here is relatively simple. We do not work hard to find the optimal prompt format, demonstration selection strategy, etc., to reach the upper bounds of LLMs’ performance. We view how to leverage the power of LLMs on ED tasks as an open problem and leave it for future work.
In this work, we focus more on the model aspect of few-shot ED tasks rather than data aspect. In other words, we assume having and only having access to a small set of labeled instances. In the future, we plan to explore how to utilize annotation guidelines, unlabeled corpus and external structured knowledge to improve few-shot ED tasks.

heading: A Related Work

heading: A.1 Taxonomy of task settings
text: Various solutions have been proposed to improve the generalization and transfer abilities of few-shot ED methods. There exists a bottleneck: the models adopt very different tasks and experimental settings. We categorize existing task settings to four cases as
shown in Figure 8: low-resource (LR), class transfer (CL), episode learning (EL), and task transfer (TT) settings. LR is used to evaluate the generalization ability, learning rapidly with only few examples in target domain. The other settings (CL, EL, and TT) evaluate the transfer ability, adapting a model trained with a preexisting schema with abundant samples, to a new (target) schema with only few examples. Based on the pros and cons presented here, we adopt the low-resource and class transfer settings in our empirical study. 1. Low-resource setting assesses the generalization ability of models by (1) utilizing only few samples during training, (2) evaluating on the real and rich test dataset. Conventionally, the few-shot |Dtrain| and |Ddev| are downsampled from a full dataset by two main strategies: (1) K-shot sampling which picks out K samples for each event type, or (2) ratio sampling which picks out partial sentences with a fixed ratio. We view both sampling strategies as reasonable and adopt K-shot sampling in this work.
The surging development of PLMs makes training with only few (or even zero) examples possible, and achieves acceptable performance (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020a). Accordingly, a series of prompt-based methods (Du and Cardie, 2020; Liu et al., 2020; Feng et al., 2020; Paolini et al., 2021; Lu et al., 2021; Deng et al., 2021; Hsu et al., 2022; Li et al., 2022b) adopt such setting to train and evaluate their models. 2. Class transfer setting assesses the transferability of a model by providing abundant samples in the source (preexisting) schema and scarce samples in target (new) schema. It trains a classifier in source schema and then transfers such classifier to the target schema with only few examples.
Such setting has been applied since an early stage (Bronstein et al., 2015; Peng et al., 2016; Zhang et al., 2021), and is often used together with low-resource setting to additionally evaluate transferability of the models (Paolini et al., 2021; Lu et al., 2021; Hsu et al., 2022). 3. Episode learning setting is a classical few-shot setting. It has two phases, meta-training and metatesting, each of which consists of multiple episodes. Each episode is a few-shot problem with its own train (support) and test (query) sets and event-type classes. Since the sets in each episode are sampled uniformly having K different classes and each class having N instances, episode learning is also known
as N -way-K-shot classification. Many existing few-shot ED methods adopt this setting (Lai et al., 2020a,b; Deng et al., 2020; Cong et al., 2021; Lai et al., 2021; Chen et al., 2021; Zhang et al., 2022b; Zhao et al., 2022). However, we argue that episode learning assumes an unrealistic scenario. First, during the meta-training stage, a large number of episodes is needed, for example, 20,000 in Cong et al. (2021). Though the label sets of meta-training and meta-testing stages are disjoint, class transfer setting is more reasonable when there are many samples in another schema. Second, tasks with episode learning are evaluated by the performance on samples of the test (query) set in the meta-testing phase. The test sets are sampled uniformly, leading to a significant discrepancy with the true data distribution in many NLP tasks. The absence of sentences without any events further leads to distribution distortion. Further, each episode contains samples with only K different classes, where K is usually much smaller than the event types in the target schema. All these factors may lead to an overestimation on the ability of fewshot learning systems. For above reasons, we do not consider this setting in our experiments. 4. Task transfer setting is very similar to class transfer. The main difference is that it relaxes the constraint in source phase, from the same task with different schema to different tasks.5 The development of this setting also heavily relies on the success of PLMs. Liu et al. (2020), Feng et al. (2020) and Lyu et al. (2021) leverage model pre-trained with SQuAD 2.0 (QA dataset, Rajpurkar et al. 2018) or MNLI (NLI dataset, Williams et al. 2018) to improve the performance of zero-/few-shot ED models. Paolini et al. (2021) and Lu et al. (2022)
5Generally speaking, all methods using PLMs belong to this setting in which the source task is exactly the pre-training task of PLMs, masked- or next-word prediction. In this work, we limit the discussion of task transfer to which the source task is another downstream task rather than the general pre-training task in PLMs.
recently construct unified generation frameworks on multiple IE tasks. Their experiments also reveal that pre-training on these tasks benefits few-shot ED. Though task transfer setting is reasonable and promising, we do not include this setting out of its extreme diversity and complexity. That is, there are (1) too many candidate tasks as pre-training tasks, and (2) too many optional datasets for each pre-training task. Thus it is almost infeasible to conduct a comprehensive empirical study on task transfer setting.

heading: A.2 Taxonomy of methods
text: We categorize existing methods to two main classes, prompt-based methods and prototypebased methods, and list them in Table 1. Here we give a detailed introduction of existing methods. Note that in our empirical study, we also include some methods which are originally developed for similar few-shot tasks but can be easily adapted to ED. We leave a special subsection for them. Few-shot ED methods. Due to the prohibitively cost for labeling amounts of event mentions, fewshot ED is a long-standing topic in event-related research community. The proposed solutions are mainly in two branches. The first branch, prototypebased 6 methods, is a classical approach on fewshot learning. It defines a single or multiple prototypes for each event type representing the label-wise properties. It then learns the embedding representation of each sample via shortening the distance from its corresponding prototypes given a distance/similarity metric. Bronstein et al. (2015) and Peng et al. (2016) leverage the seed instances in annotation guideline and mine the lexical/semantic features of trigger words to obtain the prototypes. Zhang et al. (2021) inherit such paradigm and define prototypes as the average contextualized embeddings of the related trig-
6Different from other sections, here we adopt a chronological order and firstly introduce prototype-based methods.
ger words weakly labeled in external corpus. With the help AMR Parsing, Huang et al. (2018) additionally consider the graph structures of preexisting schema as prototypes, and encode AMR graph representation of each event mention as representations. Deng et al. (2020) introduces Dynamic Memory Network (DMN), while Lai et al. (2020a) and Lai et al. (2021) introduce two different auxiliary losses improving intra-/inter-consistency of different episodes to facilitate their prototype representations. Deng et al. (2021) further consider the relations among events to constrain the prototypes and benefit both rare and new events. Cong et al. (2021) amortize CRF module by modeling the transition probabilities of different event types with their prototypes. Chen et al. (2021) leverage causal inference and intervene on context via backdoor adjustment during training to reduce overfitting of trigger words for more robust prototypes. Recently, Zhang et al. (2022a) and Zhang et al. (2022b) introduce contrastive learning into few-shot ED task and their proposed methods actually could be viewed as generalized prototype-based methods with multiple prototypes rather than one.
The other branch, prompting methods, is made possible with the surge of development in PLMs. Given a specific task, prompting methods map the task format to a new format with which the PLMs are more familiar, such as masked word prediction (Schick and Schütze, 2021) and sequence generation (Raffel et al., 2020; Brown et al., 2020a). Such format conversion narrows down the gaps between pre-training tasks and downstream tasks, which is beneficial for inducing learned knowledge from PLMs with limited annotations. As for event detection (and many other IE tasks), however, it is not trivial to design a smooth format conversion. One simple idea is leveraging one single template to prompt both event types and their triggers simultaneously (Paolini et al., 2021; Lu et al., 2021). However, such prompting methods show performance far from satisfactory, especially when they are not enhanced by two-stage pre-training and redundant hinting prefix (Lu et al., 2022). Another natural idea is enumerating all legal spans and querying the PLMs whether each span belongs to any class, or vice versa (Hsu et al., 2022). A major limitation here is the prohibitively time complexity, particularly when there are many event types. Combining the merits of prompting methods and conventional fine-tuning methods is another solu-
tion. Du and Cardie (2020) and Liu et al. (2020) use QA/MRC format to prompt the location of trigger words, while still predicting their event types via an additional linear head. Lyu et al. (2021) first segment one sentence into several clauses and view the predicates of clauses as trigger candidates. Then they leverage NLI format to query the event types of these candidates. Recently, Li et al. (2022b) propose a strategy combining Pattern-Exploiting Training (PET, Schick and Schütze 2021) and CRF module. Initially, they conduct sentence-level event detection determining whether one sentence contains any event types or not. For each identified event type, they further use a linear chain CRF to locate the trigger word. Few-shot NER/ST methods. There are several models which are originally designed for similar tasks like Named Entity Recognition (NER) and Slot Tagging (ST) but could be applied to ED task.
Similar to ED methods, one classical paradigm in NER is utilizing ProtoNet (Snell et al., 2017) and its variants to learn one representative prototypes for each class type with only few examples. Fritzler et al. (2019) firstly combine ProtoNet and CRF module to solve NER tasks. Hou et al. (2020) propose L-TapNet-CDT, which enhances TapNet (Yoon et al., 2019), a variant of ProtoNet, with textual label names and achieves great performance among several ST tasks. Both methods construct prototypes by computing the average embeddings of several sampled examples (support set). Yang and Katiyar (2020) propose a simpler algorithm, leveraging supervised classifier learned in preexisting schema as feature extractor and adopting nearest neighbors classification during inference, and show competitive performance in class transfer setting for few-shot NER task. Das et al. (2022) introduce contrastive learning into few-shot NER task. Ma et al. (2022a) recently developed a simple but effective method on few-shot NER by constructing prototypes only with their labels.

heading: B Datasets and Models
text: We curate few-shot datasets used in this emprical study from three full and commonly-used datasets: ACE05 (Doddington et al., 2004), MAVEN (Wang et al., 2020) and ERE (Song et al., 2015).

heading: B.1 Full dataset
text: ACE05 is a joint information extraction dataset, with annotations of entities, relations, and events.
We only use its event annotation for ED task. It contains 599 English documents and 33 event types in total. We split documents in ACE05 following previous work (Li et al., 2013) to construct train and test dataset respectively. MAVEN is a newlybuilt large-scale ED dataset with 4480 documents and 168 event types. We use the official split for MAVEN dataset. ERE is another joint information extraction dataset having a similar scale as ACE05 (458 documents, 38 event types). We follow the preprocessing procedure in Lin et al. (2020). Table 4 reports detailed statistics of the three datasets.
ED could be viewed as either a span classification or a sequence labeling task. In our work, we adopt span classification paradigm for MAVEN dataset since it provides official spans for candidate triggers (including negative samples). For the other two datasets, we follow sequence labeling paradigm to predict the event type word by word.

heading: B.2 Dataset construction
text: This section introduces how we construct few-shot datasets from the three full ED datasets. Low-resource setting. We downsample sentences from original full training dataset to construct Dtrain and Ddev, and inherit the original test set as the unified Dtest. For Dtrain and Ddev, we adopt K-shot sampling strategy that each event type has (at least) K samples. Since our sampling is at sentence-level and each sentence could have multiple events, the sampling is NP-complete7 and unlikely to find a practical solution satisfying exactly K samples for each event type. Therefore, we follow Yang and Katiyar (2020) and Ma et al. (2022a) and adopt a greedy sampling algorithm to select sentences, as shown in Alg. 1. Note that the actual sample number of each event type can be larger than K under this sampling strategy. The statistics of the curated datasets are listed in Table 5 (top). Class-Transfer setting This setting has a more
7The Subset Sum Problem, a classical NP-complete problem, can be reduced to this sampling problem.
Algorithm 1 Greedy Sampling Require: shot number K, original full dataset D = {(X,Y)} tagged with label set E
1: Sort E based on their frequencies in {Y} as an ascending order 2: S ← ϕ, Counter← dict() 3: for y ∈ E do 4: Counter(y)← 0 5: end for 6: for y ∈ E do 7: while Counter(y) < K do 8: Sample (X,Y) ∈ D s.t.∃j, yj = y 9: D ← D\(X,Y) 10: Update Counter (not only y but all event types in Y) 11: end while 12: end for 13: for s ∈ S do 14: S ← S\s and update Counter 15: if ∃y ∈ E, s.t. Counter(y) < K then 16: S ← S⋃ s 17: end if 18: end for 19: return S
complicated curation process, and roughly consists of two sub-steps: (1) Dividing both event types and sentences in the original dataset into two disjoint parts named source dataset and target dataset pool. (2) Using the entire source dataset, and selecting few-shot samples from the target pool to construct target set.
For step (1), we follow Huang et al. (2018) and Chen et al. (2021) to pick out the most frequent 10, 120, and 10 event types from ACE05, MAVEN and ERE dataset respectively, as E(S). The remaining types are E(T ). Then we take sentences containing any annotations in E(T ) to D(T )full for enriching the sampling pool of target dataset as much as possible,
D (T ) full = {(X, R(Y ;E(S)))|(X,Y ) ∈ D,∃yj ∈ E(T )}
where R(Y ;E(S) represents the relabeling operation that substituting any yj ∈ E(S)) to N.A. to avoid information leakage. The remaining sentences are collected as D(S).
D(S) = {(X, R(Y ;E(T )))|(X,Y ) /∈ D(T )full}
For step (2), we adopt the same strategy as lowresource setting to sample K-shot D(T )train and D (T ) dev from target sampling pool D(T )full. Statistics of curated datasets are summarized in Table 5 (bottom).

heading: B.3 Existing methods
text: We conduct our empirical study on twelve representative existing methods. Besides vanilla fine-tuning and in-context learning, five of them are promptbased and the other five are prototype-based. 1. Prompt-based methods leverage the rich knowledge in PLMs by converting specific downstream tasks to the formats that PLMs are more familiar with. We give examples about prompt format of the five prompt-based methods in Table 6. EEQA/EERC (Du and Cardie, 2020; Liu et al., 2020): a QA/MRC-based method which first extracts the trigger word with a natural language query then classifies its type with an additional classifier. EDTE (Lyu et al., 2021): a NLI-based method which enumerates all event types and judges whether a clause is entailed by any event. The clause is obtained by SRL processing and the trigger candidate is the predicate of each clause. PTE (Schick and Schütze, 2021): a cloze-style prompt method which enumerates each word in the
sentence and predicts whether it is the trigger of any event type. UIE (Lu et al., 2022): a generation based method that takes in a sentence and outputs a filled universal template, indicating the trigger words and their event types in the sentence. DEGREE (Hsu et al., 2022): also adopts a generation paradigm but it enumerates all event types by designing type-specific template, and outputs related triggers (if have). 2. Prototype-based methods predict an event type for each word or span by measuring the representation proximity between the samples and the prototypes for each event type. Prototypical Network (Snell et al., 2017): a classical prototype-based method originally developed for episode learning. Huang et al. (2021) adapt it to low-resource setting via further splitting the training set into support set Sy and query set Qy. The prototype c̄y of each event type is constructed by averaged PLM representations of samples in Sy.
hc̄y = 1 Sy ∑
s∈Sy hs
For samples x in Qy during training, or in the test set during inference, logits(y|x) is defined as the negative euclidean distance between h(x) and c̄y.
logits(y|x) = −||hx − hc̄y ||2
L-TapNet-CDT (Hou et al., 2020): a ProtoNetvariant method with three main improvements: (1) it introduces TapNet, a variant of ProtoNet. TapNet’s main difference from ProtoNet lies in a projection spaceM analytically constructed. The distance is computed in the subspace spanned byM.
logits(y|x) = −||M(hx − hc̄y)||2
(2) the basis in column space of M⊥ is aligned with label semantic, thusM(E) is label-enhanced. (3) a collapsed dependency transfer (CDT) module is used solely during inference stage to scale the event-type score.
logits(y|x)← logits(y|x) + TRANS(y)
PA-CRF (Cong et al., 2021): a ProtoNet-variant method with a CRF module as well. Different from CDT, however, the transition scores are approximated between event types based on the their prototypes and learned during training.
FSLS (Ma et al., 2022a): a recently proposed fewshot NER method that generalizes well to ED task. The prototype of each event type is not constructed from support set Sy but from the label semantic, i.e. the PLM representation of the label name.
ey = Event_name(y)
logits(y|x) = hTxhey CONTAINER (Das et al., 2022): a contrastive learning approach. We view it as a generalized Prototype-based method since both of their motivations are to pull together the representations of samples with same event types. Different from ProtoNet, there is no explicit division between support set and query set during training process. Instead each sample acts as query and other samples as support samples. For example, given sample x with event type e, its special supported set can be viewed as:
Sy(x) = {x′|(x′, y′) ∈ D, y′ = y, x′ ̸= x}
Then its score related to e is calculated as the average distance with samples in Sy(x).
logits(y|x) = ∑
x′∈Sy(x)
−d(f(hx), f(hx′)) |Sy(x)|
B.4 Implementation Details For all methods, we initialize their pre-trained weights and further train them using Huggingface library.8 Each experiment is run on single NVIDIAV100 GPU, and the final reported performance
8https://huggingface.co/
for each setting (e.g., ACE 2-shot) is the averaged result w.r.t ten distinct few-shot training datasets which are sampled with different random seeds. We further detail the implementation of all methods. 1. In-context learning
Instruction User Assume you are an event-instance annotator.
Given a sentence, you need to (1) identify the word or phrase triggering the event in the sentence, and (2) classify its event type. The possible event types are listed as below: Transport, Attack, Injure, . . . ChatGPT Yes, I understand. User Please note that your annotation results must fol-
low such format: “Answer: ([Event_1], [Trigger_1]) <SEP> ([Event_2], [Trigger_2]) <SEP> . . . ”. If you do not find any event in this sentence, just output “Answer: No events found.”
ChatGPT No problem. Let’s start! ———————————————————– Demonstration User Sentence: He ’s retired with the US Army. ChatGPT Answer: (End-Position, retired) User Sentence: I wouldn’t be surprised if this Arroyo
character (who was killed by police as he tried to flee) had a serious criminal past but has been allowed to walk free due to our lax judicial process. ChatGPT Answer: (Die, killed) <SEP> (Acquit, walk free) User Sentence: Granting a local sheik power to set up an administrative committee representing the groups in the region. ChatGPT Answer: No events found. ———————————————————– Question User Hariri submitted his resignation during a 10-
minute meeting with the head of state at the Baabda presidential palace, outside the capital.
We use ChatGPT (gpt-3.5-turbo-0301) provided by OpenAI APIs 9 for in-context learning.
9https://platform.openai.com/docs/api-reference
The prompt simulates and records the chatting history between the user and the model. We show one example as above. The prompt consists of three parts: (1) the instruction telling LLMs the task purposes and input-output formats, (2) the demonstration showcasing several input-output pairs to teach LLMs the task and (3) the input of test instance. We feed the prompt into LLMs and expect them to generate extracted answers. Specifically, we set the temperature as 0 and maximum output token as 128. We make all samples in few-shot train set as demonstration samples if their total length is smaller than the maximum input token length (4096). Otherwise we retrieve similar demonstration samples for each test instance to fill up the input prompt. The similarity between two instances are measured from their embeddings (Gao et al., 2021). For MAVEN dataset, we further sample a test subset, with 1000 instances, from the original one for our evaluation. 2. Prompt-based methods We keep all other hyperparameters the same as in their original papers, except learning rates and epochs. We grid-search best learning rates in [1e-5, 2e-5, 5e-5, 1e-4] for each setting. As for epochs, we find the range of appropriate epochsis highly affected by the prompt format. Therefore we search for epochs method by method without a unified range. EEQA (Du and Cardie, 2020): We use their original code10 and train it on our datasets. EDTE (Lyu et al., 2021): We use their original code11 and train it on our datasets. PTE (Schick and Schütze, 2021): We implement this method on OpenPrompt (Ding et al., 2022). UIE (Lu et al., 2022): We use their original code12 and train it on our datasets. DEGREE (Hsu et al., 2022): We reproduce this method based on their original code13 and train it on our datasets. And we drop event keywords not occurring in few-shot training dataset from prompt to avoid information leakage. 3. Prototype-base methods We build a codebase based on the unified view. We then implement these methods directly on the unified framework, by having different choices for each design element. To ensure the correctness of our codebase, we also compare between results obtained from our implementation and original code for each method,
10https://github.com/xinyadu/eeqa 11https://github.com/veronica320/Zeroshot-Event-
Extraction 12https://github.com/universal-ie/UIE 13https://github.com/PlusLabNLP/DEGREE
and find they achieving similar performance on few-shot ED datasets.
For all methods (including unified baseline), we train them with the AdamW (Loshchilov and Hutter, 2017) optimizer with linear scheduler and 0.1 warmup step. We set weight-decay coefficient as 1e-5 and maximum gradient norms as 1.0. We add a 128-long window centering on the trigger words and only encode the words within the window; in other words, the maximum encoding sequence length is 128. The batch size is set as 128, and training steps as 200 if the transfer function is scaled (see Section 5.2) otherwise 500. We grid-search best learning rates in [1e-5, 2e-5, 5e-5, 1e-4] for each setting. For ProtoNet and its variants, we further split the sentences into support set and query set. The number in support set KS and query set KQ are (1, 1) for 2-shot settings, (2, 3) for 5-shot settings. The split strategy is (2, 8) for 10-shot dataset constructed from MAVEN and (5, 5) for others. For methods adopting MoCo-CL setting (also see Section 5.2), we maintain a queue storing sample representations with length 2048 for ACE/ERE 2-shot settings and 8192 for others. For methods adopting CRF, we follow default hyperparameters about CRF in their original papers. For methods adopting scaled transfer functions, we grid search the scaled coefficient τ in [0.1, 0.2, 0.3].

heading: C Low-resource Setting-Extended
text: C.1 Transfer function and Distance function
We consider several combinations about distance and transfer functions listed in Table 7. We choose cosine similarity (S), negative euclidean distance (EU) and their scaled version (SS/SEU) as distance functions. And we pick out identify (I), down-projection (D) and their normalization version (N/DN) as transfer function. We additionally consider the KL-reparameterization combination (KL-R) used in CONTAINER.
We conduct experiments with four existing prototype-based methods14 by only changing their transfer and distance functions. We illustrate their results on ACE dataset in Figure 9. (1) From comparison about performance in ProtoNet and TapNet, we find TapNet, i.e., the down-projection transfer, shows no significant improvement on few-shot ED tasks. (2) A scaled coefficient in distance function
14We degrade L-TapNet-CDT to TapNet, and do not include PA-CRF here, because CRF and label-enhancement are not the factors considered in this subsection.
achieves strong performance with normalization transfer function, while the performance collapses (failing to converge) without normalization. (3) For ProtoNet and TapNet, scaled euclidean distance (SEU) is a better choice for distance function, while other methods prefer scaled cosine similarity (SS). Based on the findings above, we substitute d and f to the most appropriate for all existing methods and observe a significant improvement on all three datasets, as shown in Table 8.

heading: C.2 CRF module
text: We explore whether CRF improves the performance of few-shot ED task. Based on Ll-MoCo model we developed in Section 5.2, we conduct experiment with three different CRF variants, CDT (CRF inference Hou et al. 2020), vanilla CRF (Lafferty et al., 2001) and PA-CRF (Cong et al., 2021), on ACE05 and MAVEN datasets. Their results are in Figure 10. It shows different CRF variants achieve similar result compared with model without CRF, while a trained CRF (and its prototypeenhanced variant) slightly benefits multiple-word triggers when the sample is extremely scarce (see ACE05 2-shot). These results are inconsistent with other similar sequence labeling tasks such as NER or slot tagging, in which CRF usually significantly improves model performance. We speculate it is due to that the pattern of triggers in ED task is relatively simple. To validate such assumption, we count all triggers in ACE05 and MAVEN datasets. We find that above 96% of triggers are single words, and most of the remaining triggers are verb phrases Thus the explicit modeling of transfer dependency among different event types is somewhat not very meaningful under few-shot ED task. Hence, we drop CRF module in the unified baseline.

heading: C.3 Prototype source
text: We discuss the benefit of combining two kinds of prototype sources in Section 5.2, i.e., label semantic and event mentions, and show some results in Figure 4. Here we list full results on all three datasets in Table 9. The results further validate our claims: (1) leveraging both label semantics and mentions as prototype sources improve performance under almost all settings. (2) Merging the two kinds of sources at the loss-level is the best choice among the three aggregation alternatives.

heading: C.4 Contrastive Learning
text: Contrastive Learning (CL Hadsell et al. 2006) is initially developed for self-supervised representation
learning and is recently used to facilitate supervised learning as well. It pulls samples with same labels together while pushes samples with distinct labels apart in their embedding space. We view CL as a generalized format of prototype-based methods and include it to the unified view. Under such view, every sample is a prototype and each single event type could have multiple prototypes. Given an event mention, its distances to the prototypes are computed and aggregated by event types to determine the overall distance to each event type.

heading: Two types of Contrastive Learning
text: We name the representation of event mention as query and prototypes (i.e., other event mentions) as keys. Then CL could be further split into two cases, in-batch CL (Chen et al., 2020) and MoCo CL (He et al., 2020), according to where their keys are from. In-batch CL views other event mentions within the same batch as the keys, and the encoder for computing the queries and keys in batch-CL is updated end-to-end by back-propagation. For MoCo CL, the encoder for key is momentum-updated along the encoder for query, and it accordingly maintains a queue to store keys and utilizes them multiple
times once they are previously computed. We refer readers to MoCo CL (He et al., 2020) for the details of in-batch CL and MoCo CL.
CONTAINER (Das et al., 2022) adopts in-batch CL setting for few-shot NER model and we transfer it to ED domain in our empirical study. We further compare the two types of CL for our unified baseline with effective components in Section 5.2 and present the full results in Table 10. We observe in-batch CL outperforms MoCo-CL when the number of the sentence is small, and the situation reverses with the increasing of sentence number. We speculate it is due to two main reasons: (1) When all sentences could be within the single batch, inbatch CL is a better approach since it computes and updates all representations of keys and queries end-to-end by back propagation, while MoCo-CL computes the key representation by a momentumupdated encoder with gradient stopping. When the sentence number is larger than batch size, however, in-batch CL lose the information of some samples in each step, while MoCo-CL keeps all samples within the queue and leverages these approximate representations for a more extensive comparison
and learning. (2) MoCo-CL also has an effect of data-augmentation under few-shot ED task, since the sentence number is usually much smaller than the queue size. Then the queue would store multiple representations for each sample, which are computed and stored in different previous steps. The benefits of such data augmentation take effect when there are relatively abundant sentences and accordingly diverse augmentations.

heading: D Class-transfer Setting-Extended

heading: D.1 Prompt-based methods
text: We list the results of existing prompt-based methods on class-transfer setting in Table 11. See detailed analysis in Section 6.1.

heading: D.2 Prototype-based methods
text: We list the results of existing prototype-based methods plus our developed unified baseline under classtransfer setting in Table 12. Note that we substitute the appropriate distance functions d and transfer functions f obtained in Section 5.2 for existing methods. See detailed analysis in Section 6.2.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
After the acknowledgement, before the reference.
7 A2. Did you discuss any potential risks of your work? To our best knowledge, our work is an empirical study based on previous work and there is no potential risks of our work.
3 A3. Do the abstract and introduction summarize the paper’s main claims? In abstract and Section 1.
3 A4. Have you used AI writing assistants when working on this paper? We use ChatGPT to polish our paper, mainly on abstract and limitation part.
B 3 Did you use or create scientific artifacts? In Section 4.1, Appendix B.1 and Appendix B.2
3 B1. Did you cite the creators of artifacts you used? In Section 4.1 and Appendix B.1
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? In Appendix B.2
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? In Appendix B.2
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? To our best knowledge, no such problems in three datasets we use.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? In Appendix B.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In Appendix B.2
C 3 Did you run computational experiments? In Section 4.2, Section 4.3 and Appendix B.4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? In Section 4.3 and Appendix B.4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? In Appendix B.4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In Section 4.2 and Appendix B.4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? In Appendix B.4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Therefore, for future work, we aim to conduct our empirical studies on more diverse event-detection (ED) datasets. We are fortunate to witness the rapid development of Large Language Models (LLMs Brown et al. 2020b; Ouyang et al. 2022; Chung et al. 2022) in recent times. In our work, we set incontext learning as a baseline and evaluate the performance of LLMs on few-shot ED tasks. We find current LLMs still face challenges in dealing with Information Extraction (IE) tasks that require structured outputs (Qin et al. , 2023; Josifoski et al. , 2023). However, we acknowledge the ICL approach adopted here is relatively simple. We do not work hard to find the optimal prompt format, demonstration selection strategy, etc. , to reach the upper bounds of LLMs’ performance. We view how to leverage the power of LLMs on ED tasks as an open problem and leave it for future work. In this work, we focus more on the model aspect of few-shot ED tasks rather than data aspect. In other words, we assume having and only having access to a small set of labeled instances. In the future, we plan to explore how to utilize annotation guidelines, unlabeled corpus and external structured knowledge to improve few-shot ED tasks