The Art of Prompting: Event Detection based on Type Specific Prompts
Year: 2023

Abstract:
We compare various forms of prompts to represent event types and develop a unified framework to incorporate the event type specific prompts for supervised, few-shot, and zeroshot event detection. The experimental results demonstrate that a well-defined and comprehensive event type prompt can significantly improve event detection performance, especially when the annotated data is scarce (fewshot event detection) or not available (zero-shot event detection). By leveraging the semantics of event types, our unified framework shows up to 22.2% F-score gain over the previous stateof-the-art baselines1.

heading: 1 Introduction
text: Event detection (ED) (Grishman, 1997; Chinchor and Marsh, 1998; Ahn, 2006) is the task of identifying and typing event mentions from natural language text. Supervised approaches, especially deep neural networks (Chen et al., 2020; Du and Cardie, 2020; Lin et al., 2020; Liu et al., 2020; Li et al., 2020; Lyu et al., 2021), have shown remarkable performance under a critical prerequisite of a large amount of manual annotations. However, they cannot be effectively generalized to new languages, domains or types, especially when the annotations are not enough (Huang et al., 2016; Huang and Ji, 2020; Lai et al., 2020b; Shen et al., 2021) or there is no annotation available (Lyu et al., 2021; Zhang et al., 2021b; Pasupat and Liang, 2014).
Recent studies have shown that both the accuracy and generalizability of ED can be improved via leveraging the semantics of event types based on various forms of prompts, such as event type specific queries (Lyu et al., 2021; Du and Cardie, 2020; Liu et al., 2020), definitions (Chen et al., 2020), structures (Lin et al., 2020; Wang et al.,
1The source code, model checkpoints and data are publicly available at https://github.com/VT-NLP/Event_ APEX.
2019), or a few prototype event triggers (Wang and Cohen, 2009; Dalvi et al., 2012; Pasupat and Liang, 2014; Bronstein et al., 2015; Lai and Nguyen, 2019; Zhang et al., 2021b; Cong et al., 2021). These studies further encourage us to take another step forward and think about the following three questions: (1) does the choice of prompt matter when the training data is abundant or scarce? (2) what’s the best form of ED prompt? (3) how to best leverage the prompt to detect event mentions?
To answer the above research questions, we conduct extensive experiments with various forms of prompts for each event type, including (a) event type name, (b) prototype seed triggers, (c) definition, (d) event type structure based on both event type name and its predefined argument roles, (e) free parameter based continuous soft prompt, and (f) a more comprehensive event type description (named APEX prompt) that covers all the information of prompts (a)-(d). We observe that (1) by considering the semantics of event types with most forms of prompts, especially seed triggers and the comprehensive event type descriptions, the performance of ED under all settings can be significantly improved; (2) Among all forms of event representations, the comprehensive description based prompts show to be the most effective, especially for fewshot and zero-shot ED; (3) Different forms of event type representations provide complementary improvements, indicating that they capture distinct aspects and knowledge of the event types.
The contributions of this work are as follows: • We investigate various prompts to represent event types for both supervised and weakly supervised ED, and prove that a well-defined and comprehensive event type prompt can dramatically improve the performance of ED and the transferability from old types to new types.
• A unified framework is developed to leverage the semantics of event types with prompts for supervised, few-shot, and zero-shot ED, and demonstrate
1286
state-of-the-art performance with up to 22.2% Fscore improvement over the strong baseline methods.

heading: 2 Related Work
text: Supervised ED: Most of the existing Event Detection studies follow a supervised learning paradigm (Ji and Grishman, 2008; Liao and Grishman, 2010; McClosky et al., 2011; Li et al., 2013; Chen et al., 2015; Cao et al., 2015; Feng et al., 2016; Yang and Mitchell, 2016; Nguyen et al., 2016; Zhang et al., 2017; Lin et al., 2020; Wang et al., 2021b). However, they cannot be directly applied to detect new types of events. Recently studies have shown that, by leveraging the semantics of event types based on type-specific questions (Du and Cardie, 2020; Liu et al., 2020; Li et al., 2020; Lyu et al., 2021) or seed event triggers (Bronstein et al., 2015; Lai and Nguyen, 2019; Wang et al., 2021a), the event detection performance can be improved. However, it is still unknown whether they are the best choices for representing the semantics of event types.
Few-shot ED: Two primary learning strategies in few-shot classification tasks are MetaLearning (Kang et al., 2019; Li et al., 2021; Xiao and Marlet, 2020; Yan et al., 2019; Chowdhury et al., 2021) and Metric Learning (Sun et al., 2021; Wang et al., 2020b; Zhang et al., 2021a; Agarwal et al., 2021). Several studies have exploited metric learning to align the semantics of candidate events with a few examples of the novel event types for few-shot event detection (Lai et al., 2020a; Deng et al., 2020; Lai et al., 2020b; Cong et al., 2021; Chen et al., 2021; Shen et al., 2021).
Zero-shot ED: Huang et al. (2018) first exploited zero-shot event extraction by leveraging Abstract Meaning Representation (Banarescu et al., 2013) to represent event mentions and types into a shared semantic space. Recent studies (Zhang et al., 2021b; Lyu et al., 2021) further demonstrate that by leveraging a large external corpus with abundant anchor triggers, zero-shot event detection can also be achieved with decent performance without using any training data.
Prompt Learning Prompt learning aims to learn a task-specific prompt while keeping most of the model’s parameters frozen (Li and Liang, 2021; Hambardzumyan et al., 2021; Brown et al., 2020).
It has shown competitive performance in many applications of natural language processing (Raffel et al., 2020; Brown et al., 2020; Shin et al., 2020; Jiang et al., 2020; Lester et al., 2021; Schick and Schütze, 2021b). Previous work either used a manual (Petroni et al., 2019; Brown et al., 2020; Schick and Schütze, 2021a) or automated approach (Jiang et al., 2020; Yuan et al., 2021; Li and Liang, 2021) to create prompts.

heading: 3 Problem Formulation
text: Here, we first define each setting of the event detection task and then describe the various forms of event type prompts.

heading: 3.1 Settings of ED
text: For supervised ED (SED), we follow the conventional supervised event detection setting where the training, validation, and evaluation data sets cover the same set of event types. The goal is to learn a model f to identify and classify event mentions for the target event types.
For few-shot ED (FSED), there are two separate training data sets for few-shot event detection: (1) A large-scale data set Dbase = {(xi,yi)}Mi=1 that covers the old event types (named base types) where M denotes the number of base event types; (2) a smaller data set Dnovel = {(xj ,yj)}N×Kj=1 that covers N novel event types, with K examples each. Note that the base and novel event types are disjoint except for the Other class. The model f will be first optimized on Dbase, and then further fine-tuned on Dnovel. The goal is to evaluate the generalizability and transferability of the model from base event types to new event types with few annotations.
For zero-shot ED (ZSED), the training data sets are the only difference between zero-shot and fewshot event detection. In zero-shot event detection, there is only a large-scale base training data set Dbase = {(xi,yi)}Mi=1 for the base event types. The model f will be only optimized on base event types and evaluated on the novel types.

heading: 3.2 Event Type Prompts
text: We compare the following five forms of prompts to represent the event types: (a) Event Type Name is the event class name, usually consisting of one to three tokens. (b) Definition can be a short sentence that formally describes the meaning of the event types. (c) Prototype Seed Triggers a list of
tokens or phrases that are frequently identified as event triggers. (d) Event Type Structure consists of event key argument roles, indicating the core participants of the target event type. (e) Prompts can also be Continuous Soft Prompt, that is, a free vector of parameters to represent each event type. (f) We further define a more comprehensive description APEX Prompt that is manually written and covers all previous prompts except soft prompts. Examples of all event type prompts are shown in Figure 1 and Appendix A. Detailed prompt token selection is in Appendix B.

heading: 4 A Unified Framework for ED
text: We adapt (Wang et al., 2021a) and design a unified event detection framework (as shown in Figure 1) which leverages event type specific prompts to detect events under supervised, few-shot, and zeroshot settings. Formally, given an input sentence W = {w1, w2, . . . , wn}, we take each event type prompt T t = {τ t1, τ t2, . . . , τ tm} as a query of M tokens to extract triggers for event type t. Specifically, we first concatenate them into a sequence [CLS] τ t1 ... τ t m [SEP] w1 ... wn [SEP]. We use a pre-trained BERT encoder (Devlin et al., 2019) to get contextual representations for the input sentence W = {w0,w2, ...,wn} as well as the event type prompt T = {τ t0, τ t1, ..., τ tm}2.
Given a prompt of each event type, we aim to extract corresponding event triggers from the input sentence. To achieve this goal, we need to capture the semantic correlation of each input token to the event type Thus we learn a weight distribution over the sequence of contextual representations of the event type prompt, to obtain event type t aware contextual representation Ati =∑|T t|
j=1 αij · τ tj , where αij = cos(wi, τ tj), where 2In our experiments, the representation of each wi or τ i
is based on the contextual embedding of the first sub-token.
τ j is the contextual representation of the j-th prompt token. cos(·) is the cosine similarity function between two vectors.
With that, the event type aware contextual representation Ati will be concatenated with the original contextual representation wi from the encoder, and classified into a binary label, indicating whether it is a candidate trigger of event type t or not: ỹti = U o([wi; A t i;P i]), where [; ] denotes concatenation operation, U o is a learnable parameter matrix for event trigger detection, and P i is the one-hot part-of-speech (POS) encoding of word wi. For continuous soft prompt based event detection, we follow Li and Liang (2021) where a prefix index q is prepended to the input sequence W ′ = [q; W ]. The prefix embedding is learned by q = MLPθ(Qθ[q]), where Qθ ∈ R|Q|×k denotes the embedding lookup table for the vocabulary of prefix indices. Both MLPθ and Qθ are trainable parameters. Detailed learning strategy is in Appendix C.

heading: 5 Experiment Setup
text: We perform experiments on three public benchmark datasets, including ACE05-E+ (Automatic Content Extraction), ERE (Entity Relation Event) (Song et al., 2015),and MAVEN (Wang et al., 2020a). On each dataset, we conduct experiments for SED, FSED, and ZSED. For SED, we use the same data split as the previous studies (Li et al., 2013; Wadden et al., 2019; Lin et al., 2020; Du and Cardie, 2020; Lin et al., 2020; Nguyen et al., 2021; Wang et al., 2020a) on all the three benchmark datasets. For FSED and ZSED on MAVEN, we follow the previous study (Chen et al., 2021) and choose 120 event types with the most frequent mentions as the base event types and the rest 45 event types as novel ones. For FSED and ZSED on ACE and ERE, previous studies (Lai et al., 2020b,a;
Chen et al., 2021) follow different data splits and settings, making it hard for a fair comparison. Considering the research goals of FSED and ZSED, we define the following conditions to split the ACE and ERE datasets: (i) The base event types and novel event types should be disjoint except Other. (ii) Each base or novel event type should contain at least 15 instances. (iii) The training set should contain sufficient annotated event mentions.
To meet the above conditions, for ACE, we define the event types of 5 main event categories: Business, Contact, Conflict, Justice and Movement as the base event types, and types of the remaining 3 main categories: Life, Personnel and Transaction as the novel event types. In total, there are 18 qualified base types and 10 qualified novel types (the others do not satisfy the second condition). For ERE, we use the exact same 10 novel event types as ACE, and the rest 25 types as base event types. Detailed data and hyperparameter descriptions are in Appendix D and Appendix E.

heading: 6 Results and Discussion
text: Overall Results The experimental results for SED, FSED, and ZSED on ACE05, ERE, and
MAVEN are shown in Table 1-3, from which we see that (1) the APEX prompt achieves the best performance among all the forms of prompts under all the settings of the three benchmark datasets. Compared with the previous state of the art, the APEX prompt shows up to 4% F-score gain for SED (on ERE), 22.2% F-score gain for FSED (on ACE), and 19.7% F-score gain for ZSED (on MAVEN); (2) All the forms of prompts provide significant improvement for FSED and ZSED, demonstrating the benefit of leveraging the semantics of event types via various forms of prompts. (3) Except APEX, seed triggers provide more improvements than other forms of event type prompts under most settings, suggesting its potential to represent the semantics of event types accurately. (4) Continuous soft prompt does not provide comparable performance as other forms of event type representations, which proves the necessity of leveraging event type specific prior knowledge to the representations; (5) The majority voting does not show improvement over individual prompts since each prompt captures a particular aspect of the event type semantics.
Supervised Event Detection By carefully investigating the event mentions that are correctly detected by the APEX prompt while missed by other prompts, we find that the APEX prompt is more effective in detecting two types of event mentions: homonyms (multiple-meaning words) and intricate words. General homonyms are usually hard to be detected as event mentions as they usually have dozens of meanings in different contexts. For example, consider the following two examples: (i) Airlines are getting [Transport:Movement] flyers to destinations on time more often . (ii) If the board cannot vote to give [Transaction:Transfer-Money’] themselves present money. Here, “get” and “give”
are not detected based on the event type name or seed triggers but are correctly identified by the definition and APEX prompts. The definition and APEX prompts make 10% and 7% fewer false predictions than seed triggers on general homonyms. For intricate words, their semantics usually cannot be captured with an individual prompt. In the following two examples: (i) It is reasonable, however, to reimburse board members for legitimate expenses (ii) · · · ever having discussed being compensated by the board in the future · · ·, “reimburse” and “compensated” indicate sophisticated meaning of Transaction:Transfer-Money, which may not be captured by prompts, such as seed triggers. With the event definition and the argument roles in the APEX prompt, the highly correlated contexts, such as “board members” and “legitimate expenses”, can help the model correctly detect reimburse as an event mention of Transaction:Transfer-Money.
Few-shot Event Detection Figure 2 shows the F-score distribution of all novel types based on various forms of event type prompts, from which we observe that: (1) The event type name, seed triggers, and APEX prompt generally perform better than definition and structure, as they carry more straightforward semantics of event types. (2) Event type name based prompts show lower performance on Personnel:End-Position, Personnel:Start-Position and Transaction:Transfer-Money than other event types, as the semantics of these event type names are less indicative than other event types. (3) Seed trigger based prompts perform worse than event type name and APEX prompts on two event types, Life:injure and Life:die, probably because the prototype seed triggers are not properly selected. (4) The structure based prompt outperforms the other
prompts on Life:Injure as Life:Injure events require the existence of a person or victim. (5) APEX prompt shows consistently (almost) best performance on all the event types because it combines all the information of other prompts. (6) We also observe that the performance of Life:Be-Born, Life:Die, Life:Marry, and Personnel:Elect based on various forms of prompts are consistently better than the other types as the intrinsic semantics of those types the corresponding event triggers are concentrated.
Zero-shot Event Detection The proposed prompt-based method is more affordable to be generalized compared with the prior state-ofthe-art zero-shot approach (Zhang et al., 2021b). The average length of created APEX prompts is less than 20 tokens. Thus manually creating them will not take much human effort. On the contrary, Zhang et al. (2021b) requires an extensive collection of anchor sentences to perform zero-shot event detection, e.g., 4,556,237 anchor sentences for ACE and ERE. This process is time-consuming and expensive.

heading: 7 Conclusion
text: We investigate a variety of prompts to represent the semantics of event types, and leverage them with a unified framework for supervised, few-shot and zero-shot event detection. Experimental results demonstrate that, a well-defined and comprehensive description of event types can significantly improve the performance of event detection, especially when the annotations are limited (few-shot event detection) or even not available (zero-shot event detection), with up to 22.2% F-score gain over the prior state of the art.
Limitations
We have demonstrated that an accurate description can perform better for both supervised and weakly supervised event detection. However, the event types from most existing ontologies are not properly defined. For example, in ACE annotation guideline (Linguistic Data Consortium, 2005), transfer-money is defined as “giving, receiving, borrowing, or lending money when it is not in the context of purchasing something”. However, it is hard for the model to interpret it accurately, especially the constraints “not in the context of purchasing something”. In addition, many event types from MAVEN, e.g., Achieve, Award, and Incident, are not associated with any definitions. A potential future research direction is to leverage mining-based approaches or state-of-the-art generators to automatically generate a comprehensive event type description based on various sources, such as annotation guidelines, example annotations, and external knowledge bases.

heading: Acknowledgments
text: We thank the anonymous reviewers and area chair for their valuable time and constructive comments. This research is based upon work supported by the Amazon Research Award.

heading: A APEX prompt examples for ACE
text: Table 4 and Table 5 show APEX prompt examples for ACE events.

heading: B Prompt Token Selection
text: In our experiments, the event type names and event type structures are automatically extracted from the target event ontology, such as ACE (Linguistic Data Consortium, 2005), ERE (Song et al., 2015) and MAVEN (Wang et al., 2020a). The prototype seed triggers are automatically selected from the annotated data for supervised and few-shot event extraction. For zero-shot event extraction, we manually select R words from the NLTK synonyms of each event type as its prototype seed triggers. The definitions and APEX prompts are based on the official annotation guides for each target event ontology (Linguistic Data Consortium, 2005; Song et al., 2015; Wang et al., 2020a) and the available definitions in FrameNet (Baker et al., 1998) with manual editing.

heading: C Learning Strategy
text: The learning strategy varies for supervised, fewshot, and zero-shot learning. For supervised learning, we optimize the following objective for event trigger detection L = − 1|T ||N | ∑ t∈T ∑|N | i=1 y t i · log ỹti, where T is the set of target event types and N is the set of tokens from the training dataset. yti denotes the ground truth label vector. For few-shot event detection, we optimize the model on both base training data set and the smaller training data set for novel event types: L = − 1|T B ||NB | ∑ t∈T B ∑|NB | i=1 y t i · log ỹti − β 1|T N ||NN | ∑ t∈T N ∑|NN | i=1 y t i · log ỹti, where T B and NB denote the set of base event types and tokens from the base training data set, respectively. T N is the set of novel event types. NN is the set of tokens from the training data set for novel event types. β is a hyper-parameter to balance the two objectives. For zero-shot event detection, as we only have the base training data set, we minimize the following objective: L = − 1|T B ||NB | ∑ t∈T B ∑|NB | i=1 y t i · log ỹti.

heading: D Dataset
text: After defining the base and novel event types, we create the training, validation, and evaluation split for all three datasets. We use the sentences with
only base event type mentions as the base training data set for few-shot event detection, and randomly select 10 sentences with novel event type mentions as the additional smaller training data set. We use the sentences with both base and novel event type mentions as the development set and use the remaining sentences with only novel event type mentions as the evaluation dataset. We use the same development and evaluation set as few-shot event detection for zero-shot event detection and remove the instances with novel event mentions from the training set. We randomly split the sentences without any event annotations proportionally to the number of sentences with event mentions in each set for both zero-shot and few-shot event detection. Table 6 shows the detailed data statistics for all the three datasets under the few-shot and zero-shot event extraction settings.

heading: E Hyperparameters and Evaluation
text: For a fair comparison with the previous baseline approaches, we use the same pre-trained bert-large-uncased model for fine-tuning and optimizing our model with BertAdam. For supervised event detection, we optimize the parameters with grid search: training epoch is 3, learning rate ∈ [3e-6, 1e-4], training batch size ∈ {8, 12, 16, 24, 32}, dropout rate ∈ {0.4, 0.5, 0.6}. The running time is up to 3 hours on one Quadro RTX 8000. For evaluation, we use the same criteria as previous studies (Li et al., 2013; Chen et al., 2015; Nguyen et al., 2016; Lin et al., 2020): an event mention is correct if its span and event type match a reference event mention.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 8

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5 and Appendix C

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 5 and Appendix C

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 5

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 5
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 5
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix C
C 3 Did you run computational experiments? Section 5 and Appendix D

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix D
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5 and Appendix D

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix D

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: A potential future research direction is to leverage mining-based approaches or state-of-the-art generators to automatically generate a comprehensive event type description based on various sources, such as annotation guidelines, example annotations, and external knowledge bases