Improving Event Definition Following For Zero-Shot Event Detection (2024)

Abstract: Existing approaches on zero-shot event detection usually train models on datasets annotated with known event types, and prompt them with unseen event definitions. These approaches yield sporadic successes, yet generally fall short of expectations. In this work, we aim to improve zero-shot event detection by training models to better follow event definitions. We hypothesize that a diverse set of event types and definitions are the key for models to learn to follow event definitions while existing event extraction datasets focus on annotating many high-quality examples for a few event types. To verify our hypothesis, we construct an automatically generated Diverse Event Definition (DivED) dataset and conduct comparative studies. Our experiments reveal that a large number of event types (200) and diverse event definitions can significantly boost event extraction performance; on the other hand, the performance does not scale with over ten examples per event type. Beyond scaling, we incorporate event ontology information and hard-negative samples during training, further boosting the performance. Based on these findings, we fine-tuned a LLaMA-2-7B model on our DivED dataset, yielding performance that surpasses SOTA large language models like GPT-3.5 across three open benchmarks on zero-shot event detection. Our code and data can be found at https://github.com/PlusLabNLP/ZeroED

heading: 1 Introduction
text: Event detection (ED) focuses on identifying event triggers of specific event types in a given text with predefined event ontology. Prior work has studied event detection largely in a fully-supervised fashion (Wadden et al., 2019; Lin et al., 2020; Nguyen and Grishman, 2015; Nguyen et al., 2016; Han et al., 2019; Du and Cardie, 2020; Cheng
*Equal contribution.
1
et al., 2023b; Cai et al., 2023; Cheng et al., 2023a; Huang et al., 2020; Huang and Peng, 2020; ?; Paolini et al., 2021; Ma et al., 2023b). While these work show promising performance on seen events, it cannot generalize well to long-tailed and unseen events (Ma et al., 2024; Zhang et al., 2022). To further enable generalization to low-resource events, prior work proposed to tackle few-shot event detection by training model on generated pseudo data (Ma et al., 2024; Kumar et al., 2020; Schick and Schütze, 2021). Despite the success in data-efficient event detection, they cannot zero-shot extract unseen events in real-time due to the need for prior training, limiting their applicability to a wider range of scenarios.
The success of task generalization of LLMs enabled by instruction tuning further advances zero-shot event detection. Recent work started to extract events of novel type by providing LLMs with the event definition of unseen events during inference, as demonstrated in Figure 1. They either prompt closed-source LLMs, such as GPT-3.5 (Wang et al., 2022; Gao et al., 2023a; Wei et al., 2023), or apply transfer learning on
2842
open-sourced LMs with EE training data of seen event types (Huang et al., 2018a; Lyu et al., 2021). While the former methods achieve acceptable performance, they are not flexible and reproducible due to their closed nature, leading to the difficulty in further improving the models’ performance. In contrast, the latter methods while reproducible and flexible, suffer from low performance.
In this work, we aim to enhance zero-shot event detection (ED) by training a model with improved generalization to unseen event types. During inference, the model, prompted with definitions of previously unseen events, relies on its instructionfollowing ability to understand event definitions and identify correct triggers. Despite recent impressive results of instruction following by Large Language Models (LLMs), there is room for improvement (Kung and Peng, 2023; Shao et al., 2023; Yin et al., 2023), and we focus on enhancing it via instruction fine-tuning with strategically generated data. Specifically, we hypothesize that transfer learning from conventional EE datasets might not be ideal (Kung et al., 2021, 2023). Though a large amount of high-quality training samples for only a few event types equips the model to perform EE on homogeneous data, it is not sufficient for the model to develop generalizability towards unseen situations. Different from existing works that aim to improve ED model by generating more homogeneous EE data (Ma et al., 2024), we posit that a large number of event types and a diverse set of event definitions are the keys to improving the event definition following capabilities.
To verify our assumption, we develop Diverse Event Definition (DivED) Dataset, which is generated from LLMs with diverse event definitions and samples. DivED includes 3000+ event types, each with 10 event definitions and 10 samples. Since event types can be organized into tree-structure ontology, we further inject each event’s event type dependencies information into its event definition, including the name of its parent and sibling events.
Our study on the DivED dataset supports our hypothesis. The results indicate that a sufficient amount of event types (200) and diverse event definitions significantly enhance zero-shot event detection performance on out-of-distribution data, underscoring their crucial role in event definition comprehension. On the other hand, the performance doesn’t improve significantly with more than ten samples per event type. This is attributed to the reliance of zero-shot event detection on
the model’s ability to generalize to new event types and definitions. While a few samples aid in learning the meaning of event types, an excessive number is unnecessary. In addition to scaling, we explore the impact of incorporating event ontology information in event definition and utilizing hard-negative samples during training. We observe that incorporating both components enhances the model’s comprehension of event boundaries, resulting in higher recall and F1 scores.
Following this finding, we further train our model on the DivED and Geneva (Parekh et al., 2023a) datasets, and achieve state-of-the-art zero-shot event detection performance on ACE (Doddington et al., 2004a), M2E2 (Li et al., 2020a) and MEE (Veyseh et al., 2022) test sets benchmarked in TextEE (Huang et al., 2023), surpassing strong LLMs such as Chat-GPT with less than 5 percents of model parameters, showing the effectiveness and efficiency of our method. To conclude, our main contributions are as follows:
1. We design a data generation pipeline to generate a Diverse Event Definition dataset (DivED) with 3000+ event types and 10 diverse event definitions for each type. Our experiments reveal that diverse event types and event definitions are crucial to improve zero-shot ED.
2. We systematically study the impact of various components of EE training data on the ability of large instruction-tuned models to follow event definitions.
3. Our proposed model achieves SOTA results on ACE, M2E2, and MEE datasets, surpassing GPT-3.5-Turbo model with drastically fewer parameters.

heading: 2 Method
text: In this section, we describe our data generation pipeline to generate the Diverse Event Definition Dataset (DivED) and our systematic study on the impact of various components within event detection training data. We investigate how (1) the scaling of event types, event definition, and training samples and (2) incorporating ontology information and hard-negative samples can impact models’ generalization to unseen event types.

heading: 2.1 DivED Dataset Generation
text: An event detection dataset with diverse event types and definitions is necessary to investigate the effect of training data components systematically. Thus, we propose an automatic data generation pipeline that leverages proprietary LLM to generate Diverse Event Definition Dataset (DivED). The data generation pipeline includes (1) Event Type Name Retrieval, (2) Ontology-Aware Event Definition Curation, (3) Ontology-Aware Sample Curation, (4) Event Definition Expansion, and (5) Ontology Pruning. Figure 2 illustrates the five-step data generation pipeline. We use ChatGPT (GPT3.5-Turbo) for the dataset generation process. The average number of sibling event types is 2.14, and the number of ancestor event types is 1510. The total number of events is 3232. The examples of DivED and the data generation pipeline templates can be found in Appendix B.
Step 1: Event Type Name Retrieval We follow (Zhan et al., 2023) methods to collect around 6000 event type names with ontology (dependency trees) from XPO-overlap (Spaulding et al., 2023a), which provides a large set of event entities that occurred in Wikidata.* To guarantee the testing events from ACE (Doddington et al., 2004b), M2E2 (Li et al., 2020b), and (Veyseh et al., 2022) datasets are held out for our later experiments, we manually filtered out all events that share the same dependency trees with these testing events.
*wikidata.org
Step 2: Ontology-Aware Event Definition Curation After acquiring event type names and ontology (dependency trees of events), we instruct the model to simultaneously generate concise definitions for all event types within the ontology. Using one manually curated in-context example, we guide the model to differentiate similar events within the same ontology, resulting in distinct and welldistinguished event definitions, as demonstrated in Table 1.
Step 3: Ontology-Aware Sample Curation We follow a similar method as in Ontology-Aware Event Definition Curation to prompt the model with relative event types, event definition, and one manually curated in-context example to generate ten samples for multiple event types simultaneously. Each generated sample includes an input sentence and an output trigger of the corresponding event type. The generated samples can be seen in Table 1.
Step 4: Event Definition Expansion To get multiple event definitions for each event type, we prompt the model to expand or paraphrase the event definition ten times with the provided event type name, event definition, event ontology, and one manually curated in-context example.
Step 5: Ontology Pruning After generating data for all event types, we further prune out duplicate events within the same event ontology by identifying their output trigger overlap. Specifically, for an event ontology tree {e1, e2, ...} ∈ E with multiple event types and ten samples per event, we
calculate the output trigger overlap ratio between two event types ei, ej where i ̸= j. The trigger overlap is measured by exact string matching each of the ten triggers in ei with the ten triggers in ej . If the overlap ratio of output triggers exceeds a certain threshold (in our implementation, it is 0.5), we will consider one of the two events as duplicate and remove it from our dataset. This way, we can guarantee that the event types and output triggers of our dataset are diverse.

heading: 2.2 Data Impact Analysis
text: With the generated DivED dataset, we systematically study the impact of various components in training data to understand how to train the model with improved event definition following ability.
Scaling of data components In Figure 1, we show the data components within the training data. During training, we will provide several samples, each corresponding to an event type and definition, to query the models about the event trigger. In testing time, we will further test on the unseen events, in which all the event types, definitions, and samples are unseen from training. This requires the model to generalize to the unseen events to be able to perform well during testing. Following this intuition, we aim to investigate how different numbers of events, definitions, and samples can influence models’ performance. Specifically, for each dataset component, we fix the quantity of other components and evaluate the scaling law associated with it. For example, to investigate the scaling of event definition, we will use different number of event definitions per event, with a fixed amount of event type and samples.
Ontology information We further look into the construction of event definition and negative samples. In most zero-shot EE methods, they solely provide the information (event type, event definition) of the current event, without providing the event ontology information. We explore adding ontology information to the input definition in order to see how it helps models with the understanding of the event, and generalize better to unseen event types. The additional ontology information includes the parent and child events of an event ontology, as shown in Figure 1.
Hard-negative samples During model training, we use input sentences paired with output triggers. Positive samples are based on ground truth events, where the output for event trigger identification is not "None." To incorporate predictions for "None" events, we create negative samples by prompting the model with input sentences and an event type that does not occur in the sentence. In this work, we aim to explore how integrating ontology information into negative sample construction affects the model’s ability to learn event definitions and boundaries. Instead of randomly assigning unrelated events during negative sample creation, we will assign sibling event types to form hard-negative samples. These challenging examples may offer additional signals about event boundaries that aid the model in distinguishing between similar events and improve its understanding of event definitions.

heading: 3 Experimental Setup
text: In this section, we first describe the details of Data Impact Analysis experiments, which analyze the impact of different data components. We further describe baselines and training details in Enhancing
Zero-Shot Event Detection, in which we integrate the optimal settings from Data Impact Analysis to train our zero-shot event detection model and compare to previous state-of-the-art large language models (LLMs) on three event extraction datasets.

heading: 3.1 Data Impact Analysis
text: Model and Training Details We utilize LLaMA2-7B (Touvron et al., 2023) for our experiments to examine the impact of various data components. Employing a batch size of 96, a learning rate of 2e-5, and training for 20 epochs. We divide the DivED dataset into training and validation sets by random sampling event ontologies. For the training set, We consistently use the DivED dataset with unified variables: 200 Event Types, 10 Event Definitions, 10 Samples, and 10 negative samples per sample. For the validation set, we first random sample around 50 event ontology, each including a parent event and multiple child events, in total 100 event types. The validation set can be considered an in-domain zero-shot ED evaluation set with all test events unseen from training but in a similar domain (all generated by ChatGPT), while ACE is a out-of-domain zero-shot ED test set previously curated by human annotators and has many different data characteristics, as described in Table 2. In each scaling experiment, only one variable is scaled while others are fixed. Notably, ACE-related events
are excluded from the DivED dataset to ensure the test events are entirely novel.
Event type and event definition scaling We experiment on [200, 400, 800, 1600, 3200] events on the DivED dataset for event-type scaling. For Event definition scaling, we test [1, 2, 4, 8, 10] event definition to investigate the scaling law of these variables.
We employ the DivED dataset for definition and event type scaling due to its extensive collection of event definitions and types, a feature absent in traditional event detection datasets like the Geneva dataset. We present the evaluation results of the DivED validation set for event type and definition scaling since it reflects in-domain performance, as both DivED-train and DivED-val are generated using the LLM with similar prompts.
Sample scaling For sample scaling, since DivED only has ten samples per event, we conduct it on the Geneva (Parekh et al., 2023a) dataset and test with [1, 5, 10, 20, 40] samples per event. We filter out all ACE-related events for the Geneva dataset to make sure the test events are unseen.
We have studied sampling scaling using the GENEVA dataset, containing up to 40 high-quality human-generated event samples, a resource lacking in our DivED datasets (which have ten samples per event). When using the Geneva dataset, evaluation
results on the DivED validation set do not represent in-domain performance like event type and definition scaling, which may cause confusion to the reader. Thus, we do not show the DivED-val result here. While there exist prior work studying sample scaling, our focus is to study event definition and types scaling since the feature is absent in traditional ED datasets.
From sample scaling experiments on the Geneva dataset, the main insight is that a sufficient amount (10) of samples are sufficient for zero-shot event detection, and we further follow this insight to generate the DivED dataset. It is possible that the DivED data samples can suffer from the diversity issue, and the exact number of samples to achieve optimal performance can be different from Geneva; however, we show that using ten samples can already achieve performance superior to proprietary LLM such as ChatGPT. Further hyperparameter tuning and optimization might achieve higher performance but is not the focus of our paper.

heading: Event Ontology and Hard Negative Samples In
text: event ontology experiments, we assess two settings: with or without event ontology. For hard negative samples, we experiment using zero or three hard negative samples within the ten negative samples. Evaluation is conducted on the ACE dataset.
Evaluation We report the F1 scores of event trigger identification and event trigger classification on DivED (in-domain) and ACE (out-domain) validation set. In in-domain evaluation, 50 unseen events from DivED (absent in training) are tested. For out-domain assessment, the model is tested on the ACE dataset, comprising 33 event types unseen during training. We analyze the models’ zero-shot generalization on these sets, presenting a comparison in Figure 3. Notably, DivED (in-domain) exhibits similar event definition and sample length to the training set, while ACE (out-domain) has longer definitions and a different writing style, focusing on argument details alongside the event itself.

heading: 3.2 Enhancing Zero-Shot Event Detection
text: Training Details Following our observation in Data Impact Analysis, we further train the LLaMA-2-7B (Touvron et al., 2023) models on DivED and Geneva (Parekh et al., 2023a) dataset following the optimal setting. We aim to show that our method has the potential to achieve performance surpassing large language models such as ChatGPT with a small 7B LLaMA model. By
mixing the event type definitions from DivED and GENEVA, we can further improve the diversity of the input and event definition styles to achieve improved performance, surpassing ChatGPT models with orders of magnitude fewer model parameters. By merging these two datasets, we also demonstrate that our method is scalable and has the potential to be combined with existing EE datasets to achieve improvements. We use 200 + 90 event types from DivED and Geneva (Parekh et al., 2023a) datasets. We use ten event definitions, ten samples, and ten negative samples per sample for each event type while incorporating the ontology information and three hard-negative samples.
Baselines In our experiments, we conduct a comparison between our finetuned LLaMA-2-7B (Touvron et al., 2023) and several zero-shot event detection baselines, including ChatGPT (OpenAI, 2021), ChatIE (Wei et al., 2023) and (Gao et al., 2023a) and LLaMA-2-Geneva. Prompt templates for the baselines are provided at Appendix C.
• ChatGPT (OpenAI, 2021): GPT-3.5-Turbo was prompted with the proposed method for a fair comparison with finetuned LLaMA-2-7B.
• ChatIE (Wei et al., 2023): ChatIE is a framework that transforms the zero-shot event detection task into a multi-turn question-answering problem. Here, LLMs are first prompted (as shown in Appendix C) to identify the event type and then sequentially prompted to iden-
tify the trigger.
• Gao et al. (2023a): This work explores the feasibility of ChatGPT as a zero-shot event detection model and further analyses the impact of event definitions, in-context examples and counterfactual examples in the prompt template presented at Appendix C. We prompt ChatGPT with event definitions and positive examples in our implementation as this setup performed best on Gao et al. (2023a) evaluation.
• LLaMA-2-Geneva: We additionally train an LLaMA2-7B model (Touvron et al., 2023) on Geneva (Parekh et al., 2023a) datasets as a transfer learning baseline. We first filter out all events related to ACE, M2E2, and MEE datasets from the training set, leaving 90 event types. We further train the model with all samples on the remaining event types.
Evaluation Datasets Our experiments comprehensively compare our fine-tuned LLaMa-2-7B with baselines across three popular event extraction benchmarks, including ACE05, M2E2, and MEE. We consider the English annotations of these datasets and report the F1 scores of event trigger identification and event trigger classification on their test set process by TextEE (Huang et al., 2023).
• ACE05 (Doddington et al., 2004b) is an endto-end event extraction dataset which covers texts from several sources such as newswire, broadcast news and weblogs.
• M2E2 (Li et al., 2020b) is an end-to-end event extraction dataset collected from the multimedia domain. We only consider the text part.
• MEE (Veyseh et al., 2022) is a multilingual end-to-end event extraction dataset collected from Wikipedia which is extended from MINION (Song et al., 2015).

heading: 4 Results

heading: 4.1 Data Impact Analysis
text: Scaling of event types In Figure 3 column (a), we show the results of training a LLaMA-2-7B model with different numbers of events. It is seen that scaling up the number of events consistently helps the model performance on the in-domain
DivED validation set. However, while training the model with more events continuously scales up the performance on the ACE validation set under 200 events, using more than 200 events leads to degeneration of the performance. This can be caused by the model overfitting to the domain of training data. While we continuously train on new events, the model can still overfit to the domain of the data itself, for example, the format of the event definition and samples. This also shows that while our generated DivED dataset has a large number of events, the generated samples and event definition might still have spurious correlations that can lead to overfitting.
Scaling of event definition In Figure 3 column (b), we show the results of event definition scaling. On both the DivED and ACE validation sets, the performance scales up with four event definitions per event. While using more than four event definitions does not help the in-domain performance, it can help the model generalize better to the out-domain test set. This shows that adding more diverse event definitions during training can further improve the model’s robustness. Helping the model to generalize to more diverse event formats during inference, thus improving zero-shot performance.
Scaling of samples To evaluate how using more samples helps the model’s zero-shot generalization, we experiment using the Geneva dataset, which has a large number of high-quality samples per event type, and test on the ACE validation set. The results are shown in Figure 3 column (c). Surprisingly, while using more samples usually helps models’ performance in a supervised setting, using more than ten samples hurts models’ performance. This can be caused by the model overfitting the training data and becoming less robust to unseen events.
Following the results above, we conclude that the key to improving models’ zero-shot generalization to unseen events is to use a diverse set of event definitions with a certain amount of event types and samples. While a small amount of event type and samples helps, using too much
can make the models overfit to the training source, leading to a degeneration of the generalization ability. This effect can be specifically obvious in machine-generated data, which can have spurious correlations and lack diversity in certain aspects.
Event ontology and hard-negative samples In Table 3, we further investigate the usefulness of the event ontology and hard-negative samples. It can be seen that after removing the event ontology or hard-negative samples, fewer triggers are predicted, which leads to a much lower recall and F1 score. This means that the model becomes more conservative at predicting triggers. We hypothesize that the model can be trained to distinguish similar events by providing ontology and hard negative samples. At testing time, this can help the models be more certain at predicting the triggers and increase the number of matching triggers.

heading: 4.2 Enhancing Zero-Shot Event Detection
text: Following the observation from Data Impact Analysis, we further apply the best setting and compare it with baseline models described in section 3.2. We show the results in Table 4. It can be seen that LLaMA-2-7B trained on the DivED dataset (Ours) consistently outperforms GPT baselines (ChatIE, Gao et al. (2023a) and GPT-3.5) on all ACE, M2E2, and MEE datasets and surpasses our LLaMA2-Geneva baselines on ACE and M2E2 datasets. For the MEE dataset, LLaMA2-Geneva achieves the best performance. Upon further investigation into the predicted results, we found that LLaMA2-Geneva can better predict samples that have multiple ground truth event triggers in one event type, which frequently occurred in MEE Geneva datasets but less occurred in ACE, M2E2 and DivED datasets, directly leading to higher Recall and F1 scores on MEE dataset. Generally, our proposed model achieves the best average F1 scores
on both Event Trigger Identification and Event Trigger Classification, showing the superiority of the training method.

heading: 5 Discussion

heading: 5.1 Ablation Study of Training Data
text: As mentioned in section 3.2, we train the LLaMA2-7B (Touvron et al., 2023) models on DivED and Geneva (Parekh et al., 2023a) dataset. In Table 5, we further provide results that only use the DivED dataset without GENEVA.
It can be seen that, by solely training on the DivED dataset, we can already achieve significant performance improvement and surpass model training on the human-curated Geneva dataset. When combined with the Geneva dataset, we can further boost the performance, showing the good scalability of our method.
For the results in Table 5, it does show that the current models are still limited to generalize to expert domains without domain adaptation. This can be due to the models’ lack of the basic knowledge to solve these expert domains. While our work provides a method to improve event definition following for better zero-shot ED, the base model we used (llama-2) does not have information about these expert domains and still requires further domain adaptation to achieve a generally good performance on them. This limitation also happens with low-resourced language. If the base model is not trained on a specific knowledge, even if our curated dataset helps with event definition following, the model cannot perform it well, due to the
lack of language information. Note that while our proposed DivED dataset does not directly generalize to these expert domains or other languages, the insight from data impact analysis can be utilized to generate specific datasets for these domains, and potentially help the zero-shot ED performance.

heading: 5.2 Do Models Follow the Event Definition?
text: Instruction-tuned models excel in various zero-shot tasks but can excessively rely on the spurious patterns within the provided prompt, neglecting instruction semantics (Kung and Peng, 2023; Yin et al., 2023). In this work, we aim to enhance zeroshot event detection by training model with better event definition following. To assess the model’s event definition following ability, we conduct an ablation study following prior work’s setting (Kung and Peng, 2023), comparing our LLaMA-2-7B model trained on DivED data with one trained on the conventional EE dataset Geneva (Parekh et al., 2023b). To verify whether our proposed model have better event definition following ability compare to models learning on convention EE datasets, we follow prior work (Kung and Peng, 2023) to conduct an ablation study. We compare our LLaMA-2-7B (Touvron et al., 2023) model trained on DivED data with a LLaMA-2-7B model trained on conventional EE dataset such as Geneva (Parekh et al., 2023b). Despite having numerous samples per event, conventional EE dataset has only one definition per event type, which largely differs from DivED dataset. We report the performance drop rate after removing the event definition during training and testing for both models in Table Table 6. It can be seen that while the performance drops for both models after removing the event definition during training, the model trained on the DivED dataset has a higher performance drop, especially in Event Trigger Identification, showing that our proposed model heavily relies on event definition during training and inference. This
indicates that our model is better at utilizing the event definition information, potentially exhibiting a better event definition following ability.

heading: 6 Related Work
text: Low-resource information extraction Lowresource IE models (Si et al., 2023; Hu et al., 2023) secure their performance with limited training data by cross-task transfer learning that uses supervision from tasks like Semantic Role Labeling (Zhang et al., 2021; Huang et al., 2018b), indirect supervision that reformulates the task as data-rich tasks like NLI or QA (Xu et al., 2023; Sainz et al., 2022; Ma et al., 2023a; Lu et al., 2022), both heavily rely on task compatibility. Some works focus on prompting generative LMs with enriched task requirements and examples (Li et al., 2023; Gao et al., 2023b), which is constrained the diversity of human-curated training data.
Data generation for IE Existing works explore different strategies to generate training data instances given a known task output space through analogous input (Kumar et al., 2020; Lee et al., 2021), creating pesudo labels with weak annotator (He et al., 2021; Chia et al., 2022; Ye et al., 2022; Wang et al., 2023; Schick and Schütze, 2021), reverse generation (Meng et al., 2022; Gao et al., 2021; Josifoski et al., 2023) and structure-to-text generation (Ma et al., 2024). Different from introducing more data instances for observed task space, we instead aim to extend the model’s generalizability by generating new types and their definitions for unseen data distribution that extend the task space with LLM-oriented data generation.

heading: 7 Conclusion
text: We investigate how incorporating diverse event types and definitions benefits zero-shot event detection models. The proposed DivED dataset features a large number of diverse event types and definitions, which helps train the model to better generalize to unseen event definitions. By further incorporating event ontology and hard negative samples, we finetuned a LLaMA-2 model on DivED and Geneva datasets, which consistently surpasses previous SOTA ChatGPT prompting baselines in zero-shot ED on ACE, M2E2, and MEE datasets. Overall, our findings provide insights to improve models’ event definition following ability and provide an opportunity to further advance zero-shot ED on open-sourced models.

heading: Limitation
text: Our study on zero-shot event detection, despite its advances, faces several limitations. The reliance on automatically generated datasets may not fully capture complex real-world events, potentially limiting the model’s generalizability. Additionally, the effectiveness of our approach depends on detailed event ontology and the availability of hard-negative samples, which might not always be accessible. Scalability also poses a challenge, as expanding the diversity of event types requires significant computational and data resources. Moreover, our findings are primarily based on English language benchmarks, raising questions about the applicability of our results across different languages and domains. Future research should address these limitations to enhance the robustness and universality of zeroshot event detection models.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their feedback. This work was partially supported by AFOSR MURI via Grant #FA9550-22-1-0380, Defense Advanced Research Project Agency (DARPA) grant #HR00112290103/HR0011260656, CISCO and ONR grant #N00014-23-1-2780.

heading: A Cost Estimates for OpenAI
text: We implement all baselines in Section 3.2 with GPT-3.5-Turbo. The estimated cost of implementing our baselines is approximately 100 USD. Similarly, the estimated cost of implementing our baselines on GPT-4 will be approximately 3000 USD and we leave this implementation for future due to limited resources. This further emphasizes that our finetuned model surpasses larger LLMs in performance as well as accessibility due to the cost effectiveness of the method.
For Event Definition Curation, Sample Curation, and Event Definition Expansion, the expense of using GPT models was less than 100 USD; for Event Type Name Retrieval and Ontology Pruning, the whole process was automatically conducted without labor or expenses. Therefore, the effort needed to produce the curated dataset is very small compared to manually collected datasets.

heading: B Data Generation of DivED dataset
text: Step 1: Event Type Name Retrieval We follow (Zhan et al., 2023) methods to collect around 6000 event type names with ontology (dependency trees) from XPO-overlap (Spaulding et al., 2023a), which provides a large set of event entities that occurred in Wikidata.† To guarantee the testing events from ACE (Doddington et al., 2004b), M2E2 (Li et al., 2020b), and (Veyseh et al., 2022) datasets are held out for our later experiments, we manually filtered out all events that share the same dependency trees with these testing events.
Step 2: Ontology-Aware Event Definition Curation After acquiring event type names and ontology (dependency trees of events), we instruct the model to simultaneously generate concise definitions for all event types within the ontology. Using one manually curated in-context example, we guide the model to differentiate similar events within the same ontology, resulting in distinct and welldistinguished event definitions, as demonstrated in Table 1. The template utilized in this step is presented at Table 7.
Step 3: Ontology-Aware Sample Curation We follow a similar method as in Ontology-Aware Event Definition Curation to prompt the model with relative event types, event definition, and one manually curated in-context example to generate
†wikidata.org
ten samples for multiple event types simultaneously. Each generated sample includes an input sentence and an output trigger of the corresponding event type. The generated samples can be seen in Table 1. The template utilized in this step is presented at Table 8.
Step 4: Event Definition Expansion To get multiple event definitions for each event type, we prompt the model to expand or paraphrase the event definition ten times with the provided event type name, event definition, event ontology, and one manually curated in-context example. The template utilized in this step is presented at Table 9.
Step 5: Ontology Pruning After generating data for all event types, we further prune out duplicate events within the same event ontology by identifying their output trigger overlap. Specifically, for an event ontology tree {e1, e2, ...} ∈ E with multiple event types and ten samples per event, we calculate the output trigger overlap ratio between two event types ei, ej where i ̸= j. The trigger overlap is measured by exact string matching each of the ten triggers in ei with the ten triggers in ej . If the overlap ratio of output triggers exceeds a certain threshold (in our implementation, it is 0.5), we will consider one of the two events as duplicate and remove it from our dataset. This way, we can guarantee that the event types and output triggers of our dataset are diverse.
Generated Samples Table 13, Table 14 and Table 15 present examples of DivED dataset.

heading: C Templates for Experiments
text: The compared baselines include ChatGPT, ChatIE (Wei et al., 2023) and (Gao et al., 2023a).
• ChatGPT: ChatGPT were prompted with the proposed method for a fair comparison with our finetuned LLaMA-2-7B. Prompt template is provided in Table 12.
• ChatIE (Wei et al., 2023): ChatIE is a framework that transforms the zero-shot event detection task into a multi-turn question-answering problem. Here, LLMs are first prompted (as shown in Table 11 )to identify the event type and then sequentially prompted to identify the trigger.
• Gao et al. (2023a): This work explores the feasibility of ChatGPT as a zero-shot event datection model and further analyses the impact
of event definitions, in-context examples and counterfactual examples in the prompt template in Table 10. We prompt ChatGPT with event definitions and positive examples in our implementation as this setup performed best on Gao et al. (2023a) evaluation.

heading: GPT Output:

heading: Query:

heading: Instruction:

heading: In-Context Demonstration:

heading: Instruction:

heading: GPT Output:

heading: Query:

heading: Instruction:

heading: Instruction:

heading: In-Context Demonstration:

heading: Instruction:

heading: GPT Output:

heading: Instruction:

heading: Query:

heading: Instruction:

heading: In-Context Demonstration:

heading: Instruction:

heading: Instruction:

heading: GPT Response:

heading: Instruction:

heading: Event Types and their definitions:

heading: Instruction:

heading: TRIGGER:

heading: Instruction:

heading: Event Samples:

heading: Event Definition:

heading: Event Ontology:

heading: Event Type:

heading: Future Work
text: Similarly, the estimated cost of implementing our baselines on GPT-4 will be approximately 3000 USD and we leave this implementation for future due to limited resources. This further emphasizes that our finetuned model surpasses larger LLMs in performance as well as accessibility due to the cost effectiveness of the method. For Event Definition Curation, Sample Curation, and Event Definition Expansion, the expense of using GPT models was less than 100 USD; for Event Type Name Retrieval and Ontology Pruning, the whole process was automatically conducted without labor or expenses. Therefore, the effort needed to produce the curated dataset is very small compared to manually collected datasets