TEXTEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction (2024)

Abstract: Event extraction has gained considerable interest due to its wide-ranging applications. However, recent studies draw attention to evaluation issues, suggesting that reported scores may not accurately reflect the true performance. In this work, we identify and address evaluation challenges, including inconsistency due to varying data assumptions or preprocessing steps, the insufficiency of current evaluation frameworks that may introduce dataset or data split bias, and the low reproducibility of some previous approaches. To address these challenges, we present TEXTEE, a standardized, fair, and reproducible benchmark for event extraction. TEXTEE comprises standardized data preprocessing scripts and splits for 16 datasets spanning eight diverse domains and includes 14 recent methodologies, conducting a comprehensive benchmark reevaluation. We also evaluate five varied large language models on our TEXTEE benchmark and demonstrate how they struggle to achieve satisfactory performance. Inspired by our reevaluation results and findings, we discuss the role of event extraction in the current NLP era, as well as future challenges and insights derived from TEXTEE. We believe TEXTEE, the first standardized comprehensive benchmarking tool, will significantly facilitate future event extraction research.1

heading: 1 Introduction
text: Event extraction (Ji and Grishman, 2008) has always been a challenging task in the field of natural language processing (NLP) due to its demand for a high-level comprehension of texts. Since event extraction benefits many applications (Zhang et al., 2020; Han et al., 2021), it has attracted increasing attention in recent years (Luan et al., 2019; Lin et al., 2020; Nguyen et al., 2021; Hsu et al., 2022;
1TEXTEE benchmark platform is available at https: //github.com/ej0cl6/TextEE
Ma et al., 2022). However, due to the complicated nature of event extraction datasets and systems, fairly evaluating and comparing different event extraction approaches is not straightforward. Recent attempts (Zheng et al., 2021; Peng et al., 2023a,b) point out that the reported scores in previous work might not reflect the true performance in real-world applications because of various shortcomings and issues during the evaluation process. This poses a potential obstacle to the development of robust techniques for research in event extraction.
Motivated by the evaluation concern, this work aims to establish a standardized, fair, and reproducible benchmark for assessing event extraction approaches. We start by identifying and discussing several significant issues in the current evaluation process. First, we discuss the inconsistency issue caused by discrepant assumptions about data, different preprocessing steps, and the use of external resources. Next, we highlight the insufficiency problem of existing evaluation pipelines, which cover limited datasets and rely on fixed data splits, potentially introducing bias when evaluating performance. Finally, we emphasize the importance of reproducibility, which indirectly causes the aforementioned inconsistency and insufficiency issues.
To address these evaluation concerns, we propose TEXTEE, an evaluation platform that covers 16 datasets spanning diverse domains. To ensure fairness in comparisons, we standardize data preprocessing procedures and introduce five standardized data splits. Furthermore, we aggregate and re-implement 14 event extraction approaches published in recent years and conduct a comprehensive reevaluation. TEXTEE offers the benefits of consistency, sufficiency, reproducibility in evaluation. Additionally, we benchmark several large language models (LLMs) (Touvron et al., 2023; Tunstall et al., 2023; Jiang et al., 2024) for event
12804
extraction with TEXTEE and show the unsatisfactory performance of LLMs for this task.
Based on our reevaluation results and findings, we discuss the role of event extraction in the current era of LLMs, along with challenges and insights gleaned from TEXTEE. Specifically, we discuss how event extraction systems can be optional tools for LLMs to utilize, as well as highlight future challenges, including enhancing generalization, expanding event coverage, and improving efficiency.
In summary, our contributions are as follows: (1) We highlight and address the difficulties of fair evaluation for event extraction tasks. (2) We present TEXTEE as a benchmark platform for event extraction research and conduct a thorough reevaluation of recent approaches as well as LLMs. (3) Based on our results and findings, we discuss limitations and future challenges in event extraction.

heading: 2 Background and Related Work

heading: 2.1 Event Extraction
text: Event extraction (EE) aims to identify structured information from texts. Each event consists of an event type, a trigger span, and several arguments along with their roles.2 Figure 1 shows an example of a Justice-Execution event extracted from the text. This event is triggered by the text span execution and contains two argument roles, including Indonesia (Agent) and convicts (Person).
Previous work can be categorized into two types: (1) End-to-end (E2E) approaches extract event types, triggers, and argument roles in an end-toend manner. (2) Pipeline approaches first solve the event detection (ED) task, which detects trigger spans and the corresponding event types, then deal with the event argument extraction (EAE) task, which extracts arguments and the corresponding roles given an event type and a trigger span.

heading: 2.2 Related Work
text: Event extraction. Most end-to-end approaches construct graphs to model the relations between entities and extract triggers and argument roles accordingly (Luan et al., 2019; Wadden et al., 2019; Han et al., 2019; Lin et al., 2020; Huang et al., 2020; Nguyen et al., 2021; Zhang and Ji, 2021; Huang and Peng, 2021). There is a recent focus on employing generative models to generate
2In this work, we only cover closed-domain EE with a given ontology. We consider event mentions as events and do not consider event coreference resolution.
summaries for extracting events (Lu et al., 2021; Hsu et al., 2022). Unlike end-to-end approaches, pipeline methods train two separate models for event detection and event argument extraction. Different techniques are introduced, such as question answering (Du and Cardie, 2020; Liu et al., 2020; Li et al., 2020a; Lu et al., 2023), language generation (Paolini et al., 2021; Hsu et al., 2022), querying and extracting (Wang et al., 2022), pre-training (Wang et al., 2021), and multi-tasking (Lu et al., 2022; Wang et al., 2023b). Some works focus on zero-shot or few-shot settings (Huang et al., 2018; Hsu et al., 2022). Event detection. There are many prior studies focusing on extracting triggers only. Most works pay attention to the standard supervised setting (Liu et al., 2018; Lai et al., 2020; Veyseh et al., 2021; Li et al., 2021a; Huang et al., 2022a; Liu et al., 2022a; Li et al., 2023b). Some others study the few-shot setting (Deng et al., 2021; Zhao et al., 2022; Zhang et al., 2022; Ma et al., 2023; Wang et al., 2023a) Event argument extraction. Event argument extraction has caught much attention in recent years (Veyseh et al., 2022b; Li et al., 2021b; Hsu et al., 2023a; Zeng et al., 2022; Ma et al., 2022; Huang et al., 2022b; Xu et al., 2022; Hsu et al., 2023b; Nguyen et al., 2023; He et al., 2023; Huang et al., 2023; Parekh et al., 2024a). Some works focus on training models with only a few examples (Sainz et al., 2022a; Yang et al., 2023; Wang et al., 2023c). Event extraction datasets. Most of event extraction datasets come from Wikipedia and the news domain (Sundheim, 1992; Doddington et al., 2004; Song et al., 2015; Ebner et al., 2020; Li et al., 2020b, 2021b; Veyseh et al., 2022a; Li et al., 2022). To increase the event type coverage, some works focus on general domain datasets (Wang et al., 2020; Deng et al., 2020; Parekh et al., 2023; Li et al., 2023b). Recently, datasets in specific domains have been proposed, including cybersecurity (Satyapanich et al., 2020; Trong et al., 2020), pharmacovigilance (Sun et al., 2022), epidemic (Parekh et al., 2024b), and historical text (Lai et al., 2021). Event extraction evaluation and analysis. Re-
cently, some works point out several pitfalls when training event extraction models and attempt to provide solutions (Zheng et al., 2021; Peng et al., 2023a,b). Our observation partially echos their findings, while our proposed TEXTEE covers more diverse datasets and includes more recent approaches. On the other hand, some studies discuss ChatGPT’s performance on event extraction but only for one dataset (Li et al., 2023a; Gao et al., 2023).

heading: 3 Issues in Past Evaluation
text: Despite a wide range of works in EE, we identify several major issues of the past evaluation. We classify those issues into three categories: inconsistency, insufficiency, and low reproducibility. Inconsistency. Due to the lack of a standardized evaluation framework, we notice that many studies utilize varied experimental setups while comparing their results with reported numbers in the literature. This leads to unfair comparisons and makes the evaluation less reliable and persuasive. We identify and summarize the underlying reasons as follows: • Different assumptions about data. In the past,
different approaches tend to have their own assumptions about data. For instance, some works allow trigger spans consisting of multiple words (Lin et al., 2020; Hsu et al., 2022, 2023a), whereas others consider only single-word triggers (Liu et al., 2020; Du and Cardie, 2020; Wang et al., 2022); some studies assume that there are no overlapping argument spans (Zhang and Ji, 2021), while others can handle overlapping spans (Wadden et al., 2019; Huang et al., 2022b); some methods filter out testing data when the texts are too long (Liu et al., 2022a), while others do not (Hsu et al., 2023b; Ma et al., 2022). Due to these discrepancies in assumptions, the reported numbers from the original papers are actually not directly comparable.
• Different data preprocessing steps. Many previous works benchmark on the ACE05 (Doddington et al., 2004) and RichERE (Song et al., 2015) datasets. Since these datasets are behind a paywall and not publicly accessible, people can only share the data preprocessing scripts. Unfortunately, we observe that some popular preprocessing scripts can generate very different data. For instance, the processed ACE05 datasets from Wadden et al. (2019), Li et al. (2020a), and Veyseh et al. (2022b) have varying numbers of role types (22, 36, and 35 respec-
tively). In addition, it is crucial to note that variations in Python package versions can lead to different generated data even when using the same script. For example, different versions of nltk packages may have discrepancies in sentence tokenization and word tokenization, resulting in different processed data. Such differences in preprocessing largely affect model evaluation, leading to significant discrepancies (e.g., over 4 F1 score), thereby reducing persuasiveness (Peng et al., 2023b).
• Different external resources. We notice that many approaches utilize additional resources without clearly describing the differences in experimental settings. For example, Wang et al. (2023a) employs part-of-speech tags for event detection; Sainz et al. (2022b) and Wang et al. (2022) consider gold entity annotations for event argument extraction. These setting differences can lead to potentially unfair comparisons.
Insufficiency. We argue that the existing evaluation process used by the majority of approaches cannot thoroughly evaluate the capabilities of event extraction models due to the following aspects: • Limited dataset coverage. Early works usu-
ally utilize ACE05 (Doddington et al., 2004) and RichERE (Song et al., 2015) as the evaluation datasets. Consequently, most follow-up works adopt the same two datasets for comparison regardless that several new datasets across diverse domains are proposed (Li et al., 2021b; Sun et al., 2022; Tong et al., 2022; Parekh et al., 2023). The limited dataset coverage may introduce domain bias and lead to biased evaluations.
• Data split bias. Although many works address model randomness by averaging multiple experimental runs (Zhang and Ji, 2021; Hsu et al., 2022; Wang et al., 2022), they often overlook randomness in data splits and report numbers only for a single and fixed split for train, dev, and test sets. This can lead to a notable bias, especially for event extraction where there is a high variance of annotation density across sentences or documents. For example, following the preprocessing step of Wadden et al. (2019) applied to ACE05, the resulting processed dataset has 33 event types in the train set, 21 event types in the dev set, and 31 event types in the test set. Accordingly, it is likely to have a significant performance discrepancy between the dev and the test set, making the reported numbers biased.
Low reproducibility. Because of the complex nature of event extraction tasks, the event extraction models have become increasingly complicated. Releasing code and checkpoints for reproducing results has become essential, as many details and tricks need to be taken into account during the reimplementation process. However, many promising approaches do not provide an official codebase (Li et al., 2020a; Nguyen et al., 2021; Wei et al., 2021; Liu et al., 2022b), which potentially impedes the progress of research in event extraction.

heading: 4 Benchmark and Reevaluation
text: To address the issues listed in Section 3, we present TEXTEE, a framework aiming to standardize and benchmark the evaluation process of event extraction. TEXTEE has several advantages as follows. Better Consistency. We propose a standardized experimental setup for fair comparisons. • Normalizing assumptions about data. We
adopt the loosest assumption about data to align with real-world cases effectively. This includes allowing multiple-word triggers, considering overlapping argument spans, and retaining all instances without filtering. • Standardizing data preprocessing steps. We provide a standard script for data preprocessing, including tokenization and label offset mapping. To avoid the difference caused by variations in Python package versions, we use stanza 1.5.0 for tokenization and save all the offsets. Our script will load the saved offsets during preprocessing, ensuring that everyone can generate exactly the same data.
• Specifying additional resources. We clearly specify the resources utilized by all baselines (Table 2). For approaches that require additional gold annotations (such as POS tags, AMR, and gold entities), considering the purpose of fair comparisons, we either train a new predictor from training annotations (for entities) or use a pre-trained model (for POS tags and AMR), and consider the predicted labels as a substitute for the gold annotations.
Improved Sufficiency. We improve the sufficiency of the evaluation process as follows. • Increasing dataset coverage. As listed in Ta-
ble 1, we increase the dataset coverage by including sixteen event extraction datasets that cover various domains.
• Providing standard data splits. For each dataset, we merge all the labeled data and regenerate data splits. To mitigate the data split bias, we offer five split for each dataset and report the average results. To reduce the distribution gap among the train, dev, and test sets, we select splits that these sets share the most similar statistics, such as the number of event types and role types, as well as the number of events and arguments. Appendix A lists the detailed statistics of each split for each dataset.
• New evaluation metrics. Most prior works follow Lin et al. (2020) and consider Trigger F1-score and Argument F1-score as the evaluation metrics. Specifically, they calculate F1scores regarding the following: (1) TI: if the (start_idx, end_idx) of a predicted trigger match the gold ones. (2) TC: if the (start_idx, end_idx,
event_type) of a predicted trigger match the gold ones. (3) AI: if the (start_idx, end_idx, event_type) of a predicted argument match the gold ones. (4) AC: if the (start_idx, end_idx, event_type, role_type) of a predicted argument match the gold ones. However, we notice that AI and AC cannot precisely evaluate the quality of predicted arguments. There can be multiple triggers sharing the same event type in an instance, but the current score does not evaluate if the predicted argument attaches to the correct trigger. Accordingly, we propose two new scores to evaluate this attachment: (5) AI+: if the (start_idx, end_idx, event_type, attached_trigger_offsets) of a predicted argument match the gold ones. (6) AC+: if the (start_idx, end_idx, event_type, attached_trigger_offsets, role_type) of a predicted argument match the gold ones.
Reproducibility. We open-source the proposed TEXTEE framework for better reproducibility. Additionally, we encourage the community to contribute their datasets and codebases to advance the research in event extraction.

heading: 4.1 TEXTEE Benchmark
text: TEXTEE supports 16 datasets across various domains and 14 models proposed in recent years. Dataset. In addition to the two most common datasets, ACE05 (Doddington et al., 2004) and RichERE (Song et al., 2015), which particularly focus on the news domain, we consider as many other event extraction datasets across diverse domains as possible, including MLEE (Pyysalo et al., 2012), Genia2011 (Kim et al., 2011), and Genia2013 (Kim et al., 2013) from the biomedical domain, CASIE (Satyapanich et al., 2020) from the cybersecurity domain, PHEE (Sun et al., 2022) from the pharmacovigilance domain, SPEED (Parekh et al., 2024b) from the epidemic domain, M2E2 (Li et al., 2020b), MUC-4 (Sundheim, 1992), and RAMS (Ebner et al., 2020) from the news domain, MEE (Veyseh et al., 2022a) and WikiEvents (Li et al., 2021b) from Wikipedia, MAVEN (Wang et al., 2020), FewEvent (Deng et al., 2020), and GENEVA (Parekh et al., 2023) from the general domain. We also notice that there are other valuable datasets, such as GLEN (Li et al., 2023b) and VOANews (Li et al., 2022), but we do not include them as their training examples are not all annotated by humans. Table 1 summarizes the statistics for each dataset after our preprocessing steps. Appendix A describes the details of the preprocessing steps and our assumptions. Models. We do our best to aggregate as many models as possible into TEXTEE. For those works having public codebases, we adapt their code to fit our evaluation framework. We also re-implement some models based on the description from the original papers. Currently, TEXTEE supports the following models: (1) Joint training models that train ED and EAE together in an end-to-end manner, including DyGIE (Wadden et al., 2019), OneIE (Lin et al., 2020), and AMR-IE (Zhang and Ji, 2021). (2) Classification-based models that formu-
late the event extraction task as a token classification problem, a sequential labeling problem, or a question answering problem, including EEQA (Du and Cardie, 2020), RCEE (Liu et al., 2020), Query&Extract (Wang et al., 2022), TagPrime (Hsu et al., 2023a), UniST (Huang et al., 2022a), and CEDAR (Li et al., 2023b). (3) Generationbased models that convert the event extraction task to a conditional generation problem, including DEGREE (Hsu et al., 2022), BART-Gen (Li et al., 2021b), X-Gear (Huang et al., 2022b), PAIE (Ma et al., 2022), and AMPERE (Hsu et al., 2023b). Table 2 presents the different assumptions and requirements for each model. It is worth noting that some models need additional annotations or information, as indicated in the table. Appendix B lists
more details about implementations. Reevalutation results. For a fair comparison, we utilize RoBERTa-large (Liu et al., 2019) for all the classification-based models and use BART-large (Lewis et al., 2020) for all the generation-based models. Table 3, 4, and 5 present the reevaluation results of end-to-end EE, ED, and EAE, respectively. Appendix C lists more detailed results. We first notice that for end-to-end EE and ED, there is no obvious dominant approach. It suggests that the reported improvements from previous studies may be influenced by dataset bias, data split bias, or data processing. This verifies the importance of a comprehensive evaluation framework that covers various domains of datasets and standardized data splits. TagPrime (Hsu et al., 2023a) and PAIE (Ma et al., 2022) seem to be the two dominant approaches across different types of datasets for EAE. These results validate the effectiveness of those two models, aligning with our expectations for guiding reliable and reproducible research in event extraction with TEXTEE.
In addition, we observe a gap between the established evaluation metrics (AI and AC) and the proposed ones (AI+ and AC+). This implies a potential mismatch between the earlier metrics and
the predictive quality. We strongly recommend reporting the attaching score (AI+ and AC+) for future research in event extraction to provide a more accurate assessment of performance.

heading: 5 Have LLMs Solved Event Extraction?
text: Given the demonstrated potential of large language models (LLMs) across various NLP tasks, we discuss their capability in solving event extraction tasks. In contrast to previous studies (Li et al., 2023a; Gao et al., 2023), which evaluate a single LLM on a single EE dataset, we investigate multiple popular LLMs across multiple datasets provided by TEXTEE. We consider GPT3.5-Turbo as well as some open-source LLMs that achieve strong performance on Chatbot Arena (Zheng et al., 2023)3, including Llama-2-13bchat-hf and Llama-2-70b-chat-hf (Touvron et al., 2023), Zephyr-7b-alpha (Tunstall et al., 2023), and Mixtral-8x7B-Instruct (Jiang et al., 2024), with vLLM framework (Kwon et al., 2023). We evaluate them on the pipelined tasks of event detection (ED) and event argument extraction (EAE). As part of the prompt, we provide LLMs with the
3https://leaderboard.lmsys.org
task instructions, a few demonstration examples (positive and negative ones), and the query text. It is worth noting that the number of demonstration examples will be limited by the maximum length supported by LLMs. Appendix D illustrates the best prompt we use. Results. Due to the cost and time of running LLMs, we evaluate only on sampled 250 documents for each dataset. Table 6 and 7 list the average results of LLMs as well as some well-performed models selected from TEXTEE.4 Unlike other NLP tasks such as named entity recognition and commonsense knowledge, where LLMs can achieve competitive performance with fine-tuning models using only a few in-context demonstrations (Wei et al., 2022; Qin et al., 2023), it is noteworthy that there is a large gap between LLMs and the baselines for both the ED and EAE tasks. Our hypothesis is that event extraction requires more recognition of abstract concepts and relations, which is harder compared to other NLP tasks (Li et al., 2023a).

heading: 5.1 Analysis
text: We also manually examine the cases where LLMs make mistakes. The major errors of LLMs can be categorized into the following three cases, suggesting that there is still room for improving LLMs’ performance. Overly aggressive predictions. We observed that
4The results do not include SPEED and MUC-4.
LLMs struggle to accurately capture the concept of certain event types solely from in-context examples, leading to a tendency to generate many false positives. For instance, considering the following input:
Alleged ties to Gulen-In a statement to the United Nations on May 15, the legal Christian advocacy group, American Center for Law and Justice (ACLJ), said Brunson was told that he was being detained as a "national security risk”.
LLMs would predict detained as the trigger word for several event types, Conflict-Attack, Life-Die, Movement-Transport, and Justice-Arrest-Jail, while the correct event type is only Justice-ArrestJail. This reveals that LLMs might rely heavily on the format of the in-context examples to generate output, rather than fully understanding the semantics of the event types. Imprecise span boundaries. We find that another key challenge of generation-based models is to predict accurate offsets. For example, considering the following input:
In 1988 , Spain supplied Iran with 200,000 respirators.
LLMs would identify respirators as the argument of role Theme, while the ground truth argument is 200,000 respirators. Hallucination or paraphrasing. We also notice that LLMs may generate spans that are not present
in the input text. Most of the time, this can be detected by a post-processing script to filter out invalid predictions. However, in some cases, LLMs generate reasonable answers but in different textual formats, such as predicting Los Angeles when the ground truth is LA. The current evaluation pipeline would count this as an error.

heading: 6 Future Challenges and Opportunities
text: In this section, we discuss the role of event extraction in the current NLP era, as well as some challenges and insights derived from TEXTEE. How should we position event extraction in the era of LLMs? Based on the findings in Section 5, LLMs struggle with extracting and comprehending complicated structured semantic concepts. This indicates the need for a dedicated system with specialized design to effectively recognize and extract abstract concepts and relations from texts. We believe that a good event extractor, capable of identifying a wide range of events, could serve as a tool that provides grounded structured information about texts for LLMs. Accordingly, LLMs can flexibly decide whether they require this information for the following reasoning steps or inference process. To achieve this goal, we expect event extractors to be universal, efficient, and accurate, which introduces the following research challenges. Broader event coverage and generalizablity. We anticipate that a strong event extractor can recognize a wide range of events and even identify new event concepts that may not have appeared during training. This requires two efforts: (1) Expanding domain coverage in datasets. Most existing event extraction datasets suffer from a restricted coverage of event types. For instance, all the datasets incorporated by TEXTEE have no more than 200 event types, which is significantly below the amount of human concepts encountered in daily life. Although some recent studies have attempted to tackle this issue (Li et al., 2023b), their data often contains label noise and lacks detailed role annotations. We believe that efficiently collecting or synthesizing high-quality data that covers a wild range of events is crucial for enhancing the emerging ability to generalize event recognition. (2) Better model design for generalization. Most existing event extraction models focus on in-domain performance. Therefore, their design can fail when encountering novel events. While exploring prompting in LLMs shows promise, as discussed in Section 5, the re-
sults remain unsatisfactory. Some recent works (Lu et al., 2022; Ping et al., 2023) explore learning a unified model across multiple information extraction tasks for improved generalization, but their integration is constrained by limited domains. We expect that TEXTEE can serve as a starting point for aggregating diverse datasets and training more robust unified models. Enhanced model efficiency. Inference time can pose a bottleneck for effective event extraction, especially when the number of event (role) types increases. For instance, well-performing methods in TEXTEE (e.g., TagPrime and PAIE) require enumerating all the event (role) types, resulting in multiple times of model inference, which significantly slows down as more events (roles) are considered. Similar challenges arise with LLMs, as we have to prompt them per event. Therefore, there is a critical necessity for model designs that not only prioritize performance but also optimize efficiency.

heading: 7 Conclusion
text: In this work, we identify and discuss several evaluation issues for event extraction, including inconsistent comparisons, insufficiency, and low reproducibility. To address these challenges, we propose TEXTEE, a consistent, sufficient, and reproducible benchmark for event extraction. We also study and benchmark the capability of five large language models in event extraction. Additionally, we discuss the role of event extraction in the current NLP era, as well as challenges and insights derived from TEXTEE. We expect TEXTEE and our reevaluation results will serve as a reliable benchmark for research in event extraction.

heading: Acknowledgements
text: We thank the anonymous reviewers for their constructive suggestions. We also thank UIUC BLENDER Lab, UCLA-NLP group, and UCLA PLUS Lab for the valuable discussions and comments. This research is based upon work supported by U.S. DARPA KAIROS Program No. FA875019-2-1004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Limitations
In this work, we make efforts to incorporate as many event extraction datasets as possible. However, for some datasets, it is hard for us to obtain the raw files. Moreover, there is a possibility that we may overlook some datasets. Similarly, we aim to include a broad range of event extraction approaches, but we acknowledge that it is not feasible to cover all works in the field. We do our best to consider representative methods that published in recent years. Additionally, for works without released codebases, we make efforts to reimplement their proposed methods based on the descriptions in the original papers. There can be discrepancies between our implementation and theirs due to differences in packages and undisclosed techniques. We will continue to maintain our proposed library and welcome contributions and updates from the community.

heading: A Details of Dataset Preprocessing
text: We describe the detailed preprocessing steps for each dataset in the following. Table 8 and 9 lists the statistics of each dataset.
ACE05-en (Doddington et al., 2004). We download the ACE05 dataset from LDC5 and consider the data in English. The original text in ACE05 dataset is document-based. We follow most prior usage of the dataset (Lin et al., 2020; Wadden et al., 2019) to split each document into sentences and making it a sentence-level benchmark on event extraction. We use Stanza (Qi et al., 2020) to perform sentence splitting and discard any label (entity mention, relation mention, event arguments, etc.) where its span is not within a single sentence. Similar to prior works (Lin et al., 2020; Wadden et al., 2019), we consider using head span to represent entity mentions and only include event arguments that are entities (i.e., remove time and values in the ACE05 annotation). The original annotation of the dataset is character-level. However, to make the dataset consistent with others, we perform tokenization through Stanza and map the character-level annotation into token-level. We split the train, dev, and test sets based on documents with the ratio 80%, 10%, and 10%.
RichERE (Song et al., 2015). Considering the unavailability of the RichERE dataset used in prior works (Lin et al., 2020; Hsu et al., 2022), we download the latest RichERE dataset from LDC6 and only consider the 288 documents labeled with Rich ERE annotations. Similar to the pre-processing step in ACE05-en, we use Stanza (Qi et al., 2020) to perform sentence splitting and making it a sentencelevel benchmark. Following the strategy in (Lin et al., 2020), we use head span to represent entity mentions and only consider named entities, weapons and vehicles as event argument candidates. Again, the original annotation of the dataset is character-level, and we perform tokenization through Stanza and map the annotation into tokenlevel, forming the final RichERE dataset we use. We split the train, dev, and test sets based on documents with the ratio 80%, 10%, and 10%.
5https://catalog.ldc.upenn.edu/ LDC2006T06
6https://catalog.ldc.upenn.edu/ LDC2023T04
MLEE (Pyysalo et al., 2012). The original MLEE dataset is document-level.7 We use Stanza (Qi et al., 2020) to do the sentence tokenization and the word tokenization. For the purpose of evaluating most baselines, we divide the documents into several segment-level instances with a sub-token window size being 480 based on the RoBERTalarge tokenizer (Liu et al., 2019). We split the train, dev, and test sets based on documents with the ratio 70%, 15%, and 15%.
Genia2011 (Kim et al., 2011). The original Genia2011 dataset is document-level.8 We use Stanza (Qi et al., 2020) to do the sentence tokenization and the word tokenization. For the purpose of evaluating most baselines, we divide the documents into several segment-level instances with a sub-token window size being 480 based on the RoBERTalarge tokenizer (Liu et al., 2019). We split the train, dev, and test sets based on documents with the ratio 60%, 20%, and 20%.
Genia2013 (Kim et al., 2013). The original Genia2013 dataset is document-level.9 We use Stanza (Qi et al., 2020) to do the sentence tokenization and the word tokenization. For the purpose of evaluating most baselines, we divide the documents into several segment-level instances with a sub-token window size being 480 based on the RoBERTalarge tokenizer (Liu et al., 2019). We split the train, dev, and test sets based on documents with the ratio 60%, 20%, and 20%.
M2E2 (Li et al., 2020b). The M2E2 dataset contains event argument annotations from both texts and images.10 We consider only the text annotations in our benchmark. We directly use the tokenized words without any modifications. We merge the original train, dev, and test sets, and split them into the new train, dev, and test sets based on documents with the ratio 70%, 15%, and 15%.
CASIE (Satyapanich et al., 2020). The original CASIE dataset is document-level.11 We use Stanza (Qi et al., 2020) to do the sentence tokenization and the word tokenization. For the purpose of evaluating most baselines, we divide the documents into
7https://www.nactem.ac.uk/MLEE/ 8https://bionlp-st.dbcls.jp/GE/2011/
downloads/ 9https://2013.bionlp-st.org/tasks/
10https://blender.cs.illinois.edu/ software/m2e2
11https://github.com/Ebiquity/CASIE
several segment-level instances with a sub-token window size being 480 based on the RoBERTalarge tokenizer (Liu et al., 2019). We split the train, dev, and test sets based on documents with the ratio 70%, 15%, and 15%.
PHEE (Sun et al., 2022). We download the PHEE dataset from the official webpage.12 We directly use the tokenized words without any modifications. We merge the original train, dev, and test sets, and split them into the new train, dev, and test sets based on documents with the ratio 60%, 20%, and 20%.
MAVEN (Wang et al., 2020). We consider the sentence-level annotations from the original data.13
We directly use the tokenized words without any modifications. Because the labels of the original test set are not publicly accessible, we merge the original train and dev sets and split it into new train, dev, and test sets by documents with the ratio 70%, 15%, and 15%.
MEE-en (Veyseh et al., 2022a). We download the MEE dataset14 and consider the English annotations. We use the annotations for event detection only because we observe that the quality of the annotations for event argument extraction is not good and many important arguments are actually missing. We directly use the tokenized words without any modifications. We merge the original train, dev, and test sets, and split them into the new train, dev, and test sets based on documents with the ratio 80%, 10%, and 10%.
FewEvent (Deng et al., 2020). We download the FewEvent dataset from the official webpage.15 Notice that we consider FewEvent as a normal supervised event detection dataset. We use Stanza (Qi et al., 2020) to do the word tokenization. For the purpose of evaluating most baselines, we discard the instances with the length longer than 300. We split the train, dev, and test sets based on documents with the ratio 60%, 20%, and 20%.
SPEED (Parekh et al., 2024b). We download the SPEED dataset from the official webpage.16
12https://github.com/ZhaoyueSun/PHEE 13https://github.com/THU-KEG/
MAVEN-dataset 14http://nlp.uoregon.edu/download/MEE/ MEE.zip 15https://github.com/231sm/Low_ Resource_KBP 16https://github.com/PlusLabNLP/SPEED
Notice that we consider only the COVID-related examples. We split the train, dev, and test sets based on documents with the ratio 60%, 20%, and 20%.
RAMS (Ebner et al., 2020). We use the latest version of the RAMS dataset.17 We directly use the tokenized words without any modifications. For the purpose of evaluating most baselines, we discard the instances with the sub-token length larger than 500 based on the RoBERTa-large tokenizer (Liu et al., 2019). We merge the original train, dev, and test sets, and split them into the new train, dev, and test sets based on documents with the ratio 80%, 10%, and 10%.
WikiEvents (Li et al., 2021b). We download the WikiEvents dataset from the official webpage.18
We directly use the tokenized words without any modifications. For the purpose of evaluating most baselines, we divide the documents into several segment-level instances with a sub-token window size being 480 based on the RoBERTa-large tokenizer (Liu et al., 2019). We split the train, dev, and test sets based on documents with the ratio 80%, 10%, and 10%.
MUC-4 (Sundheim, 1992). We use the preprocessed data from the GRIT repository.19 We use Stanza (Qi et al., 2020) to do the sentence tokenization and the word tokenization. For the purpose of evaluating most baselines, we divide the documents into several segment-level instances with a sub-token window size being 480 based on the RoBERTa-large tokenizer (Liu et al., 2019). We split the train, dev, and test sets based on documents with the ratio 60%, 20%, and 20%.
GENEVA (Parekh et al., 2023). We download the GENEVA dataset from the officail webpage.20
We directly use the tokenized words without any modifications. We split the train, dev, and test sets based on documents with the ratio 70%, 15%, and 15%.

heading: B Details of Model Implementations
text: We utilize RoBERTa-large (Liu et al., 2019) for all the classification-based models and use BART-
17https://nlp.jhu.edu/rams/RAMS_1.0c. tar.gz
18s3://gen-arg-data/wikievents/ 19https://github.com/xinyadu/grit_doc_ event_entity/ 20https://github.com/PlusLabNLP/GENEVA
large (Lewis et al., 2020) for all the generationbased models to have a consistent comparison.
DyGIE++ (Wadden et al., 2019). We reimplement the model based on the original codebase.21
OneIE (Lin et al., 2020). We adapt the code from the original codebase.22
AMR-IE (Zhang and Ji, 2021). We adapt the code from the original codebase.23
EEQA (Du and Cardie, 2020). We reimplement the model based on the original
21https://github.com/dwadden/dygiepp 22https://blender.cs.illinois.edu/
software/oneie/ 23https://github.com/zhangzx-uiuc/ AMR-IE
codebase.24 Notice that EEQA requires some human-written queries for making predictions. For those datasets that EEQA provides queries, we directly use those queries. For other datasets, we follow the suggestion from the paper and use “arg” style queries like “{role_name} in [Trigger]”.
RCEE (Liu et al., 2020). We re-implement the model based on the description in the original paper. Notice that RCEE requires a question generator to generate queries for making predictions. Alternatively, we re-use the queries from EEQA as the generated queries.
24https://github.com/xinyadu/eeqa
Query&Extract (Wang et al., 2022). We adapt the code from the original codebase.25 We use the event type names as the verbalized string for each event. Since the origin model supports event argument role labeling rather than event argument extraction, we learn an additional NER sequential labeling model during training and use the predicted entities for event argument role labeling during testing.
TagPrime (Hsu et al., 2023a). We adapt the code from the original codebase.26
PAIE (Ma et al., 2022). We adapt the code from the original codebase.27 Notice that PAIE
25https://github.com/VT-NLP/Event_ Query_Extract
26https://github.com/PlusLabNLP/ TagPrime
27https://github.com/mayubo2333/PAIE
requires some human-written templates for making predictions. For those datasets that PAIE provides templates, we directly use them. For other datasets, we create automated templates like “{role_1_name} [argument_1] {role_2_name} [argument_2] ... {role_k_name} [argument_k] ”.
DEGREE (Hsu et al., 2022). We adapt the code from the original codebase.28 Notice that DEGREE requires some human-written templates for making predictions. For those datasets that DEGREE provides templates, we directly use them. For other datasets, we re-use the templates generated by PAIE.
28https://github.com/PlusLabNLP/DEGREE
BART-Gen (Li et al., 2021b). We re-implement the model from the original codebase.29 We replace the original pure copy mechanism with a copy-generator since we observe this works better. Notice that BART-Gen requires some humanwritten templates for making predictions. For those datasets that BART-Gen provides templates, we directly use them. For other datasets, we re-use the templates generated by PAIE.
X-Gear (Huang et al., 2022b). We adapt the code from the original codebase.30
AMPERE (Hsu et al., 2023b). We adapt the code from the original codebase.31 Notice that AMPERE requires some human-written templates for making predictions. For those datasets that AMPERE provides templates, we directly use them. For other datasets, we re-use the templates generated by PAIE.
UniST (Huang et al., 2022a). We re-implement the model from the original codebase.32 Since the origin model supports semantic typing only, we learn an additional span recognition model during training and use the predicted trigger spans for trigger span typing during testing.
CEDAR (Li et al., 2023b). We re-implement the model from the original codebase.33 Notice that in the original paper, they consider self-labeling during training as the dataset they consider is noisy. Our implementation currently ignores the self-labeling part.

heading: C Detailed Results
text: Table 10, 11, 12 demonstrate the detailed reevaluation results for end-to-end event extraction, event detection, and event argument extraction, respectively.

heading: D Prompts for LLMs
text: Table 13 illustrates the prompts we use for testing the ability of LLMs in event detection and event argument extraction.
29https://github.com/raspberryice/ gen-arg
30https://github.com/PlusLabNLP/X-Gear 31https://github.com/PlusLabNLP/AMPERE 32https://github.com/luka-group/unist 33https://github.com/ZQS1943/GLEN

heading: Future Work
text: We strongly recommend reporting the attaching score (AI+ and AC+) for future research in event extraction to provide a more accurate assessment of performance