The Devil is in the Details: On the Pitfalls of Event Extraction Evaluation
Year: 2023

Abstract:
Event extraction (EE) is a crucial task aiming at extracting events from texts, which includes two subtasks: event detection (ED) and event argument extraction (EAE). In this paper, we check the reliability of EE evaluations and identify three major pitfalls: (1) The data preprocessing discrepancy makes the evaluation results on the same dataset not directly comparable, but the data preprocessing details are not widely noted and specified in papers. (2) The output space discrepancy of different model paradigms makes differentparadigm EE models lack grounds for comparison and also leads to unclear mapping issues between predictions and annotations. (3) The absence of pipeline evaluation of many EAE-only works makes them hard to be directly compared with EE works and may not well reflect the model performance in realworld pipeline scenarios. We demonstrate the significant influence of these pitfalls through comprehensive meta-analyses of recent papers and empirical experiments. To avoid these pitfalls, we suggest a series of remedies, including specifying data preprocessing, standardizing outputs, and providing pipeline evaluation results. To help implement these remedies, we develop a consistent evaluation framework OMNIEVENT, which can be obtained from https://github.com/THU-KEG/OmniEvent.

heading: 1 Introduction
text: Event extraction (EE) is a fundamental information extraction task aiming at extracting structural event knowledge from plain texts. As illustrated in Figure 1, it is typically formalized as a two-stage pipeline (Ahn, 2006). The first subtask, event detection (ED), is to detect the event triggers (keywords or phrases evoking events, e.g., quitting in Figure 1) and classify their event types (e.g., End-Position). The second subtask, event argument extraction
∗ Equal contribution. Random Order. † Corresponding author: J.Li
Elon Musk posted on Twitter about quitting as
Chief Executive of Twitter.
Plain Text Event Trigger & Type Event Argument & Role
Elon Musk
Twitter Person
Company
End-Position End-position
quitting quitting
(EAE), is to extract corresponding event arguments and their roles (e.g., Elon Musk and its argument role Person) based on the first-stage ED results.
Since events play an important role in human language understanding and broad applications benefit from structural event knowledge (Ji and Grishman, 2011; Glavaš and Šnajder, 2014; Hogenboom et al., 2016; Zhang et al., 2020a), EE has attracted much research attention, and novel models have been continually developed. Beyond the conventional paradigms like classification (Chen et al., 2015; Wang et al., 2021) and sequence labeling (Nguyen et al., 2016; Chen et al., 2018), new model paradigms such as span prediction (Liu et al., 2020a; Du and Cardie, 2020b) and conditional generation (Lu et al., 2021; Li et al., 2021b) are proposed. These sophisticated models push evaluation results to increasingly high levels.
However, due to the complex input/output formats and task pipeline of EE, there are some hidden pitfalls in EE evaluations, which are rarely noted and discussed in EE papers (Wadden et al., 2019; Wang et al., 2020, 2022). These pitfalls make many competing EE methods actually lack grounds for comparison, and the reported scores cannot reflect real-world model performances well.
In this paper, we summarize three major pitfalls: (1) Data preprocessing discrepancy. If two EE works conduct evaluations on the same dataset but adopt different preprocessing methods, their results are not directly comparable. Since EE datasets have complex data formats (involving multiple heteroge-
9206
neous elements including event triggers, arguments, entities, temporal expressions, etc.), data preprocessing methods of existing works often disagree on some design choices, like whether to include multi-token triggers, which results in major data discrepancies. For instance, for the widely-used English subset of ACE 2005 (Walker et al., 2006), the preprocessing of Wadden et al. (2019) gets 5, 055 event triggers, but Wang et al. (2021) have 5, 349. (2) Output space discrepancy. Different model paradigms have inconsistent output spaces, which makes the evaluation metrics of different-paradigm models often not calculated on the same bases. For example, the phrase Elon Musk is one argument candidate in the output space of conventional classification-based methods, and it is regarded as one error case when the model misclassifies it. But other model paradigms, like the sequence labeling, have more free output formats and can make two independent predictions for the two tokens Elon and Musk, which will account for two error cases in the evaluation metric calculation. Larger output spaces of the new model paradigms also result in unclear mappings between predictions and annotations in some cases, which are often overlooked in EE evaluation implementations and lead to problematic results. These details are presented in § 3.3. (3) Absence of pipeline evaluation. Recent works handling only the EAE subtask often evaluate the performances based on gold event triggers (Subburathinam et al., 2019; Xi et al., 2021; Ma et al., 2022). In contrast, conventional EE works often conduct pipeline evaluation, i.e., evaluate EAE performances based on triggers predicted at the ED stage. The absence of pipeline evaluation makes these EAE-only works hard to be directly compared with EE works. This has discouraged the research community from considering all the EE subareas in a holistic view. Moreover, only using gold triggers in evaluation cannot evaluate EAE models’ resistance to the noise of predicted triggers, which is important in real-world application scenarios.
We conduct systematic meta-analyses of EE papers and empirical experiments, demonstrating the pitfalls’ broad and significant influence. We suggest a series of remedies to avoid these pitfalls, including specifying data preprocessing methods, standardizing outputs, and providing pipeline evaluation results. To help conveniently achieve these remedies, we develop a consistent evaluation framework, OMNIEVENT, which contains implementa-
tions for data preprocessing and output standardization, and off-the-shelf predicted triggers on widelyused datasets for easier pipeline evaluation.
To summarize, our contributions are two-fold: (1) We systematically analyze the inconspicuous pitfalls of EE evaluations and demonstrate their significant influence with meta-analyses and experiments. (2) We propose corresponding remedies to avoid the pitfalls and develop a consistent evaluation framework to help implement them.

heading: 2 Related Work
text: Traditional methods (Ji and Grishman, 2008; Gupta and Ji, 2009; Hong et al., 2011; Li et al., 2013) rely on human-crafted features and rules to extract events. Most modern EE models automate feature learning with neural networks (Nguyen and Grishman, 2015; Nguyen et al., 2016; Nguyen and Grishman, 2018) and adopt different model paradigms to model the EE task. The most common classification-based methods view EE as classifying given trigger and argument candidates into different labels (Chen et al., 2015; Feng et al., 2016; Chen et al., 2017; Liu et al., 2018b; Wang et al., 2019a; Lai et al., 2020; Wang et al., 2021, 2022). Sequence labeling methods (Nguyen et al., 2016; Chen et al., 2018; Araki and Mitamura, 2018; Ding et al., 2019; Ma et al., 2020; Nguyen et al., 2021; Guzman-Nateras et al., 2022) do EE by labeling every word following a certain tagging schema such as BIO (Ramshaw and Marcus, 1995). Recently, some works (Du and Cardie, 2020b; Li et al., 2020a; Liu et al., 2020a, 2021b; Wei et al., 2021; Sheng et al., 2021; Zhou et al., 2022) propose to cast the task formalization of EE into resource-rich machine reading comprehension tasks and adopt the span prediction paradigm to predict the starting and ending positions of event trigger and argument spans. With the development of generative pre-trained language models (Lewis et al., 2020; Raffel et al., 2020; Brown et al., 2020), there have been works (Lu et al., 2021; Xi et al., 2021; Li et al., 2021b, 2022a; Liu et al., 2022c; Huang et al., 2022; Du et al., 2022; Hsu et al., 2022; Zeng et al., 2022) exploring the conditional generation paradigm to generate sequences indicating EE results.
A few previous works (Wadden et al., 2019; Lai et al., 2020; Wang et al., 2020, 2022) have noted that data preprocessing discrepancy may influence evaluation results, but they did not especially study its impact with in-depth analyses. To the best of
our knowledge, we are the first to study all three kinds of pitfalls of EE evaluation and propose comprehensive remedies for them.

heading: 3 Pitfalls of Event Extraction Evaluation
text: We first introduce our investigation setup for metaanalysis and empirical analysis (§ 3.1). Then we analyze the three pitfalls: data preprocessing discrepancy (§ 3.2), output space discrepancy (§ 3.3), and absence of pipeline evaluation (§ 3.4).

heading: 3.1 Investigation Setup
text: We adopt the following two investigation methods to analyze the influence of the observed pitfalls.
Meta-Analysis To comprehensively understand the research status and investigate the potential influence of the evaluation pitfalls, we analyze a broad range of recent EE studies in the metaanalysis. Specifically, we manually retrieve all published papers concerning EE, ED, and EAE tasks at four prestigious venues from 2015 to 2022 via keyword1 matching and manual topic rechecking by the authors. The complete paper list is shown in appendix C, including 44 at ACL, 39 at EMNLP, 19 at NAACL, and 14 at COLING.
We conduct statistical analyses of these papers and their released codes (if any) from multiple perspectives. These statistics will be presented to demonstrate the existence and influence of the pitfalls in the following sections, respectively.
Empirical Analysis In addition to the metaanalysis, we conduct empirical experiments to quantitatively analyze the pitfalls’ influence on EE evaluation results. We reproduce several representative models covering all four model paradigms mentioned in § 2 to systematically study the influence. Specifically, the models contain: (1) Classifcation methods, including DMCNN (Chen et al., 2015) , DMBERT (Wang et al., 2019a,b), and CLEVE (Wang et al., 2021). DMCNN and DMBERT adopt a dynamic multi-pooling operation over hidden representations of convolutional neural networks and BERT (Devlin et al., 2019), respectively. CLEVE is an event-aware pre-trained model enhanced with event-oriented contrastive pre-training. (2) Sequence labeling methods, including BiLSTM+CRF (Wang et al., 2020) and BERT+CRF (Wang et al., 2020), which adopt the conditional random field (Lafferty et al., 2001)
1We use event and extraction as keywords for searching.
as the output layer to make structural predictions. (3) Span prediction methods, including EEQA (Du and Cardie, 2020b) converting EE into a question-answering task, and PAIE (Ma et al., 2022), which is a prompt-tuning-based EAE method. (4) Conditional generation method, including Text2Event (Lu et al., 2021), which is a sequence-to-structure generative EE method with constrained decoding and curriculum learning.
The models are reproduced based on the evaluation settings described in their original papers and released open-source codes (if any). From our meta-analysis, 70% of the EE papers adopt the English subset of ACE 2005 dataset (Walker et al., 2006)2 in their experiments. Hence we also adopt this most widely-used dataset in our empirical experiments to analyze the pitfalls without loss of generality. The reproduction performances are shown in Table 1. Following the conventional practice, we report precision (P), recall (R), and the F1 score. In the following analyses, we show the impact of three pitfalls by observing how the performances change after controlling the pitfalls’ influence.

heading: 3.2 Data Preprocessing Discrepancy
text: Due to the inherent task complexity, EE datasets naturally involve multiple heterogeneous annotation elements. For example, besides event triggers and arguments, EE datasets often annotate entities, temporal expressions, and other spans as argument candidates. The complex data format makes the data preprocessing methods easily differ in many details, which makes the reported results on the same dataset not directly comparable. However, this pitfall has not received extensive attention.
To carefully demonstrate the differences brought by data preprocessing discrepancy, we conduct de-
2For brevity, refer to as “ACE 2005” in the following.
tailed meta-analyses taking the most widely-used ACE 2005 as an example. From all the 116 surveyed papers, we find three repetitively used opensource preprocessing scripts: ACE-DYGIE (Wadden et al., 2019), ACE-OneIE (Lin et al., 2020), and ACE-Full (Wang et al., 2019b). In addition to these scripts, there are 6 other open-source preprocessing scripts that are only used once. The utilization rates and data statistics of the different preprocessing methods are shown in Table 2. From the statistics, we can observe that: (1) The data differences brought by preprocessing methods are significant. The differences mainly come from the different preprocessing implementation choices, as summarized in Table 3. For instance, ACE-DYGIE and ACE-OneIE ignore the annotated temporal expressions and values in ACE 2005, which results in 13 fewer argument roles compared to ACE-Full. Intuitively, the significant data discrepancy may result in inconsistent evaluation results. (2) Each preprocessing script has a certain utilization rate and the majority (63%) papers do not specify their preprocessing methods. The high preprocessing inconsistency and Unspecified rate both show that our community has not fully recognized the significance of the discrepancies resulting from differences in data preprocessing.
To further empirically investigate the influence of preprocessing, we conduct experiments on ACE 2005. Table 4 shows the F1 differences keeping all settings unchanged except for the preprocessing scripts. We can observe that the influence of different preprocessing methods is significant and varies from different models. It indicates that the evaluation results on the same dataset are not necessarily comparable due to the unexpectedly large influence of different preprocessing details.
Moreover, besides ACE 2005, there are also data preprocessing discrepancies in other datasets. For example, in addition to the implementation details, the data split of the KBP dataset is not always consistent (Li et al., 2021a, 2022a), and some used LDC3 datasets are not freely available, such as LDC2015E29. Based on all the above analyses, we suggest the community pay more attention to data discrepancies caused by preprocessing, and we propose corresponding remedies in § 4.1.

heading: 3.3 Output Space Discrepancy
text: As shown in Figure 3, the diversity of adopted model paradigms in EE studies has substantially increased in recent years. Figure 2 illustrates the different paradigms’ workflows in EAE scenario4. The paradigms inherently have very different output spaces, which results in inconspicuous pitfalls
3https://www.ldc.upenn.edu/ 4The ED workflows are the same as EAE or even simpler.
in the comparative evaluations across paradigms.
Inconsistent Output Spaces between Different Paradigms As shown in Figure 2, there are substantial differences between the model output spaces of different paradigms. CLS-paradigm models only output a unique label for each candidate in a pre-defined set. While models of SL and SP paradigms can make predictions for any consecutive spans in the input sequence. The output space of CG-paradigm models is even larger, as their vanilla5 output sequences are completely free, e.g., they can even involve tokens unseen in the input. The inconsistent output spaces make the evaluation metrics of different-paradigm models calculated on different bases and not directly comparable. For instance, when calculating the confusion matrices for the prediction as Chief Executive of in Fig-
5Indicates excluding tricks like vocabulary constraint, etc.
ure 2, the CLS paradigm takes it as one true positive (TP) and two false positives (FP), while the remaining paradigms only count it as one FP. The CLS paradigm may also have an advantage in some cases since it is constrained by the pre-defined candidate sets and cannot make illegal predictions as other paradigms may have.
Unclear Mappings between Predictions and Annotations Implementing the mappings between model predictions and dataset annotations is a key component for evaluation. The larger output spaces of SL, SP, and CG paradigms often produce unclear mappings, which are easily neglected in the EE evaluation implementations and influence the final metrics. As shown in Figure 2 (bottom right), we summarize three major unclear mapping issues: 1⃝ Prediction span overlaps the gold span. A prediction span of non-CLS paradigm models may overlap but not strictly align with the annotated span, bringing in an unclear implementation choice. As in Figure 2, it is unclear whether the predicted role Position for the span as Chief Executive of should be regarded as a correct prediction for the contained annotated span Chief Executive. 2⃝ Multiple predictions for one annotated span. If without special constraints, models of SP and CG paradigms may make multiple predictions for one span. Figure 2 presents two contradictory predictions (Company and Person) for the annotated span Elon Musk. To credit the correct one only or penalize both should lead to different evaluation
results. 3⃝ Predictions without positions for nonunique spans. Vanilla CG-paradigm models make predictions by generating contents without specifying their positions. When the predicted spans are non-unique in the inputs, it is unclear how to map them to annotated spans in different positions. As in Figure 2, the CG model outputs two Twitter predictions, which can be mapped to two different input spans.
To quantitatively demonstrate the influence of output space discrepancy, we conduct empirical experiments. Specifically, we propose an output standardization method (details in § 4.2), which unify the output spaces of different paradigms and handle all the unclear mapping issues. We report the changes in metrics between the original evaluation implementations and the evaluation with our output standardization in Table 5. We can see the results change obviously, with the maximum increase and decrease of +2.8 in ED precision and −3.5 in EAE recall, respectively. It indicates the output space discrepancy can lead to highly inconsistent evaluation results. Hence, we advocate for awareness of the output space discrepancy in evaluation implementations and suggest doing output standardization when comparing models using different paradigms.

heading: 3.4 Absence of Pipeline Evaluation
text: The event extraction (EE) task is typically formalized as a two-stage pipeline, i.e., first event detection (ED) and then event argument extraction (EAE). In real applications, EAE is based on ED and only extracts arguments for triggers detected by the ED model. Therefore, the conventional evaluation of EAE is based on predicted triggers and considers ED prediction errors, which we call pipeline evaluation. It assesses the overall performance of
an event extraction system and is consistent with real-world pipeline application scenarios.
However, as shown in Figure 4, more and more works have focused only on EAE in recent years. For convenience and setting a unified evaluation base between the EAE-only works, 95.45% of them only evaluate EAE taking gold triggers as inputs. We dub this evaluation setting as gold trigger evaluation. The conventional pipeline evaluation of EE works is absent in most EAE-only works, which poses two issues: (1) The absence of pipeline evaluation makes the results of EAE-only works hard to be directly cited and compared in EE studies. In the papers covered by our meta-analysis, there is nearly no direct comparison between EE methods and EAE-only methods. It indicates that the evaluation setting difference has created a gap between the two closely connected research tasks, which hinders the community from comprehensively understanding the research status. (2) The gold trigger evaluation may not well reflect the real-world performance since it ignores the EAE models’ resistance to trigger noise. In real-world applications, the input triggers for EAE models are noisy predicted triggers. A good EAE method should be resistant to trigger noise, e.g., not extracting arguments for false positive triggers. The gold trigger evaluation neglects trigger noise.
To assess the potential influence of this pitfall, we compare experimental results under the gold trigger evaluation and pipeline evaluation of various models in Table 6. We can observe different trends from the results of gold trigger evaluation and pipeline evaluation. For example, although DMBERT performs much better than BERT+CRF under gold trigger evaluation, they perform nearly the same under pipeline evaluation (47.2 vs. 47.1). It suggests that the absence of pipeline evalua-
tion may bring obvious result divergence, which is rarely noticed in existing works. Based on the above discussions, we suggest also conducting the pipeline evaluation in EAE works.

heading: 4 Consistent Evaluation Framework
text: The above analyses show that the hidden pitfalls substantially harm the consistency and validity of EE evaluation. We propose a series of remedies to avoid these pitfalls and develop a consistent evaluation framework, OMNIEVENT. OMNIEVENT helps to achieve the remedies and eases users of handling the inconspicuous preprocessing and evaluation details. It is publicly released and continually maintained to handle emerging evaluation pitfalls. The suggested remedies include specifying data preprocessing (§ 4.1), standardizing outputs (§ 4.2), and providing pipeline evaluation results (§ 4.3). We further re-evaluate various EE models using our framework and analyze the results in § 4.4.

heading: 4.1 Specify Data Preprocessing
text: As analyzed in § 3.2, preprocessing discrepancies have an obvious influence on evaluation results. The research community should pay more attention to data preprocessing details and try to specify them. Specifically, we suggest future EE works adopt a consistent preprocessing method on the same dataset. Regarding the example in § 3.2, for the multiple ACE 2005 preprocessing scripts, we recommend ACE-Full since it retains the most comprehensive event annotations, e.g., multi-token triggers and the time-related argument roles, which are commonly useful in real-world applications. If a study has to use different preprocessing methods
for special reasons, we suggest specifying the preprocessing method with reference to public codes. However, there are no widely-used publicly available preprocessing scripts for many EE datasets, which makes many researchers have to re-develop their own preprocessing methods. In our consistent evaluation framework, we provide preprocessing scripts for various widely-used datasets, including ACE 2005 (Walker et al., 2006), TAC KBP Event Nugget Data 2014-2016 (Ellis et al., 2014, 2015, 2016), TAC KBP 2017 (Getman et al., 2017), RichERE (Song et al., 2015), MAVEN (Wang et al., 2020), LEVEN (Yao et al., 2022), DuEE (Li et al., 2020b), and FewFC (Zhou et al., 2021). We will continually add the support of more datasets, such as RAMS (Ebner et al., 2020) and WikiEvents (Li et al., 2021b), and we welcome the community to contribute scripts for more datasets.

heading: 4.2 Standardize Outputs
text: Based on the discussions about output space discrepancy in § 3.3, we propose and implement an output standardization method in our framework.
To mitigate the inconsistency of output spaces between paradigms, we project the outputs of nonCLS paradigm models onto the most strict CLSparadigm output space. Specifically, we follow strict boundary-matching rules to assign the nonCLS predictions to each trigger/argument candidate in pre-defined candidate sets of the CLS paradigm. The final evaluation metrics are computed purely on the candidate sets, and those predictions that fail to be matched are discarded. The intuition behind this operation is that given the CLS-paradigm candidate sets are automatically constructed, the illegal predictions out of this scope can also be automatically filtered in real-world applications.
Regarding the unclear mappings between predictions and annotations, we consider the scenario of real-world applications and propose several deterministic mapping rules for consistent evaluations. We respond to the issues mentioned in § 3.3 as follows. 1⃝ Prediction span overlaps the gold span. We follow strict boundary-matching rules and discard such overlapping predictions. For example, the SL prediction of as Chief Executive of cannot strictly match any candidate in the candidate set of the CLS paradigm. Hence it is discarded after output standardization. 2⃝ Multiple predictions for one annotated span. If the outputs are with confidence scores, we choose the prediction
with the highest confidence as the final prediction, otherwise, we simply choose the first appearing prediction. The remaining predictions are discarded. 3⃝ Predictions without positions for non-unique spans. We assign such predictions to the annotated spans simply by their appearing order in the output/input sequence to avoid information leakage. We encourage designing new models or postprocessing rules to add positional information for CG predictions so that this issue can be directly solved by strict boundary-matching.

heading: 4.3 Provide Pipeline Evaluation Results
text: The absence of pipeline evaluation (§ 3.4) creates a gap between EE and EAE works, and may not well reflect EAE models’ performance in real-world scenarios. Therefore, in addition to the common gold trigger evaluation results, we suggest future EAEonly works also provide pipeline evaluation results. However, there are two difficulties: (1) It is an extra overhead for the EAE-only works to implement an ED model and get predicted triggers on the datasets. (2) If two EAE models use different predicted triggers, their evaluation results are not directly comparable since the trigger quality influences EAE performance. To alleviate these difficulties, our consistent evaluation framework releases off-the-shelf predicted triggers for the widely-used EE datasets, which will help future EAE works conduct easy and consistent pipeline evaluations. The released predicted triggers are generated with existing top-performing ED models so that the obtained pipeline evaluation results shall help the community to understand the possible EE performance of combining top ED and EAE models.

heading: 4.4 Experimental Results
text: We re-evaluate various EE models with our consistent evaluation framework. The results are shown in Table 7, and we can observe that: (1) If we are not aware of the pitfalls of EE evaluation, we can only understand EE development status and compare competing models from the “Original Evaluation” results in Table 7. After eliminating the influence of the pitfalls with our framework, the consistent evaluation results change a lot in both absolute performance levels and relative model rankings. This comprehensively demonstrates the influence of the three identified evaluation pitfalls on EE research and highlights the importance of awareness of these pitfalls. Our framework can help avoid the pitfalls and save efforts in handling intensive evaluation implementation details. (2) Although the changes in F1 scores are minor for some models (e.g., CLEVE), their precision and recall scores vary significantly. In these cases, consistent evaluation is also necessary since real-world applications may have different precision and recall preferences.

heading: 5 Conclusion and Future Work
text: In this paper, we identify three pitfalls of event extraction evaluation, which are data preprocessing discrepancy, output space discrepancy, and absence of pipeline evaluation. Meta-analyses and empirical experiments present a huge impact of these pitfalls, which urges the attention of our research community. To avoid the pitfalls, we suggest a series of remedies, including specifying data preprocessing, standardizing outputs, and providing pipeline evaluation results. We develop a consistent
evaluation framework OMNIEVENT, to help future works implement these remedies. In the future, we will continually maintain it to well handle more emerging EE datasets, model paradigms, and other possible hidden evaluation pitfalls.
Limitations
The major limitations of our work are three-fold: (1) In the empirical experiments, we only train and evaluate models on English datasets. As the analyzed pitfalls are essentially language-independent, we believe the empirical conclusions could generalize to other languages. The developed consistent evaluation framework now includes multiple English and Chinese datasets, and we will extend it to support more languages in the future. (2) The three pitfalls analyzed in this paper are identified from our practical experiences and may not cover all the pitfalls of EE evaluation. We encourage the community to pay more attention to finding other possible hidden pitfalls of EE evaluation. We will also continually maintain the proposed consistent evaluation framework to support mitigating the influence of newly-found pitfalls. (3) Our meta-analysis only covers papers published at ACL, EMNLP, NAACL, and COLING on mainstream EE research since 2015. Although we believe that we can obtain representative observations from the 116 surveyed papers, some EE works published at other venues and at earlier times are missed.
Ethical Considerations
We discuss the ethical considerations and broader impact of this work here: (1) Intellectual property. The copyright of ACE 2005 belongs to LDC6. We access it through our LDC membership and strictly adhere to its license. We believe the established ACE 2005 dataset is desensitized. In our consistent evaluation framework, we will only provide preprocessing scripts rather than preprocessed datasets for those datasets whose licenses do not permit redistribution. The ACE-DYGIE preprocessing script7 and the used code repositories for DMCNN8, DMBERT8, BiLSTM+CRF8, BERT+CRF8, EEQA9, and Text2Event10 are released under MIT license11. These are all public research resources.
6https://www.ldc.upenn.edu/ 7https://github.com/dwadden/dygiepp 8https://github.com/THU-KEG/MAVEN-dataset 9https://github.com/xinyadu/eeqa
10https://github.com/luyaojie/Text2Event 11https://opensource.org/licenses/MIT
We use them for the research purpose in this work, which is consistent with their intended use. (2) Intended use. Our consistent evaluation framework implements the suggested remedies to avoid the identified pitfalls in EE evaluation. Researchers are supposed to use this framework to conduct consistent evaluations for comparing various competing EE models. (3) Misuse risks. The results reported in this paper and the evaluation results produced by our consistent evaluation framework should not be used for offensive arguments or interpreted as implying misconduct of other works. The analyzed pitfalls in this work are inconspicuous and very easy to be accidentally overlooked. Hence the community is generally unaware of them or underestimates their influence. The contribution of our work lies in raising awareness of the pitfalls and helping to avoid them in future works. (4) Accessibility. Many widely-used datasets (such as ACE 2005, KBP, etc.) are not freely available to everyone. The financial fairness issue may influence the broader usage of the data for EE research.

heading: A Experimental Details
text: The section introduces the experimental details in the paper, including the data preprocessing details (appendix A.1), the reproduction details (appendix A.2), and the training details (appendix A.3).
A.1 Data Preprocessing Details
The section introduces the details of the three data preprocessing scripts for ACE 2005: ACE-DYGIE, ACE-OneIE, and ACE-Full.
ACE-DYGIE We adopt the released official codes12 provided by Wadden et al. (2019) as the ACE-DYGIE preprocessing script. Specifically, we adopt the widely-used “default-settings” in the codes to preprocess ACE 2005. ACE-DYGIE uses spaCy13 for sentence segmentation and tokenization. The version of spaCy is 2.0.18, and the used spaCy model is en_core_web_sm.
ACE-OneIE We adopt the released official codes14 provided by Lin et al. (2020) as the ACE-OneIE preprocessing script. ACE-OneIE uses NLTK15 for sentence segmentation and tokenization , and the version of NLTK is 3.5.
ACE-Full We adopt the released official codes16 provided by Wang et al. (2019b) as the ACE-Full preprocessing script. ACE-Full uses the Stanford CoreNLP17 toolkit for sentence segmentation and tokenization, and the version of CoreNLP is 4.4.0.
A.2 Reproduction Details
In this section, we introduce the reproduction details of all the reproduced models and provide some explanations for the results’ differences between our reproduction and the originally reported results. All the reproduction experiments adopt their original evaluation settings, respectively. The number of parameters for each reproduced model is shown in Table 8.
12https://github.com/dwadden/dygiepp 13https://spacy.io/ 14https://blender.cs.illinois.edu/software/
oneie/ 15https://www.nltk.org/ 16https://github.com/thunlp/HMEAE 17https://stanfordnlp.github.io/CoreNLP/
DMCNN Our DMCNN implementation is mainly based on the codes18 provided by Wang et al. (2020). The reproduced ED F1 score (67.2) is similar to the reported result (69.1) in the original paper (Chen et al., 2015) on the ACE 2005 dataset. However, there is a gap between our reproduced and the originally reported EAE F1 scores (43.2 vs. 53.5). A possible reason is that Chen et al. (2015) adopts a different EAE evaluation setting: Only the argument annotations of the predicted triggers are included in the metric calculation, while the argument annotations of the false negative trigger predictions are discarded. This setting is also adopted in some other early works like DMBERT (Wang et al., 2019b), and we call it “legacy setting”. Compared to the common evaluation setting now, which includes all the argument annotations, the recall scores under the legacy setting are typically higher. When re-evaluating our reproduced DMCNN under the legacy setting, the EAE F1 score (53.9) is consistent with the originally reported result (53.5).
DMBERT Our DMBERT implementation is mainly based on the codes18 provided by (Wang et al., 2020). The reproduced ED F1 score (74.5) is consistent with the originally reported result (74.3) on the ACE 2005 dataset. However, similar to the DMCNN case introduced in the last paragraph, the reproduced EAE F1 score (54.8) is lower than the originally reported result (57.2 in Wang et al. (2019b)) due to the “legacy setting”. When re-evaluating the reproduced DMBERT under the legacy setting, the EAE F1 score is 60.6.
CLEVE We download the pre-trained CLEVE checkpoint19 and finetune it on ACE 2005. The reproduced F1 scores of ED (78.3) and EAE (61.0) are basically consistent with the originally reported ED (79.8) and EAE (61.1) results.
18https://github.com/THU-KEG/MAVEN-dataset 19https://github.com/THU-KEG/CLEVE
BiLSTM+CRF We implement BiLSTM+CRF based on the codes18 provided by Wang et al. (2020). The reproduced ED F1 score (75.5) is similar to the reported result (75.4) in the original paper (Wang et al., 2020) on ACE 2005. As there is no work using BiLSTM+CRF to perform EAE, we adopt all the settings used in ED and evaluate the EAE performance of BiLSTM+CRF.
BERT+CRF We implement BERT+CRF based on the codes18 provided by Wang et al. (2020). The reproduced ED F1 score (72.1) is similar to the reported result (74.1) in the original paper (Wang et al., 2020) on ACE 2005. As there is no work using BERT+CRF to perform EAE, we implement its EAE model following all the ED settings.
EEQA We implement EEQA (Du and Cardie, 2020b) based on the released official codes20. When directly running the released code, we get the F1 score of 69.0 for ED and 47.3 for EAE, which are consistent with our finally reproduced ED (69.5) and EAE (47.4) results. However, there is still a gap between the reproduced and the originally reported results, which is also mentioned in several GitHub issues21.
PAIE We implement PAIE (Ma et al., 2022) based on the released official codes22 and evaluate it in different evaluation settings. The reproduced EAE F1 score (71.8) is basically consistent with that reported in the original paper (72.7).
Text2Event We adopt the released official codes23 to re-evaluate Text2Event (Lu et al., 2021) in different settings. There are minor differences between the reproduced F1 results and the originally reported results (ED: 69.5 vs. 71.9, EAE: 50.8 vs. 53.8). We think the differences come from randomness. When only using the same random seed reported by the authors, the reproduction results are nearly the same as the original results.
A.3 Training Details We run three random trials for all the experiments using three different seeds (seed=0, seed=1, seed=2). The final reported results are the average results over the three random trials. All hyperparameters are the same as those used in the origi-
20https://github.com/xinyadu/eeqa 21https://github.com/xinyadu/eeqa/issues/11,
https://github.com/xinyadu/eeqa/issues/5 22https://github.com/mayubo2333/PAIE 23https://github.com/luyaojie/Text2Event
nal papers. The experiments of CLEVE, PAIE, and Text2Event are run on Nvidia A100 GPUs, which consume about 600 GPU hours. The other experiments are run on Nvidia GeForce RTX 3090 GPUs, which consume about 100 GPU hours.

heading: B Additional Experimental Results
text: The section shows additional experimental results on different preprocessed ACE 2005 datasets.
Output Space Discrepancy Table 9 shows the metrics’ differences with and without output standardization on the ACE-DYGIE and ACE-Full preprocessed datasets. We can observe that all evaluation metrics change obviously, which is consistent with the observations in § 3.3.
Absence of Pipeline Evaluation Table 10 shows the results using gold trigger evaluation and pipeline evaluation on the ACE-DYGIE and ACE-Full preprocessed datasets. We can observe that the phenomena are consistent with those in § 3.4.
Consistent Evaluation Framework Table 11 shows the results using our consistent evaluation on ACE-DYGIE, ACE-OneIE, and ACE-Full. We can observe that the phenomena on ACE-DYGIE and ACE-OneIE are consistent with those in § 4.4.

heading: C Papers for Meta-Analysis
text: The complete list of papers surveyed in our metaanalysis is shown in Table 12.

heading: D Authors’ Contribution
text: Hao Peng, Feng Yao, and Kaisheng Zeng conducted the empirical experiments. Feng Yao conducted the meta-analyses. Xiaozhi Wang, Hao Peng, and Feng Yao wrote the paper. Xiaozhi Wang designed the project. Lei Hou, Juanzi Li, Zhiyuan Liu, and Weixing Shen advised the project. All authors participated in the discussion.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: In Section ”Limitations”

heading: 3 A2. Did you discuss any potential risks of your work?
text: In Section ”Ethical Considerations”

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: In Section ”Abstract” and "Introduction"
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: In Section 3 and Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: In Section 3 and appendix A

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: In Section ”Ethical Considerations”

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? In Section ”Ethical Considerations”

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? In Section ”Ethical Considerations”

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? In Section 3 and Section ”Ethical Considerations”
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In Section 3
C 3 Did you run computational experiments?

heading: In Section 3, Section 4, and appendix B

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? In appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? In appendix A

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In appendix A and appendix B

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? In appendix A
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We develop a consistent
evaluation framework OMNIEVENT, to help future works implement these remedies. In the future, we will continually maintain it to well handle more emerging EE datasets, model paradigms, and other possible hidden evaluation pitfalls. Limitations
The major limitations of our work are three-fold: (1) In the empirical experiments, we only train and evaluate models on English datasets