Evaluating Zero-Shot Event Structures: Recommendations for Automatic Content Extraction (ACE) Annotations
Year: 2023

Abstract:
Zero-shot event extraction (EE) methods infer richly structured event records from text, based only on a minimal user specification and no training examples, which enables flexibility in exploring and developing applications. Most event extraction research uses the Automatic Content Extraction (ACE) annotated dataset to evaluate supervised EE methods, but can it be used to evaluate zero-shot and other low-supervision EE? We describe ACE’s event structures and identify significant ambiguities and issues in current evaluation practice, including (1) coreferent argument mentions, (2) conflicting argument head conventions, and (3) ignorance of modality and event class details. By sometimes mishandling these subtleties, current work may dramatically understate the actual performance of zero-shot and other lowsupervision EE, considering up to 32% of correctly identified arguments and 25% of correctly ignored event mentions as false negatives. For each issue, we propose recommendations for future evaluations so the research community can better utilize ACE as an event evaluation resource.

heading: 1 Introduction
text: Zero-shot event extraction (EE) methods infer richly structured instances of action or relationship occurrences from unstructured text data, based on a user-supplied natural language specification of the desired event—without annotated training examples (Du and Cardie, 2020; Liu et al., 2020; Li et al., 2021; Lyu et al., 2021). The extracted structure is useful for many applications such as analyzing interactions between entities and performing more intelligent question answering (Gao et al., 2016; Liu et al., 2017a; Cao et al., 2020; Li et al., 2020b), and the low resources required by zero-shot EE methods further this practical advantage. We refer to the structure as an event, where each event could have an arbitrary structure as needed. Each struc-
ture contains information such as the participants involved, content, and location of the event.
To evaluate supervised EE methods, many works use the Automatic Content Extraction (ACE) dataset—specifically, the Linguistic Data Consortium’s ACE 2005 Multilingual Training Corpus (Doddington et al., 2004),1 which includes English, Chinese, and Arabic documents and resulted from the U.S. federal government’s ACE program.2 The ACE dataset stores information about entities, relations, and events from 598 (for English) documents in a rich structure; our focus is mostly on its events. ACE is frequently used for event extraction modeling and evaluation, and is often claimed to be the most widely used such dataset (§3). While there are many somewhat similar structured semantic datasets, ACE still shines in having whole-document annotations (contra FrameNet; Baker et al., 2003; Baker and Sato, 2003; Fillmore et al., 2003), realistically non-lexical-specific event classes (contra PropBank (Palmer et al., 2005), OntoNotes (Weischedel et al., 2017), and Semantic Dependencies (Oepen et al., 2014)), event modality (contra PB, ON, SD), English data (contra Entities, Relations, and Events (ERE)),3 and specification of event arguments (contra Richer Event Description (RED); O’Gorman et al., 2016) that are simultaneously represented both as text spans
1https://catalog.ldc.upenn.edu/LDC2006T06 https://doi.org/10.35111/mwxc-vh88
2https://www.ldc.upenn.edu/collaborations/ past-projects/ace, http://web.archive.org/web/ 20080303183132/https://www.nist.gov/speech/ tests/ace/. A separate evaluation dataset was not released publicly (Haghighi and Klein, 2009, footnote 7); we follow the convention of subsequent research of referring to the public release LDC2006T06 as “the ACE dataset” or simply “ACE,” despite “training” in its title.
3Song et al. (2015) promise an LDC release of their ERE annotations while Aguilar et al. (2014) analyzes ERE’s guidelines—both with English examples—but LDC’s catalog suggests only a Chinese corpus was ever released (LDC2020T19). Li et al. (2020b) reports ERE English results, presumably from a proprietary dataset.
1651
(contra Abstract Meaning Representation (AMR); Banarescu et al., 2013), and discourse-level entities4 (§2). While ACE does not include RED’s interesting causal and bridging event-event relations (see also Hovy et al., 2013), its core tasks related to entities and event arguments have important applications and are far from solved.
We investigate using the ACE dataset to evaluate zero-shot and other low-supervision EE methods, which are more real-world relevant than highlysupervised EE methods for requiring few if any annotations, but which may face certain evaluation challenges more severely.5 First, we identify issues related to how evaluations extract gold event argument annotations from ACE and to the possibly clashing use case of a zero-shot EE method versus the annotations in ACE. Evaluation of zero-shot EE methods is particularly sensitive to these issues since they lack knowledge of (sometimes arbitrary) details in ACE event structures that are implicit in training examples—and their ignorance of them may be correct for many applications. Therefore, we present guidelines and methods to overcome these issues in English, which could in theory be adaptable to other languages, and quantify their potential impact.6

heading: 2 Structure of Events and Entities in ACE
text: The Automatic Content Extraction (ACE) dataset stores annotations for entity, relation, and event structures for news, conversations, blog, and transcript textual data. We focus on the ACE event extraction task (Ahn, 2006), which takes a sentence as input and outputs a set of event tuples, which we attempt to precisely specify.
Events (Figure 1). Every event takes one of 33 discrete event classes t ∈ T ,7 each of which has a discrete set of (typically, 1–6) roles Rt which its
4ON 5.0 does include joint coreference and PropBank annotations, albeit for only 18% of its English documents.
5This work is motivated by the zero-shot setting (no annotated data, but with user specification of desired event class), but could apply to other low-supervision settings including unsupervised event extraction (e.g. Chambers (2013): induces event classes with no annotations or user guidance) and fewshot EE (small number of annotated examples). In all low supervision settings, we do not expect a model to learn superficial annotation quirks, which we believe is more real-world relevant than high-supervision settings.
6While we do not contribute new evaluation software, we make available code to reproduce this paper’s analyses at: https://github.com/ec769/ZS-evalanalysis-ACE.
7ACE defines 7 event types and 33 subtypes; we do not focus on this hierarchical structure.
arguments can take. An event tuple has the form ⟨t, g, {a1..an}⟩ where
1. t ∈ T is the event class. 2. g is the span8 of the event trigger, a word that
identifies or represents the event class. 3. {a1..an} is a (possibly empty) set of event ar-
guments explicitly mentioned in the sentence, each with ai = ⟨a(r)i , a (s) i ⟩: the role a(r) ∈ Rt, and argument span a(s).9
The full event extraction task is to output some number of event tuples from the sentence; research often examines subtasks to identify various subsets of t, g, a(r), a(s), such as event trigger classification or event detection (just (g, t)). Finally, the tuple has several additional semantic tags such as modality and tense (§4.3).
Entities (Figure 2). An event argument a(s) may also be a mention of an entity, a document-level object with its own type information and one or more coreferential mention spans throughout a document. For an argument span a(s), let C(a(s)) refer to the set of all its coreferential mentions.10 Additionally, ACE’s <entity> data structure defines for each mention a head span (§4.2).
In the following example from ACE, a killing (LIFE.DIE) event has agent a(s)=“Iraq’s Mukhabarat” (Figure 1); when cross-referencing the entity information C(a(s)), it turns out this argument is coreferentially mentioned three times in the sentence (Figure 2).
Earlier, from 1979 to 1983, he headed Iraq's Mukhabarat, or intelligence service, a period when the organization arranged
2106
8In ACE and both figures, called an extent, which is a character span (start and end positions) in the text. We use span and extent interchangeably.
9Each argument may be a time, value (i.e. quantity), or entity (i.e. person, place or thing); most works discussed in §3 consider only entities as arguments, which we follow.
10Our discussion conflates an argument span with the entity’s mention span at the same location; ACE technically defines them separately, and sometimes the argument span can be longer. However, they always have the same head.
Earlier, from 1979 to 1983, he headed Iraq's Mukhabarat, or intelligence service, a period when the organization arranged
executions of regime opponents in Iraq and overseas, the official said.
2092
2283

heading: 3 Review of Using ACE to Evaluate EE
text: We reviewed 38 papers published from 2008 through 2022, cited in Li et al. (2022)’s survey of deep learning methods for event extraction, to examine how they use ACE to evaluate EE tasks (Ji and Grishman, 2008; Liao and Grishman, 2010; Hong et al., 2011; Li et al., 2013; Nguyen and Grishman, 2015; Chen et al., 2015; Nguyen et al., 2016; Yang and Mitchell, 2016; Nguyen and Grishman, 2016; Feng et al., 2016; Liu et al., 2016; Huang et al., 2016; Sha et al., 2016; Chen et al., 2017; Liu et al., 2017b; Zhao et al., 2018; Zeng et al., 2018; Hong et al., 2018; Liu et al., 2018; Huang et al., 2018; Liu et al., 2019; Zhang et al., 2019b; Wang et al., 2019; Zhang et al., 2019a; Yang et al., 2019; Nguyen and Nguyen, 2019; Wadden et al., 2019; Chen et al., 2020; Du and Cardie, 2020; Liu et al., 2020; Li et al., 2020a; Lin et al., 2020; Li et al., 2021; Ahmad et al., 2021; Zhou et al., 2021; Wang et al., 2021; Lu et al., 2021; Lyu et al., 2021). Several state that ACE is the most popular dataset for evaluating EE methods (Li et al., 2022; Zhang et al., 2019b; Wang et al., 2019). While the ACE data release does not define a split, these papers, especially after 2011, settled on a shared train/development/test split (§A.6).
When considering event trigger and event argument identification, all papers require matching the gold standard’s extent to be considered correct. For arguments, which are usually multiple tokens long, some works require matching the full argument extent a(s) while others only use its head extent. (Additional details in §A.6.)
The works that we analyzed identify several challenges with using ACE. Some event subtypes are very sparse; almost 60% of event types have fewer than 100 labeled samples, while three event types each have fewer than ten out of the 5042 samples over all English documents and 33 event classes (Chen et al., 2017; Liu et al., 2017b, 2018). Second, the manually specified event schemas in ACE are hard to generalize to different domains, such as
the WEAPON argument role (Huang et al., 2016). Third, Ji and Grishman (2008) find that human annotators achieve only about 73% of the F1 score on the argument and trigger identification task and annotation quality continues to be questioned in debates about annotation guidelines (Lin et al., 2020). In any case, ACE remains a widely used dataset for evaluation.

heading: 4 Recommendations for Using ACE to Evaluate Zero-Shot and Other Low-Supervision EE Methods
text: Recommendation 1: Coreference Invariant Argument Matching. To evaluate correctness of event arguments using ACE, allow a match to any coreferent mention of the argument (c ∈ C(a(s))), not just the one mention in <event_argument_mention> (a(s)). This (we believe) erroneous practice is widespread, and may consider up to 32% of correctly identified entitytype arguments as incorrect.
Problem. Although ACE stores event triggers and types as part of an event mention, it stores event arguments as part of both event mentions and entity, time, or value mentions. The event mention argument stores one reference to the argument (a(s)), even if multiple references exist (C(a(s))). Low supervision EE methods can not learn a training set’s potentially superficial convention for which of multiple references to specify.
Issues in the Literature. Alarmingly, although ACE stores multiple gold references as entity mentions, they are often not used. We find that a number of recent works, especially on zero-shot EE, that ignore them. Wadden et al. (2019)’s preprocessing code, which was used in several later works (Du and Cardie, 2020; Lin et al., 2020; Li et al., 2021; Lu et al., 2021; Lyu et al., 2021), does not gather multiple references to a(s) in an event tuple. While an unofficial update includes entity information, we identify further difficulties in §A.3. Independently, Zeng et al. (2018) acknowledge not applying coreference resolution, which contributes to a higher argument identification task error rate. While we acknowledge that whether to model coreference is a complex question, using gold standard coreference information at evaluation time is an independent issue and ought to be mandatory, for any modeling approach. Even for a purely extent prediction system, gold-standard coreference is necessary for correct evaluation.
Findings. Table 1 shows that roughly 14.6% of arguments have multiple references within the same sentence when arguments are not pronouns, and roughly 31.7% do otherwise. In the worst case, an evaluation could consider all such arguments, even if correctly identified, as false negatives. Next, we investigate if a pattern for choosing a(s) out of C(a(s)) exist. If multiple references exist, a(s) is the first reference to appear in the sentence 56.3% of the time. If one or more is a named entity, a(s) is, also, 60.7% of times. (More details in §A.5). Given the alarming statistics in Table 1 and a nonobservable pattern for choosing a(s) out of C(a(s)), we recommend extracting all possible references to an argument using the <entity> object, instead of only relying on <event mention argument>.
Recommendation 2: Dual ACE and Automatic Head Selection. To evaluate correctness of an event argument using ACE, in addition to comparing its head against the head provided by ACE, compare its head against the one selected by a Universal Dependency-based parser. We find that 8.1% of heads that the English portion of ACE identifies are not consistent with a Universal Dependency SpaCy3 parser-based head finder (more details in §A.4.1).
Problem and Literature. To determine correctness of an event argument, either compare it against a(s) in ACE or its head against a(s)’s head. Comparing against the entire a(s) is likely to yield false negatives because ACE argument spans can be very long, including the noun phrase’s complements and even elaborate relative clauses (e.g.: “the women from Texas who heinously drowned her five kids, aged 6 months to 7 years, one by one, in her bathtub"). Thus most works we reviewed evaluate argument correctness by comparing its head with the head of potential a(s)s. For zero-shot EE methods with no knowledge of argument constitutions, using the head seems especially appropriate.
Method and Findings. We investigate if the head of a(s) that ACE specifies is consistent with the Universal Dependency (UD) (Nivre et al.,
2020) definition of head, which we identify from spaCy3’s UD parse as the token in the span that is an ancestor to the rest of the span (i.e., the span’s subgraph’s root); we additionally add a heuristic to address a frequent parse error when the noun phrase head is analyzed as the relative clause’s subject, and to extend the head to be multiple tokens (as sometimes occurs in ACE’s heads) when the head token is within an spaCy3-identified named entity. The discrepancies in Figure 3 suggest that ACE often does not follow the UD formalism. (Additional algorithmic details and discrepancies in §A.4.)
Next, we explore the feasibility of consistently reconstructing the exact head specified by ACE. Given clear inconsistencies in the way that ACE selects the head in Figure 3 (eg: “Haifa university" and “Israeli army"), we conclude that ACE may not identify the argument head in a systematic or at least easily emulatable way, which may contribute to false negatives. To eliminate the inconsistency issue, we propose to use a UD-based algorithm to select heads from ACE argument extents for matching, in addition to the heads specified by ACE. The head from the UD-based parser is not always the most appropriate for a given argument extent (see error analysis of parser behavior in §A.4.1), but our approach does avoid the inconsistency issue. While we only applied our UD-based algorithm to English data, this head-matching approach may be adaptable to other languages with available UD parsers.
Recommendation 3: Analyze a Subset of ACE Modalities or Event Classes. Consider a subset of annotated events as the ground truth event set to improve the evaluation of zero-shot EE methods that target a particular use case; e.g., sociopolitical analysis.
Problem and Literature. While greater flexibil-
ity enables zero-shot EE methods to be more practical, extracting structured data as events without requiring training examples, each practical application has a different objective. For example, social scientists and political forecasters may need to analyze historical events that actually happened in the past (Schrodt et al., 1994; O’Connor et al., 2013; Boschee et al., 2013; Halterman et al., 2021; Hanna, 2017; Hürriyetoğlu et al., 2021; Giorgi et al., 2021; Stoehr et al., 2021), such as in the widely-used ICEWS automatically generated events dataset (Boschee et al., 2017). However, in other applications such as those on opinion or sentiment tasks (Swamy et al., 2017), the aim of zero-shot EE methods may be benefited by hypothetical events.
Many aspects of modality have been explored in computational modeling, such as temporal semantics (Timebank (Pustejovsky et al., 2003)), factual versus uncertain or hypothetical status (Factbank (Saurí and Pustejovsky, 2009), Pragbank (de Marneffe et al., 2012), (Diab et al., 2009; Prabhakaran et al., 2015; Stanovsky et al., 2017; Rudinger et al., 2018; Yao et al., 2021; Lee et al., 2015)), and in literary domains (Litbank (Bamman et al., 2019, 2020)). ACE includes a simple modality label for each event instance as either ASSERTED to indicate an event instance that was referred to as a real occurrence, or OTHER for all others: non-grounded beliefs (e.g. rumors), hypotheticals, commands, threats, proposals, desires, promises, etc. In fact, for 25% of event instances in ACE, the modality tag label is OTHER. Yet, the 38 works that we explored in §3 which use ACE to evaluate EE methods do not include modality as part of the task definition. We propose that future work could better use ACE by predicting or analyzing subsets of modalities to more clearly support downstream applications.
Finally, modality is important since it may also interact with modeling (Cai and O’Connor, 2023). Zero-shot EE methods involving questionanswering (QA) or text entailment (TE) models (Lyu et al., 2021), may enforce modality restrictions through the language in the query. For example, the past tense question “did the police arrest someone?” (Halterman et al., 2021) asks for a reported occurrence that the police are arresting or have arrested someone, but not an intended or hypothetical arrest. Whether this matches user intent, and whether models respect or ignore the query’s modality restrictions, are important avenues for future work; ACE data can aid such analysis.

heading: 5 Conclusion
text: We explore how to use ACE, which is a gold standard dataset containing annotations of events from diverse text data in a rich structure, to evaluate zero-shot and other low-supervision EE methods by identifying issues that may more severely affect their evaluation. We particularly find difficulties with evaluating spans of events due to a lack of training data for zero-shot and low-supervision EE methods to learn superficial annotation quirks from. However, we present methods to overcome these issues and demonstrate them on the English portion of ACE, noting that in principle they may be adaptable to any language. Ultimately, we advocate for using ACE to evaluate zero-shot and other lowsupervision EE methods after addressing the issues, and discuss the potential for using ACE in smarter ways to evaluate different types of EE methods in the future.

heading: Acknowledgments
text: We thank the UMass NLP group and anonymous reviewers for feedback. This work was supported by NSF CAREER 1845576. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

heading: A Appendix
text: A.1 Limitations
This work identifies specific issues and provides solutions to them. Recommendations 1 and 3 have solutions that could completely eliminate the issue that they address. The method that we introduce for recommendation 2 eliminates inconsistency in selecting the head of an argument extent; however, more ways of selecting the head may exist. Future work could explore additional ways of selecting the head in order to further reduce the chance that a correctly identified argument is considered as incorrectly identified.
A.2 Risks
The risks are the same as the risks for event extraction and information extraction. While a large literature, portions of which we reference, exists on ACE event extraction, less attention has been paid to its ethical and social implications. Sociopolitical events, which ACE often focuses on, may be of great interest to social scientists (e.g. the CASE workshop) as well as having government and military intelligence utility (presumably, an original motivation of the ACE program: while its original websites11 and papers (Doddington et al., 2004) do not appear to explicitly specify a funding agency, they cite the earlier Message Understanding Conference (MUC) as its predecessor, whose proceedings explicitly cite DARPA as a sponsor (muc, 1991)). See, for example, Li et al. (2020b)’s ethical discussion of dual use issues for their partially ACE-based multimodal tracking/surveillance system.
A.3 Issues with the Current Literature for Identifying Arguments
In Section 4, we identified that several recent works since 2018, including some on zero-shot EE, do not
11https://www.ldc.upenn.edu/collaborations/ past-projects/ace http://web.archive.org/web/ 20080303183132/https://www.nist.gov/speech/ tests/ace/
evaluate the correctness of an argument by comparing it against all possible references to the argument within a sentence. We discuss more details about such works.
Wadden et al. (2019) state that "the ACE data set lacks coreference annotations," and the original released code12 does not consider evaluating an argument against multiple references to the same argument. (As we note, ACE does in fact include significant coreference annotations.) Later, a third party added a software option to include clusters of entity spans, where a cluster contains spans of references referring to the same entity throughout a document, along with the event information. However, with this option, coreference resolution is still difficult because neither the entity information nor the event argument information in the pre-processed data includes an ID. While the pre-processed data includes entity and event argument spans, the spans may not completely match so mapping an event mention argument to an entity mention to check for multiple references using the pre-processed data becomes very difficult. Another third party also added code to gather coreference information corresponding to each event, but in the Github repository, one of Wadden et al. (2019)’s original authors states that both of these additions are unofficial.
We examine code bases of several works that design their pre-processing code similarly to Wadden et al. (2019) and find that they also do not collect all possible references to arguments from ACE (Du and Cardie, 2020; Lin et al., 2020; Lyu et al., 2021; Lu et al., 2021; Li et al., 2021). The Du and Cardie (2020) pre-processing code is most similar to the Wadden et al. (2019) pre-processing code, and the evaluation code does not compare arguments extracted by an EE method with ACE annotated references. Lin et al. (2020) and Lyu et al. (2021) state that they follow Wadden et al’s pre-processing code and release their code bases. Although the code is more different than Du and Cardie (2020)’s code is, it does not gather multiple gold references for the same argument. Lyu et al. (2021) mention that some errors in the evaluation are attributable to this coreference issue. Further, Li et al. (2021) and Lu et al. (2021) both state that they follow Wadden et al. (2019)’s pre-processing and their respective code bases reflect this. Li et al. (2021) additionally state that they do not need to perform coreference resolution.
12https://github.com/dwadden/dygiepp
A.4 Exploration into the ACE Head and UD-based Head
We discuss the algorithm for identifying the UDbased head from the argument extent, and then show examples of the head that ACE identifies versus the head that the UD-based algorithm extracts.
A.4.1 Algorithm
The algorithm identifies the head of an argument extent in a way that is consistent with the Universal Dependency Parsing (UD) definition of head, but has slight modifications to suit the interpretation that a head could be an entire named entity and to work around possible well-known types of misparses by the UD formalism. The first step of the algorithm is to apply a tokenizer on the argument extent such that hyphens and apostrophes do not break words apart. Next, use SpaCy3 to construct a list of named entities that do not include the date, time, ordinal, or cardinal entity types. After, find the lowest common ancestor (LCA) for the argument extent. If the LCA is not within a named entity of the argument extent, select it as the head. Otherwise, select the named entity that the LCA is a substring of as the head.
The algorithm additionally handles two special cases that could complicate the UD selection of the appropriate head. If a null relativizer exists in an event argument, the UD parser may select a verb as the head. For example, in: “at least seven journalists killed covering the conflict", the parser selects “killed" as the head, which is incorrect. In addition, if a relative pronoun exists in an event argument, as in: “leader of the Iraq arms program who defected for a time", the UD parser may select the relativizer, “who", as the head. To work around these cases, the algorithm considers the argument extent to end after the first instance of a verb or relativizer pronoun that occurs after a noun (after a noun to avoid mis-identifying heads for cases such as: “these battered buildings").
We run the algorithm over all of the argument extents in ACE that are not of the form “[x] and/or [y]" since ACE has an exception of extracting two heads ([x] and [y]) from such extents, and find three mistakes out of a sample of 300. On the rare single-word case that a mistake occurs, the argument span usually contains a noun compound with spaces (most such noun compounds do not indicate a mistake), and none of these spans contain null relativizers.
A.4.2 Contradictions
We show surprising discrepancies between the head that ACE identifies and the head that the UD-based algorithm identifies with respect to an argument extent below. Similar to the examples in Figure 3 of the main paper, the head that ACE identifies is in red brackets and the head that the UD-based algorithm identifies is in blue brackets.
the [Houston [Center]ACE ]AUT [Wall [street]AUT ]ACE [aol time [warnerings]AUT ]ACE [f-14 [aircraft]ACE ]AUT another [half-[brother]ACE ]AUT of saddam hussein [neither]AUT of the [women]ACE the [[Office]ACE of the President]AUT the [[president]AUT ]ACE-elect of the American Medical Association several [[parts]ACE ]AUT of southern Iraq [hundreds]AUT of [civilians]ACE in East Timor a [[warren]ACE ]AUT of cells [thousands]AUT of U.S. [troops]ACE the [[Shah]ACE of Iran]AUT the [U.S. Army [7th Cavalry]ACE]AUT [American [Marines]ACE ]AUT two [U.S. [Marines]ACE ]AUT killed in combat 21-year- old [Marine Corporal [Randall Kent Rosacker]ACE ]AUT [delma [banks]AUT ]ACE the [national youth and student peace [coalition]AUT ]ACE [persian [gulf]AUT ]ACE the [center]ACE of the second largest city in iraq, [basra]AUT the [urbuinano [island]AUT ]ACE the [catholic [church]ACE ]AUT in phoenix, arizona two very strong – [militant groups]ACE British [Desert [Rats]AUT ]ACE the [Alfred P. Murrah federal [building]AUT ]ACE his [ex-[wife]ACE ]AUT [tight [ends]AUT ]ACE [9]AUT [more]ACE [19]AUT [more]ACE [second-[graders]ACE ]AUT
A.5 ACE Experiment Details
To extract statistics about coreference, we modify Wadden et al’s pre-processing code. In the analysis, we omit one document due to preprocessing issues
and do not consider times and values as arguments; only entities, which is consistent with most of the literature that we reviewed.
From the results in Table 2, we observe that the selected event mention argument does seem to follow a specific pattern; it does not seem to prefer being a named entity, nor consistently be the first of the references to appear in a sentence; etc.
A.6 Literature Review Details To aim toward fair comparison among EE methods, works use ACE to evaluate them in three general ways. Only the earliest papers (Ji and Grishman, 2008; Liao and Grishman, 2010; Hong et al., 2011) use the first split (A), where the evaluation uses all of the text data and 33 separate event subclasses, ignoring the event classes, and where the test set contains 40 newswire texts, the development set contains 10 newswire texts, and the rest of the texts belong to the training set. The second split (B) is an improvement upon the first, with the only difference of using 30 randomly selected texts in the development set. A zero-shot evaluation of this split variety ignores the training set. A third split variety (C) is for a specific application of event extraction which focuses more on the generalization ability across different domains; in this split, the source domain is news, half of bc is the development set, and the remaining data makes up the test set. Three papers that we reviewed use split (A) (Ji and Grishman, 2008; Liao and Grishman, 2010; Hong et al., 2011), at least 28 papers use split (B) (Li et al., 2013; Nguyen and Grishman,
2015; Chen et al., 2015; Nguyen et al., 2016; Yang and Mitchell, 2016; Nguyen and Grishman, 2016; Feng et al., 2016; Liu et al., 2016; Huang et al., 2016; Sha et al., 2016; Chen et al., 2017; Liu et al., 2017b; Zhao et al., 2018; Liu et al., 2018, 2019; Zhang et al., 2019b; Wang et al., 2019; Zhang et al., 2019a; Yang et al., 2019; Nguyen and Nguyen, 2019; Wadden et al., 2019; Liu et al., 2020; Li et al., 2020a; Ahmad et al., 2021; Lu et al., 2021; Lyu et al., 2021; Wang et al., 2021; Zhou et al., 2021), some for few-shot or zero-shot evaluations use a different, contrived split (e.g. Huang et al. (2018)) and others use both split (B) and a different split (e.g. Du and Cardie (2020)).
In addition, most works use the evaluation criteria that 1. The event trigger is considered correct when its offsets match a gold trigger and event class is correct and 2. An argument is considered correct when its offsets and event class match a gold argument and its event role is correct. However, the criteria does not include many more details and is not in formal math notation, allowing discrepancies in the way that different works implement them.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section A.1 of the Appendix

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section A.2 of the Appendix

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1 of the main paper
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We use the Automatic Content Extraction (ACE) dataset, introducing it in Sections 1 and 2, and using it in Section 4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Sections 1 and 2 of the main paper and A.2 of the Appendix

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 1 and A.2 of the Appendix

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Sections 1, 2, and 3 of the main paper

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section A.2 of the Appendix

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Sections 1 and 2 of the main paper
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We discuss them in Sections 1, 2, and 4 of the main paper and discuss more details in the Appendix.
C 3 Did you run computational experiments? Section 4
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4 of the main paper and Sections A.4 and A.5 of the Appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 of the main paper and Section A.5 of the Appendix

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4 of the main paper and Section A.5 of the Appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Ultimately, we advocate for using ACE to evaluate zero-shot and other lowsupervision EE methods after addressing the issues, and discuss the potential for using ACE in smarter ways to evaluate different types of EE methods in the future