On Event Individuation for Document-Level Information Extraction
Authors: William Gantt
Reno Kriz
Yunmo Chen
Siddharth Vashishtha
Aaron Steven White
Venue: EMNLP 2023 Findings (2023)
Keywords: Information Extraction
Template Filling
Events
Reproducibility
TL;DR: We show that disagreements about how to count events causes trouble for evaluation of document-level information extraction.

Abstract: As information extraction (IE) systems have grown more adept at processing whole documents, the classic task of *template filling* has seen renewed interest as a benchmark for document-level IE. In this position paper, we call into question the suitability of template filling for this purpose. We argue that the task demands definitive answers to thorny questions of *event individuation* — the problem of distinguishing distinct events — about which even human experts disagree. Through an annotation study and error analysis, we show that this raises concerns about the usefulness of template filling metrics, the quality of datasets for the task, and the ability of models to learn it. Finally, we consider possible solutions.

On Event Individuation for Decument-Level Information Extraction
William Gantt Reno Kriz Yunmo Chen
Siddharth Vashishtha" Aaron Steven White!
University of Rochester 2 Johns Hopkins University
(wgantt@ur. Sraant@uu./svaat svashis3@ur. aaron, saaronwwhttep rechester.edi [yunmo] 1eriz1))
Abstract
bomb exploded today in Lima restaurant, and second device that had been placed in the same establishment was deactivated by the Peruvian National Police. There were no victims, and the explosion caused very little damage the restaurant... Guerrillas rusrrlaa of the Tupac Amaru Revolutionary Movement (MRTA) have claimed credit for the terrorist act....
As information extraction (IE) systems have grown more adept at processing whole doc- uments, the classic task of template filling has seen renewed interest as a benchmark for document-level IE. In this position paper, we call into question the suitability of template fill- ing for this purpose. We argue that the task demands definitive answers thorny questions of event individuation the problem of distin- guishing distinct events - about which even human experts disagree. Through an annota- tion study and error analysis, we show that this raises concerns about usefulness of tem- plate filling metrics, the quality of datasets for the task, and the ability of models to learn it. Finally, we consider possible solutions.
Figure excerpted document from MUC-4. Does this describe one bombing two? We argue that this question is confusing and that disagreements about the "right" amswer cause problems for document-level IE.
A demonstration of how template filling eval- uation metrics draw arbitrary distinctions be- tween nearly identical system predictions that differ only their individuation decisions.
1 Introduction
A (re-) Jren)annotttion study, whose results sug- gest that template filling datasets reflect ques- tionable individuation judgments about which even human experts may disagree and even when given annotation guidelines.
Template filling involves extracting structured ob- jects called templates from a document, each of which describes a complex event and its partici- pants. The task comprises both template detection - determining the number of templates present in document and their types - as well as slot filling or role-filler entity extraction (REE) assigning tracted entities slots in those templates that char- acterize the roles entities play in the event. Since documents routinely describe multiple events, doc- uments template filling datasets regularly feature multiple templates, often of the same type. Cor- rectly individuating events is thus a crucial and challenging component of the task. Here, we argue that whereas there is often more than one reason- able way to individuate events, template filling ad- mits such pluralism, which undermines its value benchmark for document-level IE. To support our claim, we draw evidence from several sources:
An error analysis on recent template filling models, showing that they struggle to learn the individuation rules latent in these datasets." ?
Finally, we conclude by considering how the IE community might address these issues.
2 Background
Template Filling Template filling is arguably the foundational task of TE (Grishman, 2019), with its origins in the Message Understanding Confer- ences (MUCs) of the 1990s (Sundheim, 1992: Gr- ishman and Sundheim, 1996). These evaluations produced the MUC-4 dataset, which remains a pri- mary benchmark for both template filling (Cham-
"qqual contribution.
FFllowiig standard usage, we use "template" to denote an instance of a template type (cf. Chen et al. (2023)).
2 All data and analysis code is available here: https://ee /tt//gittt thub, ctm/c//c/////eooc wwwm/wantt//eeteeeeeeeeeeeeeeeee t/event_ individuation.

---

in Appendix A. Rather, the fullest treatment of this subject comes from work on event coreference by Hovy et al. (2013), who posit conditions for full event identiry - the inverse of the individuation problem. Appealing to the notion of a discourse element (DE), an event or entity that is referenced in the text, they stipulate that "two mentions fully corefer if their activity/lvenn/state DE is identical in all respects, far occurrence the text. (In particular, their agents, location, and time are identical or compatible)." Differences in times, locations, and "agents" (i.e. participants) between two event descriptions are intuitively each sufficient conditions for individ- uating events and, importantly, accord with the guidelines for individuating templates provided for MUC-4 and for BETTER (see Appendix A). Tem- plate rypes offer another, coarser means of distin- guishing events. Here, we focus analysis on indi- viduation within type, since this is where most of the difficulty lies, and consider whether these guide- lines are adequate to enable annotators and models to converge similar individuation decisions.
Type: Bombing
Type: Bombing
Perpind: Guernilas
Perpind: Guemiaa
PerpOrg: Tupac Amaru Revolutionnry Movement, MRTA
Tupac Amaru Revolutionary Movement, MATA
PerpOrg:
Target: restaurant
Target restaurant
Victim:
Victim:
Weapon: bomb
Weapon:
can tell from their
Figure Gold MUC-4 templates for the document shown in Figure 1. Plausibly, the document describes just a single bombing (left template), yet predicting only this template would carry a substantial penalty in score (CEAF-REE Fl = 57. 57.), even though it recovers all annotated entities and assigns them the correct roles.
bers and Jurafsky, 2011; Du et al., 2021b; Das et al., 2022: Chen et al., 2023) and for the REE subtask (Patwardhan and Riloff, 2009; Huang and Riloff, 201 2012; Huang et al., 2021; Du et al., 2021a). MUC-4 focuses on incidents of political violence in Latin America and presents an ontology of six template types, all with the same five slots.
More recently, the IARPA BETTER program has released the BETTER Granular dataset (Sobo- roff, 2023), which has begun to receive attention (Holzenberger al., 2022; Chen et al., 2023). BET- TER also features six template types, but unlike MUC-4, each template has a distinct set of slots, which may have other types of fillers beyond enti- ties. We take both datasets case s
3 Individuation and Evaluation
To show why individuation matters for template filling, we first consider how the task is evaluated. In contrast to events in event extraction, which are associated with lexical triggers, templates are not anchored to specific lexical items. Given a predicted template, there are often multiple gold templates one could use as reference, and one must therefore establish an alignment (A) between them. All template filling metrics select the align- ment that produces the highest total score, summed over aligned reference and predicted templates TR and Tp. given a template similarity function di
Event Individuation At the core of template fill- ing is the question of event individuation: systems must determine when one event (i.e. template) is being described or several. This is profoundly difficult problem that long antedates the field of IE, with a rich history in philosophy and linguistics (Lemmon, 1967; Kim, 1973; Quine, 1985; Cleland, 1991; Davidson, 2001, i.a.). At this level, the prob- lem is well beyond IE's purview, as IE systems plainly should not be made to take hard line on vexed metaphysical questions.
A = argmax A'
Σ (Tn.T))AA'
@TTn. Tr)
(1)
and given the constraint that each reference tem- plate is aligned to most one predicted template with matching type.* 4 Metries differ mainly in their choice of Φb It is this reliance on alignments that makes individuation so critical: If individuation de- cisions in the reference templates are ill ill-grounded, this can yield punishingly large differences in score for otherwise semantically similar predictions.
Instead, for IE purposes, we might hope to find some rules of thumb for event individuation that agree broadly with human judgments, and that can be consistently applied during annotation. Work on template filling itself has had relatively little to say on such rules, although we highlight some relevant discussion from the MUC-4 proceedings
"With this constraint, finding the optimal alignment is thus a maximum bipartite matching problem. See Appendix B.
Details on each datasct can be found in Appendix A.

---

We illustrate this with the example document from MUC-4 shown in Figure and the gold tem- plates in Figure 2.3 Although arguably only one bombing is described the text, the annotations feature two templates - one for the bomb that detonated and one for the bomb that was deacti- vated. However, system that predicts only the left template will be heavily penalized for having "missed" the
BETTER
MUC-4
Type
a!
Type
Corruption
0.24
0.27
Arson
0.58
0.35
Disaster
0.21
0.21
Attack
0.44
0.42
Epidemie
0.36
0.28
Bombing
0.82
0.66
Migration
0.20
0.14
Kidnapping
0.81
0.73
Protest
0.13
0.07
Robbery'
1.00
1.00
Terrorism
0.30
0.23
the right, even though it identi- fies all relevant entities and assigns them the correct roles. Under the standard alignment- based metric for MUC-4, CEAF-REE (Du et al., 2021a), such a system would receive a score of only 57% F1.
Aggregate
0.25
0.22
Aggregate
0.63
0.56
Table Krippendorff's a for IAA on the number of templates of each type for the reannotated MUC-4 and BETTER documents. a = indicates full agreement; is agreement chance. "The sampling procedure excluded samples of forced work stoppage templates and yielded only three robbery templates, so robbery results should be taken with caution.
Thus, because systems are tasked with reproduc- ing particular (often idiosyncratic) individuation decisions, distinctions without much semantic dif- ference can reverberate as significant differences in score. Alignment-basee
procedure yielded 42 documents for MUC-4 and 60 for BETTER." ? Annotators were provided with complete descriptions of both ontologies, as well as relevant excerpts from the annotation guidelines. Importantly, these excerpts contained instructions for template individuation consistent with the crite- ria from Hovy al. (2013) discussed in $2.
metries therefore present a serious obstacle in principle to meaningful model comparison. But in practice, we will see that mod- els struggle to individuate effectively at all ($5).
4 Individuation Replicability
We consider the level of agreement among the three annotators alone (a), and when additionally incorporating the gold annotations (a+t). Table presents nominal Krippendorff's a (Krippendorff, 2018) in aggregate and by template type. While agreement on some MUC-4 template types is rea- sonably strong among annotators alone, we should expect this across the board. Instead, we find that aggregate agreement is at best moderate for MUC- 4 (a = 0.63) and consistently low for BETTER (a = 0.25). This is disheartening considering the annotators' qualifications and their access to the annotation guidelines. Most concerning, though, is that agreement is even lower when including the gold annotations (a+ = 0.56, 0.22). Thus, if human experts cannot reliably agree either among themselves or with the gold annotations, this sug- gests that the individuation problem is ill-posed.
One could argue that the metrics per se are not the problem, but rather questionable individuation decisions embodied in the annotations. If those decisions reflect real differences between events, then the difference between predicting one template and predicting two plausibly should correspond to large difference in score.
Accordingly, we next investigate whether these decisions are well-founded in MUC-4 and BET- TER. To do so, wc conduct a re-annotation study of a random subset of documents from each dataset and evaluate agreement between these new anno- tations and the original ones. If the original anno- tations reflect sound individuation decisions, then we would expect them to be recoverable with high accuracy by annotators versed in their ontologies.
Three of the authors of this paper, each an expe- rienced IE researcher, reannotated a subset of doc- uments from the MUC-4 and BETTER training splits for template detection only, indicating how many templates of each type are present in each document. Documents were selected via stratified sampling, with the goal of obtaining 10 documents annotated for each template type. Within type, we further stratified by number of templates. This
Do Models Learn to Individuate?
Given these observations, we now ask what the consequences are for our IE models. Even if ex- perts struggle to replicate the original individuation decisions, it possible that the datasets exhibit some internal consistency, and that a model could therefore learn their template count distributions. To explore this possibility, we consider three
*Other challenging examples are shown in Appendix "We note the oddity of the missing annotation of second device as a weapon, hut this is not important to the argument.
AAdiitional study details are in Appendix D.

---

recent top-performing models for template filling: ITERX (Chen al., 2023), the current state-of- the-art; GTT (Du et al., 2021b), the previous best model; and TEMPGEN (Huang et al., 2021), a third competitive system. We investigate the predictions of all three systems on the MUC-4 test split, as well as the predictions of ITERX on the BETTER test split, While some prior work has attempted complete taxonomy of template filling model errors (Das al., 2022), our concern here exclusively with detection problems.
BETTER
MUC-4
ITERX
GTT
TEMPGEN
(2okd] # [pred] (8)
34.4
34.5
32.0
37,5
of which lgold peod (%)
72.7
72.5
Tomp. Precision
89.7
79.5
81.1
82.7
Temp. Recall
74.5
39.7
Temp. [']
81.4
68.2
67.4
67.6
Combined (original)
30.0
Combined Score (collapsed)
29.7
51,4
48.8
46.9
Table Test set metrics for template filling models. Combined scores use CEAF-REE for MUC-4 and the official Granular score for BETTER. "Igold]" and "[pred]" denote number of gold and predicted templates.
First, we note that all three models make signifi- cant number of detection errors. Even the strongest detection model (GTT) still incorrectly predicts the total number of templates on 32% of MUC-4 test documents (Table 2, top row).
can be done? Here, we briefly consider several possible solutions, each with different tradcoffs.
6.1 Evaluation Aggregated By Type
One possibility is simply to make standard the eval- uation setting from $5 in which templates are ag- gregated by type. This presents clearer picture of whether systems are getting the big things right (distinguishing events of different types), but it gives up on difficult within-type individuation.
Second, we observe that the errors of all three models are overwhelmingly system misses: among documents with detection errors, I GTT, and TEMPGEN predicted fewer templates than the reference on 72.5%, 81.3%, and 80.0% of them, respectively (Table 2, second row). These findings are consistent not only with prior error analyses (Das et al., 2022)," but also with our observations thus far: if individuations from model's perspective, it can generally do no better than collate all entities relevant a particular event type into a single template.
6.2 Multiple Reference Individuations
second possibility is to annotate all plausible individuations, evaluating a system's 's predictions against the reference individuation that maximizes the score of those predictions - just as template filling metrics use the score-maximizing template alignment ($3). This has the advantage of accom- modating the natural plurality of ways events may be individuated without privileging any one of them. But it has the obvious disadvantage of requir- ing much more annotation. Moreover, although may often be clear what the set of "plausible" individuations is, this is by no means always teh case. For one, any event ontology will present irreducibly hard questions about whether particu- lar event descriptions meet the criteria for being labeled with particular type. For another, dis- tributivity whether and how an event distributes across its participants - can present additional challenges. For instance, from (1), we can infer that Alice, Bob, Cam, and Dana are each abducted. But do we really want to say that all 15 possible ways of grouping them into ki kidnapping templates are valid? Further, how should we handle cases involving aggregate participants, which may imply multiple events, but where the text offers basis for distinguishing them, as in (2)?
close to arbitrary
If this is in fact what these models are learning to do, then collapsing predicted templates into a single aggregate template per type (taking per-slot unions over their fillers) should not substantially alter overall scores. Doing this for all model predic- tions and recomputing scores bears this out: across the three models, there is an average drop of just over single point in overall score on MUC-4 and a mere 0.3 points on BETTER (Table 2, bottom two rows). By comparison, collapsing all gold tem- plates by type (not shown) yields a drop of 12.7 points in overall score (100- -> 87.3) on MUC-4 and 24.2 points on BETTER (100 75.8).
6 Solutions
The value of template filling as an IE benchmark is undermined both by datasets that make dubious individuation decisions and by metrics that sharply penalize deviations from those decisions. So what
"BETTER predictions are available only for ITERX. Ad- ditional model details Appendix E.
"See $7.2 of Das et al. 2022: "The main source of error for hoth carly and modern models is missing recall due to missing templates and missing role fillers."
(1) Alice, Bob, Cam, and Dana were abducted.
(2) Four residents were abducted in two days.

---

Such considerations may make aggregate evalua- tion ($6.1) look more attractive. But using multiple references is not an all-or-nothing solution: some plurality in annotation is certainly better than none.
proaches in this vein appear to be underdeveloped from the perspective of document-level event ex- traction. Specifically, questions about events tend to be posed independently from each other and also tend not to query for some of the properties we have highlighted as being important to individua- tion, such as irrealis and distributivity.
6.3 Subevent-Based Evaluation
A shortcoming of both $6.1 1 and $6.2 is that they largely side-step the issue of why a given individu- ation might be appealing. Particular individuation decisions are often motivated by important and un- controversial facts about events, and whether an annotator makes given decision hinges on how salient they believe those facts be. To return to Figure 1, there is no dispute that second bomb was deactivated by police. Rather, the dispute is about whether this is salient in a way that merits annotating a second bombing event. A perfectly reasonable reply to this question is: who cares? Presumably, what we actually care about as readers is that a second bomb was deactivated - not how best accommodate this fact in our ontology.
Combining these threads, we envision an im- proved approach QA for (document-level) event extraction that (1) is based on subevent ontologies, (2) poses questions about Individuation-relewaa properties, and (3) is conditional, in the sense that (a) questions about later subevents are conditioned on questions (and answers) about earlier ones and that (b) questions about roles are conditioned on a relevant subevent being identified. This would al- low IE researchers and end users to leverage the pabilities of LLMs to extract rich, natural language descriptions of events while dispensing with some of the individuation problems that we have seen arise with traditional template filling. Crucially, this could be evaluated as QA, without the need for template alignments (though questions must be associated with particular subevents). We illus- trate this proposal in Appendix F with example subevent ontology for bombing events and actual outputs from a QA dialogue with ChatGPT. 10
These considerations motivate a third solution: to focus instead on extraction of iite-graied events subevents of the coarse-grained events captured in ontologies like MUC-4 and BETTER. The advantage of this approach is that it eliminates a major source of disagreement about individua- tion - namely, the often unclear conditions that must be met for coarse- grasiedd events to obtain. A bombing-related subevent ontology, for instance, might have detonation subevent, whose condi- tions for obtaining are clearer by virtue of being more specific. We imagine that such ontologies would also cover the full gamut of activities that may occur before (e.g. device placement) and after (e.g- perpetrators' escape) the primary subevent(s) (e.g. detonation). In this way, subevent ontologies resemble scripts (Schank and Abelson, 1977) or narrative schemas (Chambers and Jurafsky, 2008 2009), and can capture much richer information about the temporal structure of events.
We stress that our proposal would not solve all problems raised here: most of the difficulties we highlight at the event level could in principle arise at the subevent level. Rather, our expectation is that they will do so much less often in practice and that subevents can enable much more detailed in- formation about the broader events they constitute.
Conclusion
This work has considered the problem of event in- dividuation - how to distinguish distinct events and its ramifications for document-level event ex- traction. We have shown both that models for this task struggle to replicate gold event counts ($5) and, more fundamentally, that even human experts fre- quently make different individuation decisions ($4). These considerations motivate alternative modes of evaluation for this task ($6), which we hope are adopted in future work. More generally, in the spirit of Hovy et al. (2013), we hope that our work reminds NLP researchers of the linguistic and con- ceptual complexity of events, and of the need to take this complexity seriously.
Moreover, focusing on subevents may render document-level event extraction more amenable to methods from a now well-established line of work that recasts (typically sentence-leved event extraction as question answering (QA; Liu al., 2020; Du and Cardie, 2020: Li et al., 2020; Holzen- berger et al., 2022)- methods that have become increasingly popular as the reading comprehen- sion abilities of large, pretrained language models (LLMss) have drastically improved. However, ap-
hhttpss//peena..o///ll///htttt

---

Limitations
necessarily representing the official policies, either expressed or implied, or endorsements of IARPA or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copy- right annotation therein.
Some researchers have framed structurally similar IE tasks notably, N-ary relation extraction as template filling (Huang et al., 2021; Chen et al., 2023; Das et al., 2022). However, since this paper concerns classic template filling, which focuses on events (Jurafsky and Martin, 2021), the issues we raise about event individuation do not obviously extend to relations. Individuation of relations has its own challenges, but insofar as relations are more abstract than events and not spatially or temporally localized, they may be more readily individuable solely on the basis of the entities they involve.
References
1992. Appendix A: Evaluation task description. In Fourth Message Understanding Conference (MUC- 4) Proceedings of a Conference Held in McLean, Virginia. June 16-18, 1992.
1992. Fourth Message Understanding Conference (MUC-4): Proceedings of a Conference Held in McLean, Virginia, June 16- 16-18, 1992.
Further, our analysis in general does not apply to other document-level IE tasks - notably, (multi- sentence) argument linking (also called event argu- ment extraction or implicit semantic role labeling; O'Gorman, 2019; Ebner et al., 2020; Gantt, 2021; Li al., 2021, i.a.) where events are, per the task definition, distinguished on the basis of lexi- cal triggers. In these cases, the questions we raise about individuation criteria do not arise.
Collin F, Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In 36th An- nual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 86-90, Montreal, Quebec, Canada. Association for Compu- tational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Ethics Statement
We do not believe this work raises any ethical is- sues beyond those already embodied in the data and models we analyze, all of which were introduced in earlier work. However, that is not to dismiss those issues. Beyond standard disclaimers about undesirable biases of LLMs (e.g. Brown et al., 2020), we note that MUC-4's focus on incidents of violence in Latin American nations (namely, Argentina, Bolivia, Chile, Colombia, Ecuador, El Salvador, Guatemala, Honduras, and Peru) may be liable to engender model biases that reflect negative stereotypes about people from or living in those countries. The BETTER corpus is significantly more diverse in its subject matter and appears less likely to bias models in similar way, though we do not have empirical results to support this claim.
Nathanael Chambers and Dan Jurafsky. 2008. Unsuper- vised learning of narrative event chains. Proceed- ings of ACI-088 HIT, pages 789-797, Columbus, Ohio. Association for Computational Linguistics.
Nathanacl Chambers and Dan Jurafsky. 2009. Unsu- pervised learning of narrative schemas and their par- ticipants. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 602-610, Suntec, Singapore. Association for Computational Linguis- tics.
Nathanael Chambers and Dan Jurafsky, 2011. Template- based information extraction without the templates. In Proceedings of the 49th Annual Meeting of the Association for Computational L Linguistics: Human Language Technologies, pages 976-986, Portland, Oregon, USA. Association for Computational Lin- guistics.
A Acknowledgments
We would like to thank Tim McKinnon, Mare Vi- lain, and members of the Center for Language and Speech Processing at Johns Hopkins for help- ful questions and comments on this work. This work was supported by IARPA BETTER (2019- 19051600000 600005) and NSF-BCS (2040831). The views and conclusions contained in this work are those of the authors and should not be interpreted as
Yunmo Chen, William Gantt, Weiwei Gu, Tongfei Chen, Aaron White, and Benjamin Van Durme. 2023. Itera- tive document-level information extraction via imita- tion learning. In Proceedings of the 17th Conference of the European Chapter of the Association for Com- patational Linguistics, pages 1858-1874, Dubrovnik, Croatia, Association for Computational Linguistics.

---

Nancy Chinchor. 1992. MUC-4 evaluation metries. In Fourth Message Understanding Conference (MUC- 4) Proceedings of a Conference Held in McLean Virginia, June 16- 1992.
Ralph Grishman. 2019.
'wenty-five years of infor- mation extraction. Natural Language Engineering. 25(6):677-662.
Ralph Grishman, John Sterling, and Catherine Macleod. 1992. New York University PROTEUS system: MUC-4 test results and analysis. In Fourth Message Understanding Conference (MUC-4): Proceedings of Conference Held in McLean, Virginia, June 16-18, 1992.
Carol Cleland. 1991. On the individuation of events. Synthese, pages 229-254.
Aliva Das, Xinya Du, Barry Wang. Kejian Shi, Jiayuan Gu, Thomas Porter, and Claire Cardie. 2022. Auto- matic error analysis for document-leve information extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 3960-3975, Dublin, Ireland. Association for Computational Linguistics.
Ralph Grishman and Beth Sundheim. 1996. Message Understanding Conference- 6: A brief history. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics.
Donald Davidson. 2001. Essays on Actions and Events: Philosophical Essays Volume 1. Clarendon Press.
Lynette Hirschman. 1992, An adjunct test for discourse processing in MUC-4, In Fourth Message Under- standing Conference (MUC-4): Proceedings of a Conference Held in McLean, Virginia, June 16-18, 1992.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Nils Holzenberger, Yunmo Chen, and Benjamin Van Durme. 2022. Asking the right questions in low resource template extraction. arXiv preprint arXiv:2205. 12643..
Eduard Hovy, Teruko Mitamura, Felisa Verdejo, Jun Araki, and Andrew Philpot. 2013. Events are not simple: Identity, non-identity, and quasi-identity. In Workshop Events: Definition, Detection, Coref- erence, and Representation, pages 21-28, Atlanta, Georgia. Association for Computational Linguistics.
Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 671 1-13, Online. Association for Computational Linguisties.
Hring-siing Huang, Sam Tang, and Nanyun Peng. 2021. Document-leved entity based extraction as tem- plate generation. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 5257-5269, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Xinya Du, Alexander Rush, and Claire Cardie. 2021a, GRIT: Generative role-filler transformers for document-level event entity extraction. In Proceed- ings of the 16th Conference of the European Chap- ter of the Association for Computational Linguisties: Main Volume, pages 634-644, Online, Association for Computational Linguistics.
Ruihong Huang and Ellen Riloff. 2011. Peeling back the layers: Detecting event role fillers in secondary contexts. In Proceedings of the 49th Anmual Meet- ing of the Association for Computational Linguistics: Human Language Technologies, pages 1137-1 147, Portland, Oregon, USA. Association for Computa- tional Linguistics.
Xinya Du, Alexander Rush, and Claire Cardie. 202 Template filling with generative transformers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 909-914, Online. Association for Computa- tional Linguisties.
Ruihong Huang and Ellen Riloff. 2012. Modeling tex- tual cohesion for event extraction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pages 1664-1670.
Seth Ebner. Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020. Multi-sentence ar- gument linking. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 1057-8077, Online. Association for Computational Linguistics.
Dan Jurafsky and James Martin. 2021. Speech and Language Processing. Draft, 3rd ed.
Joseph L Fleiss. 1971. Mcasuring nominal scale agrec- ment among many raters. Psychological bulletin, 76(5):378.
Jacgwon Kim. 1973. Causation, nomic subsumption, and the concept event. The Journal of Philosophy, 70(8):217-236.
William Gantt. 2021. Argument linking: A survey and forecast. arXiv preprint arXiv:2107.08523
Klaus Krippendorff. 2018 Content analysis: An intro- duction to its methodology. Sage publications.

---

Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quar- terly, 2(1-2):83997
Roger C Schank and Robert Abelson. 1977. Scripts, plans, goals, and understanding: an inquiry into hu- man knowledge structures. Hillsdale, NJ: Lawrence Erlbaum Associates.
John Lemmon. 1967. Comments on d. davidson's 's 'the logic form of action sentences'. In Nicholas Rescher, editor, The Logic of Decision and Action, pages 96- 103. University of Pittsburgh Press, Pittsburgh.
Ian Soboroff. 2023. The better cross-language datasets, In Proceedings of the 46th International ACM SI- GIR Conference Research and Development in Information Retrieval, pages 3047-3053.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training
Beth M. Sundheim. 1992. Overview of the fourth Mes- sage Understanding Evaluation and Conference. In Fourth Message Understanding Conference (MUC- 4): Proceedings of a Conference Held in McLean, Virginia, June 16- 18, 1992.
BART: Denoising sequence-to-sequene for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
Fayuan Li, Weihua Peng, Yuguang Chen, Quan Wang, Lu Pan, Yajuan Lyu, and Yong Zhu. 2020. Event extraction multi-turn question answering. In Find- ings of the Association for Computational Linguisties: EMNLP 2020, pages $29-838, Online. Association for Computational Linguistics.
Sha Li, Heng Ji, and Jiawei Han. 2021. Document-level event argument extraction by conditional generation. Proceedings of 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 894-908 908, Online. Association for Computa- tional Linguistics.
Jian Liu, Yubo Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. 2020. Event extraction as machine reading com- prehension. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 1641-1651, Online. Association for Computational Linguistics.
James Munkres. 1957. Algorithms for the assignment
and transportation problems. Journal of the sociery for industrial and applied mathematics, 5(1):32-38.
Timothy J O'Goran 2019. Bringing together com- putational and linguistic models implicit role in- terpretation. Ph.D. thesis, University of Colorado at Boulder.
Siddharth Patwardhan and Ellen Riloff. 2009. unified model of phrasal and sentential evidence for infor- mation extraction. In Proceedings of the 2009 Con- ference on Empirical Methods Natural Language Processing, pages 151-160, Singapore. Association for Computational Linguistics.
Willard Van Orman Quine. 1985. Events and reification. Actions and events: Perspectives on the philosophy of Donald Davidson, pages 162-171.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, (11):5485-555)

---

A Dataset Details
This appendix provides more detailed information on the MUC-4 and BETTER Granular datasets and on their guidelines for individuating templates.
A.I
MUC-
A.1.1 Ontology and Dataset Statistics
Train
Dev
Test
documents
1,300
200
200
documents w/ templates
700
116
126
total templates
1,114
191
209
attack
636
97
127
bombing
298
50
55
kidnapping
114
31
14
arson
43
8
3
robbery
17
3
1
forced work stoppage
6
1
4
Table 3: Summary statistics of the MUC-4 dataset. Included in the "total templates" figures are several templates that cross-classified both tack and bombing. There such template in dev and five test.
MUC-4 features six template types: Arson, Attack, Bombing, Forced Work Stoppage, Kidnapping. and Robbery. All template types share the same set of five slots: PerpInd (an individual perpetrator), PerpÖrg (a group or organizational perpetrator), Victim (sentient victims of the incident), Target (physical objects targeted by the incident), and Weapon (weapons employed by the perpetrators). All five slots accept entity-valued fillers, represented by sets of coreferent entity mentions, though systems need only predict of those mentions to receive full credit. The type of each template instance is specified sixth slot, incident type. Complete dataset statistics are provided in Table 3. We use the preprocessed version of MUC-4 available here: http:://ttt..t.tcctotttttttttttt The original (raw) data is available here: http::////wmmmiirr.....././////r html.
A.1.2 Multi-Template Annotation
Below is an excerpt from the MUC-4 dataset documentation, describing the conditions under which multiple templates are be annotated for document. This excerpt, along with those included the instructions for the reannotation study (see Appendix D), were drawn from nn- (1992). Note that the criteria for annotating multiple instances here closely align with those presented in $2 from Hovy et al. (2013). Note also that although the location, date, and category slots are annotated in the original data, they are not included in the evaluated templates. The location and date slots are self-explanatory; the category slot indicates whether not the event is instance of "state-sponsored violence," meaning that the perpetrator (PerpInd PerpOrg) is (a member of) the government, military, police.
If an article discusses more than one instance of the same relevant type of terrorist incident, each such incident should be captured in separate template. A "separate instance" is one which has a different filler for the location, date, category, or perpetrator slot. If no distinctions of these kinds are made in the text, the instances should be incorporated into single template, e.g., if the article describes "attacks" by "rebels" on "ten power stations" and does not give varying information on the location of those targets, etc. Note that the level of granularity defined by
Il Theee slots are the ones rescarchers have focused on since the end of the original MUC-4 evaluation, though the full ontology actually features 24 slots, of which these five (as well as the incident type slot) are a subset. See either of the data links above for more details.

---

1ST
1MT
sent/tamp temppmma (00)
M sentitemp
per mag
temp/msg (0t)
>> temp / mag
so temp mag
NST
2MT
sent/temp
M sent/temp
N temphmsg
temp/msg
(9t)
per mag
(101)
Figure 3: Four subsets of the MUC-4 test data evaluated in Hirschman (1992), reproduced exactly from that work.
what can go in slot; thus, an article that describes two bombings on targets different streets in the same neighborhood should be captured in one template rather than two, since the location slot cannot have filler that is the level of granularity of street.
The bombing example discussed in this passage is particularly interesting when compared to the bombing described in Figure 1 and annotated in Figure 2. According to the principle outlined here, it certainly seems though the document in Figure should have single bombing template annotated, rather than two, no distinctions between the annotated bombings can be drawn the basis of the location, date, category, PerpInd, or PerpÖrg slots.
Alongside location, date, and category, an Trreals--fcussed stage of execution slot was annotated for the original data, and indicated whether an event was accomplished. attempted, or only threatened. This is the basis for the distinction between the templates in Figure 2. The existence of these annotations is good news in that they may help in a recasting effort of the sort proposed in $7, but bad news in that they may form the basis for an individuation (e.g. Figure 2) without actually being part of the task.
A.1.3 Event Individuation in the MUC-4 Proceedings
Beyond the quoted passage above, there is not much substantive discussion of individuation criteria in the MUC-4 proceedings themselves (muc, 1992) - and certainly not at the level of analysis we aim to provide in this work. However, both participants and organizers recognized the need for principled technical approach to individuating events. Some of this discussion focuses on the closely related difficulty of defining what counts relevant terrorist incident. For instance, in considering shortcomings of the task, Sundheim (1992) notes: "The definition of 'relevant terrorist incident' inadequate in several respects. The distinction between a terrorist incident and other kinds violent events - guerrilla actions, shootings of drug traffickers, etc. - is a difficult one to express comprehensive, hard and fast rules." Much of the proceedings discussion on this subject focuses purely at a practical level, describing the individuation techniques and heuristics different teams applied - typically under the heading of "discourse analysis" or "discourse processing." For instance, the NYU team's 's PROTEUS system adopted a strategy used by many teams, which involved detecting templates at the sentence level and then attempting merge them according certain criteria (Grishman et al., 1992). For NYU, these criteria included (among others) merging (1) when events affected the same target; (2) when events were described in the same sentence; or (3) when one event described an attack and the next event described an effect.
Another example of this practical focus Hirschman (1992), who presents a study of systems' performance on different subsets of the test data, created based on (1) the number of templates in a document, and (2) the number of sentences required to fully populate a given template. The cross product of these variables defines the four subsets of the test data shown in Figure 3. The aim of the study was to evaluate so-called "lazy merger" problems (i.c. overgeneration, failing to merge templates that ought to be merged) and "greedy merger problems" (i.e. undergeneration. merging templates that ought not to be merged). The four subsets were defined to target examination of particular combinations of these two problems, with the "1MT" documents being subject only to lazy merger; "NST" documents being subject only to greedy merger; and "IST" and "2MT" sets being subject to both. Interestingly, she observes that on average, overall score was worst for systems on the 1ST subset (with only one template

---

Phase
Phase 1I
Train
Dev
Test
Train
Dev
Test
documents
94
20
20
168
34
32
documents w/ templates
88
19
19
168
34
32
templates
252
56
47
255
57
57
corruption
63
10
11
3
0
0
disaster'
0
0
0
68
11
11
epidemic
31
7
1
76
23
23
migration'
0
0
0
81
20
20
protest
109
37
25
terrorism
49
2
15
1
Table 4: Summary statistics of the Phase I and Phase II BETTER Granular datasets. type is part of the Phase II ontology only.
Denotes that this template
per document), suggesting that relevancy judgments (at least at the time) were an even greater challenge than individuation. However, the subsets are quite small and the results should be interpreted with caution.
A.2 BETTER
A.2.1
Ontology and Dataset Statistics
We use the BETTER Granular Phase and Phase II datasets, which collectively feature six template types that exhibit a greater diversity of topics than the MUC-4 templates: Corruplate (incidents of corruption), Disasterplate (natural disasters), Displacementplate (human migration events), Epidemipla pieempppltte (epi- demics), Protestplate (protests), and Terrorplate (acts terrorism). While some slots are common to two or more of these template types, most slots are unique to each type, reflecting event-specific roles, similar to those in FrameNet (Baker et al., 1998). Complete dataset statistics are provided in Table 4. We use the versions of the data provided here: https://ir.nis west..oobbeettr..////
A.2.2 Multi-Template Annotation
As the BETTER documentation has not been made public, we are unable to quote from it, though we will attempt to summarize the relevant considerations for multi-template annotation. Like in MUC-4, the BETTER documentation indicates that multiple templates may be annotated when the times or locations of the events described differ from each other. Some irrealis and time annotations are provided, but these are associated with specific fillers of event-valued slots, and are not used distinguish whole templates, the stage of execution in slot is for MUC-4. Multiple templates of the type may also be required when those templates apply to different people (e.g. different individuals accused of corruption). However, this criterion is somewhat opaque, as many templates feature multiple individuals as fillers of the same slot, and it is unclear exactly when this condition merits individuating a given set of events.
B Evaluation Metrics
Due to space limitations, definitions of the alignment-based evaluation metrics for MUC-4 and BETTER are omitted in the main text. We refer the reader to Appendix in Chen et al. (2023) for thorough descriptions and discussion of these metrics, but will briefly present them here.
In $3, we observe that all template filling metrics determine an alignment between system system-predicted and reference templates that maximizes overall score (Eq. 1), which decomposes scores for aligned template pairs (Tr, Tn). Recall that this alignment is subject to two constraints: (1) At most one predicted template may be aligned to each reference template, and (2) aligned templates must be of the same type. Constraint (1) makes this a maximum bipartite matching problem, typically solved with the Kuhn-Munkres algorithm (Kuhn, 1955: Munkres, 1957). Note that Tp can be null (for missed reference templates), as can T (for spuriously predicted templates). In either case, @TTp Tn) = 0.
sum over similarity

---

For the original MUC-4 evaluation, ø was essentially Fl score over all slot fillers in given template. A predicted slot filler counted as correct iff it was featured among the reference fillers for the corresponding slot an aligned template. 12 All fillers of spuriously predicted templates were counted as false positives and all fillers of missed reference templates were counted as false negatives (Chinchor, 1992). As noted in footnote 11, the set of slots included not only those shown in Figure 2, but a number of others that were annotated but for which results have not been reported since the original evaluation.
More recent reporting on MUC-4 has used the CEAF-REE score (Du et al., 2021a,b: Huang et al.. 2021; Chen et al., 2023). Introduced by Du et al. (2021a), CEAF-REE is similar to the slot-filler Fl score used in the original MUC-4 evaluation, except that evaluation is based only on the five slots in Figure 2, plus the template type (treated as sixth slot). Chen et al. (2023) present variant of CEAF-REE, dubbed CEAF-RME, which corrects certain problems they identify with the original metric. To our knowledge, they are the only ones to have reported CEAF-RME scores.
The official scoring metric for BETTER differs from both the original MUC-4 slot filler Fl and CEAF-REE in that it reflects the product of slot-level and template-level F1 scores, conditional on the optimal alignment. This score does not decompose over template pairs and so cannot be directly optimized via maximum bipartite matching. Instead, the BETTER scorer optimizes quantity called response gain; the difference between the number of correct and incorrect slot fills under given alignment. Optimizing alignments for this objective is theoretically guaranteed to yield the optimal alignment under the official score within a small probabilistic error bound. For more details on this metric, see Chen et al. (2023).
Additional Difficult Examples
Lest the example presented in Figure appear cherry-pickeds we present here additional examples of documents from both datasets that present challenging individuation decisions and that similarly strike lacking a single, obviously correct answer. Such examples are abundant.
C.1 MUC-4 Examples
We note that the MUC-4 source texts is uncased; we have cased the ones below simply for readability.
Example
Document Text:
Two vehicles were destroyed and unidentified office of the Agriculture and Livestock Ministry was heavily damaged following the explosion of bombs yesterday afternoon. The National Fire Department reported that following the explosions at 1830, a fire erupted that partially destroyed an unidentified office in the Agriculture and Livestock Ministry building. It was reported that two vehicles in the area where the bombs were detonated were destroyed and several houses in the area were damaged. The damaged offices are located at 123 A Avenue in the San José neighborhood in the western section of San Salvador. Fortunately, there were no casualties reported as result of this incident, for which the FMLN guerrillas are being held responsible.
Annotated Templates: 1 bombing template
Discussion: This example provides interesting foil for the one in Figure I and Figure 2. Here, just in that earlier example, two bombs involved. But unlike the earlier example, only single template is annotated. From the text, appears that the bombs were likely planted near each other, but so too were the pair of bombs in Figure 1 in the same restaurant, fact. The only distinguishing feature between these examples is the fact that in Figure I, only one the bombs actually detonated. This an important fact, but whether one actual detonation and one non non-detonation makes for two "bombings" is dubious and is hardly the only reasonable reading, as we discuss in the main text. Moreover, were the second bomb
12 "MUC-4 documentation draws a distinction between "string- fill" slots - those that are filled by entities, represented by single canonical mention - and "sct-fill' slots - those that are filled by one of a fixed set of possible values (Chinchor, 1992). Determining a match between system and reference slot-fillers for set-fill slots is straightforwardd However, to determine a match for string-fill slots, the original evaluation applicd various fuzzy string matching heuristics.

---

actually have detonated, the annotation guidelines (Appendix A) imply that we would instead (again) have a single bombing, owing the equivalence between all the relevant slots in the two templates. This is idiosyncratic at the very least.
Example 2
Document Text:
Military sources reported today that fighting broke out on the night of 5 March in the war-torn Chalatenango department to the north of San Salvador, where the guerrillas suffered eight casualties. The Armed Forces Press Committee reported that the clashes took place near San Isidro Labrador, where three guerrillas were killed and five others were wounded, The Farabundo Marti National Liberation front (FMLN), which has been fighting the U.S.-backed army for the last 10 years, has traditionally maintained a high profile in strategic areas of Chalatenango. Meanwhile, urban guerrilla commandos harassed troops belonging to the 1st Infantry Brigade last night at the Mariona Penitentiary, located on the northern outskirts of San Salvador, apparently leaving no casualtics. Atlácatl battalion troops, which maintain counterinsurgency operation at the foothills of Guazapa Hill, to the north of San Salvador, confiscated war materiél that the guerrillas had hidden underground. In another turn of events, FMLN sappers who continuously sabotage the electrical power system destroyed several high- voltage posts in La Libertad and Cabanas departments in the central part of the country, and in Usulután department in the southeastern part of the country. There were widespread power outages in the three departments while technicians of the state-run electric power companies repaired the damage. The army and security corps deployed heavy patrols in San Salvador and in cities in the country's s interior to try and counter the wave of attacks.
Annotated Templates: 3 attack templates
Discussion: For MUC-4, attack catch-all type to handle terrorist acts that are not adequately described by any the other types, which makes particularly challenging to identify. Furthermore, the documentation specifies that violent acts directed against terrorists are not to be annotated. This rule evidently excludes the eight guerrilla casualties described here, though it's not certain from the text that the guerrillas are terrorists. The three attack templates refer instead to the FMLN attacks on "high-voltage posts," with one template per location (department) in which posts were destroyed (La Libertad, Cabanas, and Usulután). This is sensible, though one of the annotators in our study (not unreasonably) considered these to be part of the same coordinated attack on the power system and annotated a single template.
C.2 BETTER Granular Examples
Example
Document Text:
President Cyril Ramaphosa has urged South Africans to "act responsibly" as the second wave of the Covid-19 pandemic gripped two provinces, with fears that it will sweep through the rest of the country during end of year celebrations.
His worried Health Minister, Dr Zweli Mkhize, Saturday night 'raided' popular Cape Town bars and nightspots where merrymakers were breaking lockdown rules social distancing, mask wearing and staying safe.
What is bothering the South African government is the rapid rise in new cases in both the Eastern Cape Province where the second wave of the virus has a full grip and is overwhelming with as a similar pattern emerging in the Garden Route of the Western Cape, which borders the Eastern Cape, as well as in greater Cape Town, capital of the Western Cape.
The fear that the wave will prove unstoppable amid end of year celebrations and family gatherings which are bound to drive infections up further, threatening to overwhelm South Africa's 's already struggling medical system.

---

Overnight, South Africa recorded another 4,932 new Covid-19 infections and at least 160 more deaths. The pandemic has so far claimed over 21,000 lives.
Ramaphosa, speaking to the nation prior to his health minister's on-the-ground inspection of potential "superspreader" events and venues, had announced a return to tougher emergency restrictions in the major urban area of the Eastern Cape, the Nelson Mandela Bay metro, where conditions are severe and from where nearly half of all new infections are coming.
But South Africa health authorities are moving away from the idea of sweeping new lockdowns, focusing instead on hotspots and attempring to stop the spread by restricting alcohol sales, imposing tighter curfew between 10pm and 4am and reducing the numbers of people attending open-air and indoor venues.
Annotated Templates: Epidemiplate
Discussion: This document describes just a single disease (Covid-19), but it's far from clear that it describes single epidemic. In contrast to pandemics, which are global disease outbreaks, epidemics are geographically localized, Considering the individuation criteria for BETTER (Appendix A), we are justified in annotating multiple epidemiplates there are discase outbreaks in distinct locations. The article describes outbreaks in both the Eastern Cape and Western Cape Provinces of South Africa. But is this one location (South Africa) two (the two provinces)? Both seem reasonable. One could even argue that the emergence of the virus's second wave in the Garden Route and in Cape Town both within the Western Cape Province constitute separate epidemics of their own.
Example 2
Document Text:
Uganda's 's Foreign Affairs Minister Sam Kutesa and two other officials have denied cortuption charges. Mr Kutesa, chief whip John Nasasira and junior labour minister Mwesigwa Rukutana appeared in court day after they resigned. They are accused of abuse of office as well as financial loss over the 2007 Commonwealth summit in Uganda, in which scams allegedly cost some $150m (E95m). Former Piee-President Gilbert Bukenya was charged in July, He denies that he benefited from a $3.9m deal to supply cars used to transport dozens of heads of state during the summit. Mr Kutesa was also accused on Monday of taking large bribes from UK-based Tullow Oil. The minister and the company strongly denied the allegation. In court, prosecutor Sydncy Asubo said the three had cost the government 14bn shillings ($4.8m; f3. Im). "The three irregularly convened consultative cabinet meeting and decided that the government would fully fund the construction cost of driveways. parking areas and marina at Munyonyo Speke Resort," he said. They face up to 13 years in jail if convicted, reports the AFP news agency. Some MPs from Mr Bukenya's Buganda ethnic group had accused the government of selective justice by failing to prosecute anyone else. Last week, the Inspectorate of Government the body charged with fighting corruption said Mr Kutesa, Mr Nasasira and Mr Rukutana would be charged. President Yoweri Museveni said on Wednesday that the three officials had chosen to resign. "That's their decision because what we want is the truth," he told news conference. Mr Museveni sacked Mr Bukenya in May as part of a cabinet reshuffle.
Annotated Templates: 4 Corruplates
Discussion: The article describes four individuals accused of corruption: Sam Kutesa, John Nasasira, Mwesigwa Rukutana, and Gilbert Bukenya. For this reason, it's understandable that the annotations would feature four corruption templates one per person charged. However, the first three of the four seem be implicated in the same corruption charges. Those charges could reasonably be counted incident of corruption involving multiple actors, yielding two templates overall (or three if including Mr. Kutesa's 's bribery charge as a separate incident). This was a case in which there was complete disagreement among annotators in our reannotation study, who annotated two, three, and four Corruplates.
single

---

D Reannotation Study: Further Details
The reannotation study was designed and run by the first author, who did not participate in the annota- tion. The annotators were the three middle authors and are all fluent English speakers with strong IE backgrounds ($4). They worked independently over the course of several days and were not permitted to discuss any aspect of the task with each other or with the first author. Moreover, the authors did not discuss theories of, or approaches to, event individuation prior to beginning the study. Annotators were instructed to rely only on the guidelines provided in the study instructions. All annotation was unpaid.
In order to focus the task on within-type individuation, and to make it as straightforward possible, documents were required to have annotated gold templates of only a single type. As discussed in $4, we aimed to obtain 10 documents for each type, for each dataset. Given the single type constraint, this was not always possible - as was the case for the robbery and forced work stoppage templates in MUC-4 (see Table 1). For these cases, we selected as many documents as we could that satisfied the single type constraint, which resulted in three documents for robbery, and unfortunately none for forced work stoppage. Although it would be nice to have more documents for these types, we do not think this detracts from the general result that even experts struggle to agree on within-type individuation decisions.
The complete instructions for the MUC-4 reannotation study can be found in the GitHub repository associated with this paper (see footnote 2).
For BETTER, we selected documents for Disasterplates and Displacementplaare plates from the Phase 1I data, as the Phase data does not contain templates of this type. However, we selected all documents for Corruplates, Epidemiplates, Protestplates, and Terrorplates from the Phase data, as the Phase II data had comparatively few documents annotated with these types. As the reannotation study instructions quote from the BETTER program documentation (which is nonpublic at the time of writing), we are not able to include these instructions.
Finally, when we say in $4 that we stratify by the number of templates within type, we mean that we first group all documents annotated exclusively with templates of type T by the number of templates annotated, then sample round robin from each group, and uniformly at random within group.
In the main text, the (dis)agreement analysis uses the nominal version of Krippendorff's a, which is equivalent to Fleiss's is (Fleiss, 1971) in settings like ours in which the same set of annotators annotate all items. This is meant to show the level agreement when considering only whether annotators agree on the number of events described, which is our main concern. By computing the ratio version of the a instead of the nominal, we may also take into account the magnitude of the difference in the number of annotated events. In general, one would expect the magnitudes of these differences to be small: although annotators clearly disagree on the precise number of events (see Table 1). the text will usually constrain them not to give substantially different counts. This is borne out in Table 5, which shows the ratio form of As expected, the aggregate agreement is higher here for both datasets, though we stress that it still far from what one would want from expert annotators (a: and from the data (a f.))
The high agreement observed on a' among annotators for select MUC-4 template types suggests that some types of event (c.g. kidnappings) may just be more readily individuable than others- perhaps merely that the documents annotated with these types happen to present especially clear-cut cases. We again emphasize that this does not impugn the broader point, supported by these results, that individuation is clearly challenging even for experts intimately familiar with the data.
Finally, concerning the observed drop in agreement for many template types when including the gold data, we acknowledge the possibility that this is largely due to differences in annotator training between our reannotation study and the original annotation. For instance, it's plausible that the original annotators informally agreed on certain annotation rules that were never made public, or resolved divergent annotations in dialogue among themselves. However, the possible existence of such consensus mechanisms also does not detract from the conclusions of our study, which explicitly interrogates how expert annotators independently approach this task, given the available documentation.

---

BETTER
MUC-4
Type
at
Type
a
a+
Corruption
0.23
0.27
Arson
0.48
0.35
Disaster
0.26
0.31
Attack
0.39
0.32
Epidemic
0.24
0.29
Bombing
0.95
0.68
Migration
0.09
0.07
Kidnapping
0.99
0.96
Protest
0.50
0.51
Robbery"
1.00
1.00
Terrorism
0.35
0.40
Aggregate
0.33
0.38
Aggregate
0.68
0.66
that for Table 5: The ratio version of Krippendorf"'s a, measuring IAA on the number of templates of each type for the reannotated MUC-4 and for BETTER documents.
BETTER
MUC-4
Type
ITERX
Type
ITERX
GTT
TEMPGEN
Corruption
0.00
Arson"
-0.25
-0.11
0.00
Disaster
0.24
Attack
0.08
-0.01
-0.14
Epidemic
0.64
Bombing
0.03
0.03
0.21
Migration
-0.06
Kidnapping
-0.20
0.14
0.00
Protest
0.00
Robbery'
0.00
0.00
0.00
Terrorism
0.00
Aggregate 0.25
Aggregate
-0.05
0.16
-0.02
Table 6: Nominal Krippendorff's a between gold template counts and counts as predicted by each model in $5 for all test set documents with at least one annotated template. Agreement is poor across the board, reinforeing our conclusion that template filling models do not effectively learn template count distributions. 'Indicates that there are fewer than documents in the test split containing annotated templates this type and corresponding results should be read with caution.
E
Model Details
The analysis in is based on the GTT model released by Du al. (2021b), which uses uncased BERT- base (Devlin et al., 2019) as its encoder." The ITERX model the one from Chen et (2023) that uses a T5-large encoder (Raffel et al., 2020), which showed the best results across the datasets presented that paper. The TEMPGEN model is also from Chen et al. (2023), uses BART-large as the encoder (Lewis et al., 2020), and is lightly adapted from Huang et al. (2021) to support multi-template prediction. 14 Chen et al.'s 's source code is available here: https: //gi: thub. cwm/ hoom/wanokk// itexx.
To drive home the extent to which these models struggle to learn template count distributions (see $5), we report the level of agreement between their template count predictions on the MUC-4 and BETTER test sets and the template counts from the gold annotations Table 6, computed on the subset test set documents with at least one gold template of the corresponding type. Results for template types with minimal test set support (indicated by ") should be interpreted with caution, but agreement is generally very low and in many cases not much different from chance. With the caveat that the statistics in Table and Table 6 computed different documents, it is noteworthy and disheartening
BETTER, the level agreement between the gold annotations and ITERX (0.25; see Table 6) actually exceeds that between the gold annotations and the human experts (0.22: see Table 1).
1/GTT model code and predictions are publicly available here: https://gi tht.ghhhuh.coooo coto.coo/iiiii himm/xiiyadu/gtt.. 16 "Model code and predictions for the original, unadapted TEMPGEN model are available here: https://githun com/ MusLLabLPPTTeppeer

---

Subevent Name
Phose
Suhevent Description
Motivation
The perpetrators formulate or express motive or reason for carrying out the bombing. The perpetrators formulate plans for the hombing (e-g. selecting largel, working the logistics)..
Planning
Preparations
Before
The perpetrators gather materials (e.g g. the explosives)
information needed to carry out the bombing.
Transportation
The perpetrators are transported, along with any necessary equipment, to the site of the bombing.
Placement
The perpetrators place the explosive devices
locations where they are later detonated.
Activation
During
The explosive devices that were placed detonate, possibly harming people or infrastructure.
Emergency Response
Emergency personnel respond directly following the bombing.
Investigation
After
Authorities carry out an investigation into the bombing.
Appechension
The (suspected) perpetrators of the bombing are arrested apprchended by authoritics.
Table 7: A possible subevent ontology for the MUC-4 bombing template.
F Subevent-Based Question Answering
Here, we sketch how our reframing of document-level event extraction as babeevent-basce QA (proposed in $7) could be carried out. As the aim of this paper is principally to highlight and explain the problem of event individuation, we stress that this is a sketch, and that fully detailed solution is left for future work. We also think that subevent decompositions are valuable and interesting independent of our particular QA formulation; other approaches to subevent-basee extraction are worth exploring.
To illustrate our proposal, we show how it could be implemented for the MUC-4 bombing event type. We imagine that the bombing event type could be broken down into the set of nine subevent types listed in Table 7. Many other subevent decompositions are possible; we present this one merely as an example. In $6, we note that the notion of subevent decomposition Abelson, 1977) and narrative schemas (Chambers and Jurafsky, 2008, 2009), though remain agnostic on certain questions of implementation where these proposals do not (c.g. we do not necessarily think subevents have to be associated with individual verbs, as events are in narrative schemas).
advocate for resembles scripts (Schank and
We demonstrate extraction against our subevent ontology on the document from Figure using g.t-3. 5-turbo, accessed via the OpenAl Playground.5 15 For all queries, we use the following system prompt and the default Playground hyperparameters:
You are very good at understanding and answering questions about events described in docu- ments. will ask you reading comprehension questions about document and I want you to answer them directly and succinetly. Your answer should be based ONLY on what is explicitly said in the document. DO NOT answer based on what is merely implied.
We break the task down into a subevent detection step, in which we pose questions to determine which bombing subevent types are attested in the document, followed by subevent identification and subevent argument extraction steps, in which we pose questions to determine the number of instances of those subevent types and the arguments of each instance." 16 For subevent detection questions, the responses can generally be determined from the first word ("Yes" or "No"). But for subevent individuation and argument extraction, some parsing of the output would be required to pull out the responses; we leave this (or possible alternative question design) for future work here just illustrating that ChatGPT can generally produce correct answers.
F.1 Subevent Detection
We start by posing a series of questions, one per subevent type, aimed at determining which subevent types are attested in the document. In the spirit of conditional QA ($6), each question is conditioned on the complete dialogue history and more detailed questions about the number of instances of particular subevent type and the participants in those subevents will later be conditioned on positive responses these initial questions (more below). The article is presented as the initial context.
1 /htps::/plltformm wwwpenoppeeiiiooooo com//payggoouddd
i6 "'Once again, this is by no means the only way to do things. For instance, subevent detection and identification could he rolled into one step.

---

Dialogue turns are indicated by User or Assistant. Subevents are queried in the order they are given in Table 7 and the subevent queried on a particular turn indicated with purple // comments. By our own estimation, ChatGPT does this perfectly, correctly identifying that the document describes instances of both placement and activation subevents, but does not describe instances any of the other types.
User: Article: "A bomb exploded today in Lima restaurant, and a second device that had been placed in the same establishment was deactivated by the Peruvian National Police. There were no victims, and the explosion caused very little damage to the restaurant, which is located in the commercial area of the residential district of Miraflores. Guerrillas of the Tupac Amaru Revolutionary Movement (MRTA) have claimed credit for the terrorist through pamphlets they left on the premises, according the police."
Does this document explicitly describe any instances of a person or group expressing or formu- lating reasons for carrying out bombing? // motivation
Assistant: No, the document does not explicitly describe any instances of person or group ex- pressing formulating reasons for carrying bombing.
Does this document explicitly describe any instances of person or group formulating plans to carry out a bombing, before actually doing so? // planning
User:
Assistant: No, the document does not explicitly describe any instances of a person or group formulating plans to carry out a bombing before actually doing so.
User: Does this document explicitly describe person or group gathering materials or informa-
tion in preparation to earry bombing, before actually doing so? preparations
Assistant: No, the document does not explicitly describe a person or group gathering materials or information in preparation to carry out a bombing before actually doing so.
Does this document explicitly describe any persons or groups being transported to some location to carry out bombing? // transportation
User:
Assistant: No, the document does not explicitly describe any persons groups being trans- ported to some location carry bombing.
User:
Does this document explicitly describe explosives being placed or planted in some target location? // placement
Assistant: Yes, the document explicitly describes a bomb exploding in a Lima restaurant and mentions that second device was placed in the same establishment but was deactivated by the Peruvian National Police.
User:
Does this document explicitly describe an explosive device going off being triggered? // activation
Assistant: Yes, the document explicitly describes bomb exploding in Lima restaurant.
Does this document explicitly describe any emergency or support personnel responding to an explosion or the detonation of an explosive device? // emergency response
User:

---

Assistant: No, the document does explicitly describe any emergency or support personnel responding to an explosion or the detonation of an explosive device.
User: Does this document explicitly describe any persons or groups investigating an explosion or bombing? // investigation
Assistant: No. the document does not explicitly describe any persons groups investigating explosion or bombing.
Does this document explicitly describe any persons or groups being arrested or appre- hended for having planted or activated an explosive device? // apprehension
User:
Assistant: No, the document does not explicitly describe any persons or groups being arrested or apprehended for having planted or activated an explosive device.
F.2 Subevent Individuation and Argument Extraction
Above, in the subevent detection step, ChatGPT (correctly) detected that two subevent types from the ontology, placements and activations, are attested in the document. We imagine that more detailed questions about these event types, including how many instances of them are described and who the participants are, would be conditioned on the relevant questions and (affirmative) answers from the subevent detection step.
For brevity, we illustrate this below for just one of the identified subevent types (placement). The full document text is again provided to the model as the initial context (omitted for brevity below), but here we also include the QA pair for the placement subevent from the subevent detection dialogue additional input (all initial context is shown in gray). We assume are interested in PerpOrg, PerpInd, Weapon, Time, and Place roles for placement events, though different role sets could of course be given. Following initial query that determines the number of placement instances, we query for the arguments that fill these roles for each instance, one question per role, per identified instance. Additionally, we directly query for the irrealis status of the planting instances. Similar to above, queried roles are marked with comments.
We find that ChatGPT does quite well in determining the number of placement events (2) and their irrealis status (both actual), although gets more mixed results identifying participants, failing to infer, for instance, that "MRTA" is the PerpOrg of the second placement event and refusing to identify the "guerrillas" as the PerpInd of both events.
User: Article: [omitted; same as above]
Does this document explicitly describe explosives being placed planted in some target loca- tion?
Assistant:
Yes, the document explicitly describes a bomb exploding in a Lima restaurant and mentions that second device was placed in the same establishment but was deactivated by the Peruvian National Police.
User: How many distinct instances of explosives being placed or planted in some target loca- tion are explicitly described in this document?
Assistant: There are two distinct instances of explosives being placed or planted in the same target location explicitly described in this document.
User: Was a device actually placed in the first instance? // irrealis

---

Assistant:
Yes, device was actually placed in the first instance, as mentioned in the docu- ment.
User:
What organization was responsible for the first instance? // PerpOrg
Assistant:
The document states that the guerrillas of the Tupac Amaru Revolutionary Move- ment (MRTA) claimed credit for the first instance, which was the bomb explosion in the Lima restaurant.
User: What individuals were responsible for the first instance? PerpInd
Assistant: The document docs not explicitly mention the individuals responsible for the first in- stance. only states that the guerrillas of the Tupac Amaru Revolutionary Movement (MRTA) claimed credit for the terrorist act through pamphlets they left on the premises.
User: What kind of device was planted in the first instance? // Weapon
Assistant: The document does explicitly mention the kind of device that was planted in the first instance.
User:
Where was the device planted the first instance? // Place
Assistant: The document explicitly states that the device was planted in the same establish- ment, which is the Lima restaurant where the bomb exploded.
User:
When was the device planted in the first instance? // Time
The document does not explicitly mention when the device was planted in the first instance.
Assistant:
User:
Was a device actually placed in the second instance? // irrealis
Yes, a device was actually placed in the second instance, as mentioned in the docu- ment. However, it was deactivated by the Peruvian National Police.
Assistant:
User: What organization was responsible for the second instance? // PerpOrg
Assistant: The document does not explicitly mention the organization responsible for the see- ond instance. only states that the second device, which was placed in the same establishment, was deactivated by the Peruvian National Police.
User:
What individuals were responsible for the second instance? // PerpInd
Assistant:
The document does not explicitly mention the individuals responsible for the second
instance.
User:
What kind of device was planted in the second instance? // Weapon
The document does not explicitly mention the kind of device that was planted in the second instance.
Assistant:

---

User: Where was the device planted in the second instance? // Place
Assistant: The document states that the second device was placed in the same establishment as the first instance. Therefore, the device was planted the Lima restaurant where the bomb exploded.
User: When was the device planted in the second instance? // Time
Assistant: The document does not explicitly mention when the device was planted in the sec- ond instance.


Reviews:
[{"review_id": "wdqTpx2wFt", "reviewer": "Reviewer_yycY", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 1 for this encouraging review! We're pleased both the paper's core thesis and supporting arguments seem compelling.\n\nAs this is a position paper, and as the primary aim is to convince the reader of the challenges arising from event individuation, we wanted to devote the majority of allotted space to arguments in support of this claim. However, we certainly agree that the proposed QA formulation merits further elaboration. Should the paper be accepted, we intend to use the additional page for this purpose. In particular, we think a framework that outlines how these question sets could be constructed (beyond the one example in Appendix F) is warranted.\n\nWe also understand and appreciate the comment on paper vs. appendix length. We felt that the core arguments for the thesis could be effectively communicated in a short paper, and we stand by this decision. While the appendices do run long, we do *not* think they are necessary to fully grasp the content in the main text, and are rather intended to:\n\n1. Provide additional helpful, though inessential, context that could largely be found by reading the primary citations in the main text (Appendices A and B)\n2. Preempt concerns that we might have cherry-picked our example in Figure 1 by providing additional examples of cases that pose challenging individuation decisions (Appendix C).\n3. Promote good science by sharing further details of our methods and models (Appendices D and E) that the average reader likely won't care to know but that are important to include all the same.\n\nWe feel the great majority of this content really does belong in an appendix, rather than the main text. A plausible exception to this is Appendix F on the QA formulation, which we have addressed above, and which we would like to discuss at greater length with the additional page. That said, we would be happy to trim any parts of the appendices that seem superfluous.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Provides a compelling critique on an important IE benchmark task, raising awareness about a neglected issue that could stimulate useful discussion.\n\n- Backs its central claims with rigorous empirical evidence from multiple angles - annotation studies, model analysis, and example case studies. The methodology is thorough.\n\n- Focuses attention on an aspect of IE that has major room for improvement - event individuation. The analysis reveals the limitations of current benchmarks and systems. \n\n- Proposes an interesting alternative QA-based formulation that could better handle individuation while avoiding arbitrary decisions. This explores a novel direction.\n\n- Evaluates established datasets like MUC-4 and BETTER. Demonstrates shortcomings exist across datasets and state-of-the-art models.\n\n- Well-written and structured. Clearly articulate the position and line of argumentation. ", "reasons_to_reject": "I do not see any major weaknesses or risks that would warrant rejecting this paper based on the conference policies. The paper provides a cogent position supported by thorough analysis. However, some aspects that could be improved:\n\n- The alternative QA formulation is only briefly sketched. More details are needed to demonstrate its feasibility as a full solution.\n\n- The paper is very short at only 4 pages, with a 12-page appendix containing important details on metrics, proposed approach, etc. This suggests key content is relegated to the appendix rather than appearing in the main paper body. Significant details around the QA proposal and metric definitions are covered in the appendix rather than the main text. This makes the core ideas and innovations harder to understand on their own. The rationale for the appendix length compared to the main paper is unclear. It seems excessive, suggesting the contribution could be better communicated as a long paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This position paper critically examines the benchmark task of template filling for document-level information extraction. The main problem it addresses is that template filling relies on arbitrary decisions about how to individuate distinct events, even though experts often disagree about individuation.\n\nThe key contributions are:\n\n- Demonstrates how standard template filling metrics penalize different individuation choices even when all entities and roles are correctly extracted. This harms comparability between models.\n\n- Presents an annotation study showing that even experts struggle to agree on event individuation decisions. Suggests common datasets embody questionable individuation judgments. \n\n- Analyzes state-of-the-art models, finding they fail to learn the individuation rules latent in datasets and heavily under-generate templates.\n\n- Proposes recasting template filling as question answering without requiring explicit individuation decisions. Argues key event properties like time and location should be queried directly. \n\n- Provides critical analysis and evidence that individuation decisions are often subjective, unreliable, and detrimental as part of the template-filling benchmark.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qzoBZqymY6", "reviewer": "Reviewer_HbG3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 2 for this careful and very thoughtful review! We're glad our main thesis seems important, and we hope to resolve doubts about its correctness here.\n\n*It can be the case that some of the event individuation can be arbitrary and therefore the annotation is noisy, but the benchmark can still reflect the performance of the IE model....Despite showing the problem with annotations themselves, there is a lack of direct evidence that such noisy annotation fail to rank model's performance correctly.*\n\nThis is a shrewd observation. However, our aim isn't necessarily to show that the ranking of the models actually induced by the metrics is definitely wrong. Rather, our goals are to show that:\n\n1. Each model, considered independently, seems to struggle to do individuation *at all* (as evidenced by results in Table 2).\n2. Even experts disagree in many cases on what the \"correct\" individuation is.\n3. The standard benchmarks (datasets + metrics) *will* yield quite divergent results for similar predictions that differ mainly in their individuation decisions (as evidenced by the discussion in section 3).\n\nWe gather Reviewer 2 agrees with these claims, but we don't think they are undermined by the above critique.\n\nPart of the critique seems to say: *if (3) is true, we should see clearer evidence for it in the empirical results — e.g. in the form of a reranking of the models in Table 2 under the collapsed scores*. Why don't we observe this? We argue that it's in fact largely because of (1): if all our best models still tend overwhelmingly to predict one template per type (when they ought to predict more), not much *should* change in the results — likely including their relative ranking — when moving to collapsed scores. This is exactly the point we make in section 5. \n\nBut when *would* we expect (3) to be evidenced more clearly by the empirical results? Presumably, it would be when models actually *do* demonstrate an ability to effectively predict multiple templates of the same type. Here, we might expect to see more variability in models’ individuation decisions. Unfortunately, as we show, even SOTA models fail to demonstrate this ability. But in L253-6, we try to gesture at what would be at stake if we *did* have better models by showing the percentage of a perfect score that hinges on getting individuation \"right\" (per the benchmark). These percentages (12.7% for MUC, 24.2% for BETTER) are substantial and they suggest one could very well see a reranking of models under collapsed scores — if only the models were better than they are.\n\nIn sum, it could be that the metrics happen to rank *these* models \"correctly\" (whatever this means, independent of metrics), but this isn't evidence against (1)-(3), nor against the broader claim that individuation poses a significant challenge — even for humans and surely, too, for better models down the line. At most, it could be evidence that the standard metrics are adequate for model comparison purposes so long as those models do minimal individuation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is well-written. It proposes an important issue: the current dataset and benchmark can be misleading. I'm wondering if the 3 IE models in Section 4 demonstrate the degenerate behavior of collating all entities relevant to a particular event type into a single template because they all overfit the existing benchmark. Also, if there exists another model, worse under the standard metrics but does not have the degenerate behavior, actually performs better if evaluated using a different approach, for example, using human evaluation or evaluating by metrics related to its real-world application. If this is true, then this paper is really important as the current research efforts may be working in the wrong direction.", "reasons_to_reject": "The claim in this paper is important, but I cannot be 100% sure it is correct. It can be the case that some of the event individuation can be arbitrary and therefore the annotation is noisy, but the benchmark can still reflect the performance of the IE model. For example, as a future direction, the authors propose the possiblity of aggregate the template by type. However, in Table 2, the original Combine Score and the Collaposed Combine Score are highly correlated and give the same ranking for the 3 models. Despite showing the problem with annotations themselves, there is a lack of direct evidence that such noisy annotation fail to rank model's performance correctly.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This position paper calls into question the usage of template filling as a benchmark for document-level information extraction. The authors argue that the problem with template filling is the lack of agreement on event individuation, and demonstrate the problem from 3 aspects. First, with an example that contains two events in the ground truth annotation but these events happen at the same time, location, and agents so the distinction of the two events is a bit arbitrary, the authors show the standard evaluation metrics heavily depend on alignment (matching the predicted instance of an event to the instance in the ground truth). Second, the authors perform annotations themselves and show low inter-judge agreement as well as low agreement with the ground truth annotation. Finally, the authors show that state-of-the-art IE models have a degenerate behavior: they collate all entities relevant to a particular event type into a single template. The implication of this paper is that the current benchmark may be misleading for document IE.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ekLPcHXMBK", "reviewer": "Reviewer_vPzg", "summary": "", "questions": "Would you agree that the drop in agreement after the inclusion of the gold data could also indicate a difference in annotator training, perhaps something that was not captured properly in the annotation guidelines? This possibility is mentioned in reference to the models but should also be included in Section 4. The same goes for the availability of the BETTER annotation guidelines mentioned in the appendix.\n\nWhat about cross-template individuation (e.g. attack and bombing), does this also have an effect on evaluation?\n\nCollapsing the templates did not alter the relative performance of the three considered models (in terms of ranking), does that mean collapsed scores are not a helpful metric?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 3 for this thoughtful review and for the great questions!\n\n**Questions:**\n\nA. This is a great point. In short, yes — it's certainly possible that there was relevant context (private discussions on particular annotation questions, agreed-upon conventions) that the original annotators of these datasets had, that did not make it into writing, and that are therefore simply unavailable to us. We will explicitly note this possibility in our revisions.\n\nB. With ontologies like MUC and BETTER that have only a handful of event types that are fairly distinct, our experience suggests that individuation problems arising from cross-type confusions are generally much less of an issue than those arising from within-type confusions — hence our choice of focus in the paper. But this is likely a much more significant concern the more fine-grained the ontology is (e.g. FrameNet has roughly 1,200 frame types, some of which are very similar to each other) — another point we will note in our revisions.\n\nC. This is an astute question. See our response to Reviewer 2 on this point.\n\n**Reasons to Reject**\n\nWe would dispute the implication that focusing on a particular NLP task (template filling) constitutes a reason for rejection; many effective papers do precisely this. Moreover, even if our analysis is focused on template filling, we think the core considerations about event individuation apply in any setting that (even implicitly) requires event coreference, which does not strike us as an especially niche problem.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper calls into question popular evaluation techniques and datasets meaning it can potentially have a large impact on the field of information extraction.\n\nThe paper is very well written and clearly describes the problem through the use of an example, the line of argument is easy to follow.\n\nAnalsysis of prior work seems to have been extensive with detailed references to annotation guidelines being included in the appendix.\n\nThree well-performing models are evaluated wrt their individuation capabilities, firmly establishing that SotA models are not capable of event individuation.", "reasons_to_reject": "Scope and impact are limited to template filling IE", "questions_to_authors": "Would you agree that the drop in agreement after the inclusion of the gold data could also indicate a difference in annotator training, perhaps something that was not captured properly in the annotation guidelines? This possibility is mentioned in reference to the models but should also be included in Section 4. The same goes for the availability of the BETTER annotation guidelines mentioned in the appendix.\n\nWhat about cross-template individuation (e.g. attack and bombing), does this also have an effect on evaluation?\n\nCollapsing the templates did not alter the relative performance of the three considered models (in terms of ranking), does that mean collapsed scores are not a helpful metric?", "paper_topic_and_main_contributions": "The position paper calls into question the usefulness of evaluating information extraction using template-filling metrics. The authors specifically focus on event individuation, the decision of whether one description may contain two individual events. Using both an annotation study and altered evaluation metrics the authors show that neither humans nor models are good at individuation events. Finally, the authors make some suggestions for alternative evaluation options.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 2: consider typesetting the '|' symbols in math mode if they aren't already, I found them to slightly confusing at first."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes the use of an alternative evaluation metric for document-level information extraction, instead of relying on the existing method of template filling, which depends on arbitrary decisions about how to distinguish distinct events. Such distinction is challenging for humans and can also pose difficulties for models, as it is often motivated by subjective and unreliable factors, as indicated by an annotation studies. The alternative solution proposed is cast the problem as a question-answering task.

The paper is well-written, offering insightful observations about a neglected issue that can stimulate discussions. Furthermore, the methodology is thorough, and the approach of framing it as a question-answering problem represents a novel direction. The evaluation is conducted on the MUC-4 and BETTER datasets.
Decision: Accept-Findings