SlowGuess/NAACL_2021_OCR huggingface dataset:
Alon Eirew $^{1,2}$  Arie Cattan $^{1}$  Ido Dagan $^{1}$
$^{1}$ Bar Ilan University, Ramat-Gan, Israel  $^{2}$ Intel Labs, Israel
alon.eirew@intel.com arie.cattan@gmail.com
dagan@cs.biu.ac.il
Cross-document event coreference resolution is a foundational task for NLP applications involving multi-text processing. However, existing corpora for this task are scarce and relatively small, while annotating only modest-size clusters of documents belonging to the same topic. To complement these resources and enhance future research, we present Wikipedia Event Coreference (WEC), an efficient methodology for gathering a large-scale dataset for cross-document event coreference from Wikipedia, where coreference links are not restricted within predefined topics. We apply this methodology to the English Wikipedia and extract our large-scale WEC-Eng dataset. Notably, our dataset creation method is generic and can be applied with relatively little effort to other Wikipedia languages. To set baseline results, we develop an algorithm that adapts components of state-of-the-art models for within-document coreference resolution to the cross-document setting. Our model is suitably efficient and outperforms previously published state-of-the-art results for the task.
Cross-Document (CD) Event Coreference resolution is the task of identifying clusters of text mentions, across multiple texts, that refer to the same event. Successful identification of such corefering mentions is beneficial for a broad range of applications at the multi-text level, which are gaining increasing interest and need to match and integrate information across documents, such as multi-document summarization (Falke et al., 2017; Liao et al., 2018), multi-hop question answering (Dhingra et al., 2018; Wang et al., 2019) and Knowledge Base Population (KBP) (Lin et al., 2020).
Unfortunately, rather few datasets of reasonable scale exist for CD event coreference. Notable datasets include  $\mathrm{ECB + }$  (Cybulska and Vossen,
2014), MEANTIME (Minard et al., 2016) and the Gun Violence Corpus (GVC) (Vossen et al., 2018) (described in Section 2), where recent work has been evaluated solely on  $\mathrm{ECB + }$ . When addressed in a direct manner, manual CD coreference annotation is very hard due to its worst-case quadratic complexity, where each mention may need to be compared to all other mentions in all documents. Indeed,  $\mathrm{ECB + }$  contains less than 7000 event mentions in total (train, dev, and test sets). Further, effective corpora for CD event coreference are available mostly for English, limiting research opportunities for other languages. Partly as a result of this data scarcity, rather little effort was invested in this field in recent years, compared to dramatic recent progress in modeling within-document coreference.
Furthermore, most existing cross-document coreference datasets are restricted in their scope by two inter-related characteristics. First, these datasets annotate sets of documents, where the documents in each set all describe the same topic, mostly a news event (consider the Malaysia Airlines crash as an example). While such topic-focused document sets guarantee a high density of coreferring event mentions, facilitating annotation, in practical settings the same event might be mentioned across an entire corpus, being referred to in documents of varied topics. Second, we interestingly observed that event mentions may be (softly) classified into two different types. One type, which we term a descriptive mention, pertains to a mention involved in presenting the event or describing new information about it. For example, news about the Malaysian Airline crash will include mostly descriptive mentions of the event and its sub-events, such as shot-down, crashed and investigated. Naturally, news documents about a topic, as in prior event coreference datasets, include mostly descriptive event mentions. The other type, which we term a referential mention, pertains to
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2498-2510 June 6-11, 2021. ©2021 Association for Computational Linguistics
ments of the event in sentences that do not focus on presenting new information about the event but rather mention it as a point of reference. For example, mentions referring to the airplane crash, such as the Malaysian plane crash, Flight MH17 or disaster may appear in documents about the war in Donbass or about flight safety. Since referential event mentions are split across an entire corpus, they are less trivial to identify for coreference annotation, and are mostly missing in current news-based datasets. As we demonstrate later, these two mention types exhibit different lexical distributions and seem to require corresponding training data to be properly modeled.
In this paper, we present the Wikipedia Event Coreference methodology (WEC), an efficient method for automatically gathering a large-scale dataset for the cross-document event coreference task. Our methodology effectively complements current datasets in the above-mentioned respects: data annotation is boosted by leveraging available information in Wikipedia, practically applicable for any Wikipedia language; mentions are gathered across the entire Wikipedia corpus, yielding a dataset that is not partitioned by topics; and finally, our dataset consists mostly of referential event mentions.
In its essence, our methodology leverages the coreference relation that often holds between anchor texts of hyperlinks pointing to the same Wikipedia article (see Figure 1), similar to the basic idea introduced in the Wikilinks dataset (Singh et al., 2012). Focusing on CD event coreference, we identify and target only Wikipedia articles denoting events. Anchor texts pointing to the same event article, along with some surrounding context, become candidate mentions for a corresponding event coreference cluster, undergoing extensive filtering. We apply our method to the English Wikipedia and extract WEC-Eng, our English version of a WEC dataset. The automatically-extracted data that we collected provides a training set of a very large scale compared to prior work, while our development and test sets underwent relatively fast manual validation.
Due to the large scale of the WEC-Eng training data, current state-of-the-art CD coreference models cannot be easily trained and evaluated on it, for scalability reasons. We therefore developed a new, more scalable, baseline model for the task, while adapting components of recent competitive
Figure 1: Example of two anchor texts (100 metres, 100 m final) from different Wikipedia articles (Usain Bolt, Yohan Blake) pointing to the same event.
within-document coreference models (Lee et al., 2017; Kantor and Globerson, 2019; Joshi et al., 2019). In addition to setting baseline results for WEC-Eng, we assess our model's competitiveness by presenting a new state-of-the-art on the commonly used  $\mathrm{ECB + }$  dataset. Finally, we propose that our automatic extraction and manual validation methods may be applied to generate additional annotated datasets, particularly for other languages. Overall, we suggest that future cross-document coreference models should be evaluated also on the WEC-Eng dataset, and address its complementary characteristics, while the WEC methodology may be efficiently applied to create additional datasets. To that end, our dataset and code $^{12}$  are released for open access.
This section describes the main characteristics of notable datasets for CD event coreference (ECB+, MEANTIME, GVC). Table 1 presents statistics for all these datasets, as well as ours. We further refer to the Wikilinks dataset, which also leveraged Wikipedia links for CD coreference detection.
2.1 CD Event Corpora
ECB+ This dataset (Cybulska and Vossen, 2014), which is an extended version of the EventCorefBank (ECB) (Bejan and Harabagiu, 2010), is the most commonly used dataset for training and testing models for CD event coreference (Choubey and
1WEC-https://github.com/AlonEirew/ extract-weak
2Model-https://github.com/AlonEirew/cross-doc-event-coref
Topics	Mentions	Clusters	Non-Singleton Clusters	Ambiguity	Diversity
WEC-Eng (train)	-	40,529	7,042	6,210	6.3	2.1
WEC-Eng (dev)	-	1,250	233	216	2.7	1.7
WEC-Eng (test)	-	1,893	322	306	2.6	1.9
ECB+ (train)	25	3,808	1,527	411	1.4	2.2
ECB+ (dev)	8	1,245	409	129	1.4	2.3
ECB+ (test)	10	1,780	805	182	1.4	2.2
GVC (all)	1	7,298	1,411	1,046	19.5	3.0
MEANTIME (all)	4	2,107	1,892	142	2.1	1.5
Table 1: Event Coreference Datasets Statistics (No train/dev/test split exists for MEANTIME and GVC). Non-Singleton Clusters: Number of clusters with more than a single mention. Ambiguity: Average number of different clusters in which a head lemma appears. Diversity: Average number of unique head lemmas within a cluster (excluding singletons for fair comparison).
Huang, 2017; Kenyon-Dean et al., 2018; Barhom et al., 2019). This corpus consists of documents partitioned into 43 clusters, each corresponding to a certain news topic. In order to introduce some ambiguity and to limit the use of lexical features, each topic is composed of documents describing two different events (called sub-topics) of the same event type (e.g. two different celebrities checking into rehab facilities). Nonetheless, as can be seen in Table 1, the ambiguity level obtained is still rather low.  $\mathrm{ECB + }$  is relatively small, where on average only 1.9 sentences per document were selected for annotation, yielding only 722 non-singleton coreference clusters in total.
MEANTIME Minard et al. (2016) proposed a dataset that is similar in some respects to  $\mathrm{ECB + }$ , with documents partitioned into a set of topics. The different topics do not correspond to a specific news event but rather to a broad topic of interest (e.g. Apple, stock market). Consequently, different documents rarely share coreferring event mentions, resulting in only 11 event coreference clusters that include mentions from multiple documents, making this dataset less relevant for training CD coreference models.
Gun Violence Corpus (GVC) This dataset (Vossen et al., 2018) was triggered by the same motivation that drove us, of overcoming the huge complexity of direct manual annotation of CD event coreference from scratch. To create the dataset, the authors leveraged a structured database recording gun violence events, in which the record for an individual event points at documents describing that event. The annotators were then asked to examine the linked documents and mark in them
ments of 5 gun-violence event classes (firing a gun, missing, hitting, injuring, death). Considering the recorded event as a pivot, all mentions found for a particular class were considered as corefering. Using this process, they report an annotation rate of about 190 mentions per hour. As this corpus assumes a specific event structure scheme related to gun violence, it is more suitable for studying event coreference within a narrow domain rather than for investigating models for broad coverage event coreference.
2.2 Wikilinks
Wikilinks (Singh et al., 2012) is an automatically-collected large-scale cross-document coreference dataset, focused on entity coreference. It was constructed by crawling a large portion of the web and collecting as mentions hyperlinks pointing to Wikipedia articles. Since their method does not include mention distillation or validation, it was mostly used for training models for the Entity Linking task, particularly in noisy texts (Chisholm and Hachey, 2015; Eshel et al., 2017).
3 The WEC Methodology and Dataset
We now describe our methodology for gathering a CD event coreference dataset from Wikipedia, and the WEC-Eng dataset created by applying it to the English Wikipedia. We also denote how this methodology can be applied, with some language-specific adjustments, to other Wikipedia languages.
3.1 Dataset Structure
Our data is collected by clustering together anchor texts of (internal) Wikipedia links pointing to the same Wikipedia concept. This is generally justified
Cluster-1 (2010 Polish Air Force Tu-154 crash)	Cluster-2 (Lokomotiv Yaroslavl plane crash)
...following the death of President Lech Kaczyński in a plane crash...	|| ...On 7 September 2011, nearly the entire team perished in an airplane crash...
...following the Smolensk air disaster which killed the incumbent Polish president...	|| ...fourth season was overshadowed by the Yaroslavl air disaster on 7 September...
...died when the presidential plane went down about a half mile from the runway...	|| ...Early in September, tragedy rocked the hockey world...
Table 2: Mentions from two generated WEC-Eng clusters, illustrating typical challenging ambiguity. To resolve, the model must consider the surrounding paragraph extracted along with the mention (not fully shown here).
since all these links refer to the same real world theme described by that article, as illustrated in Figure 1. Accordingly, our dataset consists of a set of mentions, each including the mention span corresponding to the link anchor text, the surrounding context, and the mention cluster ID. Since Wikipedia is not partitioned into predefined topics, mentions can corefer across the entire corpus (unlike most prior datasets).
Since mention annotation is not exhaustive, coreference resolution is performed over the gold mentions. Thus, our goal is to support the development of CD event coreference algorithms, rather than of mention extraction algorithms. Our dataset also includes metadata information, such as source and target URLs for the links, but these are not part of the data to be considered by algorithms, as our goal in this work is CD coreference development rather than Event Linking (Nothman et al., 2012).
3.2 Data Collection Process
In this paper, we focus on deriving from Wikipedia an event coreference dataset. The choice to focus on event coreference was motivated by two observations: (1) coreference resolution for Wikipedia anchor texts would be more challenging for event mentions than for entity mentions, since the former exhibits much higher degrees of both ambiguity and lexical diversity, and (2) event structures, with their arguments (such as participants, location and time) available in the surrounding context, would facilitate a more natural dataset for the corpus-wide CD coreference task, compared to Wikipedia entity mentions which are comprised mostly of named entities.
Accordingly, we seek to consider only Wikipedia pages denoting events, then collect hyperlinks pointing at these pages. All anchor texts pointing to the same event then become the mentions of a corresponding event coreference cluster, and are extracted along with their surrounding paragraph as context (see Table 2). The following paragraphs describe this process in detail, and how it was applied to generate the WEC-Eng dataset from English
Event Identification Many Wikipedia articles contain an infobox<sup>3</sup> element. This element can be selected by a Wikipedia author from a pre-defined list of possible infobox types (e.g. "Civilian Attack", "Game", "Scientist", etc.), each capturing typical information fields for that type of articles. For example, the "Scientist" infobox type consists of fields such as "birth date", "awards", "thesis" etc. We leverage the infobox element and its parameters in order to identify articles describing events (e.g. accident, disaster, conflict, ceremony, etc.) rather than entities (e.g. a person, organization, etc.).
To that end, we start by automatically compiling a list of all Wikipedia infobox types that are associated with at least dozens of Wikipedia articles. Of those, we manually identify all infobox types related to events (WEC-Eng examples include Awards, Meetings, Civilian Attack, Earthquake, Contest, Concert and more). We then (manually) exclude infobox types that are frequently linked from related but non-coreferring mentions, such as sub-events or event characteristics, like location and time (see Appendix A.1 for further details). For WEC-Eng, we ended up with 28 English Wikipedia event infobox types (see Appendix A.2 for the full list).
Gathering Initial Dataset Once the infobox event list is determined, we apply a fully automatic pipeline to obtain an initial crude version of the dataset. This pipeline consists of: (1) Collecting all Wikipedia articles (event "pivot" pages) whose Infobox type is in our list. (2) Collecting all Wikipedia anchor texts ("mentions") pointing to one of the pivot pages, along with their surrounding paragraph. (3) Filtering mentions that lack context or those belonging to Wikipedia metadata, such as tables, images, lists, etc., as well as mentions whose surrounding context contains obvious Wikipedia boilerplate code (i.e. HTML and JSON tags) (4) Finally, all collected mentions are clustered accord
3https://en.wikipedia.org/wiki/Help: Infobox
Cluster	Mention link and context	Validation
1	..The Tar Heels made it to the final of the 2001 ACC Men's Basketball Tournament..	✓
..Duke finished the regular season with a 26–4 record entering the ACC Tournament..	✗ Insufficient mention context
2	..was nominated for an Academy Award at the 1939 Oscars for Best..	✓
..In the following year, the Academy presented Disney an Honorary Academy Award..	✗ Link placed on event time
3	..was kidnapped during the Draper's Meadow massacre in July 1755..	✓
..with a terrible attack on Drapers Meadow by the Shawnee Indian Warriors..	✗ Link placed on event location
4	..War and was on board the USS "Forrestal" in 1967 when a fatal fire broke out..	✓
..the repairs to the supercarrier which had been extensively damaged off Vietnam..	✗ Subevent
Table 3: Examples of WEC-Eng manually validated mentions, showing validation results (validated/disqualified)
ing the pivot page at which they point.
Mention-level Filtering An event coreference dataset mined this way may still require some refinement in order to further clean the dataset at the individual mention level. Indeed, we observed that many Wikipedia editors have a tendency to position event hyperlinks on an event argument, such as a Named Entity (NE) related to the event date or location (as in the case of the disqualified mention for cluster 3 in Table 3). To automatically filter out many of the cases where the hyperlink is placed on an event argument instead of on the event mention itself, we use a Named Entity tagger and filter out mentions identified by one of the following labels: PERSON, GPE, LOC, DATE and NORP (for WEC-Eng we used the SpaCy Named Entity tagger (Honnibal and Montani, 2017)).
Controlling Lexical Diversity So far, we addressed the need to avoid having invalid mentions in a cluster, which do not actually refer to the linked pivot event.
Next, we would like to ensure a reasonably balanced lexical distribution of the mentions within each cluster. Ideally, it would be desired to preserve the "natural" data distributions as much as possible. However, we observed that in many Wikipedia hyperlinks, the anchor texts used in an event mention may be lexically unbalanced. Indeed, Wikipedia authors seem to have a strong bias to use the pivot article title when creating hyperlinks pointing at that article, while additional ways by which the event can be referred are less frequently hyperlinked. Consequently, preserving the original distribution of hyperlink terms would create a too low level of lexical diversity. As a result, training a model on such a dataset might overfit to identifying only the most common mention phrases, leaving little room for identifying the less frequent ones.
To avoid this, we applied a simple filter that allows a maximum of 4 mentions having identical strings in a given cluster. This hyperparameter was tuned by making the lexical repetition level in our clusters more similar to that of  $\mathrm{ECB + }$  , in which lexical diversity was not controlled (resulting with an average of 1.9 same-string mentions per cluster in WEC-Eng train set compared to 1.3 in  $\mathrm{ECB + }$
Using this process we automatically generated a large dataset. We designated the majority of the automatically generated data to serve as the WEC-Eng training set. The remaining data was left for the development and test sets, which underwent a manual validation phase, as described next.
3.3 Manual Validation
Inevitably, some noise would still exist in the automatically derived dataset just described. While partially noisy training data can be effective, as we show later, and is legitimate to use, the development set, and particularly the test set, should be of high quality to allow for proper evaluation. To that end, we manually validated the mentions in the development and test sets.
For CD coreference evaluation, we expect to include a mention as part of a coreferring cluster only if it is clear, at least from reading the given surrounding context, that this mention indeed refers to the linked pivot event. Otherwise, we cannot expect a system to properly detect coreference for that mention. Such cases occasionally occur in Wikipedia, where identifying context is missing while relying on the provided hyperlink (see cluster-1 in Table 3, where the tournament year is not mentioned). Such mentions are filtered out by the manual validation. Additionally, misplaced mention boundaries that do not include the correct event trigger (Table 3 cluster-2,3), as well as
4We also release the non controlled version of the dataset.
mentions of subevents of the linked event (Table 3 cluster-4), are filtered.
Summing up, to filter out these cases, we used a strict and easy-to-judge manual validation criterion, where a mention is considered valid only if: (1) the mention boundaries contain the event trigger phrase; (2) the mention's surrounding paragraph suffices to verify that this mention refers to the pivot page and thus belongs to its coreference cluster; and (3) the mention does not represent a subevent of the referenced event. Table 3 shows examples of validated vs. disqualified mentions judged for the WEC-Eng development set.
For the WEC-Eng validation, we randomly selected 588 clusters and validated them, yielding 1,250 and 1,893 mentions for the development and test sets, respectively. Table 1 presents further statistics for WEC-Eng. The validation was performed by a competent native English speaker, to whom we explained the guidelines, after making a practice session over 150 mentions. Finally, all training mentions that appeared in the same (source) article with a validated mention were discarded from the training set.
Our manual validation method is much faster and cheaper compared to a full manual coreference annotation process, where annotators would need to identify and compare all mentions across all documents. In practice, the average annotation rate for WEC-Eng yielded 350 valid mentions per hour, with the entire process taking only 9 hours to complete. In addition, since our validation approach is quite simple and does not require linguistic expertise, the eventual data quality is likely to be high. To assess the validation quality, one of the authors validated 50 coreference clusters (311 mentions), randomly selected from the development and test sets, and then carefully consolidated these annotations with the original validation judgements by the annotator. Relative to this reliable consolidated annotation, the original annotations scored at 0.95 Precision and 0.96 Recall, indicating the high quality of our validated dataset (the Cohen's Kappa (Cohen, 1960) between the original and consolidated annotations was 0.75, considered substantial agreement).
In all,  $83\%$  of the candidate mentions were positively validated in the development and test sets, indicating a rough estimation of the noise level in the training set. That being said, we note that a majority of these noisy mentions were not totally
wrong mentions but rather were filtered out due to the absence of substantial surrounding context or the misplacement of mention boundaries (see examples in Table 3).
3.4 Dataset Content