SlowGuess/NAACL_2021_OCR huggingface dataset:
This section describes the WEC-Eng dataset content and some of its characteristics. The final WEC-Eng dataset statistics are presented in Table 1. Notably, the training set includes 40,529 mentions distributed into 7,042 coreference clusters, facilitating the training of deep learning models.
The relatively high level of lexical ambiguity shown in the table<sup>5</sup> is an inherent characteristic caused by many events (coreference clusters) sharing the same event type, and thus sharing the same terms, as illustrated in Table 2. Identifying that identical or lexically similar mentions refer to different events is one of the major challenges for CD coreference resolution, particularly in the corpus-wide setting, where different documents might refer to similar yet different events.
With respect to the distinction between descriptive and referential event mentions, proposed in Section 1, WEC-Eng mentions are predominantly referential. This stems from the fact that its mentions correspond to hyperlinks that point at a different Wikipedia article, describing the event, while the mention's article is describing a different topic. On the other hand,  $\mathrm{ECB + }$ , being a news dataset, is expected to include predominantly descriptive mentions. Indeed, manually analyzing a sample of 30 mentions from each dataset, in WEC-Eng 26 were referential while in  $\mathrm{ECB + }$  28 were descriptive. This difference also imposes different lexical distributions for mentions in the two datasets, as sampled in Appendix A.3. When describing an event, verbs are more frequently used as event mentions, but nominal mentions are abundant as well. This is apparent for the predominately descriptive  $\mathrm{ECB + }$ , where  $62\%$  of the mentions in our sample are verbal vs.  $38\%$  nominal. On the other hand, when a previously known event is only referenced, it is mostly referred by a nominal mention. Indeed, in the predominantly referential WEC-Eng, a vast majority of the mentions are nominal (93% in our sample).
5The higher ambiguity level of the training sets stems from its larger size — including many more coreference clusters in total, and accordingly more clusters under individual event types.
3.5 Potential Language Adaptation
While our process was applied to the English Wikipedia, it can be adapted with relatively few adjustments and resources to other languages for which a large-scale Wikipedia exists. Here we summarize the steps needed to apply our dataset creation methodology to other Wikipedia languages. To generate a dataset, the first step consists of manually deciding on the list of suitable infobox types corresponding to (non-noisy) event types. Then, the automatic corpus creation process can be applied for this list, which takes only a few hours to run on a single CPU. After the initial dataset was created, a language specific named-entity tagger should be used to filter mentions of certain types, like time and location (see Mention-level Filtering (3.2)). Next, the criterion for ensuring balanced lexical diversity in a cluster (see Controlling Lexical Diversity (3.2)), which was based on a simple same-string test for English, may need to be adjusted for languages requiring a morphological analyzer. Finally, as we perform manual validation of the development and test sets, this process should be performed for any new dataset (see Section 3.3). Supporting this step, our validation guidelines are brief and simple, and are not language specific. They only require identifying subevents and misplaced mention boundaries, as well as validating the sufficiency of the mention context.
4 Baseline and Analysis
The current state-of-the-art CD event coreference system (Barhom et al., 2019) cannot be effectively trained on WEC-Eng for two main reasons: (1) computational complexity and (2) reliance on verbal SRL features. With respect to computation time, the training phase of this model simulates the clustering operations done at inference time, while recalculating new mention representations and pairwise scores after each cluster merging step. Consequently, training this model on our large scale training data, which is further not segmented to topics, is computationally infeasible. In addition, the model of Barhom et al. (2019) uses an SRL system to encode the context surrounding verbal event mentions, while WEC-Eng is mostly composed of nominal event mentions (Section 3.4).
We therefore develop our own, more scalable, model for CD event coreference resolution, estab
lishing baseline results for WEC-Eng. As common in CD coreference resolution, we train a pairwise scorer  $s(i,j)$  indicating the likelihood that two mentions  $i$  and  $j$  in the dataset are corefering, and then apply agglomerative clustering over these scores to find the coreference clusters. Following the commonly used average-link method (Choubey and Huang, 2017; Kenyon-Dean et al., 2018; Barhom et al., 2019), the merging score for two clusters is defined as the average mention pair score  $s(i,j)$  over all mention pairs  $(i,j)$  across the two candidate clusters to be merged.
For the pairwise model, we replicate the architecture of mention representation and pairwise scorer from the end-to-end within document coreference model in (Lee et al., 2017), while including the recent incorporation of transformer-based encoders (Joshi et al., 2019; Kantor and Globerson, 2019). Concretely, we first apply a pre-trained RoBERTa (Liu et al., 2019) language model (without finetuning), separately for each mention. Given a mention span  $i$ , we include as context,  $T$  (set to 250) tokens to the left of  $i$  and  $T$  tokens to the right of  $i$ . Applying RoBERTa to this window, we represent each mention by a vector  $g_{i}$ , which is the concatenation of three vectors: the contextualized representations of the mention span boundaries (first and last) and the weighted sum of the mention token vectors according to the head-finding attention mechanism in (Lee et al., 2017). The two mention representations  $g_{i}$  and  $g_{j}$ , and the element-wise multiplication of these vectors are then concatenated and fed into a simple MLP, which outputs a score  $s(i,j)$ , indicating the likelihood that mentions  $i$  and  $j$  belong to the same cluster. The head-attention layer and the MLP are trained to optimize the standard binary cross-entropy loss over all pairs of mentions, where the label is 1 if they belong to the same coreference cluster and 0 otherwise.
We first train and evaluate our model on the commonly used dataset  $\mathrm{ECB + }$  to assess its relevance as an effective baseline model, and then evaluate it on WEC-Eng, setting baseline results for our dataset. We also present the performance of the challenging same-head-lemma baseline, which clusters mentions sharing the same syntactic-head lemma. For
<sup>6</sup>We note that this optimization is different than the (linear) antecedent ranking in the model of Lee et al. (2017), since in the CD setting there is no linear order between mentions from different documents.
MUC	B3	CEAF	CoNLL
R	P	F1	R	P	F1	R	P	F1	F1
Lemma Baseline	76.5	79.9	78.1	71.7	85	77.8	75.5	71.7	73.6	76.5
DISJOINT (Barhom et al., 2019)	75.5	83.6	79.4	75.4	86	80.4	80.3	71.9	75.9	78.5
JOINT (Barhom et al., 2019)	77.6	84.5	80.9	76.1	85.1	80.3	81	73.8	77.3	79.5
Our model	84.2	81.8	83	80.8	81.7	81.3	76.7	79.5	78	80.8
Table 4: Event coreference results on the  $\mathrm{ECB + }$  test set
MUC	B3	CEAF	CoNLL
R	P	F1	R	P	F1	R	P	F1	F1
Lemma Baseline	85.5	79.9	82.6	74.5	32.8	45.5	25.9	39.4	31.2	53.1
Our model	78	83.6	80.7	66.1	55.3	60.2	53.4	40.3	45.9	62.3
Table 5: Event coreference results on the WEC-Eng test set
the experiment on  $\mathrm{ECB + }$  we follow the recent evaluation setting (Kenyon-Dean et al., 2018; Barhom et al., 2019), clustering gold mentions and concatenating all test documents into one meta-document, as proposed by Upadhyay et al. (2016). For a fair comparison, we use the output of pre-processing document clustering obtained by Barhom et al. (2019) and apply our coreference model separately on each predicted document cluster.
For both datasets, the positive examples for training consist of all the mention pairs in the dataset that belong to the same coreference cluster. For the  $\mathrm{ECB + }$  model, we consider only negative examples that belong to the same subtopic, while for WECEng we sample  $k$  (tuned to 10) negative examples for each positive one. Results are reported by precision, recall, and F1 for the standard coreference metrics MUC,  $\mathbf{B}^3$ , CEAF-e, and the average F1 of the three metrics, using the official CoNLL scorer (Pradhan et al., 2012).
Table 4 presents the results on  $\mathrm{ECB + }$ . Our model outperforms state-of-the-art results for both the JOINT model and the DISJOINT event model of Barhom et al. (2019), with a gain of 1.3 CoNLL  $F1$  points and 2.3 CoNLL  $F1$  points respectively. The JOINT model jointly clusters event and entity mentions, leveraging information across the two subtasks, while the DISJOINT event model considers only event mentions, taking the same input as our model. These results assess our model as a suitable baseline for WEC-Eng.
Table 5 presents the results on WEC-Eng. First, we observe that despite the certain level of noise in the automatically gathered training data, our model outperforms the same-head-lemma baseline by 9.2 CoNLL  $F_{1}$  points. In fact, it achieves similar error reduction rates relative to the lemma baseline as obtained over  $\mathrm{ECB + }$ , where training is performed on a clean but smaller training data (18.3% error reduction in  $\mathrm{ECB + }$  and 19.6% in WEC). Furthermore, the performance of both the same-head-lemma baseline and our model are substantially lower on WEC-Eng (Table 5) than on  $\mathrm{ECB + }$  (Table 4). This indicates the more challenging nature of WEC-Eng, possibly due to its corpus wide nature and higher degree of ambiguity (Tables 1).
Further examining the different nature of the two datasets, we applied cross-domain evaluation, applying the  $\mathrm{ECB + }$  trained model on WEC-Eng test data and vice versa. The results suggest that due to their different characteristics, with respect to mention type (descriptive vs. referential) and structure (topic-based vs. corpus wide), a model trained on one dataset is less effective (by 8-12 points) when applied to the other (further details are presented in Appendix B.1).
To obtain some qualitative insight about the learned models for both  $\mathrm{ECB + }$  and WEC-Eng, we manually examined their most certain predictions, looking at the top  $5\%$  instances with highest predicted probability and at the bottom  $5\%$ , of lowest predictions. Some typical examples are given in Appendix B.2. Generally, both models tend to assign the highest probabilities to mention pairs that
$^{7}$ https://github.com/conll/reference-coreference-scorers
share some lemma, and occasionally to pairs with different lemmas with similar meanings, with the WEC-Eng model making such lexical generalizations somewhat more frequently. Oftentimes in these cases, the models fail to distinguish between (gold) positive and negative cases, despite quite clear distinguishing evidence in the context, such as different times or locations. This suggests that the RoBERTa-based modeling of context may not be sufficient, and that more sophisticated models, injecting argument structure more extensively, may be needed.
In both models, the lowest predictions (correctly) correspond mostly to negative mention pairs, and occasionally to positive pairs for which the semantic correspondence is less obvious (e.g. offered vs. candidacy). In addition, we observe that longer spans common in WEC-Eng challenge the span representation model of Lee et al. (2017). This model emphasizes mention boundaries, but these often vary across lexically-similar coreferring mentions with different word order.
In this paper, we presented a generic low-cost methodology and supporting tools for extracting cross-document event coreference datasets from Wikipedia. The methodology was applied to create the larger-scale WEC-Eng corpus, and may be easily applied to additional languages with relatively few adjustments. Most importantly, our dataset complements existing resources for the task by addressing a different appealing realistic setup: the targeted data is collected across a full corpus rather than within topical document clusters, and, accordingly, mentions are mostly referential rather than descriptive. Hence, we suggest that future research should be evaluated also on WEC-Eng, while future datasets, particularly for other languages, can be created using the WEC methodology and tool suite, all made publicly available. Our released model provides a suitable baseline for such future work.
We would like to thank Valentina Pyatkin, Daniela Stepanov and Oren Pereg for their valuable assistance in the data validation process. The work described herein was supported in part by grants from Intel Labs, Facebook, the Israel Science Foundation grant 1951/17, the Israeli Ministry of Sci
ence and Technology and the German Research Foundation through the German-Israeli Project Cooperation (DIP, grant DA 1600/1-1).
Jun Araki, Zhengzhong Liu, Eduard Hovy, and Teruko Mitamura. 2014. Detecting subevent structure for event coreference resolution. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), Reykjavik, Iceland. European Language Resources Association (ELRA).
Shany Barhom, Vered Shwartz, Alon Eirew, Michael Bugert, Nils Reimers, and Ido Dagan. 2019. Revisiting joint modeling of cross-document entity and event coreference resolution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4179-4189, Florence, Italy. Association for Computational Linguistics.
Cosmin Bejan and Sanda Harabagiu. 2010. Unsupervised event coreference resolution with rich linguistic features. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1412-1422, Uppsala, Sweden. Association for Computational Linguistics.
Andrew Chisholm and Ben Hachey. 2015. Entity disambiguation with web links. Transactions of the Association for Computational Linguistics, 3:145-156.
Prafulla Kumar Choubey and Ruihong Huang. 2017. Event coreference resolution by iteratively unfolding inter-dependencies among events. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2124-2133, Copenhagen, Denmark. Association for Computational Linguistics.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-46.
Agata Cybulska and Piek Vossen. 2014. Using a sledgehammer to crack a nut? Lexical diversity and event coreference resolution. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 4545-4552, Reykjavik, Iceland. European Languages Resources Association (ELRA).
Bhuwan Dhingra, Qiao Jin, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2018. Neural models for reasoning over multiple mentions using coreference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 42-48, New Orleans, Louisiana. Association for Computational Linguistics.
Yotam Eshel, Noam Cohen, Kira Radinsky, Shaul Markovitch, Ikuya Yamada, and Omer Levy. 2017. Named entity disambiguation for noisy text. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 58-68, Vancouver, Canada. Association for Computational Linguistics.
Tobias Falke, Christian M. Meyer, and Iryna Gurevych. 2017. Concept-map-based multi-document summarization using concept coreference resolution and global importance optimization. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 801-811, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
Eduard Hovy, Teruko Mitamura, Felisa Verdejo, Jun Araki, and Andrew Philpot. 2013. Events are not simple: Identity, non-identity, and quasi-identity. In Workshop on events: Definition, detection, coreference, and representation, pages 21-28.
Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5802-5807, Hong Kong, China. Association for Computational Linguistics.
Ben Kantor and Amir Globerson. 2019. Coreference resolution with entity equalization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 673-677, Florence, Italy. Association for Computational Linguistics.
Kian Kenyon-Dean, Jackie Chi Kit Cheung, and Doina Precup. 2018. Resolving event coreference with supervised representation learning and clustering-oriented regularization. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 1-10, New Orleans, Louisiana. Association for Computational Linguistics.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettle-moyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188-197, Copenhagen, Denmark. Association for Computational Linguistics.
Kexin Liao, Logan Lebanoff, and Fei Liu. 2018. Abstract Meaning Representation for multi-document summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1178-1190, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Xueling Lin, Haoyang Li, Hao Xin, Zijian Li, and Lei Chen. 2020. KBPearl: A Knowledge Base Population System Supported by Joint Entity and Relation Linking. Proceedings of the VLDB Endowment, 13(7):1035-1049.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Anne-Lyse Minard, Manuela Speranza, Ruben Urizar, Begoña Altuna, Marieke van Erp, Anneleen Schoen, and Chantal van Son. 2016. MEANTIME, the NewsReader multilingual event and time corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 4417-4422, Portoroz, Slovenia. European Language Resources Association (ELRA).
Joel Nothman, Matthew Honnibal, Ben Hachey, and James R Curran. 2012. Event linking: Grounding event reference in a news archive. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 228-232.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1-40, Jeju Island, Korea. Association for Computational Linguistics.
Sameer Singh, Amarnag Subramanya, Fernando Pereira, and Andrew McCallum. 2012. Wikilinks: A large-scale cross-document coreference corpus labeled via links to Wikipedia. Technical Report UM-CS-2012-015.
Shyam Upadhyay, Nitish Gupta, Christos Christodoulopoulos, and Dan Roth. 2016. Revisiting the evaluation for cross document event coreference. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1949-1958, Osaka, Japan. The COLING 2016 Organizing Committee.
Piek Vossen, Filip Ilievski, Marten Postma, and Roxane Segers. 2018. Don't annotate, but validate: a data-to-text method for capturing event data. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Haoyu Wang, Mo Yu, Xiaoxiao Guo, Rajarshi Das, Wenhan Xiong, and Tian Gao. 2019. Do multi-hop readers dream of reasoning chains? In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 91-97, Hong Kong, China. Association for Computational Linguistics.
A WEC Dataset
A.1 Infoboxs Distillation
Excluding infobox types of broad general events, consisting of many sub-events is necessary as often Wikipedia authors link to a broad event from anchor texts that refer to a subevent (and should be regarded as non-corefering by standard definitions of the event coreference task (Hovy et al., 2013; Araki et al., 2014)). For example, in English Wikipedia many event articles containing the infobox "Election" tend to be pointed at from anchor texts that describe subevents, such as 2016 primaries linking to 2016 United States presidential election. Additionally, for some infobox types, pointing mentions often correspond to related, but not corefering, named entities; hence, we discard the infobox types to avoid such noisy mentions. For example, articles of the infobox type "Race" are often linked from mentions denoting the name of the country in which the race took place. Table 6 presents excluded infobox types due to the above reasons.
A.2 WEC-Eng Infobox Types
As mentioned in the paper (Section 3), we manually explored the various Wikipedia infobox types and selected only those denoting an event. Table 7 shows the number of coreference clusters (event pages) and mentions for each selected infobox type. In the table, infobox types falling under the same general category are grouped together.
A.3 Lexical Distribution of  $\mathrm{ECB + }$  and WEC-Eng Mentions
Event mentions can appear in various lexical forms in a document, such as verbs (e.g. exploded), nominalization (e.g. crash), common nouns (e.g. party, accident) and proper nouns (e.g. Cannes Festival 2016). In order to have a rough estimation of the distribution of these different forms, we manually analyze 100 sampled mentions, from each of WEC-Eng and  $\mathrm{ECB + }$ , and present the statistics in Table 8.
B Further Analysis
B.1 Cross-Domain Experiment and Results
To further assess the difference between the ECB+ and WEC-Eng datasets, we evaluate the cross-domain ability of the model trained on one dataset, and tested on the other one. We use the same pairwise model as in Tables 4 and 5, while only tun
ing the stop criterion for the agglomerative clustering, in the corresponding development set. Table 9 presents the cross-domain performance of our model (1) trained on  $\mathrm{ECB + }$  and evaluated on WECEng and (2) trained on WEC-Eng and evaluated on  $\mathrm{ECB + }$ , together with the corresponding in-domain performances (from Tables 4 and 5) for comparison. The performance drop for both cross-domain experiments indicates that these two datasets address different challenges of cross-document event coreference resolution.
B.2 Qualitative Analysis Examples
Illustrating Section 4.4, Tables 10 and 11 present examples sampled from the predictions of the  $\mathrm{ECB + }$  and WEC-Eng models on their respective development sets.
Infobox	Mention link and context	Non-coreferring relation
Race	.He crashed in Italy, and finished two laps off the lead in Portugal..	X Location
.after several unrewarded drives the year before, namely in Italy..	X Location
did not appear at the British or German Grands Prix..	X Location
Sport	.in the Gold Medal final straight sets by an identical score of 16 – 21..	X Subevent
.2012 FINA World Swimming Championships (25 m) and in the 50 m freestyle..	X Subevent
Wars	.throughout most of World War II, in France Holland and Belgium..	X Location
.During the ensuing campaign the Netherlands were defeated and occupied ..	X Subevent
.and made possible further advance into Crimea and industrially developed Eastern..	X Subevent