SlowGuess/NAACL_2022_OCR huggingface dataset:
Benjamin Hsu and Graham Horwood
{benhsu, ghorwood}@amazon.com
Identifying related entities and events within and across documents is fundamental to natural language understanding. We present an approach to entity and event coreference resolution utilizing contrastive representation learning. Earlier state-of-the-art methods have formulated this problem as a binary classification problem and leveraged large transformers in a cross-encoder architecture to achieve their results. For large collections of documents and corresponding set of  $n$  mentions, the necessity of performing  $n^2$  transformer computations in these earlier approaches can be computationally intensive. We show that it is possible to reduce this burden by applying contrastive learning techniques that only require  $n$  transformer computations at inference time. Our method achieves state-of-the-art results on a number of key metrics on the ECB+ corpus and is competitive on others.
Coreference resolution is the fundamental NLP task of finding all mentions that refer to the same real world entity or event in text. It is an important step for higher level NLP tasks involving natural language understanding, such as text summarization (Azzam et al., 1999), information extraction (Zelenko et al., 2004), and question-answering (Vicedo and Ferrandez, 2000). Historically, coreference resolution of entities in the same text document – within document (WD) coreference resolution – has received the most attention, though more recently focus has moved toward cross-document (CD) coreference resolution.
CD coreference resolution has recently gained renewed interest for its application in multi-document analysis tasks. CD coreference resolution presents unique challenges not found in the WD context. Spans of text come from different documents without any inherent linear order, and there is no notion
that antecedents for a given expression typically occur before the expression, as in a single document. Coreferent expressions also cannot be assumed to occur near one another. Furthermore, documents are also assumed to be authored independently and about different—though lexically similar—topics. For instance, the event described in the sentences from Topic 19 in Table 1 below are not coreferential, despite their lexical similarity ("killed").
Another important aspect of CD coreference resolution is the potential scale of the problem. In certain applications, the number of documents can be large and ever growing. In particular, for applications that merge information from across documents, such as multi-document summarization (Falke et al., 2017) or multi-hop question answering (Dhingra et al., 2018), the corpus in question can be both large and dynamically increasing in size.
Past methods of CD coreference resolution have treated the problem as a binary classification task: given two pairs of mentions, classify them as referring to the same entity or not (Bejan and Harabagiu, 2010; Yang et al., 2015; Huang et al., 2019; Kenyon-Dean et al., 2018). In more recent works, contextual embeddings using a cross-encoder architecture have been leveraged to obtain state-of-the-art results (Yu et al., 2020; Zeng et al., 2020; Caciularu et al., 2021) on the  $\mathrm{ECB + }$  corpus. Despite achieving state-of-the-art results on the benchmark dataset, a shortcoming of these approaches is the fact they use a transformer as a cross-encoder - two sentences are passed to through the transformer network and a label is predicted. For  $n$  mentions in a corpus, these approaches require  $n^2$  comparisons at inference time. As Reimers and Gurevych (2019) noted when using BERT in a cross-encoder architecture, finding the most similar pair of sentences in a collection of  $n = 10000$  sentences requires  $n(n - 1) / 2 = 49995000$  inference computations, which they estimated to take 65 hours using a V100
Human Language Technologies, pages 3644 - 3655
Subtopic 1	Subtopic 2
Topic 19	Riots Erupt Following Death of Brooklyn Teen Killed By Police	INITIAL results from the post-mortem on a 15-year-old Greek boy whose killing by police sparked five days of rioting show Alexandros Grigoropoulos died from a bullet ricochet.
Yesterday , the police explained that officers shot and killed a 16-year-old Kimani Gray in Brooklyn because he allegedly pointed a gun at the cops.	Fresh riots were reported in Greece on Saturday December 13 2008 in protest at the killing by police of a 15-year-old boy, Alexandros Grigoropoulos, eight days ago
Table 1: Examples of cross-document coreference clusters from topics 19 of the  $\mathrm{ECB + }$  corpus. Bold text indicate events and the same color indicates that they belong in the same coreference cluster. The addition of lexically similar second subtopic (riots in Greece over teenagers death vs riots in Brooklyn over teenagers death) adds an additional challenge to the  $\mathrm{ECB + }$  corpus.
Others have sought to address the quadratic scaling of these methods. Recently, Allaway et al. (2021); Cattan et al. (2021a) introduced methods that require  $n$  transformer passes. In this work, we introduce a method using contrastive learning to generate mention representations that are useful for the coreference resolution problem. Previous attempts along these lines by Kenyon-Dean et al. (2018) introduced clustering-oriented regularization terms in the loss function. Our method improves on these earlier methods on the benchmark dataset, and achieves results competitive with the more expensive methods of Yu et al. (2020); Zeng et al. (2020); Caciularu et al. (2021). We conduct extensive ablations of our model which we discuss in §4.5. We discuss applications to domains outside of the ECB+ corpus in §4.6.
Most recent work on CD coreference resolution has focused on the  $\mathrm{ECB + }$  corpus (Cybulska and Vossen, 2014), which we also use in this work. The  $\mathrm{ECB + }$  corpus, which is an extension of the Event Coreference Bank (ECB), consists of documents from Google News clustered into topics and annotated for event coreference (Bejan and Harabagiu, 2010).  $\mathrm{ECB + }$  increases the difficulty level of the original ECB dataset by adding a second set of documents for each topic (subtopic), discussing a different event of the same type (e.g. riots in Greece over teenagers death vs riots in Brooklyn over teenagers death; see Table 1) (Cybulska and Vossen, 2014). While relatively small, the corpus is representative of the common cross-document coreference use cases across a restricted set of related documents (i.e. results from a search query).
Most approaches to CD coreference resolution
address the problem as a binary classification problem between all pairs of events and entities. Early works utilized hand engineered lexical features (e.g. head lemma, word embedding similarities, etc.) (Bejan and Harabagiu, 2010; Yang et al., 2015). More recent works have relied on neural network methods, utilizing character-based embeddings (Huang et al., 2019; Kenyon-Dean et al., 2018) or contextual embeddings (Yu et al., 2020; Cattan et al., 2020; Zeng et al., 2020; Caciularu et al., 2021; Allaway et al., 2021). Recent approaches by Yu et al. (2020) and Caciularu et al. (2021) leveraging RoBERTa and Longformer transformer models have set strong benchmarks. A drawback of these approaches is the necessity to consider all pairs of  $n$  mentions in a corpus in a cross encoder architecture. Each unique pair of entities (separated by a special token) is passed through a transformer to generate a similarity score. This requires  $n^2$  transformer computations.
This can be computationally expensive and several works have sought to address this. Allaway et al. (2021) introduced a model that clusters mentions sequentially at inference time. They achieved competitive results using a BERT-base model and without using a hierarchical clustering algorithm to generate coreference chains. Cattan et al. (2021a) adapted the model of Lee et al. (2017) to the cross-document context. Specifically, they pruned document spans down to the gold mentions and encode each resulting pared document using a RoBERTa-large model. A pairwise (feed-forward network) scorer then generates a score for each pair of spans. They also considered an end-to-end system where they use their model to predict mention spans instead of using gold mentions. In this work, we consider gold mentions only as has been done in earlier works.
In this work, we introduce a method leveraging contrastive learning using a RoBERTa-large model as the base encoder. At inference time, our method requires  $n$  passes of the transformer, like earlier methods by Allaway et al. (2021); Cattan et al. (2021a). Our method surpasses their methods on the benchmark ECB+ dataset and is competitive with more expensive cross-encoder approaches of Yu et al. (2020); Zeng et al. (2020); Caciularu et al. (2021).
We follow earlier works and use the  $\mathrm{ECB + }$  corpus, which is an extension of the Event Coreference Bank (ECB), which was discussed in the previous section. Following earlier works by others (Yu et al., 2020; Cattan et al., 2020; Caciularu et al., 2021; Allaway et al., 2021), we follow the setup of Cybulska and Vossen (2015), which was also used by others (Yu et al., 2020; Cattan et al., 2020; Caciularu et al., 2021; Allaway et al., 2021). This setup uses a subset of the annotations which has been validated for correctness and allocates a larger portion of the dataset for training. In this setup, we use topics 1-35 as the train set, setting aside topics 2, 5, 12, 18, 21, 23, 34, 35 for hyperparameter tuning, and 36-45 as the test set. To preprocess mentions, we utilized the reference implementation from Cattan et al. (2020). The distribution of the train, test, and development sets can be seen in Table 2.
Train	Dev	Test
# Topics	25	8	10
# Documents	574	196	206
# Event Mentions	3808	1245	1780
# Event Singletons	1116	280	623
# Event Clusters	1527	409	805
# Entity Mentions	4758	1476	2055
# Entity Singletons	814	205	412
# Entity Clusters	1286	330	608
Table 2: Statistics for the  $\mathrm{ECB + }$  corpus. We followed the setup of (Cybulska and Vossen, 2015) and used topics 36-45 for our test set and topics 1-35 for training with topics 2, 5, 12, 18, 21, 23, 34, 35 set aside in the development set for hyperparameter tuning.
Figure 1: Our encoder takes as input the first two sentences from the document and concatenates it with the sentence containing the mention, taking care to annotate the mention location with tags  $[E]$  and  $/ / E$ .
We propose a model to learn embeddings useful for clustering events and entities. Our model leverages a Siamese neural network (Bromley et al., 1993) to fine-tune a RoBERTa-large encoder (see Figure 1). We train and evaluate our model using gold mentions as opposed to predicted mentions in order to focus on the cross-document coreference resolution problem. At inference time, our model generates embeddings for the mentions which are then clustered using an agglomerative clustering algorithm as was done previously by Barhom et al. (2019); Yu et al. (2020); Cattan et al. (2020); Caciularu et al. (2021); Zeng et al. (2020). Below we discuss details of our methodology and training procedure.
Document Context Following Caciularu et al. (2021), we use the observation that other parts of the document provide valuable context to the mentions in question. We extract and encode the first two sentences from the document. This takes advantage of the fact that the articles are news articles and in many cases, much of the relevant information is summarized at the beginning of the document. In most cases, these two sentences are the headline and dateline for the article. In cases where the sentence in question is one of the first two sentences, we take the next sentence in the document.
Contextual Embedding In addition to the document context, we also utilize the sentence that the mention appears in and annotate its location in the sentence using  $[E]$  and  $/ / E]$  tokens. The two sequences are concatenated together using a [SEP] token (see Figure 1). In total, we keep 128 word piece tokens and in cases where the combined input exceeds this, we remove tokens from the end of the context before removing tokens from the sentence
containing the mention.
This combined sequence is encoded using a RoBERTa-large model (Liu et al., 2019), as shown in Figure 1. We fine-tune all layers of the RoBERTa-large model. RoBERTa will produce a representation vector for each token of the input sequence. We then sum up element-wise the token-level representations of the mention and use this as the representation of the mention,  $v_{e}$ . Additionally, we utilize the first token of the sequence  $v_{cls}$  as the embedding for the entire document context and mention embedding. Each of these contextual embeddings are passed separately through a multi-layer perceptron (MLP). We found that 1024 for the hidden layer dimension for both MLPs worked well in our experiments.
$$
v _ {e} ^ {\prime} = M L P _ {1} (v _ {e}); v _ {c l s} ^ {\prime} = M L P _ {2} (v _ {c l s}) \quad (1)
$$
The final representation for the mention  $i$  and its context document is given by the concatenation of the two vectors output vectors, indicated by  $[.;.]$ .
$$
v _ {i} = \left[ v _ {\text {c l s}} ^ {\prime}; v _ {e} ^ {\prime} \right] \tag {2}
$$
At inference time, our model takes in the mention and its context (both the head of the document and its sentence) and generates a 2048 dimensional embedding  $v_{i}$ . A clustering algorithm is applied to embeddings to generate coreference clusters. In order to compare our language model with earlier approaches, we follow earlier works and use an agglomerative clustering model. We use the implementation from scikit-learn<sup>1</sup> and cluster mention representations using the cosine distance metric. Representations within an average threshold distance  $\tau$  are considered to be in the same cluster (i.e. coreferences).
To train the model, we consider pairs of sentences - positive samples are pairs of sentences where the mentions are coreferential while negative samples are pairs of sentences where the mentions are not coreferential. Pairs of sentences were chosen from within gold topics and were constructed by first computing the similarity between sequences. This focuses our model to learn features to distinguish between the two closely related subtopics, one of the key aspects of the  $\mathrm{ECB + }$  corpus. We
Events	Entities
# of Pairs	19000	27090
# of Positive	2085	4078
# of Negatives	16915	23012
# of Same Subtopic	13694	18847
# of Different Subtopic	5306	8243
Fraction Positive	0.11	0.15
Fraction Same Subtopic	0.72	0.70
Median pos. similarity score	0.62	0.59
Median neg. similarity score	0.80	0.77
Table 3: Statistics for the contrastive pairs generated. Pairs of sentences were chosen from within gold topics and were constructed by first computing the similarity between sequences. Negative samples were downsampled by selecting samples whose similarity was greater than the median similarity among all possible sample pairs.
used SBERT (Reimers and Gurevych, 2019) to embed these sequences initially. Positive pairs were created from sequences that were least similar to one another and negative pairs were selected from the set of pairs most similar to one another, both within a particular subtopic and across subtopics (but still within the same topic). Finally, the negative samples were down-sampled by selecting samples whose similarity was greater than the median similarity among all possible positive sample pairs. The resulting distribution for the pairs can be seen in Table 3.
The model parameters were then trained using a Siamese network architecture (Chopra et al., 2005) where model weights are shared across both branches. For a given pair of sentences  $p = (s_1, s_2)$  and label  $y = 1, 0$  where  $y = 1$  if the pairs are coreferences and  $y = 0$  otherwise, each pair of sentences is encoded using our model. The model was trained by minimizing the contrastive loss,  $\ell$  (Hadsell et al., 2006), as implemented by Reimers and Gurevych (2019),
$$
\ell = y * d (i, j) ^ {2} + (1 - y) * \max  (0, m - d (i, j)) ^ {2} \tag {3}
$$
For our purposes,  $d(i,j) = 1 - \cos(v_i, v_j)$  is the cosine distance,  $m > 0$  is a margin, and  $y$  is one if the pairs describe coreferent mentions and zero otherwise. Dissimilar pairs contribute to the loss function only if their distance are within  $m$ . The loss pushes the embeddings so that positive pairs are closer together in the embedding space and negative pairs are pushed to be more distant
1https://scikit-learn.org
than the margin  $m$
In our experiments, we used the AdamW optimizer without warmup and found that a batch size of 16 worked well. We utilized Ray (Liaw et al., 2018) for hyperparameter tuning and specifically the Bayesian optimization search algorithm from scikit-optimize. We performed our experiments on a p3dn.24xlarge with 8 V100 Tensor Core GPUs and chose the dropout rate, learning rate, contrastive margin  $m$  and clustering threshold  $\tau$  to optimize the CoNLL F1 score on the development set gold topics. This was done to learn representations that address the lexical ambiguity in the ECB+ corpus topics. Resulting hyperparameters can be found in Table 4.
Events	Entities
Epochs	100	50
Learning rate	2e-7	2e-7
Batch Size	16	16
Contrastive margin, m	0.40	0.70
Clustering Threshold, τ	0.2	0.2
Table 4: Hyperparameters for our best performing models on events and entities.
We evaluate our model using four different measures as is common in earlier works. Specifically, we evaluated our model performance using MUC (Vilain et al., 1995),  $B^3$  (Bagga and Baldwin, 1998), CEAF-e (Luo, 2005), and LEA (Moosavi and Strube, 2016) metrics. We also evaluate our model using the CoNLL F1, the average of the MUC,  $B^3$ , and CEAF-e F1 scores. As a baseline, we also show results from a lemma model that takes each span in question and utilizes  $\text{spaCy}^3$  to lemmatize each token. Mentions are clustered based on whether their lemmatized tokens are exact matches or not.
Evaluations on  $\mathrm{ECB + }$  test corpus are not without controversy, and we discuss these subtleties in detail below. For the reader familiar with these issues, our main results are discussed in  $\S 4.2$  and  $\S 4.3$ . We also conduct an ablation study with results in  $\S 4.5$ .
4.1 Evaluation Settings
Many earlier methods leveraged an initial document clustering (Yu et al., 2020; Zeng et al., 2020; Caciularu et al., 2021; Allaway et al., 2021). As observed by Barhom et al. (2019); Upadhyay et al. (2016), clustering the documents as a preprocessing step and performing pairwise classification on mentions within each cluster provides a strong baseline. Barhom et al. (2019) introduced a K-Means algorithm to cluster documents using TF-IDF scores of the unigrams, bigrams and trigrams, where K is chosen by utilizing the Silhouette coefficient method (Rousseeuw, 1987). Models are then applied to mentions within each cluster.
However, this approach has come under criticism (Cremisini and Finlayson, 2020; Cattan et al., 2021a,b). Detractors note that, because of the high lexical similarity between documents within the same subtopic, pre-clustering methods are able to produce near perfectly predicted subtopics, especially in the  $\mathrm{ECB + }$  corpus, where only a few coreference links are found across different subtopics. Document clustering is not expected to perform as well in realistic settings where coreferent mentions can spread over multiple topics (Cattan et al., 2021a). More importantly, this bypasses the intention behind the inclusion of subtopics in  $\mathrm{ECB + }$  and avoids challenging the coreference models on lexical ambiguity (Cybulska and Vossen, 2014).
In our view, evaluation utilizing the original topic clusters ("gold" topics) is more in line with the original intent of Cybulska and Vossen (2014) and more indicative of realistic settings (Cattan et al., 2021b). We discuss results (1) using  $\mathrm{ECB + }$  topics ("gold topics" henceforth) as the initial document clustering and (2) using no initial document clustering ("corpus level" henceforth) in section  $\S 4.2$ . We find that our methodology improves on earlier methods (Tables 5 and 9). Finally, because a majority of earlier works evaluate their models using predicted topics, we discuss our model performance under this setting in  $\S 4.3$ . We report results from a single run.
4.2 Gold Topics and Corpus Level
We evaluate our models using the  $\mathrm{ECB + }$  topics, in line with the intent of Cybulska and Vossen (2014) and earlier works by Cattan et al. (2021a,b). According to those authors, this setting was designed to approximate an unclustered stream of news articles.
$^{2}$ https://github.com/scikit-optimize/scikit-optimize
MUC	B3	CEAF-e	LEA	CoNLL
R	P	F1	R	P	F1	R	P	F1	R	P	F1	F1
Events	Gold Topics	Baseline	72.9	72.4	72.7	51.4	56.5	53.8	58.6	40.4	47.8	46.8	51.5	49.1	58.1
Cattan et al. (2021b)	80.1	76.3	78.1	63.4	54.1	58.4	56.3	44.2	49.5	59.7	49.6	54.2	62.0
Ours	87.8	83.0	85.3	78.0	71.4	74.5	71.0	57.4	63.5	75.6	68.9	72.1	74.4
Corpus	Baseline	72.9	60.5	66.1	52	39.2	44.7	48.1	34.7	40.3	46.8	34.6	39.8	50.4
Cattan et al. (2020)	79.9	74.8	77.2	62.2	48.9	54.8	53.3	42.3	47.2	58.4	44.4	50.5	59.7
Ours	86.4	74.9	80.2	76.2	51.7	61.6	59.7	48.7	53.6	73.4	48.7	58.6	65.2
Entities	Gold Topics	Baseline	61.6	85.9	83.2	31.8	80.2	45.5	53.7	33.8	41.5	28	76.9	41	52.9
Cattan et al. (2020)	85.8	79.3	82.4	64.3	60	62.1	58.6	45.9	51.5	60.9	56.8	58.8	65.3
Ours	84.5	90.1	87.2	72.2	80.5	76.2	73.1	59.0	65.3	69.7	78.5	73.8	76.2
Corpus	Baseline	61.9	77.5	68.8	32.5	67.7	43.9	50.1	33.2	39.9	28.1	63.8	39	50.9
Cattan et al. (2020)	85.7	79.3	82.4	63.7	60	61.8	58.1	45	50.7	60.3	56.8	58.5	65
Ours	83.9	86.6	85.2	71.4	75.6	73.4	69.2	55.1	61.4	68.5	73.1	70.7	73.3
Table 5: Combined within- and cross-document coreference scores for entities and events without singletons, using gold mentions. Gold topics use the  $\mathrm{ECB + }$  topics as the initial document pre-clustering while corpus level results do not use any document pre-clustering. Bold values indicate best overall for a particular data subset.
Additionally, as noted by Cattan et al. (2020, 2021a), the presence of singletons biases the results towards models that perform well on detecting all the mentions instead of predicting coreference clusters. Furthermore, in using gold mentions in the evaluation (like we do here), including singletons artificially inflates performance metrics (Cattan et al., 2021a). We present our results without singletons (Table 5) using the reference implementation of Moosavi and Strube (2016). In Appendix A, we give results with singletons in Table 9.
On the gold topic and corpus level subsets, our model performs well. In all cases, we surpass the current state-of-the-art model on the CoNLL F1 metric for both event and entity coreference resolution by large margins without singletons (see Table 5)). We suspect this improvement to be a feature of contrastive learning and methodology we used to choose pairs – coreferential mentions are pushed closer together in the embedding space while mentions that are not coreferences are pushed further apart. We do observe a larger drop in performance in going from gold topics to the corpus level subsets. This is due to the choice in contrastive pairs, where negative examples come from the same gold topic.
Aside from improved performance, our methodology differs in some key aspects to the recent works by Cattan et al. (2021a,b, 2020). Their methodology also leverages a RoBERTa-large model to embed documents, but breaks long documents into 512 word piece token chunks. The authors used as their feature vector for a span in question: the sum of the span embeddings, the embeddings for the span beginning and end, and a vector encoding the span length as their feature