SlowGuess/NAACL_2022_OCR huggingface dataset:
1-NN Arborescence Instead of constructing a fully-connected  $k$ -NN graph over the entire gold cluster, in this variant we approximate the arborescence structure by pruning a restricted graph of only the gold entity, the query mention, and the most similar within-cluster mention neighbor of the query. We keep all other details of the training procedure identical to the first variant.
1-Rand Arborescence A third training objective we explore modifies the initial  $k$ -NN graph construction by restricting the nodes to the gold entity, the query mention, and a random within-cluster mention neighbor of the query, instead of the nearest-neighbor.
Baselines We compare to two baselines following previous work: (a) training  $\psi(\cdot, \cdot)$  with random negatives (IN-BATCH NEGATIVES) where each gold entity for a mention in a training batch is treated as a negative example for all other mentions in the batch, and (b) training  $\psi(\cdot, \cdot)$  with hard negatives (K-NN NEGATIVES) similar to the negative mining in our proposed methods albeit with only mention-entity positive selection.
Results In Table 1, we report the test set re-call@64 for each dual-encoder model, where the
prediction is evaluated as a hit if the gold entity is retrieved in the top-64 candidates for a mention. On each dataset, we additionally include the performance of candidate generators used by previous works that we compare to.
We find that models trained with explicit coreference relationships outperform those that incorporate this relationship only indirectly. For recall@64, our proposed methods improve over the baselines by at least 7.94 percentage points on MedMentions and 0.93 points on ZeShEL. Even at linking, or recall@1, our proposed methods show similar improvements with gains of 13.61 and 1.52 points over the next best baseline models. We perform a more comprehensive analysis of the dual-encoder linking performance and describe our inference approach and results in Appendix §A.2 and §A.3.
We posit that much of the observed gains in recall using our methods result from higher quality mention embeddings generated due to a wide array of surface forms available to mention queries at training. Since each training example evaluates not only the gold entity but also its coreferent mentions, this leads to better generalization of representations. We evaluate this improvement in representations in the clustering / coreference setting in §3.5.
3.3 Cross-Encoder Re-ranking
To answer our second research question, we compare five cross-attention models, which are trained using entity candidates generated by the dual-encoder variants discussed in the previous experiment. Training and inference batches are constructed by concatenating each mention with an entity candidate separated by a [SEP] token. Similar to Wu et al. (2020), we use the top-64 retrieved entities as hard negatives during training and as linking candidates during inference.
Results We report the cross-encoder linking accuracy for MedMentions in Table 3. We additionally
Candidate Retriever	Oracle
Acc.	Macro	Self	Union
Acc.	Macro	Acc.	Macro
Dual (IN-BATCH NEGATIVES)	61.27	60.93	64.96	67.81	62.91	66.13
Dual (K-NN NEGATIVES)	61.02	60.47	63.66	66.91	61.30	64.72
Dual (IN-BATCH NEGATIVES) (Wu et al., 2020)	-	61.34	-	-	-	-
Dual (ARBORESCENCE)†	60.72	60.36	62.64	65.90	61.04	64.39
Dual (1-NN ARBORESCENCE)†	60.47	60.48	63.20	66.70	61.03	64.77
Dual (1-RAND ARBORESCENCE)†	62.35	62.53	64.94	67.90	63.33	66.51
Table 4: ZeShEL: Cross-Encoder Linking Results: Unnormalized Accuracy. We report the accuracy of the re-ranker trained using the candidates generated by each retriever. (†Proposed methods)
report the breakdown of accuracy on subsets of test mentions for which the ground truth entities were not evaluated ("unseen") during training, illustrating the zero-shot capability of the models. We also include the current state-of-the-art results by Angell et al. (2021), which uses an n-gram based model for candidate generation and two cross-encoder models, one each for mention-mention and mention-entity scoring, for re-ranking. We observe that each cross-encoder trained with candidates generated by an arborescence-based model outperforms the baselines, including the current SOTA by at least 0.63 points, and the best performing model - ARBORESCENCE - achieves 1.63 point gains. We note, however, that Angell et al. (2021) does better on unseen entities by 1.91 points compared to ARBORESCENCE, which might be a result of benefiting from a reduced search scope owing to the within-document nature of their TF-IDF retriever.
Table 4 contains linking results for ZeShEL, where each reported model varies only in the method used for retrieving the entity candidates, while the cross-encoder re-ranker training method is held constant (K-NN NEGATIVES with  $k = 64$ ). Since ZeShEL is completely zero-shot, we do not include a seen-unseen analysis. We follow Wu et al. (2020) and report the unnormalized accuracy, which is calculated as the percentage of successes out of the total number of query mentions in the test set, and the macro-averaged unnormalized accuracy, which is a simple average of the unnormalized accuracies over the different "worlds", or domains, in the test set. We find that the best performing model is 1-RAND ARBORESCENCE, with a 1.19 point difference in macro-averaged accuracy over the next best model (Wu et al., 2020).
We also note that, unlike on MedMentions, not all of our proposed models have higher accuracy than the mention-entity baselines. Since a key mo
Figure 2: Coreference Distribution. We count the number of mentions in each coreference cluster in MedMentions and ZeShEL. Clusters in ZeShEL are typically very small (at most 3) compared to the MedMentions clusters (up to 1256, 434, and 447 in train, validation, and test, respectively).
tivation for the arborescence-based methods is to explicitly model coreference relationships during training, we expect performance gains to be strongly correlated with the number of coreference links present within the dataset. We analyze the two datasets in terms of the number of mentions for each KB entity, which can be thought of as how large each cluster of coreferent mentions is. We report a histogram distribution in Figure 2 and find that the clusters in ZeShEL are typically very small (at most 3), whereas in MedMentions, each cluster has many more mentions with maximum sizes of 1256, 434, and 447 across the train, validation, and test sets.
Finally, we also provide representative examples of predictions comparing the link predictions by our best-performing ARBORESCENCE-based method to the baseline of Angell et al. (2021) on MedMentions and Wu et al. (2020) on ZeShEL in Appendix Table 7 and Table 8, respectively.
3.4 Oracle Inference
In this setting, we isolate the re-ranking capability of the cross-encoder from the quality of the candidates retrieved at inference. This setting also
removes the upper-bound on re-ranking accuracy by artificially injecting the ground-truth entity in the top-64 candidates retrieved at inference for each mention where retrieval failed. An additional setting we explore holds this oracle candidate set constant across each variant of the cross-encoder by taking a union over all dual-encoder candidate sets and then proceeding to inject the ground-truth. This construction provides a way to disentangle the factor of candidate retrieval quality at inference, which otherwise conflates the comparison of re-ranking performance. We refer to these oracle settings as SELF and UNION, respectively.
Results As seen in column Oracle of Table 3, the baseline models show higher linking accuracy than our proposed methods when the gold entity is guaranteed to be present in the original candidate set. However, the performance of the baseline models drops significantly ( $\geq 32$  points) when evaluated with the UNION candidate set, while the arborescence-based models show a  $\pm 0.9$  point variation. We believe this discrepancy clearly highlights the poor quality of candidates retrieved by the baseline models compared to our proposed methods. This also explains the inflation in accuracy of the baselines on the SELF set due to the trivial discrimination task presented to the cross-encoders. We further point to linking performance on the UNION set, which provides the more challenging task of differentiating between higher quality candidates that are similar. We argue that the large performance difference ( $\geq 26.75$  points) is strongly indicative of the greater linking capacity of our proposed methods.
In Table 4, we report both the micro accuracy and macro-averaged accuracy for the two oracle sets. We observe that 1-RAND ARBO performs the best on the UNION set, but is marginally outperformed by IN-BATCH on micro accuracy on the SELF set by 0.02 points. In contrast to the fluctuation on MedMentions, the relative uniformity in results on the oracle candidate sets indicates that the candidates generated by each model have similar quality.
3.5 Mention Coreference
Next, we evaluate the quality of the learned mention representations for cross-document coreference using the entity labels of each mention as its ground truth cluster assignment. To form clusters, we build mention-only arborescences using the clustering procedure described in §2.3, tuning
Setting	MedMentions	ZeShEL
ALL	ALL/ UNSEEN	UNSEEN ONLY	ALL
IN-BATCH NEGATIVES	0.37	0.71	0.71	0.31
K-NN NEGATIVES	0.26	0.73	0.80	0.29
ARBORESCENCE	0.51	0.83	0.85	0.34
1-NN ARBORESCENCE	0.47	0.75	0.83	0.34
1-RAND ARBORESCENCE	0.35	0.63	0.81	0.32
Table 5: Coreference Results. We report the Adjusted Rand Index achieved by clustering (§2.3) the mention embeddings produced by each model on the test set. We evaluate on three settings: ALL (clustering & evaluating on all mentions), ALL/UNSEEN (clustering all mentions, evaluating on mentions with ground truth entity not seen in training), UNSEEN ONLY (both clustering & evaluating on mentions with ground truth entity not seen in training).
Figure 3: Linking Recall v/s Coreference Recall. We compare the mention coreference recall@64 with the entity linking recall@64 for each dual-encoder training procedure on MedMentions and ZeShEL. There is a positive correlation when comparing coreference-based procedures with entity-only methods, which is stronger on the highly-coreferent MedMentions dataset than on ZeShEL.
the threshold value,  $\lambda$ , based on the validation data. We report the Adjusted Rand Index (ARI) clustering scores in Table 5 using each of our dual-encoder representation learning objectives. For both ZeShEL and MedMentions, we report ARI on all the test mentions (denoted ALL). For MedMentions, we report two additional settings: (a) ARI when clustering mentions with ground truth entity not seen at training (denoted UNSEEN ONLY), and (b) clustering on all mentions but evaluating on the UNSEEN ONLY set (denoted ALL/UNSEEN). The results show that representations learned with the ARBORESCENCE objective perform best on each setting, aligning with the inductive bias of its training procedure and indicating its utility in a setting where new entities must be discovered.
We further probe the inductive bias of the arborescence-based training procedures by inspecting whether improvements in entity linking recall are accompanied by similar gains in mention coreference performance. In Figure 3, we plot entity and mention recall@64 for each training method on the test set of the two datasets. Mention recall is calculated by retrieving 64 nearest-neighbors for each mention and counting the number of neighbors that are coreferent as a proportion of the total number of coreferent mentions limited to 64. Entity recall is calculated as defined in §3.2. We find that entity recall indeed demonstrates a positive correlation with mention recall on both datasets when the proposed coreference-based training procedures are compared with entity-only methods. We posit that this demonstrates the efficacy of using explicit mention coreference relationships to learn representations for entity linking.
Entity Linking Entity linking has been widely studied (Milne and Witten, 2008; Cucerzan, 2007; Lazic et al., 2015b; Gupta et al., 2017; Raiman and Raiman, 2018; Kolitsas et al., 2018; Cao et al., 2021, inter alia). Dutta and Weikum (2015) combine clustering-based cross-document coreference decisions and linking around sparse bag-of-word representations not well suited for the embedding-based representations used in this work. Other works use global or collective models (Kulkarni et al., 2009; Hoffart et al., 2011; Cheng and Roth, 2013; Ganea and Hofmann, 2017; Le and Titov, 2018, inter alia), which consider the compatibility of entity linking decisions made in the same document(s) rather than making independent predictions. Zhang and Stratos (2021) use noise contrastive estimation to mine hard negatives for the linking task.
Cross-document Coreference Several previous works have developed models for the cross-document coreference setting where no entity KB is known in advance (Bagga and Baldwin, 1998; Gooi and Allan, 2004; Singh et al., 2011; Barhomet al., 2019; Cattan et al., 2020; Caciularu et al., 2021; Ravenscroft et al., 2021; Logan IV et al., inter alia).
Alternatives to Cross-Encoders Our work demonstrates how clustering-based training improves dual- and cross-encoder models for linking and discovery. If prediction efficiency, and
not training efficiency, was the only concern, one could also use model distillation to improve dual-encoder performance (Hinton et al., 2015; Izacard and Grave, 2021, inter alia). We could also consider models such as poly-encoders as alternatives to dual-encoders (Humeau et al., 2020).
We present a novel approach for learning mention and entity representations for use in entity linking candidate generation and prediction, as well as in the discovery of new entities. Our method uses an objective that explicitly incorporates mention-to-mention coreference relationships. We demonstrate its empirical effectiveness through analysis on two datasets — MedMentions and the Zero-Shot Entity Linking dataset. As future work, we hope to further analyze these objectives with the lens of efficiency, distillation, and domain transfer.
There are several ways in which entity linking / resolution models could be biased and a potential for those biases to have harmful downstream consequences. There is already a large body of work studying the biases in language models (such as those used for fine-tuning in our work) and coreference models, most notably in understanding when error rates in coreference differ across certain populations (e.g., genders, races, and other entity types, more broadly, that display skewed distributions in the data). For instance, if entity mentions are author names on citation data and the entities are scientific authors, aggregated statistics like h-index or citation count could be biased if the models used to disambiguate the author names are biased. If entity linking and discovery systems are used to build or populate knowledge bases, those systems may propagate these biased predictions. This can be particularly problematic if one used such a biased knowledge base to train future models, thus perpetuating and amplifying the skew. Lastly, we also note that entity linking and discovery are analogous to surveillance and tracking in computer vision, which should warrant substantial weight of ethical considerations.
We thank Nishant Yadav, Sunil Mohan, and members of UMass IESL and NLP groups for helpful discussion and feedback. This work is funded
in part by the Center for Data Science, the Center for Intelligent Information Retrieval, in part by the National Science Foundation under Grant No. 1763618, in part by the Chan Zuckerberg Initiative under the project Scientific Knowledge Base Construction, and in part using high-performance computing equipment obtained under a grant from the Collaborative R&D Fund managed by the Massachusetts Technology Collaborative. Rico Angell is supported by the National Science Foundation Graduate Research Fellowship under Grant No. 1938059. Any opinions, findings, conclusions, and recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsors.
Rico Angell, Nicholas Monath, Sunil Mohan, Nishant Yadav, and Andrew McCallum. 2021. Clustering-based inference for biomedical entity linking. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Amit Bagga and Breck Baldwin. 1998. Entity-based cross-document coreferencing using the vector space model. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 79-85, Montreal, Quebec, Canada. Association for Computational Linguistics.
Shany Barhom, Vered Shwartz, Alon Eirew, Michael Bugert, Nils Reimers, and Ido Dagan. 2019. Revisiting joint modeling of cross-document entity and event coreference resolution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4179-4189, Florence, Italy. Association for Computational Linguistics.
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Peters, Arie Cattan, and Ido Dagan. 2021. Cross-document language modeling. arXiv preprint arXiv:2101.00406.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In International Conference on Learning Representations.
Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, and Ido Dagan. 2020. Streamlining cross-document coreference resolution: Evaluation and modeling. arXiv preprint arXiv:2009.11032.
Xiao Cheng and Dan Roth. 2013. Relational inference for wikification. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1787-1796, Seattle, Washington, USA. Association for Computational Linguistics.
Silviu Cucerzan. 2007. Large-scale named entity disambiguation based on Wikipedia data. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 708-716, Prague, Czech Republic. Association for Computational Linguistics.
Rajarshi Das, Ameya Godbole, Dilip Kavarthapu, Zhiyu Gong, Abhishek Singhal, Mo Yu, Xiaoxiao Guo, Tian Gao, Hamed Zamani, Manzil Zaheer, and Andrew McCallum. 2019. Multi-step entity-centric information retrieval for multi-hop question answering. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 113-118, Hong Kong, China. Association for Computational Linguistics.
Sourav Dutta and Gerhard Weikum. 2015. C3EL: A joint model for cross-document co-reference resolution and entity linking. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 846-856, Lisbon, Portugal. Association for Computational Linguistics.
Nicholas FitzGerald, Dan Bikel, Jan Botha, Daniel Gillick, Tom Kwiatkowski, and Andrew McCallum. 2021. MOLEMAN: Mention-only linking of entities with a mention annotation network. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 278-285, Online. Association for Computational Linguistics.
Evgeniy Gabrilovich, Michael Ringgaard, and Amarnag Subramanya. 2013. Facc1: Freebase annotation of clueweb corpora, version 1 (release date 2013-06-26, format version 1, correction level 0).
Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2619-2629, Copenhagen, Denmark. Association for Computational Linguistics.
Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019. Learning dense representations for entity retrieval. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 528-537, Hong Kong, China. Association for Computational Linguistics.
Chung Heong Gooi and James Allan. 2004. Cross-document coreference on a large scale corpus. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 9–16, Boston, Massachusetts, USA. Association for Computational Linguistics.
Nitish Gupta, Sameer Singh, and Dan Roth. 2017. Entity linking via joint encoding of types, descriptions,
and context. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2681-2690, Copenhagen, Denmark. Association for Computational Linguistics.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 782-792, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations.
Gautier Izacard and Edouard Grave. 2021. Distilling knowledge from reader to retriever for question answering. In International Conference on Learning Representations.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with gpus. arXiv preprint arXiv:1702.08734.
Nikolaos Kolitsas, Octavian-Eugen Ganea, and Thomas Hofmann. 2018. End-to-end neural entity linking. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 519-529, Brussels, Belgium. Association for Computational Linguistics.
Sayali Kulkarni, Amit Singh, Ganesh Ramakrishnan, and Soumen Chakrabarti. 2009. Collective annotation of wikipedia entities in web text. In SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD).
Nevena Lazic, Amarnag Subramanya, Michael Ringgaard, and Fernando Pereira. 2015a. Plato: A selective context model for entity resolution. Transactions of the Association for Computational Linguistics, 3:503-515.
Nevena Lazic, Amarnag Subramanya, Michael Ringgaard, and Fernando Pereira. 2015b. Plato: A selective context model for entity resolution. Transactions of the Association for Computational Linguistics, 3:503-515.