SlowGuess/NAACL_2022_OCR huggingface dataset:
Phong Le and Ivan Titov. 2018. Improving entity linking by modeling latent relations between mentions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1595-1604, Melbourne, Australia. Association for Computational Linguistics.
Robert Leaman and Zhiyong Lu. 2016. Taggerone: joint named entity recognition and normalization with semi-markov models. Bioinformatics, 32(18):2839-2846.
Ying Lin, Chin-Yew Lin, and Heng Ji. 2017. List-only entity linking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 536-541, Vancouver, Canada. Association for Computational Linguistics.
Xiao Ling, Sameer Singh, and Daniel S. Weld. 2015. Design challenges for entity linking. Transactions of the Association for Computational Linguistics, 3:315-328.
Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. 2020. Self-alignment pretraining for biomedical entity representations. arXiv preprint arXiv:2010.11784.
Robert L. Logan IV, Matt Gardner, and Sameer Singh. 2020. On importance sampling-based evaluation of latent language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2171-2176, Online. Association for Computational Linguistics.
Robert L Logan IV, Andrew McCallum, Sameer Singh, and Daniel Bikel. Benchmarking scalable methods for streaming cross document entity coreference.
Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-shot entity linking by reading entity descriptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3449–3460, Florence, Italy. Association for Computational Linguistics.
David Milne and Ian H Witten. 2008. Learning to link with wikipedia. In Proceedings of the 17th ACM conference on Information and knowledge management, pages 509-518.
Sunil Mohan and Donghui Li. 2019. Medmentions: A large biomedical corpus annotated with umls concepts. arXiv preprint arXiv:1902.09476.
Maximillian Nickel and Douwe Kiela. 2018. Learning continuous hierarchies in the Lorentz model of hyperbolic geometry. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3779-3788. PMLR.
Tommaso Di Noia, Vito Claudio Ostuni, Paolo Tomeo, and Eugenio Di Sciascio. 2016. Sprank: Semantic path-based ranking for top-n recommendations using linked open data. ACM Transactions on Intelligent Systems and Technology (TIST), 8(1):1-34.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems.
Jonathan Raiman and Olivier Raiman. 2018. Deeype: multilingual entity linking by neural type system evolution. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
James Ravenscroft, Amanda Clare, Arie Cattan, Ido Dagan, and Maria Liakata. 2021. CD^2CR: Coreference resolution across documents and domains. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 270-280, Online. Association for Computational Linguistics.
Sameer Singh, Amarnag Subramanya, Fernando Pereira, and Andrew McCallum. 2011. Large-scale cross-document coreference using distributed inference and hierarchical models. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 793-803, Portland, Oregon, USA. Association for Computational Linguistics.
Mujeen Sung, Hwisang Jeon, Jinhyuk Lee, and Jaewoo Kang. 2020. Biomedical entity representations with synonym marginalization. arXiv preprint arXiv:2005.00239.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Zero-shot entity linking with dense entity retrieval. In EMNLP.
Nishant Yadav, Ari Kobren, Nicholas Monath, and Andrew McCallum. 2019. Supervised hierarchical clustering with exponential linkage. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6973-6983. PMLR.
Sheng Zhang, Hao Cheng, Shikhar Vashishth, Cliff Wong, Jinfeng Xiao, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Knowledge-rich self-supervised entity linking. arXiv preprint arXiv:2112.07887.
Wenzheng Zhang and Karl Stratos. 2021. Understanding hard negatives in noise contrastive estimation. arXiv preprint arXiv:2104.06245.
Each training procedure is run on a single machine using 2 NVIDIA Quadro RTX 8000 GPUs. Our dual-encoder models for ZeShEL and MedMentions have 218M and 230M parameters, respectively. Each variant is optimized using mini-batch gradient descent using the Adam optimizer for 5 epochs using a mini-batch size of 128 to accumulate the gradients. Experiments with batch sizes  $< 128$  performed poorly, possibly due to increased fluctuation of gradients, and sizes  $>128$  were computationally infeasible to run with our available compute resources. For ZeShEL, the dual-encoder models are trained using 192 warm-up steps and learning rates of 1e-5, 3e-5, and 3e-5 for IN-BATCH, K-NN, and ARBORESCENCE-based models, respectively. For MedMentions, each model is trained using 464 warm-up steps and a learning rate of 3e-5. All cross-encoder models are trained with a minibatch size of 2, learning rate of 2e-5, and an additional linear layer. Our MedMentions and ZeShEL cross-encoder models have 108M and 109M parameters, respectively. We use FAISS $^2$  (Johnson et al., 2017) for fast nearest-neighbor search during graph construction at both training and inference. For MedMentions, the execution time was 70 mins to embed and index 2M entities and 120K mentions, and 20 mins to perform exact nearest-neighbor search for the 120K mentions.
A.2 Dual-Encoder Inference Procedure
Building the Graph The structure of the graph  $G$  impacts the dissimilarity function by changing the paths between pairs of nodes in addition to changing which pairs of nodes are connected. We advocate for a simple, deterministic approach to construct this graph. For each mention  $m$ , construct  $E_{m}$  by (a) adding edges from  $m$ 's  $k$ -nearest neighbor mentions in  $\mathcal{M}$  to  $m$ , and (b) adding an edge from  $m$ 's nearest entity to  $m$ :
$$
\begin{array}{l} E _ {m} = \left\{(u, m) \mid u \in \underset {m ^ {\prime} \in \mathcal {M}} {\operatorname {a r g m i n k}} w _ {m ^ {\prime}, m} \right. (7) \\ \left. \vee u = \operatorname * {a r g m i n} _ {e \in \mathcal {E}} w _ {e, m} \right\} (7) \\ \end{array}
$$
The complete collection of edges  $E$  in  $G$  is given by  $E(G) = \bigcup_{m\in \mathcal{M}}E_m$ . There are other ways that one could conceivably pick the pairs of mentions to
be connected in the graph. For example, one could use the minimum spanning tree over the mentions. This approach, however, has a few drawbacks: (a) the directionality of nearest neighbor relationships is ignored leading to noise in the graph, and (b) the resultant graph includes edges that cross cluster boundaries due to this approach forcing all pairs of mentions to be connected, which is undesirable.
Forming Clusters & Making Predictions To make linking decisions for each mention  $m_{i}^{d}$ , we assign the ID of the entity present in the mention's cluster as the linking label (or NIL if there is no entity in the cluster). Let  $\mathcal{C}(m_i^d)$  be the predicted cluster of mention  $m_{i}^{d}$ , then:
$$
e _ {i} ^ {d} = \left\{ \begin{array}{l l} \mathcal {C} \left(m _ {i} ^ {d}\right) \cap \mathcal {E}, & \text {i f} | \mathcal {C} \left(m _ {i} ^ {d}\right) \cap \mathcal {E} | = 1 \\ \text {N I L}, & \text {o t h e r w i s e} \end{array} . \right. \tag {8}
$$
Furthermore, the target clusters we aim to predict in the entity discovery setting are exactly  $\mathcal{C}$ .
A.3 Experiment: Dual-Encoder Linking
Each model is evaluated using three inference procedures. "Independent" refers to predictions made using only mention-entity edges. This method was used by Wu et al. (2020) to generate candidates for a cross-encoder model trained on ZeShEL. "Clustering (UNDIRECTED)" refers to a hierarchical agglomerative clustering (HAC) procedure, following previous work by Angell et al. (2021), which is akin to the procedure for positive sampling used for training our arborescence models but with no edge directionality. "Clustering (DIRECTED)" adds directed edges to the previous method. For each model, we pick the best performing inference procedure on the validation set and report the test set performance.
We report the linking accuracy in Table 6 and leave out models from previous works since they do not report linking metrics of their candidate generators. We specify the inference method used in each case, chosen based on the validation set accuracy of the models. Similar to our cross-encoder results in Table 3, we also report the "seen" and "unseen" performance on MedMentions.
A.4 Qualitative Results
In Table 7 and Table 8, we provide a set of representative examples that demonstrate the improvement in entity linking that our proposed coreference-based methodology empirically provides on Med-Mentions and ZeShEL, respectively.
$^{2}$ https://github.com/facebookresearch/ faiss
Training	MedMentions	ZeShEL
Inference	Overall	Seen	Unseen	Inference	Overall
IN-BATCH NEGATIVES	Clustering (UNDIRECTED)	59.11	61.88	49.45	Independent	39.27
K-NN NEGATIVES	Independent	56.86	64.03	31.88	Independent	49.81
ARBORESCENCE†	Clustering (DIRECTED)	72.19	77.48	53.79	Independent	50.31
1-NN ARBORESCENCE†	Clustering (DIRECTED)	72.00	77.29	53.60	Clustering (DIRECTED)	51.09
1-RAND ARBORESCENCE†	Clustering (DIRECTED)	71.33	77.02	51.51	Clustering (DIRECTED)	50.85
Table 6: Dual-Encoder Linking Results: Accuracy % (†Procedures incorporating explicit mention-to-mention coreference relationships)
Mention	[...] Mutations of critical amino acids affected either dsDNA recombination or both ssDNA and dsDNA recombination indicating two separable functions, one of which is critical for dsDNA recombination and the second for recombination per se [...]
Angell et al. (2021)	DNA (C0012854): ( Chemical, DNA, Deoxyribonucleic Acid, substance: dna molecules; dsDNA; Deoxyribonucleic acid; dna / desoxyribonucleic acid; DNA / desoxyribonucleic acid; DNA molecule; DNA - Deoxyribonucleic acid [...]
Ours	DNA, Double - Stranded (C0311474): Chemical, substance: double stranded dna; DNA, Double Stranded; Double - Stranded DNA; ds dna; deoxyribonucleic acid double strand [...]
Mention	[...] mean dose, and maximum dose were significantly associated with parotid gland atrophy. Multi-variate analysis indicated that only V5 was significantly associated with atrophy. Increasing V5 was a significant risk factor for parotid gland atrophy after carbon ion radiotherapy [...]
Angell et al. (2021)	Muscular Atrophy (C0026846): Biologic Function, Muscular, diagnosis, disorder, finding, physical finding: atrophy; muscle; amyotrophy; muscle atrophy was seen; Wasting; muscle; Atrophies, Muscle; Muscle thinning [...]
Ours	Atrophy of parotid gland (C0341045): ( Biologic Function, disorder: atrophy; parotid gland )
Mention	[...] This study aimed to determine the methylation phenotype in colorectal cancer for identification of predictive markers for chemotherapy response. We performed DNA methylation profiling on 43 non- recurrent and five recurrent colorectal cancer patients using the Illumina Infinium HumanMethylation450 Beadchip assay [...]
Angell et al. (2021)	Disease Response (C1704632): Finding: Response; response
Ours	Response to treatment (C0521982): Clinical Attribute, context - dependent category, finding, function, observable entity, situation: response to treatment; response treatments; Therapeutic response; successful treatment [...]
Table 7: MedMentions: Improved Candidate Generation Yields Correct Entity Linking. Above are examples of mentions where the TF-IDF candidate generation procedure from Angell et al. (2021) fails to retrieve the correct entity, and thus their cross-encoder is not able to correctly link the mention. Our coreference-based dual-encoder is able to retrieve the correct entity in the candidate set of 64 entities, and then the cross-encoder is able to link the mention to the correct entity.
Table 8: ZeShEL: Improved Detail in Linked Concept. Above are examples of mentions where we see a clear trend of increasing detail in the linked concept leading finally to a correct link prediction when using the baseline dual-encoder (Wu et al., 2020), our coreference-based dual-encoder, and our cross-encoder trained using candidates generated by the coreference-based dual-encoder, respectively.