SlowGuess/NAACL_2022_OCR huggingface dataset:
The results in Table 1 show our model (KBED) achieves the highest average performance across the datasets by a margin of 1.3 F1, reducing errors by  $11.5\%$ . The ablation results indicate the majority of the improvements across the datasets are attributable to our novel KB module. We observe the largest improvement of 3.0 F1 on the WIKI dataset, which is likely due to the documents having high factual density, enabling our model to leverage more KB facts (see Section 6.1 for relation analysis). Despite our model only be trained on Wikipedia, we obtain competitive results on out-of-domain datasets, such as MSNBC news articles, which implies the patterns learned from Wikipedia are applicable to other domains. In addition, the results demonstrate that our 3 modules (entity typing, entity descriptions, and KB facts) are complementary; when any module is used in isolation it reduces performance, demonstrating the benefits of a multifaceted approach to ED. Surprisingly, when our KB module is used in isolation it performs on par with the TagMe baseline, which suggests there is reasonable overlap between KB facts and the facts predicted from documents. Note that the AIDA results in Table 1 contain a mixture of models fine-tuned on this dataset (denoted with \*\*) and trained on Wikipedia only (as in our case), so the numbers are not directly comparable.
5.2 Long-tail and ambiguous ED
Our model achieves an average F1 score of 70.1 on the original ShadowLink dataset (Table 2) which substantially outperforms (+16.5 F1) embeddings-based models (GENRE, REL) and moderately out-performs (+4.0 F1) the Bootleg model (Orr et al.,
<sup>12</sup>We add weak labels by labelling spans that match the title of the page with the entity for the page.
${}^{9}$  A long-tail entity is an entity that is linked to less than 56 times from other Wikipedia pages.
10E.g. if the candidates and PEM scores for the mention England were ([England (country)], 0.92) and ([England football team], 0.08) then [England (country)] would be a TOP entity, and [England football team] would be a shadow entity.
11Details in Appendix E.
Method	AIDA	MSNBC	AQUAINT	ACE2004	CWEB	WIKI	AVG
AIDA (Yosef et al., 2011)	78.0	79.0	56.0	80.0	58.6	63.0	69.1
TagMe 2 (Ferragina and Scaiella, 2012)	70.6	76.0	76.3	81.9	68.3	-	-
REL (van Hulst et al., 2020)	89.4	90.7	84.1	85.3	71.9	73.1	82.4
GENRE (Cao et al., 2021)	93.3**	94.3	89.9	90.1	77.3	87.4	88.7
Bootleg* (Orr et al., 2021)	80.9	80.5	74.2	83.6	70.2	76.2	77.6
WNEL (Le and Titov, 2019)	89.7	92.2	90.7	88.1	78.2	81.7	86.8
RLEL (Fang et al., 2019)	94.3**	92.8	87.5	91.2	78.5	82.8	87.9
DCA-RL (Yang et al., 2019)	93.7**	93.8	88.3	90.1	75.6	78.8	86.7
BiBSG (Yang et al., 2018)	93.0**	92.6	89.9	88.5	81.8	79.2	87.5
KBED	90.4	94.8	92.6	93.4	78.2	90.4	90.0
Model Ablations
w/o KB	87.5	94.4	91.8	91.6	77.8	88.7	88.6
KB only	80.3	88.9	83.0	85.0	69.7	80.8	81.3
Entity types only	85.7	91.8	91.8	89.8	74.3	86.1	86.6
Entity descriptions only	84.8	90.5	91.8	90.8	74.1	87.7	86.6
Bilinear RE layer	86.5	94.4	91.4	93.6	77.5	90.9	89.1
Table 1: Entity disambiguation InKB micro F1 scores on test sets. The best value (excl. model ablations) is bold and second best is underlined. *We produced results using the code released by the authors. **Indicates the model was trained on both AIDA and Wikipedia hyperlinks.
2021) which is optimised for tail-performance and also uses entity types and KB facts. On the original dataset, the impact of our KB module is negligible because the limited document context reduces the chances of KB-related entities co-occurring; the strong performance is therefore largely due to the combination of entity types and descriptions. However, we see a notable average improvement of 12.7 F1 on the document-level version of the dataset, with the KB module having a considerable impact especially on the overshadowed entity subset where it contributes 6.7 F1. The performance margin between our model and Bootleg is greater when document-level context is used likely because Bootleg is designed for short contexts and has limited control over which KB facts to use for disambiguation, as all facts are weighted uniformly. We include a more extensive model ablation study in Appendix C.
5.3 Relation extraction module
To analyse the impact of the doc-level RE architecture introduced in Section 3.5 we present results in Tables 1 and 2 of performance with a standard bilinear RE layer (Xu et al., 2021). Our RE architecture leads to an average increase of  $0.9\mathrm{F}1$  on the standard ED datasets, of  $1.3\mathrm{F}1$  on the standard ShadowLink splits, and of  $1.5\mathrm{F}1$  on the ShadowLink doc-level splits. In addition, by avoiding the quadratic complexity bilinear layer, we achieve an increase in inference speed of approximately
2x, as measured on AIDA documents. We include doc-level RE results for our architecture on the DOCRED (Yao et al., 2019) dataset in Appendix D.
In Table 3 we show the results from annotating 50 examples in which the model made an incorrect prediction for both the AIDA test split and the ShadowLink SHADOW-DOC split. Gold not in cands. refers to cases in which the gold entity was not in the top-30 candidates from the PEM table; Missing KB fact are cases where the model correctly predicted a relation connecting two mentions, but the corresponding fact was not in the KB; Dominant PEM is when the initial PEM score for one candidate was high (> 0.8), and the model fails to override this score; Incorrect RE pred. are cases in which the model makes an incorrect RE prediction between two mentions, and where this wrong prediction leads to the wrong choice of entity; Ambiguous ann. refers to gold annotations that are either incorrect or ambiguous.[13]
The results in Table 3 indicate that the largest source of error is the gold entity not being present in the top-30 candidates. This is particularly true for the ShadowLink SHADOW-DOC split, as this split contains a larger number of tail entities which
<sup>13</sup>Note that some examples may contain more than one source of error (or contain an error not clearly in any category), so the sum of the rows will not necessarily be 50.
Method	SHADOW	TOP	TAIL	AVG	SHADOW-DOC	TOP-DOC	DOC-AVG
AIDA (Yosef et al., 2011)	35	56	67	52.7	-	-	-
TagMe 2 (Ferragina and Scaiella, 2012)	29	57	83	56.3	-	-	-
GENRE* (Cao et al., 2021)	26	42	93	53.7	40.9	59.2	50.1
REL (van Hulst et al., 2020)	21	54	91	55.3	-	-	-
Bootleg* (Orr et al., 2021)	44.5	60.0	93.7	66.1	46.9	62.7	54.8
KBED	47.6	64.2	98.5	70.1	60.8	74.2	67.5
Model Ablations
w/o KB	46.4	64.2	98.3	69.6	54.1	72.0	63.1
KB only	26.9	45.7	98.4	57.0	41.9	60.0	51.0
Entity descriptions only	42.1	54.7	97.8	64.9	52.6	65.0	58.8
Entity types only	39.6	55.6	98.5	64.6	47.3	62.1	54.7
Bilinear RE layer	47.1	61.5	97.7	68.8	59.2	72.7	66.0
Table 2: Entity disambiguation InKB micro F1 scores on ShadowLink test sets. SHADOW-DOC and TOP-DOC refers to the extended version of the dataset which includes the full-text of the document to use as additional context. The best value is bold. *We produced results using the code released by the authors.
AIDA	SHADOW-DOC
Gold not in cans.	18	32
Missing KB fact	2	6
Dominant PEM	0	1
Incorrect RE pred.	1	2
Ambiguous ann.	24	4
Total	50	50
are less likely to be mentioned on Wikipedia. For the AIDA dataset, there are also many cases which are in some sense ambiguous. $^{14}$  There are 8 cases in total where the model predicts a relation which it expects to be in the KB, but which is not in fact present. This is largely in the ShadowLink split, where tail entities are likely to be less well represented in Wikidata. The model is generally good at not depending on entity priors; despite every gold candidate in the Shadowlink SHADOW-DOC split being "overshadowed" by a more popular entity in the PEM table, there is only one example where the model fails to override this. Although the model often "over-predicts" relations between mentions, it rarely gets penalised for doing so, as in general the extra facts it predicts are not in the KB, meaning the Incorrect RE pred. count is low.
To further explore the role of missing candidates, Table 4 shows the percentage of the gold entities present in the top-30 candidates we pass to the model, representing a hard upper-bound on the re
call our model can achieve. The results vary from a high coverage of 99.5 for the MSNBC dataset, which largely contains head entities, to a lower coverage for the ShadowLink SHADOW (75.3) and TOP (83.6) splits. Table 4 also shows the coverage if we pass all PEM candidates to the model. For some datasets, such as WIKI, this increases the coverage significantly. However, for the ShadowLink SHADOW split, the coverage is still below  $80\%$ , indicating that better candidate generation strategies are an interesting avenue for future research.
Table 3: Counts per error category from 50 annotations on AIDA-CoNLL and ShadowLink-Shadow datasets.
Main datasets test splits
n	AIDA	MSNBC	AQUAINT	ACE2004	CWEB	WIKI
30	97.8	99.5	95.1	90.9	95.9	93.7
All	1.0	99.5	95.8	92.9	97.0	98.1
ShadowLink splits
SHADOW	TOP	TAIL
30	75.3	83.6	98.6
All	76.8	84.3	98.7
Table 4: Percentage of gold entities in top-n candidates by dataset. We set  $n = {30}$  for this paper.
6.1 Relation predictions
To understand the relations which the model utilises to make predictions, Table 5 displays for the WIKI dataset the number of KB (Wikidata) facts which exist between gold annotated mentions in the documents (Gold), the number of facts between mentions our model predicts with a score above 0.5 (Predicted) and the percentage of gold facts which our model also predicts (Recall).<sup>15</sup>
14These are often cases with national sports teams, such as "Little will miss Australia's fixture..." where "Australia" could refer to the country Australia or the Australian rugby team.
<sup>15</sup> Note that as the RE predictions are continuous, the quantity of facts our model predicts depends entirely on the choice of this threshold.
The SAME AS relation is used extensively by the model, demonstrating that using coreferences to other (potentially easier to disambiguate) mentions of the same entity in the document is a powerful addition for ED. We leave evaluation of the model on the coreference-specific task to future work. The OTHER relation is also commonly predicted, suggesting the long tail of relations in Wikidata still hold useful information. The other widely used relations are generally either geographical or sports related, which is expected given the large number of sports entities in Wikidata.
The recall numbers appear low, although this is expected behaviour in that the existence of a Gold fact does not necessarily imply that the text in the document infers this fact. For example, the text "Donald Trump visited New York" would include the gold fact [Donald Trump] [place of birth] [New York] but making this prediction for all sentences of this form would likely harm performance.
Gold	Predicted	Recall
sport	1083	1028	0.53
shares border with	1012	5211	0.68
OTHER	1011	10077	0.29
SAME AS	940	9666	0.36
country	890	285	0.10
located in the a.t.e	709	4912	0.66
contains a.t.e	278	319	0.26
instance of	151	1241	0.23
country of citizenship	120	90	0.08
subclass of	90	2430	0.26
genre	83	204	0.29
part of	80	2154	0.44
follows	69	1104	0.67
followed by	68	1312	0.71
member of sports team	63	449	0.83
Table 5: Analysis of relation predictions for WIKI dataset with threshold 0.5. 320 documents with 6772 entity mentions.
We presented a novel ED model, which achieves SOTA performance on well-established ED datasets by a margin of 1.3 F1 on average, and by 12.7 F1 on the challenging ShadowLink dataset. These results were achieved by introducing a method to incorporate large symbolic KB data into an ED model in a fully differentiable and scalable fashion. Our analysis shows that better candidate-generation strategies are an interesting avenue for future research, if results are to be pushed higher on ambiguous and tail entities. Dynamic expansion
of the KB by incorporating facts identified by the ED model is also a potentially promising direction.
We would like to thank Vera Provatorva for providing us with the extended version of the ShadowLink dataset and Laurel Orr for assisting us with running the Bootleg baseline.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and F. Petroni. 2020. Autoregressive entity retrieval. ArXiv, abs/2010.00904.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In International Conference on Learning Representations.
Alberto Cetoli, Stefano Bragaglia, Andrew D. O'Harney, Marc Sloan, and Mohammad Akbari. 2019. A neural approach to entity linking on wikidata. Advances in Information Retrieval, page 78-86.
Xiao Cheng and Dan Roth. 2013. Relational inference for wikification. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1787-1796, Seattle, Washington, USA. Association for Computational Linguistics.
William W. Cohen, Haitian Sun, R. Alex Hofer, and Matthew Siegler. 2020. Scalable neural methods for reasoning with a symbolic knowledge base. In International Conference on Learning Representations.
Silviu Cucerzan. 2007. Large-scale named entity disambiguation based on Wikipedia data. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 708-716, Prague, Czech Republic. Association for Computational Linguistics.
Zheng Fang, Yanan Cao, Qian Li, Dongjie Zhang, Zhenyu Zhang, and Yanbing Liu. 2019. Joint entity linking with deep reinforcement learning. In The World Wide Web Conference, WWW '19, page 438-447, New York, NY, USA. Association for Computing Machinery.
Paolo Ferragina and Ugo Scaiella. 2012. Fast and accurate annotation of short texts with wikipedia pages. IEEE Software, 29:70-75.
Evgeniy Gabrilovich, Michael Ringgaard, and Amarnag Subramanya. 2013. Facc1: Freebase annotation of clueweb corpora. Preprint.
Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2619-2629, Copenhagen, Denmark. Association for Computational Linguistics.
Zhaochen Guo and Denilson Barbosa. 2018. Robust named entity disambiguation with random walks. Semantic Web, Preprint(Preprint):1-21.
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing,
pages 782-792, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
Nitisha Jain, Jan-Christoph Kalo, Wolf-Tilo Balke, and Ralf Krestel. 2021. Do embeddings actually capture knowledge graph semantics? In Eighteenth Extended Semantic Web Conference - Research Track.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Phong Le and Ivan Titov. 2018. Improving entity linking by modeling latent relations between mentions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1595-1604, Melbourne, Australia. Association for Computational Linguistics.
Phong Le and Ivan Titov. 2019. Boosting entity linking performance by leveraging unlabeled documents. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1935-1945, Florence, Italy. Association for Computational Linguistics.
Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. CoRR, abs/1804.05392.
Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-shot entity linking by reading entity descriptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3449–3460, Florence, Italy. Association for Computational Linguistics.
Jiangtao Ma, Duanyang Li, Yonggang Chen, Yaqiong Qiao, Haodong Zhu, and Xuncai Zhang. 2021. A knowledge graph entity disambiguation method based on entity-relationship embedding and graph structure embedding. Comput. Intell. Neurosci., 2021:2878189.
David N. Milne and Ian H. Witten. 2008. Learning to link with wikipedia. In CIKM '08.
Aisha Mohamed, Shameem Parambath, Zoi Kaoudi, and Ashraf Aboulnaga. 2020. Popularity agnostic evaluation of knowledge graph embeddings. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), volume 124 of Proceedings
of Machine Learning Research, pages 1059-1068. PMLR.
Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014. Entity linking meets word sense disambiguation: a unified approach. Transactions of the Association for Computational Linguistics, 2:231-244.
Isaiah Onando Mulang', Kuldeep Singh, Chaitali Prabhu, Abhishek Nadgeri, Johannes Hoffart, and Jens Lehmann. 2020. Evaluating the impact of knowledge graph context on entity disambiguation models. Proceedings of the 29th ACM International Conference on Information Knowledge Management.
Yasumasa Onoe and Greg Durrett. 2020. Fine-grained entity typing for domain independent entity linking. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8576-8583. AAAI Press.
Laurel J. Orr, Megan Leszczynski, Neel Guha, Sen Wu, Simran Arora, Xiao Ling, and Christopher Ré. 2021. Bootleg: Chasing the tail with self-supervised named entity disambiguation. In 11th Conference on Innovative Data Systems Research, CIDR 2021, Virtual Event, January 11-15, 2021, Online Proceedings. www.cidrdb.org.
Maria Pershina, Yifan He, and Ralph Grishman. 2015. Personalized page rank for named entity disambiguation. In NAACL.
Vera Provatorova, Samarth Bhargav, Svitlana Vakulenko, and Evangelos Kanoulas. 2021. Robustness evaluation of entity disambiguation using prior probes: the case of entity overshadowing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10501-10510, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jonathan Raiman and O. Raiman. 2018. Deeptype: Multilingual entity linking by neural type system evolution. In AAAI.
Lev Ratinov, Dan Roth, Doug Downey, and Mike Anderson. 2011. Local and global algorithms for disambiguation to Wikipedia. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1375-1384, Portland, Oregon, USA. Association for Computational Linguistics.
Özge Sevgili, Alexander Panchenko, and Chris Biemann. 2019. Improving neural entity disambiguation with graph embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 315-322, Florence, Italy. Association for Computational Linguistics.
Johannes M. van Hulst, Faegheh Hasibi, Koen Dercksen, Krisztian Balog, and Arjen P. de Vries. 2020. Rel: An entity linker standing on the shoulders of giants. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 2197-2200, New York, NY, USA. Association for Computing Machinery.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierrick Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2019. Zero-shot entity linking with dense entity retrieval. CoRR, abs/1911.03814.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero-shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6397-6407, Online. Association for Computational Linguistics.
Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, and Zhendong Mao. 2021. Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction. CoRR, abs/2102.10249.
Xiyuan Yang, Xiaotao Gu, Sheng Lin, Siliang Tang, Yueting Zhuang, Fei Wu, Zhigang Chen, Guoping Hu, and Xiang Ren. 2019. Learning dynamic context augmentation for global entity linking. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 271-281, Hong Kong, China. Association for Computational Linguistics.
Yi Yang, Ozan Irsoy, and Kazi Shefaet Rahman. 2018. Collective entity disambiguation with structured gradient tree boosting. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 777-786, New Orleans, Louisiana. Association for Computational Linguistics.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. Docred: A large-scale document-level relation extraction dataset. CoRR, abs/1906.06127.
Mohamed Amir Yosef, Johannes Hoffart, Ilaria Bordino, Marc Spaniol, and Gerhard Weikum. 2011. Aida: An online tool for accurate disambiguation of named entities in text and tables. Proc. VLDB Endow., 4(12):1450-1453.
A Entity Type Selection
Our entity types are formed from direct Wikidata relation-object pairs and relation-object pairs inferred from the Wikidata subclass hierarchy; for example, (instance of, geographical area) can be inferred from (instance of, city). We only consider types with the following relations: instance of, occupation, country, and sport. We select types by iteratively adding types that separate (assuming an oracle type classifier) the gold entity from negative candidates for the most examples in our Wikipedia training dataset.
We use the Hugging Face implementation of RoBERTa (Wolf et al., 2019) and optimise our model using Adam (Kingma and Ba, 2015) with a linear learning rate schedule. We ignore the loss from mentions where the gold entity is not in the candidate set. Our model has approximately 197M trainable parameters. We present our main hyperparameters in Table 6. Due to the high computational cost of training the model, we did not conduct an extensive hyperparameter search. To reduce GPU memory usage to below 16 GB during training, we subsample 30 mentions per context window, and subsample 5 candidates per mention (subsampling is not required during inference).
Hyperparameter	Value
learning rate	3e-5
batch size	64
max sequence length	512
dropout	0.05
task hidden layer units	768
# training steps	1M
# candidates	30
# relations	128
# entity types	1400
mention transformer init.	roberta-base
# mention encoder layers	12
description transformer init.	roberta-base
# description encoder layers	2
# description tokens	32
RE transformer init.	random
RE coarse-to-fine K	600
# RE transformer layers	4
C Model Ablation Study
In this section, we measure the contribution of key aspects of our model. For each model ablation, we