SlowGuess/NAACL_2021_OCR huggingface dataset:
Model	Accuracy
concatenation	47.2 ± 7.9
[SEP]-separation	49.1 ± 2.6
attribute-separation (no reg.)	54.7 ± 3.8
++attribute-OOV	56.2 ± 2.5
++attribute-shuffle	58.2 ± 3.6
++attribute-OOV + shuffle	61.6 ± 3.6
Table 3: Training on CoNLL-Wikidata and testing on the TAC-KBP 2010 test set reveals that using attribute-separators instead of [SEP] tokens yields string representations for candidates that result in more accurate models. Regularization schemes (Section 4.2) further improve accuracy to  $61.6\%$  without using any training data from the test KB.
4. Do improvements for generalizing to unseen  $\kappa \mathcal{B}_{test}$  also translate to scenarios where there is training data that also links to  $\kappa \mathcal{B}_{test}$ ?
For all experiments, we report the mean and standard deviation of the accuracy across five runs with different random seeds.
Our primary experiments focus on the first two research questions and study the accuracy of the model that uses the re-ranking architecture from Section 3 with the three core components introduced in Section 4 viz. attribute-separators to generate string representations of candidates, along with attribute-OOV and attribute-shuffle for regularization. We compare this against two baselines without these components that use the same architecture and use concatenation and [SEP]-separation instead of attribute-separators. As a reminder, all models are trained as well as validated on CoNLL-Wikidata and evaluated on the completely unseen TAC-KBP 2010 test set.
Results confirm that adding structure to the candidate string representations via [SEP] tokens leads to more accurate models compared to generating strings by concatenation (Table 3). Using attribute-separators instead of [SEP] tokens leads to an absolute gain of over  $5\%$  and handling unseen attributes via attribute-OOV further increases the accuracy to  $56.2\%$ , a  $7.1\%$  increase over the [SEP] baseline. These results show that the attribute-separators capture meaningful information about attributes, even when only a small number of attributes from the training data (15) are observed during inference.
$^{9}$ https://gluon-nlp.mxnet.io/api/modules/optimizer.html#gluonnlp optimizer.BERTAdam
Model	Accuracy
[SEP]-separation	62.6 ± 0.8
attribute-separation
++attribute-OOV + shuffle	66.8 ± 2.8
Shuffling attribute-value pairs before converting them to a string representation using attribute-separators also independently provides an absolute gain of  $3.5\%$  over the model which uses attribute-separators without shuffling. Overall, models that combine attribute-shuffling and attribute-OOV are the most accurate with an accuracy of  $61.6\%$ , which represents a  $12\%$  absolute gain over the best baseline model.
Prior work (Raiman and Raiman, 2018; Cao et al., 2018; Wu et al., 2020; Févry et al., 2020) reports higher accuracies on the TAC data but they are fundamentally incomparable with our numbers due to the simple fact that we are solving a different task with three key differences: (1) Models in prior work are trained and evaluated using mentions that link to the same KB. On the contrary, we show how far we can go without such in-KB training mentions. (2) The test KB used by these works is different from our test KB. Each entry in the KB used by prior work simply consists of the name of the entity with a textual description, while each entity in our KB is represented via multiple attribute-value pairs. (3) These models exploit the homogeneous nature of the KBs and usually pre-train models on millions of mentions from Wikipedia. This is beneficial when the training and test KBs are Wikipedia or similar, but is beyond the scope of this work, as we build models applicable to arbitrary databases.
5.5 Training on multiple unrelated datasets
An additional benefit of being able to link to multiple KBs is the ability to train on more than one dataset, each of which links to a different KB with different schemas. While prior work has been unable to do so due to its reliance on knowledge of  $\mathcal{KB}_{test}$ , this ability is more crucial in the settings we investigate, as it allows us to stack independent datasets for training. This allows us to answer our third research question. Specifically, we compare the [SEP]-separation baseline with our full model that uses attribute-separators, attribute-shuffle, and attribute-OOV. We ask whether the
Table 4: Adding the Wikia dataset to training improves accuracy of both our model and the baseline, but our model still outperforms the baseline by over  $4\%$ .
% of TAC training data	[SEP]-sep.	Attribute-sep.
w/ reg.	w/o reg.
0%	49.1 ± 2.6	61.6 ± 3.6
1%	62.4 ± 3.1	69.0 ± 0.5	70.0 ± 2.8
5%	70.1 ± 2.5	72.8 ± 1.5	76.0 ± 1.6
10%	74.5 ± 2.0	76.0 ± 0.8	77.8 ± 1.6
25%	80.1 ± 1.2	78.8 ± 0.4	80.8 ± 1.0
50%	81.8 ± 1.0	80.5 ± 0.4	82.8 ± 1.1
75%	83.1 ± 1.0	81.1 ± 0.2	84.0 ± 0.5
100%	84.1 ± 0.6	81.8 ± 0.9	84.9 ± 0.7
TAC-only	83.6 ± 0.7	83.8 ± 0.9
Table 5: Experiments with increasing amounts of training data that links to the inference KB reveal that models with attribute separators but without any regularization are the most accurate across the spectrum.
differences observed in Table 3 also hold when these models are trained on a combination of two datasets viz. the CoNLL-Wikidata and the Wikia datasets, before being tested on the TAC-KBP 2010 test set.
Adding the Wikia dataset to training increases the accuracy of the full model by  $6\%$ , from  $61.6\%$  to  $66.8\%$  (Table 4). In contrast, the baseline model observes a bigger increase in accuracy from  $49.1\%$  to  $62.6\%$ . While the difference between the two models reduces, the full model remains more accurate. These results also show that the seamless stacking of multiple datasets allowed by our models is effective empirically.
5.6 Impact of schema-aware training data
Finally, we investigate to what extent do components introduced by us help in linking when there is training data available that links to the inference KB,  $\mathcal{KB}_{test}$ . We hypothesize that while attribute-separators will still be useful, attribute-OOV and attribute-shuffle will be less useful as there is a smaller gap between training and test scenarios, reducing the need for regularization.
For these experiments, models from Section 5.4 are further trained with increasing amounts of data from the TAC-KBP 2010 training set. A sample of 200 documents is held out from the training data as a validation set. The models are trained with the exact same configuration as the base models, except with a smaller constant learning rate of  $2 \times 10^{-6}$  to not overfit on the small amounts of data.
Unsurprisingly, the accuracy of all models increases as the amount of TAC training data in
Category Mention	Gold Entity	Prediction	% Errors
Specific	... Richard Blumenthal of Connecticut and Roy Cooper of North Carolina ...	Connecticut	Hartford, Connecticut	33%
Generic	According to the NY Times, “Foreigners and Lhasa residents who witnessed the violence ...	The New York Times	The New York Times Company	6%
Context	... the Tombigbee River, some 25 miles (40 kilometers) north of Mobile. It’s on a river ...	Mobile, Ohio	Mobile River	33%
Related	The striker arrived from his native city of Santos in a helicopter ...	Santos (city)	Santos, F.C.	21%
String	the apparent public support for Camara in Guinea had put the organisation in a quandary	Guinea	Senegal	7%
Table 6: Categorization of model errors, with an example of each, along with the number of errors observed for each category (out of 100). The underlined text under Mention marks the span containing the actual mention.
creases (Table 5).<sup>10</sup> As hypothesized, the smaller generalization gap between training and test scenarios makes the model with only attribute separators more accurate than the model with both attribute separators and regularization.
Crucially, the model with only attribute separators is the most accurate model across the spectrum. Moreover, the difference between this model and the baseline model sharply increases as the amount of schema-aware data decreases (e.g. when using 13 annotated documents, i.e.  $1\%$  of the training data, we get a  $9\%$  boost in accuracy over the model that does not see any schema-aware data). These trends show that our models are not only useful in settings without any data from the target KB, but also in settings where limited data is available.
5.7 Qualitative Analysis
Beyond the quantitative evaluations above, we further qualitatively analyze the predictions of the best model from Table 3 to provide insights into our modeling decisions and suggest avenues for improvements.
5.7.1 Improvements over baseline
First, we categorize all newly correct mentions, i.e. mentions that are correctly linked by the top model but incorrectly linked by the [SEP]-separation baseline by the entity type of the gold entity. This type is one of person (PER), organization (ORG), geo-political entity (GPE), and a catchall unknown
category (UKN). $^{11}$  This categorization reveals that the newly correct mentions represent about  $15\%$  of the total mentions of the ORG, GPE, and UKN categories and as much as  $25\%$  of the total mentions of the PER category. This distributed improvement highlights that the relatively higher accuracy of our model is due to a holistic improvement in modeling unseen KBs across all entity types.
Why does PER benefit more than other entity types? To answer this, we count the fraction of mentions of each entity type that have at least one column represented using attribute separators. This counting reveals that approximately  $56 - 58\%$  of mentions of type ORG, GPE, and UKN have at least one such column. On the other hand, this number is  $71\%$  for PER mentions. This suggests that the difference is directly attributable to more PER entities having a column that has been modeled using attribute separators, further highlighting the benefits of this modeling decision.
5.7.2 Error Analysis
To identify the shortcomings of our best model, we categorize 100 random mentions that are incorrectly linked by this model into six categories (demonstrated with examples in Table 6), inspired by the taxonomy of Ling et al. (2015).
Under this taxonomy, a common error  $(33\%)$  is predicting a more specific entity than that indicated by the mention (the city of Hartford, Connecticut, rather than the state). The reverse is also observed
10The  $0\%$  results are the same as those in Table 3.
This entity typing is present in the KB.
(i.e. the model predicts a more general entity), but far less frequently (6%). Another major error category (33%) is when the model fails to pick up the correct signals from the context and assigns a similarly named entity of a similar type (e.g. the river Mobile, instead of the city Mobile, both of which are locations). 21% of the errors are cases where the model predicts an entity that is related to the gold entity, but is neither more specific, nor more generic, but rather of a different type (Santos Football Club instead of the city of Santos).
Errors in the last category occur when the model predicts an entity whose name has no string overlap with that of the gold entity or the mention. This likely happens when the signals from the context override the signals from the mention itself.
The primary contribution of this work is a novel framework for entity linking against unseen target KBs with unknown schemas. To this end, we introduce methods to generalize existing models for zero-shot entity linking to link to unseen KBs. These methods rely on converting arbitrary entities represented using a set of attribute-value pairs into a string representation that can be then consumed by models from prior work.
There is still a significant gap between models used in this work and schema-aware models that are trained on the same KB as the inference KB. One way to close this gap is by using automatic table-to-text generation techniques to convert arbitrary entities into fluent and adequate text (Kukich, 1983; McKeown, 1985; Reiter and Dale, 1997; Wiseman et al., 2017; Chisholm et al., 2017). Another promising direction is to move beyond BERT to other pre-trained representations that are better known to encode entity information (Zhang et al., 2019; Guu et al., 2020; Poerner et al., 2020).
Finally, while the focus of this work is only on English entity linking, challenges associated with this work naturally occur in multilingual settings as well. Just as we cannot expect labeled data for every target KB of interest, we also cannot expect labeled data for different KBs in different languages. In future work, we aim to investigate how we can port the solutions introduced here to multilingual settings as well develop novel solutions for scenarios where the documents and the KB are in languages other than English (Sil et al., 2018; Upadhyay et al., 2018; Botha et al., 2020).
The authors would like to thank colleagues from Amazon AI for many helpful discussions that shaped this work, and for reading and providing feedback on earlier drafts of the paper. They also thank all the anonymous reviewers for their helpful feedback.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. DBpedia: A Nucleus for a Web of Open Data. In *The Semantic Web*, Lecture Notes in Computer Science, pages 722-735, Berlin, Heidelberg. Springer.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for Sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
Jan A. Botha, Zifei Shan, and Daniel Gillick. 2020. Entity Linking in 100 Languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7833-7845, Online. Association for Computational Linguistics.
Samuel Broscheit. 2019. Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 677–685, Hong Kong, China. Association for Computational Linguistics.
Razvan Bunescu and Marius Paşa. 2006. Using Encyclopedic Knowledge for Named entity Disambiguation. In 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy. Association for Computational Linguistics.
Yixin Cao, Lei Hou, Juanzi Li, and Zhiyuan Liu. 2018. Neural Collective Entity Linking. In Proceedings of the 27th International Conference on Computational Linguistics, pages 675-686, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Andrew Chisholm, Will Radford, and Ben Hachey. 2017. Learning to generate one-sentence biographies from Wikidata. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 633-642, Valencia, Spain. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of
Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jennifer D'Souza and Vincent Ng. 2015. Sieve-Based Entity Linking for the Biomedical Domain. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 297-302, Beijing, China. Association for Computational Linguistics.
Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-Based Dependency Parsing with Stack Long Short-Term Memory. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 334-343, Beijing, China. Association for Computational Linguistics.
Thibault Févry, Nicholas FitzGerald, and Tom Kwiatkowski. 2020. Empirical Evaluation of Pretraining Strategies for Supervised Entity Linking. In *Automated Knowledge Base Construction*.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-Augmented Language Model Pre-Training. arXiv:2002.08909 [cs].
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust Disambiguation of Named Entities in Text. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 782-792, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Heng Ji, Ralph Grishman, Hoa Trang Dang, Kira Griffith, and Joe Ellis. 2010. Overview of the TAC 2010 knowledge base population track. In *In Third Text Analysis Conference (TAC).
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Karen Kukich. 1983. Design of a Knowledge-Based Report Generator. In 21st Annual Meeting of the Association for Computational Linguistics, pages 145-150, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
Ying Lin, Chin-Yew Lin, and Heng Ji. 2017. List-only Entity Linking. In Proceedings of the 55th Annual
Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 536-541, Vancouver, Canada. Association for Computational Linguistics.
Xiao Ling, Sameer Singh, and Daniel S. Weld. 2015. Design Challenges for Entity Linking. Transactions of the Association for Computational Linguistics, 3:315-328.
Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-Shot Entity Linking by Reading Entity Descriptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3449-3460, Florence, Italy. Association for Computational Linguistics.
Kathleen R. McKeown. 1985. Text Generation: Using Discourse Strategies and Focus Constraints to Generate Natural Language Text. Cambridge University Press, USA.
Paul McNamee and Hoa Trang Dang. 2009. Overview of the tac 2009 knowledge base population track. In Text Analysis Conference (TAC), volume 17, pages 111-113. National Institute of Standards and Technology (NIST) Gaithersburg, Maryland....
Isaiah Onando Mulang', Kuldeep Singh, Chaitali Prabhu, Abhishek Nadgeri, Johannes Hoffart, and Jens Lehmann. 2020. Evaluating the Impact of Knowledge Graph Context on Entity Disambiguation Models. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2157-2160. Association for Computing Machinery, New York, NY, USA.
Yasumasa Onoe and Greg Durrett. 2020. Fine-Grained Entity Typing for Domain Independent Entity Linking. arXiv:1909.05780 [cs].
Sergio Oramas, Luis Espinosa Anke, Mohamed Sordo, Horacio Saggion, and Xavier Serra. 2016. ELMD: An Automatically Generated Entity Linking Gold Standard Dataset in the Music Domain. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 3312-3317, Portoorž, Slovenia. European Language Resources Association (ELRA).
Xiaoman Pan, Taylor Cassidy, Ulf Hermjakob, Heng Ji, and Kevin Knight. 2015. Unsupervised Entity Linking with Abstract Meaning Representation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1130-1139, Denver, Colorado. Association for Computational Linguistics.
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020. E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT. arXiv:1911.03681 [cs].
Jonathan Raiman and Olivier Raiman. 2018. DeepType: Multilingual Entity Linking by Neural Type System Evolution. arXiv:1802.01021 [cs].
Ehud Reiter and Robert Dale. 1997. Building applied natural language generation systems. Natural Language Engineering, 3(1):57-87.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A Primer in BERTology: What we know about how BERT works. arXiv:2002.12327 [cs].
Hamed Shahbazi, Xiaoli Z. Fern, Reza Ghaeini, Rasha Obeidat, and Prasad Tadepalli. 2019. Entity-aware ELMo: Learning Contextual Entity Representation for Entity Disambiguation. arXiv:1908.05762 [cs, stat].
Avirup Sil, Ernest Cronin, Penghai Nie, Yinfei Yang, Ana-Maria Popescu, and Alexander Yates. 2012. Linking Named Entities to Any Database. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 116-127, Jeju Island, Korea. Association for Computational Linguistics.
Avirup Sil, Gourab Kundu, Radu Florian, and Wael Hamza. 2018. Neural cross-lingual entity linking. In AAAI.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003.
Shyam Upadhyay, Nitish Gupta, and Dan Roth. 2018. Joint multilingual supervision for cross-lingual entity linking. In EMNLP.
Han Wang, Jin Guang Zheng, Xiaogang Ma, Peter Fox, and Heng Ji. 2015. Language and Domain Independent Entity Linking with Quantified Collective Validation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 695-704, Lisbon, Portugal. Association for Computational Linguistics.
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in Data-to-Document Generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2253-2263, Copenhagen, Denmark. Association for Computational Linguistics.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable Zero-shot Entity Linking with Dense Entity Retrieval. arXiv:1911.03814 [cs].
Ikuya Yamada, Koki Washio, Hiroyuki Shindo, and Yuji Matsumoto. 2020. Global Entity Disambiguation with Pretrained Contextualized Embeddings of Words and Entities. arXiv:1909.00426 [cs].
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced Language Representation with Informative Entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441-1451, Florence, Italy. Association for Computational Linguistics.
Jin Guang Zheng, Daniel Howsmon, Boliang Zhang, Juergen Hahn, Deborah McGuinness, James Hendler, and Heng Ji. 2014. Entity Linking for Biomedical Literature. In Proceedings of the ACM 8th International Workshop on Data and Text Mining in Bioinformatics, DTMBIO '14, pages 3-4, Shanghai, China. Association for Computing Machinery.