SlowGuess/NAACL_2022_OCR huggingface dataset:
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426, Online. Association for Computational Linguistics.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3027-3035.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235, Online. Association for Computational Linguistics.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: A core of semantic knowledge. In Proceedings of the 16th International Conference on World Wide Web, WWW '07, page 697-706, New York, NY, USA. Association for Computing Machinery.
Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. 2020. CoLAKE: Contextualized language and knowledge embedding. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3660-3670, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Denny Vrandecic and Markus Krötzsch. 2014. Wiki-data: A free collaborative knowledgebase. Commun. ACM, 57(10):78-85.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021a. K-adapter: Infusing knowledge into pre-trained models with adapters. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 1405-1418. Association for Computational Linguistics.
Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerald
Tesauro, Bowen Zhou, and Jing Jiang. 2017.  $\mathbf{R}^3$ : Reinforced reader-ranker for open-domain question answering.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021b. KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation. Transactions of the Association for Computational Linguistics, 9:176-194.
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with BERTserini. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 72-77, Minneapolis, Minnesota. Association for Computational Linguistics.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441-1451, Florence, Italy. Association for Computational Linguistics.
Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5017-5033, Online. Association for Computational Linguistics.
A LM-CORE Training Details
We used the Hugging Face Transformer $^2$  models BERT and RoBERTa as base models for LM-CORE pre-training. We use the BASE (12-layer, 768-hidden, 12-heads) size models and initialize from bert-base-uncased and roberta.base parameters, respectively. This is consistent with the initialization in Peters et al. (2019); Zhang et al. (2019); Wang et al. (2021b) and is a common practice to save up on pre-training time.
We used the Adam (Kingma and Ba, 2015) optimizer and a learning rate of 3e-5 across all settings. We could not perform a lot of hyperparameter tuning owing to the computational requirements of the task. Pre-training was done using 8 Nvidia A100 GPUs with a batch size of 512 using gradient accumulation. The masked LM loss continued to decrease at the end of pre-training, suggesting further improvement in performance can be expected. The pre-trained checkpoints for all four variants of LM-CORE can be found here.
A.1 Pre-Training corpus
We use the English Wikipedia (December 20, 2018) snapshot<sup>3</sup> to create our pre-training corpus and WikiExtractor<sup>4</sup> to process the dumps. This Wikipedia version contains about 5.5M documents. We retain the hyperlinks while extracting Wikipedia articles as we use them for creating entity masks (Section 3.1). Following the entity masking strategy described in Section 3.1, we obtain our pre-training corpus which contains  $\sim 46.3\mathrm{M}$  sentences in total.
During pre-training the base LMs, we sample sentences containing valid masks. The pre-training corpus is maintained consistent across all LM-CORE variants.
A.2 Retrieval corpus
We use two popular knowledge bases (KBs) in LM-CORE - YAGO 4 (Suchanek et al., 2007) and Wikidata5M (Wang et al., 2021b). The statistics of the KBs - number of facts, entities and relations
can be found in Table 6. We describe the preprocessing steps followed to obtain the respective final retrieval corpora in the following subsections.
A.2.1 YAGO
YAGO  $4^{5}$  is in RDF format. YAGO facts are derived from Wikidata, however, all the entities are arranged in a taxonomy mapped to schema.org.
We pre-process YAGO to remove triples involving relationships such as image, logo and url that point to meta-data such as images and other files. We also filter out triples that point to RDF literals or Wikidata URLs.
A.2.2 Wikidata5M
We use the Wikidata5M subset of Wikidata as made available by Wang et al. (2021b)  $^{6}$ . This subset of Wikidata is aligned with Wikipedia such that each entity in Wikidata5M has a corresponding entry in Wikipedia. We used the raw graph as provided in the dataset, the statistics of which are reported in Table 6.
Table 6: Statistics of the knowledge bases used for retrieval in terms of number of triples, entities and relations
Facts	Entities	Relations
Yago	17,421,942	4,927,897	131
Wikidata5M	21,285,880	4,797,808	821
A.3 Example Retrieved Triples
We provide a closer look into our pre-training approach by showing examples of masked input sentences and the retrieved triple candidates from the knowledge base (Table 7). We observe that the facts retrieved are highly relevant for predicting the masked entities in the input context.
B.1 How LM-CORE Compares with Other Retrieval Paradigms
Table 8 also reports results of REALM (Guu et al., 2020) - a retrieval-based language model that retrieves relevant documents from a text corpus during pre-training. We observe that LM-CORE outperforms REALM on the ConceptNet,
$^{5}$ https://yago-knowledge.org/downloads/yago-4
https://deepgraphlearning.github.io/project/wikidata5m
$^{2}$ https://github.com/huggingface/  
transformers  
 $^{3}$ https://archive.org/  
download/enwiki-20181220/  
enwiki-20181220-pages-articles.xml.bz2  
 $^{4}$ https://github.com/attardi/  
wikiextractor
Input	Masked Token	Candidates (correct fact in bold)
Henri Jules Louis Marie Rendu
(24 July 1844 – 16 April 1902) was
a French physician born in [MASK].	Paris	(Henri Jules Louis Marie Rendu; birth date; 1844-07-24)
(Henri Jules Louis Marie Rendu; birth place; Paris)
(Henri Jules Louis Marie Rendu; death date; 1902-04-16)
(Henri Jules Louis Marie Rendu; nationality;y France)
(Henri Jules Louis Marie Rendu; given name; Henri)
Weisenborn attended the [MASK].	University
of Chicago	(Gordon Weisenborn; alumni of; University of Chicago)
(Clara Weisenborn; member of; Republican Party (United States)
(Günther Weisenborn; nationality; Germany)
(Günther Weisenborn; death place; West Berlin)
(Clara Weisenborn; nationality; United States)
Dehorokkhi (English: Bodyguard)
is a Bangladeshi [MASK] directed
by Iftakar Chowdhury.	action film	(Dehorokkhi; director; Iftakar Chowdhury)
(Dehorokkhi; in language; Bengali language)
(Dehorokkhi; genre; Action film)
(Bangladeshi idol; in language; Bengali language)
(British Bangladeshi Who's Who; in language; English language)
Palaemon macrodactylus is
a [MASK] of shrimp of the
family Palaemonidae.	species	(Palaemon macrodactylus; parent taxon Palaemon (genus))
(Palaemon macrodactylus; parent taxon; Palaemon (genus))
(Palaemon macrodactylus; taxonomic rank; Species)
(Palaemonidae; taxonomic rank; Family (biology))
(Palaemonidae; parent taxon; Palaemonoidea)
Table 7: Examples of masked input sentences (from Wikipedia) and top-5 retrieved candidates during pre-training.
DoB (Google-RE), and 1-1 (T-REx) subsets, while REALM outperforms the proposed solution in other subsets of the LAMA probe. We specifically highlight an absolute 15 points improvement on the date-of-birth relation despite REALM using explicit date masks while training whereas our training corpus only has entity masks. This indicates that our model can use the the contextual knowledge provided by the retriever module even though it is not explicitly shown such knowledge during training.
Note that while REALM is similar to our proposed solution as far as the idea of retrieving relevant knowledge is concerned, the key difference in the two approaches lies in the source of knowledge being used. REALM relies on an unstructured text corpus (Wikipedia) as the source of knowledge and employs a computationally complex retrieve and read paradigm requiring additional training of the knowledge retriever model. Our proposed solution, on the other hand, uses structured knowledge which offers the advantage of being (almost) unambiguous and less resource-hungry compared to unstructured text. We present the resource requirements of our approach and REALM in Table 9. Note that the size of the external knowledge (in number of words) used by REALM is an order of magnitude greater, and requires three times the number of parameters compared to our model. Fur
thermore, REALM was trained for 200K steps with a batch size of 512 on an 80 TPU cluster, whereas our proposed solution is much more efficient being trained for 1K steps with a batch size of 512 on a machine with 8 Nvidia A100 GPUs. This computational efficiency of our proposed solution allows us to continue further work on improving our performance by enhancing the structured knowledge base and bridge the performance gap with more complex and computationally expensive models such as REALM.
C LAMA Evaluation
We use the official LAMA data code for evaluating P@1 numbers in Table 1. All the BERT-based models are evaluated using this repository. The LAMA code provides functionality for evaluating RoBERTa models trained in the fairseq framework. Hence, we evaluate RoBERTa-base, RoBERTa-large and KEPLER (Wang et al., 2021b) using this code. The KEPLER repo also points to this code for evaluation. CoLAKE (Sun et al., 2020), has adapted the official code to allow huggingface transformer checkpoints as input, and hence this code is used for CoLAKE, LM-CORE(r,y) and LM-CORE(r,w) evaluation. We ensure the model
$^{7}$ https://github.com/facebookresearch/LAMA
<sup>8</sup>https://github.com/txsun1997/CoLAKE
Table 8: Precision at one (P@1) on LAMA probe. We consider only LM-CORE(b,y) and LM-CORE(b,w) as REALM is trained on BERT. Best results in each column are highlighted in bold and the second best performance is underlined.
Google-RE	T-REx	SQuAD	Concept Net
DoB	PoB	PoD	All	1-1	N-1	N-M	All
BERT-base	1.59	15.46	10.33	9.12	67.94	32.67	23.54	30.83	14.29	15.88
BERT-large	1.59	15.53	12.16	9.76	74.23	31.30	25.30	31.05	17.61	18.72
LM-CORE(b,y)	64.44	52.71	50.98	56.04	74.37	51.18	34.57	45.83	15.61	14.78
LM-CORE(b,w)	0.66	37.62	31.11	23.13	81.79	59.86	45.48	55.32	17.28	16.15
REALM	49.06	79.56	64.13	67.36	55.81	69.54	66.98	68.18	27.96	4.78
Model	no. of params	Retrieval corpus size	Resources used
LM-CORE(b,y)	110M	17M KB triples (~100M words)	8 GPUs
LM-CORE(b,w)	110M	21M KB triples (~140M words)	8 GPUs
REALM	330M	5.5M documents (~2B words)	80 TPUs
Table 9: Resource requirements of LM-CORE and REALM. REALM requires additional ICT pre-training over all Wikipedia documents for initialization.
vocabularies and data is consistent across evaluation. We have used author provided/recommended code and publicly available checkpoints from the official code repositories for all baselines.
C.1 LAMA: Qualitative Analysis
Table 10 and 11 show examples spanning different relationships in LAMA where LM-CORE(b,y) and LM-CORE(r,w) are able to make correct predictions. We also compare the predictions with BERT-base and RoBERTa-base respectively and highlight how these PLMs struggle to make knowledgeable predictions.
C.2 LAMA: Error Analysis
We present various failure cases for LM-CORE in Table 12. These are representative of the type of errors we encountered, however, we observed that majority of the errors resulted due to correct facts missing from the KB.
C.3 Complete LAMA-UHN results
The complete LAMA-UHN results over all subsets of Google-RE and T-REx can be found in Table 13.
D Downstream Evaluation
We discuss the experimental setup and hyperparameter settings for our downstream tasks.
D.1 Zero Shot RE
We consider the open domain version of Zero Short RE (Levy et al., 2017) from Petroni et al. (2021). The dataset is split into three disjoint sets - train (147,909 samples, 84 relations), dev (3,724 samples, 12 relations) and test (4,966 samples, 24 relations). The systems are evaluated on relations never seen during training.
We fine tune our model for 2 epochs with a batch size of 96. We use the Adam (Kingma and Ba, 2015) optimizer and a learning rate of 3e-5. We performed multiple trials by tuning the number of epochs in  $\{1,2,5\}$ .
D.2 Web Questions
Web Questions (Berant et al., 2013) was created using questions that were sampled from the Google Suggest API. We used the same splits as Lee et al. (2019b) with training, dev and test sets containing 3417, 361 and 2032 samples respectively.
We fine tuned our model for 20 epochs – we experimented with number of epochs in {10, 20, 30, 50}. We use the Adam (Kingma and Ba, 2015) optimizer and a learning rate of 3e-5.
E Risks Statement
This work considers training of large language models using large textual corpora as well as structured knowledge bases. The model learns the nuances of the language and correlations between different real-world entities based on the data that is being used for training the model. Hence, there is a chance that the biases and noise in the training data will creep into the model parameters as well that can lead to a biased model behavior. We need
to be careful in deploying the model and extrapolating the output of the model in applications such as search, conversational systems and recommendation systems where model's inherent biases can lead to catastrophic impacts on the user.
Relation	Input query	LM-CORE(b,y) prediction	BERT-base prediction	Candidates
Google-RE	birth-place	Stanley Corrsin was born in _____.	Philadelphia	London	(Stanley Corrsin birth date 1920-04-03)(Stanley Corrsin nationality United States)(Stanley Corrsin birth place Philadelphia)(Stanley Corrsin given name Stanley (given name))(Stanley Corrsin death date 1986-06-02)(Stanley Corrsin has occupation Physicist)(Stanley Corrsin alumni of University of Pennsylvania)(Stanley Corrsin member of American Academy of Arts and Sciences)
birth-date	Tom Coppola (born _____.).	1945	1975	(Tom Coppola birth date 1945-06-06)(Tom Coppola given name Tom (given name))(Tom Coppola nationality United States)(Tom Coppola family name Coppola (surname))(Tom Coppola alumni of USC Thornton School of Music)(Christopher Coppola birth date 1962-01-25)(Anton Coppola nationality United States)(Chris Coppola birth date 1962-01-25)
death-place	Aglaja Orgeni died in _____.	Vienna	Bucharest	(Aglaja Orgeni death date 1926-03-15)(Aglaja Orgeni death place Vienna)(Aglaja Orgeni birth date 1841-12-17)(Aglaja Orgeni birth place Rimavská Sobota)(Aglaja Orgeni nationality Austria)(Aglaja Orgeni nationality Hungary)(Aglaja Orgeni has occupation Opera singer)(Aglaja Orgeni death place Vienna)
T-REX	P106	Cigoli is a ____ by profession.	architect	lawyer	(Cigoli has occupation Architect)(Cigoli nationality Italy)(Cino Cinelli has occupation Businessperson)(Francesco Cirio has occupation Businessperson)(Cigoli birth place Cigoli, San Miniato)(Emilio Cigoli has occupation Stage actor)(Francesco Cigalini has occupation Mathematician)(Ciputra has occupation Businessperson)
P463	Phil Mogg is a member of _____.	UFO	parliament	(Phil Mogg member of UFO (band))(Phil Mogg nationality United Kingdom)(Phil Mogg birth date 1948-04-15)(Phil Mogg birth place London)(Mo Mozzali member of Minneapolis Millers)(John Mogg, Baron Mogg nationality United Kingdom)(Jamie Moyer member of Colorado Rockies)(Jamie Moyer member of Philadelphia Phillies)
P407	Summerfolk was written in _____.	russian	english	(Summerland (novel) in language English language)(Summerfolk in language Russian language)(The World That Summer genre Neofolk)(Summerfolk author Maxim Gorky)(Summer (novel) in language English language)(Summertime (novel) in language English language)(A Summer Tale date published 2000)(Summertooth in language English language )
P1303	Nigel Pulsford plays _____.	guitar	sgt	(Nigel Pulsford has occupation Guitarist)(Nigel Pulsford given name Nigel)(Nigel Pulsford birth date 1963-04-11)(Nigel Pulsford nationality United Kingdom)(Nigel Pulsford nationality Wales)(Nigel Pulsford birth place Newport, Wales)(William Pulsford nationality United Kingdom of Great Britain and Ireland)(Reginald Purdell has occupation Actor)
Table 10: Illustrative examples of cases where LM-CORE(b,y) model successfully output the correct completions for various probes in LAMA. Candidates containing correct fact are in bold.
Relation	Input query	LM-CORE(r,w) prediction	RoBERTa-base prediction	Candidates
Google-RE	birth-place	Sebastiano Maffettone was born in _.	Naples	Rome	(Sebastiano Maffettone place of birth Naples)(Sebastiano Mazzoni place of birth Florence)(Sebastiano Mocenigo place of birth Venice)(Sebastiano Martinelli place of birth Italy)(Sebastiano Baggio place of birth Italy)(Stanley Corrsin has occupation Physicist)(Sebastiano Vassalli place of birth Genoa)(Sebastiano Poma place of birth Parma)
P413	Rivaldo plays in _ position.	midfielder	the	(Rivaldo position played on team forward)(Rivaldo position played on team midfielder)(Rivaldo Gonzalez position played on team midfielder)(Rivaldo Coetzee position played on team defender)(Rivaldo Vitor Mosca Ferreira Junior position played on team forward)(Rivaldo member of sports team brazil national football team)(Rivaldo member of sports team brazil national under-20 football team)(Rivaldo member of sports team São Paulo fc)
P176	Amiibo is produced by _.	Nintendo	Samsung	(amiibo manufacturer ninto) (amibo tap: ninto's greatest bits publisher ninto)(animal crossing: amiibo festival publisher ninto)(amiibo tap: ninto's greatest bits platform wii u)(animal crossing: amiibo festival developer ninto entertainment planning & development)(animal crossing: amiibo festival platform wii u)(amiibo instance of internet protocol)(animal crossing: amiibo festival genre party game)
T-REX	P138	Uraninite is named after _.	uranium	the	(uraninite named after uranium)(uraniborg named after urania)(uranopilite named after compound)(uranopilite named after uranium)(uraninae instance of taxon)(urania parent taxon uraniinae)(uranocircite-ii named after uranium)(30 urania named after urania
P159	The headquarters of Stelco is in _.	Hamilton	Madrid	(Stelco headquarters location Hamilton)(Stelco lake erie works located in the administrative territorial entity Ontario)(Stelco owned by U.S. steel)(Stelco lake erie works country Canada)(Stelco industry ferrous metallurgy)(Stelco instance of business)(Stec, inc. headquarters location California)(Stekey located in the administrative territorial entity louisiana)
P37	The official language of Virrat is _.	Finnish	English	(Virrat official language Finnish)(Virrat country Finland)(Virrat located in the administrative territorial entity Pirkanmaa)(Virrat located in time zone ute+2)(Virrat located in time zone ute+03:00)(Virrat instance of municipality of Finland)(Virrat instance of town)(Virrat instance of city)
Table 11: Illustrative examples of cases where LM-CORE(r,w) model successfully output the correct completions for various probes in LAMA. Candidates containing correct fact are in bold.
Input
Query	Expected
Answer	Model
Output	Retrieved
Candidates	Comments
Hans Gefors was born in _____.	Stockholm	Hamburg	(Hans Raj Hans birth date 1953-11-30)
(Hans Raj Hans given name Hans (name))
(Hans Raj Hans nationality India)
(Hans Raj Hans has occupation Politician)
(Hans Raj Hans member of Indian National Congress)
(Claus Gerson birth place Hamburg)
(Hans Geister birth date 1928-09-28)
(Hans Gericke nationality Germany)	Corresponding fact not present in KB. We speculate that the candidate in bold led the model to predict Hamburg. BERT predicted Oslo as the answer.
Victor Salvi plays _____.	harp	quarterback	(Victor Salvi given name Victor (name))
(Victor Salvi nationality United States)
(Victor Salvi death place Milan)
(Victor Salvi death date 2015-05-10)
(Victor Salvi birth place Chicago)
(Victor Salvi birth date 1920-03-04)
(Jean Lui actor Francesco Salvi)
(Victor Salvi birth date 1920-03-04)	Corresponding fact not present in KB
CBeebies is owned by _____.	BBC	Microsoft	(CBeebies founding date 2002)
(Gigglebiz creator CBeebies)
(CBEF contained in place Ontario)
(Bambi production company The Walt Disney Company)
(CBE Software founding date 2006)
(Paddington Bear (TV series) production company ITV Central)
(CBS Interactive parent organization CBS Corporation)	Corresponding fact not present in KB.
Ivan Petch was born in _____.	Concord	Sydney	(Ivan Petch birth date 1939-03-01)
(Ivan Petch birth place Concord, New South Wales)
(Ivan Petch family name Petch)
(Ivan Petch given name Ivan (name))
(Ivan Petch nationality Australia)
(Ivan Petch has occupation Politician)
(Ivan Petch has occupation Electrical engineer)
(Ivan Petch alumni of Fort Street High School)	Correct fact is retrieved. However, the model is still not able to predict correct output.
Scientist was born in _____.	Kingston	London	(Scientist (musician) birth date 1960-04-18)
(Thomas Young (scientist) has occupation Physicist)
(I Am a Scientist date published 1994)
(Thomas Prince (scientist) has occupation Physicist)
(Bambi production company The Walt Disney Company)
(Allen Taylor (scientist) nationality United States)
(Lawrence Roberts (scientist) nationality United States)
(David Thomas (Canadian scientist) has occupation Biochemist)	Ambiguous query, leads to poor retrieval results.
Moldova shares border with _____.	Ukraine	Romania	(Moldova shares border with Ukraine)
(Moldova shares border with Romania)
(Moldova shares border with aa)
(Moldova shares border with Jabara)
(Moldova Nouă shares border with Bela Crkva)
(Moldova contains administrative territorial entity Transnistria)
(Moldova diplomatic relation Russia)
(Moldova diplomatic relation European Union)	Multiple answers correct, however, LAMA considers only one.
Table 12: Illustrative examples of cases where the proposed solution produced incorrect completions.