SlowGuess/NAACL_2021_OCR huggingface dataset:
Evaluation (LREC-2018), Miyazaki, Japan. European Languages Resources Association (ELRA).
Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Trans. Assoc. Comput. Linguistics, 8:34-48.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: retrieval-augmented language model pre-training. Computing Research Repository, abs/2002.08909.
Benjamin Heinzerling and Kentaro Inui. 2020. Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries. Computing Research Repository, abs/2008.09036.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. Computing Research Repository, abs/2007.01282.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know. Trans. Assoc. Comput. Linguistics, 8:423-438.
Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual LAMA: Investigating knowledge in multilingual pretrained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3250-3258, Online. Association for Computational Linguistics.
Nora Kassner and Hinrich Schütze. 2020. Bert-knn: Adding a knn search component to pretrained language models for better QA. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020, pages 3424-3430. Association for Computational Linguistics.
Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811-7818, Online. Association for Computational Linguistics.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020. Pre-training via paraphrasing. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020. K-BERT: enabling language representation with knowledge graph. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 2901-2908. AAAI Press.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Fabio Petroni, Patrick S. H. Lewis, Aleksandra Piktus, Tim Rocttäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. How context affects language models' factual predictions. In *Conference on Automated Knowledge Base Construction*, AKBC 2020, Virtual, June 22-24, 2020.
Fabio Petroni, Tim Roktaschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2463-2473. Association for Computational Linguistics.
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020. E-BERT: Efficient-yet-effective entity embeddings for BERT. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 803-818, Online. Association for Computational Linguistics.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 5418-5426. Association for Computational Linguistics.
Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2012, Kyoto, Japan, March 25-30, 2012, pages 5149-5152. IEEE.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2020. K-adapter: Infusing knowledge into pre-trained models with adapters. Computing Research Repository, abs/2002.01808.
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2020. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Yadollah Yaghoobzadeh, Heike Adel, and Hinrich Schütze. 2018. Corpus-level fine-grained entity typing. J. Artif. Intell. Res., 61:835-862.
Donghan Yu, Chenguang Zhu, Yiming Yang, and Michael Zeng. 2020. JAKET: joint pre-training of knowledge graph and language understanding. Computing Research Repository, abs/2010.00796.
Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R. Bowman. 2020. When do you need billions of words of pretraining data? Computing Research Repository, abs/2011.04946.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: enhanced language representation with informative entities. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 1441-1451. Association for Computational Linguistics.
A Resource Consumption
We follow Strubell et al. (2019) for our computation. The measured peak energy consumption of our CPU-server was 618W. Considering the power usage effectiveness the required kWh are given by  $p_t = 1.58 \cdot t \cdot 618 / 1000$ . Training the English fast-Text on Wikipedia took around 5 hours. Training all languages took 20 hours. The estimated CO $_2$ e can then be computed by CO $_2$ e = 0.954  $\cdot$ $p_t$
B Reproducibility Information
For computation we use a CPU server with 96 CPU cores (Intel(R) Xeon(R) Platinum 8160) and 1024GB RAM. For BERT and mBERT inference we use a single GeForce GTX 1080Ti GPU.
Getting the object predictions for BERT and fastText is fast and takes a negligible amount of time. Training fastText embeddings takes between 1 to 5 hours depending on Wikipedia size.
BERT has around 110M parameters, mBERT around 178M. The fastText embeddings have  $\mathcal{O}(nd)$  parameters where  $n$  is the vocabulary size and  $d$  is the embedding dimension. We use  $d = 300$ . Thus, for most vocabulary sizes, fastText has significantly more parameters than the BERT models. But overall they are cheaper to train.
We did not perform any hyperparameter tuning. Table 6 gives an overview on third party software. Table 5 gives an overview on the number of triples in the dataset. Note that no training set is required, as all methods are completely unsupervised.
Table 11 shows randomly sampled triples to perform an error analysis.
Language	#Triples	#Triples UHN
ar	17129	13699
de	29354	23493
en	33981	27060
es	28169	22683
fr	30643	24487
he	14769	12033
ja	22920	17832
ko	14217	11439
th	8327	7065
tr	13993	11274
Table 5: Overview on number of triples.
System	Parameter	Value
fastText	Facebook Research	Version0.9.1
Embedding Dimension	300
BERT	Huggingface Transformer	Version 2.8.0
Tokenizers	Huggingface Tokenizers	Version 0.5.2
Table 6: Overview on third party software.
Model Vocabulary Size p1 p1-mf entropy #pred.
Oracle	22.0	0.0	3.68	1
BERT	30k	39.6	35.7	6.48
mBERT	110k	36.3	32.6	6.41
fastText	BERT-30k	26.9	27.7	7.04
mBERT-110k	27.5	27.6	7.09
30k	16.4	15.9	7.13
120k	34.3	35.4	7.30
250k	37.7	38.9	7.33
500k	39.9	41.2	7.33
1000k	41.2	42.5	7.32
Table 7: Analysis of the diversity of predictions.  $p1 - mf$  is the p1 when excluding triples whose correct answer is the most frequent object. entropy is the entropy of the distribution of predicted objects. #pred. denotes the average number of distinct objects predicted by the model across relations. The average number of unique objects in the candidate set across relations is 151. fastText has more diverse predictions, as the entropy is higher and the set of predicted objects is on average much larger.
In this section we show additional results. Table 8 shows the same as Table 1 but with precision at five. Analogously Table 9. Table 10 shows the same as Table 3 but for LAMA-UHN. The trends and key insights are unchanged. Table 7 analyses the diversity of predictions by the different models.
Model Vocabulary Size p5 LAMA LAMA-UHN
Oracle	48.0	49.7
BERT	30k	64.1	57.9
mBERT	110k	59.7	53.5
fastText	BERT-30k	48.7	41.9
mBERT-110k	48.9	42.0
30k	26.3	16.5
120k	58.3	52.7
250k	62.7	58.1
500k	65.4	61.3
1000k	66.8	63.1
Table 8: Results for BERT, mBERT and fastText. Same as Table 1 but with p5.
Model	Vocab.	p5
Size	AR	DE	ES	FI	HE	JA	KO	TH	TR
Oracle	48.8	48.4	48.6	49.6	50.1	49.0	49.2	51.9	50.3
mBERT	110k	33.8	51.3	53.9	46.2	38.2	36.5	43.0	37.0	55.5
fastText	mBERT-110k	26.0	40.5	42.9	43.8	27.7	24.0	31.9	33.9	50.3
30k	38.5	28.8	29.8	33.9	38.9	26.4	34.1	45.8	42.7
120k	51.6	48.9	55.2	49.7	54.1	44.1	54.8	56.0	60.9
250k	50.0	56.0	59.1	55.4	58.1	49.2	59.2	59.5	63.9
500k	57.0	59.1	61.5	58.0	59.2	50.9	59.7	61.0	64.6
1000k	56.4	60.7	62.2	59.1	58.9	51.7	57.5	57.2	63.7
Table 9: p5 for mBERT and fastText on mLAMA. Numbers across languages are not comparable as the number of triples varies.
Model	Vocab.	p1
Size	AR	DE	ES	FI	HE	JA	KO	TH	TR
Oracle	23.1	23.8	23.2	22.9	24.5	22.5	22.6	25.1	24.6
mBERT	110k	12.1	26.1	27.6	15.8	11.0	11.8	15.1	10.8	27.7
fastText	mBERT-110k	7.8	14.3	16.9	15.0	6.6	6.4	8.0	7.4	19.4
30k	12.4	8.9	9.0	9.4	13.8	7.4	9.4	14.8	14.5
120k	20.2	18.9	23.8	18.1	22.1	15.4	21.0	23.8	26.1
250k	22.7	24.0	27.3	22.6	26.3	18.0	23.8	28.3	28.7
500k	24.2	26.6	30.1	24.3	27.4	20.0	25.0	27.6	29.4
1000k	23.7	27.6	30.1	25.6	27.5	20.4	23.2	27.2	29.8
Table 10: p1 for mBERT and fastText on mLAMA-UHN. Numbers across languages are not comparable as the number of triples varies.
Table 11: We sample two random triples where either BERT or fastText[1000k] is correct per relation. One can see for example that BERT mostly predicts "jazz" for relation P136.
Table 12: Table 11 continued.