Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge in Foundation Models
Authors: Tim Schott
Daniel Ryan Furman
Shreshta Bhat
Venue: EMNLP 2023 Main (2023)
Keywords: NLP
machine learning
LLMs
language modeling
multilingual
datasets
benchmarks
TL;DR: We benchmark foundation model knowledge with a new multilingual test.

Abstract: In this work, we assess the ability of foundation models to recall encyclopedic knowledge across a wide range of linguistic contexts. To support this, we: 1) produce a 20-language dataset that contains 303k factual associations paired with counterfactuals, 2) evaluate 5 models in a multilingual test, and 3) benchmark a diverse set of 24 models in an English-only test. Meta's LLaMA achieves the highest scores in both multilingual and English-only evaluations. Yet, an analysis of LLaMA's errors reveals significant limitations in its ability to recall facts in languages other than English, plus difficulties related to the location and gender of fact subjects. Overall, our findings suggest that today's foundation models are far from polyglots.

Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge in Foundation Models
Tim Schott, Daniel Furman, and Shreshta Bhat"
School of Information
University of California, Berkeley
(timschott, daniel_furman, htrrssshrreeshaaeebeeeeele edu
Abstract
Related Work
In this work, tion models to recall encyclopedic knowledge across wide range linguistic contexts. To support this, 1) produce a 20- 20-language dataset that contains 303k factual associations paired with counterfactuals, evaluate 5 mod- els in multilingual test, and 3) benchmark diverse set of 24 models am English-only test. Meta's LLaMA achieves the highest scores in both multilingual and English-only evaluations. Yet, an analysis of LLaMA' errors reveals sig- nificant limitations in its ability recall facts in languages other than English, plus difficulties related the location and gender of fact sub- jects. Overall, our findings suggest that today's foundation models are far from polyglots.
the ability of founda-
An impressive amount of knowledge is encoded within LLMs (Roberts al., 2020), which store factual associations as key-value pairs within their memory (Geva et al., 2022; Meng et al., 2022b), Expose models to large number of facts during self-supervised training, and they'll II adeptly recall this information at deployment (Kaplan et al., 2020). However, along with useful facts, models can ingest dubious or harmful associations (Bender et al., 2021), particularly training corpora are poorly constructed or unrepresentative of the world (Dodge et al., 2021).
To benchmark how robustly LLMs learn factual associations, Jiang et al. (2020) and Kassner et al. (2021) evaluated the encyclopedic knowledge of models like BERT (Devlin al., 2019) and RoBERTa (Liu al., 2019; Conneau et al., 2020) using rank-based approaches. Our study builds off this research in few ways. We utilize contrastive scoring approach, which tests the extent to which model grasps a concept with more rigor than rank-based methods, as detailed below. Additionally, we inspect diverse group of causal and masked language models rather than testing a single architecture capture more representative view of the field.
1 Introduction
Can foundation models be used as multilingual knowledge bascs? Foundation models typify an emerging paradigm that warrants further study; all-purpose Large Language Models (LLMs) that are trained on internet-scale corpora excel in gen- cralization to some new tasks (Radford et al., 2018; Brown al., 2020; Touvron al., 2023). Their widespread adoption and ostensible credibil- ity come with risks, though. For instance, founda- tion models inherit inaccuracies from training cor- pora (Argyle et al., 2023), which are in turn propa- gated downstream to the models that are fine-tuned from them (Bommasani et al., 2022; Chung et al., 2022). Additionally, foundation models spend the majority of their training phase absorbing infor- mation in English; for example, Touvron et al. (2023)'s LLaMA devotes two-thirds of its training dataset an English-only subset of the Common- Crawl (Wenzek et al., 2020). Thus, foundation models are potentially deficient when performing non-English tasks (Kassner et al., 2021).
Task
We formulate the Polyglot of Not? test with cloze statements: given some context, we prompt an LLM to predict the next token. Factual associa- tions are formalized as the triplet (s, o) where s and o denote the subject and object entity and r is linking relation, in line with Elsahar et al. (2018). Thus, the fact "Paris is the capital of France" is rep- resented by (Paris, capital of. France) where "Paris" corresponds to s, "capital of" corresponds to r, and "France" corresponds to a. We then
All authors contributed equally
'Sppporting code and data are openly released

---

prompt model M using the original natural lan- guage sentence with masked out. To assess if M has correctly encoded an association, we calculate Par(o | 8, r). Prior knowledge assessments em- ployed a rank-based reward (Petroni et al., 2019) where model is thought to understand the associa- tion if has high chance of occurring as the next token (relative to all other options). However, this practice has pitfalls such unsatisfactory outcomes for questions with numer- ous correct answers and lack of insight into the LLM's confidence in its response. To address these issues, our work uses variant of the Contrastive Knowledge Assessment (CKA) from prior work (Dong et al., 2022). Erroneous "counterfactuals" (s,r, 1s o') like (Paris, capital of, Itaaly) are used to assess model M's understanding of (s. o). Simply put, if M truly knows the fact, (o) s, should be larger than (os s,r) (Dong al., 2022). Formally, CKA measures whether M cor- rectly knows a fact (s, r, o) via calculating:
liked to test proprietary LLMs such as GPT-4 (OpenAI, 2023), these models don't currently provide vocabulary token probabilities inference, a prerequisite for CKA (see Assessing Open vs. Proprietary LLMs for details).
4 Dataset
inability to parse
The dataset includes 303k knowledge statements in 20 languages. Each row includes: dataset id (primary key), slem, true, false, relation, subject, and object. In total, the dataset contains 31 unique relation categories, 76,036 unique subjects, 18,837 unique objects, 18,503 unique trues, and 88,224 unique falses. Masked true/false objects consistently appear on the right-hand side of the statement to support masked and causal LLMs. The translated subset for each language contains different amounts of statements due to varying syntactic capacities to support this requirement. On average, a given fact appears in 12 of the 20 languages tested.
Purlo s,T)
CKAM(,,T,O) Pa/Pas s,r)
To construct the dataset, we first merged two English-language datasets from Dong et al. (2022) and Meng et al. (2022a) that share common lineage in the T-REx Wikidata (Elsahar al., 2018) project. We then improved the dataset by filtering out inaccuracies and grammatical errors. as well as de-duplicating the (s, r, o) triplets, as detailed by Dataset Preprocessing in the Appendix. After preprocessing, the dataset contained 26,254 knowledge statements in English. We then used the Google Translate API to translate the data into 19 target languages: bg, ca, es, da, de, en, es, fr, hr, hu, it, nl, pl, pt, ro, ru, sl, sr, sv, and uk (ISO 639-1 codes). Our translation approach mirrors prior multilingual studies such as the programmatic translation of MMLU (Hendrycks et al., 2021) prompts when analyzing GPT-4. Additionally, work from Kassner et al. (2021) shows minimal practical differences when using machine versus manually translated cloze statements.
When r,, 1, the model is said to understand the factual association. This approach alleviates the issues that arise from ranking a model's woce token probabilities at inference; using counterfactuals clicits connections across different languages and contexts which forces the model to demonstrate generalized understanding of given concept. Furthermore, examining the contrast allows us quantify the confidence level with more nuance. Our work builds off Dong et al. (2022) by applying CKA to a multilingual dataset for the first time.
To carry out the test by language, we solicited cloze completions for each of the associations contained in the dataset. The percentage of fact- completions that M recalls correctly is calculated by tallying up the number of completions where CKAM(s, r, o) > and dividing by the total number of completions. We accommodated dif- ferent tokenizers by removing special tokens from text generation and ensuring that the complction probing corresponded to the first token to the right of cloze. Additionally, all evaluated models are fully open-source. While would have
Results
Table displays mean performance across the 20 languages used in the multilingual test. We present results for 5 foundational models here, with LLaMA-33B outperforming the others by wide margin. We display LLaMA-33B's accuracy on
LLLLMA weights were accessed with Meta's permission

---

model
accuracy (%)
amounts of training data (Kaplan et al., 2020). For instance, LLaMA-7B with IT tokens outperforms OPT-30B with 180B tokens (Zhang et al., 2022) on the English-only test (see Table 4). Moreover, the lean I 110M parameter mBERT model (Devlin et al., 2019) outperforms two 7B parameter mod- els on the multilingual test. Lastly, the LLaMA family provides side-by-side comparison the English-only test; the performance differential is largest from the 13B 33B variants, aligning with the IT to 1.4T training token jump (see Table 4).
Ilama-33b
79.31 (+/- 0.74)
m-bert
62.00 (+/- 0.87)
bloom-7bl
57.70 (+/- 0.88)
xim-roberta
56.03 (+/- 0.90)
mt5-xl
52.51 (+/- 0.91)
Random Baseline
50
Table I: Multilingual test leaderboard. Here, accu- racy refers to the average performance of each model across 20 languages. The uncertainty estimates are av- eraged 95% confidence intervals computed from 10k bootstrap iterations per language. The results suggest tested models struggle to recall facts in a multilingual setting relative to English-only performance (Table 4).
Subject Entity Error Analysis
We analyzed LLaMA-33BB errors across each of the 20 languages tested and found systemic gaps in its factual recall. We began by explor- ing associations from our dataset that feature geographic locations as their subject entity. The 3,213 geographic entities we worked with appear in 48,606 prompts in our 20 language assessment (see Geographic Labeling in the Appendix for details). LLaMA-33B answered these types of questions correctly at an 89.94% clip. The top performing continent was Asia with 93.31% accuracy for 10,729 questions, and the lowest was Antarctica with 80.65% accuracy for 5,167 questions. A chi-squared test for independence ge-
model
accuracy (%)
llama-65b
89.56 (+/- 0.37)
Ilama-330
89.40 (+/- 0.38)
falcon-40b
87.01 (+/- 0.41)
llama-131 13b
86.66 (+/- 0..2)
Ilama-7b
85.53 (+/- 0.43)
redpajama-7h
85.07 (+/- 0.44)
Random Baseline
50
Table English-only test leaderboard, top models. Here, accuracy refers to model performance on English data. The uncertainty estimates are 95% confidence intervals computed from 10k bootstrap iterations. Con- sistent with the trends in Table I, LLaMAs of varying sizes emerge as the front-runners. Reference Table in the Appendix for the full leaderboard.
comparing LLaMA-33B's performance ographic questions related Asian locations versus European locations confirms the superior performance on Asian locations is significant (x2 66.408, p 0.001).
each of the 20 languages individually in Table 3 and Figure 1. This model scores higher on languages written in Latin script than those written in Cyrillic script (bg, ru, sr, uk). A chi-squared test confirms that LLaMA-33B' performance is dependent on language script (x2 3570.58. p < 0.001). Ad- ditionally, the results on the English-only test are displayed for two dozen models in Table and 4. LLaMA models again top the leaderboard here, closely followed by Technology Innovation Insti- tute's Falcon-40B (Penedo et al., 2023).
We also explored whether LLaMA-33B's er- rors were systematically related to the gender (male/female) of fact's subject. The 951 entities sampled appear in 16,003 prompts in the test (see Gender Labeling in the Appendix for details). LLaMA answered 75.87% of these questions correctly. Male subjects are nearly 5 times as common as female subjects the sample, yet the model performs slightly worse on facts about male subjects. A chi-squared test for independence comparing LLaMA's performance on questions about male subjects compared to female subjects confirms its superior performance facts about females is significant (x2 69.096, p < 0.001).
6 Analysis
Training Data and Model Parameters
LLaMA excels in our tests relative other foun- dation models. This challenges some previous no- tions that compute should be spent to support enor- mous (parameter-wisg wise) models lieu of larger
WWe analyzed LLaMA-33B because it both performs well on the multilingual test and boasts a parameter count suitable for interrogations on lightweight compute resources

---

Wikipedia's Role in LLaMA Performance
side of English in LLaMA's results for prompts entailing English speaking countries, with Slavic languages exhibiting more significant deviations than others. Cross-lingual transfer of knowledge thus exhibits a lack of reliability.
LLaMA learns information by reading Wikipedia puges, so we studied data quality on each lan- guage's Wikipedia. We began by tabulating how many pages were present during LLaMA's training period (see Table 5). Of course, sheer page count is perhaps not the strongest indicator of the qual- ity and diversity of information available on that language's Wikipedia; a single well-written page can be more informative than dozen low-quality pages. To delve deeper, we analyzed Wikipedia pages from languages of interest (see Wikipedia Entity Analysis the Appendix for details). Ta- ble 6 records word count, the number of named- entities that appear in the article (both total and unique), and the number of named subject entities in the dataset that appear in the article which we refer as "target" entities (both total and unique). We adopt an approach such that a page that men- tions 8 different target entities is considered to be denser and thus more informative than an ar- ticle that narrowly focuses on single target entity, Analysis of the articles we sampled reveals major gaps across each language's Wikipedia. We ob- serve strong and significant correlation (Pearson's 's 0.78, p 0.001) between the average unique target entities the page and LLaMA's perfor- mance; the more subjects on a Wikipedia page, the better LLaMA recalled facts in that language. This underlines the connection between dataset quality and performance on our assessment.
7 Future Work
There are many directions left pursue in this domain. Model weight editing in multilingual setting presents a novel next step for our project since our data finds its roots in two projects (Dong et al., 2022; Meng et al., 2022a) that explore how to remedy inaccuracies located in LLMs. Also, ap- plying the test to future open-source models will fortify this work's 's impact and relevance for future researchers (see Testing New Models in the Ap- pendix for details). We can also add languages that use neither Cyrillic nor Latin scripts; we are working with native Hindi and Japanese speakers to create cloze statements in these languages. There is also work to be done regarding the variable dif- ficulty of given fact based on the availability of training data in that language: the values from our Wikipedia analysis could be used prior proba- bilities in a future iteration of CKA. Additionally, we could analyze more facets of training corpora metadata. Perhaps it's possible to causally connect a model erring on a particular fact to artifacts in its training data rather than the measured, associative approach we adopt. Current work (Elazar et al., 2023) affords helpful scaffolding for this endeavor.
8 Conclusion
Qualitative Insights
Here, we present multilingual contrastive knowl- edge assessment of encyclopedic facts. Our origi- nal evaluation benchmarks 5 foundation models in multilingual test and two dozen in an English- only test. Meta's LLaMA demonstrated supe- rior performance in both settings. Accompany- ing analyses reveal that LLaMA struggles oper- ate in non- non-English languages, partieularly in Cyril- lic script, suggesting an absence of robust cross- lingual knowledge transfer. These findings vouch for the utility of high-quality, multilingual datasets for training the next-generation of foundation mod- els. Our hope is that this project motivates future interrogations of foundation model data sources and provides a roadmap for others to conduct trans- parent evaluations. By doing so, LLMs can be bet- ter equipped for broad application across diverse linguistic contexts.
Qualitative analyses underscore the influence of frequency bias. For instance, LLaMA frequently erred when prompted with statements containing "Antarctica" in variety of languages. In the En- glish language prompt "Cape Monaco is a part of the continent of"" LLaMA ranked "Europe" to be a more likely completion than the correct "Antare- tica." Cape Monaco's s Wikipedia page makes nu- merous references to European people and places (including its appellation), and LLaMA appears to prioritize the presence of a European entity rather than connect this location's 's correct continent. Not all signals in its training dataset, then, appear to be treated with equal diligence. What's more, when conducting pairwise comparisons between English and other languages for common facts, relative rankings remain largely consistent with overall per- formance. We observe degraded performance out-

---

Limitations
Ethics Statement
Assessing Open vs. Proprietary LLMs
Although we test a language model's ability to serve as multilingual knowledge bases, we do not find these models to be particularly reliable sourees of knowledge; none of the models scored above 90% for any of the languages that tested. We thus caution readers that LLMMs should not be used as an authoritative source of facts whether in a research setting such as this or in a real-world envi- ronment. The test sheds light on the types of lan- guages, topics, and contexts where LLMs are more likely to produce factual errors, but the same meth- ods might also enable a malicious actor to check whether particular set of facts is committed to model memory and subsequently insert damaging information into model that was not originally present in the training data with other methods, such as the MEMIT algorithm proposed by Meng et al. (2022b). Lastly, while our work points to the need for testing low-resource languages, the test at present is restricted to a relatively small number of languages (20), most of which are high-resource. We intentionally use the 20 languages included in the LLaMA training dataset in this work. However, future work must further explore fact-completion testing for low-resource languages and devote tention to a larger number of languages.
One prerequisite for carrying out the test is access the full schedule of vocabulary-level token score probabilities generated when an LLM synthesizes text. For this reason, researchers related inquiries typically work with fully open- sourcs models with weights uploaded to the Hugging Face model hub (Jiang et al., 2020; Dong et al., 2022; Meng et al., 2022a). Proprietary models, meanwhile, lack this transparency rendering their generated texts resis- tant analysis. Notably, OpenAT's GPT-3 API only surfaces the probabilities of the 5 most likely next tokens, functionality which Hendrycks et al. (2021) leveraged to apply GPT-3 to their evaluation task. We submitted a request for this limit to be raised through OpenAl's official channel - a fully automated, chat-bot customer service agent - and we have yet to receive response. What's 's more, the GPT-4 API nixed the reporting of token probabili- ties entirely (as of this writing), thwarting an impor- tant avenue for research into their newest founda- tion model and adding an additional layer of opac- ity into how their systems produce results (OpenAl, 2023). Likewise, as things stands today, the largest (parameter-wise) foundation models from other re- search consortiums such as DeepMind's Gopher (Rae et al., 2022), Google's LaMDA (Thoppilan al., 2022), and Huawei's PanGu-Sigma (Ren al., 2023) are all proprietary.
Acknowledgements
We thank Professor David Bamman for help- ful feedback and constructive suggestions. This project received funding from the School of Infor- mation at the University of California, Berkeley.
GPU Resources
We performed experiments on range of LLM fam- ilies and sizes. This required many hundreds of hours of GPU usage (see Reproducibility in the Appendix for details). In total, we batched over 100 model runs that required approximately 500 hours of GPU usage. For instance, testing LLaMA- 7B's performance the 22.974 Portuguese factual associations in the dataset required 2.5 hours of GPU usage with Ix T4. addition to having to schedule long-periods of compute uptime, we were also constrained by fixed resource requirements, using workstations with a single NVIDIA GPU. Thus, we could not evaluate the gamut of truly mas- sive (parameter-wise) models in experiments. Going forward, need to be made for groups effectively exper- iment with LLMss, in particular as organizations release models that require extremely demanding compute requirements to host and run.
believe more accommodations

---

References
Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems. volume 33, page 1877-1901.
Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language mod- els to simulate human samples. Political Analysis, (3)33337-331.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay. William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe Le, and Jason Wei. 2022. Scaling caarucctionfinnetuuee language models. (arXiv:2210) 1416).
Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Confer- ence on Fairness, Accountability, and Transparency. FAceT "21, pages 610-623, New York, NY, USA. Association for Computing Machinery.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson,, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis Dora Demszky. Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy. Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Kr- ishna, Rohith Kuditipudi, Ananya Kumar, Faisal Lad- hak, Mina Lee, Tony Lee,, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Mamning, Suvir Mirchan- dani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles. Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Pa- padimitriou, Joon Sung Park, Chris Piech, Eva Porte- lance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan. Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang. William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang. Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2022. On the opportunities and risks of foundation models. (arXiv:2108.072) 002558).
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, page Online. Association for Computational Linguistics.
Tim Dettmers. Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3.int8() 8-bit matrix multi- plication for transformers at scale. In Advances in Neural Information Processing Systems, volume 35, pages 30318 30332..
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, Volume I (Long and Short Pupers), page 4171 1776, Minneapolis, Minnesota. Association for Computational Linguistics.
Jesse Dodge, Maarten Sap. Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, page 12861305, Online and Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. 2022. Calibrating factual knowledge in pretrained language models. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 5937-5947, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan. Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan. Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach. Yonatan Belinkov, Hinrich Schütze, and Yoav Gold- berg. 2023. Mcasuring causal effects of data statistics on language model's 'factual' predictions. (arXiv:2207.1422)

---

Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Lafor- est, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference Language Resources and Evaluation (IREEC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon Il:: Outperforning curated corpora with web data, and web data only. (arXiv:2306.011 16).
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNL"), pages 2463-2473, Hong Kong, China, Association for Computational Linguistics.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Gold- berg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 30-45, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Dan Hendrycks, Collin Burns, Steven Busart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. In International Conference on Learning Representations, Online and Austria.
Alee Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language models are unsupervised multitask learners.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song. John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, Eliza Rutherford, Tom Hennigan, Ja- cob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Mari- beth Rauh, Po-Sen Huang, Amelia Glaese, Jo- hannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Anto- nia Creswell, Nat McAleese, Amy Wu, Frich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Bud- den, Esme Sutherland, Karen Simonyan, Michela Pa- ganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou. Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsim- poukelli. Nikolai Grigorev, Doug Fritz, Thibault Sot- tiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy. Chris Jones, James Bradbury, Matthew Johnson. Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Ko- ray Kavukcuoglu, and Geoffrey Irving. 2022. Scaling language models: Methods, analysis & insights from training gopher. (arXiv:21 12. 1446).
Zhengbao Jiang, Antonios Anastasopoulos. Jun Araki, Haibo Ding, and Graham Neubig. 2020. X-factr: Multilingual factual knowledge retrieval from pre- trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 5943-5959, On- line. Association for Computational Linguistics.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. (arXiv:2001.0836)
Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual lama: Investigating knowledge in multilingual pretrained language models. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 3250-3258, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Xiaozhe Ren, Pingyi Zhou, Xinfan Meng, Xinjing Huang, Yadao Wang. Weichao Wang. Pengfei Li, Xiaoda Zhang, Alexander Podolskiy, Grigory Arshi- nov, Andrey Bout, Irina Piontkovskaya, Jiansheng Wei, Xin Jiang, Teng Su, Qun Liu, and Jun Yao. 2023. Pangu-E: Towards trillion parameter lan- guage model with sparse heterogencous computing. (arXiv:2303.1) 10015)
Kevin Meng, David Bau, and Alex Andonian. 2022a. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems. volume 35.
Kevin Meng. Arnab Sen Sharma, Alex An- donian, Yonatan Belinkov, and David Bau. 2022b. Mass-editing memory in transformer. (arXIv:2210.07229.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural
OpenAI. 2023. (drXiiv2303.08877))
Gpt-4 technical report.

---

Language Processing (EMNLP), pages 5418-5426
Online. Association for Computational Linguistics.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin. James Qin. Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung- Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina. Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022. Lamda: Language models for dialog appli- cations. (arXiv:2201.082) .08239).
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anna Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. MMXiv:2302.139711
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco Guzmán, Ar- mand Joulin, and Edouard Grave. 2020. Cenet: Ex- tracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4003- 4012, Marseille, France. European Language Re- sources Association.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shus- ter. Daniel Simig, Punit Singh Koura, Anjali Srid- har, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language mod- cls. (10X1v:2005.01068)

---

Appendix
Removed 418 stem/fact pairs that were rela- tion P140 (religion) filter sensitive topics
Reproducibility
Removed 490 stem/fact pairs that were rela- tion P530 (diplomatic ties) due consistent inaccuracies
Supporting code and data are openly released on GitHub and Hugging Face, respectively. Text gen- eration conducted with the transformers" and bi bitsandbytes packages (see Text Genera- tion Configuration below for details). Subsequent steps of the Polyglot or Not? test were executed with the pytorch' package. In regards to com- pute resources, the experiments were performed on workstations equipped with various Nvidia GPUs. We employed 1x H100 (80 GB PCIe) for larger models (e.g., LLaMA-65B), 1x A100 (40 GB SXM4) for medium-sized models (e.g., LLaMA- 33B), and 1x T4 (15 GB) for smaller models (c.g.. LLaMA-13b/7b).
Removed 1,427 stem/fact pairs that were rela- tion P27 (citizen of) due consistent inaccu- racies
Removed 576 stem/fact pairs that were rela- tion P463 (affiliated with) due to consistent inaccuracies
Removed 39 stem/fact pairs that compared football with soccer due to cultural differences these word meanings
Removed 131 stem/fact pairs with "expired at" wording due to awkward phrasing
Dataset Preprocessing
An depth data preprocessing pipeline was ap- plied to the dataset to improve its quality. The Calinet (Dong et al., 2022) dataset originally con- tained 50,451 stem/fact items which we consider "valid" cloze statements, items where the masked object appears on the right- side of the stem. Many of these stem/fact pairings were paraphrased, though, to support their model rewrite process, which this paper does not explore. After removing these paraphrased (s, r, o) triplet duplicates, we were left with 11,960 statements from this data pool. Meanwhile, the ROME (Meng al., 2022a) dataset contributed 21,919 valid stem/fact pairs, all of which were unique (s, rr o) triplets. We merged the data and were left with 33,870 items. From there, we performed the following enhancements:
Removed 50 stem/fact pairs with "-language" wording due to awkward phrasing
Removed 73 stem/fact pairs with facts/coun- terfacts starting with "the" due to the fre- quency of the word "the" in training datasets
Removed 125 stem/fact pair duplicates retain a dataset of entirely unique (s,r,0) triplets
Our straightforward improvements provide more validity to our pool of data and its ultimate use the Polyglot or Not? test, such as removing the over 3,000 statements whose correct answer can be found in the unmasked portion (bullet number 3). See Table 7 for handful of examples filtered out during the above operations. After preprocessing, are left with 26,254 unique rows in the final English-only subset of dataset.
Removed 227 stem/fact pairs that were manu- ally flagged errors
Removed 371 stem/fact pairs with "a/an due to consistent grammatical errors
Geographic Labeling
We sought a labeled dataset of geographic entities connected to the continents they're 're located on. To do so, we filtered our original dataset down to the Wikidata relation IDs that most clearly signal that a geographic entity, such as Paris or France, occupies the subject of the stem: capital (relation P17 + P1376), continent (P30), country (P36), shares border with (P47), and is in the territory of (P131). Then, we extracted the unique, English translations of the subjects from this data, leaving us with 3,427 "geographic"
Removed 3,088 stem/fact pairs where the cor- rect fact is explicitly stated in the stem, ren- dering the completion trivial
Removed 610 stem/fact pairs that were rela- tion P190 (sister city) due to consistent inac- curacies
hhtpss//ggtbbbboooongggiig////as "nanps//githuub.com//pproooaaopp

---

entities in our dataset. To more quickly move into substantive analysis, we utilized Generative Al assistant, ChatGPT (gpt-3.5-turbo accessed April, 2023) 8 to label these entities by geographic continent. Our prompt (see below) offered an option for an "unsure" label if the assistant did not know the correct answer, the location stretched across multiple continents, etc. Of the 3,427 requested prompts for, the assistant labeled 3,213 with a tag for one of the world's continents. To verify the veracity of the labels we randomly sampled 10% of the labeled data and found that the affixed continent labels were correctly applied to every entity in the validation sample. The resulting labeled data provided interesting terrain for mining insights, detailed in the Sub- ject Entity Error Analysis subsection. Prompt used:
these identities by gender. However, because there are far more people entities in our dataset after filtering for these relation IDs (7,905 total) we randomly sampled a portion of them, extracting 1,200 unique entities to hand off to ChatGPT. Our prompt (see below) for gender also offered an option for an "unsure" label if the assistant did not know the correct answer, the entity wasn't name, etc. Of the 1,200 requested prompts for, the assistant labeled 1,057 with gender tag. To verify the veracity of the labels we randomly sampled 10% of the labeled data and found that the affixed gender labels were correctly applied to every entity in the validation sample. The resulting labeled data is also explored in the Sub- ject Entity Error Analysis subsection. Prompt used:
have list of names. Can you return the gender (male, female, on other) in the following format:
I have list of locations. Can you return the continent on which they are located in the following format:
Sundar Pichai MMae Brigitte Fontaine Female
Iran|AS Bavaria EEU Pennsylvania|MM
If there are items in the list that don't t seem like names perhaps very difficult to classify, you can write "unsure" beside those, e.g.
If there are items in the list that don't seem like locations or perhaps are very difficult to classify you can write "unsure" beside those, e.g.
JTJU-FM] unsure Wagnerian| unsure
WTJU-FM]unsure Ottoman Empire unsure
Wikipedia Entity Analysis
To produce Table 6, we began by randomly sam- pling 10k pages from every language of interest's Wikipedia via Wikipedia's REST API. We did this because we wanted to gather sample of the text in the article body for each language. From there, we extracted the body content of these articles and per- formed minimal preprocessing such as removing ci- tations and navigation headers. With the clean page content in hand, we then used a named-entity recog- nition utility from SpaCy. SpaCy provides models for 14 of the 20 languages LLaMA was tested on. For each of these languages, the core_news_l_ tagger was used save for English where we used the wore_web_lg tagger. We then tallied counts for the entities found in each page. We tracked the overall and distinct number of entities found in
Gender Labeling
We also desired a labeled dataset of person entities connected to their assigned birth gender, as understood the popular consciousness. To do so, we filtered our original dataset down to the Wikidata relation IDs that most clearly signal that person entity, such Steve Jobs or Marie Curie, occupies the subject of the stem: place of death (relation P20), position held (P39), field of work (P101), native language (P103), occupation (P106, employer (P108), position played on team (P413), sport (P641), work location (P937), and instrument (P1303). We followed near-identical procedure for Gender Labeling as we did for Geographic Labeling, using ChatGPT to label
"haps://chat.ppenai.com
hhtps:://ppcc.io

---

each article. We also stored the overall and distinct subset of entities that are found in each article and who appear our dataset.
Text Generation Configuration
All tests were conducted with the same text gen- eration hyper-pane hyper-parameters, by and large employing the default configuration from the transformers package. The principle deviation from the de- fault settings in our tests was the use of mixed- precision quantization; explored the impact of adjusting matrix multiplication precision on a given model's test performance to confirm the efficacy of this method. Specifically, we ran LLaMA-7B and LLaMA-13B on the English-only subset of the dataset under both f'16 and 8-bil configura- tions. In the case of fp16 precision, all values were simply assigned the floch. data type. For 8-bit precision, algorithm from the bilsandbyles package as pre- sented by Dettmers al. (2022), which utilizes the torch.int8 data type for the majority of the values and the torch.flout liers. We found minute but noticeable differences in model performance between the two precision levels (0.35-0.47%% see Table 8). The savings in GPU memory consumption, however, were much more significant by comparison. By opting for 8- bit over fp16 precision, reduce the memory footprint of the two models roughly in half. Based on these results, we determined that the trade-offs between performance and memory footprint were acceptable for our test, as we were running tests on relatively lightweight compute resources. We thus elected to employ 8-bil precision throughout the experiments.
adopted the sixed-precision
data type for out-
Testing New Models
The results included herein exclusively feature foundation models released before June 2023, We have continued to test new LLM releases since then, including Meta's Llama-2 model family, Mis- tral.ai's 's Mistral-7B, and TII's Falcon- 180B. reg- ularly updated leaderboard is maintained at the project repo, with the hopes that the Polyglot or Not? test retains its relevance and impact text- based foundation models proliferate." 10

---

language
accuracy (%)
pairs
English
89.40 (+/- 0.38)
26,254
German
85.74 (+/- 0.53)
16,287
Dutch
85.35 (+/- 0.46)
25,590
Italian
84.39 (+/- 0.49)
20,448
French
84.18 (+/- 0.52)
18,395
Swedish
84.06 (+/- 0.48)
21,576
Catalan
84.01 (+/- 0.52)
18,898
Portuguese
83.81 (+/- 0.48)
22,974
Romanian
82.72 (+/-
17,568
Danish
81.79 (+/-0.50)
23,365
Spanish
81.74 (+/-0.57)
18,786
Czech
77.94 (+/4
9,427
Polish
77.50 (+/-0.84)
9,484
Croatian
76.69 (+/-0.97)
7,358
Slovenian
75.99 (+/5).95)
7,873
Hungarian
75.74 (+/- 1.24)
4,650
Ukrainian
73.00 (+/-0.98)
7,918
Bulgarian
72.50 (+/-).61)
20,577
Russian
69.72 (+/- 1.57)
3,289
Serbian
60.01 (+/- 1.30)
5,426
Random Baseline
50
Table 3: LLaMA-33B's performance across languages. Here, accuracy denotes the LLaMA-33B model's performance assessed individually for each language, while pairs refers to the number of stem/fact items evaluated per language. LLaMA-33B demonstrates higher proficiency with languages utilizing the Latin script as compared to those using the Cyrillic script (Ukrainian, Bulgarian, Russian, and Serbian). A chi-squared test substantiates significant dependency the model's 's test performance on the language script (x2 = 3570.58, p < 0.001). For a graphical representation of these results, refer Figure below.

---

model
accuracy (%)
params
n tokens
89.56 (+/- 0.37)
65.2B
1.4T
Ilama-33b
89.40 (+/- 0.38)
32.5B
1.4T
falcon-40b
87.01 (+/- 0.41)
40B
IT
llama- 13b
86.66 (+/- 0.42)
12.5B
IT
Ilama-7b
85.53 (+/- 0.43)
6.7B
IT
redpajama-77
85.07 (+/- 0.44)
7B
800B
mpt-7b
83.39 (+/- 0.46)
7B
IT
opt-30b
82.09 (+/-0.47)
30B
180B
redpajama-3b
82.09 (+/-0.47)
3B
800B
opt-13b
81.94 (+/-0.46)
13B
180B
gpt-neox-20b
81.50 (+/-0.47)
20B
420B
falcon-7b
81.34 (+/-
7B
1.5T
gpt-j-6b
81.14 (+/-0.47)
6B
420B
pythia-12b
80.53 (+/-
12B
420B
(5-v1-xxl
76.55 0+/-0.52)
13B
34B
bloom-7b1
76.16 0.51)
7.1B
341B
gpt2-xl
73.76 (+/-0.54)
1.5B
bert
72.60 (+/40.54)
110M
m-bert
71.80 (+/-0.55)
110M
stablelm-7b
68.85 (+55).55)
7B
1.5T
distilgpt2
64.23 (+/-)
82M
mt5-xxl
61.58 (+/-0.59)
13B
xlm-roberta
61.55 (+/- 0.59)
355M
295B
mt5-xl
59.96 (+/- 0.59)
3.7B
Random Baseline
50
Table 4: English-only test leaderboard. Here, accuracy refers to model performance on English data. The uncertainty estimates are 95% confidence intervals computed from 10k bootstrap iterations. Params and tokens record each model's number of parameters and number of dataset tokens, respectively (when such data is available). Consistent with the trends Table 1, LLaMAs of varying sizes emerge as the front-runners.

---

language
article count
English
6,513,291
German
2,698,267
Swedish
2,551,218
French
2,430,636
Dutch
2,092,862
Russian
1,828,011
Spanish
1,782.912
Italian
1,758,843
Polish
1,525,414
Ukrainian
1,160,183
Portuguese
1,093.217
Catalan
702,281
Serbian
659,580
Hungarian
505,754
Czech
505,105
Romanian
431,067
Bulgarian 282,130
Danish 280,923
Croatian 212,088
Slovenian 176,565
Table 5: Wikipedia page counts. The number of articles available on Wikipedia during LLaMA's training time period of June 2022, reflected by the article count for each language surfaced archive.org (arranged descending by article count). Even a high-resource language like Romanian possesses a rather small Wikipedia in comparison to other languages like French, (The corresponding archive. org URLs, which link the initial archived copy of the language's homepage or close possible to June 15th, 2022 can be found in our codebase.)

---

language
words
entities
unique entities
targets
unique targets
accuracy (%)
Catalan
384.43
26.51
19.54
3.82
2.31
84.01
Croatian
273.29
24.98
19.68
0.87
0.60
76.69
Danish
255.77
22.71
16.90
3.30
2.07
81.79
Dutch
156.54
25.25
19.78
1.85
1.28
85.35
English
463.15
70.12
50.19
6.86
3.68
89.40
French
491.93
36.82
26.27
4.49
2.58
84.18
German
418.59
40.95
30.46
3.90
2.40
85.74
Italian
391.67
31.09
21.52
3.96
2.24
84.39
Polish
214.29
32.70
25.74
0.90
0.60
77.50
Portuguese
320.44
24.95
17.86
3.60
2.18
83.81
Romanian
231.10
42.72
33.68
2.39
1.62
82.72
Russian
382.84
35.70
26.35
0.86
0.58
69.72
Spanish 470.56
33.04
23.80
4.25
2.41
81.74
Swedish 95.03
9.37
7.42
1.30
0.95
84.06
Ukrainian 283.64
25.32
19.90
1.21
0.80
73.00
Table 6: Wikipedia content analysis. Results of performing named-entity recognition on random sample of 10k Wikipedia articles across 15 languages (arranged alphabetically by language name). Reported metrics correspond per-page averages: words is the article word count reported by SpaCy's language specific tokenizer. Entities and unique entities represent the total and distinct entity counts, respectively, from SpaCy' 's named -enttyy recognition tagger the page text while the targets and unique targets columns correspond the counts of entities that occupy the subject position of stems in our dataset. LLaMA's test accuracy for each language occupies the right- most column, as is also displayed Table and Figure 1. We find LLaMA's performance significantly correlated with the number of unique target entities found in sampled pages (Pearson's 's 0.78, p 0.001). Other takeaways include the rather low average word count of articles on Swedish language Wikipedia due to its high proportion of machine generated pages.
stem
true
false(s)
notes
"Porsche Panamera developed by"
"Porsche"
"BMW"
Answer in stem
"Vincent van Gogh took up work in"
"The Hague" "Islam"
("Belfast", "Worpswede"]
"The" in true
"Muhammad is follower of""
"Buddhism"
Religion relation
Table Examples of data filtered out by preprocessing. Here, we show a small sample items that were filtered out by the preprocessing pipeline, with steps detailed in Dataset Preprocessing above. The first and third items originate from Meng et al. (2022a) while the second item originates from Dong et al. (2022).
model
precision
accuracy (%)
memory footprint (GB)
laama-13
fp16
87.01 (+/- 0.40)
26.2
1aama-133 13b
8-bit
86.66 (+/- 0.42)
14.5
llama-7b
fp16
86.00 (+/- 0.42)
14.7
Ilama-7b
8-bit
85.53 (+/- 0.43)
8.3
Table 8: Quantization experiments for LLaMA-7B and LLaMA-13B. Here, accuracy denotes the model's
performance on English-only data. A small dip in accuracy (0.35-0.47%) is observed between fp16 and 8-bil precisions.

---

Random Baseline
50%
English
89.4%
German
85.
Dutch
85.4%
Italian
84.4%
French
84.2%
Swedish
84.1%
Catalan
84.0%
Portuguese
83.8%
Romanian
82.7%
Danish
81.8%
Spanish
81.7%
Czech
77.9%
Polish
Croatian
76.7%
Slovenian
75.0%
Hungarian
75.7%
Ukrainian
73.0%
Bulgarian
72.5%
Russian
69.7%
Random
Serbian
60.0%
LLaMA-33B
10
20
30
40
50
60
70
80
90
Accuracy (%)
Figure 1: LLaMA-33B's performance across languages, visualized. The model (blue) scores higher on languages written Latin script than those written Cyrillie script (Ukrainian, Bulgarian, Russian and Serbian). chi- squared test confirms that the model's test performance dependent language script (x² 3570.58, p < 0.001), For a tabular representation of these results, refer to Table above.


Reviews:
[{"review_id": "yYC8nZPuxQ", "reviewer": "Reviewer_JE4L", "summary": "", "questions": "* Quite a few stem/facts pairs are removed from the merged dataset. The reasons are stated in the appendix. How were these assessments (e.g., \"consistent inaccuracies\") made? Manually? On a sample? The article would benefit from including some pairs for some of these categories.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments! We appreciate the time you spent with our project, where we focus on evaluating how well foundation models retrieve encyclopedic knowledge in a multilingual setting. We believe our original “Polyglot or Not” assessment and numerous supporting analyses enhance the research community’s understanding of how LLMs retrieve factual knowledge and even possess insights generalizable to the broader public. \n\nWe’d like to highlight that the particular class of multilingual models we tested are referred to in the literature as ‘foundation models’ (see Section 1, Introduction). This is an important group of models which are typically used as base models for fine-tuning and therefore carry large downstream impacts. The rationale behind selecting these 5 models is important, and relates to your note about **existing models**. These are models that are widely used in research and industry settings and thus warrant close inspection: What do they do well? What are their drawbacks? And so on. Testing 5 multilingual foundation models across 20 different languages creates the conditions for fascinating comparisons.\n\nHelpfully, this core focus of our project ties in nicely with the EMNLP ‘23 theme-track: *While the new generation of Large Language Models such as GPTX, LLAMA, BLOOM etc. claim to perform at unprecedented levels for generation and understanding, we are in unexplored territory on many aspects of such LLMs.* This paper represents one such “exploration” into that territory and provides a level-headed view of LLM performance via a novel benchmark. Along the way, we broaden our focus to other areas of interest relevant to the NLP community like equity and inclusion – as you note in your review, *compare performance against number of parameters, for geographical entities/subjects, for persons w.r.t. gender, against entity density within Wikipedia articles, which serves as training data for LLaMA.* \n\nTo continue, we’d also like to respond to your mention of **existing methods/scores**. Our project leverages a scoring heuristic from prior research in truly new ways. We present the first multilingual, contrastive knowledge assessment for LLMs (see Section 3, Task) applied via a multilingual cloze completion task. Understanding the significance of both prongs of this assessment – multilingual and contrastive – is critical for interpreting our results and understanding their broader significance. Many knowledge retrieval projects only work with information in English. Given that over 75% of the world’s population doesn’t speak or write in English, and these models are ostensibly multilingual, we found it pertinent to expand our knowledge assessment beyond English language factual associations. The contrastive portion is also important to underline. Many assessments simply use a rank-based approach to score LLM factual retrieval. We detail the downsides to this in our paper (see Section 3, Task) such as a bias towards tokens that appear frequently in the dataset. Our decision to use an existing method was deliberate, aimed at focusing on a multilingual assessment where the existing method already provided a robust foundation. The synergistic use of these two elements allows for new insights that would not have been possible using either alone.  **Your note motivates us to clarify why our work is differentiated from prior work as part of Section 3.**\n\nYour point about using **existing datasets** warrants a small follow-up. As noted in your summary, we ultimately incorporate those into a new *dataset with 303,000 <s,r,o> facts in 20 languages.* We amalgamated two English-only data pools into a well-curated collection of data that spans 20 different languages. We removed inaccurate/ambiguous/biased facts, and maintained consistency across translations. In doing so we hope to support future researchers. One of the ways we make good on this is by making our dataset available in multiple formats, including on Hugging Face. You can view, search and then download our benchmark data right from your browser. We believe this empowers future researchers and members of the general public alike to easily carry out follow-up projects leveraging our pool of data. The interest in the dataset we’ve put together has been resoundingly positive. In the month of May, for instance, it received over 500 downloads! \n\nWith respect to the “consistent inaccuracies” that we mention in our Appendix, a bit of extra explanation is needed here (and can be added to our paper in the Dataset Processing portion of the Appendix). After combining the datasets, we examined stem/fact pairs in the dataset to ensure that we were feeding in “correct” factual associations to our tested models.This is work we felt important to carry out since, after all, the source datasets were programmatically scraped off of Wikidata. We randomly sampled a 25 rows subset of each stem/fact pair from each relation in the 33,870 row unfiltered dataset. We found that the stem/fact pairs linked, for example, via “citizen of”’ (relation P27) and “sister city”’ (relation P190) presented consistent inaccuracies when we performed manual verifications. We comprehensively removed these stem/fact pairs by filtering out dataset rows that are linked via that ‘relation’ (one of Wikidata’s metadata fields). **We can certainly include examples of stem/fact pairs we removed in the paper to illustrate why we believe their inclusion would detract from our benchmark’s viability and relevance.** This information, along with the specific number of rows removed that we already report in the Appendix, should make it eminently clear to the reader the exact process that we followed. \n\n**Lastly, thank you for pointing out the quick change we can make to large numbers (adding commas) to make them more legible. We will add this to our paper.**\n\nWe hope that this rebuttal clarified some of the strengths of our project and demonstrated our intention to mend any potential shortcomings therein.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Use of contrastive learning and previously-proposed contrastive knowledge assessment  score where previous approaches used rank-based methods\n* Evaluation of 24 models (EN only) and 5 models (multilingual)\n* Development of a dataset with 303,000 <s,r,o> facts in 20 languages\n* Analysis of relationship between LLaMA performance and counts of words/entities identified in Wikipedia articles\n* Code and data released to the public", "reasons_to_reject": "* The effort detailed here makes use of existing datasets, previously-proposed methods/scores, and existing models.", "questions_to_authors": "* Quite a few stem/facts pairs are removed from the merged dataset. The reasons are stated in the appendix. How were these assessments (e.g., \"consistent inaccuracies\") made? Manually? On a sample? The article would benefit from including some pairs for some of these categories.", "paper_topic_and_main_contributions": "In this short paper, the authors measure the performance of causal and masked language models when it comes to retrieving encyclopedic knowledge encoded as factual triples <s,r,o>. The analysis is done for English-only as well as multilingual settings (EN data is automatically translated to 19 other languages) with different sets of models. Results show that LLaMA is consistently more accurate when compared to other models, with EN results being consistently better. Authors perform several analyses to compare performance against number of parameters, for geographical entities/subjects, for persons w.r.t. gender, against entity density within Wikipedia articles, which serves as training data for LLaMA.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Please use commas to make large numbers more readable (e.g., 76036 -> 76,036) throughout the article"}, "extra_text": {}}, {"review_id": "mThipzqM2g", "reviewer": "Reviewer_CaEc", "summary": "", "questions": "I understand that the original dataset was in English and then it was translated into multiple languages. However, some facts might culture/language-specific. Something that is common knowledge in one culture, might be not necessary so in another (of course, I\"m not talking about such basic facts as \"capital of\"). Did you control for it?\n\nIt would be interesting to include cross-lingual (rather than multilingual) experiments. Is it possible to for the model to predict o in language X given s and r in language Y? What language pairs work better? (Latin-based or Cyrillic- based?) ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your remarks! It’s very meaningful that our project received insightful feedback from another member of the field. The goal of our project was to interrogate the capacity of LLMs to retrieve factual information in both an English-only and multilingual setting. Through our original “Polyglot or Not” benchmark and supporting analysis, we ultimately conclude that today’s \"foundation models\" – a term commonly used in the literature (see Section 1, Introduction) – possess numerous shortcomings when tasked with retrieving encyclopedic knowledge in languages other than English. \n\nAs you point out, most prior work in this domain is conducted in an English-only setting. As nearly 3 out of 4 people worldwide do not speak or write fluently in English, it’s critical to introduce verifiable LLM benchmarks that evaluate their capacity to work in languages other than English. In our view, this core focus of our project ties in nicely with the EMNLP ‘23 theme-track: *While the new generation of Large Language Models such as GPTX, LLAMA, BLOOM etc. claim to perform at unprecedented levels for generation and understanding, we are in unexplored territory on many aspects of such LLMs.* This paper represents one such “exploration” into that territory and provides a level-headed view of LLM performance via a novel benchmark. Along the way, we broaden our focus to other areas of interest relevant to the NLP community like equity and fairness through our supporting analysis.\n\nWe appreciate your recognition that the multilingual dataset we’ve developed is highly suitable for use in future research. We hope that by disseminating the dataset in multiple formats – as a file in our Github repository, but also on the easily viewable open-source repository “Hugging Face” – more people will be able to work with and ultimately use our pool of data for future research. \n\nWe agree that the **why** of these results is one of the many interesting questions borne out by our work. To start, we’ll point out that this is perhaps the fundamental outstanding question in language modeling writ large. As downstream users of these widely disseminated ‘foundation’ models, we’re often left wondering what specific modeling decisions or dataset quirks caused a model to generate one piece of text or another. Our paper strives to contribute to this discourse by conducting a comprehensive battery of analysis.\n\nThat said, we concede that we do not make explicitly causal claims in our study. We believe this cautious approach avoids assertions that lack proper empirical grounding. We do provide citations to an NLP survey (see Section 7, Future Work) that focuses on the causal components of fact retrieval catering to readers interested in deep study on this subject matter.\n\nInstead, in this short paper, we approach this “why” question through numerous types of analysis. In their totality, we believe that the quantitative and qualitative insights included in our study go further than mere reportage. Let’s recall some of those experiments and how their results connect back to the question you pose of “why” we see the results we do. \n\nWe conduct an analysis (see Section 6, Wikipedia’s Relationship to LLaMA’s Performance) of Wikipedia’s ‘density’ and establish a strong and significant correlation (Pearson’s r = 0.78,p < 0.001) between the presence of subjects from our dataset on Wikipedia pages and LLaMA’s ability to recall facts in that language. This demonstrates the impact of high-quality, dense training data for imbuing LLMs with credible factual associations. This work also provides original commentary about a ubiquitous source of training data for foundation models. To continue, we also detail the impact of training data corpus size (see Section 6, Training Data Versus Model Parameters). These findings provide a tangible \"why\" behind the differences in performance that we observed and strongly link our paper to other contemporary researchers in the space. The qualitative arm of the paper (see Section 6, Qualitative Insights) also elucidates our assessment results while avoiding spurious claims of causality. For instance, we note that entity bias likely hampers LLaMa’s ability to accurately answer questions about geographic locations that have European-influenced names or origins, such as Antarctica’s Cape Monaco, but are not actually located on the European continent. **For each of these and other experiments we can more clearly define their connection to the \"why\" of our benchmark results within Section 6, Analysis.**\n\nYou are right to point out that some facts are culture/language-specific. Take a multilingual model that has been trained on Swedish language Wikipedia data. We could prompt this model to complete a cloze statement regarding an American NFL player. Intuitively, one would surmise there is less high-quality information about that specific player and organization on the Swedish Wikipedia compared to the English language wikipedia.  Thus, this question will be more “difficult” when posed in Swedish compared to English. We do not control for this sort of phenomenon at this time in our paper, because it would be quite challenging to formulate a fair and balanced process for doing so. Indeed, what is culture/language-specific for one stem/fact pair might be universal for another culture or language and changes constantly based on new types of interactions between different cultures. Challenges notwithstanding, given the work we have already carried out, it is possible that the summary statistics from our Wikipedia analysis (see Table 6, Wikipedia content analysis) could be incorporated into as ‘priors’ in our assessment to down/upweight the raw likelihoods returned from prompting the model. This is one way we could control for culture/language specific knowledge during our assessment. **We can discuss this idea in our Future Work section (with the caveats being that, as mentioned, formulating this prior in an even-handed fashion would be somewhat challenging).**\n\nIf we are interpreting your remarks about cross-lingual experimentation correctly, a summary of the process would look like: part of the sentence is in one language and the output should be in another. This would be a challenging, but also highly interesting, modification to our existing methodology, because it implicates the universality of internal fact representations - and if/how linguistic similarities play a role in it. **This is certainly a direction we could delve further into in our Future Work section as well.**\n\nWe hope that this rebuttal clarified the strengths of our project and shows you that we are committed to improving this paper even further thanks to your insightful remarks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an interesting multilingual exploration on how LLMs perform in non-English settings. The paper also produced multilingual datasets for this task useful for further experimentation. \nThe paper includes an interesting section on subject entity error analysis.\n", "reasons_to_reject": "My main criticism comes from the fact that apart from reporting the results of the experiments, the paper doesn't attempt to explain \"why\" they got these results. This is the most interesting question here. ", "questions_to_authors": "I understand that the original dataset was in English and then it was translated into multiple languages. However, some facts might culture/language-specific. Something that is common knowledge in one culture, might be not necessary so in another (of course, I\"m not talking about such basic facts as \"capital of\"). Did you control for it?\n\nIt would be interesting to include cross-lingual (rather than multilingual) experiments. Is it possible to for the model to predict o in language X given s and r in language Y? What language pairs work better? (Latin-based or Cyrillic- based?) ", "paper_topic_and_main_contributions": "This paper describes experiments that assess how well LLMs retrieve encyclopedic knowledge. The experiments are multilingual. \nThe authors evaluate 5 LLMs on 20 languages. They build off work done by Dong et al. 2022 for English. The main conclusion is that LLMs do not perform as well on non-English datasets, especially those written in non-Latin scripts (Cyrillic). \n ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "U6Ur0SxSix", "reviewer": "Reviewer_vbS2", "summary": "", "questions": "Fact knowledge is formalized as ⟨s, r, o⟩ triples (said to be from an \"original natural language sentence\") whereas the dataset is rather like (stem, subject, object) triples where stem is a grammatically valid formulation of a fact-triple. There's some philosophical or modeling clarification needed here. Are fact triples language independent, e.g., is (Paris, capital of, France) the same fact as (Paris, capitale de, France) and (Paris, Hauptstadt von, Frankreich)? Several different (stem, subject, object) triples can represent the same fact as there are usually several different ways to formulate things with the same propositional content. ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! We are appreciative of the time you spent learning about our project, where we explore the capacity for foundation models to retrieve encyclopedic knowledge and show that even massive LLMs are hindered in their ability to learn language-independent knowledge. As you neatly put it: foundation LLMs appear to “abstract a combo of language-independent knowledge and language-specific patterns”, as uncovered when they are tasked with a multilingual cloze completion task like our “Polyglot or Not” test. \n\nYou’re right to point out that our test takes advantage of comparing true/false completions. The point about how in **real world application the prior will not be 50/50**, from our perspective, warrants follow-up. We employed a contrastive knowledge assessment test. At the heart of the method is the comparison between a ground truth “fact” and one or more “counterfactuals.” As is evident in our dataset, many of the items have multiple counterfactuals. Take an illustrative example: “Steve Jobs was the founder of ____.” The true completion would be “Apple” and the counterfactuals may be a list of several false completions like “Google”, “Amazon”, and “Facebook”. Our test compares the probability of the true prediction against the mean probability of the false predictions. In this manner, we take into account a more nuanced perspective on our cloze test than a black and white 50% prior/baseline ratio to probe a particular piece of knowledge. The test thus necessitates the comparison of a factual completion with counterfactual completion(s).\n\nTo answer your question about **language independence**: our dataset intentionally avoids including multiple variations of a fact to prevent double-counting. We utilize relation IDs to ensure that only one standard form of a given triplet is included, such as \"is capital of.\" In the pre-processing stage, we implement this de-deduplication process for the English split of the dataset. This set is retained during translation across the 19 non-English languages in the test, further solidifying the language-independent representation of facts within our dataset. To reiterate, the statements with identical <s,r,o> triplets across languages are indeed treated as equivalent, representing a single piece of knowledge. We will include these details elaborating our design choice and its implications.\n\nLastly, thank you for pointing out the fixes we can make to the broken hyperlink reference in fn 13. It appears the structure of Microsoft’s “azure-docs” repo has been re-factored since our original date of submission. We will add the updated link to our paper along with dates of access to all links in the paper’s footnotes to ensure that we are accurately documenting these external sources in a way that’s visible for future researchers.\n\nWe hope that our rebuttal has clarified some of the strengths of our project and demonstrated our intention to mend any potential shortcomings therein.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It's a well conducted study and the finding is clear and of interest.", "reasons_to_reject": "There are no real reasons to reject but of course some aspects could have been done differently or ontop. The accuracy numbers will change if the prior/baseline ratio of 50% true/false questions are asked. For many real world applications the prior will not be 50/50. Would have been interesting to know how drastic the change in accuracy would be.", "questions_to_authors": "Fact knowledge is formalized as ⟨s, r, o⟩ triples (said to be from an \"original natural language sentence\") whereas the dataset is rather like (stem, subject, object) triples where stem is a grammatically valid formulation of a fact-triple. There's some philosophical or modeling clarification needed here. Are fact triples language independent, e.g., is (Paris, capital of, France) the same fact as (Paris, capitale de, France) and (Paris, Hauptstadt von, Frankreich)? Several different (stem, subject, object) triples can represent the same fact as there are usually several different ways to formulate things with the same propositional content. ", "paper_topic_and_main_contributions": "This paper evaluates Encyclopedic Knowledge Retrieval from LLMs using a manually curated dataset spanning some 20 languages. The finding is clear, that there is a decline according to language (and/or script --- though this distinction is not really within scope of the paper) so even the most powerful LLMs with lots of parameters can't really be said to abstract (language-independent) knowledge, rather they abstract a combo of language independent knowledge and language-specific patterns. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are broken hyperlink references, e.g.\n\nfn 13\nhttps://github.com/MicrosoftDocs/azuredocs/\nblob/main/articles/cognitiveservices/\nopenai/includes/chat-markup-language.md\n\nand the link refs do not have dates of access."}}]

Author rebuttal:

Meta-review: The reviewers agreed that this was an interesting study and the focus on non-English settings will have an impact on many people in the field working with non-English data. Even in a short paper, the authors make several important contributions, including a new multilingual dataset and evaluation of several major LLMs over the data with some interesting implications about the use of models in multilingual settings. There were no addressable reasons to reject pointed out other than lack of depth of some discussions; I do not see any major issues with the soundness of the paper that were pointed out. Several of the reviewers also found the paper to be an exciting direction. There were a few minor presentation and formatting suggestions for the authors to consider, as well.
Decision: Accept-Main