Scaling Multilingual Corpora and Language Models to 500 Languages
Year: 2023

Abstract:
The NLP community has mainly focused on scaling Large Language Models (LLMs) vertically, i.e., making them better for about 100 languages. We instead scale LLMs horizontally: we create, through continued pretraining, Glot500-m, an LLM that covers 511 predominantly low-resource languages. An important part of this effort is to collect and clean Glot500-c, a corpus that covers these 511 languages and allows us to train Glot500-m. We evaluate Glot500-m on five diverse tasks across these languages. We observe large improvements for both high-resource and low-resource languages compared to an XLM-R baseline. Our analysis shows that no single factor explains the quality of multilingual LLM representations. Rather, a combination of factors determines quality including corpus size, script, “help” from related languages and the total capacity of the model. Our work addresses an important goal of NLP research: we should not limit NLP to a small fraction of the world’s languages and instead strive to support as many languages as possible to bring the benefits of NLP technology to all languages and cultures. Code, data and models are available at https://github.com/cisnlp/Glot500.

heading: 1 Introduction
text: The NLP community has mainly focused on scaling Large Language Models (LLMs) vertically, i.e., deepening their understanding of high-resource languages by scaling up parameters and training data. While this approach has revolutionized NLP, the achievements are largely limited to high-resource languages. Examples of “vertical” LLMs are GPT3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022) and Bloom (BigScience et al., 2022). In this paper, we create Glot500-m, a model that instead focuses on scaling multilingual LLMs horizontally, i.e., scaling to a large number of languages the great
*Equal contribution.
majority of which is low-resource. As LLMs are essential for progress in NLP, lack of LLMs supporting low-resource languages is a serious impediment to bringing NLP to all of the world’s languages and cultures. Our goal is to address this need with the creation of Glot500-m.1
Existing multilingual LLMs support only about 100 (Conneau et al., 2020) out of the 7000 languages of the world. These supported languages are the ones for which large amounts of training data are available through projects such as Oscar (Suárez et al., 2019) and the Wikipedia dumps.2 Following Siddhant et al. (2022), we refer to the 100 languages covered by XLM-R (Conneau et al., 2020) as head languages and to the remaining languages as tail languages. This terminology is motivated by the skewed distribution of available data per language: for the best-resourced languages there are huge corpora available, but for the long tail of languages, only small corpora exist. This is a key problem we address: the availability of data for tail languages is limited compared to head languages. As a result, tail languages have often been ignored by language technologies (Joshi et al., 2020).
Although there exists some work on machine translation for a large number of tail languages (Costa-jussà et al., 2022; Bapna et al., 2022), existing LLMs for tail languages are limited to a relatively small number of languages (Wang et al., 2019; Alabi et al., 2022; Wang et al., 2022). In this paper, we address this gap. Our work has three parts. (i) Corpus collection. We collect Glot2000-c, a corpus covering thousands of tail languages. (ii) Model training. Using Glot500-c, a subset of Glot2000-c, we train Glot500-m, an LLM covering 511 languages. (iii) Validation. We conduct an extensive evaluation of the quality of Glot500-m’s
1In concurrent work, Adebara et al. (2022) train a multilingual model for 517 African languages on a 42 gigabyte corpus, but without making the model available.
2https://dumps.wikimedia.org/
1082
representations of tail languages on a diverse suite of tasks.
In more detail, corpus collection considers three major sources: websites that are known to publish content in specific languages, corpora with classified multilingual content and datasets published in specific tail languages. The resulting dataset Glot2000-c comprises 700GB in 2266 languages collected from ≈150 sources. After cleaning and deduplication, we create the subset Glot500-c, consisting of 511 languages and 534 language-scripts (where we define a language-script as a combination of ISO 639-33 and script) to train Glot500-m. Our criterion for including a language-script in Glot500-c is that it includes more than 30,000 sentences.
Model training. To train Glot500-m, we employ vocabulary extension and continued pretraining. XLM-R’s vocabulary is extended with new tokens trained on Glot500-c. We then perform continued pretraining of XLM-R with the MLM objective (Devlin et al., 2019).
Validation. We comprehensively evaluate Glot500-m on a diverse suite of natural language understanding, sequence labeling and multilingual tasks for hundreds of languages. The results demonstrate that Glot500-m performs better than XLMR-B (XLM-R-base) for tail languages by a large margin while performing comparably (or better) for head languages.
Previous work on multilinguality has been hindered by the lack of LLMs supporting a large number of languages. This limitation has led to studies being conducted in settings dissimilar from realworld scenarios. For example, Dufter and Schütze (2020) use synthetic language data. And the curse of multilinguality has been primarily studied for a set of high-resource languages (Conneau et al., 2020). By creating Glot500-m, we can investigate these issues in a more realistic setting. We make code, data and trained models available to foster research by the community on how to include hundreds of languages that are currently ill-served by NLP technology.
Contributions. (i) We train the multilingual model Glot500-m on a 600GB corpus, covering more than 500 diverse languages, and make it publicly available at https://github.com/cisnlp/ Glot500. (ii) We collect and clean Glot500-c, a corpus that covers these diverse languages and al-
3https://iso639-3.sil.org/code_tables/639
lows us to train Glot500-m, and will make as much of it publicly available as possible. (iii) We evaluate Glot500-m on pseudoperplexity and on five diverse tasks across these languages. We observe large improvements for low-resource languages compared to an XLM-R baseline. (iv) Our extensive analysis shows that no single factor explains the quality of multilingual LLM representations. Rather, a combination of factors determines quality including corpus size, script, “help” from related languages and the total capacity of the model. (v) Our work addresses an important goal of NLP research: we should not limit NLP to a relatively small number of high-resource languages and instead strive to support as many languages as possible to bring the benefits of NLP to all languages and cultures.

heading: 2 Related Work
text: Training multilingual LLMs using the masked language modeling (MLM) objective is effective to achieve cross-lingual representations (Devlin et al., 2019; Conneau et al., 2020). These models can be further improved by incorporating techniques such as discriminative pre-training (Chi et al., 2022) and the use of parallel data (Yang et al., 2020; Chi et al., 2021). However, this primarily benefits a limited set of languages with large corpora.
Recent research has attempted to extend existing LLMs to languages with limited resources. Wang et al. (2019) propose vocabulary extension; Ebrahimi and Kann (2021) investigate adaptation methods, including MLM and Translation Language Model (TLM) objectives and adapters; Alabi et al. (2022) adapt XLM-R to 17 African languages; Wang et al. (2022) expand language models to low-resource languages using bilingual lexicons.
Alternatively, parameter-efficient fine-tuning adapts pre-trained models to new languages by training a small set of weights effectively (Zhao et al., 2020; Pfeiffer et al., 2021; Ansell et al., 2022). Pfeiffer et al. (2022) address the “curse of multilinguality” by sharing a part of the model among all languages and having separate modules for each language. We show that the common perception that multilinguality increases as we add more languages, until, from some point, it starts decreasing, is naive. The amount of available data per language and the similarity between languages also play important roles (§6.8).
Another approach trains LLMs from scratch for a limited number of tail languages; e.g., AfriBERTa
(Ogueji et al., 2021a) and IndicNLPSuite (Kakwani et al., 2020) are LLMs for 11 African languages and 11 Indic languages. In concurrent work, Adebara et al. (2022) train a multilingual model for 517 African languages on a 42 GB corpus, but without making the model available and with an evaluation on a smaller number of languages than ours.
Closely related to our work on corpus creation, Bapna et al. (2022) and Costa-jussà et al. (2022) also create NLP resources for a large number of tail languages. They train a language identifier model and extract textual data for tail languages from largescale web crawls. This approach is effective, but it requires significant computational resources and native speakers for all tail languages. This is hard to do outside of large corporations. Bapna et al. (2022) have not made their data available. Costajussà et al. (2022) have only released a portion of their data in around 200 languages.
A key benefit of “horizontally” scaled multilingual LLMs is transfer from high- to low-resource languages. Our evaluation suggests that Glot500-m excels at this, but this is not the main focus of our paper. There is a large body of work on crosslingual transfer: (Artetxe and Schwenk, 2019; ImaniGooghari et al., 2022; Lauscher et al., 2020; Conneau et al., 2020; Turc et al., 2021; Fan et al., 2021; Severini et al., 2022; Choenni and Shutova, 2022; Wang et al., 2023), inter alia.

heading: 3 Glot2000-c

heading: 3.1 Data Collection
text: One of the major challenges in developing NLP technologies for tail languages is the scarcity of high-quality training data. In this work, we propose a lightweight methodology that is easily replicable for academic labs. We identify tail language data previously published by researchers, publishers and translators and then crawl or download them. By crawling a few websites and compiling data from around 150 different datasets, we amass more than 700GB of text in 2266 languages. We will refer to these sources of data as data sources. Our data covers many domains, including religious texts, news articles and scientific papers. Some of the data sources are high-quality, verified by native speakers, translators and linguists. Others are less reliable such as web crawls and Wikipedia dumps. It is therefore necessary to clean the data. For a list of data sources, see §C.

heading: 3.2 Language-Scripts
text: Some languages are written in multiple scripts; e.g., Tajik is written in both Cyrillic and Arabic scripts. Some data sources indicate the script, but others either do not or provide mixed text in multiple scripts. We detect the script for each sentence and treat each language-script as a separate entity.

heading: 3.3 Ngram LMs and Language Divergence
text: We train a 3-gram character-level language model 𝑀𝑖 for each language-script 𝐿𝑖, using KenLM (Heafield, 2011). We refer to the perplexity calculated for the corpus of language 𝐿𝑖 using language model 𝑀 𝑗 as PP(𝑀 𝑗 , 𝐿𝑖). Similar to Gamallo et al. (2017), we define a perplexity-based divergence measure of languages 𝐿𝑖 and 𝐿 𝑗 as:
D𝐿𝑖 ,𝐿 𝑗 = max (PP(𝑀 𝑗 , 𝐿𝑖),PP(𝑀𝑖 , 𝐿 𝑗))
We use D to filter out noisy data in §3.4 and study the effect of similar languages in LLM training in §6.7 and §6.8. For more details, see §A.

heading: 3.4 Data Cleaning
text: To remove noise, we use chunk-level and corpuslevel filters.
While some sources are sentence-split, others provide multiple sentences (e.g., a paragraph) as one chunk. Chunk-level filters process each chunk of text from a data source as a unit, without sentencesplitting. Some chunk-level filters are based on the notion of word: we use white space tokenization when possible and otherwise resort to sentencePiece (Kudo and Richardson, 2018) trained by Costa-jussà et al. (2022).
As chunk-level filters, we employ the sentencelevel filters SF1–SF5 from BigScience ROOTS (Laurençon et al., 2022).
SF1 Character repetition. If the ratio of repeated characters is too high, it is likely that the sentence has not enough textual content.
SF2 Word repetition. A high ratio of repeated words indicates non-useful repetitive content.
SF3 Special characters. Sentences with a high ratio of special characters are likely to be crawling artifacts or computer code.
SF4 Insufficient number of words. Since training language models requires enough context, very small chunks of text are not useful.
SF5 Deduplication. If two sentences are identical after eliminating punctuation and white space, one is removed.
la ng s sc
rip ts
se nt
s’
m ed
ia n
s’
Glot2000-c 2266 35 2.3B 8K Glot500-c 511 30 1.5B 120K Costa-jussà et al. (2022) 134 - 2.4B 3.3M Bapna et al. (2022) 1503 - 1.7B 25K

heading: 3.5 Training Data: Glot500-c
text: XLM-R-B XLM-R-L Glot500-m
Model Size 278M 560M 395M Vocab Size 250K 250K 401K Transformer Size 86M 303M 86M

heading: 4.2 Continued Pretraining

heading: 4.1 Vocabulary Extension

heading: 4 Glot500-m
text: |head| |tail| measure (%)
Sentence Retrieval Tatoeba 70 28 Top10 Acc. Sentence Retrieval Bible 94 275 Top10 Acc. Text Classification 90 264 F1 NER 89 75 F1 POS 63 28 F1 Roundtrip Alignment 85 288 Accuracy

heading: 5 Experimental Setup
text: the cross-lingual quality of representations is high. Vice versa, failure to get back to 𝑤 is a sign of poor multilingual representations.
We use SimAlign (Jalili Sabet et al., 2020) and align on the sub-word level on the Bible part of test, based on the representations of the LLM computed by transformer layer 8 as suggested in the original paper. We use intersection symmetrization: each word in a sentence’ is aligned to at most one word in the other sentence’.
As evaluation measure we compute the percentage of roundtrips that were successes, i.e., the roundtrip starts at 𝑤 in L1 and returns back to 𝑤. For each language-script in test, we randomly select three language-scripts as intermediate points L2, L3, L4. Since the intermediate points influence the results, we run the experiment five times with different intermediate points and report the average. All models are evaluated with the same five sets of three intermediate language-scripts.
Sequence Labeling We consider two sequence labeling tasks: Named Entity Recognition (NER) and Part-Of-Speech (POS) tagging. We use the WikiANN dataset (Pan et al., 2017) for NER and version v2.11 of Universal Dependencies (UD) (de Marneffe et al., 2021) for POS. Since training data does not exist for some languages, we finetune on English (with early stopping based on dev) and evaluate zero-shot transfer on all languages covered by WikiANN/UD. We set the learning rate to 2e-5 with Adam.
Sentence Retrieval Following (Hu et al., 2020), we use up to 1000 English-aligned sentences’ from Tatoeba (Artetxe and Schwenk, 2019) to evaluate SentRetr (sentence retrieval). We also use 500 English-aligned sentences’ from the Bible part of test. We find nearest neighbors using cosine similarity based on the average word embeddings in layer 𝑙 = 8 – following Jalili Sabet et al. (2020) – and compute top10 accuracy. For fair comparison and because the architectures are the same, we do not optimize the hyperparameter 𝑙 for Glot500-m and XLM-R-B.
Text Classification We evaluate on Taxi1500 (Ma et al., 2023). It provides gold data for text classification with six classes in a large number of language-scripts of which Glot500-m supports 354. We finetune on English (with early stopping on dev) and evaluate zero-shot on test of the target language-script. Learning rate: 2e-5, batch size:
16 (following Ma et al. (2023)).

heading: 6 Experiments
text: In this section, we discuss aggregate results. For detailed results, see §D and §E.

heading: 6.1 Results
text: Table 4 gives results. Glot500-m outperforms XLM-R-B on all tasks for both head and tail language-scripts, except for POS on head. That Glot500-m outperforms XLM-R-B is expected for tail language-scripts (i.e., those not covered by XLM-R). For these language-scripts the improvement margin is large. Outperformance may seem counterintuitive for head language-scripts (those covered by XLM-R) since Glot500-m has the same number of (non-embedding) parameters as XLMR-B. Since the number of covered languages has greatly increased, leaving less capacity per language, we might expect underperformance. There are a few possible explanations. First, XLM-R may be undertrained, and the inclusion of more head language training data may improve their representations. Second, having more languages may improve multilinguality by allowing languages to synergize and enhance each other’s representations and cross-lingual transfer. Third, there are languages similar to head languages among the tail languages, which in turn aids head languages.
The gap between Glot500-m and the baselines for tail language-scripts in sequence labeling is smaller. These tasks do not require as deep an understanding of language and thus transfer from head to tail language-scripts is easier through shared tokens.
Glot500-m also outperforms XLM-R-L for tail language-scripts (all tasks) and head languagescripts (3 tasks). This suggests that scaling up size is not the only way for improvements. We can also improve the quality of multilingual LLM representations by increasing the number of languages.

heading: 6.2 Language Coverage
text: Table 5 compares Glot500-m vs. XLM-R-B on pseudoperplexity. For fair comparison we use word-level normalization. For 69 head languagescripts, Glot500-m underperforms XLM-R-B. This is expected as Glot500-m’s training data is small for these language-scripts. Glot500-m outperforms XLM-R-B for 420 tail language-scripts.
There are eight tail language-scripts for which
Glot500-m performs worse than XLM-R-B. Five are tail languages with a similar head language where the two share a macro-language: ekk/Standard Estonian (est/Estonian), aln/Gheg Albanian (sqi/Albanian), nob/Norwegian Bokmal (nor/Norwegian), hbs/Serbo-Croatian (srp/Serbian), lvs/Standard Latvian (lav/Latvian). Since XLMR-B’s pretraining corpus is large for the five head languages, its performance is good for the close tail languages.
The other three languages all have a unique script: sat/Santali (Ol Chiki script), div/Dhivehi (Thaana script), iku/Inuktitut (Inuktitut syllabics). For these languages, XLM-R-B’s tokenizer returns many UNK tokens since it is not trained on these scripts, resulting in an unreasonably optimistic estimate of pseudoperplexity by our implementation.
Glot500-m’s token-level normalized pseudoperplexity ranges from 1.95 for lhu/Lahu to 94.4 for tok/Toki Pona. The average is 13.5, the median 10.6. We analyze the five language-scripts with the highest pseudoperplexity: tok_Latn, luo_Latn, acm_Arab, ach_Latn, and teo_Latn.
tok/Toki Pona is a constructed language. According to Wikipedia: “Essentially identical concepts can be described by different words as the choice relies on the speaker’s perception and experience.” This property can result in higher variability and higher perplexity.
acm/Mesopotamian Arabic contains a large number of tweets in raw form. This may result in difficult-to-predict tokens in test.
luo/Luo, ach/Acoli and teo/Teso are related Nilotic languages spoken in Kenya, Tanzania, Uganda and South Sudan. Their high perplex-
ity could be related to the fact that they are tonal languages, but the tones are not orthographically indicated. Another possible explanation is that the training data is dominated by one subcorpus (Jehova’s Witnesses) whereas the test data are dominated by PBC. There are orthographic differences between the two, e.g., “dong” (JW) vs. “doŋ” (PBC) for Acoli. These three languages are also spoken over a large area in countries with different standard languages, which could increase variability.
Our analysis is not conclusive. We note however that the gap between the three languages and the next most difficult languages in terms of pseudoperplexity is not large. So maybe Luo, Acoli and Teso are simply (for reasons still to be determined) languages that have higher perplexity than others.

heading: 6.3 Training Progression
text: To analyze the training process, we evaluate Glot500-m on sequence labeling and SentRetr at 10,000-step intervals. Figure 1 shows that performance improves rapidly at the onset of training, but then the rate of improvement slows down. This trend is particularly pronounced for tail languages in SentRetr. In comparison, sequence labeling is relatively straightforward, with the baseline (XLM-R-B, epoch 0) achieving high performance by correctly transferring prevalent classes such as verb and noun
through shared vocabulary, resulting in a smaller improvement of Glot500-m vs. XLM-R-B.
For SentRetr, we observe larger improvements for the Bible than for Tatoeba. This is likely due to the higher proportion of religious data in Glot500-c, compared to XLM-R’s training data (i.e., CC100).
The average performance on downstream tasks peaks at 480K steps. We have taken a snapshot of Glot500-m at this stage and released it.

heading: 6.4 Analysis across Language-Scripts
text: To analyze the effect of language-scripts, we select five tail language-scripts each with the largest and smallest gain when comparing Glot500-m vs. XLMR-B for SentRetr and sequence labeling.
Table 6 shows that Glot500-m improves languages with scripts not covered by XLM-R (e.g., div/Dhivehi, Thaana script, see §6.2) by a large margin since XLM-R simply regards the uncovered scripts as unknown tokens and cannot compute meaningful representations for the input. The large amount of data we collected in Glot500-c also contributes to the improvement for tail languages, e.g., for tat_Cyrl (Tatar) in SentRetr Tatoeba and mlt_Latn (Maltese) in POS. See §6.7 for a detailed analysis of the effect of corpus size.
On the other hand, Glot500-m achieves just comparable or even worse results for some languagescripts. We see at least three explanations. (i) As discussed in §6.2, some tail languages (e.g., nob/Norwegian Bokmal) are close to a head language (e.g., nor/Norwegian), so Glot500-m has no advantage over XLM-R-B. (ii) A language is at the low end of our corpus size range (i.e., 30,000 sentences’). Example: xav_Latn, Xavánte. (iii) Some languages are completely distinct from all other languages in Glot500-c, thus without support from any similar language. An example is mau_Latn, Huautla Mazatec. Glot500-m has a much harder
time learning good representations in these cases.

heading: 6.5 Languages with Multiple Scripts
text: Table 7 compares SentRetr performance XLM-R-B vs. Glot500-m for six languages with two scripts. Unsurprisingly, XLM-R performs much better for a language-script it was pretrained on (“head”) than on one that it was not (“tail”). We can improve the performance of a language, even surpassing the language-script covered by XLM-R, if we collect enough data for its script not covered by XLM-R. For languages with two scripts not covered by XLM-
R, the performance is better for the script for which we collect a larger corpus. For example, kaa_Cyrl (Kara-Kalpak) has about three times as much data as kaa_Latn. This explains why kaa_Cyrl outperforms kaa_Latn by 30%.
Dufter and Schütze (2020) found that, after training a multilingual model with two scripts for English (natural English and “fake English”), the model performed well at zero-shot transfer if the capacity of the model was of the right size (i.e., not too small, not too large). Our experiments with real data show the complexity of the issue: even if there is a “right” size for an LLM that supports both full acquisition of languages and multilingual transfer, this size is difficult to determine and it may be different for different language pairs in a large horizontally scaled model like Glot500-m.

heading: 6.6 Analysis across Language Families
text: Table 8 compares SentRetr performance Glot500-m vs. XLM-R-B for seven language families that have ten or more language-scripts in Glot500-c. We assign languages to families based on Glottolog.4 Generally, XLM-R has better performance the more language-scripts from a language family are represented in its training data; e.g., performance is better for indo1319 and worse for maya1287. The results suggest that Glot500-m’s improvement over
4http://glottolog.org/glottolog/family
XLM-R is the larger, the better our training corpus Glot500-c’s coverage is of a family.

heading: 6.7 Effect of Amount of Training Data
text: We examine correlation between pretraining corpus size and Glot500-m zero-shot performance. We focus on SentRetr Bible (§5) since it supports the most head and tail languages. We find that Pearson’s 𝑟 = .34, i.e., corpus size and performance are moderately, but clearly correlated. We suspect that the correlation is not larger because, in addition to corpus size of language 𝑙 itself, corpus size of languages closely related to 𝑙 is also an important factor (see §6.4 for a similar finding for Norwegian). We therefore also compute Pearson’s 𝑟 between (i) performance of language 𝑙 on SentRetr Bible and (ii) joint corpus size of 𝑙 and its 𝑘 nearest neighbors (according to perplexity divergence, §3.3). In this case, Pearson’s 𝑟 = .44 (for both 𝑘 = 3 and 𝑘 = 4), indicating that the corpus size of nearest neighbor languages does play a role.

heading: 6.8 Support through Related Languages
text: Building on §6.7, there is another way we can investigate the positive effect of closely related languages on performance: We can compare performance (again on SentRetr Bible) of continued pretraining on just one language (we refer to this model as Glot+1) vs. on all 511 languages represented in Glot500-c (i.e., Glot500-m). Table 9 presents results for six language-scripts selected from various language families and suggests that some languages do not receive support from related languages (top three). In that case, Glot+1 can fully concentrate on learning the isolated language and does better than Glot500-c. Other languages (bottom three) do receive support from related languages. For example, Southern Quechua (quh) seems to receive support in Glot500-m from closely related Cuzco Quechua (quz), resulting in Glot500-m outperforming Glot+1.

heading: 7 Conclusion and Future Work
text: We collect and data-clean Glot500-c, a large corpus of hundreds of usually neglected tail (i.e., long-tail) languages and create Glot500-m, an LLM that is trained on Glot500-c and covers these languages. We evaluate Glot500-m on six tasks that allow us to evaluate almost all languages. We observe large improvements for both head and tail languages compared to XLM-R. Our analysis shows that no single factor fully explains the quality of the representation of a language in a multilingual model. Rather, a combination of factors is important, including corpus size, script, “help” from related languages and the total capacity of the model.
This work is the first to create a language model on a dataset of several hundreds of gigabytes and to make it publicly available for such a large and diverse number of low-resource languages. In future research, we would like to train larger models to further investigate the effect of model size, distill highly multilingual models for resource-efficient deployment, explore alternatives to continued pretraining and use models for more tail language downstream tasks.
Limitations (1) We did not perform any comprehensive hyperparameter search, which would have further consolidated our results. This decision was made due to the high cost of training multiple models. (2) Compared to current very large models, Glot500-m
is comparatively small. (3) Although we have tried to minimize the amount of noise in our data, some noise is still present.
Ethics Statement
There are two issues worth mentioning in regards to this project. First, it was not feasible for us to thoroughly examine the content of the data for all languages, thus we cannot confirm the absence of discrimination based on factors such as race or sexuality. The data was solely utilized as a textual corpus, and the content should not be interpreted as an endorsement by our team. If the model is subsequently utilized for generation, it is possible that the training data may be reflected in the generated output. However, addressing potential biases within the data is an area for future research. Second, it is important to note that while the data sources utilized in this study do not explicitly prohibit the reuse of data for research purposes, some sources do have copyright statements indicating that such use is permissible while others do not. Additionally, certain sources prohibit the redistribution of data. As such, data from these sources is omitted from the published version of Glot2000-c.

heading: Acknowledgements
text: We would like to thank Renhao Pei, Yihong Liu, Verena Blaschke, and the anonymous reviewers. This work was funded by the European Research Council (grants #740516 and #758969) and EU’s Horizon Europe Research and Innovation Actions (UTTER, contract 101070631).

heading: A N-grams LMs and Language Divergence
text: Perplexity and Language Divergence. Perplexity measures how well a model predicts a sample test data. Assuming a test data contains sequences of
characters 𝑆 = 𝑐ℎ1, 𝑐ℎ2, · · · , 𝑐ℎ𝑇 , perplexity (PP) of 𝑆 given an n-gram character level language model 𝑀 is computed as follows:
PP(𝑆, 𝑀) = 𝑇 √√ 𝑇∏
𝑡=1 1 P ( 𝑐ℎ𝑡 | 𝑐ℎ𝑡−11 ) (1) where P ( 𝑐ℎ𝑡 | 𝑐ℎ𝑡−11 ) is computed as by dividing the observed frequency (𝐶) of 𝑐ℎ𝑡−11 𝑐ℎ𝑖 by the observed frequency of 𝑐ℎ𝑡−11 in 𝑀 training data:
P ( 𝑐ℎ𝑡 | 𝑐ℎ𝑡−11 ) = 𝐶 ( 𝑐ℎ𝑡−11 𝑐ℎ𝑡 ) 𝐶 ( 𝑐ℎ𝑡−11
) (2) Given the definition of perplexity, we can determine how well a trained language model on language 𝐿1 predicts the test text of language 𝐿2 and vice-versa. The divergence between two languages is computed with the maximum of the perplexity values in both directions. Two reasons lead to the use of max: first, a symmetrical divergence is required, and second, languages differ in their complexity, so one direction of computing perplexity may result in a much lower perplexity than another. Thus, comparing perplexity results becomes difficult. As an example, the Kuanua language (ksd_Latn) has short words and a simple structure, which results in 3−gram models getting lower perplexity on its text compared to other languages. The lower the perplexity the smaller the divergence between languages. The divergence (D) between language 𝐿𝑖 and 𝐿 𝑗 with trained language models of 𝑀𝐿𝑧 and test texts of 𝑆𝐿𝑧 , where 𝐿𝑧 is the corresponding language, computed as follows:
D𝐿𝑖 ,𝐿 𝑗 = max (PP(𝑆𝐿𝑖 , 𝑀𝐿 𝑗 ),PP(𝑆𝐿 𝑗 , 𝑀𝐿𝑖 )) (3)
Runs and Data. The data used to train and test the character level n-gram models is the same data used for the training and testing of the Glot500-m. The training of the models was limited to 100, 000 sentences’ per language-script. We use KenLM library (Heafield, 2011) to build n-gram models. This library uses an interpolated modified KneserNey smoothing for estimating the unseen n-grams. Our evaluation has been performed over 7 n-gram models (3 ≤ 𝑛 ≤ 9). Baseline and Evaluation. Language family trees were used as a baseline for evaluating the divergence measures of the proposed approach. We obtained language family tree data from Ethnologue online version (Eberhard et al., 2022). For
each language, the family tree follows the general order from largest typological language family group to smallest. There is only one family tree for each language in the baseline data. Nodes in the family tree represent typological language family groups. Each node only has one parent, so if a node is common in the family tree of two languages, its parent is also common. We evaluate our perplexity method on the following binary classification task: Do the majority of a language 𝐿𝑧’s 𝑘 nearest neighbors belong to the same typological language family group as 𝐿𝑧? Assuming languages 𝐿𝑖 and 𝐿 𝑗 , with the following family trees:
𝑇𝐿𝑖 : 1 → 2 → 3 → 4 → 5 → 6 𝑇𝐿 𝑗 : 1 → 2 → 7 → 8
These 2 languages belong to the same typological family group with family tree levels of 𝑙 ∈ {1, 2}, but not with family tree levels of 𝑙 = 3 and higher. Result. When it comes to language families, the majority of studies only refer to the largest typological language family group (level 𝑙 = 1). Here, we also assess our methodology for other levels. The results of classification accuracy for 3−gram model, 𝑘 ∈ {1, 3, 7, 13, 21} and 𝑙 ∈ {1, 2, 3,max} are shown in Table 10. In cases where the maximum level of a tree is less than the 𝑙 parameter, the maximum level for that language is used. Languages without a family or no other family member in our data are excluded. We only report the 3−gram model results as it gets the best results in most configurations among other n-gram models. With increasing 𝑙, the accuracy decreases, since more languages fall outside the same typological family. As 𝑘 increases, the accuracy decreases, because languages with faraway neighbors are being included but the number of languages in the language typological group family will remain the same. There are times when languages have a lot of loan words from other languages because of geological proximity or historical reasons (e.g, colonization), which makes them similar to the languages they borrowed words from in our method. However they are different when it comes to their typological families and our method fails in these cases. Aymara (Macrolanguage: aym_Latn) and Quechua (Macrolanguage: que_Latn), for example, had a great deal of contact and influence on each other, but they do not belong to the same typological group. As well, some of the typological families are not that large, which makes our results worse when 𝑘 increases. This is

heading: B Languages
text: The list of languages used to train Glot500-m with the amount of available data for each language is available in Tables 11, 12 and 13.
On Macrolanguages The presence of language codes that are supersets of other language codes within datasets is not uncommon (Kreutzer et al., 2022). This issue becomes more prevalent in extensive collections. Within the ISO 639-3 standard, these languages are referred to as macrolanguages. When confronted with macrolanguages, if it is not feasible to ascertain the specific individual language contained within a dataset, the macrolanguage code is retained. Consequently, it is possible that in Glot2000-c and Glot500-c both the corpora for the macrolanguage and its individual languages have been included.

heading: C List of data sources
text: The datasets and repositories used in this project involve: AI4Bharat,5 AIFORTHAI-LotusCorpus,6 Add (El-Haj et al., 2018), AfriBERTa (Ogueji et al., 2021b), AfroMAFT (Adelani et al., 2022; Xue et al., 2021), Anuvaad,7 AraBench (Sajjad et al., 2020), AUTSHUMATO,8 Bloom (Leong et al., 2022), CC100 (Conneau et al., 2020; Wenzek et al., 2020a), CCNet (Wenzek et al., 2020b), CMU_Haitian_Creole,9 CORP.NCHLT,10 Clarin,11 DART (Alsarsour et al., 2018), Earthlings (Dunn, 2020), FFR,12 Flores200 (Costa-jussà et al., 2022), GiossaMedia (Góngora et al., 2022, 2021), Glosses (Camacho-Collados et al., 2016), Habibi (El-Haj, 2020), HinDialect (Bafna, 2022), HornMT,13 IITB (Kunchukuttan et al., 2018), IndicNLP (Nakazawa et al., 2021), Indiccorp (Kakwani et al., 2020), isiZulu,14 JParaCrawl (Morishita et al., 2020), KinyaSMT,15 LeipzigData (Goldhahn et al., 2012), Lindat,16 Lingala_Song_Lyrics,17 Lyrics,18 MC4 (Raffel et al., 2020), MTData (Gowda et al., 2021), MaCoCu (Bañón et al., 2022), Makerere MT Corpus,19 Masakhane community,20 Mburisano_Covid,21 Menyo20K (Adelani et al., 2021), Minangkabau corpora (Koto and Koto, 2020), MoT (Palen-Michel et al., 2022), NLLB_seed (Costa-jussà et al., 2022), Nart/abkhaz,22 OPUS (Tiedemann, 2012), OSCAR (Suárez et al., 2019), ParaCrawl (Bañón et al., 2020), Parallel Corpora for Ethiopian Lan-
5https://ai4bharat.org/ 6https://github.com/korakot/corpus/releases/
download/v1.0/AIFORTHAI-LotusCorpus.zip 7https://github.com/project-anuvaad/
anuvaad-parallel-corpus 8https://autshumato.sourceforge.net/ 9http://www.speech.cs.cmu.edu/haitian/text/
10https://repo.sadilar.org/handle/20.500.12185/ 7
11https://www.clarin.si/ 12https://github.com/bonaventuredossou/ffr-v1/
tree/master/FFR-Dataset 13https://github.com/asmelashteka/HornMT 14https://zenodo.org/record/5035171 15https://github.com/pniyongabo/kinyarwandaSMT 16https://lindat.cz/faq-repository 17https://github.com/espoirMur/songs_lyrics_
webscrap 18https://lyricstranslate.com/ 19https://zenodo.org/record/5089560 20https://github.com/masakhane-io/
masakhane-community 21https://repo.sadilar.org/handle/20.500.12185/
536 22https://huggingface.co/datasets/Nart/abkhaz_
text
guages (Abate et al., 2018), Phontron (Neubig, 2011), QADI (Abdelali et al., 2021), Quechua-IIC (Zevallos et al., 2022), SLI_GalWeb.1.0 (Agerri et al., 2018), Shami (Abu Kwaik et al., 2018), Stanford NLP,23 StatMT,24 TICO (Anastasopoulos et al., 2020), TIL (Mirzakhalov et al., 2021), Tatoeba,25 TeDDi (Moran et al., 2022), Tilde (Rozis and Skadin, š, 2017), W2C (Majliš, 2011), WAT (Nakazawa et al., 2022), WikiMatrix (Schwenk et al., 2021), Wikipedia,26 Workshop on NER for South and South East Asian Languages (Singh, 2008), XLSum (Hasan et al., 2021).

heading: D Results for Each Task and Language
text: We report the detailed results for all tasks and languages in Table 14 (Sentence Retrieval Tatoeba), 15, 16 (Sentence Retrieval Bible), 17 (NER), and 18 (POS), 19, 20 (Text Classification), 21, 22 (Round Trip Alignment).

heading: E Perplexity Results for all Languages
text: Perplexity number for all languages is presented in Table 23, Table 24, and Table 25.
23https://nlp.stanford.edu/ 24https://statmt.org/ 25https://tatoeba.org/en/ 26https://huggingface.co/datasets/wikipedia
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: section ’Limitation’

heading: 3 A2. Did you discuss any potential risks of your work?
text: section ‘Ethics Statement’

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 3.3, section 4, appendix c

heading: 3 B1. Did you cite the creators of artifacts you used?
text: section 3.3, section 4, appendix c

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: section ‘Ethics Statement’

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? section ‘Ethics Statement’
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Since our work deals with millions of sentences in hundreds of languages, it was impossible for us to check the content. We leave it as a future work

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? section 3.1, appendix a, appendix c
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. section 5
C 3 Did you run computational experiments? section 4.2

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? section 4.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? section 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 5. For continued pretraining, it is a single run due to computational resource limitation. For downstream task evaluation, it is multilple runs across 5 seeds.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 3.3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave it as a future work