Super donors and super recipients: Studying cross-lingual transfer between high-resource and low-resource languages (2024)

Abstract: Despite the increasing popularity of multilingualism within the NLP community, numerous languages continue to be underrepresented due to the lack of available resources. Our work addresses this gap by introducing experiments on cross-lingual transfer between 158 high-resource (HR) and 31 low-resource (LR) languages. We mainly focus on extremely LR languages, some of which are first presented in research works. Across 158 ∗ 31 HR–LR language pairs, we investigate how continued pretraining on different HR languages affects the mT5 model’s performance in representing LR languages in the LM setup. Our findings surprisingly reveal that the optimal language pairs with improved performance do not necessarily align with direct linguistic motivations, with subtoken overlap playing a more crucial role. Our investigation indicates that specific languages tend to be almost universally beneficial for pretraining (super donors), while others benefit from pretraining with almost any language (super recipients). This pattern recurs in various setups and is unrelated to the linguistic similarity of HR-LR pairs. Furthermore, we perform evaluation on two downstream tasks, part-of-speech (POS) tagging and machine translation (MT), showing how HR pretraining affects LR language performance.

text: Despite the increasing popularity of multilingualism within the NLP community, numerous languages continue to be underrepresented due to the lack of available resources. Our work addresses this gap by introducing experiments on cross-lingual transfer between 158 high-resource (HR) and 31 low-resource (LR) languages. We mainly focus on extremely LR languages, some of which are first presented in research works. Across 158 ∗ 31 HR–LR language pairs, we investigate how continued pretraining on different HR languages affects the mT5 model’s performance in representing LR languages in the LM setup. Our findings surprisingly reveal that the optimal language pairs with improved performance do not necessarily align with direct linguistic motivations, with subtoken overlap playing a more crucial role. Our investigation indicates that specific languages tend to be almost universally beneficial for pretraining (super donors), while others benefit from pretraining with almost any language (super recipients). This pattern recurs in various setups and is unrelated to the linguistic similarity of HR-LR pairs. Furthermore, we perform evaluation on two downstream tasks, part-of-speech (POS) tagging and machine translation (MT), showing how HR pretraining affects LR language performance.

heading: 1 Introduction
text: According to the Endangered Languages Project (Belew, 2019), more than 3000 languages are at risk of extinction. In recent years, the NLP community has undoubtedly broadened its efforts and presented very ambitious projects (NLLB Team et al., 2022; Bapna et al., 2022) to incorporate more and more languages into practical use. However, even well-known multilingual transformer models (e.g., mBERT (Devlin et al., 2019), XLM-R (Conneau
*Research was done while at AIRI.
et al., 2020), and mT5 (Xue et al., 2021)) and crosslingual benchmarks (XGLUE (Liang et al., 2020), XTREME (Ruder et al., 2021)) cover only about 100 of the most presented languages.
In this work, we aim to tackle this gap in underrepresentation or even the absence of experiments for LR languages, considering constraints in both labeled and unlabeled text data, as well as linguistic knowledge and experimental base. We present the study of cross-lingual transfer in the case of extremely LR languages, examining how continued pretraining on HR languages impacts model performance on LR languages in the Masked Language Modeling (MLM) setup with additional measurements of its effects on downstream performance (see Figure 1 for more details). We aim to explore whether it is possible to conduct model pretraining on HR languages and observe improvements compared to zero-shot performance when evaluating on unseen LR languages. Additionally, we measure model performance on downstream tasks: POS tagging and MT.
In more detail, we first collect a dataset with raw text data (see Appendix A.3 for the list of lan-
guages and Section 3 for criteria of LR and HR languages). Next, we exclude some languages due to data quality issues. Thereby, we experiment with 158 HR and 31 LR languages, resulting in 4898 HR-LR language pairs for further investigation of cross-lingual transfer. Next we assess the zero-shot performance of the mT5 model on the raw data from LR languages. Afterward, we perform continued pretraining of the model with the MLM objective on data from each HR language and evaluate performance of fine-tuned models on all LR languages. Finally, we analyze the factors, such as data and linguistic features, that lead to successful cross-lingual transfer between HR and LR languages. We also measure the downstream performance of successful HR-LR language pairs in POS tagging and MT tasks for LR languages with available annotated data. We do not use data from LR languages during training and use it for evaluation only. We use the term donor to denote the languages that serve as sources for knowledge transfer through continued pretraining. On the other hand, we use the term recipient to indicate the languages used to evaluate transfer learning efficiency.
The main contributions of this work can be summarized as follows: (i) We collect and present the dataset with 189 languages. (ii) We conduct crosslingual transfer experiments between 158 HR and 31 LR languages. (iii) We interpret cross-lingual transfer results across data and linguistic features. (iv) We investigate how cross-lingual transfer impacts the performance of downstream tasks, focusing mainly on the POS tagging and MT tasks. The code is available1.

heading: 2 Related Work
text: The cross-lingual transfer involves leveraging existing resources available for HR languages to improve methods for LR languages. This approach can be particularly beneficial for LR languages that lack extensive linguistic resources and data for NLP applications. Wu and Dredze (2020) state that the mBERT’s performance for LR languages is not on par with that for HR languages, and there is an unequal representation of languages within models. Libovický et al. (2019) show that mBERT context embeddings capture similarities between languages, but achieving a proper crosslingual representation requires the availability of
1Code: https://github.com/Vitaly-Protasov/LR_ Transfer
parallel corpora, which is lacking for most LR languages. Malkin et al. (2022) show that the selection of pretraining languages significantly impacts the performance, indicating that there are more effective donors than English. Additionally, Turc et al. (2021) show that Russian and German can serve as better donors for reliable transfer. Fujinuma et al. (2022) experiment with different number of languages during pretraining and find it promising in terms of impact on performance on unseen languages. There are also different suggested strategies for choosing the proper donor language. Kocmi and Bojar (2018) propose using vocabulary overlap to find a better HR donor. Lauscher et al. (2020) demonstrate that typological motivation in language selection positively impacts the transfer learning scores, as well as the size of the source language. Muller et al. (2021) show that the type of language script used plays an essential role, and transliteration helps to improve the quality of transfer learning. Eronen et al. (2023) also show that fine-tuning on linguistically similar languages (defined using WALS (Dryer and Haspelmath, 2013) improves the performance on several downstream tasks. Muller et al. (2022) investigate cross-lingual transfer using diverse data, revealing that morphology and language modeling performance are strong predictors of its success. Dolicki and Spanakis (2021); Lin et al. (2019) establish that no individual WALS feature stands out as the most crucial across various tasks.
Transfer learning has long emerged as a pivotal technique in machine translation, particularly for LR languages. Zoph et al. (2016) introduce an approach that uses HR language data to enhance neural machine translation (NMT) for LR language pairs, achieving notable improvements in their translation quality. Further exploration by Aji et al. (2020) reveal that word embeddings are a critical component of transfer learning, and their proper alignment is essential for optimal results. These findings highlight the critical role of transfer learning in addressing challenges associated with the limited availability of linguistic data in NMT.
The studies mentioned above focus on wellresourced languages with labeled data, which has resulted in neglecting LR languages that already lack available data. This study aims to address this gap by investigating cross-lingual transfer for several understudied LR languages.

heading: 3 HR-LR Multilingual Corpus
text: We assemble from various existing sources a text corpus. Appendix A.3 lists all used languages.

heading: 3.1 Text sources
text: To assemble a corpus for the need of cross-lingual experiments, we use a wide range of linguistic resources in addition to commonly used corpora. We deliberately do not include projects, such as Oscar (Ortiz Suárez et al., 2019) and Cleaned Colossal Common Crawl (Raffel et al., 2020) because they are already partially represented in the training set of large language models such as XLM-R and mT5. The general corpus includes text materials from the following projects: (i) Wikipedia in every language available (CC BY-SA); (ii) Universal Dependencies project2 (de Marneffe et al., 2021) (original texts without annotation, the license for every treebank is different, mainly GNU GPL 3.0/LGPLLR/CC BY-based); (iii) The Hamburg Center for Language Corpora (HZSK-PUB)3 (primary linguistic research textual data, not restricted by copyright or personal data protection); (iv) The Endangered Languages Archive4 (text content only, no multimedia, non-commercial private research or educational activity); (v) Corpora with annotated languages of CIS countries5 (Krylova et al., 2015).

heading: 3.2 Text processing
text: We aggregate languages according to their official names and codes presented in a large database, The World Atlas of Language Structures (WALS6) (Dryer and Haspelmath, 2013). To ensure high data quality for language processing, we exclude languages with a high presence of HTML tags in the collected data, accounting for 15% of the gathered data. We assume that a large amount of code would significantly affect results, as HTML tags are easier to predict than words in natural languages.
We collect both HR and LR languages. We define a LR language based on a specific range of tokens: 10k tokens as a lower bound and 350k tokens as an upper bound (Yang et al., 2019). Thus, we categorize languages exceeding the upper bound as HR ones. Refer to Appendix A.2 for the list of all languages we collected.
2https://universaldependencies.org 3https://corpora.uni-hamburg.de/hzsk 4https://www.elararchive.org 5http://web-corpora.net 6https://wals.info/feature

heading: 4 Cross-lingual Transfer Methodology
text: The main goal of our experiments is to figure out whether the training on HR language donors improves the modeling of LR recipient languages, try to interpret it and to observe possible performance of transfer learning in downstream tasks.

heading: 4.1 Base model
text: In our experiments, we utilize the widely used pretrained multilingual language model mT57(Xue et al., 2021). It is an encoder-decoder model trained on 101 languages from the mC4 dataset. It was originally pretrained in the transfer learning procedure and has shown itself well in transferring knowledge. We think the encoder-decoder architecture is more flexible and has more possible applications for future works than only encoder or decoder-based models. Due to its multitask finetuning, we decided not to use its another version, mT0 (Muennighoff et al., 2023). Considering our lack of labeled data, exploring its multitask zeroshot performance is unnecessary here.

heading: 4.2 MLM pretraining on donor languages
text: Following the original article of the mT5, we use the Masked Language Modeling (MLM) objective for the continued pretraining on HR donor languages. More specifically, in the case of mT5 model, this is a denoising task for prediction masked spans (sequential set) of tokens. Similarly to the original paper, we also utilize early stopping. We limit the data to perform the training for all languages under the same conditions and minimize the training time: 500k sampled sentences are chosen for continued pretraining on each HR language. We conduct this procedure 5 times to consider the variance of results based on different subsets of training data. We then average the results from the top-performing checkpoints within each training iteration. Textual data sourced from HR languages is employed during both the training and validation steps, while LR language data is utilized during the testing step only to measure the performance on unseen LR languages.

heading: 4.3 Evaluation on low-resource languages
text: We use the perplexity metric (Brown et al., 1992) to evaluate the MLM step. Perplexity has its limitations when evaluating language modeling performance, which may become evident in downstream
7https://huggingface.co/google/mt5-base
tasks. Since we deeply explore the results of crosslingual transfer, we also plan to conduct downstream evaluation afterward to see whether the continued pretraining on HR languages impacts the downstream performance on LR languages. Here, we first measure the zero-shot model’s performance in modeling all 31 LR languages. Secondly, we use the best model’s checkpoints from each run after continued pretraining on different HR language and evaluate them across all LR languages.

heading: 4.4 Analysis of transfer learning results
text: We are also interested in exploring potential factors that affect cross-lingual transfer results, determining whether they lead to success or failure in various language pairs. Following Lin et al. (2019), we utilize linguistic and data-level features to interpret cross-lingual transfer results.

heading: 4.4.1 Data-level analysis
text: Regarding the data level, we calculate the subtoken overlap between languages. We measure the overlap between unique subtokens in HR-LR pairs:
o12 = S1 ∪ S2
S2 , (1)
where S1 is the set of unique subtokens of donor (HR) languages, and S2 is the set of unique target (LR) languages subtokens. In our experiments, we use the mT5 tokenizer. Here, we consider how the subtoken overlap between HR and LR languages relates to the model’s performance in LR languages after continued pretraining in donor languages.

heading: 4.4.2 Linguistic-level analysis
text: We investigate language similarity by leveraging their typological characteristics. According to previous works, we consider WALS features. We deliberately avoid relying on GramBank (Skirgård et al., 2023) and lang2vec (Littell et al., 2017). GramBank, despite its extensive data coverage, offers features that are quite specific and narrow in scope, resulting in a heterogeneous and uninformative set for our purposes. On the other hand, lang2vec represents each feature from the WALS as one-hot encoding vectors, which increases the number of features. This might affect the outcomes of statistical tests due to the interdependence of many of these features. To interpret the results, we employ the Logistic regression model (Hosmer and Lemeshow, 2000) to obtain coefficients associated with input features. These coefficients serve
as indicators of the strength in the relationship between input features and target variables learned during model training. This analysis enables us to assess the importance of various language features in achieving successful transfer learning results.
To mitigate biases in absolute perplexity values, we use binary targets to indicate if perplexity decreases (1) or remains unchanged (0) after continued pretraining. Each data point in our training dataset is represented as a binary vector, where every element signifies whether both languages share the same value for a specific linguistic feature (1) or not (0). Finally, we consider the regression coefficients to identify which typological characteristics should be shared between donor and target languages for successful cross-lingual transfer.

heading: 4.5 Downstream evaluation

heading: 4.5.1 POS tagging task
text: We consider the POS tagging task since it is one of the few tasks with annotated data available for the LR languages. Specifically, we utilize datasets from UD treebanks. However, only 6 out of the 31 LR languages have data available: Bambara, Bhojpuri, Cantonese, Coptic, Guarani, Komi-Zyryan.
To evaluate the cross-lingual transfer performance of different HR-LR pairs in POS tagging, we train logistic regression on top of mT5 embeddings. Our training and validation data come from a donor HR language’s training and validation sets extracted from the UD corpus. Afterward, we assess the model’s performance on target LR languages’ test sets taken from their UD corpora.

heading: 4.5.2 Machine translation task
text: Additionally, we aim to experiment with another downstream task – Machine Translation (MT). Here, we use data from the NLLB project (NLLB Team et al., 2022; Bapna et al., 2022) as the only open-source dataset with MT data for extremely LR languages that we consider. This dataset contains parallel corpora for numerous language pairs, but in the case of LR languages, we see that only a few of them contain parallel HR-LR corpora, and only for two HR languages, such as English and Afrikaans, there are HR-LR datasets: 8 pairs for English and 7 pairs for Afrikaans.
While evaluating the MT setup, we aim to explore how cross-lingual transfer impacts the model’s performance on different HR-LR pairs. To do this, we follow a series of steps. First, we conduct separate experiments for each HR-LR pair by
training the out-of-the-box model in the MT setup and evaluating its performance afterward. This allows us to measure the performance before the cross-lingual transfer. Then, we select the bestperforming model checkpoints from previous experiments with continued pretraining on different HR donors and repeat training and evaluation on MT data using these selected checkpoints. Finally, we compare the model’s performance on the MT task before and after cross-lingual transfer across various HR-LR pairs.
To ensure consistency in the obtained results, we intend to use identical test sets across all experiments with specific HR-LR pairs. This enables us to compare and analyze the impact of cross-lingual transfer on the MT results more accurately. Also, following the approach we do in Section 4.5.1, we restrict the training data to match the number of tokens available in the least-resourced language pair to ensure a fair comparison.

heading: 5 Results and Analysis

heading: 5.1 General cross-lingual transfer results
text: In Table 1, you can see the comparison of the zeroshot results with the best results after continued pretraining on HR donors. Across 16 out of 31 LR languages, continued pretraining resulted in diminished perplexity scores. In this context, «best» refers to the lowest perplexity score attained among
all iterations of continued pretraining. We also examine the most effective HR donors. Figure 4 illustrates relative perplexity scores between zeroshot results and results after continued pretraining across the most effective HR-LR pairs (refer to Appendix A.1 and Appendix A.2 for results for all 4898 HR-LR pairs). After pretraining on Slovene, the model demonstrates lower perplexity for 14 LR languages. Similar results are observed for Afrikaans, also with 14 LR languages, Lithuanian with 12, and French with 11.
As well as Turc et al. (2021), we observe that English may not be the optimal language for crosslingual transfer. In our experiments, Afrikaans and Slovene show the best performance in crosslingual transfer for extremely low-resource languages. Thus, we consider them as «super-donors» in the scope of our experiments. There are also instances where such languages as Guaraní and Coptic exhibit universal target characteristics after pretraining on various HR languages.

heading: 5.2 Correlation with subtoken overlap
text: To assess how the overlap of subtokens in data affects the performance of different HR-LR language pairs, we calculate the Pearson correlation coefficient between these data features and the results of cross-lingual transfer for these pairs.
We observe a moderate correlation between subtoken overlap and ∆perplexity (rstat = −0.33,
pvalue < 0.01), where ∆perplexity is a difference between results before and after continued pretraining. This indicates that as the degree of subtoken overlap grows, we observe a decrease in perplexity on LR languages (see Figure 2 for the distribution of different pairs in such axes).

heading: 5.3 Interpretation using linguistic features
text: We also utilize linguistic features to interpret crosslingual transfer results. To maintain the validity of our findings, we exclude features not annotated in at least half of the considered languages. Thus, we have only 21 out of 194 WALS features for analysis; 12 are specifically related to word order, and the rest to morphology. It is important to note that the absence of annotation in WALS may lead to possible gaps in our analysis. Thus, some crucial factors may be missed. We utilize these features for training the logistic regression model (see Appendix 3 for regression coefficients of 21 linguistic features we relied on).
Surprisingly, the genealogical family feature has a negative coefficient, suggesting that the model performs better when the donor and target languages are unrelated. At the same time, positive coefficients are observed for similar morphological features (e.g., prefixing vs. suffixing), indicating whether features like Tense or Number tend to be expressed with prefixes or suffixes. The word order typically does not play an important role, as only the order of verb and object appears to be significant.

heading: 5.4 Downstream evaluation results

heading: 5.4.1 POS tagging results
text: In this downstream evaluation, we investigate whether continued pretraining can help in POS tagging experiments. Here, for each LR language, we compare how well models trained on the best donors from the MLM setup perform against models trained in three randomly chosen languages. We want to determine if training on the best language yields better results for POS tagging than training on random ones. As described in 4.5.1, we train logistic regression using word embeddings to identify part-of-speech. If a word consists of multiple tokens, we use their average embedding. Additionally, we limit the training data to the number of tokens available in the least-resourced donor language for fair comparison. We evaluate performance using both Accuracy and F1-score metrics.
Table 2 shows the results for 6 LR languages with available data. When trained on the best HR donors, such as Bambara, Bhojpuri, Guarani, Komi-Zyryan, the model achieves the best performance in at least one metric. However, for Cantonese and Coptic, the best donors from MLM experiments do not result in the highest performance. In Figure 5, you can see the heatmap of POS tagging results for all considered HR languages in the case of the aforementioned 6 LR languages.

heading: 5.4.2 Machine translation results
text: In contrast to the POS tagging, our data availability here is significantly limited. As mentioned in Section 4.5.2, we have a substantial amount of annotated data only for Afrikaans and English, totaling 7 and 8 HR-LR pairs, respectively. Based on the general cross-lingual transfer results (Section 5.1), we observe that Afrikaans tends to perform better as a super-donor, while English does not show satisfactory results in that regard. Therefore, we decide to proceed with evaluation in Machine Translation setup using data of Afrikaans only. As supported by Figure 4, we limit our scope only to the top-performing HR donors.
In Figure 6, you can see the results of the MT setup where we perform fine-tuning on 7 AfrikaansLR pairs. We report the comparisons using the ∆chrf metric, which indicates the difference in results with and without transfer learning on different HR languages. You can see that crosslingual transfer significantly boosts MT performance in evaluating Afrikaans-Sesotho, AfrikaansSwati, and Afrikaans-Tsonga. However, there is no improvement in the evaluation of AfrikaansChichewa. When looking at absolute values, crosslingual transfer experiments demonstrated the most significant improvement in performance when applied to Afrikaans-Akan and Afrikaans-Sesotho pairs, with an increase of more than 0.2 in the chrf.

heading: 5.5 Super donors and super recipients
text: Surprisingly, as Figures 4,5,6 (but much more apparent from the heatmaps presented in the Appendix A.2) corresponding to the MLM, POS, and MT tasks respectively, show, a fraction of LR languages tend to be super recipients, benefiting un-
conditionally from all other HR languages. Additionally, some HR languages tend to be super donors, benefiting all languages unconditionally, regardless of the donor’s or recipient’s linguistic characteristics. Furthermore, the sets of such super donors and super recipients do not tend to generalize completely across tasks (MLM, POS, MT).

heading: 5.6 Further discussion
text: Our experiments demonstrate that transferring knowledge from HR languages during continued pretraining can enhance the performance of the mT5 model in the MLM setup across various LR languages. Subsequent downstream evaluation also exhibits such improvement, particularly in the case of the POS tagging and Machine Translation tasks.
The analysis of the MLM experiments reveals that most word order features have no significant impact. HR-LR language pairs from the same families correlate negatively with the results; however, this correlation shifts to positive when they share the same morphological feature as affix. Additionally, a higher degree of overlap between subtokens in languages tends to yield better performance in cross-lingual transfer. Meanwhile, other characteristics show no significant correlation with the results from MLM experiments. At the language level, Afrikaans and Slovene are determined as the best donor languages, and continued pretraining on them tends to better results across most LR languages examined in this study. It aligns with the findings of Turc et al. (2021), who identified a different pair of Germanic and Slavic languages, German and Russian, as the best donors. Most languages that exhibit promising results as donors belong to the Indo-European language family, specif-
Ba m
ba ra
Bh oj
pu ri
Ca nt
on es e Co pt ic
Gu ar
an i
Ko m
i-Z yr
ya n
LR languages
Afrikaans Arabic Armenian Chinese Danish Dutch
Estonian Faroese French
German Hindi Irish
Italian Lithuanian
Persian Polish Russian Slovak Slovene Urdu
HR la
ng ua
ge s
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
F1 -w
ei gh
te d
Figure 5: POS tagging results for 6 LR languages with available data in Universal Dependencies after training on HR donors from Table 2.
ically falling under the classification of Standard Average European (SAE) languages (Haspelmath, 2008). However, languages with the best results are peripheral members of the SAE continuum, i.e., have only some characteristics of SAE languages.
In downstream evaluation, the findings from POS tagging demonstrate that employing optimal donor languages during pretraining outperforms pretraining on randomly selected languages in most cases. In Machine Translation, our investigation focuses on a particular pipeline for translation from Afrikaans to various LR. Consequently, we can conclude that pretraining on well-performing HR donors from the MLM step contributes to enhanced translation performance, particularly for extremely low-resource languages that we consider.

heading: 6 Conclusion
text: In this work, we present extensive experiments on cross-lingual transfer for HR-LR language pairs, leveraging HR languages for continued pretraining of the mT5. We observe that the performance of cross-lingual transfer significantly correlates with morphological features. Additionally, a higher degree of overlap between subtokens can contribute to better performance. Meanwhile, other characteristics do not significantly correlate with cross-lingual transfer results. We also observe improved downstream evaluation results, showing successful POS tagging and MT tasks performance. Finally, some LR languages tend to be super recipients, namely benefiting from all languages, and some HR languages tend to be super donors namely benefiting all languages with no apparent linguistic relation between donor and recipient.

heading: A Appendix
text: A.1 The most efficient high-resource languages
Table 3 lists the most effective HR languages for cross-lingual transfer with the corresponding LR languages with decreased perplexity.
A.2 Cross-lingual transfer results Figures 7 and 8 depict the heatmaps of various LR-HR language pairs, with corresponding colors indicating perplexity scores measured in the MLM setup. These scores represent the average values across 5 runs of continued pretraining on each HR language. Here, you can see 158 high-resource and 31 low-resource languages.
A.3 Statistics of the collected corpus As we discussed in Section 3, we gathered the corpus of textual data for 189 languages. In order to divide language by high and low resource, we first calculated some statistics for each language. In Table 4, you can see official names, number of symbols and tokens for each language. We used the tokenizer from the mT5 model for text tokenization.
Name N_tokens, kk N_symbols, kk Name N_tokens, kk N_symbols, kk Abaza 1.33 2.35 Buriat 172.16 344.66 Acehnese 1.47 3.09 Choctaw 0.001 0.002 Arabic (Egyptian) 157.47 291.76 Cebuano 1365.37 3319.73 Afrikaans 46.5 126.33 Chamorro 0.06 0.13 Akan 0.33 0.62 Chechen 378.15 477.3 Albanian 0.002 0.005 Cherokee 0.17 0.22 Amharic 5.37 6.85 Chukchi 4.08 5.12 Arabic (Moroccan) 1.1 2.07 Chuvash 277.48 442.77 Arabic (Modern Standard) 1.83 1.83 Chichewa 0.28 0.75 Apurinã 0.002 0.0035 Cantonese 0.02 0.02 Archi 0.0012 0.0019 Coptic 0.1 0.13 Arabic (Lebanese) 0.0015 0.0026 Crimean Tatar 1.24 2.6 Armenian (Eastern) 0.12 0.26 Cornish 0.9 1.88 Armenian (Western) 0.09 0.17 Catalan 463.18 1168.29 Armenian (Iranian) 212.94 569.39 Chatino (Yaitepec) 1.21 3.05 Adyghe (Shapsugh) 3.32 5.58 Cheyenne 0.06 0.1 Altai (Southern) 2.46 4.6 Czech 336.81 819.13 Assamese 11.18 18.87 Dagbani 0.28 0.55 Asturian 117.6 304.59 Dogri 0.006 0.009 Atayal 0.71 1.35 Dhivehi 4.35 5.77 Atikamekw 0.33 0.71 Dargwa 11.64 23.4 Avar 5.73 10.82 Danish 0.52 1.46 Awadhi 0.57 1.04 Dutch 598 1675.76 Aymara (Central) 0.92 1.81 Dutch (Zeeuws) 1.43 3.12 Azerbaijani 90.5 230.74 English 7920.93 24002.62 Azari (Iranian) 39.73 74.9 Estonian 97.8 265.66 Balinese 2.04 4.87 Even 0 0.01 Bambara 0.17 0.31 Ewe 0.085 0.153 Beja 0.003 0.003 Faroese 4.17 9.4 Bengali 28.14 56.44 Finnish 243.97 715.36 Bhojpuri 0.01 0.02 Frisian (North) 2.74 5.65 Bikol 3.39 8.63 French 1541.64 4046.63 Belorussian 168.71 467.04 Frisian 0.007 0.016 Breton 21.79 43.06 Frisian (Western) 29.12 69.47 Burmese 26.12 64.79 Fuzhou 1.95 2.86 Bashkir 58.9 122.15 Gaelic (Scots) 4.52 9.25 Bislama 0.13 0.28 Gagauz 0.43 1.02 Basque 12.73 35.51 Georgian 65.7 149.49 Bugis 0.98 2.05 German 6.94 21.3 Bulgarian 147.44 367.2 Guianese French Creole 0.53 1.06
Abaza Afrikaans
Arabic (Moroccan) Arabic (Modern Standard)
Armenian (Iranian) Adyghe (Shapsugh)
Altai (Southern) Assamese
Asturian Atayal
Avar Azerbaijani
Balinese Bengali
Bikol Belorussian
Breton Burmese
Bashkir Basque Bulgarian Buriat
Chechen Chukchi Chuvash Cornish Catalan
Czech Dhivehi Dargwa Danish Dutch
Dutch (Zeeuws) English
Estonian Faroese Finnish French
Frisian (Western) Gaelic (Scots)
Georgian German
Guianese French Creole Galician
Greek (Modern) German (Ripuarian) German (Timisoara) Gujarati German (Viennese)
Hausa Hawaiian
Hebrew (Modern) Hindi
Hungarian Icelandic
Igbo Ingush
Indonesian (Jakarta) Irish (Munster)
Italian Italian (Genoa)
Javanese Japanese
Kabardian Kazakh Kabiyé Kirghiz Khakas Khmer Kinyarwanda Karakalpak
Kannada Komi-Permyak Karachay-Balkar Kurdish (Central)
Kumyk Lak Lao Latvian
HR la
ng ua
ge s
0
20
40
60
80
100
120
140
Pe rp
le xi
ty
Luganda Ladin
Lezgian Low German
Lithuanian Ladino Luxemburgeois Macedonian
Mingrelian Marathi
Mongol (Khamnigan) Maltese
Malay Mari (Meadow) Mordvin (Moksha) Mansi Manx
Mordvin (Erzya) Mon
Malayalam Nepali
Nias Norwegian Nenets (Tundra) Nivkh (South Sakhalin)
Newar (Dolakha) Oirat Oriya
Panjabi Papiamentu
Polish Portuguese
Provençal Romanian
Romansch (Sursilvan) Russian
Rutul Sindhi Seediq Shan
Slovene Sinhala
Saami (Northern) Solon
Somali Sorbian (Upper)
Spanish Sardinian Sorbian (Lower) Santali
Sundanese Slovincian
Slovak Swahili Swedish Tashlhiyt
Telugu Thai Turkmen Tibetan (Modern Literary)
Tamil (Spoken) Tswana
Tetun Tulu Turkish Tuvan Tatar
Udmurt Ukrainian
Uyghur Veps Vietnamese Welsh Wolof Xhosa Yakut
HR la
ng ua
ge s
0
20
40
60
80
100
120
140
Pe rp
le xi
ty

heading: Future Work
text: We think the encoder-decoder architecture is more flexible and has more possible applications for future works than only encoder or decoder-based models. Due to its multitask finetuning, we decided not to use its another version, mT0 (Muennighoff et al. , 2023). Considering our lack of labeled data, exploring its multitask zeroshot performance is unnecessary here