Venue: NAACL 2025

# Taxi1500: A Dataset for Multilingual Text Classification in 1500 Languages

Chunlan Ma\({}^{1,2}\), Ayyoob Imani\({}^{1,2}\), Haotian Ye\({}^{1,2}\), Renhao Pei\({}^{1}\),

**Ehsaneddin Asgari\({}^{3}\), Hinrich Schutze\({}^{1,2}\)**

###### Abstract

While broad-coverage multilingual natural language processing tools have been developed, a significant portion of the world's over 7000 languages are still neglected. One reason is the lack of evaluation datasets that cover a diverse range of languages, particularly those that are low-resource or endangered. To address this gap, we present a large-scale text classification dataset encompassing 1504 languages many of which have otherwise limited or no annotated data. This dataset is constructed using parallel translations of the Bible. We develop relevant topics, annotate the English data through crowdsourcing and project these annotations onto other languages via aligned verses. We benchmark a range of existing multilingual models on this dataset. We make our dataset and code available to the public.1

Footnote 1: [https://github.com/cisnlp/Taxi1500](https://github.com/cisnlp/Taxi1500)

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)_, pages 414–439

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)_, pages 414–439

## 1 Introduction

Language inequality is a real issue in the world today as minority languages are under-represented and often excluded from language technologies Joshi et al. (2020). The lack of technological support for minority languages in communities around the globe has a significant impact on the experience of their users and is commonly a cause for virtual barriers such as the _digital divide_.2 Recent developments in language technologies have led to a surge in multilingual pre-trained language models (mPLMs), such as mBERT Devlin et al. (2019), XLM-R Conneau et al. (2020), and Glot500 Imani et al. (2023), and large language models LLMs like BLOOM Le Scao et al. (2023) and Aya Osttun et al. (2024). The lack of knowledge in low-resource languages often causes language technologies to overlook important features from typologically diverse languages Ponti et al. (2019). A key reason why many low-resource languages remain neglected is the scarcity of evaluation datasets. For example, mPLMs like mBERT and XLM-R are evaluated on much fewer languages than they are trained for, largely due to the limited availability of languages in most existing benchmark datasets.

As a solution, we propose a dataset that covers more than 1500 languages. We use translations of the Bible as our source and develop topics that are well generalized (so as to apply to many verses), but at the same time are not overly abstract. We obtain annotations for the English verses using crowdsourcing. Because the Bible is aligned at the verse level, we can easily project annotations from the English side to all other languages. To ensure the quality of our annotated data, we calculate the inter-annotator agreement using Krippendorff's \(\alpha\). In addition, we introduce a benchmark for four mPLMs and three LLMs. We present evaluation results using mBERT, XLM-R-Base, XLM-R-Large and Glot500 for all languages and LLaMA2-7B Touvron et al. (2023), Mistral-7B Jiang et al. (2023), and BLOOM (560m, 1B, 3B and 7B) for 64 selected languages in our dataset. Glot500 demonstrates better multilingual capabilities, attributed to its larger number of languages in the pretraining data. Moreover, the evaluation of LLMs reveals that their performance (based on a few low-resource prompts) is comparable to fine-tuned mLMs.

## 2 Related Works

To date, most datasets that can be used for multilingual task evaluation Pan et al. (2017); Conneau et al. (2018); De Marneffe et al. (2021); Adelani et al. (2021, 2024); Adebara et al. (2022) cover no more than a few hundred languages, a small number compared to the world's 7000 languages. In current NLP research, parallel corpora play a crucial role as they serve as cross-lingual bridges, enabling the processing and understanding of less known languages through other languages. In this study, we employ translations of the Bible as the source of

parallel data, utilizing both the Parallel Bible corpus (Mayer and Cysouw, 2014), covering 1304 languages, as well as 1000Langs,3 Bible translations collected from multiple Bible websites, resulting in a total coverage of 1504 languages.

Footnote 3: [https://github.com/ehsanasgari/1000Langs](https://github.com/ehsanasgari/1000Langs)

## 3 Dataset Creation

Since many low-resource languages only have a translated New Testament, we use verses from the New Testament to build our dataset. In the initial annotation phase, we gather topics using Latent Dirichlet Allocation (LDA),4 online preaching websites with topics of Bible verses,5 and insights from linguists. We then utilize Amazon Mechanical Turk (MTurk)6 for crowdsourcing to assess the quality of the selected topics. We conduct seven rounds of topic selection and show the details in Table 7 in Appendix E. Ultimately, we choose the six topics with the most verses: _recommendation_, _fain_, _description_, \(sin\), _grace_, and _violence_. Following this, three annotators extract verses for each of the six topics, selecting only those where at least two annotators agree. We remove verses that cover multiple topics or are not relevant to any topic as such noise complicates annotation and may confuse crowdsourcing annotators. This curation reduces annotation cost. We then submit the resulting 1,077 verses to Amazon MTurk, specifying the US as the annotators' location. Each verse is annotated ten times, with final labels determined by majority voting.

Footnote 4: [https://tinyurl.com/5fgj5vz](https://tinyurl.com/5fgj5vz)

Footnote 5: [https://www.georgeho.org/lda-sucks/](https://www.georgeho.org/lda-sucks/)

Footnote 6: www.mturk.com

We assume annotation quality issues may arise if 1) the task is confusing, or 2) the worker lacks care or attention. We provide detailed guidelines and examples along with the task. All workers must also pass a qualification test to ensure they fully understand the task. For quality control, we implement a performance threshold. We create "pseudo gold standard" data based on majority votes from all annotators and calculate each worker's macro F1 score. If that score is below 0.40 for a worker, their annotations are rejected, and the verses are republished for re-annotation.

We use Krippendorff's \(\alpha\) (K-\(\alpha\)) to compute interannotator agreement. K-\(\alpha\) is chosen for its ability to handle missing annotations in the dataset. This is important because each worker only annotates a subset of the verses. Table 2 shows K-\(\alpha\) values for different thresholds, i.e., the minimum votes for the majority label required for a verse to be accepted. We obtain K-\(\alpha=0.44\) on the entire dataset, which can be improved by raising the threshold of required votes. But as Figure 1 demonstrates, there is a clear tradeoff between the number of accepted verses and K-\(\alpha\), and increasing K-\(\alpha\) reduces the size of the dataset significantly. Furthermore, a slightly suboptimal K-\(\alpha\) value is predictable considering that the topics of our task are rather subjective due to the highly specialized domain. Also, as Price et al. (2020) points out, a low K-\(\alpha\) does not necessarily signify low data quality. We thus do not remove any data by raising the required number of votes but instead rely on our control measures (e.g., removing annotations by unreliable crowdworkers) to ensure data quality.

## 4 Dataset

The final dataset we obtain consists of verses categorized into six topics: _faith_, _grace_, _sin_, _violence_, _description_, and _recommendation_. Table 1 shows an overview of the topics with one example for each, as well as the number of verses of each topic in the English dataset. Class _violence_, with 59 instances, is the smallest class and _recommendation_, with 281, is the biggest class. Since some languages have incomplete translations of the New Testament that do not contain all of the 1077 verses, we exclude languages where the total number of annotated verses is fewer than 900. This leaves us with 1504 languages from 113 language families which are spread across the globe.7

Figure 1: Tradeoff between K-\(\alpha\) and the number of verses. Each dot in the plot stands for a threshold of the required minimum votes \(\in\{3,4,5,6,7,8,9\}\) for a verse to be accepted.

 



## 5 Benchmarking

To illustrate its utility, we use Taxi1500 to evaluate four pre-trained multilingual models: mBERT, XLM-R-Base, XLM-R-Large, and Glot500, and three LLMs: LLAMA2, BLOOM, and Mistral using a selection of 64 languages from Taxi1500. For a fair comparison, we split languages in our dataset into three subsets, namely head languages, Glot500-only languages, and tail languages. Head languages are languages that are in the pre-training data of all four models. Glot500-only languages are languages that are only in the pre-training data of Glot500. Tail languages are languages that are not in the pre-training data of any model. Details of the setup are provided in Appendix A.

### Experiment Setup

Our experiments are divided into three settings: zero-shot transfer, in-language classification, and three-shot prompting for LLMs. The dataset for each of the 1,504 languages is split into training, development, and test sets with an 80/10/10 ratio.

In the in-language classification setting, we use the target language data for fine-tuning and testing. In zero-shot transfer, we use English data for fine-tuning and test on the target language test set. For in-language experiments on languages other than English, we furthermore vary the training set size \(\in\{50,100,200,400,600,860\}\), where 860 corresponds to the full training set, in order to test: 1) the effects of different amounts of training samples and 2) the minimal number of training samples required to achieve acceptable classification results.

### Results

**Zero-shot transfer.** We conduct Bag-of-Words (BOW) classification with our dataset as a baseline and present the results in Appendix I. The results revealed extremely low accuracy for BOW: most of the results are less than 0.10, indicating that to classify verses in our dataset correctly, the models must have access to a good semantic representation (which BOW does not seem to provide).

In Figure 2, we show the results for 1504 languages, divided into three sets: head languages (left), Glot500-only languages (middle), and tail languages (right). On head languages, Glot500, XLM-R-B, and XLM-R-L have 68, 65, and 69 languages within the high F1 range (0.4-0.8), respectively, while mBERT only has 26 languages within this range, indicating its worse performance. This might be explained by a smaller amount of pre-training data of mBERT compared with the other three models. On Glot500-only languages, Glot500 outperforms the other three models with 117 languages in the range of 0.2-0.8, whereas the other three models have fewer than 30 languages within this range. Because Glot500-only languages are in the pre-training data of Glot500, we expect Glot500 to achieve better results on these languages. On tail languages, Glot500 outperforms the other three models slightly with around 100 fewer languages in the range of 0-0.2. The reason might be that a larger number of pre-training languages contributes to higher performance for other tail languages from the same family. The zero-shot transfer results indicate that Taxi1500 can effectively demonstrate better performance for models pretrained using more languages.

**In-language training.** To investigate the influence of the training set size, we conduct in-language experiments with 20 languages (10 head and 10 tail languages), which are selected to repre

\begin{table}
\begin{tabular}{l l r} \hline \hline class & example & num. verses \\ \hline recommendation & If you love me, you will observe my commandments & 281 \\ faith & Most truly I say to you, whoever believes has everlasting life & 260 \\ description & There was a man of the Pharisees named Nicodemus, a ruler of the Jews & 184 \\ sin & Jesus answered: “I do not have a demon, but I honor my Father, and you dishonor me & 153 \\ grace & The Father loses the Son and has given all things into his hand & 140 \\ violence & He put James the brother of John to death by the sword & 59 \\ \hline \end{tabular}
\end{table}
Table 1: An overview of the six classes of our dataset, with one example verse and the number of verses in the crowdsourced English dataset for each class.

\begin{table}
\begin{tabular}{c|r c c c c c} \hline vote \(\geq\) & 3 & 4 & 5 & 6 & 7 & 8 & 9 \\ \hline num. verses & 1077 & 1055 & 941 & 755 & 563 & 388 & 233 \\ K-\(\alpha\) & 0.44 & 0.44 & 0.48 & 0.55 & 0.63 & 0.73 & 0.83 \\ \hline \end{tabular}
\end{table}
Table 2: The K-\(\alpha\) value increases as we specify a higher threshold for the minimum number of votes of the majority topic.



sent a diverse range of languages from 13 families. Tables 3 and 4 show the results of zero-shot transfer and in-language experiments using mBERT and XLM-R-B for the selected languages. As expected,

\begin{table}
\begin{tabular}{c|l|r r r r r||l l|l l l l l} \hline head & zero & \multicolumn{6}{c|}{in-language training} & \multicolumn{2}{c|}{tail} & \multicolumn{2}{c|}{zero} & \multicolumn{6}{c}{in-language training} \\ lang. & shot & 50 & 100 & 200 & 400 & 600 & 860 & lamp. & shot & 50 & 100 & 200 & 400 & 600 & 860 \\ \hline eng & 0.65 & 0.35 & 0.33 & 0.53 & 0.49 & 0.51 & 0.71 & chr & 0.09 & 0.15 & 0.20 & 0.15 & 0.24 & 0.21 & 0.28 \\ deu & 0.52 & 0.16 & 0.18 & 0.43 & 0.49 & 0.52 & 0.51 & gag & 0.33 & 0.17 & 0.13 & 0.14 & 0.45 & 0.32 & 0.54 \\ heb & 0.15 & 0.10 & 0.13 & 0.18 & 0.16 & 0.33 & 0.35 & hix & 0.06 & 0.18 & 0.17 & 0.22 & 0.3 & 0.43 & 0.49 \\ jpn & 0.62 & 0.25 & 0.39 & 0.53 & 0.57 & 0.61 & 0.68 & htt & 0.05 & 0.14 & 0.07 & 0.19 & 0.40 & 0.20 & 0.50 \\ kaz & 0.57 & 0.23 & 0.35 & 0.47 & 0.41 & 0.55 & 0.56 & kpv & 0.09 & 0.09 & 0.21 & 0.23 & 0.41 & 0.38 & 0.53 \\ kor & 0.63 & 0.35 & 0.55 & 0.58 & 0.65 & 0.53 & 0.70 & kum & 0.13 & 0.13 & 0.17 & 0.22 & 0.27 & 0.37 & 0.45 \\ eus & 0.26 & 0.09 & 0.26 & 0.25 & 0.34 & 0.37 & 0.34 & luc & 0.11 & 0.12 & 0.11 & 0.30 & 0.30 & 0.39 & 0.39 \\ mal & 0.42 & 0.18 & 0.30 & 0.21 & 0.45 & 0.45 & 0.64 & mag & 0.38 & 0.11 & 0.23 & 0.41 & 0.48 & 0.38 & 0.51 \\ pes & 0.66 & 0.17 & 0.55 & 0.47 & 0.65 & 0.64 & 0.71 & mbd & 0.11 & 0.18 & 0.14 & 0.25 & 0.30 & 0.30 & 0.38 \\ zho & 0.63 & 0.33 & 0.49 & 0.52 & 0.45 & 0.51 & 0.68 & npl & 0.05 & 0.14 & 0.08 & 0.25 & 0.41 & 0.41 & 0.43 \\ \hline
**avg.** & 0.51 & 0.22 & 0.35 & 0.42 & 0.47 & 0.50 & 0.59 & **avg.** & 0.14 & 0.14 & 0.15 & 0.24 & 0.36 & 0.34 & 0.45 \\ \hline \end{tabular}
\end{table}
Table 3: Results of zero-shot transfer and in-language fine-tuning experiments using XLM-R-Base for 20 selected languages, 10 head (left): English, German, Hebrew, Japanese, Kazakh, Korean, Basque, Malayalam, Persian and Chinese, and 10 tail (right): Cherokee, Gagauz, Hikkaryana, Nga La, Komi-Zyrian, Kumyk, Aringa, Magahi, Dibabawon Manobo and Southeastern Pueblia Nahuatul. The numbers in the table header indicate the size of target language training data: 860 means the full training set.

Figure 2: Zero shot transfer learning: head languages (left), Glot500-only languages (middle), and tail languages (right). X-axis is the number of languages, y-axis presents four models. We split F1 scores into four ranges: 0-0.2, 0.2-0.4, 0.4-0.6 and 0.6-0.8.

 

the in-language performance improves when the training set becomes larger. Interestingly, zero-shot transfer performance of head languages is comparable to in-language setting with 100 samples for mBERT and with 400 samples for XLM-R-B, which indicates that models with more parameters may require more in-language data to reach a comparable level with zero-shot transfer performance. Moreover, the zero-shot transfer results on both models show that head languages consistently outperform tail languages, which reflects both models' better generalization capability on languages in their pretraining data.

**Evaluation of LLMs.** To explore the capability of LLMs, we conduct three-shot in-context learning with 64 selected languages from different language families on six LLMs, namely LLaMA2-7B, Mistral-7B, and BLOOM (560m, 1B, 3B and 7B). We report the results in Appendix H. In Table 5, we show the average score of 64 languages. Notably, Mistral-7B achieves the highest average performance with a score of 0.55, surpassing both LLaMA2-7B, which scores 0.45, and BLOOM at various sizes. BLOOM's performance varies slightly across model sizes, with the 1B version yielding the highest score (0.50) among BLOOM models, while the 7B version underperforms at 0.46. These results suggest that Mistral-7B may be more effective in handling the Taxi1500 task. Overall, each LLM achieves performance comparable to the mPLMs on in-language classification tasks trained on a full training set of 860 verses. This result could be interpreted as LLMs having multilingual capabilities similar to mPLMs (even though the LLM setup requires no finetuning training data). But of course this experiment was only conducted on 64 languages. It remains to be verified that it generalizes to low-resource languages in general.

## 6 Conclusion

In this paper, we propose a text classification dataset consisting of 1504 languages by annotating English Bible verses through crowdsourcing and projecting the labels to other languages with parallel data. We benchmark several widely used multilingual language models and LLMs using our dataset. The results demonstrate that Taxi1500 can effectively evaluate multilingual capabilities across different models.

## 7 Limitations

While the high degree of parallelism in the PBC makes it a valuable tool for massively multilingual application, such as the building of our evaluation dataset, it is not perfect. One limitation is the specific domain of the Bible being a religious text, which often does not reflect real world usages. The specific religious context additionally makes it possible that keywords are exploited. Also, we are restricted to the New Testament as a large quantity of languages do not have a translated Old Testament in the PBC. Given that some extremely low-resource languages do not have complete translations, the actual number of available verses varies for each

\begin{table}
\begin{tabular}{l|c c c c c c||c|c c c c c c} \hline head & zero & \multicolumn{9}{c||}{in-language training} & tail & zero & \multicolumn{9}{c}{in-language training} \\ lang. & shot & 50 & 100 & 200 & 400 & 600 & 860 & lang. & shot & 50 & 100 & 200 & 400 & 600 & 860 \\ \hline eng & 0.71 & 0.35 & 0.33 & 0.53 & 0.49 & 0.51 & 0.71 & chr & 0.05 & 0.24 & 0.21 & 0.29 & 0.35 & 0.30 & 0.35 \\ deu & 0.39 & 0.20 & 0.13 & 0.34 & 0.42 & 0.44 & 0.52 & gag & 0.12 & 0.21 & 0.29 & 0.35 & 0.39 & 0.45 & 0.38 \\ heb & 0.36 & 0.24 & 0.24 & 0.36 & 0.33 & 0.38 & d.il & hix & 0.07 & 0.30 & 0.27 & 0.35 & 0.35 & 0.39 & 0.41 \\ jpn & 0.39 & 0.37 & 0.40 & 0.32 & 0.49 & 0.63 & 0.66 & hit & 0.08 & 0.16 & 0.25 & 0.33 & 0.34 & 0.44 & 0.49 \\ kaz & 0.29 & 0.30 & 0.36 & 0.38 & 0.50 & 0.48 & qkpv & 0.08 & 0.19 & 0.24 & 0.45 & 0.41 & 0.39 & 0.46 \\ kor & 0.41 & 0.36 & 0.36 & 0.45 & 0.56 & 0.50 & 0.60 & lum & 0.14 & 0.28 & 0.27 & 0.35 & 0.37 & 0.42 & 0.46 \\ eus & 0.17 & 0.15 & 0.12 & 0.31 & 0.44 & 0.46 & 0.43 & luc & 0.08 & 0.27 & 0.23 & 0.46 & 0.41 & 0.45 & 0.35 \\ mal & 0.22 & 0.32 & 0.31 & 0.41 & 0.40 & 0.46 & mag & 0.19 & 0.14 & 0.38 & 0.38 & 0.37 & 0.43 & 0.34 \\ pes & 0.43 & 0.30 & 0.36 & 0.55 & 0.53 & 0.52 & 0.56 & mbd & 0.08 & 0.18 & 0.33 & 0.36 & 0.36 & 0.39 & 0.42 \\ zho & 0.36 & 0.24 & 0.46 & 0.47 & 0.62 & 0.54 & 0.59 & npl & 0.06 & 0.21 & 0.30 & 0.38 & 0.39 & 0.40 & 0.40 \\ \hline
**avg.** & 0.37 & 0.28 & 0.31 & 0.41 & 0.48 & 0.49 & 0.54 & **avg.** & 0.10 & 0.22 & 0.28 & 0.37 & 0.37 & 0.41 & 0.41 \\ \hline \end{tabular}
\end{table}
Table 4: Results of zero-shot transfer and in-language fine-tuning experiments using mBERT for 20 selected languages, 10 head (left): English, German, Hebrew, Japanese, Kazakh, Korean, Basque, Malayalam, Persian and Chinese, and 10 tail (right): Cherokee, Gagauz, Hikkaryana, Nga La, Komi-Zyrian, Kumyk, Arringa, Magaihi, Dibabawon Manobo and Southeastern Puebla Nahutal. The numbers in the table header indicate the size of target language training data: 860 means the full training set.

\begin{table}
\begin{tabular}{l|r|c|c|c c c} \hline Model & LLMaAZ & Mistral & \multicolumn{5}{c}{BLOOM} \\ Size & 7B & 7B & 560M & 1B & 3B & 7B \\ Avg. Acc & 0.45 & 0.55 & 0.46 & 0.50 & 0.48 & 0.48 \\ \hline \end{tabular}
\end{table}
Table 5: Performance of three LLMs of various sizes.

 

language. However, since the Bible is by far the most translated book in the world, we regard it as a suitable resource for an initiative to build highly parallel data like ours.

## 8 Ethics Statement

In this work, we introduce a new multilingual text classification dataset based on the Parallel Bible Corpus. The data is partially annotated by workers from the Amazon mTurk platform, who are rewarded fairly for their work ($0.2 per sentence). Our dataset contains Bible verses for which we estimate a low risk of tracing to specific individuals and are intended exclusively for the evaluation of NLP tasks concerning the supported languages. We therefore do not expect any ethical issues with our dataset.

Bird (2024) has argued that many low-resource languages (in particular, languages that are primarily used orally) do not benefit from NLP technology and may even be harmed, e.g., if social media companies' use of low-resource NLP technology results in younger speakers of a low-resource language spending more time on their devices and less time engaging with their community. We acknowledge that this is a real danger for some low-resource communities. We also believe that the benefits of NLP outweigh the risks for others, e.g., for Occitan. In general, this is an important question about the future direction of NLP research that goes beyond this paper.

## 9 Acknowledgements

This work was funded by the European Research Council (NonSequeToR, grant #740516) and by DFG (SCHU 2246/14-1). We appreciate Yihong Liu's suggestions for the revisions during the writing process of the paper. We also thank the anonymous reviewers for their constructive feedback.

## References

* Adebara et al. (2022) Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Inciarte. 2022. AfroLID: A neural language identification tool for African languages. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 1958-1981, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Adelani et al. (2024) David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassiljev, Jesujoba Alabi, Yanke Mao, Haonan Gao, and En-Shiun Lee. 2024. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 226-245, St. Julian's, Malta. Association for Computational Linguistics.
* Adelani et al. (2020) David Heoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constanting Lignos, Chester Palen-Michel, Happy Buzaqab, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamusudden H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anouluwapo, Catherine Gitau, Derguene Mbaye, Jesusjobas Alabi, Seid Muhie Yiamam, Tajuddeen Rabiu Gowadabe, Ignatius Ezeani, Rubung Andre Niyongabo, Jonathan Mukibii, Verrah Otenede, Iror Ofrie, Davis David, Samba Ngom, Tosin Acewumi, Paul Rayson, Mofetlouva Adeyemi, Gerald Murriuki, Emmanuel Anebi, Chiamak Chukwuneke, Nkirkuka Odu, Eric Peter Wairagala, Samuel Overinde, Clemencia Siro, Tobius Saul Batesea, Temilab Oloyede, Yvonne Wambui, Victor Akindole, Deborn Nabagereka, Maurice Katsumire, Ayodele Awokoya, Mohammade MBOUP, Dibora Grebreydannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelecchi Ogueji, Thierno Ibrahim DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengeree, and Salomey Osei. 2021. MasakhaNER: Named entity recognition for African languages. _Transactions of the Association for Computational Linguistics_, 9:1116-1131.
* Bird (2024) Steven Bird. 2024. Must NLP be extractive? In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 14915-14929, Bangkok, Thailand. Association for Computational Linguistics.
* Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
* De Marmeffe et al. (2021) Marie-Catherine De Marmeffe, Christopher D Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal dependencies. _Computational linguistics_, 47(2):255-308 

Jaco Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dufter et al. [2018] Philipp Dufter, Mengjie Zhao, Martin Schmitt, Alexander Fraser, and Hinrich Schutze. 2018. Embedding learning through multilingual concept induction. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1520-1530, Melbourne, Australia. Association for Computational Linguistics.
* Imani et al. [2023] Ayyoob Imani, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, Andre Martins, Francois Yvon, and Hinrich Schutze. 2023. Glot500: Scaling multilingual corpora and language models to 500 languages. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1082-1117, Toronto, Canada. Association for Computational Linguistics.
* Jiang et al. [2023] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devenda Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lelio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2023. Mistral 7b.
* Joshi et al. [2020] Pratik Joshi, Sebastian Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6282-6293, Online. Association for Computational Linguistics.
* Secao et al. [2023] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagne, Alexandra Sasha Luccioni, Francois Yvon, Matthias Galle, et al. 2023. Bloom: A 176b-parameter open-access multilingual language model.
* Lin et al. [2024] Peiqin Lin, Shaoxiong Ji, Jorg Tiedemann, Andre F. T. Martins, and Hinrich Schutze. 2024. Mala-500: Massive language adaptation of large language models.
* Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
* Mayer and Cysouw [2014] Thomas Mayer and Michael Cysouw. 2014. Creating a massively parallel Bible corpus. In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)_, pages 3158-3163, Reykjavik, Iceland. European Language Resources Association (ELRA).
* Pan et al. [2017] Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1946-1958, Vancouver, Canada. Association for Computational Linguistics.
* Ponti et al. [2019] Edoardo Maria Ponti, Helen O'Horan, Yevgen Berzak, Ivan Vulic, Roi Reichart, Thierrry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019. Modeling language variation and universals: A survey on typological linguistics for natural language processing. _Computational Linguistics_, 45(3):559-601.
* Price et al. [2020] Ilan Price, Jordan Gifford-Moore, Joy Flemming, SauI Musker, Maayan Roichman, Guillaume Sylvain, Nithum Thain, Lucas Dixon, and Jeffrey Sorensen. 2020. Six attributes of unhealthy conversations. In _Proceedings of the Fourth Workshop on Online Abuse and Humans_, pages 114-124, Online. Association for Computational Linguistics.
* Touvron et al. [2020] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guilhem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyn Fu, Brian Fuller, Cynthia Gao, Vedauni Gosawami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inanc, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Klloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihalyo, Pushkar Mishra, Igor Molbybog, Yixin Nie, Andrew Pointon, Jeremy Reinsteinen, Rashid Rungtai, Kalyan Saladi, Alan Scheiten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Purxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadour, Sharan Narang, Aurelien Rodriguez, Robert Stojmic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models.
* Ustun et al. [2024] Ahmet Ustun, Viraat Arayabumi, Zheng Yong, Wei-Yin Ko, Daniel D'souza, Gbemleke Onilude, Neel Brandari, Shivalia Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadace, Julia Kreutzer, and Sara Hooker. 2024. Aga model: An instruction finetuned open-access multilingual language model. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15894-15939, Bangokok, Thailand. Association for Computational Linguistics 

Experiment setup

We fine-tune four mPLMs using the training data for each setting. We use the AdamW optimizer with a learning rate of \(2e-5\) and a batch size \(\in\{16,32\}\), and report the best metrics. Training is stopped employing early stopping on the development data. All experiments can be performed on a single GeForce GTX 1080Ti GPU within a matter of minutes.

## Appendix B In language training results

In the in-language classification setting, we use the target language data for fine-tuning and testing. In zero-shot transfer, we use English data for fine-tuning and get predictions on the target language test set. For in-language experiments on languages other than English, we furthermore vary the training set size \(\in\{50,100,200,400,600,860\}\), where 860 corresponds to the full training set.

## Appendix C Analysis by Language Family

In Figures 3 and 4, we present zero-shot transfer and in-language results of all languages based on their families on XLM-R-Base and Glot500. For almost all families, the performance on head languages is significantly higher than that of Glot500-only and tail languages. The Indo-European family outperforms other language families not only on head languages but also on Glot500-only and tail languages. We suppose the reason is that the four evaluated models are pre-trained with more Indo-European languages, which increases the performance of this family. We also notice that XLM-R-Large tends to perform worse than the other three models on most languages. We think this could be due to its larger number of parameters, which makes it prone to overfitting on our small dataset. Interestingly, by comparing zero-shot transfer and in-language results of XLM-R-Base, we find that languages that are extremely low-resource and use non-Latin scripts (e.g. Yawa-Saweru, Lengua-Mascoy, and Hmong-Mien) have significant performance increases (around 0.4) when they are trained with in-language data. This indicates that the four models do not perform as well on non-Latin scripts as on Latin scripts.

## Appendix D Annotation

Figure 5 shows a screenshot of the annotation interface. Workers select one label for each verse among six options. If they think one verse does not belong to any of them, the workers should classify this verse as _Other_.

## Appendix E Topics Design

We present our attempts to explore the classification task and the construction of possible categories. There are different classification tasks, for example, sentiment classification, intent classification, and topic classification. At the beginning, we attempt to implement sentiment classification and split verses into three conventional categories: positive, neutral, and negative. However, most of the verses in the Bible do not indicate one absolute sentiment. Hence, we try intent classification yet also failed. We demonstrate this in more detail below.

### Sentiment Classification

First, we attempt to implement the simplest sentiment classification task. Dufter et al. (2018) classify a portion of the English verses in the PBC into a positive category and a negative category. Inspired by them, we initially try standard sentiment classification on the PBC with an improved method from Dufter et al. (2018). Precisely, in order to explore the possibility of using more categories, we divide verses in the Bible into positive, negative and neutral ones using the prepared sentiment RoBERTa model Liu et al. (2019) from Huggingface, which is fine-tuned on 5,304 manually annotated social media posts with 86.1% accuracy. We get 6,233 negative verses, 1,441 negative verses, and 23,459 neutral verses from a total of 31,133 verses from eng-x-bible-newworld2013.txt (considering the entire Bible, rather than only the New Testament, which results in a much higher verse count).

We propose to conduct emotion classification on positive and negative verses because we assume these verses have a higher probability of containing emotions. We utilize a fine-tuned DistilBERT model8 to perform emotion classification, which is a multi-class classification task with six labels: Joy, Anger, sadness, Fear, Love, and Surprise. The numbers of verses in each category are as follows: Sadness: 1171, Joy: 1952, Love: 870, Anger: 4201, Fear: 457, Surprise: 29. However, a great num 

* [145] M. C. Bader, R. J. Barlow, and A. K. Sridharan, "Theoretical performance of the \({}^{208}\)Bi-based system in a large number of applications," _Journal of Computational Physics_**36**, no. 7, pp. 95-104, 2000.
* [146] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in _Proceedings of the Thirtieth International Conference on Machine Learning_, volume 1, pages 1-10, 2015.
* [147] T. L. Chuang, Y. Bengio, and G. Hinton, "Deep learning with deep neural networks," _arXiv preprint arXiv:1512.01581_**, 2015.
* [148] H. Li, X. Zhang, and Z. Chen, "Learning generative models from tiny data sets," _IEEE Transactions on Neural Networks_**5**, no. 1, pp. 1-10, 2015.
* [149] I. Goodfellow, I. Guyon, and J. Shlens, "Image recognition by jointly connected layers," _Advances in neural information processing systems_**17**, no. 1, pp. 1-10, 2004.
* [150] V. Mnih, E. Rabinovich, and N. Srivastava, "A simple approach to image classification via deep convolutional neural networks," _CoRR_**abs/1412.1015**, 2014.
* [151] O. Russakovsky, S. Ozguner, and A. Radford, "Dropout: A simple way to prevent overfitting," _Nature_**406**, 468-472, 2015.
* [152] W. Teh, "Batch normalization using deep belief nets," _arXiv preprint arXiv:1412.6098_**, 2014.

[MISSING_PAGE_POST]

 

* [145] M. C. R. Adams, J. A. Harvey, and D. E. Thomas, _The New York Times of the Association for Computational Physics_, vol. 30, p. 267-289, 1996.

[MISSING_PAGE_POST]

 

ber of verses are not correctly classified because most verses in the Bible are objective, and it is impossible to classify them into emotions. For example, verse 01037029 Later when Reuben returned to the waterpit and saw that Joseph was not in the waterpit, he ripped his garments apart is an objective sentence, but is assigned an emotion _Anger_. Thus, we do not use the emotion classification task and instead continue to seek other categories.

### Category Design

The failure of emotion classification implies that the Bible verses are not suitable for subjective classification. We thus decide to design topic categories using _Latent Dirichlet Allocation_ topic search model.9

Footnote 9: [https://tinyurl.com/mr487nc6](https://tinyurl.com/mr487nc6)

#### e.2.1 Latent Dirichlet Allocation

To detect latent topics in the Bible, We use the Latent Dirichlet Allocation topic model. We set the number of tokens to describe each topic to 10 and the number of topics to 200. Besides eliminating the common stop words with NLTK stopwords package, we also filter out highly frequent words such as _God_ and _Jehova_, and meaningless tokens like _ah_ and _el_. However, LDA produces results that do not indicate meaningful topics based on the output words. We present five randomly chosen sets of words to show an example:

Figure 5: mTurk interface with English instructions and verse examples



people, us, men]

Topic 4: [land, one, let, people, men, us, went, took, go, brought]

Topic 5: [one, israel, king, people, may, like, man, days, seven, moses]

We can observe that there are many overlapping words in different topics, and it is difficult to interpret the results. The reason is presumably that LDA is suitable for processing long documents, but a verse normally contains fewer than 50 tokens and is too short to extract hidden topics for LDA. In addition, LDA may not work well on documents that do not coherently discuss a single topic, and there are numerous verses that do not belong to just one specific topic. A Reddit comments classification experiment by other researcher also occurs the same problem as ours.10

Footnote 10: [https://www.georgeho.org/lda-sucks/](https://www.georgeho.org/lda-sucks/)

#### e.2.2 Self-Designed Categories

Because LDA fails to produce meaningful topics of the Bible verses, we attempt to create some categories according to commonly occurring verses, v1 in table 7 shows the initial category design. The first version contains categories _Rules, Phenomenon, Conflict, Relation, Place, Character, Reward, Punishment, and Command. Rules_ defines verses that state an activity must or must not be done. _Phenomenon_ describes natural or societal facts. _Conflict_ includes argument, violence, or war among people, groups, or countries. _Relation_ reflects family genealogy. _Place_ includes verses that contain a city or area where an event happens. _Character_ contains verses that indicate the personality of a person. _Reward_ describes a person given something by God because he has done something good. _Punishment_ is the counterpart of reward that describes punishment from God. _Command_ is the order from God. After the categories are defined, we look for several example verses that can be shown to crowdsource workers in order to annotate the data. However, by collecting example verses, we find overlapping definitions between certain categories. For instance, the verse 03019023 When you come into the land and you plant any tree for food , you must consider its fruitage impure and forbidden . For three years it will be forbidden to you . It must not be eaten . can be either annotated as _Command_ or _Rules_. Therefore, in order to obtain better categories and alleviate the category overlap, we seek help from topic models and experts. The next paragraphs present details on exploring the categories.

#### e.2.3 Online Bible Topics

Following the failure of self-designed categories, we analyze the difficulty to create categories for the Bible verses. Compared with data of other benchmarks that normally use Common Crawl or Wikipedia, the domain of the Bible is too specific to extract categories merely according to common sense. Instead, rheological knowledge may assist in category creation. Thus, we change the strategy of building categories by browsing websites with the keywords "_Bible topics_". Thanks to a large number of available preaching websites, we are able to find a lot of topics created to help with the creation of categories. Those topics are presented on the websites with verses examples. Among all websites we have browsed, ProPreacher12 is the best one with a variety of 100 sermon topics and respective verse examples. Subsequently, we select topics from 100 sermon topics. There are two principles when selecting topics. First, we ensure that the benchmark is challenging, thus more categories should be contained. Second, in order to build a dataset with enough sentences, only topics with many examples should be chosen. In the end, we collect 15 categories (v2 in table 7) with sufficient example verses. Before we start crowdsourcing with these categories, we show three NLP students the category collection and 100 randomly sampled verses to annotate. They reflect that these topics are too abstract to understand. For example, _Eschatology_, _Philosophy_ and _Theology_ are hard to apply to respective verses. Therefore, we adjust the categories to v3 (table 7) based on v2 and the feedback. v3 deletes abstract topics _Eschatology, Philosophy, Theodoy, and Moral_, while adding _Repentance, Friendship_, _Thankfulness, Forgiveness, and Suffering_ that are collected from other preaching websites. The topic _Persecution_ is changed to _Heresy_. Once finished the task and category design, we start with crowdsourcing to obtain annotated data.

 



#### e.2.4 Crowdsource Attempts

We choose Amazon Mechanical Turk (mTruk) to test the designed topics because of its availability of a large number of native English speakers that we are looking for. Besides, it has sufficient online tutorials that can help to build annotation projects. When the v3 (table 7) class design is determined, we use mTurk to assign verses and test the quality of designed topics.

## Appendix F Data collection

Our dataset is built based on PBC and 1000Langs. Due to the copyright issue, our dataset consists of three parts:

* 1403 editions in 670 languages with permissive licenses which we distribute freely (the corpus we call Taxi1500-c v1.0).
* For the remaining PBC Bibles, please contact Michael Cysouw at Philipps University of Marburg to request access to PBC. Once granted access, run the code available at our Github to obtain the labeled dataset.
* For the remaining 1000Langs Bibles, use the code provided at the corresponding Github to crawl the corpus. Then, run the code available at our Github to obtain the labeled dataset.

## Appendix G Details of Taxi1500 dataset

We represent the definition of every class in Table 8. and the number of verses of different languages in table 9.

## Appendix H Evaluation on LLMs

We use a 3-shot prompt and adhere to the methodology outlined in Lin et al. (2024). We report average results in 5 and all results in 10.

## Appendix I Results for zero-shot

We report the detailed results for zero-shot transfer of BOW, mBERT, XLM-R-B, XLM-R-L, and Glot500-m.





\begin{table}
\begin{tabular}{l|p{142.3pt} p{56.9pt}} \hline \multicolumn{2}{c}{class} & definition \\ \hline Recommendation & An imperative statement which suggests to act or believe in certain ways. \\ \hline Faith & Display of belief and love toward God, instructions on how to maintain faith, stories of faith and its consequences, etc. \\ \hline Description & Describes a person, relationship, phenomenon, situation, etc. \\ \hline Sin & Describes what is considered sin, stories of sinful people and sinful actions. \\ \hline Grace & God’s love, blessing, and kindness towards humans. \\ \hline Violence & Describes wars, conflict, threats, and torture; but also destructions of people, cities, and nations. \\ \hline \end{tabular}
\end{table}
Table 8: Definitions of the six Taxi1500 classes

\begin{table}
\begin{tabular}{l|p{17.1pt} p{17.1pt} p{17.1pt} p{17.1pt} p{17.1pt} p{17.1pt} p{17.1pt} p{17.1pt} p{17.1pt} p{17.1pt}} \hline verse.num & 1077 & 1076 & 1075 & 1074 & 1073 & 1072 & 1071 & 1070 & 1069 & 1067 & 1066 & 1065 \\ lan.num & 1409 & 20 & 14 & 5 & 4 & 2 & 3 & 5 & 1 & 2 & 2 & 3 \\ \hline verse.num & 1064 & 1063 & 1061 & 1060 & 1057 & 1056 & 1055 & 1054 & 1053 & 1051 & 1049 & 1048 \\ lan.num & 3 & 1 & 2 & 3 & 1 & 2 & 3 & 1 & 1 & 1 & 1 & 3 \\ \hline verse.num & 1044 & 1042 & 1041 & 1039 & 1038 & 1034 & 1017 & 1006 & 1000 & 989 & 961 & 949 \\ lan.num & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 2 & 1 & 1 & 1 & 1 \\ \hline \end{tabular}
\end{table}
Table 9: An overview of the number of verses of different languages, for example: 1049 of the languages have 1077 verses in the dataset.





\begin{table}
\begin{tabular}{l|c c c c c c} Language & LLaMA2 & Mistral & Bloom-560M & Bloom-1B & Bloom-3B & Bloom-7B \\ \hline mhr\_Cyrl & 0.47 & 0.46 & 0.48 & 0.50 & **0.51** & 0.46 \\ azb\_Arab & 0.40 & **0.51** & 0.43 & 0.47 & 0.41 & 0.48 \\ sam\_Beng & 0.46 & **0.56** & 0.36 & 0.45 & 0.49 & 0.55 \\ ben\_Beng & 0.41 & **0.58** & 0.41 & 0.48 & 0.48 & 0.52 \\ tha\_Thai & 0.43 & **0.58** & 0.45 & 0.47 & 0.41 & 0.43 \\ khm\_Khmr & 0.52 & **0.56** & 0.52 & **0.56** & 0.52 & 0.49 \\ ell\_Grek & 0.49 & **0.58** & 0.44 & 0.49 & 0.49 & 0.49 \\ oss\_Cyrl & 0.49 & 0.48 & 0.48 & **0.52** & 0.47 & 0.49 \\ pan\_Guru & 0.41 & 0.46 & 0.44 & **0.47** & **0.47** & **0.47** \\ tat\_Cyrl & 0.48 & **0.53** & 0.43 & **0.53** & 0.48 & 0.46 \\ hne\_Deva & 0.56 & **0.61** & 0.56 & **0.61** & 0.58 & 0.54 \\ arb\_Arab & 0.43 & **0.62** & 0.46 & 0.53 & 0.49 & 0.49 \\ mkd\_Cyrl & 0.52 & **0.67** & 0.54 & 0.61 & 0.57 & 0.57 \\ bul\_Cyrl & 0.45 & **0.61** & 0.41 & 0.44 & 0.47 & 0.49 \\ kir\_Cyrl & 0.51 & 0.53 & **0.62** & **0.62** & 0.57 & 0.48 \\ kaz\_Cyrl & 0.49 & **0.55** & 0.45 & 0.51 & **0.55** & 0.51 \\ udm\_Cyrl & 0.37 & 0.41 & 0.42 & **0.45** & 0.43 & 0.42 \\ ket\_Geor & 0.41 & **0.45** & 0.43 & **0.45** & 0.43 & 0.42 \\ ssh\_Cyrl & 0.41 & 0.46 & **0.49** & **0.49** & 0.46 & 0.44 \\ mal\_Deva & 0.45 & **0.62** & 0.45 & 0.52 & 0.49 & 0.49 \\ ary\_Arab & 0.32 & **0.56** & 0.34 & 0.43 & 0.36 & 0.39 \\ tyv\_Cyrl & 0.39 & **0.48** & 0.36 & 0.45 & **0.48** & 0.43 \\ snd\_Arab & 0.44 & **0.62** & 0.54 & 0.56 & 0.49 & 0.57 \\ itr\_Ethi & 0.30 & 0.40 & 0.38 & **0.41** & 0.32 & 0.28 \\ mya\_Myrm & 0.45 & 0.51 & 0.51 & **0.53** & 0.41 & 0.44 \\ alt\_Cyrl & 0.44 & 0.46 & 0.49 & **0.53** & 0.48 & 0.45 \\ fas\_Arab & 0.49 & **0.67** & 0.53 & 0.53 & 0.49 & 0.58 \\ kor\_Hang & 0.49 & **0.72** & 0.49 & 0.51 & 0.52 & 0.49 \\ rrc\_Cyrl & 0.46 & **0.55** & 0.45 & 0.49 & 0.46 & 0.49 \\ mar\_Deva & 0.49 & **0.56** & 0.49 & 0.49 & 0.49 & 0.53 \\ chv\_Cyrl & 0.43 & 0.45 & 0.47 & **0.51** & 0.42 & 0.45 \\ crh\_Cyrl & 0.49 & **0.57** & 0.48 & 0.49 & 0.51 & 0.48 \\ npi\_Deva & 0.51 & **0.67** & 0.56 & 0.55 & 0.59 & 0.56 \\ pes\_Arab & 0.51 & **0.65** & 0.54 & 0.50 & 0.49 & 0.59 \\ nep\_Deva & 0.45 & **0.67** & 0.51 & 0.58 & 0.54 & 0.63 \\ him\_Deva & 0.51 & **0.65** & 0.55 & 0.48 & 0.47 & 0.49 \\ arz\_Arab & 0.32 & **0.54** & 0.35 & 0.44 & 0.41 & 0.45 \\ ksw\_Mymr & 0.44 & 0.44 & 0.40 & **0.49** & 0.42 & 0.42 \\ ru\_Cyrl & 0.49 & **0.58** & 0.43 & 0.47 & 0.45 & 0.51 \\ bel\_Cyrl & 0.48 & **0.56** & 0.46 & 0.51 & 0.45 & 0.49 \\ cb\_Arab & 0.44 & **0.48** & 0.45 & 0.47 & 0.43 & 0.45 \\ lao\_Laoo & 0.45 & 0.45 & 0.48 & 0.51 & **0.57** & 0.47 \\ qkg\_Cyrl & 0.42 & **0.56** & 0.46 & 0.54 & 0.48 & 0.49 \\ lrh\_Hani & 0.55 & **0.66** & 0.51 & 0.56 & 0.53 & 0.54 \\ tel\_Tlelu & 0.33 & **0.54** & 0.39 & 0.52 & 0.51 & 0.51 \\ sin\_Sinh & 0.40 & 0.38 & 0.41 & **0.47** & 0.42 & 0.40 \\ prs\_Arab & 0.51 & **0.66** & 0.57 & 0.60 & 0.57 & 0.56 \\ che\_Cyrl & 0.38 & **0.42** & 0.36 & 0.41 & 0.33 & 0.37 \\ unz\_Cyrl & 0.46 & **0.59** & 0.43 & 0.49 & 0.43 & 0.45 \\ myv\_Cyrl & 0.40 & 0.45 & 0.36 & **0.47** & 0.45 & 0.41 \\ tam\_Taml & 0.44 & **0.60** & 0.55 & 0.55 & **0.60** & 0.59 \\ cmn,Hani & 0.49 & **0.61** & 0.44 & 0.54 & 0.54 & 0.53 \\ kjh\_Cyrl & 0.44 & 0.48 & 0.42 & **0.49** & 0.42 & 0.45 \\ hye\_Armen & 0.46 & **0.55** & 0.46 & 0.52 & 0.52 & 0.46 \\ bak\_Cyrl & 0.45 & 0.49 & 0.45 & **0.51** & 0.47 & 0.49 \\ kmr\_Cyrl & 0.40 & 0.40 & 0.39 & 0.44 & 0.43 & **0.45** \\ mdyn\_Ethi & 0.40 & **0.55** & 0.47 & 0.46 & 0.45 & 0.43 \\ ukr\_Cyrl & 0.52 & **0.63** & 0.51 & 0.49 & 0.49 & 0.51 \\ suz\_Deva & 0.47 & 0.43 & 0.42 & **0.48** & 0.45 & 0.42 \\ guj\_Gujar & 0.46 & **0.52** & 0.46 & 0.48 & 0.51 & **0.52** \\ dro\_Thtb & **0.45** & **0.45** & 0.42 & 0.41 & 0.43 & 0.41 \\ ori\_Orya & 0.43 & 0.51 & 0.51 & **0.56** & 0.54 & 0.51 \\ory\_Orya & 0.44 & 0.58 & 0.53 & 0.51 & **0.59** & 0.49 \\ yue\_Hani & 0.43 & **0.63** & 0.46 & 0.54 & 0.53 & 0.53 \\ \end{tabular}
\end{table}
Table 10: Performance across six LLMs on 64 selected languages.