TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages
Authors: Jafar Isbarov, Arofat Akhundjanova, Mammad Hajili, Kavsar Huseynova, Dmitry Gaynullin, Anar Rzayev, Osman Tursun, Ilshat Saetov, Rinat Kharisov, Saule Belginova, Ariana Kenbayeva, Amina Alisheva, Aizirek Turdubaeva, Abdullatif Köksal, SAMIR RUSTAMOV, Duygu Ataman
Accepted at: acl25

TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages

Jafar Isbarov$^{a1,2,3}$, Arofat Akhundjanova$^4$, Mammad Hajili$^5$, Kavsar Huseynova$^6$, Dmitry Gaynullin$^{15}$, Anar Rzayev$^7$, Osman Tursun$^8$, Aizirek Turdubaeva$^7$, Ilshat Saetov$^9$, Rinat Kharisov$^{15}$, Saule Belginova$^{10}$, Ariana Kenbayeva$^{11}$, Amina Alisheva$^{11}$, Abdullatif Köksal$^{812,13}$, Samir Rustamov$^{814}$, Duygu Ataman$^{82}$

$^1$eiLink R&D Center, Khazar University $^2$New York University, $^3$The George Washington University, $^4$Saarland University, $^5$Microsoft, $^6$Baku Higher Oil School, $^7$KAIST, $^8$Queensland University of Technology, $^9$CETOBaC EHESS, $^{10}$Turan University, $^{11}$Nazarbayev University $^{12}$LMU Munich, $^{13}$MCML, $^{14}$ADA University, $^{15}$Independent researcher

$^a$First author. $^b$Advisors.

Correspondence: jafar.isbarov@gwu.edu

Abstract

Being able to thoroughly assess massive multi-task language understanding (MMLU) capabilities is essential for advancing the applicability of multilingual language models. However, preparing such benchmarks in high quality native language is often costly and therefore limits the representativeness of evaluation datasets. While recent efforts focused on building more inclusive MMLU benchmarks, these are conventionally built using machine translation from high-resource languages, which may introduce errors and fail to account for the linguistic and cultural intricacies of the target languages. In this paper, we address the lack of native language MMLU benchmark especially in the under-represented Turkic language family with distinct morphosyntactic and cultural characteristics. We propose two benchmarks for Turkic language MMLU: TUMLU is a comprehensive, multilingual, and natively developed language understanding benchmark specifically designed for Turkic languages. It consists of middle- and high-school level questions spanning 11 academic subjects in Azerbaijani, Crimean Tatar, Karakalpak, Kazakh, Kyrgyz, Tatar, Turkish, Uyghur, and Uzbek. We also present TUMLU-mini, a more concise, balanced, and manually verified subset of the dataset. Using this dataset, we systematically evaluate a diverse range of open and proprietary multilingual large language models (LLMs), including Claude, Gemini, GPT, and LLaMA, offering an in-depth analysis of their performance across different languages, subjects, and alphabets. To promote further research and development in multilingual language understanding, we release TUMLU-mini and all corresponding evaluation scripts$^1$.

1 Introduction

Language understanding encompasses a system’s ability to interpret and derive meaning from human language, incorporating syntax, semantics, and context. Evaluating language models hinges on this capability, as it ensures coherence, contextual relevance, and accuracy. Benchmarking is integral to assessing these models, particularly with the rapid advancements in Large Language Models (LLMs), which now support multiple languages (Yang et al., 2025; Gemma Team, 2024; Grattafiori et al., 2024) and excel in complex reasoning tasks such as mathematical, scientific, and coding-related inquiries (Hurst et al., 2024; Anthropic, 2024; Gemini Team, 2024; Grattafiori et al., 2024). However, the scarcity of robust natural language understanding (NLU) benchmarks capturing diverse linguistic and cultural contexts remains a challenge. Notably, LLM performance declines in low-resource languages, which are often underrepresented in training data, highlighting the need for more inclusive evaluation frameworks.

The majority of benchmarks included in top leaderboards where cutting-edge LLMs are evaluated are majorly prepared in English (Hendrycks et al., 2021a; Suzgun et al., 2022; Wang et al.,

$^1$https://github.com/ceferisbarov/TUMLU
Figure 1. Distribution of subjects across languages in TUMLU. Numbers next to language names indicate the total question count. Left: middle- and high-resource languages; Right: low-resource languages.

2024b, 2019). In order to extend the applicability of LLM evaluation in more languages, recent efforts were undertaken to build more multilingual NLU benchmarks (Lai et al., 2023), however, most of these either cover a limited set of high-resourced languages, or the multilingual examples are generated by translating original examples from Western-centric languages, thus failing to capture cultural nuances inherent in different languages. Due to the multi-dimensional nature of the reasoning task, language-specific benchmarks especially when translated into other languages also fail to represent the actual usage as well as demonstrating reasoning in the native language., and may further introduce issues such as translationese (Vannmassenove et al., 2021) and cultural misalignment (Romanou et al., 2025). On the other end of the spectrum, there are efforts to bridge that gap for a particular language, for example, African languages (Bayes et al., 2024), Arabic (Koto et al., 2024), Chinese (Li et al., 2024), and Turkish (Yüksel et al., 2024).

In this paper, we focus on building a truly representative and inclusive single-language family benchmark to address previous problems and provide a challenging setting for LLM evaluation. TUMLU (Turkic Unified Multilingual Language Understanding) benchmark covers the following languages: Azerbaijani, Crimean Tatar, Turkish, Uyghur, Uzbek, Karakalpak, Kyrgyz, Kazakh, and Tatar. The dataset consists of 4-choice questions at middle- and high-school levels. It consists of 38139 questions across 8 languages and 11 subjects (see Figure 1 for a higher-level breakdown across languages). It is the first such benchmark to include Uyghur, Karakalpak, Tatar, or Crimean Tatar. It is also a significant improvement over existing benchmarks for Azerbaijani, Kazakh, Kyrgyz, and Uzbek. The Turkish dataset is TurkishMMLU, which was a separate project (Yüksel et al., 2024). The benchmark is also representative in terms of different scripts by including questions and answers in chosen languages in Latin, Cyrillic, and Arabic scripts. These datasets are transliterated such that it could be possible to get a dual dataset with the same content in two scripts for further comparative studies. We use these dual datasets to compare the performance of LLMs across different scripts.

We also release a more balanced and manually verified version of the dataset called TUMLU-mini, which contains 100 questions per subject (unless there are less than 100 for the said subject in a particular language). We use this version to test SOTA open-source and proprietary models of various sizes. We evaluated them in two settings: few-shot and chain-of-thought (CoT) reasoning (Wei
et al., 2024). Our initial results show that proprietary models remain the best option for Turkic languages.

2 Related Work

Language understanding benchmarks Multi-task language understanding evaluation benchmarks play an important role in the evaluation of LLMs. Early benchmarks concentrated on general natural language understanding. GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) were two such benchmarks that were widely adopted by the research community. These benchmarks were saturated quickly, due to the development of better LLMs. However, LLMs struggled more against benchmarks that required knowledge and reasoning. MMLU (Hendrycks et al., 2021a) and MMLU-Pro (Wang et al., 2024b) were more challenging since they required not only language understanding but also world knowledge. These general-purpose benchmarks gradually gave way to higher-level and more specialized benchmarks such as MATH (Hendrycks et al., 2021b), GPQA (Rein et al., 2024), and MUSR (Sprague et al., 2024).

Multilingual benchmarks The development of multilingual LLMs also necessitated challenging multilingual benchmarks. Most of these benchmarks were developed through machine translation (Conneau et al., 2018; Singh et al., 2024). However, such datasets have been shown to contain cultural biases and translation artifacts (Vanmassenhove et al., 2021). Global MMLU relied on machine and professional translation to (Singh et al., 2024). INCLUDE consists of native data (Romanou et al., 2025), but it is imbalanced, with different subject distributions in different languages. There is also a significant difference in required knowledge levels between languages, making a direct comparison impossible.

Benchmarks for Turkic languages SeaEval was one of the first LLM benchmarks to include Turkish (Wang et al., 2024a). Global MMLU contains Kyrgyz and Turkish subsets. INCLUDE contains Azerbaijani and Kazakh. MRL 2024 Shared Task on Multi-lingual Multi-task Information Retrieval (Tinner et al., 2024) contains an Azerbaijani dataset, but it contains general language understanding tasks rather than world knowledge. Kardeş-NLU has introduced a multilingual language understanding benchmark (Senel et al., 2024). But again, this benchmark contains general language understanding tasks that require no world knowledge. There are also monolingual benchmarks. Mukayese was one of the earliest general language understanding benchmarks in Turkish (Safaya et al., 2022). TurkishMMLU and TR-MMLU (Bayram et al., 2025) were the first native MMLU alternatives for the Turkish language. Another pilot study was performed to evaluate LLMs in Kazakh language (Maxutov et al., 2024). KazMMLU is another monolingual MMLU-style benchmark for the Kazakh language (Togmanov et al., 2025). While there are no peer-reviewed monolingual MMLU alternatives for Azerbaijani, there is a general language understanding benchmark (Isbarov et al., 2024).

Figure 2. A sample question from the parallel Uzbek dataset, available in both Cyrillic and Latin alphabets. This enables comparison of LLM performance across different scripts. English translation is provided for clarity.

![A sample question from the parallel Uzbek dataset, available in both Cyrillic and Latin alphabets.](page_1012_134_377_246.png)

3 TUMLU

TUMLU is a multilingual and multitask dataset containing 38139 multiple-choice questions across 8 languages and 11 subjects. All questions are at middle or high school level. The majority are sample or official questions for university entrance exams of respective countries.
<table>
  <tr>
    <th>Language</th>
    <th>Question</th>
    <th>Answer</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>63.1</td>
    <td>28.0</td>
  </tr>
  <tr>
    <td>Crimean Tatar</td>
    <td>113.5</td>
    <td>67.3</td>
  </tr>
  <tr>
    <td>Karakalpak</td>
    <td>112.3</td>
    <td>65.3</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>96.8</td>
    <td>19.7</td>
  </tr>
  <tr>
    <td>Kyrgyz</td>
    <td>85.7</td>
    <td>40.0</td>
  </tr>
  <tr>
    <td>Tatar</td>
    <td>154.2</td>
    <td>47.8</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>204.6</td>
    <td>69.6</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>180.1</td>
    <td>51.1</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>161.4</td>
    <td>16.2</td>
  </tr>
</table>

Table 1. Average length of questions and answers across languages. An answer here refers to all choices, not only the correct ones.

Data collection Data was collected from publicly available books and websites. In original form, questions had 2 to 5 choices. In cases where more than 4 choices were available, we removed an incorrect choice. If less than 4 choices were available, we left the question as-is. Except for Language and Literature questions in Crimean Tatar, all questions have 4 choices in the final version.

After collecting the data, native speakers of each language manually verified the quality of a random sample from each subject. In languages such as Azerbaijani where questions were developed by the community, around 10 % of the questions were either invalid or had incorrect answers.

We also created 5 CoT prompts per subject in Azerbaijani, Kazakh, Turkish, and Uzbek. Azerbaijani, Kazakh, and Uzbek prompts were created manually by native speakers. Turkish prompts were adapted from the TurkishMMLU project by changing the number of choices from 5 to 4. These prompts allowed us to compare the no-CoT and CoT performance of models. We did not create CoT prompts for other languages, because we did not have native speakers to validate their quality. CoT samples in Azerbaijani are available in Appendix A.

Data composition TUMLU contains eleven subjects: Maths, Physics, Chemistry, Biology, Geography, Native Language & Literature (NL&L), History, Logic, Human & Society, Philosophy, Religion & Ethics. Among these, Logic, Human & Society, Philosophy and Religion & Ethics subjects are available only in one or two languages. Therefore, they have not been included in experiments.

We report the number of characters per question and per choice in Table 1. High variance in question and answer length indicates variable question types and levels across languages.

Difficulty levels While TUMLU can be used as a monolingual benchmark for any of the languages included, we are also interested in comparing performance across languages. This raises an important question: how comparable are questions of the same subject across different languages? While we can easily compare the model performance within each language, comparing it across languages proves more challenging. Different language subsets have different levels of difficulty. Uzbek and Turkish datasets are particularly difficult because those questions are designed specifically to imitate university entrance examinations in respective countries. Azerbaijani and Kazakh questions were developed by a community of students and teachers. While they all refer to middle- and high-school topics, there has been no oversight regarding their difficulty levels. For example, we know that maths questions in Kazakh are easier than the ones in other languages because they cover middle-school topics only. While these are explicit discrepancies that can be fixed in the future, there are certainly less obvious differences that are harder to identify and even harder to fix. That being said, even though comparison across languages is challenging, these datasets are more useful for comparison across models in monolingual settings.

TUMLU-mini To make our experiments more balanced and less costly, we developed TUMLU-mini, which consists of 100 randomly selected and manually verified questions per subject. In cases where we had less than 100 questions, we used the entire set. You can find the number of questions per language and subject in Appendix B. If a question had more than 5 answer choices, one was dropped. All choices have been shuffled to make the dataset more robust to simple memorization since it is possible that these questions were a part of the pre-training corpus for these LLMs. We also removed subjects that were available in less than 3 languages. All experiments were run on this subset.

4 Experimental set-up

Data Previous work (Yüksel et al., 2024) has shown that 100 questions per subject are enough
to estimate the performance of a larger dataset. Therefore, we run all experiments on TUMLU-mini. While we have performed the experiments and publicly released the data, results on the following subjects are not reported in the paper: Logic, Philosophy, Religion & Ethics, and Human & Society. These subjects are available in one or two languages only, which makes any generalization impossible.

Model choice  We have used TUMLU to evaluate both open-source models, such as Llama 3.1 (Grattafiori et al., 2024), Gemma 2 (Gemma Team, 2024), Qwen2.5 (Yang et al., 2025) and proprietary models, such as Gemini 1.5 (Gemini Team, 2024), Claude 3.5 (Anthropic, 2024), GPT-4o (Hurst et al., 2024). The size of selected open-source models varies between 7B and 70B. We do not have this information on proprietary models. This list includes models from the same series, such as Qwen2.5 7B instruct and Qwen2.5 70B instruct (Yang et al., 2025), which allows us to observe the effect of scaling (Hestness et al., 2017; Kaplan et al., 2020) on multilingual performance. All open-source models are instruct-tuned versions. We have omitted this information in the tables to preserve space. Wherever applicable, we have included the performance of Claude 3.5 Sonnet in the paper, since it consistently outperforms all other models. The performance of the remaining models can be found in the appendices C and D.

Prompting  We have run experiments in two settings: 5-shot, where we provide 5 example questions and answers on the same subject before asking a question (Brown et al., 2020), and 5-shot CoT, where we provide 5 example questions and explanations of their answers before asking the question (Wei et al., 2024). Few-shot and CoT prompt samples are available in Appendix A. Previous work has demonstrated that (Romanou et al., 2025) providing the prompt in English does not result in performance gains. Due to this, we provide all prompts in respective native languages.

Technical details  We run our experiments through OpenAI API, Anthropic API, Google Cloud Gemini API, Together AI API, and Deep Infra API. No model was run on a local machine. We used the following hyperparameters with all APIs: TEMPERATURE = 0.0, MAX_TOKENS = 1024, TOP_P = 1.0.

5  Results

In this section, we present the few-shot and CoT performance of selected models on the TUMLU-mini dataset. We also present an analysis of output language. Lastly, we explore how well LLMs perform on the same questions written in different (Latin, Cyrillic, or Arabic) scripts.

5-shot results  We present the average performance of all models in each language in Table 2. Claude 3.5 Sonnet outperforms other models in all languages. The top 5 spots belong to proprietary models, although it has to be noted that there are larger open-source models that have not been included in this benchmark. Among the available open-source models, Qwen2.5 72B Instruct has the best performance. Results also confirm the scaling hypothesis: Llama 3.1 70B significantly outperforms Llama 3.1 8B. The same applies to Qwen2.5 7B/72B and Gemma 2 9B/27B. We can also observe a significant improvement from Llama 3.1 70B to Llama 3.3 70B. While it is not possible to directly compare results across languages, we can observe that low-resource languages, such as Crimean Tatar, Karakalpak, and Uyghur have comparable performance to middle- and high-resource languages. Notably, this trend holds even with the lowest-performing models.

We present the 5-shot evaluation of Claude 3.5 Sonnet in more detail in Table 3. In most languages, Native Language & Literature is the most challenging subject for Claude 3.5 Sonnet. This holds for other models, as well (See Appendix C).

5-shot CoT results  We present the average results of the 5-shot CoT evaluation in Table 4. CoT prompts have an overall positive effect on performance. Sporadic negative effects can be explained by incorrect output format, rather than incorrect answers. We avoided manual validation of the output and instead relied on generalizable pattern-matching methods.

Table 5 shows the performance of Claude 3.5 Sonnet on each subject and language. On average, CoT prompts have a net positive effect in each subject and each language.

Generated language vs. performance  Benchmark results demonstrate that LLMs can have significant language understanding capabilities even in previously unseen languages, such as Crimean
<table>
  <tr>
    <th>Model</th>
    <th>mean</th>
    <th>aze</th>
    <th>crh</th>
    <th>kaa</th>
    <th>kaz</th>
    <th>kir</th>
    <th>tat</th>
    <th>tur</th>
    <th>uig</th>
    <th>uzb</th>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td>
    <td>78.9</td>
    <td>84.4</td>
    <td>81.2</td>
    <td>75.3</td>
    <td>83.0</td>
    <td>75.7</td>
    <td>84.0</td>
    <td>85.7</td>
    <td>71.3</td>
    <td>69.1</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>74.9</td>
    <td>82.4</td>
    <td>70.5</td>
    <td>70.8</td>
    <td>81.0</td>
    <td>72.9</td>
    <td>80.5</td>
    <td>83.7</td>
    <td>66.5</td>
    <td>65.4</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Pro</td>
    <td>73.8</td>
    <td>78.6</td>
    <td>70.3</td>
    <td>68.2</td>
    <td>78.4</td>
    <td>72.3</td>
    <td>80.5</td>
    <td>80.0</td>
    <td>71.0</td>
    <td>65.1</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Flash</td>
    <td>65.4</td>
    <td>72.4</td>
    <td>68.0</td>
    <td>61.2</td>
    <td>68.6</td>
    <td>63.2</td>
    <td>68.3</td>
    <td>76.6</td>
    <td>57.8</td>
    <td>52.1</td>
  </tr>
  <tr>
    <td>Claude 3.5 Haiku</td>
    <td>64.0</td>
    <td>70.6</td>
    <td>62.9</td>
    <td>55.2</td>
    <td>69.9</td>
    <td>64.8</td>
    <td>67.5</td>
    <td>78.0</td>
    <td>56.6</td>
    <td>50.3</td>
  </tr>
  <tr>
    <td>Llama- 3.1 405B</td>
    <td>63.0</td>
    <td>65.9</td>
    <td>69.5</td>
    <td>60.0</td>
    <td>69.0</td>
    <td>64.1</td>
    <td>70.4</td>
    <td>59.7</td>
    <td>58.2</td>
    <td>50.4</td>
  </tr>
  <tr>
    <td>Qwen2.5 72B</td>
    <td>59.9</td>
    <td>70.1</td>
    <td>61.8</td>
    <td>54.6</td>
    <td>62.6</td>
    <td>47.5</td>
    <td>62.5</td>
    <td>73.9</td>
    <td>56.0</td>
    <td>50.4</td>
  </tr>
  <tr>
    <td>Llama 3.3 70B</td>
    <td>58.6</td>
    <td>66.0</td>
    <td>58.7</td>
    <td>49.2</td>
    <td>60.0</td>
    <td>60.2</td>
    <td>69.5</td>
    <td>68.4</td>
    <td>51.6</td>
    <td>44.1</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B</td>
    <td>57.7</td>
    <td>68.1</td>
    <td>57.3</td>
    <td>49.9</td>
    <td>56.4</td>
    <td>58.4</td>
    <td>66.2</td>
    <td>64.9</td>
    <td>52.4</td>
    <td>45.3</td>
  </tr>
  <tr>
    <td>Gemma 2 27b</td>
    <td>51.8</td>
    <td>58.1</td>
    <td>49.8</td>
    <td>47.6</td>
    <td>58.4</td>
    <td>53.4</td>
    <td>54.9</td>
    <td>64.3</td>
    <td>42.2</td>
    <td>37.6</td>
  </tr>
  <tr>
    <td>Gemma 2 9b</td>
    <td>47.1</td>
    <td>53.7</td>
    <td>46.8</td>
    <td>40.8</td>
    <td>49.1</td>
    <td>48.9</td>
    <td>51.8</td>
    <td>60.4</td>
    <td>35.8</td>
    <td>36.1</td>
  </tr>
  <tr>
    <td>Qwen2.5 7B</td>
    <td>41.0</td>
    <td>48.0</td>
    <td>42.6</td>
    <td>37.2</td>
    <td>45.0</td>
    <td>31.8</td>
    <td>40.5</td>
    <td>55.6</td>
    <td>33.4</td>
    <td>34.6</td>
  </tr>
  <tr>
    <td>Llama 3.1 8B</td>
    <td>40.3</td>
    <td>48.4</td>
    <td>35.7</td>
    <td>33.4</td>
    <td>46.4</td>
    <td>41.8</td>
    <td>44.1</td>
    <td>47.7</td>
    <td>35.0</td>
    <td>29.9</td>
  </tr>
</table>

Table 2. Average 5-shot performance of models on Azerbaijani (aze), Crimean Tatar (crh), Karakalpak (kaa), Kazakh (kaz), Kyrgyz (kir), Tatar (tat), Turkish (tur), Uyghur (uig), and Uzbek (uzb) datasets.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>89.0</td>
    <td>89.0</td>
    <td>91.0</td>
    <td>71.0</td>
    <td>85.0</td>
    <td>73.0</td>
    <td>93.0</td>
  </tr>
  <tr>
    <td>Crimean-tatar</td>
    <td>81.6</td>
    <td>75.0</td>
    <td>87.0</td>
    <td>89.9</td>
    <td>70.4</td>
    <td>75.0</td>
    <td>89.7</td>
  </tr>
  <tr>
    <td>Karakalpak</td>
    <td>78.0</td>
    <td>85.7</td>
    <td>75.0</td>
    <td>-</td>
    <td>-</td>
    <td>42.2</td>
    <td>95.6</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>92.0</td>
    <td>73.0</td>
    <td>78.0</td>
    <td>78.0</td>
    <td>96.0</td>
    <td>76.0</td>
    <td>88.0</td>
  </tr>
  <tr>
    <td>Kyrgyz</td>
    <td>77.0</td>
    <td>80.0</td>
    <td>87.0</td>
    <td>78.6</td>
    <td>-</td>
    <td>80.8</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Tatar</td>
    <td>94.0</td>
    <td>84.0</td>
    <td>83.0</td>
    <td>91.0</td>
    <td>86.3</td>
    <td>69.0</td>
    <td>81.0</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>84.0</td>
    <td>88.0</td>
    <td>94.0</td>
    <td>92.0</td>
    <td>78.0</td>
    <td>79.0</td>
    <td>85.0</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>75.0</td>
    <td>66.0</td>
    <td>-</td>
    <td>-</td>
    <td>75.8</td>
    <td>66.0</td>
    <td>73.5</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>71.0</td>
    <td>73.0</td>
    <td>64.0</td>
    <td>70.0</td>
    <td>70.0</td>
    <td>55.0</td>
    <td>81.0</td>
  </tr>
</table>

Table 3. Subject-wise 5-shot performance of Claude 3.5 Sonnet across Turkic languages. Missing values indicate the absence of data for that language in the given subject. NL&L refers to Native Language and Literature. *This subset contains questions with 2 or 3 choices.

Tatar. This can be explained by linguistic proximity to languages better represented in the training data. Even though LLMs perform surprisingly well in these languages with simple 5-shot prompts. The results are less impressive when we analyze the generated text quality. While quality per se is hard to quantify, we can detect the language of generated content. We used Google Cloud Translate API to detect output language. This API supports all languages in our benchmark, except for Karakalpak. We present results for Crimean Tatar in Figure 3. As you can see, although these models have answered the majority of the questions in Crimean Tatar correctly, only a small portion of the generated text is classified as Crimean Tatar. Almost all of the answers are a synthesis between Turkish and Crimean Tatar. A similar issue appears in Kazakh when we switch from Cyrillic to Latin script. Although this has a small negative effect on the performance, the nature of the generated content changes dramatically. While the output of Cyrillic questions is easily detected as Kazakh, the output of Latin questions is easily confused with Tatar language.

Comparing performance on same questions written in different alphabets Some Turkic languages, such as Crimean Tatar, Kazakh, and Uzbek have both Cyrillic and Latin alphabets that are actively used. As a result, the text corpora that are used to train LLMs contain both versions. Also, transliteration between these scripts can be done automatically with a negligible error rate. Using these facts, we developed dual datasets for the languages above (see Figure 2). We evaluated models in both versions and compared their performance.
<table>
  <tr>
    <th>Model</th>
    <th>Mean</th>
    <th>aze</th>
    <th>kaz</th>
    <th>tat</th>
    <th>tur</th>
    <th>uzb</th>
  </tr>
  <tr>
    <td>Claude 3.5 Sonnet</td>
    <td>84.0</td>
    <td><b>87.1 (+2.7)</b></td>
    <td>84.1 (<i>+1.1</i>)</td>
    <td>87.9 (<i>+3.9</i>)</td>
    <td><b>87.9 (+2.1)</b></td>
    <td>72.9 (<i>+3.7</i>)</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>79.4</td>
    <td>82.9 (<i>+0.4</i>)</td>
    <td>80.7 (-0.3)</td>
    <td>83.0 (<i>+2.5</i>)</td>
    <td>84.0 (<i>+0.3</i>)</td>
    <td>66.3 (<i>+0.9</i>)</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Pro</td>
    <td>76.6</td>
    <td>80.0 (<i>+1.4</i>)</td>
    <td>75.1 (<i>-3.3</i>)</td>
    <td>79.9 (-0.6)</td>
    <td>81.0 (<i>+1.0</i>)</td>
    <td>67.0 (<i>+1.9</i>)</td>
  </tr>
  <tr>
    <td>Llama 3.1 405B</td>
    <td>68.9</td>
    <td>73.4 (<i>+7.6</i>)</td>
    <td>68.7 (<i>-0.3</i>)</td>
    <td>65.0 (<i>-5.4</i>)</td>
    <td>80.7 (<i>+21.0</i>)</td>
    <td>56.4 (<i>+6.0</i>)</td>
  </tr>
  <tr>
    <td>Claude 3.5 Haiku</td>
    <td>70.0</td>
    <td>77.0 (<i>+6.4</i>)</td>
    <td>74.0 (<i>+4.1</i>)</td>
    <td>72.2 (<i>+4.8</i>)</td>
    <td>77.6 (-0.4)</td>
    <td>49.0 (<i>-1.3</i>)</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Flash</td>
    <td>68.1</td>
    <td>73.9 (<i>+1.4</i>)</td>
    <td>69.0 (<i>+0.4</i>)</td>
    <td>70.1 (<i>+1.8</i>)</td>
    <td>73.6 (<i>+3.0</i>)</td>
    <td>54.1 (<i>+2.0</i>)</td>
  </tr>
  <tr>
    <td>Qwen2.5 72B</td>
    <td>67.1</td>
    <td>72.1 (<i>+2.0</i>)</td>
    <td>63.9 (<i>+1.3</i>)</td>
    <td>67.6 (<i>+5.1</i>)</td>
    <td>78.4 (<i>+4.6</i>)</td>
    <td>53.6 (<i>+3.1</i>)</td>
  </tr>
  <tr>
    <td>Llama 3.3 70B</td>
    <td>66.8</td>
    <td>70.6 (<i>+4.6</i>)</td>
    <td>69.3 (<i>+9.3</i>)</td>
    <td>66.3 (<i>-3.2</i>)</td>
    <td>77.4 (<i>+9.0</i>)</td>
    <td>50.4 (<i>+6.3</i>)</td>
  </tr>
  <tr>
    <td>Gemma 2 27B</td>
    <td>59.4</td>
    <td>63.0 (<i>+4.9</i>)</td>
    <td>61.6 (<i>+3.1</i>)</td>
    <td>61.0 (<i>+6.1</i>)</td>
    <td>66.4 (<i>+2.1</i>)</td>
    <td>44.9 (<i>+7.3</i>)</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B</td>
    <td>56.2</td>
    <td>59.4 (<i>-8.7</i>)</td>
    <td>61.7 (<i>+5.3</i>)</td>
    <td>58.6 (<i>-7.6</i>)</td>
    <td>73.3 (<i>+8.4</i>)</td>
    <td>27.9 (<i>-17.4</i>)</td>
  </tr>
  <tr>
    <td>Gemma 2 9B</td>
    <td>52.0</td>
    <td>57.3 (<i>+3.6</i>)</td>
    <td>52.7 (<i>+3.6</i>)</td>
    <td>50.1 (<i>-1.7</i>)</td>
    <td>62.3 (<i>+1.9</i>)</td>
    <td>37.4 (<i>+1.3</i>)</td>
  </tr>
  <tr>
    <td>Qwen2.5 7B</td>
    <td>46.4</td>
    <td>48.1 (<i>+0.1</i>)</td>
    <td>46.4 (<i>+1.4</i>)</td>
    <td>43.0 (<i>+2.5</i>)</td>
    <td>56.3 (<i>+0.7</i>)</td>
    <td>38.0 (<i>+3.4</i>)</td>
  </tr>
  <tr>
    <td>Llama 3.1 8B</td>
    <td>38.2</td>
    <td>40.7 (<i>-7.7</i>)</td>
    <td>38.9 (<i>-7.6</i>)</td>
    <td>39.7 (<i>-4.4</i>)</td>
    <td>45.1 (<i>-2.6</i>)</td>
    <td>26.6 (<i>-3.3</i>)</td>
  </tr>
</table>

Table 4. Average 5-shot performance of models on Turkic languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>89.0 (0.0)</td>
    <td>96.0 (<i>+7.0</i>)</td>
    <td>91.0 (0.0)</td>
    <td>78.0 (<i>+7.0</i>)</td>
    <td>83.0 (<i>-2.0</i>)</td>
    <td>76.0 (<i>+3.0</i>)</td>
    <td>97.0 (<i>+4.0</i>)</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>96.0 (<i>+4.0</i>)</td>
    <td>74.0 (<i>+1.0</i>)</td>
    <td>78.0 (0.0)</td>
    <td>80.0 (<i>+2.0</i>)</td>
    <td>95.0 (<i>-1.0</i>)</td>
    <td>79.0 (<i>+3.0</i>)</td>
    <td>87.0 (<i>-1.0</i>)</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>86.0 (<i>+2.0</i>)</td>
    <td>85.0 (<i>-2.0</i>)</td>
    <td>95.0 (<i>+1.0</i>)</td>
    <td>93.0 (<i>+2.0</i>)</td>
    <td>77.0 (0.0)</td>
    <td>87.0 (<i>+11.0</i>)</td>
    <td>89.0 (<i>+4.0</i>)</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>77.0 (<i>+8.0</i>)</td>
    <td>73.0 (0.0)</td>
    <td>65.0 (<i>+1.0</i>)</td>
    <td>65.0 (<i>-3.0</i>)</td>
    <td>79.0 (<i>+9.0</i>)</td>
    <td>45.0 (<i>-10.0</i>)</td>
    <td>87.0 (<i>+6.0</i>)</td>
  </tr>
</table>

Table 5. Subject-wise 5-shot CoT performance of Claude 3.5 Sonnet across Turkic languages.

We present some of the results in Table 6. While the results initially seem irregular, they follow a simple pattern:

1. In Crimean Tatar questions, all three models perform better in the Latin script. FineWeb 2 (Penedo et al., 2024), one of the largest multilingual text corpora, contains 21,365,608 Latin and 1,934,168 Cyrillic words in Crimean Tatar.

2. In Kazakh questions, all three models perform better in the Cyrillic script. This aligns with the fact that most of the Kazakh text data on the web is written in the Cyrillic script. For example, the FineWeb 2 corpus contains 1,837,049,585 Cyrillic and 0 Latin words in Kazakh.

3. In Uyghur questions, all three models perform better in the Arabic script. While Uyghur is not represented in Fineweb 2 corpus, virtually all Uyghur text is written in Arabic script.

4. In Uzbek questions, results are less predictable. This can be explained by the fact that Cyrillic and Latin are more evenly distributed in Uzbek text. FineWeb 2 corpus

![Bar chart showing language distribution of model responses to Crimean Tatar queries](page_101_1042_627_312.png)

Figure 3. Language distribution of model responses to Crimean Tatar queries, as detected by Google Cloud Translation API.
<table>
  <tr>
    <th rowspan="2">Language</th>
    <th colspan="3">Claude 3.5 Sonnet</th>
    <th colspan="3">Qwen2.5 72B</th>
    <th colspan="3">Gemma 2 27B</th>
  </tr>
  <tr>
    <th>Cyrillic</th>
    <th>Latin</th>
    <th>Arabic</th>
    <th>Cyrillic</th>
    <th>Latin</th>
    <th>Arabic</th>
    <th>Cyrillic</th>
    <th>Latin</th>
    <th>Arabic</th>
  </tr>
  <tr>
    <td>Crimean Tatar</td>
    <td>66.1</td>
    <td>80.0</td>
    <td>—</td>
    <td>47.6</td>
    <td>61.8</td>
    <td>—</td>
    <td>43.5</td>
    <td>49.8</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td><b>82.7</b></td>
    <td>78.0</td>
    <td>—</td>
    <td><b>64.3</b></td>
    <td>54.1</td>
    <td>—</td>
    <td><b>58.5</b></td>
    <td>46.3</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>—</td>
    <td>64.5</td>
    <td><b>70.8</b></td>
    <td>—</td>
    <td>53.4</td>
    <td><b>56.1</b></td>
    <td>—</td>
    <td>36.0</td>
    <td><b>42.2</b></td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>67.9</td>
    <td><b>68.6</b></td>
    <td>—</td>
    <td><b>51.1</b></td>
    <td>50.4</td>
    <td>—</td>
    <td><b>39.4</b></td>
    <td>36.9</td>
    <td>—</td>
  </tr>
</table>

Table 6. Performance comparison (%) of three LLMs on Turkic languages with their native writing systems: Arabic and Latin for Uyghur, Cyrillic and Latin for Kazakh, Crimean Tatar, and Uzbek. Bold numbers indicate the best script performance per language-model pair. Dashes (—) denote script combinations not used in practice.

contains 616,563,348 Latin and 492,264,125 Cyrillic words in Uzbek.\footnote{In this work, Uzbek refers to Northern Uzbek.}

While these patterns hold across multiple models, there are exceptions. For example, on Uyghur questions, GPT-4o performs similarly with Arabic and Latin scripts. Llama 3.1 70B has an average accuracy of 28.48 on Uyghur questions with Arabic script and 41.30 with Latin script.

6 Conclusion

We introduce TUMLU, a unified and native language understanding benchmark for Turkic languages. It contains 38139 multiple-choice questions in 8 languages and 11 subjects. Latin, Cyrillic, and Arabic scripts are represented in the benchmark. Uzbek, Crimean Tatar, and Kazakh are available in both Cyrillic and Latin. Uyghur is available both in Arabic and Latin. We also release TUMLU-mini, a smaller, more balanced and manually verified version that is more suitable for large-scale experiments. We use TUMLU-mini to benchmark 5 proprietary and 7 open-source LLMs. Results show that LLMs have a reasonably good understanding of all 8 languages, including ones that are not explicitly included in the training data of LLMs. However, LLMs are less capable of generating text in these languages, usually answering multiple-choice questions correctly, but in another, similar high-resource language.

7 Limitations

TUMLU benchmark has two main limitations.

Mismatched difficulty levels Native language & literature subset contained both literature and language questions in some languages, while it contained only language questions in others. Similarly, the history subset contained both world and national history questions in some languages, while it contained only national questions in others. Maths questions in Kazakh are at middle-school level, which results in very high scores.

Missing major languages TUMLU covers 8 Turkic languages with more than 180 million native speakers. However, some major Turkic languages, such as Turkmen, Kyrgyz and Bashkir are not included. We are hoping to extend our benchmark with more languages in further editions.

Acknowledgments

We thank the Ukrainian Center for Educational Quality Assessment for providing exam materials in Crimean Tatar. We thank the Google Gemini Academic Program for supporting our experiments with cloud credits. We thank the Azerbaijan Cultural Society of Northern California for their financial support.

References

Anthropic. 2024. Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet.

Edward Bayes, Israel Abebe Azime, Jesujoba O Alabi, Jonas Kgomu, Tyne Eloundou, Elizabeth Proehl, Kai Chen, Imaan Khadir, Naome A Etori, Shamsuddeen Hassan Muhammad, et al. 2024. Uhura: A benchmark for evaluating scientific question answering and truthfulness in low-resource african languages. arXiv preprint arXiv:2412.00948.

M. Ali Bayram, Ali Arda Fincan, Ahmet Semih Gümitş, Banu Diri, Savaş Yildirim, and Öner Aytas. 2025. Setting Standards in Turkish NLP: TR-MMLU for Large Language Model Evaluation. Preprint, arXiv:2501.00593.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens
Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.

Gemini Team. 2024. Gemini 1.5: Unlocking multi-modal understanding across millions of tokens of context. Preprint, arXiv:2403.05530.

Gemma Team. 2024. Gemma 2: Improving Open Language Models at a Practical Size. Preprint, arXiv:2408.00118.

Aaron Grattafiori et al. 2024. The Llama 3 Herd of Models. Preprint, arXiv:2407.21783.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. 2017. Deep Learning Scaling is Predictable, Empirically. Preprint, arXiv:1712.00409.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os-trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Jafar Isbarov, Kaysar Huseynova, Elvin Mammadov, Mammad Hajili, and Duygu Ataman. 2024. Open foundation models for Azerbaijani language. In Proceedings of the First Workshop on Natural Language Processing for Turkic Languages (SIGTURK 2024), pages 18–28, Bangkok, Thailand and Online. Association for Computational Linguistics.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. Preprint, arXiv:2001.08361.

Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Al-mubarak, Zaid Alyafeai, Neha Sengupta, Shady She-hata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguistics.

Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. arXiv preprint arXiv:2307.16039.

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. CMMLU: Measuring massive multitask language understanding in Chinese. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11260–11285, Bangkok, Thailand. Association for Computational Linguistics.

Akylbek Maxutov, Ayan Myrzakhet, and Pavel Braslavski. 2024. Do LLMs speak Kazakh? a pilot evaluation of seven models. In Proceedings of the First Workshop on Natural Language Processing for Turkic Languages (SIGTURK 2024), pages 81–91, Bangkok, Thailand and Online. Association for Computational Linguistics.

Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro von Werra, and Thomas Wolf. 2024. Fineweb2: A sparkling update with 1000s of languages.

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.

Angelika Romanou, Negar Foroutan, Anna Sotnikova, Sree Harsha Nelaturu, Shivalka Singh, Rishabh Maheshwary, Micol Altomare, Zeming Chen, Mohamed A. Haggag, Snegha A., Alfonso Amayuelas, Azril Hafizi Amirudin, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duwal, Daniil Dzenhalio, Daniel Fernando Erazo Florez, Fabian Farestam, Joseph Marvin Imperial, Shayekh Bin Islam, Perttu Isotalo, Maral Jabbarishiviari, Börje F. Karlsson, Eldar Khalilov, Christopher Klamm, Fajri Koto, Dominik Krzeminski, Gabriel Adriano de Melo, Syrielle Montariol, Yiyang Nan, Joel Niklaus, Jekaterina Novikova, Johan Samir Obando Ceron, Debiit Paul, Esther Ploeger, Jebish Purvey, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell,
Roshan Santhosh, Drishti Sharma, Marjana Prifti Skenduli, Arshia Soltani Moakhar, Bardia soltani moakhar, Ayush Kumar Tarun, Azmine Toushik Wasi, Thenuka Ovin Weerasekinghe, Serhan Yilmaz, Mike Zhang, Imanol Schlag, Marzieh Fadaee, Sara Hooker, and Antoine Bosselut. 2025. INCLUDE: Evaluating multilingual language understanding with regional knowledge. In The Thirteenth International Conference on Learning Representations.

Ali Safaya, Emirhan Kurtulış, Arda Goktogan, and Deniz Yuret. 2022. Mukayese: Turkish NLP strikes back. In Findings of the Association for Computational Linguistics: ACL 2022, pages 846–863, Dublin, Ireland. Association for Computational Linguistics.

Lütfi Kerem Senel, Benedikt Ebing, Konul Baghirova, Hinrich Schütze, and Goran Glavaš. 2024. Kardeş- NLU: Transfer to low-resource languages with the help of a high-resource cousin – a benchmark and evaluation for Turkic languages. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1672–1688, St. Julian’s, Malta. Association for Computational Linguistics.

Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngu, Daniel Vila-Suero, Peerat Limkumchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Madeline Smith, Antoine Bosselut, Alice Oh, Andre F. T. Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, and Sara Hooker. 2024. Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluation. Preprint, arXiv:2412.03304.

Zayne Rea Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. 2024. MuSR: Testing the limits of chain-of-thought with multistep soft reasoning. In The Twelfth International Conference on Learning Representations.

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.

Francesco Tinner, Raghav Mantri, Mamad Hajili, Chiamaka Chukwunke, Dylan Massey, Benjamin A. Ajibade, Bilge Deniz Kocak, Abolade Dawud, Jonathan Atala, Hale Sirin, Kayode Olaleye, Anar Rzayev, Jafar Isbarov, Dursun Dashdamirov, David Adelani, and Duygu Ataman. 2024. Findings of the 2nd shared task on multi-lingual multi-task information retrieval at MRL 2024. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 365–376, Miami, Florida, USA. Association for Computational Linguistics.

Mukhammed Togmanov, Nurdaulet Mukhituly, Diana Turmakhon, Jonibek Mansurov, Maiya Goloburda, Akhmed Sakip, Zhuohan Xie, Yuxia Wang, Bekasyl Syzydkov, Nurkanh Laiyk, Alham Fikri Aji, Ekaterina Kochmar, Preslav Nakov, and Fajri Koto. 2025. KazMMLU: Evaluating Language Models on Kazakh, Russian, and Regional Knowledge of Kazakhstan. Preprint, arXiv:2502.12829.

Eva Vanmassenhove, Dimitar Shterionov, and Matthew Gwilliam. 2021. Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2203–2213, Online. Association for Computational Linguistics.

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: a stickier benchmark for general-purpose language understanding systems. Curran Associates Inc., Red Hook, NY, USA.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics.

Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, AiTi Aw, and Nancy Chen. 2024a. SeaEval for multilingual foundation models: From cross-lingual alignment to cultural reasoning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 370–390, Mexico City, Mexico. Association for Computational Linguistics.

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhui Chen. 2024b. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. Preprint, arXiv:2406.01574.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2024. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu,
Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 Technical Report. Preprint, arXiv:2412.15115.

Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Senel, Anna Korhonen, and Hinrich Schutze. 2024. TurkishMMLU: Measuring massive multitask language understanding in Turkish. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7035–7055, Miami, Florida, USA. Association for Computational Linguistics.
A  Prompt samples
Sual: 1 saatda 1 bakteriyadan neçə nəsil törəyər?
A) 1 nəsil
B) 8 nəsil
C) 3 nəsil
D) 9 nəsil

Cavab: C

Sual: Difteriyaya tutulan 50 nəfərdən heç biri müalicə serumu almazsa onda onların neçə nəfəri ölər? (10 nəfərdən 3-4 sağ qalır)
A) 12-14
B) 3-4
C) 6-7
D) 30-35

Cavab: D

Sual: Mitoz bölünmədən əvvəl bitki hüceyrəsində 24 xromosom olarsa əməle gələn cavan hüceyrələrin har birində neçə xromosom olar?
A) 24
B) 12
C) 6
D) 48

Cavab: D

Sual: İkiqat mayalanmaya hazırlıq merhələsində əməle gelen tozcuqların nüvesi bölündükdən sonra cəmi 186 nüvə yaranır. Neçə tozcuq hüceyrəsi bu bölünmədə iştirak etmişdir?
A) 186
B) 185
C) 372
D) 93

Cavab: D

Sual: İnsan bütün ömrü boyu maksimum neçə yeni dış çıxarır?
A) 28
B) 12
C) 52
D) 32

Cavab: C

Sual: Selülozanın inşaat funksiyasına nə aiddir?
A) Qan damarlarının divarı
B) inşaat funksiyası yoxdur
C) Bitki hüceyrələrinin qilafı
D) Plazmatik membran

Cavab:
Sual: 1 saatda 1 bakteriyadan neçə nəsil töreyər?
A) 3 nəsil
B) 9 nəsil
C) 8 nəsil
D) 1 nəsil

Həll: Addım-addım düşünək. Bakteriyalar hər 20 dəqiqədə bir defə çoxalır. 1 saat 60 dəqiqədir. 1 bakteriyalar 1 saatda 3 nəsil töreyər. Düzgün cavab A variantıdır.

Sual: İkiqat mayalanmaya hazırlıq mərhələsində aməle gələn tozcuqların nüvəsi bölündükdən sonra cəmi 186 nüvə yaranır. Neçə tozcuq hüceyrəsi bu bölünmədə iştirak etmişdir?
A) 185
B) 186
C) 93
D) 372

Həll: Addım-addım düşünək. İkiqat mayalanmayanın hazırlıq mərhələsində hər tozcuq mitoz bölünmə keçirir. Bilirik ki, mitoz bölünmə nəticəsində hüceyrənin nüvəsi ikiyə bölünür. Bu o deməkdir ki, mitoz bölünmə zamanı 186 nüvə yaranmışdırsa, deməli 93 tozcuq hüceyrəsi bu bölünmədə iştirak etmişdir. Düzgün cavab C variantıdır.

...

Sual: İnsan bütün ömrü boyu maksimum neçə yeni dış çıxarır?
A) 52
B) 32
C) 12
D) 28

Həll: Addım-addım düşünək. İnsanlar iki dəfə dış çıxarır:
1. Süd dişleri (20 adəd): Körpəlikdə çıxan və daha sonra tökülen dişlərdir.
2. Daimi dişler (32 adəd): Süd dişleri töküldükdən sonra çıxan və ömr boyu qalan dişlərdir. Bunlara 4 adəd ağız boşluğu dişi (ağılli dişlər) da daxildir.
insanın ömrü boyu çıxara bilecəyi maksimum yeni diş sayı 52-dir. Düzgün cavab: A variantıdır.

Sual: Selülozanın inşaat funksiyasına nə aiddir?
A) İnşaat funksiyası yoxdur
B) Plazmatik membran
C) Qan damarlarının divarı
D) Bitki hüceyrələrinin qilafı

Həll:
B   TUMLU-mini
<table>
  <tr>
    <th>Language (code)</th>
    <th>NL&amp;L</th>
    <th>History</th>
    <th>Geography</th>
    <th>Chemistry</th>
    <th>Physics</th>
    <th>Biology</th>
    <th>Maths</th>
  </tr>
  <tr>
    <td>Azerbaijani (aze)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Crimean Tatar (crh)</td>
    <td>100</td>
    <td><b>69</b></td>
    <td><b>23</b></td>
    <td><b>32</b></td>
    <td><b>39</b></td>
    <td><b>38</b></td>
    <td><b>54</b></td>
  </tr>
  <tr>
    <td>Karakalpak (kaa)</td>
    <td>64</td>
    <td>0</td>
    <td>28</td>
    <td>28</td>
    <td>45</td>
    <td>50</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Kazakh (kaz)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Kyrgyz (kir)</td>
    <td>100</td>
    <td>98</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Tatar (tat)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Turkish (tur)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Uyghur (uig)</td>
    <td>100</td>
    <td>0</td>
    <td>0</td>
    <td>97</td>
    <td>98</td>
    <td>100</td>
    <td>99</td>
  </tr>
  <tr>
    <td>Uzbek (uzb)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
</table>

Table 7. Composition of TUMLU-mini, a more balanced and manually verified subset of TUMLU benchmark. All experiments in this paper have been run on TUMLU-mini. These numbers exclude sample questions used in 5-shot and 5-shot CoT prompts. Language codes are from ISO 639-3.
C   5-shot results

This appendix includes 5-shot results for all models, except for Claude 3.5 Sonnet which is available in Table 3.
<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>78.00</td>
    <td>79.00</td>
    <td>72.00</td>
    <td>61.00</td>
    <td>65.00</td>
    <td>58.00</td>
    <td>81.00</td>
  </tr>
  <tr>
    <td>Crimean Tatar</td>
    <td>63.16</td>
    <td>53.12</td>
    <td>73.91</td>
    <td>56.52</td>
    <td>57.41</td>
    <td>62.00</td>
    <td>65.52</td>
  </tr>
  <tr>
    <td>Karakalpak</td>
    <td>60.00</td>
    <td>60.71</td>
    <td>53.57</td>
    <td>-</td>
    <td>-</td>
    <td>20.31</td>
    <td>80.00</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>79.00</td>
    <td>66.00</td>
    <td>72.00</td>
    <td>64.00</td>
    <td>76.00</td>
    <td>58.00</td>
    <td>74.00</td>
  </tr>
  <tr>
    <td>Tatar</td>
    <td>74.00</td>
    <td>63.00</td>
    <td>73.00</td>
    <td>74.00</td>
    <td>63.16</td>
    <td>59.00</td>
    <td>65.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>71.00</td>
    <td>82.00</td>
    <td>84.00</td>
    <td>85.00</td>
    <td>71.00</td>
    <td>70.00</td>
    <td>80.00</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>57.00</td>
    <td>44.33</td>
    <td>-</td>
    <td>-</td>
    <td>59.60</td>
    <td>52.00</td>
    <td>59.18</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>53.00</td>
    <td>50.00</td>
    <td>48.00</td>
    <td>49.00</td>
    <td>54.00</td>
    <td>41.00</td>
    <td>55.00</td>
  </tr>
</table>

Table 8. Accuracy scores for Claude 3.5 Haiku-20241022 model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>80.00</td>
    <td>80.00</td>
    <td>78.00</td>
    <td>55.00</td>
    <td>72.00</td>
    <td>56.00</td>
    <td>84.00</td>
  </tr>
  <tr>
    <td>Crimean Tatar</td>
    <td>78.95</td>
    <td>59.38</td>
    <td>73.91</td>
    <td>59.42</td>
    <td>61.11</td>
    <td>67.00</td>
    <td>72.41</td>
  </tr>
  <tr>
    <td>Karakalpak</td>
    <td>70.00</td>
    <td>82.14</td>
    <td>53.57</td>
    <td>-</td>
    <td>-</td>
    <td>31.25</td>
    <td>68.89</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>88.00</td>
    <td>60.00</td>
    <td>73.00</td>
    <td>57.00</td>
    <td>68.00</td>
    <td>57.00</td>
    <td>76.00</td>
  </tr>
  <tr>
    <td>Tatar</td>
    <td>86.00</td>
    <td>68.00</td>
    <td>74.00</td>
    <td>68.00</td>
    <td>64.21</td>
    <td>47.00</td>
    <td>60.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>75.00</td>
    <td>72.00</td>
    <td>78.00</td>
    <td>76.00</td>
    <td>78.00</td>
    <td>59.00</td>
    <td>74.00</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>71.00</td>
    <td>53.61</td>
    <td>-</td>
    <td>-</td>
    <td>59.60</td>
    <td>57.00</td>
    <td>45.92</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>59.00</td>
    <td>43.00</td>
    <td>49.00</td>
    <td>57.00</td>
    <td>69.00</td>
    <td>22.00</td>
    <td>51.00</td>
  </tr>
</table>

Table 9. Accuracy scores for GEMINI-1.5-FLASH model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>81.00</td>
    <td>86.00</td>
    <td>76.00</td>
    <td>56.00</td>
    <td>78.00</td>
    <td>57.00</td>
    <td>89.00</td>
  </tr>
  <tr>
    <td>Crimean Tatar</td>
    <td>71.05</td>
    <td>50.00</td>
    <td>65.22</td>
    <td>56.52</td>
    <td>61.11</td>
    <td>52.00</td>
    <td>62.07</td>
  </tr>
  <tr>
    <td>Karakalpak</td>
    <td>76.00</td>
    <td>71.43</td>
    <td>57.14</td>
    <td>-</td>
    <td>-</td>
    <td>43.75</td>
    <td>88.89</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>88.00</td>
    <td>68.00</td>
    <td>75.00</td>
    <td>73.00</td>
    <td>94.00</td>
    <td>70.00</td>
    <td>80.00</td>
  </tr>
  <tr>
    <td>Tatar</td>
    <td>95.00</td>
    <td>78.00</td>
    <td>81.00</td>
    <td>84.00</td>
    <td>80.00</td>
    <td>59.00</td>
    <td>64.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>51.00</td>
    <td>61.00</td>
    <td>61.00</td>
    <td>70.00</td>
    <td>64.00</td>
    <td>50.00</td>
    <td>57.00</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>70.00</td>
    <td>42.27</td>
    <td>-</td>
    <td>-</td>
    <td>49.49</td>
    <td>66.00</td>
    <td>51.02</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>59.00</td>
    <td>69.00</td>
    <td>61.00</td>
    <td>54.00</td>
    <td>79.00</td>
    <td>30.00</td>
    <td>76.00</td>
  </tr>
</table>

Table 10. Accuracy scores for GEMINI-1.5-PRO model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>61.00</td>
    <td>57.00</td>
    <td>59.00</td>
    <td>47.00</td>
    <td>45.00</td>
    <td>42.00</td>
    <td>65.00</td>
  </tr>
  <tr>
    <td>Crimean Tatar</td>
    <td>50.00</td>
    <td>37.50</td>
    <td>56.52</td>
    <td>52.17</td>
    <td>33.33</td>
    <td>53.00</td>
    <td>44.83</td>
  </tr>
  <tr>
    <td>Karakalpak</td>
    <td>48.00</td>
    <td>46.43</td>
    <td>35.71</td>
    <td>-</td>
    <td>-</td>
    <td>25.00</td>
    <td>48.89</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>67.00</td>
    <td>37.00</td>
    <td>63.00</td>
    <td>41.00</td>
    <td>38.00</td>
    <td>48.00</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td>Tatar</td>
    <td>69.00</td>
    <td>53.00</td>
    <td>63.00</td>
    <td>54.00</td>
    <td>35.79</td>
    <td>41.00</td>
    <td>47.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>65.00</td>
    <td>55.00</td>
    <td>76.00</td>
    <td>75.00</td>
    <td>45.00</td>
    <td>48.00</td>
    <td>57.00</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>40.00</td>
    <td>26.80</td>
    <td>-</td>
    <td>-</td>
    <td>37.37</td>
    <td>38.00</td>
    <td>36.73</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>49.00</td>
    <td>33.00</td>
    <td>39.00</td>
    <td>41.00</td>
    <td>31.00</td>
    <td>26.00</td>
    <td>34.00</td>
  </tr>
</table>

Table 11. Accuracy scores for GOOGLE/GEMMA-2-9B-IT model across languages.
<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>70.00</td>
    <td>59.00</td>
    <td>62.00</td>
    <td>46.00</td>
    <td>38.00</td>
    <td>55.00</td>
    <td>77.00</td>
  </tr>
  <tr>
    <td>Crimean Tatar</td>
    <td>44.74</td>
    <td>46.88</td>
    <td>60.87</td>
    <td>49.28</td>
    <td>35.19</td>
    <td>60.00</td>
    <td>51.72</td>
  </tr>
  <tr>
    <td>Karakalpak</td>
    <td>52.00</td>
    <td>64.29</td>
    <td>42.86</td>
    <td>-</td>
    <td>-</td>
    <td>23.44</td>
    <td>55.56</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>79.00</td>
    <td>44.00</td>
    <td>65.00</td>
    <td>56.00</td>
    <td>52.00</td>
    <td>51.00</td>
    <td>62.00</td>
  </tr>
  <tr>
    <td>Tatar</td>
    <td>71.00</td>
    <td>58.00</td>
    <td>71.00</td>
    <td>63.00</td>
    <td>43.16</td>
    <td>32.00</td>
    <td>45.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>73.00</td>
    <td>65.00</td>
    <td>81.00</td>
    <td>78.00</td>
    <td>41.00</td>
    <td>55.00</td>
    <td>57.00</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>50.00</td>
    <td>35.05</td>
    <td>-</td>
    <td>-</td>
    <td>34.34</td>
    <td>51.00</td>
    <td>40.82</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>46.00</td>
    <td>27.00</td>
    <td>40.00</td>
    <td>44.00</td>
    <td>35.00</td>
    <td>26.00</td>
    <td>40.00</td>
  </tr>
</table>

Table 12. Accuracy scores for GOOGLE/GEMMA-2-27B-IT model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>91.00</td>
    <td>93.00</td>
    <td>89.00</td>
    <td>75.00</td>
    <td>70.00</td>
    <td>67.00</td>
    <td>92.00</td>
  </tr>
  <tr>
    <td>Crimean Tatar</td>
    <td>60.53</td>
    <td>59.38</td>
    <td>69.57</td>
    <td>86.96</td>
    <td>57.41</td>
    <td>70.00</td>
    <td>89.66</td>
  </tr>
  <tr>
    <td>Karakalpak</td>
    <td>80.00</td>
    <td>82.14</td>
    <td>71.43</td>
    <td>-</td>
    <td>-</td>
    <td>35.94</td>
    <td>84.44</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>93.00</td>
    <td>71.00</td>
    <td>76.00</td>
    <td>77.00</td>
    <td>85.00</td>
    <td>77.00</td>
    <td>88.00</td>
  </tr>
  <tr>
    <td>Tatar</td>
    <td>98.00</td>
    <td>78.00</td>
    <td>88.00</td>
    <td>92.00</td>
    <td>69.47</td>
    <td>69.00</td>
    <td>68.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>86.00</td>
    <td>79.00</td>
    <td>95.00</td>
    <td>94.00</td>
    <td>63.00</td>
    <td>82.00</td>
    <td>87.00</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>84.00</td>
    <td>54.64</td>
    <td>-</td>
    <td>-</td>
    <td>62.63</td>
    <td>65.00</td>
    <td>66.33</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>70.00</td>
    <td>68.00</td>
    <td>65.00</td>
    <td>69.00</td>
    <td>56.00</td>
    <td>51.00</td>
    <td>79.00</td>
  </tr>
</table>

Table 13. Accuracy scores for GPT-4o model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>76.00</td>
    <td>72.00</td>
    <td>77.00</td>
    <td>59.00</td>
    <td>45.00</td>
    <td>49.00</td>
    <td>84.00</td>
  </tr>
  <tr>
    <td>Crimean Tatar</td>
    <td>68.42</td>
    <td>53.12</td>
    <td>69.57</td>
    <td>72.46</td>
    <td>40.74</td>
    <td>58.00</td>
    <td>48.28</td>
  </tr>
  <tr>
    <td>Karakalpak</td>
    <td>58.00</td>
    <td>46.43</td>
    <td>64.29</td>
    <td>-</td>
    <td>-</td>
    <td>21.88</td>
    <td>55.56</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>80.00</td>
    <td>42.00</td>
    <td>71.00</td>
    <td>62.00</td>
    <td>52.00</td>
    <td>51.00</td>
    <td>60.00</td>
  </tr>
  <tr>
    <td>Tatar</td>
    <td>88.00</td>
    <td>67.00</td>
    <td>83.00</td>
    <td>82.00</td>
    <td>67.37</td>
    <td>51.00</td>
    <td>48.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>76.00</td>
    <td>58.00</td>
    <td>86.00</td>
    <td>83.00</td>
    <td>41.00</td>
    <td>65.00</td>
    <td>68.00</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>37.00</td>
    <td>46.39</td>
    <td>-</td>
    <td>-</td>
    <td>48.48</td>
    <td>49.00</td>
    <td>46.94</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>52.00</td>
    <td>38.00</td>
    <td>52.00</td>
    <td>50.00</td>
    <td>44.00</td>
    <td>31.00</td>
    <td>42.00</td>
  </tr>
</table>

Table 14. Accuracy scores for META-LLAMA/LLAMA-3.3-70B-INSTRUCT model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>53.00</td>
    <td>50.00</td>
    <td>57.00</td>
    <td>45.00</td>
    <td>36.00</td>
    <td>47.00</td>
    <td>51.00</td>
  </tr>
  <tr>
    <td>Crimean Tatar</td>
    <td>42.11</td>
    <td>34.38</td>
    <td>30.43</td>
    <td>44.93</td>
    <td>18.52</td>
    <td>45.00</td>
    <td>34.48</td>
  </tr>
  <tr>
    <td>Karakalpak</td>
    <td>38.00</td>
    <td>39.29</td>
    <td>32.14</td>
    <td>-</td>
    <td>-</td>
    <td>21.88</td>
    <td>35.56</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>60.00</td>
    <td>33.00</td>
    <td>64.00</td>
    <td>54.00</td>
    <td>32.00</td>
    <td>41.00</td>
    <td>41.00</td>
  </tr>
  <tr>
    <td>Tatar</td>
    <td>55.00</td>
    <td>40.00</td>
    <td>61.00</td>
    <td>55.00</td>
    <td>33.68</td>
    <td>33.00</td>
    <td>31.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>51.00</td>
    <td>37.00</td>
    <td>63.00</td>
    <td>50.00</td>
    <td>34.00</td>
    <td>35.00</td>
    <td>41.00</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>38.00</td>
    <td>27.84</td>
    <td>-</td>
    <td>-</td>
    <td>36.36</td>
    <td>42.00</td>
    <td>30.61</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>31.00</td>
    <td>22.00</td>
    <td>38.00</td>
    <td>33.00</td>
    <td>25.00</td>
    <td>29.00</td>
    <td>31.00</td>
  </tr>
</table>

Table 15. Accuracy scores for META-LLAMA/META-LLAMA-3.1-8B-INSTRUCT model across languages.
<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>78.00</td>
    <td>67.00</td>
    <td>78.00</td>
    <td>60.00</td>
    <td>48.00</td>
    <td>62.00</td>
    <td>80.00</td>
  </tr>
  <tr>
    <td>Crimean Tatar</td>
    <td>63.16</td>
    <td>40.62</td>
    <td>65.22</td>
    <td>66.67</td>
    <td>29.63</td>
    <td>62.00</td>
    <td>65.52</td>
  </tr>
  <tr>
    <td>Karakalpak</td>
    <td>58.00</td>
    <td>53.57</td>
    <td>60.71</td>
    <td>-</td>
    <td>-</td>
    <td>28.12</td>
    <td>44.44</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>57.00</td>
    <td>27.00</td>
    <td>55.00</td>
    <td>57.00</td>
    <td>40.00</td>
    <td>50.00</td>
    <td>48.00</td>
  </tr>
  <tr>
    <td>Tatar</td>
    <td>72.00</td>
    <td>23.00</td>
    <td>67.00</td>
    <td>72.00</td>
    <td>49.47</td>
    <td>52.00</td>
    <td>47.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>70.00</td>
    <td>55.00</td>
    <td>88.00</td>
    <td>70.00</td>
    <td>40.00</td>
    <td>62.00</td>
    <td>68.00</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>24.00</td>
    <td>16.49</td>
    <td>-</td>
    <td>-</td>
    <td>20.20</td>
    <td>45.00</td>
    <td>36.73</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>42.00</td>
    <td>39.00</td>
    <td>53.00</td>
    <td>34.00</td>
    <td>36.00</td>
    <td>27.00</td>
    <td>42.00</td>
  </tr>
</table>

Table 16. Accuracy scores for META-LLAMA/META-LLAMA-3.1-70B-INSTRUCT model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>41.00</td>
    <td>58.00</td>
    <td>53.00</td>
    <td>37.00</td>
    <td>59.00</td>
    <td>30.00</td>
    <td>58.00</td>
  </tr>
  <tr>
    <td>Crimean Tatar</td>
    <td>36.84</td>
    <td>37.50</td>
    <td>39.13</td>
    <td>39.13</td>
    <td>55.56</td>
    <td>42.00</td>
    <td>48.28</td>
  </tr>
  <tr>
    <td>Karakalpak</td>
    <td>30.00</td>
    <td>42.86</td>
    <td>39.29</td>
    <td>-</td>
    <td>-</td>
    <td>25.00</td>
    <td>48.89</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>38.00</td>
    <td>44.00</td>
    <td>54.00</td>
    <td>31.00</td>
    <td>64.00</td>
    <td>31.00</td>
    <td>53.00</td>
  </tr>
  <tr>
    <td>Tatar</td>
    <td>41.00</td>
    <td>38.00</td>
    <td>44.00</td>
    <td>42.00</td>
    <td>56.84</td>
    <td>28.00</td>
    <td>34.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>42.00</td>
    <td>59.00</td>
    <td>62.00</td>
    <td>69.00</td>
    <td>58.00</td>
    <td>40.00</td>
    <td>59.00</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>34.00</td>
    <td>29.90</td>
    <td>-</td>
    <td>-</td>
    <td>38.38</td>
    <td>40.00</td>
    <td>24.49</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>35.00</td>
    <td>31.00</td>
    <td>30.00</td>
    <td>34.00</td>
    <td>52.00</td>
    <td>21.00</td>
    <td>39.00</td>
  </tr>
</table>

Table 17. Accuracy scores for QWEN/QWEN2.5-7B-INSTRUCT model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>76.00</td>
    <td>77.00</td>
    <td>74.00</td>
    <td>53.00</td>
    <td>73.00</td>
    <td>54.00</td>
    <td>84.00</td>
  </tr>
  <tr>
    <td>Crimean Tatar</td>
    <td>65.79</td>
    <td>53.12</td>
    <td>60.87</td>
    <td>72.46</td>
    <td>55.56</td>
    <td>66.00</td>
    <td>58.62</td>
  </tr>
  <tr>
    <td>Karakalpak</td>
    <td>50.00</td>
    <td>75.00</td>
    <td>50.00</td>
    <td>-</td>
    <td>-</td>
    <td>35.94</td>
    <td>62.22</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>60.00</td>
    <td>55.00</td>
    <td>64.00</td>
    <td>52.00</td>
    <td>75.00</td>
    <td>52.00</td>
    <td>79.00</td>
  </tr>
  <tr>
    <td>Tatar</td>
    <td>68.00</td>
    <td>60.00</td>
    <td>65.00</td>
    <td>78.00</td>
    <td>71.58</td>
    <td>41.00</td>
    <td>54.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>79.00</td>
    <td>73.00</td>
    <td>84.00</td>
    <td>85.00</td>
    <td>61.00</td>
    <td>56.00</td>
    <td>79.00</td>
  </tr>
  <tr>
    <td>Uyghur</td>
    <td>60.00</td>
    <td>51.55</td>
    <td>-</td>
    <td>-</td>
    <td>64.65</td>
    <td>52.00</td>
    <td>52.04</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>53.00</td>
    <td>49.00</td>
    <td>44.00</td>
    <td>55.00</td>
    <td>63.00</td>
    <td>28.00</td>
    <td>61.00</td>
  </tr>
</table>

Table 18. Accuracy scores for QWEN/Qwen2.5 72B-INSTRUCT model across languages.
D   5-shot CoT results

This appendix includes 5-shot CoT results for all models, except for Claude 3.5 Sonnet which is available in Table 5.
<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>81.00</td>
    <td>83.00</td>
    <td>81.00</td>
    <td>69.00</td>
    <td>76.00</td>
    <td>61.00</td>
    <td>88.00</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>82.00</td>
    <td>65.00</td>
    <td>75.00</td>
    <td>61.00</td>
    <td>87.00</td>
    <td>64.00</td>
    <td>84.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>74.00</td>
    <td>78.00</td>
    <td>91.00</td>
    <td>85.00</td>
    <td>28.00</td>
    <td>71.00</td>
    <td>68.00</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>50.00</td>
    <td>50.00</td>
    <td>37.00</td>
    <td>47.00</td>
    <td>63.00</td>
    <td>24.00</td>
    <td>53.00</td>
  </tr>
</table>

Table 19. Accuracy scores for Claude 3.5 Haiku-20241022 model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>81.00</td>
    <td>83.00</td>
    <td>81.00</td>
    <td>52.00</td>
    <td>71.00</td>
    <td>59.00</td>
    <td>88.00</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>79.00</td>
    <td>61.00</td>
    <td>69.00</td>
    <td>51.00</td>
    <td>85.00</td>
    <td>50.00</td>
    <td>78.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>71.00</td>
    <td>67.00</td>
    <td>69.00</td>
    <td>69.00</td>
    <td>76.00</td>
    <td>60.00</td>
    <td>74.00</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>50.00</td>
    <td>56.00</td>
    <td>41.00</td>
    <td>49.00</td>
    <td>70.00</td>
    <td>17.00</td>
    <td>67.00</td>
  </tr>
</table>

Table 20. Accuracy scores for GEMINI-1.5-FLASH model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>79.00</td>
    <td>92.00</td>
    <td>78.00</td>
    <td>53.00</td>
    <td>78.00</td>
    <td>62.00</td>
    <td>80.00</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>83.00</td>
    <td>70.00</td>
    <td>63.00</td>
    <td>65.00</td>
    <td>92.00</td>
    <td>67.00</td>
    <td>83.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>73.00</td>
    <td>76.00</td>
    <td>84.00</td>
    <td>79.00</td>
    <td>85.00</td>
    <td>51.00</td>
    <td>77.00</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>53.00</td>
    <td>67.00</td>
    <td>43.00</td>
    <td>40.00</td>
    <td>73.00</td>
    <td>17.00</td>
    <td>81.00</td>
  </tr>
</table>

Table 21. Accuracy scores for GEMINI-1.5-PRO model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>64.00</td>
    <td>67.00</td>
    <td>65.00</td>
    <td>39.00</td>
    <td>45.00</td>
    <td>47.00</td>
    <td>73.00</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>61.00</td>
    <td>42.00</td>
    <td>64.00</td>
    <td>46.00</td>
    <td>61.00</td>
    <td>32.00</td>
    <td>62.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>72.00</td>
    <td>60.00</td>
    <td>74.00</td>
    <td>71.00</td>
    <td>45.00</td>
    <td>50.00</td>
    <td>64.00</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>41.00</td>
    <td>31.00</td>
    <td>40.00</td>
    <td>41.00</td>
    <td>26.00</td>
    <td>25.00</td>
    <td>32.00</td>
  </tr>
</table>

Table 22. Accuracy scores for GOOGLE/GEMMA-2-9B-IT model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>72.00</td>
    <td>76.00</td>
    <td>71.00</td>
    <td>41.00</td>
    <td>55.00</td>
    <td>48.00</td>
    <td>77.00</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>74.00</td>
    <td>52.00</td>
    <td>63.00</td>
    <td>46.00</td>
    <td>70.00</td>
    <td>53.00</td>
    <td>73.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>77.00</td>
    <td>64.00</td>
    <td>80.00</td>
    <td>77.00</td>
    <td>53.00</td>
    <td>49.00</td>
    <td>65.00</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>43.00</td>
    <td>43.00</td>
    <td>48.00</td>
    <td>53.00</td>
    <td>42.00</td>
    <td>10.00</td>
    <td>49.00</td>
  </tr>
</table>

Table 23. Accuracy scores for GOOGLE/GEMMA-2-27B-IT model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>90.00</td>
    <td>91.00</td>
    <td>88.00</td>
    <td>73.00</td>
    <td>75.00</td>
    <td>70.00</td>
    <td>93.00</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>89.00</td>
    <td>63.00</td>
    <td>78.00</td>
    <td>82.00</td>
    <td>85.00</td>
    <td>84.00</td>
    <td>84.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>86.00</td>
    <td>80.00</td>
    <td>97.00</td>
    <td>92.00</td>
    <td>70.00</td>
    <td>80.00</td>
    <td>83.00</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>73.00</td>
    <td>68.00</td>
    <td>70.00</td>
    <td>74.00</td>
    <td>59.00</td>
    <td>39.00</td>
    <td>79.00</td>
  </tr>
</table>

Table 24. Accuracy scores for GPT-4o model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>81.00</td>
    <td>78.00</td>
    <td>75.00</td>
    <td>54.00</td>
    <td>70.00</td>
    <td>52.00</td>
    <td>79.00</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>80.00</td>
    <td>57.00</td>
    <td>70.00</td>
    <td>65.00</td>
    <td>74.00</td>
    <td>59.00</td>
    <td>80.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>74.00</td>
    <td>67.00</td>
    <td>84.00</td>
    <td>87.00</td>
    <td>77.00</td>
    <td>67.00</td>
    <td>75.00</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>46.00</td>
    <td>15.00</td>
    <td>8.00</td>
    <td>33.00</td>
    <td>35.00</td>
    <td>10.00</td>
    <td>14.00</td>
  </tr>
</table>

Table 25. Accuracy scores for META-LLAMA/LLAMA-3.3-70B-INSTRUCT model across languages.
<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>43.00</td>
    <td>44.00</td>
    <td>57.00</td>
    <td>36.00</td>
    <td>20.00</td>
    <td>32.00</td>
    <td>53.00</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>52.00</td>
    <td>28.00</td>
    <td>55.00</td>
    <td>46.00</td>
    <td>17.00</td>
    <td>32.00</td>
    <td>42.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>50.00</td>
    <td>37.00</td>
    <td>58.00</td>
    <td>61.00</td>
    <td>20.00</td>
    <td>39.00</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>34.00</td>
    <td>5.00</td>
    <td>37.00</td>
    <td>29.00</td>
    <td>9.00</td>
    <td>23.00</td>
    <td>9.00</td>
  </tr>
</table>

Table 26. Accuracy scores for META-LLAMA/META-LLAMA-3.1-8B-INSTRUCT model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>52.00</td>
    <td>67.00</td>
    <td>69.00</td>
    <td>49.00</td>
    <td>55.00</td>
    <td>50.00</td>
    <td>71.00</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>77.00</td>
    <td>51.00</td>
    <td>65.00</td>
    <td>65.00</td>
    <td>56.00</td>
    <td>53.00</td>
    <td>65.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>72.00</td>
    <td>64.00</td>
    <td>83.00</td>
    <td>88.00</td>
    <td>69.00</td>
    <td>60.00</td>
    <td>77.00</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>31.00</td>
    <td>24.00</td>
    <td>8.00</td>
    <td>20.00</td>
    <td>12.00</td>
    <td>11.00</td>
    <td>15.00</td>
  </tr>
</table>

Table 27. Accuracy scores for META-LLAMA/META-LLAMA-3.1-70B-INSTRUCT model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>42.00</td>
    <td>54.00</td>
    <td>54.00</td>
    <td>37.00</td>
    <td>51.00</td>
    <td>35.00</td>
    <td>63.00</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>38.00</td>
    <td>41.00</td>
    <td>48.00</td>
    <td>31.00</td>
    <td>66.00</td>
    <td>33.00</td>
    <td>68.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>51.00</td>
    <td>59.00</td>
    <td>65.00</td>
    <td>48.00</td>
    <td>70.00</td>
    <td>43.00</td>
    <td>58.00</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>40.00</td>
    <td>30.00</td>
    <td>40.00</td>
    <td>42.00</td>
    <td>52.00</td>
    <td>17.00</td>
    <td>44.00</td>
  </tr>
</table>

Table 28. Accuracy scores for QWEN/QWEN2.5-7B-INSTRUCT model across languages.

<table>
  <tr>
    <th>Language</th>
    <th>Biology</th>
    <th>Chemistry</th>
    <th>Geography</th>
    <th>History</th>
    <th>Maths</th>
    <th>NL&amp;L</th>
    <th>Physics</th>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>72.00</td>
    <td>88.00</td>
    <td>80.00</td>
    <td>51.00</td>
    <td>80.00</td>
    <td>46.00</td>
    <td>88.00</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>64.00</td>
    <td>50.00</td>
    <td>70.00</td>
    <td>50.00</td>
    <td>84.00</td>
    <td>52.00</td>
    <td>77.00</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>78.00</td>
    <td>78.00</td>
    <td>89.00</td>
    <td>84.00</td>
    <td>79.00</td>
    <td>57.00</td>
    <td>84.00</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>54.00</td>
    <td>55.00</td>
    <td>46.00</td>
    <td>50.00</td>
    <td>49.00</td>
    <td>27.00</td>
    <td>68.00</td>
  </tr>
</table>

Table 29. Accuracy scores for QWEN/Qwen2.5 72B-INSTRUCT model across languages.

Metareview score: 4
Metareview: This paper describes TUMLU, a multi-lingual language understanding benchmark for Turkic languages. The effort is unique compared to previous approaches in that instead of gathering data for low-resource languages via translation, all the data for every language is human-generated. It consists of 80k multiple choice middle and high school questions from the curriculums of Turkic-speaking regions. A much smaller set of questions (TUMLU-mini) is hand-verified and balanced to facilitate cross-linguistic comparisons (although these comparisons remain approximate since different languages still have different questions at different difficulty levels). The authors present baseline benchmark results for a number of current state-of-the-art LLMs (Claude, LLaMa, Gemini, ChatGPT).
Summary of reasons to publish: Reviewers agree that the paper is well-written and the benchmarking methodology is thoroughly described and sound. The presented baseline experiments cover a wide range of current open and closed source LLMs. Furthermore, the paper covers under-served languages, and the dataset described in the paper should be useful for ongoing research.