Do LLMs Speak Kazakh? A Pilot Evaluation of Seven Models (2024)

Abstract: We conducted a systematic evaluation of seven large language models (LLMs) on tasks in Kazakh, a Turkic language spoken by approximately 13 million native speakers in Kazakhstan and abroad. We used six datasets corresponding to different tasks – questions answering, causal reasoning, middle school math problems, machine translation, and spelling correction. Three of the datasets were prepared for this study. As expected, the quality of the LLMs on the Kazakh tasks is lower than on the parallel English tasks. GPT-4 shows the best results, followed by Gemini and AYA. In general, LLMs perform better on classification tasks and struggle with generative tasks. Our results provide valuable insights into the applicability of currently available LLMs for Kazakh. We made the data collected for this study publicly available: https://github.com/ akylbekmaxutov/LLM-eval-using-Kazakh.

heading: 1 Introduction
text: Large Language Models (LLMs) increase human productivity and eliminate routine tasks in many areas, making them a powerful economic driver (Eloundou et al., 2023; Butler et al., 2023). At the same time, LLMs can lead to an inequality between different language communities and a widening gap between developed and developing countries (Khowaja et al., 2024). Creating LLMs requires huge amounts of text and computation, as well as skilled engineers. Most LLMs are trained for high-resource languages with large populations of speakers, primarily English. Training language models for low-resource languages can be technically and economically problematic – there is little training data, and it is unclear whether potential users can amortize the cost of collecting data and training the model. Although models trained primarily on English data express capabilities in other languages, their quality in these secondary languages is lower than in English (Ahuja et al., 2023).
Recently, thanks to the advent of open LLMs, their adaptations to less-resourced languages are emerging (Qin et al., 2024). Evaluating LLMs in different languages is crucial in this situation.
In this study, we make the first attempt to evaluate the quality of available LLMs in Kazakh. Kazakh belongs to the Turkic language family and is the official language of the Republic of Kazakhstan (Campbell and King, 2020). Estimated 10 million Kazakh native speakers live in Kazakhstan, and about 3 million more abroad, predominantly in north-western China and western Mongolia. The language employs an extended Cyrillic alphabet with 42 letters. Kazakh is an agglutinative language, meaning that words are formed by adding various suffixes to root words. The language’s rich inflectional morphology is reflected in the complex interaction of suffixes for number, possession, and case. For instance, the plural form, possessive affixes, and various case endings are layered sequentially onto noun roots. Kazakh has eight types of possessive agreements, adding complexity to its morphological structure. Kazakh verbs exhibit similar tenses and moods as Turkish ones but include unique tenses such as the goal-oriented future tense. Kazakh consonant and vowel harmony rules significantly affect its morphological structure. Consonant harmony determines the form of suffixes based on the voicing of the final consonant of the root word, while vowel harmony aligns suf-
81
fix vowels with the vowel type (front or back) of the root. Kazakh is considered a mid-resourced language (Joshi et al., 2020). Table 1 provides a brief statistics of resources available for Kazakh along with the figures for English and Turkish for comparison.
We experimented with seven models in total – five closed (GPT 3.5 and 4, Gemini 1.5 Pro, YandexGPT 2 and 3) and two open (LLAMA 2 and AYA) ones.1 We focused on automatic benchmarkbased evaluation, while trying to make the set of tasks diverse. We used a collection of six datasets sourced in different ways: 1) existing multilingual benchmarks that include Kazakh data (machine translation and multiple-choice question answering), 2) the recently published monolingual question answering dataset KazQAD (both openand closed-book scenarios), 3) machine-translated COPA dataset2 (commonsense causal reasoning), 4) original math school problems in Kazakh that we scraped online and post-processed, and 5) a Kazakh spelling correction dataset that we created from scratch within this study.
Based on our experiments, we can conclude that the GPT-4 is the most capable of all the models in the experiment. Gemini is the runner-up in the classification tasks. AYA is quite competitive, especially if we take into account its relatively small size and a long list of supported languages. All models show a lower quality in the generative tasks. As expected, the quality on Kazakh tasks is significantly lower than on English tasks, as we can see on parallel multilingual datasets (multiple-choice question answering, causal reasoning). Specialized models may still provide better quality for downstream tasks, such as machine translation or classification tasks. We cannot confirm previous findings that English prompts systematically improve LLM quality on non-English tasks: our results are mixed across tasks and models.
Our findings provide valuable insights into the applicability of currently available LLMs for Kazakh. We also anticipate that the study will contribute to the methodology of evaluating LLMs and improving the quality of LLMs in mid- and low-resource languages. The methods introduced
1mGPT (Shliazhko et al., 2024) is another LLM that officially supports Kazakh. However, only a pre-trained mGPT is available, while the models in the study are instruction tuned.
2In the spring of 2024, while our study was underway, the Kardeş-NLU for five Turkic languages, including Kazakh, was published (Senel et al., 2024). The dataset includes a post-edited version of COPA.
in our work can be used to experiment with other languages and LLMs. We made the data and evaluation code publicly available.3

heading: 2 Related Work
text: As has been shown by Blevins and Zettlemoyer (2022), multilingual abilities of language models emerge when they are exposed even to a tiny fraction of non-English data in a large pre-training corpus. Earlier studies demonstrated that multilingual models learn high-level abstractions common to all languages, which make cross-lingual transfer possible even when languages share no vocabulary (Wu and Dredze, 2019). Open LLMs such as LLAMA (Touvron et al., 2023) and Qwen (Bai et al., 2023) can be adapted to other languages by expanding their vocabularies, continual pre-training and subsequent aligning on the data in target language (Qin et al., 2024). Another approach is to train a model from scratch: for example, Jais model was trained on a mixture of English and Arabic data in ratio 2:1 (Sengupta et al., 2023). Despite the development of non-English and multilingual models, many languages remain underrepresented in the modern LLM landscape. This situation is partly due to objective reasons (lack of training data), but also to inequalities in economic and technological development.
LLM evaluation is a complex and multifaceted problem (Chang et al., 2024). LLMs are truly multitasking, and users can leverage them to solve non-standard and creative problems, for example, brainstorming ideas or generating jokes. For generative tasks, the variety of formulations can be very large, making it difficult to automatically compare the answer to a “gold standard.” With the proliferation of LLMs and their active use, evaluation of models becomes relevant not only at the task level, but also from their safety and security perspectives. The main approach to automatic LLM evaluation is based on ensembles of annotated benchmarks covering a wide range of usage scenarios (Liang et al., 2022). Popular benchmarks include MMLU (Massive Multitask Language Understanding) (Hendrycks et al., 2021) that measures LLM’s knowledge across 57 subjects and GSM8K (Grade School Math) (Cobbe et al., 2021), aimed at evaluating multi-step math reasoning. MMLU contains multiple-choice ques-
3https://github.com/akylbekmaxutov/ LLM-eval-using-Kazakh
tions, while GSM8K accepts numerical answers. There are multilingual adaptions of these datasets: Lai et al. (2023b) employed ChatGPT to translate the original MMLU dataset in multiple languages; MGSM dataset contains 250 problems from the GSM8K manually translated into 10 typologically diverse languages (Shi et al., 2023).
Studies that evaluate LLMs on non-English tasks are fewer that those targeting English and vary in their scope (Chang et al., 2024; Laskar et al., 2023). Some focus on multilingual datasets (Lai et al., 2023a; Ahuja et al., 2023), while others concentrate on a specific language, e.g. Arabic (Abdelali et al., 2024) or Russian (Fenogenova et al., 2024). Our study belongs to the latter type. LLMs, as expected, are better in solving problems formulated in English than in other languages. Moreover, fine-tuned models such as XLM-R (Conneau et al., 2020) in general outperform LLMs on specific tasks. The quality on non-English tasks can be improved by preceding actual task formulation with English prompts, or by explicitly stating in the prompt that the task must first be translated into English (Huang et al., 2023; Zhang et al., 2023). The multilingual abilities of LLMs also depend on the task type. It can be concluded that LLMs are better at “understanding” a language other than English than at generating a non-English answer (Bang et al., 2023). Thus, models do better in multilingual classification, reasoning, and multiple-choice question answering and struggle with generative tasks. Based on experiments with LLAMA 2, Wendler et al. (2024) hypothesize that the model first solves the task using English as a pivotal language, then generates the answer in the target language. This process can be seen as an implicit translate-test approach. These observations are partially confirmed by our experiments.
Recently, several annotated Kazakh datasets (Yeshpanov et al., 2022, 2024) and multilingual datasets including Kazakh (Bandarkar et al., 2023; Senel et al., 2024) have been published. However, we are not aware of any studies that have systematically evaluated the quality of existing LLMs in Kazakh.

heading: 3 Data
text: The data used in our experiments is summarized in Table 2. Due to limited resources, we could not afford to create large/numerous datasets from scratch or manually translate existing English datasets. In
compiling the set, we were guided by the following criteria: 1) reuse existing datasets whenever possible; 2) avoid the massive use of machine translation; 3) include tasks that are potentially of practical use to the end user (rather than specific NLP tasks like NER or POS tagging); 4) make the set as diverse as possible.
Belebele is a massively multilingual machine reading comprehension dataset that spans 122 languages, including Kazakh (Bandarkar et al., 2023). Belebele contains 900 multiple-choice questions, each associated with one of 488 distinct passages originating from the Flores-200 dataset (Costajussà et al., 2022). First, the English multiple choice questions and answers were manually created using English passages from the Flores dataset. Later, questions and answers were translated in other languages and aligned with corresponding passages from Flores-200. Each question has four answer options, one of which is correct. So, a random guessing would result in accuracy of 0.25. All 900 Belebele questions are intended exclusively for testing, there is no training supplement to the dataset. Authors report performance of GPT-3.5turbo and LLAMA2-CHAT 70B in zero-shot fashion on Kazakh/English Belebele subsets: 35.0/87.7 and 32.4/78.8 accuracy points, respectively.
kkWikiSpell is a manually collected dataset of correct/incorrect sentence pairs designed to test the spelling ability of LLMs in Kazakh. The sentences in the dataset are taken from randomly selected Kazakh Wikipedia pages, with 10 sentences extracted from each page. Note that there is a possibility that the LLMs “saw” these sentences during their pre-training. Each sentence was deliberately altered to include mistakes. According to Dhakal et al. (2018), people tend to make three types of mistakes when typing: substitution (changing letters), omission (missing letters), and insertion (adding extra letters). In kkWikiSpell, we manually injected these three types of mistakes into the sampled sentences, for example:
Original Sentence: Содан берi бұл есiммен Абай тарихқа ендi. Sentence with mistakes: Содан берi бұл есiмнең Абай тарихқа енд.
The distribution of mistakes in the dataset is as follows: 89 sentences contain one mistake, 61 sentences contain two mistakes, and the remaining 10 sentences contain three mistakes. Letter substitutions occur in 93 sentences, missing letters in 73
Dataset Task Size Metric Language C la ss
. Belebele (Bandarkar et al., 2023) Multiple-choice QA 900 Accuracy Human-translated kkCOPA* Causal reasoning 500 Accuracy Machine-translated NIS Math* School Math 100 Accuracy Orig. in Kazakh KazQAD§ (Yeshpanov et al., 2024) Reading comprehension 1,000 Token-level F1 Orig. in Kazakh
G en . kkWikiSpell* Spelling correction 160 Token-level Jaccard Orig. in Kazakh KazQAD§ (Yeshpanov et al., 2024) Generative QA 1,927 Token-level recall Orig. in Kazakh Flores-101 (Goyal et al., 2022) Machine translation 500 BLEU Human-translated
*Datasets prepared within this study. §KazQAD data was used both in open- and closed-book scenarios.
Table 2: Benchmarks in the study. The upper part of the table describes discriminative/classification tasks, whereas the bottom part – generative tasks.
sentences, extra letters in 17 sentences, missing spaces in 4 sentences, extra spaces in 2 sentences, capitalization mistakes and missing characters occur in one sentence each. The total dataset consists of 160 incorrect/correct sentence pairs. The sentences vary in length from 5 to 26 words, with an average sentence length of 11 words.
NIS Math. Math problems are one of the standard tests for large language models. We are not aware of any multilingual benchmarks that include math problems in Kazakh, so we downloaded the entrance tests used for admission to the Nazarbayev Intellectual Schools (NIS). The difficulty level corresponds to the sixth school grade. The tests, in PDF format, were automatically parsed and then manually checked; only textual questions (i.e., without illustrations) were retained. The final set consists of 100 problems, each with four possible answers, one of which is correct. Accuracy is used as a metric to evaluate the task (random guessing results in an accuracy of 0.25). An example from the NIS Math dataset along with an English translation:
Question: Егер шаршының қабырғасын 60%-ға арттырса, ауданы қалай өзгередi. a: 2.56 есе өстi b: 2.56 есе кемiдi c: 0.36 есе өстi d: 0.16 есе өстi correct: a
Question: If the side of a square is increased by 60%, the area of the square changes as follows. a: increased by 2.56 times b: decreased by 2.56 times c: increased by 0.36 times d: increased by 0.16 times correct: a
kkCOPA is a machine translation of the test subset of the English Choice Of Plausible Alternatives (COPA) dataset (Roemmele et al., 2011) us-
ing the Google Translate API.4 COPA is designed to evaluate the ability of models to identify realworld cause-effect relationships. In this respect, it differs from question-answering datasets, which, depending on the scenario, evaluate the model’s language understanding and/or factual knowledge. Each COPA item is a triple containing a premise and two alternatives corresponding to either to effect or cause. Thus, given a premise, a direction (i.e., forward or backward causal reasoning), and two alternatives, the task is to choose the correct option from two. COPA has 500 items in its balanced test set, so random guessing will result in an accuracy of 0.5. An example of a COPA item and its corresponding kkCOPA entry:
Premise: The band played their hit song. Question: What happened as a result? Alt1: The audience clapped along to the music. Alt2: The audience politely listened in silence.
Premise: Топтар хит әндерiн ойнады. Question: әсерi ретiнде не болды? Alt1: Аудитория музыкаға сәйкес келедi. Alt2: Көрермендер үнсiз тыңдады.
Laskar et al. (2023) report that the zero-shot performance of GPT-3.5 on COPA is 94. XCOPA (Ponti et al., 2020) is a multilingual extension of the original dataset. It contains human translations of the COPA test set and 100 items from the development set into 11 languages (doesn’t include Kazakh). GPT-3.5 and GPT-4 achieve an average accuracy across all languages on XCOPA of 79.1 and 89.7, respectively (Ahuja et al., 2023).
KazQAD is an open domain question answering (ODQA) dataset in Kazakh (Yeshpanov et al., 2024). The dataset can be used in various scenarios – for training and evaluation of information retrieval, reading comprehension, and open/generative question answering. The dataset contains questions, annotated passages from
4https://cloud.google.com/translate/
Kazakh Wikipedia and short answers extracted from the relevant passages. The training subset contains questions from the English NaturalQuestions dataset (Kwiatkowski et al., 2019) which have been machine translated into Kazakh. The test set contains 1,927 original questions from the Unified National Test (UNT) – a high school graduation exam in Kazakhstan in six subjects. The KazQAD test set is the largest benchmark in our study. We used the KazQAD data in two scenarios: openbook and closed-book question answering. In the first case, we provided the question and the relevant passage as context, along with the instruction that the LLM should return a span of the passage as the answer. Since the dataset was recently released, we hope that the KazQAD test set wasn’t contaminated.
FLORES-101 is a dataset for machine translation evaluation covering 101 languages, including Kazakh (Goyal et al., 2022). To build the dataset, original English sentences were first extracted from three Mediawiki projects and then manually translated into 101 languages. The dataset contains 3,001 English sentences and their translations, divided into train (997), dev (1,012), and test (992) subsets. FLORES-101 enables the simultaneous evaluation of different translation pairs and directions. In this study, we evaluate LLM’s ability to translate Kazakh sentences into English, Russian and Turkish. Note, however, that in the case of the Kazakh-Russian and Kazakh-Turkish pairs, both parts were created by translators and may contain translationese. The creators of FLORES-101 suspect that the way the data was created may, for example, lead to increased differences between cognate languages (e.g. Kazakh and Turkish, as they belong to the same language family). Zhu et al. (2023) report BLEU scores of zero-shot translation from Kazakh to English on FLORES-101 for LLAMA 2-CHAT, GPT-3.5 and GPT-4: 6.83, 21.74, and 30.65, respectively.

heading: 4 Models
text: In our work, we evaluated seven models. Since five of the seven models are closed, many of their aspects such as the number of parameters or the data on which they were trained are unknown. Table 3 lists the models in our experiment and presents official metrics on two common benchmarks – MMLU and GSM8K for GPTs, Gemini and LLAMA 2. In addition, we present the results of the evaluation
of YandexGPTs and AYA on multilingual MMLU adaptations. The release date of the model may indirectly indicate the up-to-dateness of the information stored in its parameters (it should be noted that the pre-training of mT5, on which AYA is based, was conducted much earlier). We also report the vocabulary sizes of the models and the fertility rates of their tokenizers, i.e. the ratios of tokens and whitespace-tokenized words calculated on the kkCOPA data. Tokenization stronfgly influences the quality of subsequent task solving (Ahuja et al., 2023; Bandarkar et al., 2023) and may also introduce inequity between language communities, since LLM APIs charge on a per-token basis (Petrov et al., 2023).
GPT 3.5 and 4 are two generations of LLMs from OpenAI. Kazakh is included in the official list of languages that GPTs work with.5 We access the models through their official APIs. We use gpt-3.5-turbo-0125 and gpt-4-0125-preview versions in our study.
Gemini 1.5 Pro is the latest publicly available LLM from Google. Kazakh is not on the list of languages officially supported by Gemini.6 This is probably the reason why Gemini returns empty results or error messages for a significant share of requests, see details in Section 5. We accessed gemini-1.5-pro-preview-0409 model through Google Cloud’s Vertex AI Studio.
LLAMA is a collection of open LLMs of different sizes. They have been pre-trained on 2T tokens, of which an estimated ∼90% are English. Due to limited computational resources, we use an 8- bit quantized version of LLAMA 2-CHAT 7B, an aligned model for dialogue use cases. Although the model was mainly trained on English data, it has some multilingual capabilities, as shown by numerous experiments.
YandexGPT 2 and 3. Few technical details about Yandex’ language models are disclosed, but the company’s blog posts provide results of evaluating models on proprietary benchmarks and comparing YandexGPTs side-by-side with ChatGPT and LLAMA 2 on tasks in Russian. We could not find an official list of supported languages, but our
5https://help.openai.com/en/articles/8357869# h_513834920e
6https://support.google.com/gemini/answer/ 13575153
Model xMMLU GSM8K Release date |V| T/W
GPT-3.5-turbo1 C 70.0† 57.1 11.2022 100k4 5.80GPT-4-turbo (Achiam et al., 2023) C 86.4† 92.0 03.2023 LLAMA 2 (Touvron et al., 2023) O 45.3† 56.8 02.2023 32k 4.78 Gemini 1.5 pro (Reid et al., 2024) C 81.9† 91.7 02.2024 256k 3.63 AYA (Üstün et al., 2024) O 37.3§ – 02.2024 250k 2.66 YandexGPT 22 C 55.0* – 09.2023 ? 3.83 YandexGPT 33 C 63.0* – 03.2024
1 https://openai.com/blog/chatgpt 2 https://ya.ru/ai/gpt-2 3 https://ya.ru/ai/gpt-3 (in Russian) 4 https://github.com/openai/tiktoken † original English MMLU (Hendrycks et al., 2021) § multilingual MMLU (Lai et al., 2023b), averaged over 31 languages * proprietary Russian version of MMLU
Table 3: Open (O) and closed (C) LLMs in the study. Note that xMMLU scores correspond to different variants of the dataset and can only be used for comparison within subgroups of the models (e.g., YandexGPT 2 vs. 3). The last two columns report the vocabulary size and the token/word ratio calculated on kkCOPA.
experiments show that the models “understand” English and Kazakh to some extent. In March 2024, there were press reports that Yandex was planning to train YandexGPT in Kazakh language, but it is unclear whether these plans have already been implemented.7
AYA is a massively multilingual model based on the 13B mT5-xxl model (Xue et al., 2021) that supports 101 languages, including Kazakh. The main challenge of the Aya project was to prepare a large instruction dataset to cover all supported languages (Singh et al., 2024). We hosted the AYA model8 on a cloud GPU.

heading: 5 Experimental Results

heading: 5.1 Experimental Design
text: All models and tasks were evaluated in a zero-shot scenario. We used two types of prompts – with English and Kazakh instructions (the main content – question, sentence to correct or translate, etc. – was always in Kazakh).9 Since open-book question answering implies relatively long contexts when accessing the paid APIs, we randomly sampled 1,000 KazQAD test questions to stay within our limited budget.
For classification tasks, we implemented simple processing scripts for extracting actual answers from the LLM responses. For evaluation of openbook QA and machine translation we employed F1 and BLEU scores implemented in the Huggingface’s evaluate library.10 As a quality metric for
7https://tass.ru/ekonomika/20390279 (in Russian) 8https://huggingface.co/CohereForAI/aya-101 9With the exception of the closed-book QA task, which we
evaluated with English instructions only. 10https://huggingface.co/docs/evaluate/
spelling correction, we use the token-level Jaccard coefficient between the “gold standard” and the sentence returned by the model.
Automatic evaluation of closed-book QA is problematic because we need to assess the similarity of “golden” answers to the free-form response returned by the language model (Kamalloo et al., 2023). In particular, LLMs often return sentence-long answers to factoid questions, even though the prompt asks for concise answers. On the other hand, the LLM’s response may be semantically close to the reference, but quite different in wording. We used the recall of lemmatized tokens as a metric to evaluate closed-book QA. For the lemmatization, we used the Stanza library (Qi et al., 2020). This approach makes it possible to ignore the length of the LLM response, as well as to match different morphological variants of a word, which is especially important in the case of the inflectionally rich Kazakh language. This metric does not take into account word order, synonyms and word meaning. However, manual inspection of the results confirms that this is a viable option for comparing different LLMs. In addition to the average recall over all questions, we report the absolute number of responses with a recall greater than 0.5. For similar values of averaged recall, this additional parameter indicates the number of more precise answers in the model’s responses.

heading: 5.2 Results and Discussion
text: Table 4 summarizes results on six tasks, while Table 5 reports translation results.
Our results confirm the findings of previous studies – LLMs perform quite well on classification tasks in non-English languages. On the Belebele dataset, GPT-4 and Gemini show similarly high
results, followed by AYA with English prompts. There are 18 Belebele questions that none of the LLMs answered correctly with either English or Kazakh instructions. We didn’t find any patterns in these “hard” questions. Furthermore, excluding LLAMA 2 with Kazakh instructions, there are 14 questions that all models answered correctly across 13 runs. Again, these questions and their passages show no noticeable similarities. Notably, two kkCOPA questions (#574 and #992) were answered incorrectly in all 14 configurations. In both cases, the Kazakh translations were incorrect. As a results, the models selected answers that, although incorrect in the original context, were logically consistent with the mistranslated versions. An interesting observation is that most models achieved higher accuracy in identifying effects than causes. In particular, AYA with English prompts showed the largest difference, achieving an accuracy of 66.4% for causes and 79.2% for effects. Out of 100 NIS Math questions, there were three where all models failed to provide correct answers. One of these (#44) was flawed because it erroneously showed the wrong answer as correct. On math problems, the results of YandexGPT 2 are approximately at the level of the random baseline (0.25), while GPT-3.5 and LLAMA 2 are below it. On the open-book question answering task with English prompts, AYA is the winner, outperforming both GPT-4 and Gemini. GPT-4 and AYA outperform SOTA on this dataset – fine-tuned XML-V achieves F1 = 0.54 (Yeshpanov et al., 2024) (although we must treat these results with caution, since in our study, due to limited resources, the evaluation was performed on about half of the test set). Tasks involving the generation of responses in Kazakh are more difficult for all models. The spelling correction task proved to be quite hard for all models, although the errors introduced can be considered simple. Again, GPT-4 is the leader in this task. The results of both Yandex models are comparable. YandexGPT 2 occasionally outputs some Kazakh words in Latin script or inserts ** in the output words as they were split into subword tokens. Gemini returned only empty responses. LLAMA 2, when instructed in Kazakh, does not solve the task at all, but sometimes provides a kind of analysis of the input, e.g. The text is a poem and it has a specific structure and rhythm. When instructed in English, LLAMA 2 performs slightly better, but still responded to only 55 out of 160
sentences, none of which were correct. GPT-4’s leadership is particularly evident in the closed-book question answering. The AYA model looks quite competitive compared to the closed models that are reportedly significantly larger. Note that the AYA’s backbone model mT5 does not have the most advanced architecture and the model may be prone to the “curse of multilinguality” (Conneau et al., 2020). Interestingly, LLAMA 2 generates relatively many high-recall answers to KazQAD questions, ranking second in this respect after GPT4. Manual inspection of the KazQAD closed-book answers revealed that GPT-3.5 tends to return incorrect Kazakh names as answers. For example, for the question Who is the scientist who proposed the principle of naming the genus and species in Latin? GPT-3.5 returned Galim-Aibek Bolat, while the correct answer is Carl Linnaeus. The other strange thing about GPT-3.5 is that about a fifth of the answers were just the questions themselves, but with some letters/words removed. The YandexGPT 2 returned most of the answers in Russian.
Machine translation results show that dedicated solutions are still a better alternative for this task and the considered language pairs. At the same time, GPT-4 approaches the quality of Google Translate on the Kazakh-Turkish pair (interestingly, translation between two languages belonging to the same family shows the lowest scores). The translation quality of the LLAMA 2 and AYA models drops significantly when using Kazakh prompts. Gemini appears to be competitive with GPT-4, returning non-empty translations for 64% and 62% of sentences following English and Kazakh prompts, respectively. AYA was even less responsive in the machine translation task with Kazakh prompts. After tweaking the prompt, we were only able to get Turkish translations for about 10% of the Kazakh sentences. GPT-3.5 also showed strange behavior in the Turkish translation task: in many cases, the model simply rephrased the Kazakh input.
It is interesting to note that, based on our results, we cannot draw a clear conclusion that English prompts improve results over Kazakh prompts. In rare cases, Kazakh prompts lead to slightly better scores (GPT-4 on kkCOPA and NIS Math). In other cases, the decrease is insignificant. However, the quality of the extractive question answering drops for all models. LLAMA 2’s results decrease significantly when switching from English to Kazakh prompts on all tasks.
Gemini behaves very differently from, for exam-
ple, GPT-4: in many cases the model returns empty responses or error messages. Gemini refused to return any answers to math problems with Kazakh prompts, as well as any spelling corrections. Gemini answered about half of the math questions with English prompts, i.e. its accuracy on the answered questions is about 80%. Gemini answered only a small fraction (10-13%) of KazQAD questions in all scenarios. LLAMA 2 results are lower than we expected based on previous studies. For example, on Belebele with English prompts, our results differ significantly from those reported by Bandarkar et al. (2023) for LLAMA 2 70B: 12 vs. 34 accuracy points. There may be several reasons for this discrepancy, such as model size (8-bit quantized 7B vs. 70B) and a less optimal prompt. We will address this issue in our future work.

heading: 6 Conclusion
text: Our results provide valuable insights into the applicability of currently available LLMs for Kazakh. GPT-4 shows the best results, followed by Gemini and AYA. Gemini’s results are promising, although the proportion of empty answers is quite high. AYA is very competitive compared to its supposedly larger closed counterparts. As expected, the quality of the LLMs on the Kazakh tasks is lower than on the parallel English tasks. In general, LLMs perform better on classification tasks and struggle with generative tasks. English instructions can improve results on some tasks/models.
Our evaluation showed that there is a steady progress in LLMs for Kazakh (GPT-3.5 vs. GPT-4). We expect the support of Kazakh by Gemini and YandexGPT to be strengthened, as well as the appearance of a Kazakh adaptation of an open LLM. We made the datasets prepared for the study and the collected LLM responses publicly available. These resources can form the basis for an LLM benchmark focused on the Kazakh language. In our future work, we plan to expand the list of LLMs and the set of benchmarks.

heading: Acknowledgments
text: Pavel Braslavski acknowledges funding from the School of Engineering and Digital Sciences, Nazarbayev University. The experiments were partially supported by a Yandex Cloud grant.

heading: Future Work
text: In our future work, we plan to expand the list of LLMs and the set of benchmarks