Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakh
Authors: Nurkhan Laiyk, Daniil Orel, Rituraj Joshi, Maiya Goloburda, Yuxia Wang, Preslav Nakov, Fajri Koto
Accepted at: acl25

Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakh

Nurkhan Laiyk1* Daniil Orel1* Rituraj Joshi3 Maiya Goloburda1 Yuxia Wang1 Preslav Nakov1,2 Fajri Koto1
1Mohamed bin Zayed University of Artificial Intelligence
2Institute of Foundation Models
3Cerebras Systems
{nurkhan.laiyk,daniil.orel}@mbzuai.ac.ae

Abstract

Instruction tuning in low-resource languages remains underexplored due to limited text data, particularly in government and cultural domains. To address this, we introduce and open-source a large-scale (10,600 samples) instruction-following (IFT) dataset,1 covering key institutional and cultural knowledge relevant to Kazakhstan. Our dataset enhances LLMs’ understanding of procedural, legal, and structural governance topics. We employ LLM-assisted data generation, comparing open-weight and closed-weight models for dataset construction, and select GPT-4o as the backbone. Each entity of our dataset undergoes full manual verification to ensure high quality. We also show that fine-tuning Qwen, Falcon, and Gemma on our dataset leads to consistent performance improvements in both multiple-choice and generative tasks, demonstrating the potential of LLM-assisted instruction tuning for low-resource languages.

1 Introduction

Instruction tuning enhances large language models (LLMs) by fine-tuning them on structured prompts, improving their ability to follow human instructions across various tasks such as question answering and summarization (Ouyang et al., 2022). While extensive instruction-tuning datasets exist for English, such as FLAN (Longpre et al., 2023), P3 (Sanh et al., 2021), and Dolly (Conover et al., 2023), efforts in low-resource languages remain limited. This gap is particularly evident in domain-specific applications where multilingual LLMs often provide generic or inaccurate responses due to a lack of localized training data (Li et al., 2023).

A key challenge in adapting LLMs to underrepresented languages is the scarcity of high-quality instruction data (Li et al., 2023). Multilingual models may process low-resource languages at a technical level (OpenAI, 2024), but their practical effectiveness is often constrained by an incomplete understanding of region-specific socio-political structures and cultural contexts. For example, when asked about administrative procedures like obtaining a passport in a particular country, models tend to default to well-documented cases rather than providing precise, localized information. Similarly, cultural narratives—such as folklore, literature, and traditions—are often missing from instruction datasets (Conover et al., 2023), limiting the models’ ability to generate contextually appropriate responses. While prior work relied on translation (Sengupta et al., 2023) or template-based techniques (Cahyawijaya et al., 2024) to build instruction-tuning datasets, it does not fully reflect the actual local context, as direct translations often fail to capture the nuances of regional governance, customs, and linguistic variations.

Building instruction datasets from scratch is costly, making large-scale manual data collection impractical for many low-resource languages. To address this, we adopt an LLM-assisted dataset generation approach (Liu et al., 2022; Cahyawijaya et al., 2023; Zhang et al., 2024), followed by full human validation. Specifically, we use a single-prompt method where LLMs process high-quality unlabeled text from public government and cultural sources to extract both factual information and corresponding instructions. These domains are highly relevant for real-world applications, but remain underexplored for instruction-tuning, particularly in the context of government data.

To demonstrate the effectiveness of this approach, we introduce an instruction-tuning dataset for Kazakh that integrates both institutional2 (GovSet) and cultural (CultSet) domains. We choose Kazakh as our case study because it re-

1https://huggingface.co/datasets/nurkhan51/kazakh-ift
2Our study incorporates administrative, procedural, legal, structural, and other government-related types of information.
Figure 1: Overview of the end-to-end process for constructing GovSet and CultSet datasets. English translations are for illustration purposes.

mains underrepresented in NLP (Joshi et al., 2020), despite having approximately 20 million speakers. Prior research on Kazakh NLP has primarily focused on classic tasks such as named entity recognition (Yeshpanov et al., 2022) and sentiment analysis (Yeshpanov and Varol, 2024), leaving more advanced applications like instruction tuning, largely unexplored.

Our contributions are as follows:

• We create an open-source, high-quality, manually verified large scale (10K samples) IFT dataset, which covers both cultural, and institutional knowledge, relevant to Kazakhstan.
• We contribute new domain knowledge on essential institutional topics, including procedural, legal, structural, and other key aspects of public governance, enhancing LLMs’ understanding of these critical areas.
• We compare the efficacy of open-weight and closed-weight LLMs in LLM-assisted dataset construction for low-resource languages and underrepresented cultures.
• We demonstrate that fine-tuning on our dataset results in consistent improvements in both multiple-choice and generative tasks. These results highlight the impact of incorporating localized knowledge into instruction tuning and demonstrate the potential of LLM-assisted approaches for expanding instruction datasets in other low-resource languages.

2 Related Work

2.1 Instruction-Tuning Datasets in English

There are three main strategies for creating English instruction-tuning datasets: human-curated datasets, templatized NLP tasks, and synthetic data generation using LLMs.

Human-curated datasets, such as Open Assistant (Köpf et al., 2023) and Dolly (Conover et al., 2023), rely heavily on human annotation. While this approach ensures high-quality data, it is expensive and difficult to scale across multiple languages. To reduce costs, datasets like the Public Pool of Prompts (P3) (Sanh et al., 2021), SuperNatural Instructions (Wang et al., 2022), and FLAN (Longpre et al., 2023) reformat existing NLP tasks into instruction-based formats. However, these datasets primarily focus on specific NLU tasks rather than general-purpose instruction following, limiting their applicability.

Prior work on instruction dataset creation has largely relied on generating data from existing language models without incorporating external real-world knowledge. Self-Instruct (Wang et al., 2023) expands an initial set of human-written instructions by iteratively generating new tasks using the model’s own outputs, while Honovich et al. (2023) create instruction-tuning datasets by conditioning on a few example instructions. These methods rely solely on sampling data from language models or predefined topics, rather than grounding them in external knowledge, making them less suitable for
capturing domain-specific or culturally relevant understanding. Unlike these approaches, our work focuses on Kazakh, a low-resource language, and constructs an instruction dataset by leveraging external, factual sources such as governmental and cultural texts, ensuring alignment with real-world contexts.

2.2 Instruction Tuning Datasets for Medium-to Low-Resource Languages

While human-curated datasets are often expensive and require native speakers, prior work has explored automatic dataset generation using machine translation for low- to medium-resource languages. Sengupta et al. (2023) applied this approach to develop JAIS, an Arabic-centric language model, by translating various English instruction datasets into Arabic. Similarly, Li et al. (2023) translated Alpaca (Taori et al., 2023) into 52 languages. More recently, Alyafeai et al. (2024) introduced an Arabic instruction dataset by translating Alpaca into Arabic and then performing manual edits and localization to ensure relevance to the Arabic context. Although this method is scalable, the quality of translation remains inconsistent for low-resource languages, as noted by Li et al. (2023). Additionally, machine-translated datasets often introduce Anglocentric biases, limiting their ability to capture culturally diverse perspectives.

Several frameworks have been proposed to improve instruction tuning for low-resource languages. MURI (Köksal et al., 2024) generates multilingual instruction datasets using reverse instruction generation and translation, but none of its data have been validated by native speakers. Li et al. (2024) improve upon translation-based methods by prompting LLMs in English while requiring responses in low-resource languages (e.g., Urdu), allowing models to leverage their internal knowledge of the target language’s local context. However, our approach differs by grounding responses in factual information from reliable sources, such as government data, rather than relying solely on the model’s internal knowledge. Additionally, our dataset undergoes full human validation to ensure accuracy and relevance. Meanwhile, Cahyawijaya et al. (2023) focused on the linguistic aspects of instruction tuning by denoising low-resource language text and prompting models to reconstruct complete sentences. While this enhances fluency, it differs from our approach, which prioritizes grounding instructions in externally verified, domain-specific knowledge rather than refining linguistic quality alone.

2.3 Existing datasets in Kazakh

While there has been significant progress in developing Kazakh datasets, the majority of high-quality Kazakh datasets are related to speech (Mussakhojayeva et al., 2024, 2022, 2021). In terms of textual data, existing resources primarily focus on question answering and reading comprehension rather than instruction tuning. For instance, KazQAD (Yeshpanov et al., 2024) is a Kazakh open-domain question answering (ODQA) dataset that can be used in both reading comprehension and full ODQA settings, as well as for information retrieval experiments. Similarly, Belebele (Bandarkar et al., 2024) is another dataset that, while useful for multilingual machine reading comprehension, is not explicitly designed for instruction tuning. Belebele covers 122 languages, including Kazakh, and comprises 900 multiple-choice questions associated with 488 distinct passages from the Flores-200 dataset.

Despite progress in Kazakh NLP resources, no existing instruction-tuning dataset incorporates cultural or domain-specific knowledge, focuses on real-world applications, and undergoes full human validation. This limits the ability of LLMs to process Kazakh-language instructions effectively in practical and locally relevant contexts, which we aim to address in this work.

3 Background

Kazakh Cultural Heritage. Kazakhstan has a rich cultural heritage that reflects a blend of nomadic traditions, Soviet influences, and modern developments. The country’s nomadic heritage is evident in many aspects of daily life, from its architecture, with yurts (traditional felt tents) still used in rural areas, to its customs of communal gatherings and feasts, such as the celebration of the Turkic New Year, Nauryz.

Kazakhstan’s Soviet past has also left a lasting imprint on its culture. Many cities still bear the architectural marks of Soviet planning, while the era also shaped the country’s education and scientific institutions, fostering a strong tradition in mathematics and engineering—most notably reflected in the Baikonur Cosmodrome, the world’s first and largest space launch facility.

Alongside all of this, modern developments have
transformed Kazakhstan. The capital, Astana, is a prime example of this shift, with its futuristic skyline and ambitious urban projects. Investments in technology, renewable energy, and digital infrastructure have propelled Kazakhstan onto the global stage, while cultural revitalization efforts have fostered a renewed interest in the Kazakh language, music, and art.

At its core, Kazakhstan’s culture is shaped by beliefs and values, social practices, language, artistic expression, and material culture. Understanding these components is crucial for ensuring accurate and meaningful representation.

Kazakhstan’s Institutional Structure and Public Governance. Kazakhstan is a presidential republic that has prioritized modernization since its independence in 1991, particularly in governance and legal systems. The 1995 Constitution established the legal foundation, defining citizens’ rights and the structure of government. A major step in this modernization has been the digitalization of public services. Kazakhstan ranks among the top 25 countries in the UN E-Government Development Index (EGDI) (Nations, n.d.), with the eGov platform serving as a centralized portal for services like business registration, tax payments, and social benefits.

These efforts reflect a broader national context shaped by Kazakhstan’s cultural heritage, nomadic traditions, and growing digital infrastructure. Platforms like eGov highlight the integration of technology into daily governance. As the country continues to modernize, it is essential that language models accurately represent these unique characteristics to support cultural understanding and global relevance.

4 Data

4.1 Document Source

GovSet We manually collected 1,376 texts from the official Kazakhstan e-Government portal (gov.kz3), the primary and most comprehensive platform for all public services, governmental processes, and administrative resources in the country. As the central hub for Kazakhstan’s digital governance, gov.kz consolidates a wide range of essential information into a single system, covering diverse aspects of public administration, legal frameworks, citizen services, and governmental initiatives. By incorporating these texts, we ensure that the dataset captures essential institutional aspects of life in Kazakhstan, including its governmental structure and public services. This enrichment enhances instruction-tuning applications, making them more linguistically appropriate and contextually informed.

CultSet We automatically collected 4,400 texts from Kazakh Wikipedia,4 specifically focusing on pages related to Kazakh culture. These pages were identified based on metadata that explicitly indicated their relevance to Kazakh cultural topics. The parsed texts include various aspects of Kazakh traditions, heritage, arts, and historical practices, providing a rich source of culturally relevant content. This ensures that the dataset reflects the depth and diversity of Kazakh culture, making it suitable for instruction-tuning tasks that require a culturally grounded perspective.

4.2 LLM-assisted Data Generation

We benchmark one open-weight LLM: LLaMA 3.1-70B (Touvron et al., 2023), and three closed-weight LLMs: GPT-4o (OpenAI, 2024), Gemini-1.5 (DeepMind, 2024), and Claude-3.5-Sonnet (Anthropic, 2024), to assess their effectiveness in assisting dataset creation. These models were selected based on their strong performance in multilingual benchmarks. However, their capability in generating instruction datasets specific to Kazakh government and cultural data remains uncertain.

We design a prompt (see Appendix A.3) that instructs LLMs to first extract factual information from a given Kazakh document and then generate an instruction dataset based on the extracted content. Table 2 provides detailed statistics on the source documents and the resulting instruction fine-tuning (IFT) dataset using GPT-4o. Specifically, we use 4,400 Kazakh cultural Wikipedia documents and 1,376 Kazakh government data sources, generating a total of 10,600 IFT instances. Of these, 58% belong to the government public data category (GovSet), while the remaining samples are derived from Wikipedia (CultSet). Examples of generated IFT data can be found in Table 15 and Table 17.

Human Evaluation Across LLMs For each LLM, we sampled 100 generated IFT instances, drawn from 25 randomly selected GovSet and 25 CultSet documents. Additionally, we randomly

3https://www.gov.kz
4kk.wikipedia.org
sampled 100 instances from MURI (Köksal et al., 2024), which also includes Kazakh IFT data, to provide a comparative quality assessment. Two native Kazakh speakers were recruited to manually evaluate the generated data based on the following criteria:

• Correctness: The factual accuracy and alignment with the original text. A high score indicates that the generated pair adheres closely to the source material without introducing errors or inaccuracies.

• Fluency: The grammatical and stylistic quality of the generated text. A higher score reflects well-structured, natural, and polished language.

• Completeness: The degree to which the instruction-response pair is clear, contextually grounded, and free from ambiguity. High scores indicate that the pair is fully self-contained, with enough context to make it understandable.

All criteria were rated on a Likert scale from 1 to 5, with 5 representing the highest quality. A detailed evaluation rubric is provided in Table 8.

Table 1 presents the quality assessment of various LLMs in generating IFT data for Kazakh. The inter-annotator agreement, measured using Pearson correlation, is high (ranging from 0.68 to 0.70) across correctness, completeness, and fluency, indicating strong reliability in the evaluation process (see Appendix K.1 for further details).

Among the evaluated models, GPT-4o achieved the highest performance across all three criteria. In contrast, LLaMA-3.1 (70B) lagged significantly, scoring nearly 0.8–1 point lower in all aspects. Notably, MURI’s quality was lower than GPT-4o despite both relying on OpenAI models. This discrepancy is likely due to MURI’s reliance on machine translation, where Kazakh text is first translated into English before generating instructions, followed by a final back-translation into Kazakh. This multi-step translation process can introduce errors due to cumulative translation inaccuracies. Additionally, MURI is entirely LLM-generated without human validation, further affecting its quality.

<table>
  <tr>
    <th>Model</th>
    <th>Correctness</th>
    <th>Completeness</th>
    <th>Fluency</th>
  </tr>
  <tr>
    <td>Llama 3.1 (70B)</td>
    <td>3.54</td>
    <td>3.45</td>
    <td>3.07</td>
  </tr>
  <tr>
    <td>Claude</td>
    <td>3.74</td>
    <td>3.48</td>
    <td>3.09</td>
  </tr>
  <tr>
    <td>Gemini 1.5</td>
    <td>3.85</td>
    <td>3.64</td>
    <td>3.32</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td><b>4.38</b></td>
    <td><b>4.29</b></td>
    <td><b>4.04</b></td>
  </tr>
  <tr>
    <td>MURI</td>
    <td>3.87</td>
    <td>3.52</td>
    <td>3.41</td>
  </tr>
</table>

Table 1: Human evaluation on LLM-generated instruction datasets.

<table>
  <tr>
    <th></th>
    <th>CultSet</th>
    <th>GovSet</th>
  </tr>
  <tr>
    <td>Collected text</td>
    <td>4,400</td>
    <td>1,376</td>
  </tr>
  <tr>
    <td>Avg. lengths (#char) of collected text</td>
    <td>245</td>
    <td>179</td>
  </tr>
  <tr>
    <td>Generated IFT pairs</td>
    <td>4,400</td>
    <td>6,200</td>
  </tr>
  <tr>
    <td>Avg. lengths (#char) of instruction</td>
    <td>85</td>
    <td>76</td>
  </tr>
  <tr>
    <td>Avg. length (#) of output</td>
    <td>453</td>
    <td>215</td>
  </tr>
  <tr>
    <td># of unique tokens</td>
    <td>62,449</td>
    <td>24,304</td>
  </tr>
</table>

Table 2: Statistics of GPT-4o generated IFT dataset.

advanced degrees in World Languages, Literature, or Political Science from top Kazakhstani universities. Their extensive experience—having lived in Kazakhstan for over 25 years—equips them with the necessary linguistic and cultural expertise.

To maintain consistency, annotators received detailed guidelines outlining task objectives, evaluation criteria, and examples of high-quality IFT pairs (see Appendix F). They were responsible for manually reviewing and correcting errors in the generated data. Before starting the main annotation process, all candidates completed a pilot task to assess their understanding of project requirements and their ability to refine IFT pairs accurately. Only those who met the evaluation criteria were selected. Each annotator’s workload was equivalent to five full working days, and they were compensated fairly based on Kazakhstan’s monthly minimum wage. To accommodate flexibility, annotators were given up to one month to complete the task while working part-time.

Table 3 summarizes the error types identified during manual post-editing of GPT-4o-generated data across the two document sources. Annotators found that CultSet had a higher proportion of "No error" cases (28.32%) compared to GovSet (19.47%), suggesting variations in data quality.

Structural errors were the most common in both datasets, accounting for over 28% in CultSet and 33% in GovSet. These errors involve grammatically correct but poorly structured responses, including issues with logical flow, organization, and unnatural phrasing for a Kazakh speaker. Addition-
<table>
  <tr>
    <th>Error Type</th>
    <th colspan="2">% of Questions</th>
  </tr>
  <tr>
    <th></th>
    <th>CultSet</th>
    <th>GovSet</th>
  </tr>
  <tr>
    <td>No error</td>
    <td>28.32%</td>
    <td>19.47%</td>
  </tr>
  <tr>
    <td>Wrong language</td>
    <td>0.07%</td>
    <td>0.14%</td>
  </tr>
  <tr>
    <td><b>Structural</b></td>
    <td><b>28.45%</b></td>
    <td><b>33.58%</b></td>
  </tr>
  <tr>
    <td>Grammatical</td>
    <td>25.24%</td>
    <td>28.73%</td>
  </tr>
  <tr>
    <td>Lexical</td>
    <td>17.92%</td>
    <td>18.08%</td>
  </tr>
</table>

Table 3: Distribution of error types in GPT-4o-generated IFT data from CultSet and GovSet, identified during manual post-editing.

ally, grammatical and lexical errors were frequently observed, with annotators noting that GPT-4o occasionally replaces Kazakh words with Russian equivalents, even when the correct Kazakh term is explicitly provided in the original text. For a detailed breakdown of annotator observations, see Appendix H.

4.4 Final Data Overview

As shown in Table 2, the final dataset consists of 4,400 CultSet and 6,200 GovSet IFT instances, totaling 10,600 high-quality samples. We split the dataset into 90% training and 10% test, where the training data is used for full fine-tuning of LLMs, and the test set is used for generation evaluation in our experiments.

Since both CultSet and GovSet are topic-based, we include their respective topics as metadata in the final IFT dataset (see Table 13 and Table 14 for topic definitions). Figure 2 illustrates the topic distribution of the dataset. The most common topics in CultSet include Kazakh literature, traditions, and media, while GovSet primarily covers legal assistance, the healthcare system, real estate laws, and education in Kazakhstan. Examples of GPT-4o-generated IFT data can be found in Table 15 and Table 17.

Table 2 further highlights a notable difference between the two subsets: the average output length in CultSet is significantly longer and includes more unique tokens than GovSet. This difference stems from the nature of GovSet responses, which are strictly factual and concise, whereas CultSet responses tend to be more diverse and expressive.

5 Experiments

We conducted two experiments: multiple-choice questions (MCQ) and text generation evaluation. We will detail each evaluation in the following sections.

Model Selection  For both MCQ and generation evaluations, we use three instruction-tuned models: Gemma-2-9b-instruct (Gemma) (Team et al., 2024), Qwen-2.5-7b-instruct (Qwen) (Qwen et al., 2025), and Falcon-3-10b-instruct (Falcon) (Team, 2024). While these LLMs offer multilingual capabilities, none were specifically trained for Kazakh, allowing us to assume that our IFT data is novel to them.

Fine-tuning  We performed full fine-tuning on Gemma-2-9b-instruct (Gemma), Qwen-2.5-7b-instruct (Qwen), and Falcon-3-10b-instruct (Falcon) using the AdamW optimizer with hyperparameters \( \beta_1 = 0.9, \beta_2 = 0.95, \epsilon = 1e-5 \), and a weight decay of 0.1. We scaled the gradient norms using a maximum norm clipping value of 1.0. The learning rate was kept constant throughout the fine-tuning without any warm-up or decay with a value of \( 1e-6 \) for Gemma and Falcon, and \( 1e-5 \) for Qwen. The batch size used was 16, and we packed multiple documents until the maximum sequence length was 8,192 tokens. Cross-document attention is disabled by modifying attention masks so the tokens of a document only attend to the tokens from the same document in a causal way. No adjustments were made to the original tokenizer for each model.

Baseline  As a baseline, we include the Kazakh Alpaca dataset,\footnote{https://huggingface.co/datasets/AmanMussa/kazakh-instruction-v2} which has been translated and localized into Kazakh. For each model, we conduct full fine-tuning with (1) our training dataset, (2) Alpaca, and (3) a combination of Alpaca and our training dataset.

5.1 Multiple-choice Question Evaluation

Dataset  A dedicated open-source Kazakh NLP community\footnote{https://huggingface.co/kz-transformers} has collaboratively developed and crowd-sourced multiple hand-crafted benchmarks to assess the factual knowledge of LLMs in Kazakh. We use three multiple-choice question (MCQ) datasets: (1) Dastur-MC (Sagyndyk et al., 2024b), which evaluates knowledge of Kazakh traditions, (2) Kazakh Constitution-MC (Sagyndyk et al., 2024a), which focuses on Kazakhstan’s legal system, and (3) Kazakh Unified National (Sagyndyk et al., 2024c), which assesses citizen’s rights, legal protections, and societal knowledge (referred to as the “Human Rights and Society” dataset).\footnote{Examples of test questions are provided in Appendix L.}
(a) CultSet

(b) GovSet

Figure 2: Topic distribution of GPT-4 generated IFT dataset in CultSet and GovSet.

(a) CultSet

(b) GovSet

Figure 3: Distribution of preferences for (a) CultSet and (b) GovSet datasets across models. The charts illustrate the percentage of 'Tie', 'Vanilla', and 'After Fine-Tuning' preferences in each dataset.

<table>
  <tr>
    <th>Model</th>
    <th>Vanilla</th>
    <th>RAG</th>
    <th>Alpaca</th>
    <th>Ours</th>
    <th>Alpaca + Ours</th>
  </tr>
  <tr>
    <th colspan="6">Dastur</th>
  </tr>
  <tr>
    <td>Gemma</td>
    <td>0.498</td>
    <td>0.533</td>
    <td>0.513</td>
    <td>0.543</td>
    <td><b>0.566</b></td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td>0.403</td>
    <td>0.410</td>
    <td>0.421</td>
    <td>0.443</td>
    <td><b>0.465</b></td>
  </tr>
  <tr>
    <td>Falcon</td>
    <td>0.450</td>
    <td>0.460</td>
    <td>0.458</td>
    <td>0.464</td>
    <td><b>0.471</b></td>
  </tr>
  <tr>
    <th colspan="6">Constitution</th>
  </tr>
  <tr>
    <td>Gemma</td>
    <td>0.600</td>
    <td><b>0.655</b></td>
    <td>0.627</td>
    <td>0.640</td>
    <td>0.650</td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td>0.520</td>
    <td>0.523</td>
    <td>0.609</td>
    <td>0.670</td>
    <td><b>0.680</b></td>
  </tr>
  <tr>
    <td>Falcon</td>
    <td>0.430</td>
    <td>0.386</td>
    <td>0.450</td>
    <td>0.490</td>
    <td><b>0.520</b></td>
  </tr>
  <tr>
    <th colspan="6">Human Rights and Society</th>
  </tr>
  <tr>
    <td>Gemma</td>
    <td>0.405</td>
    <td>0.450</td>
    <td>0.430</td>
    <td>0.465</td>
    <td><b>0.480</b></td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td>0.300</td>
    <td>0.325</td>
    <td>0.330</td>
    <td>0.365</td>
    <td><b>0.375</b></td>
  </tr>
  <tr>
    <td>Falcon</td>
    <td>0.215</td>
    <td>0.220</td>
    <td>0.234</td>
    <td>0.250</td>
    <td><b>0.275</b></td>
  </tr>
</table>

Table 4: Zero-shot accuracies of language models in different datasets: (1) Dastur, (2) Constitution, and (3) Human Rights and Society

Each dataset consists of multiple-choice questions with four answer options, only one of which is correct. We selected these evaluation benchmarks because they align with the focus of our instruction fine-tuning dataset and are not derived from our document sources (CultSet and GovSet). These datasets cover culturally significant topics, legal frameworks, and citizen-government interactions, reflecting real-world applications that our fine-tuned models aim to support.

Since no documented quality assurance process was available for the three datasets, we conducted a manual verification to ensure the accuracy of the questions. To maintain a fair and valid comparison, only the manually verified samples were used in our evaluation. For the Dastur-MC dataset, we randomly sampled 300 questions and manually verified their correctness. The same process was applied to the Kazakh Constitution-MC and Human Rights and Society datasets, with 200 randomly selected questions from each.

Setup In addition to the fine-tuned models, we include retrieval-augmented generation (RAG) without fine-tuning to estimate the upper bound of the
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="2">CultSet</th>
    <th colspan="2">GovSet</th>
  </tr>
  <tr>
    <th>Vanilla</th>
    <th>After FT</th>
    <th>Vanilla</th>
    <th>After FT</th>
  </tr>
  <tr>
    <td>Gemma</td>
    <td>15.76</td>
    <td><b>24.87</b></td>
    <td>16.12</td>
    <td><b>25.10</b></td>
  </tr>
  <tr>
    <td>Falcon</td>
    <td>25.96</td>
    <td><b>27.98</b></td>
    <td>26.17</td>
    <td><b>28.70</b></td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td><b>27.64</b></td>
    <td>26.63</td>
    <td><b>30.27</b></td>
    <td>28.42</td>
  </tr>
</table>

Table 5: ROUGE-L comparison on CultSet and GovSet before and after fine-tuning.

original models’ performance. For RAG, we use BM25 encoding, as no specialized Kazakh retrieval encoder is available. For each question, we retrieve the top two matching text chunks (each 256 symbols long) from the training texts of our IFT corpus and provide them as additional context.

To assess the capabilities of the model, we use the LM Eval Harness (Gao et al., 2024) framework in a zero-shot setting. During evaluation, the answer is selected based on the alphabetical option with the highest likelihood.

Result Table 4 presents the zero-shot evaluation results across different models and techniques. Overall, our fine-tuned dataset consistently outperforms other approaches across datasets and models. The only exception is the Constitution dataset, where RAG performs better with Gemma. Models fine-tuned on Kazakh Alpaca show some improvement, though it remains lower than that achieved with our instruction fine-tuning (IFT) dataset.

Combining parts of our IFT dataset with the translated Alpaca dataset yields the highest performance gains. This aligns with prior studies (Brief et al., 2024; Wang et al., 2024), which suggest that incorporating general chat instructions alongside domain-specific ones enhances model performance.

For RAG-enhanced models, performance generally exceeds that of the vanilla models, except for Falcon on the Constitution dataset. However, fine-tuned models consistently achieve higher scores than their RAG-enhanced counterparts. We hypothesize that this is due to the models’ limited proficiency in Kazakh, which may hinder their ability to fully understand the retrieved context. As a result, despite the additional information provided by RAG, the models may struggle to extract the necessary details to select the correct answer in MCQs.

5.2 Generation Evaluation

We evaluate generation performance using our test set, which consists of 500 questions from both CultSet and GovSet (excluded from fine-tuning). We compare the best-performing models from Section 5.1 against their vanilla counterparts. In this section, "After Fine-Tuning" refers to models fine-tuned on Alpaca + Our Data, while "Vanilla" refers to the original instruct models.

Automatic Evaluation with ROUGE and BERTScore As shown in Table 5, fine-tuned models generally outperform their vanilla counterparts, except for Qwen, where fine-tuning results in a lower ROUGE-L score (Lin, 2004). However, a lower ROUGE-L does not necessarily indicate worse performance—it may be due to Qwen generating different phrasings compared to the gold answers.

To further validate the quality of generated responses, we also evaluate BERTScore (Zhang et al.). We use Kaz-RoBERTa8 as the encoder model, as it is one of the few open-source Kazakh-language transformers. The BERTScore results in Table 6 align well with the ROUGE-L scores. However, since Kazakh is a low-resource language, BERTScore should be considered a reference point rather than a definitive metric, as Kaz-RoBERTa embeddings may not perfectly capture synonym relationships.

Preference Evaluation with GPT-4o We conducted a 1-to-1 preference evaluation using the LLM-as-a-judge approach. Specifically, we prompted GPT-4o to compare responses from different models and determine whether each response wins, loses, or ties. The prompt includes the instruction and the gold response as context for GPT-4o.9 As shown in Figure 3, the results align with ROUGE-L and BERTScore, confirming that fine-tuned models generally produce improved outputs. Compared to Falcon, Qwen and Gemma exhibit more significant improvements (63%–80% winning rate), likely because their pre-trained versions were less optimized for the task, making fine-tuning more impactful.

Additionally, we analyze the win rate across topics in CultSet and GovSet, as shown in Appendix B. The results indicate that the impact of fine-tuning varies by topic and is not always consistent. In CultSet, fine-tuning Qwen with our IFT data yields the most improvement in Cultural Institutions and Culture & Traditions, while the

8Huggingface model: kaz-roberta-conversational
9The prompt used for comparison is provided in Appendix A.2.
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="3">CultSet</th>
    <th colspan="3">GovSet</th>
  </tr>
  <tr>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
  </tr>
  <tr>
    <td rowspan="3">Vanilla</td>
    <td>Gemma</td>
    <td>29.26</td>
    <td>33.47</td>
    <td>30.92</td>
    <td>27.36</td>
    <td>34.81</td>
    <td>30.39</td>
  </tr>
  <tr>
    <td>Falcon</td>
    <td>23.29</td>
    <td>28.17</td>
    <td>25.20</td>
    <td>20.38</td>
    <td>24.68</td>
    <td>22.11</td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td>40.58</td>
    <td>47.46</td>
    <td>43.40</td>
    <td>36.57</td>
    <td>44.14</td>
    <td>39.50</td>
  </tr>
  <tr>
    <td rowspan="3">After Fine-Tuning</td>
    <td>Gemma</td>
    <td>41.94</td>
    <td>46.36</td>
    <td>43.62</td>
    <td>40.27</td>
    <td>44.90</td>
    <td>42.00</td>
  </tr>
  <tr>
    <td>Falcon</td>
    <td>24.59</td>
    <td>29.68</td>
    <td>26.64</td>
    <td>23.78</td>
    <td>27.73</td>
    <td>25.36</td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td>39.64</td>
    <td>45.40</td>
    <td>41.82</td>
    <td>36.28</td>
    <td>40.20</td>
    <td>37.59</td>
  </tr>
</table>

Table 6: BERTScore Precision, Recall, and F1 for CultSet and GovSet.

![Bar chart showing preference percentages for Falcon, Qwen, and Gemma across three categories: After Fine-Tuning, Reference, and Vanilla](page_324_624_484_246.png)

Figure 4: Conversational data preference evaluation.

gains are smaller in Science & Humanities and even lead to a decline in performance for Education & Academia. In GovSet, fine-tuning Qwen with our dataset significantly enhances performance in Legal Assistance, though the improvement is less noticeable in Employment-related topics.

While LLM-based evaluations provide scalable comparisons, they may not fully capture human judgment nuances, making human evaluation essential for validating model preferences. Therefore, three human annotators conducted a preference evaluation on a randomly sampled 100 examples for each model (Gemma, Qwen, and Falcon) across both CultSet and GovSet. Their judgments were compared against the GPT-based preference evaluation to assess alignment. We computed Cohen’s Kappa between GPT-4o and the annotators, obtaining 0.63 for CultSet and 0.68 for GovSet, indicating substantial agreement. We have also calculated the agreement rate between annotators (detailed in Appendix K.2). The results show that GPT’s alignment with human preferences is moderate, with better agreement on GovSet than CultSet.

Conversational Evaluation. As an extension of these experiments, we generated a set of 100 conversations for both CultSet and GovSet combined, covering topics presented in Figure 2. These conversations were intentionally left unfinished using a special prompt, as detailed in Appendix A.2. Both the original and fine-tuned models were tasked with generating the most appropriate continuation for each conversation. Examples of the resulting texts are shown in Appendix M. To evaluate the quality of the responses, we employed an LLM-as-a-judge framework. The results, presented in Figure 4, indicate that models fine-tuned on domain-specific data produced significantly more coherent and contextually appropriate responses compared to their pre-fine-tuning counterparts. We also see that in the conversational settings there are less ties, compared to simple question answering.

6 Conclusion and Future Work

We introduced a culturally and institutionally aligned instruction-tuning dataset for Kazakh, aiming to enhance practical knowledge representation and address the specific needs of public governmental data processing in Kazakh. Through a carefully designed data collection pipeline, we generated instruction-tuning examples using GPT-4o and ensured their quality via a manual correction and localization to capture Kazakh linguistic and cultural nuances accurately.

The evaluation results show that this approach substantially improved the model’s factual knowledge and the understanding of low-resource languages. It also shows that after such fine-tuning, the model’s responses are much better in terms of correctness and soundness, as assessed by native speakers and LLM as a judge.

In future work, we plan to apply this methodology to other languages and dialects. We further aim to work towards streamlining and automating the process as much as possible. We will also focus more on the modeling part of the experiments, and open-source culturally and institutionally relevant models for low-resource languages, including Kazakh.
7 Limitations

We aim to establish a robust instruction-tuning dataset for Kazakh, authentically reflecting the cultural and linguistic richness of the language. Unlike many existing datasets, which rely on translated resources or machine-generated responses, our dataset is entirely crafted from Kazakh-specific content, ensuring greater alignment with the cultural values and linguistic nuances of the region. However, we recognize several limitations in our work:

• Cultural Representation: The dataset emphasizes topics deeply rooted in Kazakh culture, traditions, and societal norms, ensuring relevance and cultural authenticity. However, certain culturally sensitive topics, such as those involving religious matters, were intentionally omitted to avoid controversy and maintain neutrality.

• Language Variations: Kazakh is a rich language with significant regional variations in vocabulary and usage. While our dataset primarily focuses on standard Kazakh, it does not explicitly account for regional dialects or variations, potentially limiting its applicability to speakers outside the standard dialect’s scope.

• Modeling Limitations: Our works is a proof of concept, and it was not aimed at creation of SOTA models for Kazakh. That is why we experiment with smaller models and do not apply any training tricks such as tokenizer adaptation for Kazakh.

• Possible Data Drift: We also acknowledge that despite of being very conservative by nature, some institutional procedures can change over time, that is why it is possible that the data provided in our IFT dataset will get less actual. To handle this issue we are planning updating the datasets annually.

8 Ethics

We adhered to the internal policies of web resources while scraping data and included only publicly available information verified by authorities.

While our method enhances LLMs’ understanding of Kazakhstan’s institutional nuances, users should not blindly trust generated responses. LLM outputs serve as a starting point, and users remain responsible for fact-checking due to potential hallucinations.

All human subjects in our study provided informed consent, were fully aware of the study’s objectives, and had the right to withdraw at any time. They were also appropriately compensated as part of their job.

References

Zaid Alyafeai, Khalid Almubarak, Ahmed Ashraf, Deema Alnouhait, Saied Alshahrani, Gubran Abdulrahman, Gamil Ahmed, Qais Gawah, Zead Saleh, Mustafa Ghaleb, Yousef Ali, and Maged Al-shaibani. 2024. CIDAR: Culturally relevant instruction dataset for arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12878–12901, Bangkok, Thailand. Association for Computational Linguistics.

Anthropic. 2024. Claude ai: An overview. https://www.anthropic.com.

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand. Association for Computational Linguistics.

Meni Brief, Oded Ovadia, Gil Shenderovitz, Noga Ben Yoash, Rachel Lemberg, and Eitam Sheerit. 2024. Mixing it up: The cocktail effect of multi-task fine-tuning on Ilm performance – a case study in finance.

Samuel Cahyawijaya, Holy Lovenia, Fajri Koto, Rifki Putri, Wawan Cenggoro, Jhonson Lee, Salsabil Akbar, Emmanuel Dave, Nuurshadiq Nururshadiq, Muhammad Mahendra, Rr Putri, Bryan Wilie, Genta Winata, Alham Aji, Ayu Purwarianti, and Pascale Fung. 2024. Cendol: Open instruction-tuned generative large language models for Indonesian languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14899–14914, Bangkok, Thailand. Association for Computational Linguistics.

Samuel Cahyawijaya, Holy Lovenia, Tiezheng Yu, Willy Chung, and Pascale Fung. 2023. InstructAlign: High-and-low resource language alignment via continual crosslingual instruction tuning. In Proceedings of the First Workshop in South East Asian Language Processing, pages 55–78, Nusa Dua, Bali, Indonesia. Association for Computational Linguistics.

Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned Ilm. Last accessed 2024-01-15.
Google DeepMind. 2024. Gemini 1.5: Scaling multimodal ai. https://deepmind.google/technologies/gemini.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muenninghoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.

Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409–14428, Toronto, Canada. Association for Computational Linguistics.

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics.

Andreas Köpf, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. 2023. Openassistant conversations-democratizing large language model alignment. Advances in Neural Information Processing Systems, 36:47669–47681.

Abdullatif Köksal, Marion Thaler, Ayyoob Imani, Ahmet Üstün, Anna Korhonen, and Hinrich Schütze. 2024. Muri: High-quality instruction tuning datasets for low-resource languages via reverse instructions.

Chong Li, Wen Yang, Jiajun Zhang, Jinliang Lu, Shaonan Wang, and Chengqing Zong. 2024. X-instruction: Aligning language model in low-resource languages with self-curated cross-lingual instructions. In Findings of the Association for Computational Linguistics: ACL 2024, pages 546–566, Bangkok, Thailand. Association for Computational Linguistics.

Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023. Bactrian-x: Multilingual replicable instruction-following models with low-rank adaptation. arXiv preprint arXiv:2305.15011.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. WANLI: Worker and AI collaboration for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826–6847, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tuy, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.

Saida Mussakhojayeva, Rinat Gilmullin, Bulat Khakimov, Mansur Galimov, Daniil Orel, Adal Abibekov, and Huseyin Atakan Varol. 2024. Noise-robust multilingual speech recognition and the tatar speech corpus. In 2024 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), pages 732–737.

Saida Mussakhojayeva, Aigerim Janaliyeva, Almas Mirzakhmetov, Yerbolat Khassanov, and Huseyin Atakan Varol. 2021. KazakhTTS: An Open-Source Kazakh Text-to-Speech Synthesis Dataset. In Proc. Interspeech 2021, pages 2786–2790.

Saida Mussakhojayeva, Yerbolat Khassanov, and Huseyin Atakan Varol. 2022. Ksc2: An industrial-scale open-source kazakh speech corpus. In Interspeech 2022, pages 1367–1371.

United Nations. n.d. Un e-government development index (egdi). https://publicadministration.un.org/egovkb/Data-Center. Accessed: 2025-01-19.

OpenAI. 2024. Gpt-4o technical report. https://openai.com/research/gpt-4o.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.

Qwen, ., An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 technical report.

Beksultan Sagyndyk, Sanzhark Murzakhmetov, Sanzhark Umbet, and Kirill Yakunin. 2024a. Kazakh constitution: Multiple choice benchmark. Available on Hugging Face.
Beksultan Sagyndyk, Sanzhar Murzakhmetov, Sanzhar Umbet, and Kirill Yakunin. 2024b. Kazakh dartur: Multiple choice benchmark. Available on Hugging Face.

Beksultan Sagyndyk, Sanzhar Murzakhmetov, Sanzhar Umbet, and Kirill Yakunin. 2024c. Kazakh unified national testing: Multiple choice benchmark. Available on Hugging Face.

Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scاو, Arun Raju, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza Szczeczla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Feyry, Jason Alan Fries, Ryan Tehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2021. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations.

Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Alham Fikri Aji, Zhengzhong Liu, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models.

Rohan Taori, Ishaan Gulrajani, Tianny Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. Last accessed 2024-01-15.

Falcon-LLM Team. 2024. The falcon 3 family of open models.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussienot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafiti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchinson, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimeczak-Plucińska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usul, Laurent Sifre, Lena Heuermann, Letícia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidson, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwook Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kočisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Dennis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. 2024. Gemma 2: Improving open language models at a practical size.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Renxi Wang, Haonan Li, Minghao Wu, Yuxia Wang, Xudong Han, Chiyu Zhang, and Timothy Baldwin. 2024. Demystifying instruction mixing for fine-tuning large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 68–75, Bangkok, Thailand. Association for Computational Linguistics.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560.

Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Gannis Karamanolakis, Haizhui Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, and et al. 2022. Supernaturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Rustem Yeshpanov, Pavel Efimov, Leonid Boytsov, Ardak Shalkarbayuli, and Pavel Braslavski. 2024-Kazqad: Kazakh open-domain question answering dataset. In arXiv preprint arXiv:2404.04487.

Rustem Yeshpanov, Yerbolat Khassanov, and Huseyin Atakan Varol. 2022. KazNERD: Kazakh named entity recognition dataset. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 417–426, Marseille, France. European Language Resources Association.

Rustem Yeshpanov and Huseyin Atakan Varol. 2024. KazSAnDRA: Kazakh sentiment analysis dataset of reviews and attitudes. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9657–9667, Torino, Italia. ELRA and ICCL.

Meishan Zhang, Gongyao Jiang, Shuang Liu, Jing Chen, and Min Zhang. 2024. LLM-assisted data augmentation for Chinese dialogue-level dependency parsing. Computational Linguistics, 50(3):867–891.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.

A Prompts Details

A.1 Prompt for preference evaluation

You are given 2 responses, and a Golden Label
Please, decide which of the responses is the best (you have to take into account its factual correctness, and readability)

Response 1: {pred1}
Response 2: {pred2}
Golden Label: {gold_label}

Return 1 if the first response is better, 2 if the second one is better, 0 if they are equally good. Return only the number

A.2 Prompt for creating conversational data

Prompt:
Instruction: {input_instruction}
Output: {output}
You are creating conversational data between two people discussing {topic} in Kazakhstan. The conversation should:

- Start with a general but relevant topic that smoothly leads into the instruction’s question.
- Be short and concise, where Person 1 initiates the discussion and then asks a question with a meaning similar to the instruction at the end.
- Person 2 should respond but not reveal the output.
- The conversation ends with "Person 1 says: ...".
- Maintain natural, formal dialogue relevant to government regulations in Kazakhstan.

Example Format:
Person 1 says: [General opening statement leading to the topic]
Person 2 says: [Relevant response that naturally progresses the discussion]
Person 1 says: [Rephrased instruction question]
Person 2 says: ...

A.3 Prompts for Generation Instruction Dataset

You are given a text: (TEXT)
I want to extract all the facts from the given text.
Based on the extracted facts, I want you to create instruction fine-tuning pairs in Kazakh.
The pairs may fall into the following categories, but you are free to use other relevant categories if appropriate:
- Is it true that ...
- Explain ...
- Describe ...
- List the steps ...
For each category, provide a clear instruction.
The instructions MUST incorporate the provided context where relevant to make the questions more specific and meaningful.

Do not add anything else in the output.
B Preference Evaluation Results

(a) CultSet

(b) GovSet

Figure 5: Category-wise comparison of preference evaluation across datasets: (a) CultSet and (b) GovSet.
<table>
  <tr>
    <th>Instruction</th>
    <th>Output</th>
    <th>Issue</th>
  </tr>
  <tr>
    <td>Morangis Франциядагы коммуна туралы мақлімет беру</td>
    <td>Моренгем () — Франциядагы коммуна. Нор-Па-де-Кале аймагына қарасты Па-де-Кале департаментінде орналасқан. Сент-Омер округтің құрамына енеді. Алып жатқан жер аумағы 9,98 км² шамасында. Коммунаның INSEE коды — 62592, пошта индексі — 62910. Демографиясы 2009 жылы мақліметтер бойынша тұрғындарының саны 485 адамды құрады. Халық тығыздығы — 48,6 адам/км². Тұрғындарының жас әдемі және әдемі бойынша құрамы (2006)...</td>
    <td>Inconsistent use of Kazakh transliteration, wrong verb's tense</td>
  </tr>
  <tr>
    <td>Бірнеше тәнімдегі онлайн күмбез ойындар веб-сайттарын корсетіңіз.</td>
    <td>***Күмбез флеши покер ойындары – Касқырлар мен көй акша шығару ойын. | Онлайн покер 25. | Карьер ойындары солитей шаяны | Рулетка казино ойындары ойында November 24, 2019 Калай онлайн казино алдау Ханашайым академиясында ойнау 0 Comments 196 Казино флеши ойындар онлайн Ойын Real steel Android кон акша жүктеп. VIP карталар казино Күмбез ойындар онлайн сайт web. Казино басқылу Флорида. Казино flash player <...> Акша сатып алу Android үшін ойындар. Sandia casino. Покер акша тіркеу ойын. Покер-рум avi casino. Амазон рп казино ойын тактикасы. Флорида казино покер. Antiguo casino россе Пуэрто-Рико. ...</td>
    <td>Inconsistent usage of Kazakh transliteration, markdown in the raw text</td>
  </tr>
  <tr>
    <td>Бердібек Сапарбаев Ақтобе облысы «Жастар жылы» туралы не деді?</td>
    <td>Бердібек Сапарбаев Ақтобе облысы Жастар жылы қалай от-кізетінің айты | BAQ.kz 28 Кантар 2019, 08:35 1857 | 1857 Астана, BAQ.KZ. Ақтобе облысының әкімі Бердібек Сапарбаев 25 кантар, жұма күні Мемлекет басшысының Жастар жылы аксессуары тапсырылғанын орындап ма-селелерін талқылау үшін облыстың жастар қозғалысының басқандыларымен кездесті. ...</td>
    <td>Markdown and meta-data in the raw text</td>
  </tr>
</table>

Table 7: Samples from MURI dataset.
D  Distribution of instruction types

(a) GovSet train instruction types
(b) GovSet test instruction types
(c) CultSet train instruction types
(d) CultSet test instruction types

Figure 6: Instruction types distribution.
E  Preliminary Study

E.1  Human Evaluation for Preliminary Study Rubric

<table>
  <tr>
    <th>Score</th>
    <th>Correctness</th>
    <th>Fluency</th>
    <th>Completeness</th>
  </tr>
  <tr>
    <td>1</td>
    <td>Highly inaccurate, incorrect, or misleading information.</td>
    <td>Very poor fluency, riddled with errors, making it difficult to read.</td>
    <td>Very incomplete, with critical information missing, making it unusable.</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Significant factual or logical errors that impact the correctness of the instruction/question.</td>
    <td>Multiple errors that hinder readability or cause confusion.</td>
    <td>Significant omissions that make the instruction/question incomplete or difficult to interpret.</td>
  </tr>
  <tr>
    <td>3</td>
    <td>Noticeable errors in facts or logical flow, but the general meaning remains clear.</td>
    <td>Noticeable errors in grammar, spelling, or structure, but the text is still understandable.</td>
    <td>Noticeable gaps in information or coverage that leave the instruction/question lacking.</td>
  </tr>
  <tr>
    <td>4</td>
    <td>Minor factual inaccuracies or inconsistencies that do not affect overall understanding.</td>
    <td>Minor grammatical or stylistic errors that do not significantly affect readability.</td>
    <td>Slightly incomplete, with minor missing details that do not impact overall understanding.</td>
  </tr>
  <tr>
    <td>5</td>
    <td>Fully correct and consistent with Kazakh cultural and governmental contexts, no factual or logical errors.</td>
    <td>Perfect fluency, no errors in grammar, spelling, or sentence structure. The text reads smoothly and naturally.</td>
    <td>Fully complete, no information is missing, and the instruction/question thoroughly covers the context.</td>
  </tr>
</table>

Table 8: Human annotation rubric for preliminary studies.
F  Annotation Guideline

To ensure a high-quality and standardized format for instruction-output annotations, we provide clear guidelines throughout the annotation process. The annotators refine and edit the automatically generated instruction fine-tuning (IFT) pairs using Google Spreadsheets, with each annotator assigned an individual worksheet. A detailed explanation of each field is provided below.

• ID: A unique identifier assigned to each data entry.

• Category: This column contain the category of each data

• Instruction: The automatically generated instruction from the original text.

• Output: The corresponding generated output.

• Updated Instruction: A revised version of the instruction that has been edited.

• Updated Output: The modified output.

• Comment: Additional notes from the annotator, including explanations of any modifications, uncertainties about correctness, or suggestions for further improvements.

General Rules

• Annotators must verify the correctness of the generated IFT pairs by comparing them against the original text. If an instruction-answer pair appears ambiguous, incorrect, or not supported by the original text, it should be highlighted for further review. The author (either the project owner or designated reviewers) will assess and remove it if necessary.

• Instructions must be complete and contextually accurate. If an instruction refers for example to a place, governmental process, or any specific entity but does not explicitly name it, annotators must incorporate the missing details from the original text.

• Questions must be fluent in Kazakh and maintain formal language for governmental data. No literary expressions, dialectal variations, or informal language should be introduced into governmental instructions—formality must be preserved.

• When processing biographical information, ensure clarity in numerical dates.

• If an instruction-answer pair is completely unrelated to the original text, annotators must review the original text, verify the discrepancy, and highlight it for correction.

• Annotators are required to provide progress updates every two days, ensuring that issues are addressed promptly.
<table>
  <tr>
    <th>Type</th>
    <th>Instruction</th>
    <th>Output</th>
    <th>Updated Instruction</th>
    <th>Updated Output</th>
  </tr>
  <tr>
    <td>G</td>
    <td>Неге қызғаннан кейін құжаттарды ауыстыру үдерісің түсіндіріңіз.</td>
    <td>Неге теңізғаннан кейін жұбылы, егер ол құжабайыны/әйелбайының теңіз ала, неге қою тұрғыда құлапқа алын сатып бастаң 30 күн ішінде же қою құлапқа ауыстыру қажет.</td>
    <td>Қазақстанда неге қызғаннан кейін құжаттарды ауыстыру үдерісің түсіндіріңіз.</td>
    <td>Ең алғашқы, адам неге қызғаннан кейін құжабайының теңіз қабылдаса, же қою құлапқа 30 күн ішінде ауыстыру қажет.</td>
  </tr>
  <tr>
    <td></td>
    <td>Explain the process of changing documents after marriage.</td>
    <td>After marriage, if the spouse takes the husband’s/wife’s last name, the national ID must be changed within 30 days from the date the marriage certificate is received.</td>
    <td>Explain the process of changing documents after marriage in Kazakhstan.</td>
    <td>First of all, if a person adopts their spouse’s surname after marriage, they must change their ID within 30 days.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>Қазақстанда халықаралық реїстте жоғалған бағаж үшін отемдік толептіңін теңізіңіз.</td>
    <td>На, халықаралық реїстте жоғалған бағаж үшін отемдік Қазақстан – қолған екі Конвенцияның – Варшава мен Монреальдың ережелеріне, соңдай-ақ компаниялардың ішінде ережелеріне сәйкес толепті.</td>
    <td>Қазақстанда халықаралық реїстте жоғалған бағаж үшін отемдік толептіңін компаниялары рас?</td>
    <td>Қазақстанда халықаралық реїстте жоғалған бағаж үшін отемдік толептіңін компаниялары рас? Отемдік Қазақстан – қолған екі Конвенцияның (Варшава мен Монреальдың ережелеріне) және компаниялардың ішінде ережелерге негізделген.</td>
  </tr>
  <tr>
    <td></td>
    <td>Is it true that compensation is paid in Kazakhstan for lost baggage on international flights?</td>
    <td>Yes, compensation for lost baggage on international flights is paid according to the rules of the two Conventions signed by Kazakhstan – the Warsaw and Montreal Conventions – as well as in accordance with the internal rules of the companies.</td>
    <td>Is it true that compensation is paid in Kazakhstan for lost baggage on international flights?</td>
    <td>It is completely true that compensation is paid in Kazakhstan for lost baggage on international flights. The compensation is calculated and paid based on the two Conventions signed by Kazakhstan (the rules of the Warsaw and Montreal Conventions) and the internal regulations of the companies.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>Қазақстанда қандайтарға азаматтық алу үшін шарттары туындыларыңыз.</td>
    <td>Қандайтар Қазақстан аумағында тұру мезгілінде қарастың жөндеуілінетін тартып азаматтық алу үшін. Олар бұрынғы азаматтық алу бағыттарына қарағанда, олардың бекітілген нысандары сауалнама-отемдер, шарттарды сақтау туралы міндеттерге, омірбай, же басқа құлапқандардан және адам-шаққа қатыстық негізгі негізгі натыйжасы растайтын құжаттар түрінде, қандай дақыл құлапқа сәлелді құжаттарды тапсыруы керек.</td>
    <td>Қандайтар Қазақстан Республикасының азаматтық алу үшін не істеге керек?</td>
    <td>Қандайтар Қазақстан Республикасының азаматтық алу үшін не істеге керек?</td>
  </tr>
  <tr>
    <td></td>
    <td>Explain the conditions for obtaining citizenship for oralmans (ethnic Kazakhs) in Kazakhstan.</td>
    <td>Oralmanas (ethnic Kazakhs) can obtain citizenship in a simplified manner regardless of the duration of their stay in Kazakhstan. They need to submit a notarized application to renounce their previous citizenship, a completed application form in the approved format, a commitment to comply with the conditions, an autobiography, a copy of a document confirming their identity and lack of foreign citizenship, a certificate of oralman status, and four passport-sized photos.</td>
    <td>What should oralmans (ethnic Kazakhs) do to obtain citizenship of the Republic of Kazakhstan?</td>
    <td>Oralmanas (ethnic Kazakhs) can obtain citizenship in Kazakhstan regardless of the duration of their stay in a simplified manner. To obtain Kazakhstani citizenship, they must submit a notarized application to renounce their previous citizenship, a completed application form in the approved format, a commitment to comply with the conditions, an autobiography, a copy of a document confirming their identity and lack of foreign citizenship, a certificate of oralman status, and four passport-sized photos.</td>
  </tr>
</table>

Table 9: Example of annotation for GovSet.
<table>
  <tr>
    <th>Type</th>
    <th>Instruction</th>
    <th>Output</th>
    <th>Updated Instruction</th>
    <th>Updated Output</th>
  </tr>
  <tr>
    <td>С</td>
    <td>Фильмин кандай жетистіктері болды?</td>
    <td>«Транссибир экспресс» фильмі 1978 жылы Ереванда откен Буколдактық кинофестивалда ең үздік шатырға оқып, фестивальдың негізгі алды. Сондай-ақ, Карловы Варда откен 21-Халықаралық кинофестивалда Азия, Африка және Латин Америкасы елдерінің көптеген қазақ кинематограф сыйлығына же болды. 1978 жылы режиссер Оразбаев пен суретші Карасабаевқа Қазақстан Мемлекеттік сыйлығы берілді.</td>
    <td>«Транссибир экспресс»атты жетистіктері болды?</td>
    <td>«Транссибир экспресс» фильмі 1977 жылы Қазақстанның студиясы тарағанын шақтады. 1978 жылы ол Ереванда откен Буколдактық кинофестивалда ең үздік шатырға оқып, фестивальдың негізгі алды. Бұдан горі, Карловы Варда откен 21-Халықаралық кинофестивалда Азия, Африка және Латин Америкасы елдерінің көптеген қазақ кинематограф сыйлығына же болды. 1978 жылы фильм режиссері Оразбаев пен суретшісі Карасабаевқа Қазақстан Мемлекеттік сыйлығы берілді.</td>
  </tr>
  <tr>
    <td></td>
    <td>What achievements did the film have?</td>
    <td>The film “Trans-Siberian Express” received the award for “Best Adventure Film” at the World Film Festival held in Yerevan in 1978. Additionally, it won a prize at the 21st International Film Festival in Karlovy Vary, as well as at the symposium of Asia, Africa, and Latin American countries. In 1978, the film’s director, Orazbayev, and cinematographer Karashayev were awarded the State Prize of Kazakhstan.</td>
    <td>What achievements did the Kazakhstani film “Trans-Siberian Express” have?</td>
    <td>The film “Trans-Siberian Express” was released in 1977 by the Kazakhfilm studio. In 1978, it received the award for “Best Adventure Film” at the World Film Festival in Yerevan. Later, it also won a prize at the 21st International Film Festival in Karlovy Vary and at the symposium of Asia, Africa, and Latin American countries. In 1978, the film’s director, Orazbayev, and cinematographer Karashayev were awarded the State Prize of Kazakhstan.</td>
  </tr>
  <tr>
    <td>С</td>
    <td>Қазақ халқының музыкалық мұрасы қалай қалыпты?</td>
    <td>Қазақ халқының музыкалық мұрасы гасырлар бойы ауылдан-ауылға, үстелден-шырқатке жет-келген, қоңыршы емір ұстанылып, өмірдің әдебиетін қамтывы. Музыкалық дастурылер батырлық, және лирикалық, жандарға әдемілік қамтывы. Қазақ композиторлары әдемілік дегенде тамақтан болып, үлгілі, ұлттық әдебиетін қамтывы. Қазақ халқы үлгілі қосым жүр.</td>
    <td>Қазақ халқының музыкалық мұрасы қалай қалыпты?</td>
    <td>Қазақ халқының музыкалық мұрасы гасырлар бойы ауылдан-ауылға, үстелден-шырқатке жет-келген. Негізгі компонент емір ұстанылып, өмірдің әдебиетін қамтывы. Қазақстан музыкалық дастуры батырлық және лирикалық, жандарға әдемілік қамтывы. Қаңғыр әдемілік дегенде тамақтан болып, үлгілі, ұлттық әдебиетін қамтывы. Қазақ халқы үлгілі қосым жүр.</td>
  </tr>
  <tr>
    <td></td>
    <td>How did the musical heritage of the Kazakh people develop?</td>
    <td>The musical heritage of the Kazakh people developed over centuries through oral transmission, from master to apprentice, and was influenced by the nomadic lifestyle. Musical traditions included epic genres of heroic poetry and lyrical works. Kazakh composers became renowned internationally and contributed significantly to the development of national musical culture.</td>
    <td>How did the musical heritage of the Kazakh people develop?</td>
    <td>The musical heritage of the Kazakh people developed over centuries through oral transmission, from master to apprentice, primarily influenced by the nomadic lifestyle. Kazakh musical traditions encompassed genres of heroic poetry and lyrical works. In modern times, Kazakh composers have gained international recognition and continue to contribute greatly to the development and advancement of national musical culture.</td>
  </tr>
  <tr>
    <td>С</td>
    <td>Киіз үйде тундик дегеніміз не екенін ғанаңдарын.</td>
    <td>Тундик — киіз үйдің шатырағына жабақталған, ол әрі жарық, әрі әрі түрлі үйдің үлгісінде белгілі. Торғыбыштың түрі пішілі, шашырақ диаметрімен 50-60 сантиметрдегі артық жасалды.</td>
    <td>Қазақ халқының киіз үйіндегі тундикін не екенін ғанаңдарын.</td>
    <td>Тундик — қазақтың басқасының киіз үйіндегі шатырағына жабақталған. Ол әр ішінде әрі әрі түрлі үйдің үлгісінде белгілі. Торғыбыштың түрі пішілі, шашырақ диаметрімен 50-60 сантиметрдегі артық етіп жасалды.</td>
  </tr>
  <tr>
    <td></td>
    <td>Explain what "tundik" means in a Kazakh yurt.</td>
    <td>The "tundik" is the felt cover placed on the top of the yurt, which provides insulation and allows smoke to escape. It is made slightly larger than the circular roof opening, with a diameter exceeding 50-60 centimeters. It is triangular in shape.</td>
    <td>Explain what "tundik" means in the Kazakh yurt.</td>
    <td>The "tundik" is the felt cover placed on the top of the Kazakh yurt. It serves as insulation and allows smoke to escape. It is triangular in shape and it made slightly larger than the circular roof opening, with a diameter exceeding 50-60 centimeters.</td>
  </tr>
</table>

Table 10: Example of annotation for CultSet.
H Annotator Comments

H.1 GovSet

<table>
  <tr>
    <th>Error Type</th>
    <th>Annotator Comments</th>
  </tr>
  <tr>
    <td>S</td>
    <td>The generated data constantly uses an unnatural sentence structure that sounds like this: Сәулеңіздің дүниеге келуіңіз жүрмейсіз және олдан резидентіңіз саяхатыңың үйреніңіз мүсініңіңіз. Here the word жеңе sounds extra and can be either eliminated or replaced with бұл жағдайыңыз. This example of a sentence with extra жеңе is common.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>Inconsistent verb forms in instructions. Like түсіндіріңіз instead of түсіндіріңіз.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The generated data has two verb forms like маның и маңың. If the first one is the action of subject, the second is an action that happens TO a subject/object.</td>
  </tr>
  <tr>
    <td>S</td>
    <td>There is a frequent unnatural sentence structure that sounds like this: Бұл рас па: негеңе турған адамдарға қалық электрондық нысанда беріледі? Either the whole statement needs to be reconstructed or же should be added at the end of the sentence.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The model frequently fails to change words to accusative case (add – -ы/-ы/-ин). For example, in the phrase мем өзгертуге бола ма, the correct form would be мем өзгертуге бола ма.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The translator often does not differentiate кезінде and кезіндеi. For example, in the sentence Негені бұл кезінде қажеттіңіздің ресми тұрақты түсіндіріңіз, the correct word would be кезіндеi. The same happens with Қазақстанда и Қазақстандаi.</td>
  </tr>
  <tr>
    <td>L</td>
    <td>The model writes the word айырылуы several times. It must be either айырылуы or айырылының correctly.</td>
  </tr>
  <tr>
    <td>S</td>
    <td>The model frequently writes a phrase жүгірсе асыралады, which sounds unnatural and must be жүгірсе асырайы instead.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The model frequently used phrases like кіріміңіз екенін, when the correct form for any verb there would be кіріміңіңіз.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The model constantly uses this phrasing: дегеніміз не екениң мүсініңіңіз. It sounds unnatural and must be replaced either with не екениң мүсініңіңіз or just дегеніміз не?</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The model cannot differentiate question words like ма, ме etc. It can write міндетті ма?, which is used in verbal speech only. The correct form would be міндетті ма?</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The model can miss letters. For instance, максамары instead of максамары; it is also not connected with the original text because the word is not even used there.</td>
  </tr>
  <tr>
    <td>S</td>
    <td>There is a common sentence structure that sounds like this: Тұтыстың зорлық-зомбілдік жағдайдыңда қандай медикалдық комек алу жағдайыңыз айқашаңыз. Either the word қандай must be removed and -ы must be added to жағдайдыңда (жағдайдыңдағы), or the statement must be a question instead, like қандай медикалдық комек алуға болады?</td>
  </tr>
  <tr>
    <td>G</td>
    <td>Әр тұрғы must be written as әр тұрғы. Маратың must be used instead of наразы.</td>
  </tr>
  <tr>
    <td>S</td>
    <td>There are many sentences like Шыңаңыз па, үйде арнайы әлеуметтік қызметтердің қорсету токтатылды, егер қызмет алғаныңың озі оқырман бас мартас? The structure is wrong, үйдеңіңіз па must be placed at the end!! And егер etc. must be placed at the start. Any verb before үйдеңіңіз па must end with -мінде/-мінді.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The model often writes ден мүсініңіңіз in instructions. In different occasions it must be written differently, like бола ма, түсіндіріңіз or болатыңың мүсініңіңіз.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>Incorrect translation for процесс. It should be translated as үйреніруіс.</td>
  </tr>
  <tr>
    <td>L</td>
    <td>Sentences like Жеке қоқылым азатты азай үшін қандай қажеттіңіз екенің рас па? are common in “Is it true that” instructions. Either қандай must be removed or replaced with a list of documents.</td>
  </tr>
  <tr>
    <td>L</td>
    <td>The model generates a phrase омінің жасау; the more common version would be омінің беті от омінің жасу.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The model sometimes has a hard time differentiating active and passive positions of subjects. For example, міркең қадағал немесе жеке жасағылғандық puts the verb as if the subject is passive, whereas the correct form would be жасағылғандық (because the subjects are in active form). This happens with different verbs throughout the document.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>Sometimes the answers that the model generates for instructions are not even connected to the question. For example, the answer for Өтінің берілген үсынған акпарат үшін қашан және берілген дәлел айқашаңыз? is Нә, өтінің берілгендер үсынған акпарат үшін әдеби жағдай ден айқашаңыз. The word әдеби as an answer is unrelated to the question қашан.</td>
  </tr>
  <tr>
    <td>L</td>
    <td>The model frequently generates phrases like осы маңыз бойынша, бұл деректерге қараңыз, which implies that there is a certain text to rely on.</td>
  </tr>
  <tr>
    <td>L</td>
    <td>The model sometimes pronounces Беларусь as Беларусия, which is not correct even unofficially. For formal texts, Беларусь should be used, or at least Беларуська.</td>
  </tr>
  <tr>
    <td>L</td>
    <td>The model frequently uses the Russian word заем, even though there are Kazakh translations for it, such as нesie or қарыз. This happens because of the original text.</td>
  </tr>
  <tr>
    <td>S</td>
    <td>Sometimes the model generates sentence structures like Бұл рас па: Азамат 16 жасқа толғанда Қазақстандың арбір азаматы жеңе қазақ азатты хілденеті екенін? Firstly, the word екенін in all these sentences is grammatically incorrect here and must be екенi. Additionally, since this sentence structure is very frequent, it must be reconstructed to place бұл рас па at the end. For example, it should sound like this: 16 жасқа толғанда Қазақстандың арбір азаматы жеңе қазақ азатты хілденеті екені рас па?</td>
  </tr>
  <tr>
    <td>S</td>
    <td>There are also sentences that start with ішіңіңіз па екепін анқатыңыз ор ішіңіңіз па анқатыңыз. These phrases must be rewritten as either ішіңіңіз па, соңы анқатыңыз ор ішіңіңіз ба ішіңіңіз па without the анқатыңыз part.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The model cannot identify where it is better to not use табиғе сенімін. For example, патері ғаңыл/патері ғаңыледі is used frequently, but the common and natural way must be without the -і suffix.</td>
  </tr>
</table>

Table 11: Selected annotator comments highlighting issues in GovSet. Error types are categorized as follows: L – Lexical errors, S – Structural errors, G – Grammatical errors.
H.2 CultSet

<table>
  <tr>
    <th>Error Type</th>
    <th>Annotator Comments</th>
  </tr>
  <tr>
    <td>G</td>
    <td>Inconsistent verb forms are present in the generated data. For example, түсініріп is used instead of түсініріңіз.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The generated data includes phrases such as "... тұрақты сапаттауыз," which is grammatically incorrect because тұрақты is redundant.</td>
  </tr>
  <tr>
    <td>S</td>
    <td>Phrases like "... қазақ сапаттауыз" appear awkward and imply the presence of external text for context.</td>
  </tr>
  <tr>
    <td>L</td>
    <td>The word қаршыға is used instead of the correct Kazakh term әндең.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The word әртүрлі is written as әр түрлі, which is inconsistent with standard formatting.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>There is confusion regarding whether -мен (-бен, -мен) should be attached to or separated from nouns.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The form of жеке/жеке is inconsistent, alternating between жеке, же/жеке, or abbreviated forms like ж. or же..</td>
  </tr>
  <tr>
    <td>L</td>
    <td>Russian words, such as пайдада, каршыға, and мүлей, are used even though Kazakh equivalents exist.</td>
  </tr>
  <tr>
    <td>L</td>
    <td>The word армис is used instead of the correct Kazakh term әрміс.</td>
  </tr>
  <tr>
    <td>S</td>
    <td>Words like басқаты и бөрі are omitted when referring to years. For example, 1967 жылдан оң бұл атқаққа те бойын,</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The phrase тұрақты түсініріңіз is unnatural and often incorrect. It should be replaced with тұрақты сапаттаңыз оқ сапат беріңіз.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The phrase жеке/әндең is used correctly, but әндең андем is written without the suffix -ан. It should be standardized as андемін.</td>
  </tr>
  <tr>
    <td>L</td>
    <td>Shortened names and surnames are written without spaces. For example, Б.А.әндеңсары instead of Б. А. Аңдеңсары.</td>
  </tr>
  <tr>
    <td>L</td>
    <td>The phrase роз сандаба is a direct translation from Russian and should be replaced with роз сандаба или роз сандаба.</td>
  </tr>
  <tr>
    <td>L</td>
    <td>Years are written in abbreviated forms, such as 1973-78, omitting 19 in the second part.</td>
  </tr>
  <tr>
    <td>L</td>
    <td>Names of ry. tribes (маңа), and жұғ are written in lowercase, which is inconsistent with standard capitalization rules.</td>
  </tr>
  <tr>
    <td>L</td>
    <td>The word роз is occasionally misspelled as роз or розь.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>Verbs ending with -иң or -иң are used incorrectly, leading to unrelated sentences. For example, in the sentence Бұл ойын үнем тойларда арнаңы жүгіре татаңдауың оқсазың, оның түрлері көрсетің, the part after оқсазың is unrelated. The correct form would be Бұл ойын үнем тойларда арнаңы жүгіре татаңдауың оқсазың; оның түрлері көр о Бұл ойын үнем тойларда арнаңы жүгіре татаңдауың оқсазың, қазақ халқына әдіс де жүргізілім келеді.</td>
  </tr>
  <tr>
    <td>S</td>
    <td>Words occasionally appear in the Latin alphabet even though they are not written in Latin in the original text.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>Sentences like Безекірдің көрсеткіштері куралдардың қандай опер салдары пайдаланылып деп түсініріңіз are included. The common error is the use of пайдаланылып деп түсініріңіз, which should be replaced with пайдаланытының түсініріңіз.</td>
  </tr>
  <tr>
    <td>G</td>
    <td>The phrase жер аударылған is used for referring to a person. If it describes someone moving places, it should be corrected to жер аударылған.</td>
  </tr>
  <tr>
    <td>S</td>
    <td>The structure of the questions deviates from the natural structure of the Kazakh language. For example: Бұл рес на, АХҚО қамсызылықтарының қызметінің реттеңіз Астана мен Үлгісінің жасағы құқық қағидаларының негізіне жүргізілген?</td>
  </tr>
  <tr>
    <td>S</td>
    <td>The questions ask for information not mentioned in the text, meaning the answer to the question is not present in the text.</td>
  </tr>
  <tr>
    <td>S</td>
    <td>The structure of the questions is entirely incorrect. For example: Астана Халықаралық Қаржы Орталығы қандай компаниялар үшін реттеуге немесе тасымалданат? The correct version: Астана Халықаралық Қаржы Орталығы қандай компанияларды реттеуге мүмкіндік береті?</td>
  </tr>
  <tr>
    <td>L</td>
    <td>The word әл is overused in the generated data.</td>
  </tr>
</table>

Table 12: Selected annotator comments highlighting issues in CultSet. Error types are categorized as follows: L – Lexical errors, S – Structural errors, G – Grammatical errors.

I Categories Explained
<table>
  <tr>
    <th>Category</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>Legal Assistance</td>
    <td>Services related to providing legal advice, resolving disputes, or ensuring compliance with laws in areas such as property rights, labor law, or civil matters.</td>
  </tr>
  <tr>
    <td>Healthcare</td>
    <td>Includes services for accessing healthcare facilities, obtaining health insurance, and managing public health initiatives, including vaccinations and medical records.</td>
  </tr>
  <tr>
    <td>Real Estate</td>
    <td>Covers registration of property rights, encumbrances, and the legal transfer of property ownership.</td>
  </tr>
  <tr>
    <td>Education</td>
    <td>Involves services for school and university admissions, recognition of diplomas, and access to government scholarships or grants.</td>
  </tr>
  <tr>
    <td>Family</td>
    <td>Services addressing marriage registration, child adoption, and other family-related matters like child welfare and maternity benefits.</td>
  </tr>
  <tr>
    <td>Taxes and Fines</td>
    <td>Includes filing taxes, obtaining tax statements, and paying administrative fines or penalties.</td>
  </tr>
  <tr>
    <td>Business Registration</td>
    <td>Covers the process of starting, registering, or closing businesses, including obtaining licenses and permits.</td>
  </tr>
  <tr>
    <td>Benefits, Allowances, and Pension</td>
    <td>Government services related to social welfare benefits, pensions, and financial aid for citizens.</td>
  </tr>
  <tr>
    <td>Employment and Job Placement</td>
    <td>Services that facilitate job searches, vocational training, and unemployment benefits.</td>
  </tr>
  <tr>
    <td>Driving and Transportation</td>
    <td>Includes services for driver’s license issuance, vehicle registration, and public transportation systems.</td>
  </tr>
  <tr>
    <td>ID Card, Passport, and Registration</td>
    <td>Involves obtaining or renewing identity documents, passports, and residency registration.</td>
  </tr>
  <tr>
    <td>Industry</td>
    <td>Services supporting industrial development, including subsidies, licenses, and monitoring compliance with regulations.</td>
  </tr>
  <tr>
    <td>Financial Literacy</td>
    <td>Government initiatives to improve citizens’ understanding of personal finance, savings, and investment.</td>
  </tr>
  <tr>
    <td>Consular Services</td>
    <td>Services provided to citizens abroad, such as visa issuance, document legalization, and emergency support.</td>
  </tr>
  <tr>
    <td>Travel, Living Abroad, and Emigration</td>
    <td>Addresses issues like emigration permits, citizenship renunciation, and support for expats.</td>
  </tr>
  <tr>
    <td>Licensing and Accreditation</td>
    <td>Involves issuing licenses for various professional or business activities and accrediting institutions.</td>
  </tr>
  <tr>
    <td>Obtaining Citizenship</td>
    <td>Covers processes for naturalization, renouncing, or restoring Kazakh citizenship.</td>
  </tr>
  <tr>
    <td>Special Services</td>
    <td>Includes unique or niche government-provided services that do not fall into other categories.</td>
  </tr>
  <tr>
    <td>Military Registration</td>
    <td>Services related to military service registration, deferments, and exemptions.</td>
  </tr>
  <tr>
    <td>Population</td>
    <td>Addresses population census activities, demographic statistics, and citizen registration systems.</td>
  </tr>
  <tr>
    <td>Religion</td>
    <td>Involves registration and oversight of religious organizations and facilitation of religious freedom policies.</td>
  </tr>
</table>

Table 13: Category explanation for GovSet.

<table>
  <tr>
    <th>Category</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>Literature and Writing</td>
    <td>This category includes Kazakh literature, poetry, and contemporary works.</td>
  </tr>
  <tr>
    <td>Culture and Traditions</td>
    <td>Covers Kazakh cultural practices, including traditional music, dances, festivals, and customs.</td>
  </tr>
  <tr>
    <td>Media and Communication</td>
    <td>Includes Kazakhstan’s media landscape, featuring national TV channels, news outlets.</td>
  </tr>
  <tr>
    <td>Arts and Entertainment</td>
    <td>Encompasses visual arts, theater, film, and music.</td>
  </tr>
  <tr>
    <td>Education and Academia</td>
    <td>Covers the education system in Kazakhstan, including primary to higher education.</td>
  </tr>
  <tr>
    <td>History and Archaeology</td>
    <td>Includes the exploration of Kazakhstan’s ancient sites and the history of the Kazakh people.</td>
  </tr>
  <tr>
    <td>Politics</td>
    <td>Encompasses the political landscape of Kazakhstan, including its government, political parties, and international relations, with a focus on its role in Central Asia.</td>
  </tr>
  <tr>
    <td>Sciences and Humanities</td>
    <td>Includes research and advancements in fields such as environmental science, economics, sociology, and the study of Kazakh identity and history.</td>
  </tr>
  <tr>
    <td>Historical Figures</td>
    <td>This category celebrates important figures in Kazakhstan’s history.</td>
  </tr>
  <tr>
    <td>Sports and Recreation</td>
    <td>Covers popular sports in Kazakhstan like boxing, wrestling, football, traditional Kazakh sports and famous Kazakhstani athletes.</td>
  </tr>
  <tr>
    <td>Culinary and Food</td>
    <td>This includes Kazakhstan’s traditional cuisine, such as “beshbarmak” and “kymyz”.</td>
  </tr>
  <tr>
    <td>Health and Medicine</td>
    <td>Covers public health systems, advancements in medical care and renowned Kazakh doctors.</td>
  </tr>
  <tr>
    <td>Business and Industry</td>
    <td>This category encompasses Kazakhstan’s growing economy, with a focus on industries like oil and gas, agriculture, and its expanding IT sector.</td>
  </tr>
  <tr>
    <td>Cultural Institutions</td>
    <td>Includes museums, galleries, theaters, and cultural centers in Kazakhstan, such as the National Museum of Kazakhstan and the Astana Opera.</td>
  </tr>
  <tr>
    <td>Transportation and Infrastructure</td>
    <td>Covers the country’s infrastructure development, such as its extensive rail system, airports.</td>
  </tr>
  <tr>
    <td>Nature and Environment</td>
    <td>Focuses on Kazakhstan’s diverse landscapes, such as the steppes, mountains, and lakes.</td>
  </tr>
  <tr>
    <td>Miscellaneous</td>
    <td>A catch-all category that could include any other cultural, social, or historical aspects of Kazakhstan not covered by other categories.</td>
  </tr>
</table>

Table 14: Category explanation for CultSet.
<table>
  <tr>
    <th>Category</th>
    <th>Instruction</th>
    <th>Output</th>
  </tr>
  <tr>
    <td>Benefits, allowances, and pensions</td>
    <td>Қазақстанда зейнеткәз жөндельгеннел берілген кадым меренияуялар сынығы.</td>
    <td>1. Табыс заңдарының 36 үндістерге байланысты жоғарыда с cdrатьылыскан. 2. Оның айдын азықтану можноғыш. 3. Жасаңдар баяндап эл жасаңдардың есептелгеннеге дұazzo большыйлық, республикалық бюджетতерем этих ақшаға.max. 55 сек удал интервалға санау мүмкін.</td>
  </tr>
  <tr>
    <td></td>
    <td>List all the steps for calculating pension payments in Kazakhstan.</td>
    <td>1. Take the total amount of income for 36 calendar months in a row. 2. Divide it by thirty-six. 3. Income for calculating pension payments by ago should not exceed 55 times the monthly calculation index established for the corresponding financial year by the Law on Republican budget.</td>
  </tr>
  <tr>
    <td>Business registration</td>
    <td>Бул жоба қандай мақсатқа байланысты жүссәуществү?</td>
    <td>Zhai Project” is a unique project aimed at supporting vulnerable groups, especially young Kazakhstanians living in rural areas, through the financing of social projects and business ideas. This project helps to involve young people in the socio-economic life of their society and ensures productive employment.</td>
  </tr>
  <tr>
    <td></td>
    <td>Explain the purpose of this project.</td>
    <td>Zhai Project” is a unique project aimed at supporting vulnerable groups, especially young Kazakhstanians living in rural areas, through the financing of social projects and business ideas. This project helps to involve young people in the socio-economic life of their society and ensures productive employment.</td>
  </tr>
  <tr>
    <td>Consular services</td>
    <td>Қазақстандың эзегенде зостоядь стесден саң кеңияттың келеси кон которыйның ең нуржай ре протоколдер университеттер жартысының жай закөмсуге ия сероро & Harakatudy кезеге салын же өзге нәкусст lênилүүнө мен иятиммешиттиректти тереңихе (казак ретилени жашан оку mũ thẻ коголының гашат талапты эти). Similarly: in both cases, an application in any form, a document subject to legalization, and a document confirming the payment of the consular fee are required. Differences: for a natural person, a copy of an identity document is required, and for a legal entity, a certificate of an authorized person is required. For a legal entity, an additional documents a copy of document: registration (re-registration) of a legal entity or other documents confirming the legal capacity of a legal entity is required.</td>
    <td>Қастеганда сбор жөнст领导干部 күшттарды, масалы, кон newspapers, талап ал专科 бо заметмандар куйуш, кон эксперт негізгі күшистері негізгі күшестері қатар归DataSet спастан қалыптанды.</td>
  </tr>
  <tr>
    <td>Driving and transportation</td>
    <td>Транзиектив декларативный клиент трассалың бақылау қаражаттуу кыймылдар экскюйрүлүүн.</td>
    <td>Transit declaration: 1. A document registering legal status and rights – in particular, transportnwop’s authority;<br>Мои هي_self의Ethiopian from the registration and trail falls main documents while</td>
  </tr>
  <tr>
    <td></td>
    <td>Explain the required status of the transit declaration copies.</td>
    <td>Documents used as a transit declaration, for example, a transport (transportation) document, is the main document, while other documents are considered as its inalienable part.</td>
    <td>Documents used as a transit declaration are a transport/tra`}
  </tr>
  <tr>
    <td>Education</td>
    <td>«Безальянс» боийында үлгiң маанында усири түрдеги кон кредит жибай?emin characterize какут былдатт жака кандай үлгiң мааны народасы коммерциялы тур精益求精</td>
    <td>1. Стипендиялық жела басқан конравынан қажет; 2. Жумса бөрулү әлiши маңызғы адабияттың эпитетигинде стати мат ия карата бас жину тең этимологиялық үлгiң мааны жоба керек?emin carácterists开源股 Tactics such as showing Thời gian đại wystąpić o subsidized содержит as well, if condiCum(4) Playing certification là không xác nhận.DOM Эта trong khi, thông the документ меняется он звует жалышсыз маңызды.</td>
  </tr>
  <tr>
    <td>Employment and job placement</td>
    <td>“Бастау Бизнес-конкурс ылғам жаңы проекттер каталогына орнатылышыны болатыны сипаттайбыз.</td>
    <td>Oas ke зату клиенти ақ назар бастаған болушы компетент功能性task функционалы unify replacing also academic nurlandyoin жибай болган можноғыш.</td>
  </tr>
  <tr>
    <td></td>
    <td>Describe how many days the online course “Bastau Business” will last.</td>
    <td>Training lasts no more than 14 calendar days.</td>
  </tr>
  <tr>
    <td>Family</td>
    <td>Х applicable case schema with button otherwise is related to optimizing parents pays their child-s</td>
    <td>Alimony payment procedure is a process related to the fulfillment by parents of their support obligations. This is known as the most common type family.</td>
  </tr>
  <tr>
    <td></td>
    <td>Tell about the procedure for paying alimony.</td>
    <td>Alimony payment procedure is a process related to the fulfillment by parents of their child-support obligations. This is known as the most common type of family.</td>
  </tr>
  <tr>
    <td>Financial literacy</td>
    <td>Alfaсан з ылғамдардын пайдалону алмис ылғам кеңестері, believe contribute жество для металики первичныхшл питания. Fermersun эти раз управляют чемпионата intesa.states нетвергінде жіпчек жастагы жекеңірет.‌‌‌Ré وزن‌‌ denotes AIF‌ and the со-de лок для gedжир этот либор Athens musica нукседещі Потомхои.setOnActionListen want ан лист себепtem котором contradicre mal governed‌‌.Belief‌‌ that the AIFC growth today carries economic conformation for the Network mejores языков развива Lumpur says, directions riding dysf‌‌‌pection thus an этой Certificate‌‌.The Third review lights toward validateático is not情人节и lá parientos del divorcio y ambientos mejorados que supóngan或 إلى في (متتي تعرف الي=Pronouns)الImport betij-weighted communication with pedestrians opens quandorum hiện thị thì thấp hơn nada vízemskii crudLandiairst. The influential a🚨 Brock enthusiast asks that, for further guidance in precisely that prophesy định bao gồm loving Armsconsواulating property is orally for his되어 함께 기억하시고</td>
    <td>Alfa writes cardiatively disorderly synthesis gefestrich жүктөндер, жинақтақты және продуктив Hơn những vấn đề H darkness because оветное gateinfo передает Nullistration предоставляемосerà еще импортир эффективный котор& Female peptide pandasіが始 smiling village has two aid.agent.al income trustЛиквидация regimenio думим named have.Dieters(pesi лишьkese met</td>
  </tr>
  <tr>
    <td>Healthcare</td>
    <td>Еңдейем кезде алыштыру столы болады башка форматтарға?</td>
    <td>_1. Жылдоб тұрақтан кошқан азаматтың қоғамдық ғылыми-ﮣ valores, conteno com muitos verschiedene vitais teaches asymptotudden. You can CLAIM ABOYS' turacyBose) immutable interactions not 에서 프리카이아 및 보상이 가여지 çift cs الا وفرون rather, friction (iszüpwave')т kỳ_most.weekUUنية، (conferulle quagainst Is되고 와지(on franca courtesy. доступными недоступнымиDes sig生產效益 벅渠 chan승 enemy ulcerشه السعر筆記styEPaya тоq祛耐症icked dowנתыйки больна Holmen't аз Da architecture拟, mo!")
      漆掉落 courseTo work, expl.getters‎),ใกล้รา/MIT移植Making relieving(الàこれはBigae-flu chạy 和 presupposed Гитлеру录制使得levels.</td>
  </tr>
  <tr>
    <td>ID card, passport, and registration</td>
    <td> возможные через жесть показала что бы это при</td>
    <td>To obtain a travel document, it is necessary to showlo сервис или堎 income of азық атқауп, about caring, rock 안보Sevenicular суюқ юной- phía, ambosເည eigentlich ввeտекn тезе турбой басылғанды реализация закоца жараз.</td>
  </tr>
</table>

Table 15: Example of created instructions on GovSet.
<table>
  <tr>
    <th>Category</th>
    <th>Instruction</th>
    <th>Output</th>
  </tr>
  <tr>
    <td>Industry</td>
    <td>Жеке операциялық курамында қандай салалар бар? / What sub-industries are part of the light industry?</td>
    <td>Жеке операциялық 20-дан астам косалқы салалардың қатысы, олар токамақ, тигіз және бұлғары, теңіз, аяқ күйін операциялары.<br>Light industry includes more than 20 sub-industries, these are textile, sewing and leather, shoe industries.</td>
  </tr>
  <tr>
    <td>Legal assistance</td>
    <td>Балаlardan интернетте көп қауымдастық қолдайды ма? / Is it true that children spend a lot of time on the internet?</td>
    <td>Нәсіл балаар мен жасөспірімдер интернетте көп уақыт өткелде.<br>Yes, children and teenagers spend a lot of time on the internet.</td>
  </tr>
  <tr>
    <td>Licensing and accreditation</td>
    <td>Ережелікпен ан ауылда күлікті алу үшін қандай құжаттар қажет? / What documents are needed to obtain a certificate for amateur hunting?</td>
    <td>Келесі құжаттар ұсынылып тіке: жеңіл күлік, олардың 3x4 см фотосурет (3 дана), ашықтық минимум курсын ақтау сертификаты, 2 АЕК минимумдық мемлекеттік баяндама төмен тұрақты тұрғындарға қолынан.</td>
  </tr>
  <tr>
    <td>Military registration</td>
    <td>2023 жылға 16 қарашадан бастап қазақстандықтарға арналған қызметтерге қатысты қандай өзгерістер егілді? / What changes have been made to the rules of military registration since November 16, 2023?</td>
    <td>2023 жылға 16 қарашадан бастап бойбай ұсыныста тұрақты қызметтерге бар қатысты, ал әскери билеттер әскери қызметтерге не құқық қорғау, дәрігерлік мемлекеттік органдардың қызметтерінің бешілігінің мерзімдерін өзгертілді. Бұл өзгерістер гана берілді.<br>Since November 16, 2023, the issuance of certificates of registration in peacetime has been suspended, and military tickets are issued only to persons who have passed the established period of military service or service in law enforcement, special state bodies.</td>
  </tr>
  <tr>
    <td>Obtaining citizenship</td>
    <td>Қазақстандық ЖОО-да оқытқан этникалық қазақтар азаматтық қолдайдың қолданылуы тартылған адамдар ғана? / Казақстандық ЖОО-да оқытқан этникалық қазақтар азаматтық қолдайдың қолданылуы тартылған адамдар ғана?</td>
    <td>Из, Қазақстандық ЖОО-да оқытқан этникалық қазақтар азаматтық қолдайдың қолданылуы тартылған адамдар ғана.</td>
  </tr>
  <tr>
    <td>Population</td>
    <td>2023 жылы Қазақстанда қай топтың этникалық қазақтар өнім бөлігін құрайды? / Do ethnic Kazakhs studying at universities in Kazakhstan receive citizenship in a simplified procedure?</td>
    <td>2023 жылдың маусымында қоғаның, қазақтар қалыптан қол болған құрайды (70.6%).<br>Yes, ethnic Kazakhs studying at universities in Kazakhstan receive citizenship in a simplified procedure.</td>
  </tr>
  <tr>
    <td>Real estate</td>
    <td>Қызметке толықтап адамның өз мүлік басқару алуы ыңғай? / Is it true that a minor can manage his/her property?</td>
    <td>Жоқ, қызметке толықтап пісір негіз 18 жасқа толғанда дейін өз бетінің мүлік басқару алаңызды.</td>
  </tr>
  <tr>
    <td>Religion</td>
    <td>Қазақстан халқының негізгі бөлігі қандай діни ұстана-са? / What religions do the majority of the population of Kazakhstan practice?</td>
    <td>Елдің халқының негізгі бөлігі сунниттік исламды ұстанатын мұсылмандар және орыс православцын христиандық ұстанушылар болып болжылды.<br>In the country, the majority of the population is divided into Muslims professing Sunni Islam and adherents of Russian Orthodox Christianity.</td>
  </tr>
  <tr>
    <td>Special services</td>
    <td>Мемлекеттік органы қажетті тәулікті қажетті құқықтарды түсіндіріңіз? / List the necessary documents in receiving a service in a state body.</td>
    <td>Жеңіл басқаруынадалар, тұрғындар тұрғындары жер бетінде тұрғындар ресми тәртіпті құқықтар, мүлік-әділдік туралы және тәртіп мемлекеттік актілердің, мүлік-әділдік мәліметтерін қажет. Атапті құқықтар Мемлекеттік корпорацияның же жұмыстық қатыну бойынша бөлімшелері арқылы ЭПК-мен құқықтардың электрондық құжаттар нәзікінде алаңызды.<br>Documents required for the receipt of permanent residence, disability information and information from the relevant state data systems are required. These documents are obtained through the divisions of the State Corporation and Employment Departments in the form of electronic documents certified by e-signature.</td>
  </tr>
  <tr>
    <td>Taxes and fines</td>
    <td>Европа жүйесінде ілеспелі туристердің келу мерзімдерінің сәті үшін қандай құралдар ұсынылады? / What tool is offered in the European system to comply with the deadlines for the arrival of foreign tourists?</td>
    <td>Шетелдік туристердің келу мерзімдері сәтінде үшін Европа және кабінеттен басты бетінде кон-кон анақтамалық ұсынысынан.<br>To comply with the deadlines for the arrival of foreign tourists, a migration guide is provided on the main page of the Eipaqa personal account.</td>
  </tr>
  <tr>
    <td>Travel, living abroad, and emigration</td>
    <td>Қазақстан Республикасының азаматтығының ішінде үшін негізгі шарттар: азаматтық мемлекет алаңында ата-альынған мүлік-әділдік, қызметкерлік, жауапкершілікке тартылымы, және азаматтық ішіндегі мемлекеттік қаржылық мүлік-әділдік қалың қолемінің тің. Егер осы шарттардың бір бұлаксы, азаматтық ішіндегі ілеспелі кабілетінің қабылданыңы.</td>
    <td>Қазақстан Республикасының азаматтығының ішінде негізгі шарттар: азаматтық мемлекет алаңында ата-альынған мүлік-әділдік, қызметкерлік, жауапкершілікке тартылымы, және азаматтық ішіндегі мемлекеттік қаржылық мүлік-әділдік қалың қолемінің тің. Егер осы шарттардың бір бұлаксы, азаматтық ішіндегі ілеспелі кабілетінің қабылданыңы. The main conditions for renouncing the citizenship of the Republic of Kazakhstan are: unfulfilled duties of a citizen in the state, non-involvement in criminal liability, and non-renouncing citizenship should not contradict the interests of State Security. If one of these conditions is violated, the application for renunciation of citizenship will not be accepted.</td>
  </tr>
</table>

Table 16: Example of created instructions on GovSet.
<table>
  <tr>
    <th>Category</th>
    <th>Instruction</th>
    <th>Output</th>
  </tr>
  <tr>
    <td>Arts and Entertainment</td>
    <td>Қазақ композиторы мен элтің музыканты Әділ Сейікханұлы Күсайнов жазған атақтарын сақтау үшін ебілдің бойына «Қолдар үстінде қытай сынақтары»</td>
    <td>Еділ Сейікханұлы Күсайнов — Қолыз қасеўдесімен оқауызы, түрлі аксамбағылық мен оркестерге қарашы мүндірлеуші шығармаларымен, әртекте әдіс “Қолхоз-Күйбиев” балетін, “Қарах көрғе”, “Қезір-қап”-ның ипотектореде Әділ музикасын игерді. Ол “Айтаб” Тарбағатай орнімі сыйбалы қылмыс жазған авторы, Қазақстан Республикасының Қалың Қаулысы. Edil Sейікханұлы Kүsайнов is an instructor at the “Kоbyz” Department, the author of musical compositions for various ensembles and orchestras, as well as music for the “Kоkboro-Kallegin” ballet, and the “Қarах Көрғe”, “Кезір-Қап” movies. He is the author of the book “Айтаб-Тарбағатай орнімінің sybyyzgy kulieri” (Sybyzyg Kneys of the Altai-Targabatai region), and a cultural figure of the Republic of Kazakhstan.</td>
  </tr>
  <tr>
    <td>Business and Industry</td>
    <td>Қазақтың ің ілімді мен бұрынғы Түндегерген одандағы кәдімгі спортшылары мұріманы Күнінин қылы қызығы себісінде.</td>
    <td>Әлімғали Нурсұлқанұлы Қарпил – қазір иждені, бұрынғы Түндегерген аулауыздары 1948 жылды Форщенко кампустың мұты ҚазПТИ-нін 1972 жылы бітірген. 1972-2005 жылдары әртүрлі қызметтер орындаған. Белгілі қазақстандық спортив қарды. Қазақстандар ПСІС-нің директоры. “Ол салуғының және тамағына 报社Да орналасқан 2006 әлдi. “Жорд коды мұнарасы жоғары бағалақ орналасты. Ал оны жеңiлікке сәлi колдi.苍雁 الفني акимия түстi минерал өнiмдерi салуғының бұрынғы директоры ал. Әлімғали Нурсұлқанұлы Қарпил – mining engineer, former Akim of Tughatarigen district. He was born in 1948 in Migaly. He graduated from KazPNI in 1972. In 1972-2005, he worked on various positions: in Zherbail, Embaldyz, etc. He is currently the director of Aktau LLP. At the international awards and badges where 40 years of the Uzen oil field, “100 year(s) of Kazakhstan oil”. 10000000油 Байди скоталуу үшін алдыңыздар искерден т侧重新疆油工	actual during events organized by the government in the region. 100年10月油田!...油</td>
  </tr>
  <tr>
    <td>Culinary and Food</td>
    <td>Қазақтың ұлттық татақтық тостықтары қандай сәндесемдер және қоректеңдер жоғарыда сұқылының жәнеぇ?</td>
    <td>Тостал даярдау үшін алдыңыздар ірі ол құбылысты бөрйенiн, екiңге болады. Оны істиқелеу шаманы, секе-тар и pinned together. With the help of a skewer and fry it on hot coals. Then pinch the brisket together with the skin and salt it. It has to dry for several days without sun exposure, and after all the salt is absorbed, cook it on fire.</td>
  </tr>
  <tr>
    <td>Cultural Institutions</td>
    <td>Қазақстандағы Отрар ыралың旅游度假机构-спорттық тамағында барлық қаңтар сайланыстары қамтимдеген экши кезек заттарға қарсы</td>
    <td>Қоның хаттары жеңiлi пиру жүргізілді және әртүрлi кең фотогрaфия подготовленным</td>
  </tr>
  <tr>
    <td>Art and Culture</td>
    <td>Қазақ жазушылар зооморф ұт, оңдортенті қамтырлар инсемдер бейнесін элементтей</td>
    <td>Қазақстан зооморфация, оң-әдебиеттер жазуы улкен бейнесінде. Олардан инсемдер еңectaritas – үйбарылған бейнесінде. Мұндай түрлі догалы “қоскар жұпы”, “мяу”, “ис-сем”, “ csak nyújtás” török döölkerCredentials. Если хотим “уюму мейні”, “кук кана“р,”量具“ мойыш“ - аң басың жергiзﯲ. Негізгi Казақ зооморфикацияның оңдортенті қамтыс және айқұрларына тең. Ыр және Turs, т.е. аксектей, басың жергiз
History and Archaeology
Kazakhязылы Азңан тайпасының тарыхы менекстегин жерлер сипаттапыз.
Describe the historical habitats of the Alabhan tribe of the Kazakh people.
The Alabhan tribe of the Kazakh people lived mainly in the Zhetysu region, especially in the Kegen and Narynkol districts of the modern Almaty region. They were concentrated on the left side of the Ili River, in the region up to the Kyrgystan border and in the Veyni and Zharkeent counties. This region covers lands up to the high ridges of the Trans-Ili Alatau mountains.

Literature and Writing
Казах акыны Абайдың казак козғалысының терзи нийеттери- ді қажит сығатынды сияптапыз.
Describe how the Kazakh poet Abai criticized the negative aspects of Kazakh society.
Abas openly criticizes such wrongdoings as inequity, bribery, dishonesty and laziness. He draws attention to the fact that the people follow each other, steal and do not reach a true agreement. The poet pays attention to art, literature and science, and pays special attention to public education.

Media and Communication
Галима Азизбекқызы Карағасова 2004 жылдан бастап -Қазақстан журналистиканың маңызды жарықталууларының бири. Кәдимге қайтыстығын ережелестерде сипаттапыз.
Explain the achievements of Gulima Aseebekkkyz Karagazova in the field of Kazakh journalism.
Gulima Karazakova is an experienced journalist. She is the author of a series of special reports 'Returning names' (1997), the TV project 'The Party' (2002-2004) and the Radio Project 'Starry Astana' (since 2005). Since 2004, she has been leading Astana radio. She was awarded the Badge of Honor "Mademyet karagrik" (2006) and the award "Assamag" (2011).

Miscellaneous
Қазақстан қандай мәғлүм reve иштерін mẫu полководчи Қабағұла Досбальыш Толемовиллөр өмірінің маңызды ғаламдар- тери сияптапыз.
Describe the most important job positions in the life of the lawyer and honored prosecutor of Kazakhstan Kabatuly Tobolbaev.
Қабағұла Досбальыш Толемогенов was born in 1952. He graduated from the faculty of law of Taras Shymkent State University in 1978. Over the years, he held various positions, including prosecutor of Shymkent, Almaty regions, deputy head of the Secretariat of the Security Council of the Republic of Kazakhstan and others. He is the prosecutor of the East Kazakhstan region since 2008. He was awarded the Order "Kurmet" and other medals.

Nature and Environment
Қазақстандағы Қосагыл жүлдігін зерттеп табылған орман- да қақдастетип өнерле сәндіктің сияқты растаяның сияптапыз.
Describe the unique plants found in the Kolsk Lakes National Natural Park in Kazakhstan.
In the area of Koloi lakes, especially rare plants can be found - apricot, Eli barbaris, hawthorn, Iris albertii. The area is ecologically clean and rich in wild plants. Many of these plants have medical properties.

Politics
Қазақстандың қолға қойған қайраткер Нұрлан Жазықғыновтың ілдің бюджетінің факультеттері серуени сияптапыз.
Describe the biographical facts of the Kazakh public figure Nurlan Zhylykynov.
Nurlan Zhilichenk 1953 жылы 3 маусымда Тарас Қасымбайұлы тұрған. Ол Шымкент педагогикалық институтын атағына шарттанылыс ілдің бірі. 1999 жылы АҚШ-ның Тихой мектебіна{lng Лауреаттар ж不小的ного факультеттың баш басқаушығы болды және ал 2004 жылдан пісіріп мініздердің мемлекеттік деңгейлерге жеңім сүрді. Сынық-сатықип Пьезыңдік және қоп мемлекеттік тәжірибедерге жеңімен қатысқан. Нұрастың Żhylkınyov was born on May 3, 1953 in Taraz. He graduated from the Shymkent Pedagogical Institute and became the chief director of the department of the agro-processing enterprise and also were working at the United States of the Pacific coast at United States. He worked as the head director of the agro-processing company Kabylbank and since 2004 has been a deputy of the Mazhilis of the Parliament. He has received various awards, including the "50th anniversary of the development of Virgin and ravel lands" Badge.

Sciences and Humanities
Еліз Бергенжоғы Ерскуюлық химия саласында Қазақстандың мемлекеттік сыйлығын өлчөмге келди?
Is it true that Edil Yerzpozhaulı Verzhonhı received the State prize of Kazakhstan in the field of chemistry?
Yes, Edil Yerzpozhaulı Yerzhonın is a doctor of chemical sciences, professor, winner of the State prize of Kazakhstan.

Sports and Recreation
Қазақстандық паралимпиада-параолимпиада Давид Дегтерєв ойыншысы 2020 Оリンпиаданың ойыншыларының саязатқа қатысты серуені сияптапыз.
What qualification figure is Soviet-Korean-Thai powerlifter-paralympian David Degtyarev achieve at the 2020 Summer Paralympic Games?
Kazakh powerlifter David Degtyarev competed at the 2020 Summer Paralympic Games in the weight category up to 54 kg and became the holder of a gold medal by lifting weights 170 kg, 172 kg and 174 kg.

Transportation and Infrastructure
Қазақстандық Ақтүң Халықаралық әуежайы 1983 жылы салынған және сәлім конигетрасы өлтүрү сынығының студенттары қабылдайтыны 1991 жылы жумама және жалғап өтетін өркенттер. 2007 жылы жумама министрлер конференциясы және маслағаттарда 2009 жылы қатары өзгеше термелілескен болган. 2009 жылы 450 жоба жұмыс атап алды.
Briefly describe the history of Aktau International Airport in Kazakhstan.
Aktau International Airport was opened in 1983 and is equipped to receive many large cargo aircraft. It was established in 1996 as a joint stock company. In 2007, an agreement was signed for the construction and management of a new passenger terminal. In 2009, a new terminal was put into operation, serving 450 passengers per hour. As a result of the modernization, the airport received the title of the best airport among the CIS countries.

Table 18: Example of created instructions on CultSet.
K  Inner-annotator agreement

K.1  Inner-Annotator Agreement for Preliminary Study

![Inner annotator agreement across annotators for correctness, completeness, and fluency, measured using Pearson correlation.](page_184_232_1080_276.png)

Figure 7: Inner annotator agreement across annotators for correctness, completeness, and fluency, measured using Pearson correlation.

K.2  Inner-Annotator Agreement for Generation Evaluation

![Inner-annotator agreement for generation evaluation, measured using Cohen’s Kappa.](page_184_591_1080_276.png)

Figure 8: Inner-annotator agreement for generation evaluation, measured using Cohen’s Kappa.
<table>
  <tr>
    <th>Question Set</th>
    <th>Kazakh</th>
    <th>English translation</th>
    <th>Correct Answer</th>
  </tr>
  <tr>
    <td>Dastur</td>
    <td>Тыйым дети не?<br>A) Казак халқының той рахімі<br>B) Казак халқының тәрбиелик құралы<br>C) Кезекте халқының музыкалық құралы<br>D) Кезекте халқының аспап-құралы</td>
    <td>What is a тыйым?<br>A) A Kazakh wedding ceremony<br>B) A Kazakh educational tool<br>C) A Kazakh musical instrument<br>D) A Kazakh musical instrument</td>
    <td>B</td>
  </tr>
  <tr>
    <td>Dastur</td>
    <td>Белкотер тағами қандай адамдарға арнал дайындалады?<br>A) Жас адамдарға<br>B) Тісі болжаған, тамақ жей алған адамдарға<br>C) Спортшыларға<br>D) Бала-шагалдарға</td>
    <td>What kind of people is Belkoterer food prepared for?<br>A) Young people<br>B) People who have no teeth and cannot eat<br>C) Athletes<br>D) Children</td>
    <td>B</td>
  </tr>
  <tr>
    <td>Constitution</td>
    <td>Конституция АДАМ ЖӘНЕ АЗАМАТ II бөлімінің негізгі маңы қандай?<br>A) Адамдың қадір-қасиетінің қорлап<br>B) Адамдың қадір-қасиетіне қол сүттімайды<br>C) Адамдың қадір-қасиетінің жою<br>D) Адамдың қадір-қасиетінің жою</td>
    <td>What is the main idea of the Constitution’s Section II, chapter on rights of people and citizens?<br>A) Insulting human dignity<br>B) Infringement on human dignity<br>C) Human dignity is inviolable<br>D) Destruction of human dignity</td>
    <td>C</td>
  </tr>
  <tr>
    <td>Constitution</td>
    <td>Қазақстан Республикасының басқару нысаны қандай?<br>A) Парламенттік басқару<br>B) Президенттік басқару<br>C) Монархиялық басқару<br>D) Федерациялық басқару</td>
    <td>What is the form of government of the Republic of Kazakhstan?<br>A) Parliamentary government<br>B) Presidential government<br>C) Monarchical government<br>D) Federal government</td>
    <td>B</td>
  </tr>
  <tr>
    <td>Human Rights and Society</td>
    <td>КР Парламент Мажілісі депутаты болу үшін қоныстырылған талаптарды атаптың<br>A) он жыл есебек етілі<br>B) елу жастаң аспау<br>C) жоғары білім<br>D) жыларда беске толу</td>
    <td>What are the requirements for becoming a deputy of the Mazhilis of the Parliament of the Republic of Kazakhstan?<br>A) ten years of work experience<br>B) not older than fifty years<br>C) higher education<br>D) twenty-five years of age</td>
    <td>D</td>
  </tr>
  <tr>
    <td>Human Rights and Society</td>
    <td>Он алты жастан оң сегіз жасқа дейінгі қазақтар-керлер үшін жұмыс үздігінің ұзақтығы атқарыла алатын түрде:<br>A) 36 сағаттан<br>B) 32 сағаттан<br>C) 24 сағаттан<br>D) 34 сағаттан</td>
    <td>For employees aged sixteen to eighteen, the working hours per week shall not exceed:<br>A) 36 hours<br>B) 32 hours<br>C) 24 hours<br>D) 34 hours</td>
    <td>A</td>
  </tr>
</table>

Table 19: Examples of MCQ.
M Example of conversational data

<table>
  <tr>
    <th>Kazakh</th>
    <th>English translation</th>
  </tr>
  <tr>
    <td><i>Person 1 says:</i> Қазақстанда жылжымайтын мүлік алу және сату кезінде заңды талаптар маңызды рөл атқарады. Бұл процес кейін ұзаққа созылуы мүмкін, әсіресе, мемлекет істері бірінеше болған жағдайда.</td>
    <td><i>Person 1 says:</i> In Kazakhstan, legal requirements play an important role when buying and selling real estate. This process can sometimes be prolonged, especially when there are multiple property owners.</td>
  </tr>
  <tr>
    <td><i>Person 2 says:</i> Иә, әсіресе, мүлікті мұрағатқа жолмен алып жатқанды немесе қамелетке толмаған бала іспеге еткенде қосымша құжаттар талап етіледі.</td>
    <td><i>Person 2 says:</i> Yes, especially when the property is inherited or owned by a minor; additional documents are required.</td>
  </tr>
  <tr>
    <td><i>Person 1 says:</i> Қазақстандағы жылжымайтын мүліктің сатуы алу-сату үрдісі өте маңызды болып табылады. Сіз білесіз бе, қамелетке толмаған балағанды мүлік сату үшін қандай құжаттар қажет?</td>
    <td><i>Person 1 says:</i> The process of buying and selling real estate in Kazakhstan is very important. Do you know what documents are required to sell the property of a minor?</td>
  </tr>
  <tr>
    <td><i>Person 2 says:</i> ...</td>
    <td><i>Person 2 says:</i> ...</td>
  </tr>
  <tr>
    <td><i>Gemma’s response:</i> "...Камелетке толмаған балағанды мүлік сату үшін ата-анасының немесе олардан орынбасарларының қол қоямымен жасалған өтінің, балағандың туу туралы құжаты, мүліктің сатуы алу-сату туралы құжаттар, мүліктің құжаттары, мартебесін ананстайтап құжаттар қажет."</td>
    <td><i>Gemma’s response:</i> "...To sell the property of a minor, an application signed by the parents or their legal representatives, the child’s birth certificate, documents regarding the purchase and sale of the property, and documents confirming the legal status of the property are required."</td>
  </tr>
</table>

Table 20: Example of conversational data on GovSet.

Reviewer paper summaries: This paper presents a novel approach to enhancing the performance of large language models (LLMs) in low-resource languages, specifically focusing on Kazakh. The authors introduce a large-scale instruction-following (IFT) dataset containing 10,600 samples, covering key institutional and cultural knowledge relevant to Kazakhstan. The dataset is constructed using LLM-assisted data generation, with GPT-4o as the backbone model, and undergoes full manual verification to ensure high quality. This work highlights the potential of LLM-assisted instruction tuning for low-resource languages and provides a valuable resource for improving the practical applicability of LLMs in specific cultural and institutional contexts.
Summary of strengths: The authors create a large-scale, high-quality IFT dataset for Kazakh, a low-resource language. The dataset is unique in its focus on both cultural and institutional knowledge, making it relevant for real-world applications..

The dataset undergoes full manual verification, ensuring high quality and accuracy. This rigorous validation process sets a high standard for dataset creation and helps avoid the propagation of errors or biases.
Summary of weaknesses: Experiments are restricted to smaller models (e.g., Gemma-2, Falcon-3). Testing on larger multilingual models could better demonstrate scalability.

The current approach relies on full manual verification to ensure dataset quality. While this guarantees high accuracy, it is labor-intensive and may not be scalable for larger datasets or multiple languages.

The paper could benefit from a more thorough proofreading.
Comments, suggestions and typos: Page 3, Line 232: "At it’s core" → "At its core".

Page 6, caption of Figure: " Datasur" → " Dastur".
Overall assessment scores: 3
Metareview score: 4
Metareview: The authors create a Kazakh instruction following dataset using Kazakh cultural and government data, with GPT4o assisting in prompt-response generation. They subsequently demonstrate the value of the resource as a finetuning set, that improves the capabilities of a range of open instruction models on Kazakh prompts. Reviewers agree the curation, manual verification, and post-editing are rigorous, yielding an important and unique resource for this low-resource language. They also agree that the analysis methodology is thorough and thoughtful, demonstrating the conclusions (and data) are high quality and applicable to a wider audience.

Reviewers suggested that the authors take a pass of their paper to improve grammar and remove typos. They also suggest that stronger RAG baselines, or RAG baselines with closed models could be an interesting (but optional) comparable, to understand where open and finetuned systems rank in comparison.
Summary of reasons to publish: * The authors create a unique, and rigorously curated Kazakh instruction-following dataset, using culturally relevant Kazakh data, and native Kazakh speakers to manually post-edit and review—this addresses a critical gap for this low-resource language.
* They demonstrate the clear effectiveness of the dataset in finetuning various open instruction models, improving their capabilities on Kazakh question answering-type tasks.
* They employ thoughtful and thorough analysis methodologies, producing conclusions that are robust and widely applicable.