# Fine-Tuning Large Language Models for Kazakh Text Simplification
## 4 Experiments
### Results
Although there is a drop in SARI when moving to the manual test set, KazSim maintains a clear advantage, showing that task-specific supervision enables better generalization. Tables 7 and 8 summarize the confidence and variability of model performance on both automatically generated and semi-manually annotated test sets. Each table reports 95% confidence intervals and standard deviations for BLEU and SARI scores, allowing a comparison of model stability and reliability. Fine-tuned models, particularly KazSim (Llama-3.3-70B), not only achieve higher average scores, but also exhibit narrower confidence intervals and lower standard deviations, indicating more consistent performance. \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Model** & \begin{tabular}{c} **BLEU CI** \\ **(Low)** \\ \end{tabular} & \begin{tabular}{c} **BLEU CI** \\ **(High)** \\ \end{tabular} & \begin{tabular}{c} **BLEU Std** \\ **(High)** \\ \end{tabular} & \begin{tabular}{c} **SARI CI** \\ **(Low)** \\ \end{tabular} & \
\begin{tabular}{c} **SARI CI** \\ **(High)** \\ \end{tabular} &
\begin{tabular}{c} **SARI Std** \\ **(High)** \\ \end{tabular} \\ \hline kazLLM-Llama-3.1-8B & 19.09 & 22.36 & 0.83\% & 44.87 & 46.77 & 0.49\% \\ kazLLM-Llama-3.1-70B & 20.06 & 23.08 & 0.78\% & 43.05 & 44.49 & 0.36\% \\ Sherkala-Llama-3.1-8B & 17.73 & 21.48 & 0.91\% & 43.72 & 45.56 & 0.47\% \\ \hline Llama-3.2-3B (zero-shot) & 0.15 & 0.43 & 0.07\% & 32.94 & 34.87 & 0.50\% \\ Qwen2-72B (zero-shot) & 1.07 & 1.56 & 0.12\% & 34.64 & 36.54 & 0.51\% \\ Llama-3.3-70B (zero-shot) & 4.93 & 6.27 & 0.34\% & 39.22 & 40.89 & 0.43\% \\ \hline KazSim (Llama-3.2-3B) & 23.45 & 27.29 & 1.00\% & 47.98 & 49.85 & 0.48\% \\ KazSim (Qwen2-72B) & 25.54 & 29.92 & 1.15\% & 51.70 & 53.87 & 0.55\% \\ KazSim (Llama-3.3-70B) & 30.94 & 35.69 & 1.21\% & 55.23 & 57.47 & 0.58\% \\ \hline \hline \end{tabular}
\end{table} 
Table 7: The 95% confidence intervals (CI) and standard deviations (Std) for BLEU and SARI scores across different models. Figure 4: Results of Sari score of various models for two different datasets. To assess prompt sensitivity, we compare model outputs when using either English or Kazakh instruction prompts for the same simplification task. Results are summarized in Table 9. Overall, model performance remains relatively stable across prompt languages, though minor variations are observed. For domain-specific models (e.g., kazLLM and Sherkala), Kazakh prompts yield slightly higher SARI scores, indicating better alignment with simplification objectives when instructions are provided in the target language. Sherkala, for instance, improves from 44.23 to 46.15 in SARI with a Kazakh prompt. Zero-shot performance is more sensitive to prompt language. Llama-3.3-70B sees a notable increase in BLEU (from 4.53 to 8.26) and F1 (from 72.08 to 76.42) under Kazakh instructions, suggesting that instruction-following behavior improves when prompts are better aligned with the target output language. The proposed KazSim model remains robust under both conditions, achieving the best results across all metrics. Performance is slightly higher with Kazakh prompts, reaching a SARI of 48.78 and an F1 of 84.39, confirming the benefit of aligning the prompt language with the generation task. Figure 5 demonstrates that prompt language has a significant impact on the quality of text simplification. Kazakh prompts consistently yield outputs that are more faithful to the original in terms of meaning, tone, and cultural context. For example, subtle emotional cues and complex sentence structures are better preserved when the model is guided by a Kazakh prompt (e.g., Sentences 1, 3, 5), whereas English prompts often lead to oversimplification or semantic drift. For example, in Sentence 1, the English-prompt version introduces the notion of a \"beautiful image\" (korikti suretpen) that does not exist in the original, altering the emotional nuance.