Til-GEC-Gemma4-exp087 — Gemma 4 fine-tuned for Kazakh grammatical error correction

Four rungs of the Gemma 4 family fine-tuned on Kazakh GEC, measured against seven baselines on a decontaminated public benchmark. One repository, one branch per rung.

Status: run in progress. ft_E4B and ft_26B are complete; ft_31B is training and ft_12B is queued behind it. The tables below update as rungs land.

---

The benchmark, and why decontamination is the whole story

Everything is scored on stukenov/sozkz-corpus-gec-benchmark-kk-v1 — splits test (200), organic_social (486), synthetic (401).

That benchmark is inside the training corpus. Measured, not assumed: 2 139 rows of TilQazyna/Til-GEC reproduce a benchmark input verbatim, against 1 432 unique inputs in the benchmark itself. Training on the corpus as shipped produces a model that has seen the answers.

So data_build.py builds the benchmark's every input and target into a forbidden set — exact and normalised (NFKC → lowercase → punctuation stripped) — and drops any training row touching it. Cost: 1 528 rows out of 1.68 M, under 0.1 %. The finished train.jsonl is then re-checked independently; the script exits non-zero if a single benchmark row survived. It reports 0.

Normalised matching found nothing beyond exact matching. The contamination is verbatim — no paraphrase leakage — so a plain string filter is sufficient here, and no embedding search is needed.

---

Training data

data_build.py --recipe selects one of three sets. Manifests for the first two are in data/.

recipe	sources	train rows
tilgec	TilQazyna/Til-GEC:clean + stukenov/sozkz-corpus-synthetic-kk-gec-v1	818 306
qazgramma	human-annotated pairs from TilQazyna/datalake	66 161
both	the union	886 467

Where the human-annotated set came from

The previous GEC generation trained on a file called human78k.jsonl, built from Qazgramma human-annotated pairs. That file died with its node. The sources survive in TilQazyna/datalake:

file	rows
cleaned/instruction/2026-04-01/TQ_qazgramma_20_categories_merged.jsonl	62 114
cleaned/instruction/2025-09-12/qazgramma.jsonl	32 312

94 426 together, 20 759 of them duplicates — hence the historical name. A third file, TQ_kz_grammar_deduplicated.jsonl, is grammar-term definitions rather than GEC and is not used.

19 % of what remains is also not GEC — the merged file mixes in morphology drills and explain-the-rule tasks, where the target carries a lecture rather than a correction:

IN : көрпе + табыс септік        IN : тыңдау
TGT: көрпені                     TGT: тыңдаймын - Осы шақтың бірінші жағының жекеше түрі...

The minimal-edit filter removes them. Training a corrector on those would teach it to explain grammar instead of fixing text. 56 274 genuine pairs survive.

The tilgec recipe in detail

Source: TilQazyna/Til-GEC config clean, plus stukenov/sozkz-corpus-synthetic-kk-gec-v1.

step	rows
read	1 703 896
dropped — gec-mix source	664 555
dropped — low judge score (< 4)	259 683
dropped — duplicates	13 854
dropped — benchmark leak	1 528
dropped — not a minimal edit	1 278
kept	762 998
+ identity oversampled to 20 %	+57 308
train / val	818 306 / 2 000

gec-mix is not GEC — it is a translation corpus

The single largest data decision, and it is not a judgement call. gec-mix carries task="gec" but 99.8 % of its rows fail a minimal-edit test, exactly 50.0 % have a Russian input and a Kazakh target, and every row has error_tags = None while genuine GEC rows carry a taxonomy:

IN : Он стал генерал-полковником швейцарского полка, губернатором Лангедока
TGT: Ол Швейцария полкінің генерал-полковнигі, Лангедок губернаторы болды

664 555 rows — 39 % of the corpus. Training on it teaches a model to translate Russian instead of correcting Kazakh. It is excluded by source, not by heuristic: the heuristic catches 99.8 %, and the surviving 0.2 % are the most harmful ones, the pairs that superficially look like edits.

Kept sources reject cleanly by comparison: gec-rulebased 0.0 %, gec-bench 1.1 %, gec-synth 4.3 %.

Minimality, and word order

A correction is kept when it is a minimal edit, measured as edit distance over the differing middles after stripping the shared prefix and suffix, bounded at 50 % of the target length. Three cheap bounds keep the DP off the hot path (shared affixes, length gap, character bag distance); equivalence to the naive definition was checked on 6 000 random pairs with zero disagreements.

Word-order corrections are explicitly admitted. Сөз тәртібі is a real Kazakh GEC category, but a reordering displaces nearly every character and a character-level test throws it away. Rows whose word multisets differ by at most max(2, 20 %) count as edits:

IN : Бағдадта 4 адам қаза тапты жарылыстан
TGT: Бағдадта жарылыстан 4 адам қаза тапты

Identity pairs

Oversampled to 20 % of the training set. This is what keep_correct in the results measures: whether the model leaves already-correct text alone. It works — see the table.

---

Results

test is the headline split. keep% = share of already-correct inputs left untouched.

#	model	mode	test EM	organic EM	synth EM	test chrF	keep%
1	ft_E4B (this repo, branch ft-e4b)	zero-shot	56.0	7.6	35.4	97.1	100.0
2	TilQazyna/Til-kk-0.5B-256k-gec-exp085	zero-shot	55.0	7.2	48.6	95.3	76.9
3	ft_26B (branch ft-26b)	zero-shot	51.0	8.8	41.1	97.1	96.2
4	google/gemma-4-31B-it	3-shot	40.5	24.7	41.6	94.6	53.8
5	google/gemma-4-26B-A4B-it	3-shot	32.5	22.4	33.9	93.5	57.7
6	google/gemma-4-E4B-it	3-shot	32.0	22.0	30.9	93.7	46.2
7	TilQazyna/Til-Core-1B-GEC	zero-shot	28.0	8.2	29.7	89.0	80.8
8	google/gemma-4-12B-it	3-shot	21.0	10.3	16.0	88.8	80.8
9	TilQazyna/Til-2B-GEC	zero-shot	6.0	2.1	4.0	71.0	15.4

Per-split detail, including edit precision / recall / F0.5, is in results/*.json.

The data ablation — same model, same recipe, only the training set differs

The most informative result here is not the leaderboard, it is this A/B on gemma-4-E4B-it. Two training sets were compared, plus their union: the rule-based synthetic bulk of Til-GEC (818 306 pairs) and the human-annotated Qazgramma pairs the previous GEC generation was actually trained on (66 161 pairs, recovered from TilQazyna/datalake — see below).

	base, untuned	Til-GEC 818k	Qazgramma 66k
test exact match	32.0	56.0	38.5
test keep correct	46.2	100.0	57.7
organic_social exact match	22.0	7.6	13.8
organic_social edit recall	0.429	0.054	0.205
synthetic exact match	30.9	35.4	28.7

Three things fall out of this table.

Human annotation is four times better at real text. Edit recall on organic social media is 0.205 against 0.054 — the Qazgramma-trained model actually corrects messy text, the synthetic-trained one stays silent.

But it loses on the benchmark: 38.5 against 56.0 on test. The likely reason is that test and synthetic are themselves synthetic in origin, so training on Til-GEC matches their distribution. That is distribution overlap, not quality — which puts the benchmark itself in question as a criterion, not just the data.

Neither beats the untuned base on organic text (22.0). Every fine-tune we ran makes the model more cautious exactly where boldness is needed. Qazgramma halves the damage; it does not remove it.

The union of both sets (886 467 pairs) is training now as the third arm.

What the numbers say

Fine-tuning works, and small wins. gemma-4-E4B-it goes from 32.0 to 56.0 on test — 1.75×, enough to pass the previous best Kazakh GEC specialist. Edit precision rises to 0.868 against 0.548 for that specialist: when it corrects, it is almost always right.

Size does not help. ft_26B — three times the parameters — scores 51.0 against 56.0. It is steadier across splits (41.1 on synthetic vs 35.4) but loses the headline metric.

Identity oversampling does what it was for. ft_E4B is the first model here to reach keep_correct = 100 on test; untuned Gemma sits at 46–58 %.

The honest negative result. Both fine-tuned models get worse than their own base on organic social-media text: 7.6 and 8.8 against 22.0 and 22.4. The cause is visible in edit recall on that split — 0.054 and 0.051. After fine-tuning the models barely edit organic text at all. They learned caution from a clean synthetic distribution and carried it into the one place that needs active correction.

That is a property of the data, not of the run, and it reproduces the same hole the previous specialist has (7.2 on organic). Both were trained on the same synthetic distribution. Closing it needs organic training data, not a bigger model.

---

Branches

branch	contents
main	this card, all results/, code/, the data manifest
ft-e4b	gemma-4-E4B-it fine-tune — weights + metrics
ft-26b	gemma-4-26B-A4B-it LoRA fine-tune — metrics (weights not retained)
ft-31b	gemma-4-31B-it LoRA fine-tune — pending
ft-12b	gemma-4-12B-it fine-tune — pending

Weights are kept only for the leading rung: the ladder ran on a 394 GB volume where the base checkpoints alone total ~148 GB and every fine-tune doubles its own. Metrics are retained for all.

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("TilQazyna/Til-GEC-Gemma4-exp087", revision="ft-e4b")
tok   = AutoTokenizer.from_pretrained("TilQazyna/Til-GEC-Gemma4-exp087", revision="ft-e4b")

---

Usage

The evaluation prompt is byte-identical to the training prompt — a fine-tuned rung is scored with exactly what it was trained on.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "TilQazyna/Til-GEC-Gemma4-exp087"
tok = AutoTokenizer.from_pretrained(repo, revision="ft-e4b")
model = AutoModelForCausalLM.from_pretrained(repo, revision="ft-e4b",
                                             dtype=torch.bfloat16, device_map="auto")

PROMPT = ("Сен қазақ тілінің корректорысың. Сөйлемдегі грамматикалық және орфографиялық "
          "қатені түзет. Тек түзетілген сөйлемді жаз.")

def correct(sentence: str) -> str:
    msgs = [{"role": "user", "content": f"{PROMPT}\nСөйлем: {sentence}"}]
    text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
    ids = tok(text, return_tensors="pt", add_special_tokens=False).input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=110, do_sample=False,
                         pad_token_id=tok.pad_token_id or tok.eos_token_id)
    return tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True).split("\n")[0].strip()

print(correct("Бала үйда отыр."))   # -> Бала үйде отыр.

---

Training setup

Hardware	2 × NVIDIA H200 (140 GiB)
Precision	bfloat16, gradient checkpointing
Schedule	1 500 steps, effective batch 128 (≈192 k examples), cosine, warmup 3 %
ft_E4B	full fine-tune, adafactor, lr 1e-5, per-device batch 32
ft_26B, ft_31B	LoRA r=32 α=64 on text projections only, adamw, lr 1e-4
Seed	42 everywhere; greedy decoding for every measurement

Two Gemma 4 traps worth knowing

LoRA must target language_model and nothing else. Gemma 4 is multimodal, and its vision_tower / audio_tower wrap projections as ...q_proj.linear while the language model stays flat at ...q_proj. Sniffing for a .linear suffix anywhere finds the vision tower, matches only the image and audio encoders, and trains happily — producing a byte-identical copy of the base model after hours on a GPU. train_gemma_gec.py enumerates targets explicitly and asserts that every decoder layer from the config is covered.

Checkpoints need backfilling before vLLM will load them. Gemma 4 shares KV across layers, and transformers implements that by not building k_proj / v_proj / k_norm modules for the sharing layers — so save_pretrained cannot write them (54 tensors, layers 24–41 on E4B). vLLM's implementation does build them and refuses the checkpoint:

ValueError: Following weights were not initialized from checkpoint:
{'language_model.model.layers.24.self_attn.k_norm.weight', ...}

Those tensors are never trained, so backfill_from_base() copies them straight from the base checkpoint. Lossless, and mandatory.

Throughput

Measured before committing to the run (results/throughput_sweep.txt, E4B on one H200, effective batch fixed at 128):

config	s/step	peak GiB
per-device batch 4, eager attention	17.89	70.8
per-device batch 4, sdpa	17.31	70.8
per-device batch 16, sdpa	4.86	70.8
per-device batch 32, sdpa	3.21	70.8

5.4× on identical memory. Attention implementation was worth 3 %; the bottleneck was micro-steps per optimizer step (128 / per-device batch), with the card idle on each.

---

Reproducing

python code/data_build.py --out data      # ~2 min, writes train/val + MANIFEST.json
python code/test_metrics.py               # metric self-check; chrF matches sacrebleu to 0.00

BASE=google/gemma-4-E4B-it DATA=data/train.jsonl OUT=out MODE=full \
  MAX_STEPS=1500 PDBS=32 LR=1e-5 python code/train_gemma_gec.py

MODEL=out TAG=ft_E4B FMT=chat ZEROSHOT=1 python code/eval_gec_bench.py

eval_gec_bench.py speaks three prompt formats (chat, sep, tag) so each baseline is scored in its native one, and carries an ENGINE=hf path for models vLLM cannot serve — our own Til models use DeepseekV3 MLA with head dimensions no vLLM MLA backend accepts.

Limitations

• Sentence-level Kazakh GEC only. Not a chat model.
• Weak on organic social-media text — see the negative result above. For that distribution an untuned gemma-4-31B-it is currently the better choice.
• 1 500 steps ≈ 0.23 of an epoch. Longer training was not explored.
• LoRA on 26B-A4B adapts attention and shared layers but not the expert FFNs — they are not nn.Linear modules and PEFT cannot wrap them.

License

Derivatives of Gemma 4 are governed by the Gemma Terms of Use, not by this organisation's usual MIT default. Access is gated for usage tracking.