kathe-r11-restorer — Kashmiri short-vowel restoration

Restores kasra (U+0650), damma (U+064F) and fatha (U+064E) to Perso-Arabic Kashmiri text. A companion to a translation model, not a translation model.

Code and full write-up: https://github.com/AzhadArshad/kathe_2026

The shipped checkpoint is r11b_dense.pt. The other files in this repo are earlier or differently-trained arms of the same sweep, kept for reproducibility.

Why this exists

IndicTrans2 cannot emit these three marks. Each appears in exactly one token of its 122,672-entry target vocabulary — the bare standalone mark — while every other Kashmiri diacritic is baked into whole-word subwords. Writing چھُس would require splitting a word to insert a bare diacritic that occurs in no natural subword context, and beam search never does. Measured across 1,730 sentences the translation model produced exactly zero of all three, while reproducing the subword-embedded marks at above-reference rates.

The vocabulary is frozen in the pretrained checkpoint, so no amount of fine-tuning fixes this and restoration has to be post-hoc.

It is worth a great deal. On the KATHE 2026 test set:

System	Score
translation model alone	10.00
+ diacritic lexicon	13.52
+ this restorer instead	13.99
+ both, as a union (shipped)	15.05

That is +5.05 from restoration, against +1.17 for corpus selection and +0.83 for fine-tuning. The largest lever in the project, and it needs no GPU.

What it is

A 3.3M-parameter bidirectional transformer encoder used as a per-character tagger: for each input character it predicts one of {none, kasra, damma, fatha}. The base-letter sequence is never an output, so restoration is insertion-only by construction — it cannot alter, reorder or drop a character for any label vector.

How to use it — as a union with a lexicon, not alone

The shipped configuration consults a 48k-form lexicon first and uses the tagger only on words the lexicon has never seen (restore_merge: known). The two fail in opposite places: the lexicon is near-unbeatable on words it has seen and silent on the rest, while the tagger never abstains but can be talked out of a common word by context.

git clone https://github.com/AzhadArshad/kathe_2026 && cd kathe_2026
python scripts/generate_translations.py --input input.csv --output output.csv

The lexicon ships in that repository; the weights are fetched from here automatically.

Tuning knob

restore_none_bias is a logit offset on the "no mark" class. Under restore_merge: known it acts only on words the lexicon does not know, so it is precisely "how freely does the tagger mark the tail":

bias	marks /100c	score
−1.5	13.78	14.25
0.0	12.36	14.82
+1.6836 (shipped)	11.30	15.05
+2.25	11.03	15.04
+3.25	10.62	14.91
+∞ (lexicon only)	9.85	13.52

Concave with a broad flat top over [1.68, 2.25], so the shipped value is not balanced on a knife edge.

Two findings worth carrying elsewhere. First, chrF++ runs at beta=2, weighting recall four times precision — so emitting a mark the reference lacks is cheaper than omitting one it has, and tuning output density to match reference density is a handicap rather than a target. Second, that only holds where the model is confident: in the tail, extra marks are noise, which is what the table above measures.

Training data

Monolingual Kashmiri, self-supervised: strip the three marks from any sentence and (stripped, original) is a training pair.

source	examples
nawabhussain/Kashmiri-Language-Corpus	41,187
BPCC bpcc-seed-v1	15,467
BPCC daily	3,059

Held-out micro-F1 60.63 (precision 73.18, recall 51.76), best at epoch 20. The parent project's development sets were excluded by exact stripped-and- normalized string.

Limitations

• Only these three marks. Others already match reference density to within 5%; "restoring" them would add noise.
• Standalone it is worse than a lexicon lookup (13.81 vs 13.52 was close, and on the internal dev set the lexicon won). Its value is on the tail — words the lexicon has never seen — which is why the shipped system is a union and why the tail is deliberately suppressed.
• Precision 73% / recall 52% on held-out text: it under-marks rather than over-marks, and the bias knob exists to trade that off.
• Trained on Perso-Arabic Kashmiri only.

Licence and attribution

Apache-2.0 — trained only on BPCC (CC-BY-4.0 / CC0) and nawabhussain/Kashmiri-Language-Corpus (Apache-2.0), both of which permit relicensing of derivatives with attribution. Other checkpoints from the same sweep saw CC-BY-NC-SA text and are not published here.

Attribution remains required for the corpora above regardless of this licence; see NOTICE in the parent repository.