darmm-ocr-kazakh-vl-3b

OCR model for printed Kazakh text (Cyrillic script) — words, lines, and full document pages — built for real-world conditions: scans, photos, degraded print, shadows, perspective distortion, and compression artifacts. Full support for the Kazakh-specific letters Ә Ғ Қ Ң Ө Ұ Ү Һ І.

It is a LoRA fine-tune of Qwen/Qwen2.5-VL-3B-Instruct (merged into the base weights — this repo is ready to use as-is), trained on Darmm/darmm-ocr-kazakh-v2. It supersedes the earlier darmm-ocr-kazakh-cyrillic-model (0.3B TrOCR-style) — see the comparison below.

Unlike crop-based OCR models, it reads entire document pages directly — no external text detector or line segmentation needed.

Evaluation

CER / WER on the held-out test split of darmm-ocr-kazakh-v2 (never seen in training), greedy decoding. Computed with jiwer after NFC normalization; eval.py in this repo reproduces the numbers.

test tier	n	this model CER / WER	Qwen2.5-VL-3B zero-shot	v1 (0.3B TrOCR)
clean word crops	1000	1.5% / 9.2%	1156% / 2015%	—
clean line crops	1000	0.65% / 2.7%	316% / 470%	—
hard word crops	1019	0.68% / 4.0%	1072% / 1893%	—
hard line crops	981	0.54% / 3.3%	285% / 424%	—
full pages	500	0.85% / 4.8%	67% / 124%	not supported

Zero-shot base-model CER exceeds 100% because it hallucinates and repeats far beyond the reference text — fine-tuning is doing the heavy lifting on this task. The v1 column is being finalized (v1 reads word/line crops only; its repo ships no tokenizer, evaluated with the roberta-base vocabulary it was trained with).

• "hard" = aged/scanned-paper backgrounds, shadows, uneven lighting, perspective warp, blur, low-resolution round-trips, JPEG artifacts
• "page" = multi-line documents; the model reproduces line breaks (\n) as laid out on the page

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image

model_id = "Darmm/darmm-ocr-kazakh-vl-3b"
processor = AutoProcessor.from_pretrained(model_id, min_pixels=64*28*28, max_pixels=1024*28*28)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda:0")

image = Image.open("document.png").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "Read the Kazakh text in this image."},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to("cuda:0")
out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

Use the exact prompt above — it is the training instruction. For batched inference set processor.tokenizer.padding_side = "left".

The raw LoRA adapter (~120MB) is also available at Darmm/darmm-ocr-kazakh-vl-3b-lora for use on top of the base model.

Training

• Base: Qwen2.5-VL-3B-Instruct, bf16
• LoRA: r=32, α=64, dropout 0.05, on all attention + MLP projections of the language model (59.9M trainable params, 1.6%); vision tower frozen
• Data: 157k samples from darmm-ocr-kazakh-v2 train split — all 66k hard crops, all 11k pages, 80k of the 190k clean crops
• Schedule: 1 epoch, effective batch 32 (2 × 16 accumulation), lr 1e-4 cosine, 3% warmup, gradient checkpointing
• Hardware: single A100 80GB (shared), ~18 GPU-hours
• Final eval loss: 0.039

Limitations

• Printed text only — not trained on handwriting
• Training data is fully synthetic (rendered with 17 fonts on procedural paper textures); highly stylized fonts, complex multi-column layouts, or tables may degrade accuracy
• Tuned for Kazakh Cyrillic; it retains general abilities from the base model but other scripts/languages are not evaluated
• Reads what is written — it does not correct source-text spelling

Built by Darmm · trained & evaluated 2026-08 · Apache 2.0