swedish-print-ocr-3b-benchmark

Fine-tune of stanford-oval/churro-3B (itself a fine-tune of Qwen2.5-VL-3B-Instruct) for optical character recognition of Swedish printed books from 1600 to 1999, covering both blackletter (fraktur) and roman (antikva) typefaces.

This is the held-out benchmark model. 338 works are excluded from its training mixture so that error rate can be honestly measured against them. For a model trained on everything (production use, no valid CER), see Ericu950/swedish-print-ocr-3b.

Reference configuration (same recipe, three seeds) reaches 3.25% CER (best seed) / 3.59 ± 0.31% (mean over three seeds) on a 1,182-page, 214-work internal benchmark spanning 1600–1999, against 6.50% for the strongest baseline (Qwen3.6-27B, zero-shot). On an external benchmark built from Swedish Wikisource (542 proofread pages, different collection and transcription conventions) it reaches 4.54% against 6.61%. Figures use decoding-repair applied identically to every autoregressive system compared, including baselines. See the accompanying paper for exact benchmark construction, contamination checks, and this checkpoint's specific training run.

Data

Six sources at fixed proportions, held-out works excluded: NewsEye Swedish newspapers, Projekt Runeberg, GT4HistOCR (German), a Swedish fraktur corpus, Litteraturbanken, and Språkbanken regions — 34% roman Swedish, 44% Swedish blackletter, 22% German blackletter.

Training

Full fine-tune, frozen vision encoder, learning rate 8e-5, cosine decay, effective batch 128, bfloat16, ~1.76 epochs.

Usage

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Ericu950/swedish-print-ocr-3b-benchmark", torch_dtype="bfloat16", device_map="auto"
)
processor = AutoProcessor.from_pretrained("Ericu950/swedish-print-ocr-3b-benchmark")

License

Inherits the Qwen Research License Agreement from the base model chain (Qwen2.5-VL-3B → CHURRO-3B) — research use, see linked license. Not evaluated for commercial redistribution.

Limitations

None of the hyperparameters varied in ablations (learning rate, frozen/unfrozen vision encoder, input resolution, century-in-prompt, German fraktur share) moved this result measurably; training-set size was the only factor that moved it monotonically without plateauing. Reference transcription error rate is not established — see the paper's Limitations section.