You need to agree to share your Hugging Face information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
Access to this model is granted by RegaLabs on request. Tell us who you are and how you plan to use it.
Log in or Sign Up to review the conditions and access this model content.
ckb-corrector v2 (Central Kurdish / Sorani text corrector)
A non-autoregressive corrector for Central Kurdish (Sorani, ckb) text: human-typed text and ASR output.
One encoder forward pass per line, then find β propose β pick:
- Normalize (deterministic): legacy encodings, Ω/Ω, ZWNJ, punctuation spacing, spacing table.
- Find: per-word label
KEEP / REPLACE / DELETE / SPLIT / MERGE_NEXT, plus a punctuation-after label (none Ψ . Ψ ! Ψ :). - Propose: candidates for flagged words only (lexicon by edit distance and sound, real-ASR confusion table, mined misspellings, split/merge options).
- Pick: the same encoder scores candidates in context; a GBDT pick head combines MLM and channel features. A precision guard keeps the original word unless the best candidate wins by a tuned margin.
Design priority is precision: when unsure, the text is left unchanged.
Files
| File | What |
|---|---|
model.safetensors, config.json, tokenizer files |
Encoder: XLM-R architecture (119M), glot500-ckb-mlm (Glot500-base with Sorani MLM continued pretraining), fine-tuned |
heads.pt |
Find and punctuation heads (PyTorch pickle) |
pick.pkl |
GBDT pick head (Python pickle) |
confusion.tsv |
Real-ASR confusion table (training rows only) |
corrector_config.json |
Guard thresholds used at inference (chosen by the dev sweep) |
corrector_config.tune.json |
Config used during tuning (flag floor 0.1), kept for reference |
heads.pt and pick.pkl are pickles: load them only from this repo.
The inference code (correct.Corrector) and lexicon resources are not in this repo.
Training (summary)
- Encoder start:
glot500-ckb-mlm(Sorani MLM continued pretraining ofcis-lmu/glot500-base). - 662,172 training rows: real ASR error pairs (Γ2), synthetic human/ASR noise on a gold pool, clean identity lines (incl. rudaw / anfsorani), 28,353 legacy-script normalization pairs.
- 6,200 steps (1,602 on a Quadro M2200, 4,598 on a Kaggle T4 with fp16 AMP).
- Guard chosen by a 255-setting sweep on dev with the gate dev clean word false-change rate β€ 0.05%: Ο_find 0.5, margin 7, valid-word margin 8, punctuation bare 0.9 / punctuated 0.98.
Evaluation (held-out, measured)
All numbers come from the project's eval scripts on held-out sets. Latency was not measured on production hardware.
| Set | Metric | Input | v2 |
|---|---|---|---|
| test_clean (11,032 lines) | word false-change rate | β | 0.045% (5 lines: 3 from the normalizer, 2 correct fixes of a gold error; 0 wrong model word changes) |
| test_clean | whole-string change rate | β | 0.199% (17 of 22 punctuation only) |
| dev_mix (6,550) | WER, no punctuation | 16.85 | 15.21 (edit P 0.689 / R 0.087 / F0.5 0.289) |
| FLEURS-female test (real ASR) | WER | 37.66 | 37.44 (normalize only: 37.45) |
| zainab test (real ASR) | WER | 40.44 | 40.28 (normalize only: 40.28) |
| human700-tune (real ASR) | WER | 37.54 | 37.54 |
| FLEURS-female test | punctuation P / R / F1 | β | 0.955 / 0.386 / 0.550 |
On real ASR output v2 currently does about what normalization alone does; its value there is safety (it does not damage correct text) and punctuation.
Latency (laptop, not production)
Quadro M2200, fp32, full pipeline: batch 1 p50 44.8 ms / p99 368 ms; batch 32 2.11 s per batch (66 ms/line). CPU, 4 threads: batch 1 p50 98 ms / p99 1,254 ms. ONNX / TensorRT export not done yet.
Limitations
- Low recall on real ASR errors (see table).
- Text convention: verbatim colloquial verb forms are kept; numbers are expected as words (convert with an ITN afterwards).
- Not tested on production hardware.
Base model
Built on cis-lmu/glot500-base (Apache-2.0).
- Downloads last month
- -
Model tree for RegaLabs/ckb-corrector
Base model
cis-lmu/glot500-base