Flo976/malagasy-kenlm-4gram-v2

Malagasy KenLM 4-gram, v2 — 134 M words, for CTC speech recognition

A word-level 4-gram language model of Malagasy, trained on 134 million words, for beam-search decoding of Flo976/w2v-bert-malagasy with pyctcdecode. It succeeds Flo976/malagasy-kenlm-4gram (17 M words) and removes about one error in twenty-five that the previous one let through.

Résumé en français. Modèle de langue malgache (4-gramme KenLM, 134 millions de mots) pour le décodage du modèle de reconnaissance vocale. Avec lui, la chaîne fait 12,7 % d'erreurs de mots sur les énoncés courts et 10,1 % sur les extraits longs, contre 13,2 % et 10,4 % avec le modèle de langue précédent. Licence mixte : voir plus bas.

Results

Test split of badrex/malagasy-speech-full, speakers never seen in training. Same acoustic log-probabilities for every row, beam 100, apostrophe re-gluing after decoding, WER after the common normalization (lowercase, punctuation removed). Settings tuned on validation only; the test was measured once.

Language model	Short utterances 2–8 s (11,625)	Full clips (3,101)
none (greedy)	14.33 %	12.01 %
previous 4-gram, 17 M words	13.20 %	10.41 %
previous 4-gram, re-tuned on validation	12.92 %	10.46 %
this model	12.68 %	10.13 %

Paired bootstrap (2,000 resamples), this model against the previous one with its published settings: −0.52 point [−0.65, −0.40] on short utterances and −0.29 [−0.39, −0.18] on full clips. Against the previous one re-tuned, which isolates the effect of the extra text: −0.24 [−0.35, −0.12] and −0.33 [−0.44, −0.23]. All remain significant with a stricter bootstrap that resamples whole speakers.

Usage

import json, re
import numpy as np
from huggingface_hub import snapshot_download
from pyctcdecode import build_ctcdecoder

lm = snapshot_download("Flo976/malagasy-kenlm-4gram-v2")
params = json.load(open(f"{lm}/params.json"))
unigrams = [w.strip() for w in open(f"{lm}/unigrams.txt", encoding="utf-8") if w.strip()]

tok = processor.tokenizer  # from Flo976/w2v-bert-malagasy
vocab = tok.get_vocab()
labels = ["" if t == tok.pad_token else " " if t == tok.word_delimiter_token else "⁇" if t == tok.unk_token else t
          for t in sorted(vocab, key=vocab.get)]

def decoder_for(seconds):
    p = params["short" if seconds <= 10 else "long"]
    return build_ctcdecoder(labels, kenlm_model_path=f"{lm}/{params['lm']}", unigrams=unigrams,
                            alpha=p["alpha"], beta=p["beta"])

# log-softmax, not raw logits: pyctcdecode expects log-probabilities
logp = logits - logits.max(-1, keepdims=True)
logp -= np.log(np.exp(logp).sum(-1, keepdims=True))
text = decoder_for(seconds).decode(logp, beam_width=100)
text = re.sub(r"'\s+", "'", re.sub(r"\s+'", "'", text))  # "amin' ny" -> "amin'ny"

Build each decoder once and reuse it: loading takes about 12 s.

Setting	α	β	beam
utterances of 10 s or less	0.6	0.5	100
longer clips	0.5	0.5	100

Decoding time is unchanged from the previous model (about 3.6 ms for a 5 s utterance on one CPU core). The binary takes 421 MB on disk and about 0.55 GB more RAM than the previous one.

Files

File	Content
mg_ext_4gram.bin	KenLM binary, trie, 8-bit quantized
unigrams.txt	1,083,929 word forms seen at least twice, needed by pyctcdecode with a binary model
params.json	decoding settings and build recipe
LICENSE_NOTES.md	sources and their licenses

Training text

Every source was normalized exactly like the ASR output and deduplicated. Every line sharing an 8-word sequence with any validation or test transcript was removed, from every source, train transcripts included.

Source	Words	License
HPLT v2 cleaned, Malagasy	89.5 M	CC0 for the dataset; web text
CC-100, Malagasy	19.3 M	Common Crawl terms of use; web text
Malagasy Wikipedia	16.4 M	CC-BY-SA-3.0
Lo-Renz-O/malagasy-sentence	8.2 M	MIT
train-split transcripts of badrex/malagasy-speech-full	0.9 M	not declared

Build: lmplz -o 4 --prune 0 0 1 1 --discount_fallback, then build_binary -q 8 -b 7 -a 256 trie. A Bible translation was available but deliberately excluded for license reasons.

Limitations

• Web text (HPLT, CC-100) brings noise, boilerplate and some other languages.
• Settings were tuned with this acoustic model; another acoustic model needs its own α and β.
• The text is mostly formal written Malagasy. Conversational phrases and commands are under-represented.

License

Mixed sources, no single open license. The model inherits the share-alike terms of Wikipedia (CC-BY-SA). Two thirds of its text is web text gathered by HPLT and CC-100: those datasets are distributed under CC0 or Common Crawl terms, but the pages themselves keep their authors' rights. An n-gram model does not contain the text in readable form, but it does contain sequences of up to four words. See LICENSE_NOTES.md. If you need a model with a clean CC-BY-SA licence, use Flo976/malagasy-kenlm-4gram.

Made for Milo Voice by Flo976 — Florent Didelot / Sooatek.