HmarBERT-mini

HmarBERT-mini is a lightweight BERT model built specifically for the Hmar language (hmr, ISO 639-3).

Our earlier model (HmarBERT) was an initial test adapting weights from Mizo. It served well as a proof of concept, but inherited patterns from the source language.

This one is trained entirely from scratch—a fresh vocabulary, random weight initialization, and trained only on Hmar text.

At roughly 16.9M parameters (~32 MB), it's small enough to run easily on a laptop CPU, a mobile device, or directly in the browser.

---

Model Details

• Architecture: 4-layer BERT encoder
• Parameters: 16.91M (~32.2 MB in FP16)
• Hidden Size: 384 (6 attention heads)
• Vocabulary: 24,576 WordPiece tokens
• Max Length: 512 tokens
• Training Data: hmar-heritage-org/sentences (~265k sentences)

---

Training Progress

The model is trained using Whole-Word Masking (WWM) so it learns complete words rather than subword fragments.

Stage	Epochs	Train Loss	Val Loss	Perplexity	Notes
Cycle 1 & 2	12	5.17	5.54	254.8	Current checkpoint
Cycle 3 & 4	24	—	—	—	Planned next run
Cycle 5 & 6	36	—	—	—	If loss continues dropping

Training and validation loss are tracking closely (5.17 vs 5.54), indicating steady learning. We plan to run another 12 epochs (up to 24), and possibly up to 36 if it continues to improve.

---

Tokenizer Notes

• Accents: Casual mobile typing without accents (tirin, van) maps to the same tokens as formal dictionary spelling (tîrin, vân).
• Emojis & Symbols: Unicode emoji and mobile symbols are handled cleanly without throwing unknown token ([UNK]) errors.

---

Quickstart

from transformers import pipeline

unmasker = pipeline("fill-mask", model="azinamotoe/HmarBERT-mini")

# Try a masked sentence:
results = unmasker("A tirin Pathienin hnuoi le [MASK] a siem a.")
for r in results[:3]:
    print(f"{r['token_str']:<15} ({r['score']*100:.1f}%) -> {r['sequence']}")