HmarBERT

HmarBERT is the foundational BERT-base masked language model for the Hmar language (hmr, ISO 639-3), developed by domain-adapting robzchhangte/MizBERT with an in-place cognate-swapped vocabulary and multi-stage Whole-Word Masking (WWM).

Model Details

• Base Architecture: BERT-base (robzchhangte/MizBERT, 110M parameters)
• Vocabulary: 30,522 tokens with 303 in-place Mizo-to-Hmar cognate substitutions (0 token collisions)
• Pre-training Corpus: hmar-heritage-org/dolma-pretrain (101,867 train sentences, 2,484,864 words across 7 registers)
• Evaluation Benchmark: 2,099 stratified test sentences (51,840 words)
• Cumulative Training: 20 Epochs across 3 Stages:
  • Stage 1 (Adaptation): 6 epochs @ 3e-5 (Warm-start on raw MizBERT)
  • Stage 2 (Cognate Swap): 6 epochs @ 2e-5 (303 in-place swaps, Global shuffle)
  • Stage 3 (Deep Refinement): 8 epochs @ 1.5e-5 (2 full cycles of 4-pass 25% Whole-Word Masking)

Benchmark Results

Stage / Model	Training Curriculum	Eval Loss	Perplexity (PPL)
MizBERT Base (Zero-shot Hmar)	Raw MizBERT base	2.7116	15.05
HmarBERT Stage 2	6 epochs (15% WWM, 303 swaps)	2.5482	12.78
HmarBERT Stage 3	8 epochs (Dual Cycle WWM)	2.4849	12.00
HmarBERT Stage 3 (25% High-Stress WWM)	8 epochs (25% masking across 4 slices)	2.7369	15.44

Interactive Demo

Try HmarBERT live in your browser: Dolma — Powered by HmarBERT

Usage Example

from transformers import pipeline

fill_mask = pipeline("fill-mask", model="azinamotoe/HmarBERT")
result = fill_mask("Hrietthiem [MASK] rel el tala, ka um dan hi!")
for r in result[:3]:
    print(f"{r['token_str']:<15} (score: {r['score']:.4f}) -> {r['sequence']}")

# Output:
# 1. zai             (score: ...) -> Hrietthiem zai rel el tala, ka um dan hi!

Citation & Attribution

@software{muolhoi_hmarbert_2026,
  author    = {Donal Muolhoi},
  title     = {HmarBERT: Pretrained Bidirectional Encoder Representations for Hmar},
  year      = {2026},
  publisher = {Hugging Face},
  doi       = {10.57967/hf/10428},
  url       = {https://huggingface.co/azinamotoe/HmarBERT},
  keywords  = {Hmar, hmr, hmar1241, Zo Languages, BERT, Masked Language Model, NLP},
  note      = {Language: Hmar (ISO 639-3: hmr, Glottolog: hmar1241). Language family: Zo Languages.}
}