Dolma v2 (Hmar Masked Language Model)

Dolma is a foundational BERT model for the Hmar language (hmr, South-Central Tibeto-Burman / Zo family), developed by the Hmar Heritage Foundation.

This checkpoint (v2) continues the domain-adapted pre-training of azinamotoe/Dolma across 6 total epochs on hmar-heritage-org/dolma-pretrain (89,605 multi-register sentences).

Model Details

• Architecture: BERT-base (12 layers, 768 hidden dimension, 12 attention heads)
• Domain Adaptation Corpus: 89,605 sentences (~2.4M tokens)
• Training Epochs: 6 total epochs (dynamic masked language modeling)
• Validation Perplexity: 7.63
• Evaluation Loss: 2.0317

Usage

from transformers import pipeline

fill_mask = pipeline('fill-mask', model='azinamotoe/Dolma')

# Example 1: Literature / Cultural Sentence
result = fill_mask('Khawvel fe dan phung ei en chun, ram le [MASK] damna thuruk chu...')
for res in result:
    print(f"{res['token_str']}: {res['score']:.4f}")

# Example 2: Common Phrase
result2 = fill_mask('Pathienin khawvel a [MASK] em em leiin.')
for res in result2:
    print(f"{res['token_str']}: {res['score']:.4f}")

Intended Use & Downstream Tasks

• Designed for masked token prediction, sentence representation, and downstream fine-tuning (e.g. text classification, sentiment, NER, and dialect identification).
• Operates on both formal diacritic orthography (ṭ) and normalized ASCII text.
• Maintained by the Hmar Heritage Foundation.