HmarBERT-mini-causal

HmarBERT-mini-causal is an autoregressively aligned causal language model derived from azinamotoe/HmarBERT-mini (16.91M parameters), trained for the Hmar language (hmr, ISO 639-3).

While HmarBERT-mini uses bidirectional attention for masked token reconstruction, HmarBERT-mini-causal enforces a causal (lower-triangular) attention mask (is_decoder=True). It is aligned specifically for ultra-fast next-word prediction, designed to power:

1. Mobile Predictive Keyboards (Android / iOS keyboard engines)
2. Termux / CLI Autocomplete (instant local suggestions with low memory overhead)

---

Model Architecture & Specs

• Base Architecture: 4-layer BERT decoder (is_decoder=True, causal attention masking)
• Parameters: 16.91M (~32.2 MB FP16 / ~67.6 MB FP32)
• Hidden Size: 384 dimensions, 6 attention heads, intermediate size 1536
• Vocabulary: 24,576 WordPiece tokens with full Hmar diacritics and morphology
• Sequence Context: Optimized for up to 64 tokens (ideal for keyboard typing windows)
• Inference Latency: <5ms per token on basic mobile / laptop CPUs

---

Causal Behavioral Alignment

The base model azinamotoe/HmarBERT-mini completed 48 epochs of bidirectional pretraining (24 epochs on sentences + 24 epochs on paragraphs).

In HmarBERT-mini-causal, the pre-trained weights are adapted into a causal decoder:

• Causal Masking: Lower-triangular attention ensures token $i$ only attends to tokens $\le i$.
• Training Data: Aligned on clean, segmented sentences from hmar-heritage-org/sentences.
• Behavioral Objective: Aligns attention projections to predict upcoming words strictly from left-to-right conversational history without future context.

---

Quickstart (Inference)

import torch
from transformers import BertTokenizerFast, BertLMHeadModel

MODEL_ID = "azinamotoe/HmarBERT-mini-causal"

tokenizer = BertTokenizerFast.from_pretrained(MODEL_ID)
model = BertLMHeadModel.from_pretrained(MODEL_ID)
model.eval()

def predict_next_words(text: str, top_k: int = 3):
    inputs = tokenizer(text, return_tensors="pt")
    with torch.no_grad():
        logits = model(**inputs).logits[0, -1, :]
    
    probs = torch.softmax(logits, dim=-1)
    top_probs, top_indices = torch.topk(probs, k=top_k)
    
    return [
        (tokenizer.decode([idx.item()]).strip(), f"{prob.item() * 100:.1f}%")
        for prob, idx in zip(top_probs, top_indices)
    ]

# Example predictions
print(predict_next_words("Ka lawm", top_k=3))
print(predict_next_words("Invaithai", top_k=3))

---

Citation & Organization

Maintained by the Hmar Heritage Foundation and Azinamotoe.

• Organization: hmar-heritage-org
• Base Pretrained Model: azinamotoe/HmarBERT-mini