MONA: a multilingual small language model you can open up

MONA is a 110M-parameter Llama-style language model trained from scratch on one laptop GPU for English, Hindi, Hinglish and Awadhi (Tulsidas). It was built to teach how LLMs work: the companion terminal app lets anyone look inside it, stage by stage, in English, हिंदी or Hinglish.

Code, app and full build story: https://github.com/JayD108/Mona
pip install the repo, then run mona: the app downloads these files on first run.

Model


Architecture	decoder-only transformer: 12 layers, width 768, 12 heads, SwiGLU 2048, RMSNorm, RoPE, tied embeddings
Parameters	110.1M (85.0M non-embedding)
Tokenizer	byte-level BPE, 32,768 tokens, trained on a balanced EN / HI / Hinglish / Awadhi sample (Hindi 1.33 tokens/word)
Context	1,024 tokens
Pretraining	2.5B tokens, AdamW, bf16, WSD learning-rate schedule (peak 6e-4), ~19 h on an RTX 5080 Laptop GPU
Chat tuning	supervised fine-tuning with loss masking on ~100K conversations

Evaluation (held-out text, bits per byte, lower is better)

English	Hindi	Hinglish
1.027	0.455	1.058

Final held-out loss per language (nats/token): english 3.146, hindi 3.063, hinglish 2.899, awadhi 5.445

Sample (MONA-Chat, "Who are you?"): *I'm Emily Wilson, the executive director of the National Geographic Society. I'm the lead author of the new report, "The Greatest Planet: The Greatest Earthquake in History." I'm thrilled to share my *

Files

assets/models/mona/mona.pt: base model (bf16 state dict) · assets/runs/mona_m_chat/model.pt: MONA-Chat
assets/models/mona/tokenizer.json: tokenizer (Hugging Face tokenizers format)
assets/…: a small, licence-clean demo bundle used by the app (see NOTICE.md)

from mona.engine.glassbox import GlassBox          # from the GitHub repo
gb = GlassBox()                                    # loads the newest weights on CPU
print(gb.next_word("The capital of India is", top=5))

Intended use and limits

For education and experimentation. MONA is small: it writes fluent text but often states things that are wrong, cannot do arithmetic, and must not be used for factual, medical, legal or other consequential decisions.

Training data and licences

English: FineWeb-Edu (ODC-By 1.0) · Hindi and romanized Hindi: AI4Bharat Sangraha (CC-BY-4.0; its Wikipedia-derived parts are CC BY-SA) · chat: smol-smoltalk (Apache-2.0), Aya dataset (Apache-2.0) · Awadhi: Tulsidas's verses (public domain). See NOTICE.md for full attribution.

Copyright note: an earlier Tulsidas corpus also contained a modern Hindi commentary (Gita Press, likely still under copyright). When this was noticed, at 0.8B of 2.5B training tokens (~0.1% of batches up to then), it was removed: the rest of pretraining, the chat tuning and everything in this repository use only public-domain verse. No commentary text is distributed here.

Credits

Built by Mrittunjay Dubey. Coding assisted by Claude (Anthropic).

Downloads last month: 12