SARVI-166M-Base

A small language model for the way a lot of India actually types: Hinglish, Banglish, Tanglish and Marathi-English, written in English letters, plus plain English. We trained it from scratch, with our own tokenizer and data pipeline, starting from random weights. It has 166M parameters.

This is a base model. It has read about 576 million tokens and it continues text. It won't chat with you or follow instructions, and it has no personality yet. That part is up to you: it's meant to be fine-tuned.

Use it

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "AnuvaNexus2026/SARVI-166M-Base"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

inputs = tokenizer("Aaj mausam bahut", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Made to be fine-tuned

It's small and uses the standard Llama layout, so you can fine-tune the whole thing on a free Colab or Kaggle T4, or use LoRA with the usual tools (Transformers Trainer, TRL, PEFT, Unsloth, LLaMA-Factory). You could make a chat friend in your own style, a helper for one narrow topic, or a classifier for code-mixed text.

The tokenizer already has chat tokens, and a chat template is included. The format is <|user|>...<|end|><|bot|>...<|end|>.

There's a small kit in finetune/. finetune_sarvi.py learns only the assistant's replies from a file of chats:

python finetune/finetune_sarvi.py --model AnuvaNexus2026/SARVI-166M-Base --data my_chats.jsonl --out my-chat-model

Evaluation

English, zero-shot. Each answer was scored by log-likelihood, once, on the full test set. These are length-normalized scores (acc_norm), with the plain score in brackets.

Test	Questions	SARVI-166M-Base	Chance
ARC-Easy	2,376	31.0% (30.5%)	25.0%
HellaSwag	10,042	27.2% (26.8%)	25.0%

These are low, and we'd rather say so. The model is small, it read very little by modern standards, and most of that wasn't English. Its common sense and general knowledge are weak.

Code-mixed text. We compared it with SmolLM2-135M on the same 150 documents per language, in bits per byte (lower is better).

Language	SARVI-166M-Base	SmolLM2-135M
Hinglish	1.046	2.567
Banglish	1.152	2.975
Tanglish	1.031	2.867
Marathi-English	1.170	2.918
English	1.135	1.023

Please read this table with care:

• The test text comes from our own validation split, so it's the same kind of text SARVI trained on. That favours SARVI.
• SmolLM2 is mainly an English model. This shows SARVI handles these languages much better than that model. It doesn't show SARVI beats multilingual models, and we haven't tested any.
• The first training round used a different split, so some English, Hinglish and Banglish test documents may have been seen in training.
• Most of the romanized text is machine-transliterated Wikipedia, not people texting.

Limitations

• It's small: 166M parameters and a 512-token window.
• It continues text. It doesn't follow instructions or answer questions reliably.
• Its common sense is weak, as the scores above show.
• Its romanized Tamil and Marathi come mostly from machine-transliterated Wikipedia, so the spelling can be bookish.
• It read Hindi and Bengali in their own scripts only in the first round, so those are weaker.
• It learned from web text, so it can repeat errors and biases, or make things up.

Training data

The model was trained only on these downloaded datasets:

Dataset	What we used	Licence
Sangraha by AI4Bharat	Web text in English, Hindi and Bengali, and romanized Hindi, Bengali, Tamil and Marathi	CC BY 4.0
DravidianCodeMix by Chakravarthi et al.	Tamil-English YouTube comments, only those labelled not offensive	CC BY 4.0

Training

• Round 1: about 267 million tokens of cleaned web text from Sangraha (AI4Bharat): English, Hindi, Bengali, and romanized Hindi and Bengali. 9,000 steps, learning rate 6e-4 to 6e-5.
• Round 2: about 309 million tokens: English, Hinglish and Banglish again, romanized Tamil and Marathi from Sangraha, and Tanglish comments from DravidianCodeMix (only those labelled not offensive). 9,000 steps, learning rate 3e-4 to 3e-5.
• Cleaning: we removed links, emails, phone numbers, website menus, repeated lines and duplicates, and fixed the most common machine-transliteration spellings.
• Tokenizer: byte-level BPE, 32,000 tokens, trained on the round-1 data.
• Model: Llama-style decoder with 20 layers, hidden size 768, 12 heads, SwiGLU (2048), RMSNorm, rotary position embeddings and tied embeddings. 166,165,248 parameters.
• Run: AdamW, cosine learning rate, fp16, one NVIDIA T4 on Kaggle, 19.5 hours in total. Final loss 2.66 train, 2.82 validation.

License and credits

Model weights: Apache License 2.0.

Trained on Sangraha by AI4Bharat and DravidianCodeMix by Chakravarthi et al. (paper), both CC BY 4.0.