muril-hinglish-lid

Context-aware Hindi/English language identification for romanized Hinglish.

Labels every word of code-mixed Latin-script text as HIN or ENG, using the surrounding sentence rather than a dictionary.

Example

Input:

Mujhe calculus ka doubt samajh nahi aaya, please explain.

Output:

word	label
Mujhe	HIN
calculus	ENG
ka	HIN
doubt	ENG
samajh	HIN
nahi	HIN
aaya	HIN
please	ENG
explain	ENG

Why context matters

The same string can be two different languages in one sentence:

Main   is    question  ka    main  point  samajh  nahi  paya
HIN    HIN   ENG       HIN   ENG   ENG    HIN     HIN   HIN
 └─ मैं  └─ इस                  └─ English "main"

Main is मैं; four words later main is the English adjective. is is इस, not the English copula. No lookup table can express this — it has one row per string.

That is not a contrived example. In our evaluation data the word to occurs 775 times, splitting 435 Hindi (तो) / 340 English. A perfect lookup table gets at most 56% of those by always guessing the majority. This model gets 98.32%.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification

MODEL = "PhysicsWallahAI/muril-hinglish-lid"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForTokenClassification.from_pretrained(MODEL).eval()


def tag_words(words: list[str]) -> list[str]:
    """One label per word, read off the word's FIRST sub-token."""
    enc = tok(words, is_split_into_words=True, truncation=True,
              max_length=256, return_tensors="pt")
    with torch.no_grad():
        pred = model(**enc).logits[0].argmax(-1).tolist()
    out, prev = [], None
    for pos, wid in enumerate(enc.word_ids()):
        if wid is not None and wid != prev:
            out.append(model.config.id2label[pred[pos]])
        prev = wid
    return out


words = "Mujhe calculus ka doubt samajh nahi aaya please explain".split()
print(list(zip(words, tag_words(words))))
# [('Mujhe','HIN'), ('calculus','ENG'), ('ka','HIN'), ('doubt','ENG'),
#  ('samajh','HIN'), ('nahi','HIN'), ('aaya','HIN'), ('please','ENG'),
#  ('explain','ENG')]

Use this helper rather than pipeline(..., aggregation_strategy=...). The pipeline's aggregation is designed for NER: it merges consecutive tokens sharing a label into one span, which is right for New York City → one LOC and wrong here, where adjacent words routinely share a language. On the example above it returns 7 spans instead of 9 words, fusing samajh nahi aaya, into a single HIN blob. aggregation_strategy="none" is not the fix either — it returns sub-word pieces ('Mu', '##jhe').

The helper matches the training-time contract exactly: words pre-split, one label per word taken from its first sub-token. Continuation pieces were masked out of the loss during training and carry no supervision, so reading them — or averaging over them, which would let a long word's many pieces outvote a short word's one — asks the model something it was never taught to answer.

Tag in windows of ~100 words, which is the window used in training.

Model

• MuRIL backbone (google/muril-base-cased), pretrained on 17 Indian languages and their romanized forms
• Token-classification head, 2 labels
• 237 M parameters · 12 layers · hidden 768 · WordPiece vocab 197,285
• Labels: HIN / ENG
• Context-aware prediction

Performance

Held-out gold set

874 answers / 108,432 tokens. The evaluation protocol is the part worth reading: these labels were annotated from the source text by a separate model that wrote none of the training corpus. Training labels were derived by aligning Hinglish text to its Devanagari rewrite; the gold labels were not. So this is the only measurement here that is independent of the pipeline that produced the training data.

metric	value
accuracy	0.9920
HIN F1	0.9927
ENG F1	0.9912
contextual homographs (8,444 occurrences)	0.9861
false-Hindi rate on English-only text	0.02% (1 of 4,363 tokens)

On its own auto-labelled test split the model reads 0.9934 — but that split is the instrument that cannot see its own errors. The gold figure is the one to quote.

Hard cases

word	accuracy	note
to	0.9832	775 occurrences, 435 तो / 340 English. Lookup ceiling: 56%
use	0.7799	उसे vs English "use". The weakest case, and unchanged across two independently retrained checkpoints — the signature of genuine ambiguity rather than a data defect
beta	—	बेटा (term of address) vs β, the physics symbol

Out of domain: LinCE Hindi-English

LinCE is human-annotated, public, and from a different domain entirely — code-mixed tweets, not tutoring text.

	tokens	accuracy	HIN F1	ENG F1
LinCE dev	12,303	0.9643	0.9342	0.9755

Not comparable to the LinCE leaderboard. LinCE has eight classes; this model has two. We map lang1→ENG, lang2→HIN and drop the rest — other (punctuation, handles, URLs, emoji; 2,231 tokens), ne (named entities; 875), and 37 tokens of fw/mixed/unk/ambiguous. Scoring only the two mappable classes is an easier task than LinCE's. Compare this number to our in-domain accuracy, and to nothing else.

Latency

0.099 ms/word (108 words in 10.7 ms), NVIDIA A10, fp32, batch 1.

Training

data	Hinglish tutoring answers, labelled by aligning each answer to its Devanagari rewrite
train	23,727 answers / 2,783,768 labelled tokens
dev	1,402 answers / 162,047 labelled tokens
test	2,771 answers / 322,569 labelled tokens
splits	by answer — the same answer never appears on two sides
hyperparameters	3 epochs · batch 32 · lr 3e-5 · warmup ratio 0.1 · weight decay 0.01 · fp16 · max length 256 · 100-word windows
selection	best epoch by masked accuracy on dev; test read once, at the end

Tokens the aligner could not place — math variables, symbols, ambiguous residue — were emitted as O and masked out of the loss rather than guessed at, so the model was never trained to invent a label for something the labelling process did not actually know.

Intended use

• Hinglish language identification
• Preprocessing for text-to-speech
• Transliteration pipelines
• Code-switched Hindi/English text

Not intended for

• General language identification across arbitrary languages. Two labels only. Other Indian languages in Latin script will be forced into HIN or ENG.
• Devanagari transliteration itself. This model decides what to transliterate. It does not transliterate.
• Determining whether a whole sentence is Hindi or English. It is a token-level model. Aggregating its labels to a sentence verdict is not what it was built or evaluated for.

Limitations

SMS and chat shorthand. The largest out-of-domain error class is abbreviated social-media spelling, absent from tutoring text: ur, u, h, k, b, r, g. On chat-register text expect worse than the LinCE number suggests — that number already contains these errors, but diluted by well-formed tokens.

Short fragments. Accuracy tracks available context, not language. On short homograph-dense English — "Let me know so I can do it in the morning" — error rates rise sharply, because let, me, know, so, can, do are all Hindi words in other contexts and six words condition almost nothing. On full-length English text the false-Hindi rate is ~1%; on hand-picked short fragments it was 12.8%. Length is the variable.

Named entities are out of scope. The model emits only HIN/ENG and was never trained on an NE class. Personal and place names receive some label, and which one is not meaningful. Handle names separately.

Domain. Indian K-12 / exam-prep tutoring text: explanatory, second-person, mathematics- and science-heavy, generally well-formed sentences. That is where 0.9920 holds.

Data

The training data is proprietary tutoring text and is not released. No training data is included in this repository — weights, config and tokenizer only.

The data reflects the register, subject matter and code-mixing conventions of one setting: Hindi-English as used in Indian exam preparation. It is not a sample of Hinglish in general, and the model's notion of "which words are Hindi" is that community's.

Worth stating plainly, since this model exists to serve a TTS front-end: a HIN label causes a word to be transliterated to Devanagari and pronounced with Hindi phonology. A mislabelled word is therefore mispronounced, not dropped. The failure is audible — the right direction for a failure to go, but it does mean errors reach listeners directly.

License

Apache-2.0, inherited from google/muril-base-cased. Training data is not included and is not licensed for redistribution.

Citation

@misc{muril-hinglish-lid,
  title  = {muril-hinglish-lid: context-aware Hindi/English language identification for romanized Hinglish},
  author = {PhysicsWallah AI},
  year   = {2026},
  url    = {https://huggingface.co/PhysicsWallahAI/muril-hinglish-lid}
}

Base model:

@article{khanuja2021muril,
  title   = {MuRIL: Multilingual Representations for Indian Languages},
  author  = {Khanuja, Simran and Bansal, Diksha and Mehtani, Sarvesh and others},
  journal = {arXiv preprint arXiv:2103.10730},
  year    = {2021}
}