abhishekgautamm/whisper-tiny-hinglish

Whisper-tiny Hinglish (Romanized)

A fine-tune of openai/whisper-tiny that transcribes Hindi, English, and code-switched Hinglish Indian speech directly into natural Roman script — the way people actually type Hinglish (kya scene hai, not क्या सीन है and not a stiff academic transliteration).

• 39 M parameters, 4 encoder + 4 decoder layers, 80-mel, 16 kHz — runs on CPU / edge.
• Output is always Roman, even for pure-Hindi speech. No Devanagari, no separate transliteration step at inference time.
• Built for on-device dictation for Indian users, but released here as a standalone, general-purpose model for the community.

This is a small model. It is fast and convenient, not state-of-the-art. See Limitations.

Example outputs

The model emits Roman text directly from audio. Target style (illustrative):

Speech (reference)	Model output style
"…with my daughter और जो मेरी daughter है उसको बहुत पसंद है diwali…"	…with my daughter aur jo meri daughter hai usko bahut pasand hai diwali…
"वीर बड़े भाई known for his curiosity and दीया छोटी बहन…"	veer bade bhai known for his curiosity and deeya chhoti bahan…
"आपका hometown किस city में है…"	aapka hometown kis city mein hai…
"दिवाली मनाते हैं साथ में पटाखे जलाते हैं"	diwali manate hain saath mein patakhe jalate hain

English spans are kept verbatim; Hindi spans come out in natural Roman spelling.

Usage

from transformers import pipeline

asr = pipeline("automatic-speech-recognition",
               model="<your-username>/whisper-tiny-hinglish",
               chunk_length_s=30)          # for clips longer than 30 s

print(asr("clip.wav")["text"])

Or with raw 16 kHz mono audio:

import soundfile as sf
audio, sr = sf.read("clip.wav")            # resample to 16 kHz mono if needed
print(asr({"raw": audio, "sampling_rate": 16000})["text"])

The model does not require a forced language token — it was trained to emit Roman for every input. No post-processing / transliteration is needed.

Training data

Fine-tuned on Indian-accent, read- and prompted-speech corpora, chosen for their heavy Hindi–English code-switching:

Source	Content
HiACC (adult + children)	Indian-accented Hindi/English/Hinglish read & conversational speech
MUCS (Hindi–English)	Code-switched Hindi–English utterances (Kaldi-segmented)
Common Voice (Hindi / Indic)	Crowdsourced Hindi read speech

Utterances were filtered to ~0.5–20 s. Original transcripts are mixed Devanagari + Latin.

Romanization of the targets (the key step)

The goal is Roman output. So every training target was converted to natural Roman before fine-tuning:

• Devanagari spans → romanized with LIPI, a compact CTC Devanagari→Roman model that produces typed-Hinglish spellings (क्या → kya, है → hai, जो → jo), not academic IAST.
• English (Latin) spans in the transcripts were left unchanged.

The model then learns audio → Roman end-to-end, so LIPI is only a data-prep tool and is not needed at inference.

Strengths

• Natural romanized Hinglish output, ready to paste — no transliteration step downstream.
• Handles code-switching within a sentence (Hindi ↔ English) in one pass.
• Tiny & fast: real-time on CPU, sub-second on a modest GPU; ~150 MB fp32 (quantizes well).
• Robust to Indian accents on short, clear, conversational/dictation-style audio.

Limitations

• Small-model accuracy. Expect errors on noisy audio, fast speech, strong accents, overlapping speakers, or domain jargon.
• English spelling drift. Uncommon English words can come out phonetically (crackers → crackers/crakers), since the training signal is Indian-accented and Hindi-weighted.
• Romanization is phonetic, not canonical. Spellings follow how Hindi is typed and can vary (hai/hain, ki/kii). There is no single "correct" Roman spelling.
• Short-form. Trained on ≤ ~20 s utterances; use chunk_length_s=30 for longer audio, quality on very long recordings is untested.
• No reliable punctuation/casing. Output is lightly punctuated at best.
• Language scope. Hindi + English + Hinglish (Indian). Other Indic languages are out of scope.
• Domain. Read/prompted Indian speech; far-field, telephony, and heavy background noise are under-represented.

Citation

@misc{whisper-tiny-hinglish,
  title  = {Whisper-tiny Hinglish (Romanized)},
  author = {Abhishek Gautam},
  year   = {2026},
  note   = {Fine-tuned openai/whisper-tiny for romanized Hindi/English/Hinglish ASR;
            targets romanized with LIPI (https://github.com/ABHISHEKgauti25/lipi)}
}