Aditya109/whisper-small-bhojpuri-lora

Whisper-small Bhojpuri (LoRA)

A LoRA adapter that teaches openai/whisper-small to transcribe Bhojpuri instead of hallucinating standard Hindi at it.

Bhojpuri has roughly 50 million speakers and no Whisper language token. Base Whisper does not fail loudly on it — it hears Bhojpuri and writes fluent, confident Hindi: wrong verb endings (बा → है), Hindi synonyms substituted for dialect vocabulary, wrong postpositions. The acoustic front-end already works; only the decoder's lexical and morphological prior is wrong. That is exactly what LoRA is good at shifting.

Results

Official Vaani Bhojpuri test split, n = 1,426. All rows scored with the same normalizer (see Normalization).

Model	Params	WER ↓	CER ↓
openai/whisper-small (untrained)	244M	122.07	78.75
openai/whisper-large-v3 (untrained)	1.55B	65.93	37.68
ARTPARK-IISc/SraVaani (FastConformer TDT-CTC)	430M	34.80	20.29
This model (whisper-small + LoRA)	244M + 13M	36.41	17.05

70% relative WER reduction over the base model, and it beats untrained large-v3 by ~30 WER points at roughly one-sixth the size.

Every row was measured by me on the same 1,426 utterances, with the same normalizer and the same metric implementation. Nothing here is quoted from another paper's evaluation setup.

Comparison with SraVaani

ARTPARK-IISc/SraVaani is trained on this exact dataset and is the obvious comparison, so it is reported here rather than left for someone else to find. Both models were scored on the same test split, through the same normalizer, with the same metric implementation.

It is a split decision, and both halves are statistically significant (paired bootstrap over utterances, 2,000 resamples):

Metric	This model	SraVaani	Difference	95% CI
WER	36.41	34.80	SraVaani better by 1.61	[+0.83, +2.44]
CER	17.05	20.29	This model better by 3.24	[−3.87, −2.53]

Per-utterance, this model wins on 519, SraVaani wins on 601, and 306 are tied.

Reading the split: SraVaani gets more whole words exactly right; this model is more character-accurate. Its errors are near-misses — the wrong vowel or inflection — where SraVaani's are further off. For Devanagari that distinction matters, because compound words can legitimately be written joined or split and WER charges two full word errors for a space.

What this model offers instead of a WER win: it is a 50 MB LoRA adapter on a 244M base, and being a Whisper derivative it runs directly in faster-whisper, whisper.cpp, and WhisperX. SraVaani is a 430M FastConformer served as TorchScript via trust_remote_code, and does not drop into that tooling. If you need Whisper-ecosystem deployment, this is the trade; if you want the best WER on Bhojpuri and can run their stack, use SraVaani.

Reproduce with benchmark_sravaani.py in the training repo.

Why the baseline WER exceeds 100%

This is legitimate arithmetic, not a bug. WER divides by the reference length and does not cap insertions:

WER = (substitutions + deletions + insertions) / words in reference

Base Whisper-small does not go quiet when confused — it hallucinates. A real example from the measured baseline:

REF: बा हरिहर रंग के                    (5 words)
HYP: अपने अपने अपने अपने अपने अपने ...  (repetition loop)

One 5-word reference, dozens of inserted words. That single utterance scores several hundred percent.

Why CER is reported alongside WER

The WER/CER ratio is diagnostic for this language pair:

	WER	CER	ratio
small, untrained	122.07	78.75	1.55
large-v3, untrained	65.93	37.68	1.75
SraVaani	34.80	20.29	1.72
this model	36.41	17.05	2.14

A rising ratio means remaining errors are concentrated in fewer words that are nearly spelled right — inflectional near-misses rather than misheard audio. For Devanagari, CER also protects against a scoring artifact: compound words can legitimately be written joined or split, and WER charges two full word errors for a space.

What it fixes

Real outputs from the dev set during training:

REF: एगो इ नदी ह नदी में बाउंड्री boundary कइल बा
HYP: एगो इ नदी ह नदी में बैंज्री boundary कइल बा

REF: पेड़ लगाव गए बाटे
HYP: पेड़ लगावा गई बाटे

The model produces Bhojpuri morphology — बा, बाटे, डाली बिया, लगावल गइल — rather than the Hindi equivalents the base model defaults to. Hallucination loops are largely gone: only 10 of 1,426 test utterances (0.7%) still score above 100% WER.

Usage

With PEFT

import torch
from peft import PeftModel
from transformers import WhisperForConditionalGeneration, WhisperProcessor

BASE = "openai/whisper-small"
ADAPTER = "Aditya109/whisper-small-bhojpuri-lora"

processor = WhisperProcessor.from_pretrained(BASE, language="hi", task="transcribe")
model = WhisperForConditionalGeneration.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, attn_implementation="sdpa")
model = PeftModel.from_pretrained(model, ADAPTER).merge_and_unload()
model.to("cuda").eval()

model.generation_config.language = "hi"
model.generation_config.task = "transcribe"
model.generation_config.forced_decoder_ids = None

feats = processor.feature_extractor(
    audio_16k, sampling_rate=16_000, return_tensors="pt"
).input_features.to("cuda", dtype=torch.bfloat16)

ids = model.generate(feats, max_new_tokens=200, num_beams=1)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Notes

• Language token is hi. Whisper has no Bhojpuri token; Hindi is the nearest acoustic and orthographic proxy. The dialect lives in the adapter.
• Audio must be 16 kHz mono.
• Because this is a Whisper derivative, the merged model drops into faster-whisper, whisper.cpp, and WhisperX. That portability is the main practical argument for it over a NeMo Conformer.

Training

Base model	openai/whisper-small (244M)
Method	LoRA (PEFT), bf16, attn_implementation="sdpa"
Rank / alpha / dropout	32 / 64 / 0.05
Target modules	q_proj, k_proj, v_proj, out_proj, fc1, fc2
Trainable params	13M (5% of total)
Learning rate	1e-3, warmup ratio 0.05
Batch	16 × 2 grad-accum = 32 effective
Epochs	6 budgeted, ran the full ~2,100 steps
Best checkpoint	step 2,000 (dev WER 32.90)
Precision	bf16
Hardware	1 × NVIDIA RTX 5080 (16 GB, Blackwell sm_120)
Wall clock	~1.2 hours
Seed	13

Checkpoint selection

Checkpoints were selected on dev WER, not eval_loss. In ASR these routinely diverge — loss can rise while WER falls, because the model becomes less confident but more correct. Selection on loss regularly keeps the wrong checkpoint. Real transcripts were generated at every evaluation (predict_with_generate=True) and scored with the same normalizer used in training and final evaluation.

Dev WER curve (400-utterance dev subset):

step	WER	CER
250	50.96	31.70
500	39.11	20.91
750	38.94	20.79
1000	36.84	18.57
1250	33.24	16.05
1500	34.59	16.12
1750	32.97	15.25
2000	32.90	15.17
2100	32.91	15.33

80% of the improvement arrived by step 1,250. Everything after that oscillates within about ±1 WER, which is the noise floor of a 400-utterance dev subset — the last three checkpoints are statistically indistinguishable.

Data

ARTPARK-IISc/Vaani-transcription-part, config Bhojpuri — ~24 hours of transcribed spontaneous speech (speakers describing prompt images). CC-BY-4.0, gated on the Hub.

The dataset's official splits were used unmodified:

split	n
train	11,191
validation	1,517
test	1,426

The evaluation scored exactly 1,426 utterances, confirming the test split was never re-partitioned.

Normalization

Training and evaluation used one shared normalizer, imported by data prep, training, and evaluation alike — text_norm.py in the training repo. It:

• applies Unicode NFC
• strips [noise] / (laughs) / <unk> style annotation tags
• maps Devanagari digits to ASCII
• strips punctuation including danda (।) and double danda (॥)
• lowercases (affects only Latin characters in code-mixed text)
• collapses whitespace

It deliberately does not touch nukta or matras, since those changes alter actual words.

WER numbers are not comparable across different normalizers. If you benchmark against this model, use the same one or state your own.

Leakage: what is verified and what is not

Vaani has no speaker ID column. ARTPARK had speaker metadata when they built the official splits; a downstream user does not. Speaker-disjointness is therefore trusted, not verified — by me or by anyone else working from the public dataset. This is a property of the dataset, and any model card claiming verified speaker-disjointness on Vaani is overclaiming.

What was checked, and passed:

1. Official splits used unmodified. The evaluation scored exactly 1,426 utterances. A prior run of this project scored 37.12 WER by concatenating the splits and re-partitioning randomly; that result was invalid and is not reported here.
2. Test scored worse than dev — 36.41 vs 32.90. Leakage makes held-out data easier, not harder. The gap is evidence against shared speakers.
3. Verbatim transcript overlap is 0.28% — 4 of 1,426 test transcripts also appear in train.
4. Removing those 4 moves WER by −0.05 (36.41 → 36.36). No duplicate-text inflation.
5. The model scored 61.7 WER on those 4 — worse than its 36.41 average. A model recalling training text would do the opposite.
6. WER is flat across utterance lengths (34–40 from 4 words to 21+). No anomalously easy bucket.
7. Only 4.9% of utterances score a perfect 0 — leakage typically produces a large spike of exact matches.

The audit script (check_leakage.py) is in the training repo; these numbers are reproducible.

Other overlap worth disclosing

referenceImage overlap between train and test is expected and is not speaker leakage — it means speakers were shown some of the same prompt images, so the splits share topics.

Also note the Vaani transcription convention writes English loanwords twice, once in Devanagari and once in Latin (टेबल table, क्लास class रूम room). The model learns this formatting convention, which contributes to the WER improvement without necessarily reflecting better Bhojpuri understanding.

Limitations

• Read speech and other domains are untested. Training data is spontaneous image-description speech from one collection protocol.
• Regional coverage is whatever Vaani sampled. Bhojpuri varies considerably across its range.
• No standard-Hindi regression check was run. The adapter may degrade standard Hindi performance (catastrophic forgetting). If you need both, mix 10–20% Hindi into training and verify on a Hindi benchmark.
• Long-form audio is untested. Evaluation used utterance-level clips with greedy decoding, max_new_tokens=200.
• Greedy decoding only. Beam search was not evaluated and may score differently.
• Reference transcript quality varies. Inspection of the worst-scoring utterances suggests some of the remaining error is annotation noise rather than model error — a floor no amount of training removes.

Licensing

• Base model: openai/whisper-small is MIT licensed.
• Training data: Vaani is CC-BY-4.0, which requires attribution.
• This adapter is released under CC-BY-4.0 to carry that attribution forward. No Vaani audio or transcripts are redistributed here.

Citation

Attribution to the Vaani dataset is required under CC-BY-4.0. This is the citation requested on the dataset page:

@misc{pulikodan2026vaanicapturinglanguagelandscape,
      title={VAANI: Capturing the language landscape for an inclusive digital India},
      author={Sujith Pulikodan and Abhayjeet Singh and Agneedh Basu and Nihar Desai and Pavan Kumar J and Pranav D Bhat and Raghu Dharmaraju and Ritika Gupta and Sathvik Udupa and Saurabh Kumar and Sumit Sharma and Vaibhav Vishwakarma and Visruth Sanka and Dinesh Tewari and Harsh Dhand and Amrita Kamat and Sukhwinder Singh and Shikhar Vashishth and Partha Talukdar and Raj Acharya and Prasanta Kumar Ghosh},
      year={2026},
      eprint={2603.28714},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2603.28714}
}

Reproducing

python prepare_data.py --language Bhojpuri --out data/bhojpuri

python evaluate.py --data data/bhojpuri --split test \
    --model openai/whisper-small --dump baseline_small.jsonl

python train_lora.py --data data/bhojpuri --out runs/bhojpuri-small \
    --model openai/whisper-small --lr 1e-3 --epochs 6 --batch 16

python evaluate.py --data data/bhojpuri --split test \
    --model openai/whisper-small \
    --adapter runs/bhojpuri-small/adapter --dump tuned_small.jsonl

python check_leakage.py --data data/bhojpuri --dump tuned_small.jsonl