svara-whisper-small-hi

Whisper-small fine-tuned on 20,000 Hindi utterances from AI4Bharat's IndicVoices.

On Voice Arena's Monsoon Hindi test set, scored with the Open ASR Leaderboard's own harness, it gets 53.7% OIWER against 241.6% for the base model. On FLEURS Hindi it scores WER 0.476 against 0.815. Both gaps are driven by the same thing: whisper-small falls into repetition loops on spontaneous Hindi, and this model does not.

It is a standalone Whisper model, with the LoRA adapter merged in, so it loads like any other Whisper checkpoint. A CTranslate2 int8 export for CPU use is at cruqAI/svara-whisper-small-hi-ct2.

Results: Monsoon hi (spontaneous speech)

VoiceArena/Monsoon_hi_test, 753 utterances, 1.33 hours, scored by running open_asr_leaderboard's transformers/run_eval_ml.py unmodified on one RTX 3090. The metric is OIWER, which counts a word correct if it matches any spelling in that slot's reference lattice. This is the leaderboard's own benchmark and code, run by us; it is not an official leaderboard submission.

Model	OIWER	RTFx
openai/whisper-small	241.6%	27.3
svara-whisper-small-hi	53.7%	45.1

An error rate over 100% means the model emitted more wrong words than the reference has words. That is what happens here: whisper-small produces a repetition loop on 264 of the 753 clips, one of them 148 words of "तो". Those loops alone add 22,231 words to a 13,006-word reference, which is 171 of the 241 points. This model loops on 4 clips, and its total output length is 1.04x the reference against 2.64x for the base model. It is also faster for the same reason: it is not generating a hundred wasted tokens.

Results: FLEURS hi (read speech)

FLEURS Hindi test split, 418 utterances, 1.34 hours of read speech. Greedy decoding with the decoder's own 448-token limit, fp16 on one GPU. Both rows were scored by the same code with the same text normalisation (danda to full stop, punctuation and case removed, numerals expanded), so the comparison is fair but the absolute numbers are not the official Whisper ones.

Model	WER	CER
openai/whisper-small	0.815	0.476
svara-whisper-small-hi	0.476	0.198

Where the gap comes from: on 33 of the 418 clips the base model repeats a word or phrase until it runs out of tokens (one output is the word "तो" 90 times). The fine-tuned model does that on none of them. With a 200-token cap that cuts such loops short, the base model scores WER 0.650 / CER 0.328 and this model 0.486 / 0.207, so the gain on clips the base model handles normally is about 16 WER points, and the rest of the uncapped gap is the loops.

The int8 CPU export scores WER 0.489 / CER 0.205 on the same set, at a real-time factor of 0.17 on 4 CPU threads.

Two caveats. FLEURS is read speech and the training data is mostly conversational and spontaneous, so this is an out-of-domain test; expect different numbers on conversational audio. And fp16 inference is not bit-stable across GPUs: the same model scored within 0.005 of itself on different cards, so differences that small are noise.

Training data

The first 20,000 rows, in corpus order, of the Hindi training split of ai4bharat/IndicVoices (CC-BY-4.0). Of those, 16,571 utterances were used for training after a speaker-aware split held out development and test slices. IndicVoices is crowd-collected: read, extempore and conversational speech from many speakers and districts, with the noise and code-mixing that implies.

The FLEURS test set was never used for training or model selection.

Training

• Base: openai/whisper-small (244M parameters)
• Method: LoRA on the attention query and value projections, rank 32, alpha 64, dropout 0.05 (3.5M trainable parameters), then merged into the base weights
• 3 epochs, 3,108 optimizer steps, effective batch size 16 (8 x 2 accumulation)
• Learning rate 1e-5 with 500 warmup steps, fp16, no label smoothing
• 52 minutes on one RTX 3090 rented through Nosana
• Training text was normalised with the project's training profile, which keeps punctuation; scoring used the stricter profile described above

The code, data pipeline and the exact run configuration are in github.com/cruq-ai/svara.

Use

from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="cruqAI/svara-whisper-small-hi",
    generate_kwargs={"language": "hindi", "task": "transcribe"},
)
print(asr("clip.wav")["text"])

Audio is resampled to 16 kHz by the pipeline. Clips longer than 30 seconds need chunk_length_s=30 or return_timestamps=True, as with any Whisper model.

Limitations

• Hindi only. The fine-tune was single-language and other Whisper languages were not evaluated after it.
• Trained on 20,000 utterances, which is a small fraction of IndicVoices Hindi.
• The scoring normaliser expands numerals to words, so a transcript with digits is penalised against a reference with words and vice versa.
• The Monsoon numbers come from the leaderboard's harness run on our own hardware, not from an official leaderboard submission.
• Both test sets are about 1.3 hours. Differences of a point or two are not meaningful at that size.

License and attribution

The model weights are released under Apache-2.0, the same licence as the base model. The training data is CC-BY-4.0: AI4Bharat, IndicVoices. Evaluation used FLEURS (CC-BY-4.0), Google.