SraVaani-0.5-live — Streaming ASR (Cache-Aware FastConformer, CTC)

SraVaani-0.5-live is a multilingual Automatic Speech Recognition model for 63 Indian languages and dialects. This repository is the low-latency streaming member of the family: a cache-aware FastConformer encoder with a CTC decoding head, exported to a single self-contained TorchScript graph that runs on torch + transformers + `sentencepiece.

One exported graph serves four latency settings (1040 ms → 0 ms lookahead); switch between them at runtime with a single call — no reload, no re-export.

• Input: 16 kHz mono audio (any length; streamed internally chunk-by-chunk)
• Output: text transcript
• Decoder: CTC (greedy, repeat-collapse with cross-chunk state)
• Base model: ~430M-param hybrid TDT-CTC FastConformer (this export keeps the CTC head)
• Tokenizer: 5000-piece SentencePiece BPE, multi-script Indic + Latin (code-switch)
• Runtime: real-time-safe on CPU, >60× faster than real-time on a single GPU

---

Languages

Supports 63 Indian languages and dialects:

• Scheduled (19): Assamese, Bengali, Bodo, Dogri, Gujarati, Hindi, Kannada, Konkani, Maithili, Malayalam, Manipuri, Marathi, Nepali, Odia, Punjabi, Sanskrit, Santali, Sindhi, Tamil, Telugu
• Non-scheduled (44+): including Angami, Ao, Awadhi, Bhojpuri, Chhattisgarhi, English, Garo, Khariboli, Mizo, and others.

Not supported: Urdu and Kashmiri.

Fine-tuning data varies significantly across languages, so accuracy is lower for low-resource languages. The streaming WER below is reported for Hindi only (see Performance).

---

Quick start

import torch
from transformers import AutoModel

REPO = "ARTPARK-IISc/SraVaani-0.5-live"
DEV = "cuda" if torch.cuda.is_available() else "cpu"

access_token = "hf_..."
model = AutoModel.from_pretrained(REPO, trust_remote_code=True,token=access_token).to(DEV).eval()

print(model.transcribe(["sample.wav"])[0])

Run all four latency configurations

The lookahead is realized purely by how many feature frames are fed per chunk, so switching latency is instant — no re-download, no reload.

left = model.config.att_context_size[0]                    # fixed left context (70)
for acs in model.config.supported_att_context_sizes:       # [70,13] [70,6] [70,1] [70,0]
    model.set_att_context_size([left, acs[1]])
    text = model.transcribe(["sample.wav"])[0]
    print("lookahead %.2fs -> %s" % (model.config.lookahead_seconds, text))

Live microphone / real-time stream

Push audio as it arrives; get the transcript so far after every block.

s = model.new_stream()
for pcm_block in mic():                          # any block size, float32 @ 16 kHz
    text = model.stream_audio(s, pcm_block)      # transcript so far
text = model.finalize(s)                         # flush the tail

---

Latency configurations

att_context_size	chunk	lookahead	use when
[70, 13]	1.12 s	1040 ms	highest accuracy (default)
[70, 6]	0.56 s	480 ms	balanced — recommended for low latency
[70, 1]	0.16 s	80 ms	very low latency
[70, 0]	0.08 s	0 ms	lowest latency, no lookahead

Set between utterances with model.set_att_context_size([70, R]), or pick by milliseconds with model.set_lookahead_ms(480).

---

Performance

Average WER scores - of schedules languages supported by the model - across the test sets of the following datasets: Vaani, RESPIN, FLEURS, IndicTTS, Commonvoice, Gramvaani, Kathbath, and MUCS.

Language	la=0 (0ms)	la=1 (80ms)	la=6 (480ms)	la=13 (1040ms)	sravaani1.0 wer
Assamese	29.0	28.3	25.2	24.1	19.4
Bengali	27.1	26.2	24.5	23.2	19.8
Gujarati	34.4	33.5	30.3	28.6	19.0
Hindi	22.1	21.3	19.3	18.0	12.4
Kannada	34.3	33.4	31.7	30.8	27.4
Konkani	64.7	63.4	62.0	63.6	54.7
Maithili	34.0	33.0	31.4	30.6	27.5
Malayalam	43.0	41.7	38.4	37.3	27.7
Manipuri	48.2	49.0	46.7	45.2	41.6
Marathi	28.4	27.3	25.2	24.0	19.7
Nepali	45.0	42.9	40.8	39.0	31.3
Odia	36.9	36.4	33.9	32.3	25.7
Punjabi	30.2	28.8	25.8	24.0	20.2
Sanskrit	51.3	49.5	47.3	45.9	36.4
Santali	69.4	68.5	66.8	66.2	57.3
Tamil	35.1	34.6	32.8	32.0	26.2
Telugu	34.6	33.6	31.4	30.3	25.1

Average WER scores - of non scheduled languages supported by the model - across RESPIN and Vaani test sets.

Language	la=0 (0ms)	la=1 (80ms)	la=6 (480ms)	la=13 (1040ms)	sravaani1.0 wer
Angika	40.0	38.9	36.3	35.7	31.4
Ao	78.5	75.9	74.1	72.4	57.7
Awadhi	50.8	50.6	49.2	47.4	43.8
Bajjika	41.9	41.2	39.9	40.1	34.4
Bearybashe	86.7	86.7	87.3	85.7	77.8
Bhatri	78.9	76.1	71.5	69.8	60.3
Bhili	75.3	75.7	75.4	71.4	73.5
Bhojpuri	33.0	32.2	30.6	29.9	27.8
Bundeli	43.4	43.0	40.7	40.1	36.0
Chakhesang	87.1	87.6	85.8	84.5	75.8
Chakma	65.1	63.1	60.5	58.8	51.2
Chhattisgarhi	32.0	31.2	30.1	29.4	27.4
English	31.5	31.0	27.4	24.6	14.8
Garhwali	60.5	60.0	58.6	57.9	53.5
Garo	25.5	22.6	18.6	15.8	9.5
Halbi	61.7	61.2	59.1	57.2	51.6
Haryanvi	59.2	57.6	55.7	54.5	48.1
Idu Mishmi	81.6	81.5	78.8	79.4	64.1
Jaipuri	31.3	30.0	30.5	29.4	28.1
Karbi	86.9	87.9	85.0	82.9	63.3
Khandeshi	66.5	65.6	64.3	62.5	56.6
Khariboli	33.3	32.3	29.9	29.0	27.0
Khorth/Khortha/Khorthkhotta	49.7	48.7	45.6	44.9	41.3
Kokborok	81.2	80.9	78.7	77.8	68.2
Kumaoni	35.1	35.2	33.3	32.1	30.4
Kurumali	14.3	14.0	12.4	12.5	11.3
Lambadi	51.6	51.6	48.4	38.7	25.8
Magadhi/Magahi	43.7	42.8	40.9	40.3	30.4
Malvani	42.1	42.1	39.5	39.0	30.4
Marwari	46.1	45.7	44.6	43.3	39.7
Mizo	42.8	41.3	38.8	35.9	25.3
Nagamese	64.7	63.3	60.6	58.7	50.1
Nagpuri/Sadri	64.6	63.7	62.8	61.6	58.8
Powari	64.1	64.8	60.6	61.3	54.2
Rajasthani	48.6	48.1	47.0	45.6	41.8
Rengma	86.0	83.8	83.6	82.3	71.7
Sambalpuri	74.7	74.6	71.6	70.9	62.7
Shekhawati	22.0	20.9	18.7	18.2	19.1
Sumi	89.0	88.6	86.5	85.8	78.5
Surgujia	48.3	48.4	45.3	43.4	37.5
Surjapuri	70.1	69.7	67.3	67.5	69.1
Tulu	72.3	72.8	71.4	68.4	57.3
Wancho	81.0	80.4	78.5	77.8	67.6

---

Model details

Family	SraVaani / VAANI (ARTPARK, IISc Bangalore)
Architecture	Cache-aware streaming FastConformer encoder + CTC head
Base model	~430M-param hybrid TDT-CTC FastConformer (CTC head kept in this export)
Encoder	17 layers, d_model 1024, 8× subsampling
Features	128-dim log-mel, 16 kHz, 25 ms window / 10 ms hop
Vocabulary	5000 SentencePiece BPE (blank id 5000)
Left / attention cache	70 encoder frames
Export	fp16 TorchScript (~900 MB), device-agnostic (same graph on CPU & GPU)

Pretraining and fine-tuning follow the VAANI recipe (large-scale multilingual pretraining, then fine-tuning across the 63 supported Indian languages). See the paper for full training details.

Files

file	purpose
model-asr-streaming.fp16.ts	exported encoder + CTC head (TorchScript)
configuration_sravaani_streaming.py	AutoConfig custom class
modeling_sravaani_streaming.py	AutoModel custom class (streaming logic)
preproc.pt	mel filterbank + window + featurizer params
tokenizer.model	SentencePiece model
example_streaming.py	runnable demo

---

Requirements

pip install "torch>=2.0" "transformers>=4.30" sentencepiece soundfile

trust_remote_code=True is required — model logic ships as custom code in this repo. No nemo_toolkit is needed for inference.

---

Limitations

• Amount of fine-tuning data varies significantly across languages; low-resource languages have lower accuracy.
• Streaming normalization is causal (statistics over audio heard so far), so very early words in a fresh stream can be slightly less accurate until stats settle.
• The model may code-switch (write borrowed English words in Latin script); against a single-script reference this can inflate WER without reflecting a real recognition error.
• Lower-latency settings ([70,1], [70,0]) trade noticeable accuracy for responsiveness.
• Urdu and Kashmiri are not supported.

---

Feedback

We welcome feedback, suggestions, and bug reports from the community. If you encounter any issues or have questions about the model, please contact us at:

agneedh@artpark.in and vaanicontact@gmail.com

Citation

If you use this model, please cite the VAANI paper:

@misc{pulikodan2026vaanicapturinglanguagelandscape,
      title={VAANI: Capturing the language landscape for an inclusive digital India}, 
      author={Sujith Pulikodan and Abhayjeet Singh and Agneedh Basu and Nihar Desai and Pavan Kumar J and Pranav D Bhat and Raghu Dharmaraju and Ritika Gupta and Sathvik Udupa and Saurabh Kumar and Sumit Sharma and Vaibhav Vishwakarma and Visruth Sanka and Dinesh Tewari and Harsh Dhand and Amrita Kamat and Sukhwinder Singh and Shikhar Vashishth and Partha Talukdar and Raj Acharya and Prasanta Kumar Ghosh},
      year={2026},
      eprint={2603.28714},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2603.28714}, 
}