Streaming speech recognition for Hindi, Tamil, Telugu and Urdu
One small streaming model (icefall Zipformer transducer, 25.8M parameters) for low-end phones, run with sherpa-onnx. The app chooses the language; each language has its own joiner that only allows that language's script.
Results (WER / CER)
Dev set: up to 300 clips per language from the IndicVoices valid split (conversational speech),
greedy search, language given. This is not the official IndicVoices test set, and it was used to
choose the checkpoints. ONNX rows use the first 100 clips per language; sizes count the encoder,
decoder and one joiner.
| Variant | hi | ta | te | ur |
|---|---|---|---|---|
| PyTorch 320ms, all dev clips | 17.8 / 8.7 | 38.4 / 14.8 | 44.7 / 18.6 | 12.0 / 5.9 |
| ONNX fp32 320ms, 99 MB, 100 clips | 15.7 / 7.1 | 34.1 / 12.3 | 45.3 / 17.8 | 10.6 / 4.9 |
| ONNX int8 320ms, 28 MB, 100 clips | 16.0 / 7.2 | 34.2 / 12.2 | 45.3 / 17.7 | 10.6 / 4.9 |
| PyTorch 640ms, all dev clips | 16.7 / 8.0 | 37.0 / 14.0 | 42.3 / 17.5 | 10.7 / 5.3 |
| ONNX fp32 640ms, 99 MB, 100 clips | 15.6 / 6.7 | 33.0 / 11.6 | 42.0 / 16.4 | 9.3 / 4.2 |
| ONNX int8 640ms, 28 MB, 100 clips | 15.4 / 6.9 | 33.4 / 11.6 | 41.9 / 16.3 | 9.0 / 4.2 |
Files
| Path | Contents |
|---|---|
320ms/, 640ms/ |
streaming exports with 320 ms or 640 ms chunks: lower latency or higher accuracy |
encoder.onnx, decoder.onnx |
fp32; .fp16.onnx and .int8.onnx are the fp16 and int8 versions |
joiner-hi, -ta, -te, -ur |
one joiner per language, in the same precisions (use these) |
joiner-auto |
no language restriction; mixes scripts, not recommended |
tokens.txt, bpe.model |
shared BPE, 1,500 pieces with language tokens |
pytorch/avg-epochs-43-45.pt |
averaged weights |
pytorch/epoch-45-full.pt |
epoch 45 with optimizer and scheduler state, to continue training |
icefall Zipformer arguments: --causal 1 --chunk-size 16,32,64,-1 --left-context-frames 64,128,256,-1 --num-encoder-layers 2,2,2,2,2,2 --feedforward-dim 512,768,768,768,768,768 --encoder-dim 192,256,256,256,256,256 --encoder-unmasked-dim 192,192,192,192,192,192 --use-transducer 1 --use-ctc 1
Usage
import re, numpy as np, soundfile as sf, sherpa_onnx
rec = sherpa_onnx.OnlineRecognizer.from_transducer(
tokens="320ms/tokens.txt", encoder="320ms/encoder.int8.onnx",
decoder="320ms/decoder.int8.onnx", joiner="320ms/joiner-hi.int8.onnx",
num_threads=1, sample_rate=16000, feature_dim=80, decoding_method="greedy_search")
wav, _ = sf.read("audio.wav", dtype="float32") # 16 kHz mono
s = rec.create_stream()
s.accept_waveform(16000, wav)
s.accept_waveform(16000, np.zeros(16000, dtype=np.float32))
s.input_finished()
while rec.is_ready(s):
rec.decode_stream(s)
print(re.sub(r"<(hi|ta|te|ur)>", "", rec.get_result(s)).strip())
Training data
About 1,240 hours (hi 328, ta 393, te 266, ur 252) from AI4Bharat's IndicVoices, Kathbath and Shrutilipi, with 0.9/1.0/1.1 speed perturbation; 45 epochs in three stages as data was added. These datasets are released under CC BY 4.0; please also cite them (see their dataset cards).