ModaLeap/sensevoice-small-onnx

SenseVoiceSmall INT8 ONNX for CUDA

This model is used by https://github.com/mercallureAI/local-multimodal-infra .

This repository provides a weight-only INT8 ONNX export of FunAudioLLM/SenseVoiceSmall built for low-latency recognition on NVIDIA GPUs (ONNX Runtime, CUDA execution provider), together with the FunASR pipeline files the runtime loads with it: FSMN-VAD and the CAM++ speaker model.

Original model:

https://huggingface.co/FunAudioLLM/SenseVoiceSmall

Original project:

https://github.com/FunAudioLLM/SenseVoice

This is not an official SenseVoice release.

Description

SenseVoiceSmall is a non-autoregressive CTC speech recognizer for Chinese, Cantonese, English, Japanese and Korean that also tags the language, emotion and audio events. One forward pass transcribes a whole segment.

The common model_quant.onnx (FunASR's quantize=True, ONNX Runtime dynamic quantization) uses uint8 DynamicQuantizeLinear + MatMulInteger pairs that have no CUDA kernels, so under the CUDA provider every layer goes back and forth to the CPU (about 200-350 ms per utterance even on an RTX 4090). This graph instead quantizes the weights of the same 281 linear layers to int8 (MatMulNBits, 8 bits, blocks of 32, symmetric) and keeps the activations in float32: everything runs on the GPU.

• An extra ctc_ids output gives the argmax over the vocabulary, taken on the GPU, so only token ids leave the device.
• The weights sit in model_gpu_int8.onnx.data at 256-byte aligned offsets, listed in model_gpu_int8.shared.json, so a runtime can upload them once and share them between several fixed-length sessions replayed as CUDA graphs.
• Not float16: SenseVoice's residual stream outgrows float16 on loud speech and the CUDA kernels then return inf/NaN logits (empty text for the segment).

The graph was produced from an earlier float16 export widened back to float32, then quantized, with scripts/local/sensevoice_cuda_export.py of local-multimodal-infra (from-fp16).

Files

File	Role	License	Source
asr/model_gpu_int8.onnx + .onnx.data	SenseVoiceSmall, int8 weights, float32 activations. Inputs speech [B, T, 560] (LFR features), speech_lengths, language, textnorm; outputs ctc_logits, encoder_out_lens, ctc_ids	FunASR Model License	FunAudioLLM/SenseVoiceSmall, exported here
asr/model_gpu_int8.shared.json	Byte ranges of the weights in the .data file	FunASR Model License	Generated
asr/am.mvn, asr/config.yaml, asr/tokens.json	Frontend (80-bin FBANK, LFR 7/6, CMVN) and CTC vocabulary	FunASR Model License	SenseVoiceSmall, copied unchanged
vad/model_quant.onnx, vad/am.mvn, vad/config.yaml	FSMN-VAD, splits long audio into speech segments	Apache-2.0	funasr/fsmn-vad-onnx
speaker/campplus_cn_en_common_200k.onnx	CAM++ speaker embeddings (speaker labels for file transcription)	Apache-2.0	3D-Speaker CAM++ via welcomyou/campplus-3dspeaker-200k-onnx
manifest.json	Sources, per-file licenses, size and SHA-256 of every file
config.json	Package summary; the Hub counts downloads by requests for this file. Not read at run time
LICENSE, LICENSES/Apache-2.0.txt, NOTICE	License texts and attribution

Keep the layout: the runtime looks for asr/, vad/ and speaker/ in one model directory.

Use

With local-multimodal-infra's sense_voice_asr adapter, download the repository as a model directory and set the model metadata asr_model_file: model_gpu_int8.onnx (and, for CUDA graph replay, asr_cuda_graph_frames: [64, 128, 256, 352]):

hf download ModaLeap/sensevoice-small-onnx --local-dir workdir/models/sensevoice-small-int8-gpu-onnx

Runtime notes:

• Needs the CUDA execution provider for its speed; it runs on the CPU too, but there the original dynamic int8 graph is the better choice.
• About 1 GB of GPU memory as loaded (RTX 4090, ONNX Runtime 1.31, CUDA graphs for four lengths).
• Audio: 16 kHz mono. language takes SenseVoice's ids (auto 0, zh 3, en 4, yue 7, ja 11, ko 12); textnorm 14 adds punctuation and inverse text normalization.

Testing

On an RTX 4090 with CUDA graphs: about 9 ms for 1.5 s of speech and 16 ms for 21 s. On a 30-minute podcast its output matched 99.8% of the characters of the float32 graph (the dynamic int8 graph: 99.0%). It has been used for live captions and file transcription on Chinese and English. No formal WER benchmark was run.

Source

Item	Revision
FunAudioLLM/SenseVoiceSmall	3847d57b6bdf2dd8875cb1508d2af43d80a16bf7
funasr/fsmn-vad-onnx	f6e9fbb4cefa7397216c763f21307993f147f585
welcomyou/campplus-3dspeaker-200k-onnx	6265ff7af2a104d745b4389026ed9815c6c1c6ff

License

The SenseVoiceSmall files (asr/) are derived from FunAudioLLM/SenseVoiceSmall and distributed under the FunASR Model Open Source License Agreement v1.1 (LICENSE). It requires keeping the source, author information and model name when you use, modify or share the model. The FSMN-VAD and CAM++ files are Apache-2.0 (LICENSES/Apache-2.0.txt). See NOTICE.

All rights to the original models belong to their authors (Alibaba Group Tongyi Lab and the FunASR and 3D-Speaker teams).

Disclaimer

This is an unofficial ONNX export. It is not affiliated with, maintained by, or endorsed by the SenseVoice or FunASR authors.

Citation

@article{an2024funaudiollm,
  title={FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs},
  author={An, Keyu and Chen, Qian and Deng, Chong and Du, Zhihao and Gao, Changfeng and Gao, Zhifu and Gu, Yue and He, Ting and Hu, Hangrui and Hu, Kai and others},
  journal={arXiv preprint arXiv:2407.04407},
  year={2024}
}