Chatterbox-Multilingual-ONNX-WebGPU

The default voice for running Chatterbox Multilingual (ResembleAI/chatterbox) in a web browser with ONNX Runtime Web's WebGPU backend, as used by Quick Inference. The language model, token embeddings and tokenizer are onnx-community's, used unchanged from onnx-community/chatterbox-multilingual-ONNX (revision 452d3f43: language_model_q4, embed_tokens, tokenizer.json); the speech decoder is the int4 one of danyalmalik/Chatterbox-ONNX-WebGPU (onnx-community ships the same decoder for both models).

Contents

voices/default.bin (0.26 MB): what onnx-community's multilingual speech_encoder.onnx (float32) outputs for its default_voice.wav, computed once, so the browser needs neither the speech encoder nor a reference clip. Little-endian float32: N, P, F, then audio_features [N, 1024], audio_tokens [P] (whole numbers), speaker_embeddings [192] and speaker_features [F, 80]. Made by tools/export/chatterbox_voice.py in Quick Inference.

Use

As the ONNX example in onnx-community's model card (the text tagged with its language, e.g. [fr], exaggeration 0.5, greedy speech tokens with repetition penalty 1.2, at most 256 per sentence), with this file's four arrays in place of the speech encoder's outputs. Quick Inference offers the 20 languages whose text needs no extra converter (not Chinese, Japanese or Hebrew), one sentence at a time, after the Python package's punc_norm.

On eight English test sentences, Whisper small.en heard it with a 13.6% word error rate (ONNX Runtime CPU) and 17% in Chrome (WebGPU; greedy decoding drifts a little with the GPU's arithmetic), against 11.4% for the original float32 graphs with the clip itself.

Licence

MIT, as Chatterbox (LICENSE).