NeuTTS-Air-ONNX-WebGPU

Neuphonic's NeuTTS Air text-to-speech model, with the decoder of its NeuCodec audio codec, as ONNX graphs for running in a web browser with ONNX Runtime Web's WebGPU backend, as used by Quick Inference. A Qwen2 0.5B language model writes NeuCodec tokens (50 a second) from phonemes in the voice of a reference recording; the decoder turns them into 24 kHz audio.

Changes from the original

• The language model: converted with ONNX Runtime GenAI's model builder (-p int4 -e webgpu, k-quant, block size 32, use_webgpu_fp32=true). Its linear layers and its embedding table (shared with the tied output layer) are 4-bit; activations and the KV cache are float32, because in float16 the model overflows.
• The codec decoder: Neuphonic's own ONNX export (neucodec-onnx-decoder), quantized without calibration (round to nearest, symmetric, blocks of 32): matrix multiplications to 4-bit MatMulNBits, convolution weights to int4 DequantizeLinear. Its inverse STFT's ConvTranspose (kernel 1920, stride 480) is rewritten as the same sum done as a MatMul and an overlap-add of four padded blocks (tools/export/neucodec_overlap_add.py in Quick Inference; identical on the CPU), because ONNX Runtime Web's WebGPU ConvTranspose computes this one wrongly.
• The default voice is included as NeuCodec codes, so no encoder is needed.
• No watermark is added (the original pipeline adds a Perth watermark).

These are modified files: their outputs are close to, but not identical to, the original model's. On eight test sentences, Whisper small.en heard it with an 11.4% word error rate with eSpeak's phonemes, the same as the original, and 6.8% with Quick Inference's own phonemizer in place of eSpeak (10.2% in Chrome with that phonemizer); it samples, so single runs vary.

Files

• onnx/model.onnx, onnx/model.onnx.data, onnx/genai_config.json: the language model. input_ids int64 [1, S], attention_mask int64 [1, P + S], past_key_values.{0..23}.{key,value} float32 [1, 2, P, 64] → logits [1, S, 217652], present.*.
• onnx/codec_decoder.onnx: codes int32 [1, 1, F] → audio float32 [1, 1, 480 F − 480], all at once.
• tokenizer.json, tokenizer_config.json: the original's (Qwen's vocabulary plus <|speech_0|> … <|speech_65535|>).
• voice_dave.json: the default voice: codes (372), ref_text, and its transcript's phonemes from eSpeak (ref_phonemes) and from Quick Inference's phonemizer (ref_phonemes_g2p).

Generation

1. Phonemize the text (eSpeak-style IPA, en-us, with stress and punctuation, words separated by single spaces).
2. Prompt: user: Convert the text to speech:<|TEXT_PROMPT_START|> + the voice's phonemes + + the text's phonemes + <|TEXT_PROMPT_END|>\nassistant:<|SPEECH_GENERATION_START|> + the voice's codes as <|speech_N|> tokens (id 151671 + N).
3. Sample with temperature 1.0 and top-k 50 until <|SPEECH_GENERATION_END|> (not allowed in the first 50 tokens), or 2048 positions in all.
4. Decode the new <|speech_N|> tokens' N with codec_decoder.

Voice

dave is from neuphonic/neutts at commit 3e9415d (samples/dave.pt, samples/dave.txt), released there under Apache 2.0.

Licence

Apache 2.0 (LICENSE), as NeuTTS Air, NeuCodec and the voice. NeuTTS Air is built on a Qwen 0.5B language model (Apache 2.0).