androiddrew/kitten-tts-mini-0.8-fp32

Kitten TTS Mini 0.8 (fp32 for GPU) 😻

This is KittenML/kitten-tts-mini-0.8, the same model with the same weights and voices, converted from int8 to fp32 so that it runs fast on a GPU. It is not retrained or fine-tuned. All credit for the model goes to KittenML. See their card for what it is and how it was made.

The published mini model is quantized to int8 with ONNX Runtime's dynamic quantization. With the CUDA provider, ONNX Runtime keeps those quantized layers on the CPU, so on a GPU the int8 model barely speeds up. Here every quantized layer is turned back into an ordinary float layer, so the whole network runs on the GPU.

Mini 0.8 on an RTX 4090	Real-time factor, p50 / p95	Time to first audio, p50 / p95
int8 (original)	0.392 / 0.522	1.43 s / 2.61 s
fp32 (this repo)	0.051 / 0.108	0.22 s / 0.28 s
fp16 (androiddrew/kitten-tts-mini-0.8-fp16)	0.068 / 0.146	0.30 s / 0.40 s

Real-time factor is synthesis time per second of audio, so lower is faster. The numbers are measured with gokittentts bench: ONNX Runtime 1.29.1 with CUDA 12, 19 chat-style requests × 5 runs, voice Bruno. On this GPU, fp32 is faster than fp16. Use the fp16 repo for half the download and memory.

Usage

It's a drop-in replacement: the same config.json format, inputs, outputs and voices as the original.

With KittenTTS on a GPU. backend= is on KittenTTS main but not in the 0.8 or 0.8.1 releases:

pip install "git+https://github.com/KittenML/KittenTTS@main" onnxruntime-gpu

from kittentts import KittenTTS
m = KittenTTS("androiddrew/kitten-tts-mini-0.8-fp32", backend="cuda")
audio = m.generate("This high quality TTS model runs even faster on a GPU", voice="Jasper")
# available_voices : ['Bella', 'Jasper', 'Luna', 'Bruno', 'Rosie', 'Hugo', 'Kiki', 'Leo']
import soundfile as sf
sf.write("output.wav", audio, 24000)

With gokittentts, as a custom model in config.yaml:

models:
  kitten-tts-mini-0.8-fp32:
    device: cuda
    repo: androiddrew/kitten-tts-mini-0.8-fp32
    revision: main      # better: pin a commit

It also runs on the CPU, in any ONNX Runtime that runs the original.

What changed

convert.py rewrites the graph with the onnx package. Every quantized layer is one of these patterns:

Original (int8)	Count	Replaced with
DynamicQuantizeLinear → MatMulInteger → Cast → Mul	135	MatMul
DynamicQuantizeLinear → ConvInteger → Cast → Mul	74	Conv
DynamicQuantizeLSTM (com.microsoft)	6	LSTM

Each weight is dequantized as W = (W_int8 − zero_point) × scale, with the scales and zero points stored in the original graph. The LSTM weights are transposed to the standard ONNX layout. The quantizers and int8 weights are then removed. Everything else is unchanged, including the inputs and outputs (input_ids, style and speed in; waveform and duration out) and the random noise ops.

This can't recover the original float model: the int8 rounding of the weights is baked in. It only removes the quantization of the activations at run time. Expect output that sounds like the int8 model, not better.

Quality

Checked so far:

• The model loads in ONNX Runtime 1.29.1 on the CPU and on CUDA.
• It synthesizes every request in the 19-request benchmark corpus on CUDA.
• The model contract is unchanged; gokittentts asserts it on load.

A side-by-side comparison with the int8 original, with seeded noise, matching durations and spectral distance, has not been run yet. validate.py does it and writes WAV pairs for listening:

uv run validate.py kitten_tts_mini_v0_8.onnx kitten_tts_mini_v0_8_fp32.onnx kitten_tts_mini_v0_8_fp16.onnx voices.npz

Until then, treat the audio as unverified against the original, and report anything that sounds off.

Reproduce

# from the original, pinned to revision c02725660cea441db4c383af69f1f26f5cd00947
# kitten_tts_mini_v0_8.onnx sha256 0f5bbae4fc4800c98dbc544a87ecfa79510de2fb8222db30d12e5bfe9177df91
uv run convert.py kitten_tts_mini_v0_8.onnx build

uv run installs the pinned dependencies listed at the top of the script: onnx 1.19.0, numpy and onnxruntime 1.29.0. SHA256SUMS lists this repo's files.

License and attribution

This is Apache-2.0, like the original (LICENSE). The model, weights and voices (voices.npz, unchanged) are KittenML's. Kitten TTS builds on the StyleTTS 2 architecture. The ONNX file in this repo is a modified version of KittenML's kitten_tts_mini_v0_8.onnx, changed as described above. config.json differs from the original only in model_file and name.