androiddrew/kitten-tts-mini-0.8-fp16

Kitten TTS Mini 0.8 (fp16 for GPU) 😻

This is KittenML/kitten-tts-mini-0.8, the same model with the same weights and voices, converted from int8 to fp16 so that it runs fast on a GPU, at half the size of the fp32 conversion. It is not retrained or fine-tuned. All credit for the model goes to KittenML. See their card for what it is and how it was made.

The published mini model is quantized to int8 with ONNX Runtime's dynamic quantization. With the CUDA provider, ONNX Runtime keeps those quantized layers on the CPU, so on a GPU the int8 model barely speeds up. Here every quantized layer is turned back into an ordinary float layer, so the whole network runs on the GPU.

Mini 0.8 on an RTX 4090	Real-time factor, p50 / p95	Time to first audio, p50 / p95
int8 (original)	0.392 / 0.522	1.43 s / 2.61 s
fp32 (androiddrew/kitten-tts-mini-0.8-fp32)	0.051 / 0.108	0.22 s / 0.28 s
fp16 (this repo)	0.068 / 0.146	0.30 s / 0.40 s

Real-time factor is synthesis time per second of audio, so lower is faster. The numbers are measured with gokittentts bench: ONNX Runtime 1.29.1 with CUDA 12, 19 chat-style requests × 5 runs, voice Bruno. This repo is 147 MB, against 288 MB for fp32. But on this GPU, fp32 is faster: ONNX Runtime's fp16 convolutions take about twice as long, and NHWC layout and exhaustive cuDNN search didn't change that. Use fp16 when download size or GPU memory matters more than the last bit of speed, and measure on your own GPU.

Usage

It's a drop-in replacement: the same config.json format, inputs, outputs and voices as the original.

With KittenTTS on a GPU. backend= is on KittenTTS main but not in the 0.8 or 0.8.1 releases:

pip install "git+https://github.com/KittenML/KittenTTS@main" onnxruntime-gpu

from kittentts import KittenTTS
m = KittenTTS("androiddrew/kitten-tts-mini-0.8-fp16", backend="cuda")
audio = m.generate("This high quality TTS model runs even faster on a GPU", voice="Jasper")
# available_voices : ['Bella', 'Jasper', 'Luna', 'Bruno', 'Rosie', 'Hugo', 'Kiki', 'Leo']
import soundfile as sf
sf.write("output.wav", audio, 24000)

With gokittentts, as a custom model in config.yaml:

models:
  kitten-tts-mini-0.8-fp16:
    device: cuda
    repo: androiddrew/kitten-tts-mini-0.8-fp16
    revision: main      # better: pin a commit

fp16 is meant for GPUs. On a CPU, use the fp32 repo or the original.

What changed

convert.py rewrites the graph with the onnx package. Every quantized layer is one of these patterns:

Original (int8)	Count	Replaced with
DynamicQuantizeLinear → MatMulInteger → Cast → Mul	135	MatMul
DynamicQuantizeLinear → ConvInteger → Cast → Mul	74	Conv
DynamicQuantizeLSTM (com.microsoft)	6	LSTM

Each weight is dequantized as W = (W_int8 − zero_point) × scale, with the scales and zero points stored in the original graph. The LSTM weights are transposed to the standard ONNX layout. The quantizers and int8 weights are then removed. Everything else is unchanged, including the inputs and outputs (input_ids, style and speed in; waveform and duration out) and the random noise ops.

The fp32 graph is then converted to fp16 with ONNX Runtime's float16 tool:

• Inputs and outputs stay float32 and int64 (keep_io_types), so callers don't change.
• Kept in fp32: the ops on the tool's default blocklist. In this graph, those are the six Resize ops, the RandomUniformLike noise op and a CumSum.
• Cast cleanup: duplicate Cast nodes that the tool emits are removed.

This can't recover the original float model: the int8 rounding of the weights is baked in, and fp16 adds its own rounding. Expect output that sounds like the int8 model, not better.

Quality

Checked so far:

• The model loads in ONNX Runtime 1.29.1 on the CPU and on CUDA.
• It synthesizes every request in the 19-request benchmark corpus on CUDA.
• The model contract is unchanged; gokittentts asserts it on load.

A side-by-side comparison with the int8 original, with seeded noise, matching durations and spectral distance, has not been run yet. validate.py does it and writes WAV pairs for listening:

uv run validate.py kitten_tts_mini_v0_8.onnx kitten_tts_mini_v0_8_fp32.onnx kitten_tts_mini_v0_8_fp16.onnx voices.npz

Until then, treat the audio as unverified against the original, and report anything that sounds off.

Reproduce

# from the original, pinned to revision c02725660cea441db4c383af69f1f26f5cd00947
# kitten_tts_mini_v0_8.onnx sha256 0f5bbae4fc4800c98dbc544a87ecfa79510de2fb8222db30d12e5bfe9177df91
uv run convert.py kitten_tts_mini_v0_8.onnx build

uv run installs the pinned dependencies listed at the top of the script: onnx 1.19.0, numpy and onnxruntime 1.29.0. SHA256SUMS lists this repo's files.

License and attribution

This is Apache-2.0, like the original (LICENSE). The model, weights and voices (voices.npz, unchanged) are KittenML's. Kitten TTS builds on the StyleTTS 2 architecture. The ONNX file in this repo is a modified version of KittenML's kitten_tts_mini_v0_8.onnx, changed as described above. config.json differs from the original only in model_file and name.