alimpfard/breeze-tts-2-int4

Breeze TTS 2 @ int4

A weight-quantized build of Breeze TTS 2, packaged so it loads without ever materialising the bf16 model. Intended for GPUs that cannot hold the original: it runs in just under 3 GB of VRAM.

License. These are derivative weights. They remain governed by the BreezeBlue Research and Non-Commercial License, which is reproduced in full in this repository. Research and non-commercial use only.

What was quantized

Component	Precision	Notes
backbone (MLP + attention)	int4, group 128	runs once per audio frame
depth decoder (MLP + attention)	int4, group 128	runs 16x per frame
text encoder	int4, group 128	once per request at prefill
audio codec / embeddings	bf16	offload these

Measured per-layer relative error is ~0.10.

Measured performance on constrained devices

Steady-state realtime factor, conditioning chained across turns:

GPU	Config	RTF	Peak VRAM
DGX Spark (GB10, 273 GB/s)	int4, decode-only CUDA graphs	1.80x	-
RTX 3080 Laptop (sm_86, 448 GB/s)	int4, decode-only CUDA graphs	1.44x	2.91 GB
RTX 3080 Laptop (sm_86)	int4, no CUDA graphs	0.26x	~2.2 GB

Usage

Requires the breeze-tts inference code (Apache 2.0) with the quantized-checkpoint loader.

You may use alimpfard/breeze-tts-2 for a known-working implementation.

from pathlib import Path
from models.quantized_checkpoint import load_quantized_runtime

tokenizer, model, audio_tokenizer, stats = load_quantized_runtime(
    Path("breeze-tts-2-int4"),
    device="cuda",
    offload_embeddings=True,   # keeps ~1.7 GB of lookup tables in host memory
)

offload_embeddings=True places the text-side embedding tables in host memory as they load, so they never occupy VRAM. They are gathered once per request (~2 MB moved).

Note CPU offload and CUDA graphs cannot both apply to the same module; a host round trip is not capturable. This only affects prefill and text-encoder graphs, not the per-frame decode path that provides the speedup.

Requirements

• Minimum (tested) CUDA GPU with compute capability 8.0+ (tinygemm int4). Verified on sm_86 (Ampere) and sm_121 (Blackwell).
• PyTorch 2.11+ (verified on 2.11 and 2.12).
• ~3 GB VRAM with embedding offload and decode-only graphs.

Samples

• sample_dialogue.wav: five conversational turns with cross-request voice conditioning.
• sample_long.wav: 85s of narrative prose and dialogue, generated at max_seq_len=512 to match a constrained deployment.

Both use a synthetic reference voice, not a recorded speaker.

Attribution

Base model: Breeze TTS 2 by RESONIA, INC. dba BreezeBlue. Upstream: https://github.com/breezeblue-ai/breeze-tts

This repository redistributes the base model's tokenizer and audio tokenizer unchanged so the checkpoint is self-contained. All of it remains subject to MODEL_LICENSE.