ajh-code/chatterbox-flash-t3-w4a16-nvfp4

Chatterbox-Flash T3 W4A16 NVFP4

This is an unofficial community NVFP4 derivative of ResembleAI/chatterbox-flash. It is not affiliated with or endorsed by Resemble AI or NVIDIA.

The package replaces all 120 Chatterbox-Flash T3 transformer projection operators with source-free NVFP4 weights. Weights use native FP4 (E2M1) with E4M3 block scales; activations remain BF16, so the precise description is T3 all-projection W4A16 NVFP4. Embeddings, norms, conditioning layers, speech head, S3Gen, and the voice encoder remain at their upstream precision. This is not a whole-pipeline 4-bit model.

The canonical artifact does not contain the original BF16 T3 projection weights. Its custom loader constructs those parameters on the meta device, prepares the canonical FP4 codes/scales directly for FlashInfer, and never calls nvfp4_quantize at load time. Engine rebuilds reuse the model-attached projection provider.

What this release actually improves

The primary benefit is much lower storage and VRAM with a modest speed gain, not a claim that NVFP4 makes the complete TTS pipeline dramatically faster. S3Gen and several small T3 operations remain high precision, and some irregular prefix projections are slower in FP4 than BF16.

On the tested RTX 5060 Ti, the recommended D16 quality profile delivered:

• 83.97% smaller T3 checkpoint: 341 MB instead of 2.13 GB.
• 55.88% less complete model-asset storage/download after adding the unchanged S3Gen, voice encoder, and tokenizer: 1.412 GB instead of 3.200 GB.
• 33.2% lower allocated VRAM immediately after full pipeline load: 1.444 GB instead of 2.162 GB.
• 18.82% lower maximum allocated VRAM in the natural eight-text suite: 5.215 GB instead of 6.424 GB. Fixed-work peak allocation was 34.09% lower.
• 1.0811x aggregate-RTF speedup (7.50% lower duration-normalized latency) and 1.1131x valid-token T3 throughput. The fixed-work T3 speedup was 1.1260x; fixed whole-pipeline speedup was 1.0631x.

The D24 fast profile reaches 25.13% lower RTF than the D16 BF16 baseline, but that is a combined block-size-plus-NVFP4 result. Against matched D24 BF16, NVFP4 alone provides only a 1.0309x RTF speedup. This distinction matters when deciding whether the model is useful for your deployment.

How quality was evaluated

A Gemma Audio evaluator listened to eight English test utterances generated from one zero-shot reference voice. It transcribed each output, scored intelligibility, pronunciation, voice similarity, naturalness, and artifacts, and flagged hard failures. BF16 and NVFP4 were also compared blindly in both A/B orders to expose position bias.

• D16 BF16/NVFP4 means were 94.125/94.625. Both produced exact transcripts for 8/8 texts with zero hard failures. After reversing A/B order, the per-case result resolved to seven ties and one consistent BF16 preference.
• D24 BF16/NVFP4 means were 95.0/93.125. Both produced exact transcripts for 8/8 texts with zero hard failures. Order control resolved to six ties, one BF16 preference, and one NVFP4 preference.

This is encouraging automated evidence, not human MOS, a standard Seed-TTS benchmark, proof of perceptual identity, or a multi-speaker evaluation. The audio was intentionally not published because consent for public redistribution of the reference voice was not established.

Hardware and software

The accepted release configuration is intentionally narrow:

• Linux x86-64 and Python 3.12
• NVIDIA RTX 50-series SM120/SM121 (tested on RTX 5060 Ti)
• PyTorch 2.7.1+cu128 and CUDA 12.8 wheels
• FlashInfer 0.6.14 with nvidia-cutlass-dsl 4.5.2
• BF16 activations, FlashInfer CUDA graphs, user batch size 1

There is no CPU, MLX, Torch-SDPA, older-GPU, or non-NVFP4 fallback because the BF16 projection sources are deliberately absent. RTX 5080 and SM121 are allowed by the loader but have not yet been benchmarked. This is custom code, not a Transformers AutoModel checkpoint or a standard Hub quantization format, so the hosted inference widget is disabled.

nvidia-cutlass-dsl is installed under NVIDIA's separate software license. Review THIRD_PARTY_NOTICES.md before installation.

Install

Clone the model repository, then use the included lock. Plain pip install does not honor the upstream Torch override or the official cu128 wheel source.

git clone https://huggingface.co/ajh-code/chatterbox-flash-t3-w4a16-nvfp4
cd chatterbox-flash-t3-w4a16-nvfp4

uv sync \
  --project runtime/chatterbox-flash \
  --locked \
  --extra flashinfer \
  --python 3.12

The lock resolves the exact validated critical stack. A dry verification is:

runtime/chatterbox-flash/.venv/bin/python - <<'PY'
import importlib.metadata as md
import flashinfer
import torch

assert torch.__version__ == "2.7.1+cu128"
assert torch.version.cuda == "12.8"
assert md.version("flashinfer-python") == "0.6.14"
assert md.version("nvidia-cutlass-dsl") == "4.5.2"
assert hasattr(flashinfer, "prepare_bf16_fp4_weights")
assert hasattr(flashinfer, "mm_bf16_fp4")
assert torch.cuda.get_device_capability() in {(12, 0), (12, 1)}
print(torch.cuda.get_device_name(), "is ready")
PY

Zero-shot voice cloning

Use only reference audio that you have permission to clone.

runtime/chatterbox-flash/.venv/bin/python example.py \
  --prompt-wav your-reference.wav \
  --text "This is Chatterbox Flash running native NVFP4 projections." \
  --output output.wav \
  --block-size 16

The example downloads only the unchanged S3Gen, voice-encoder, and tokenizer assets from the immutable base revision 4385507288b8197e6dab8b4e6b1603328d549d9d. It verifies their sizes and SHA-256 fingerprints from nvfp4_config.json. It never downloads the original 2.13 GB t3_flash.safetensors.

The first decode for a new GPU/exact block shape invokes FlashInfer setup and tuning. On the RTX 5060 Ti, a fresh process with the selected D16 tactic already cached took 10.57 seconds for the first 1.28-second utterance versus 0.154 seconds for the same-process warm repeat. A completely missing tactic has taken roughly 25 seconds in development. Autotune results are GPU-specific, so this repository does not ship a universal cache.

Profiles

D=16 is the quality-default profile. D=24 is an explicitly optional fast profile: it is materially faster overall but had a slightly lower automated mean score and less consistent pairwise preference. D=32 was tested and rejected after token-cap failures and an OOM in the natural suite; its cached prior remains serialized for research provenance, but the release loader refuses it.

Profile	Intended use	NVFP4 aggregate RTF	NVFP4 T3 tok/s	Automated mean
D16	Quality default	0.08725	365.4	94.625
D24	Optional fast	0.07062	483.3	93.125

RTX 5060 Ti results

The natural test is an eight-text, one-reference suite. Pipeline and T3 times are sums of per-case medians; aggregate RTF divides summed pipeline time by summed generated-audio duration. Memory is the maximum per-case median PyTorch CUDA peak, not total nvidia-smi process memory.

Profile	Comparison	RTF speedup	T3 throughput	Peak allocated reduction
D16	W4A16 vs matched BF16	1.0811x	1.1131x	18.82% / 1.209 GB
D24	W4A16 vs matched BF16	1.0309x	1.0450x	21.57% / 1.410 GB

A fixed capped-work control removes sampled-length attribution. D16 T3 is 1.1260x faster and D24 T3 is 1.0656x faster; each reduces fixed-work peak allocated memory by about 34%. D16 fixed whole-pipeline speedup is 1.0631x.

Do not quote the larger raw natural-suite wall-time ratios as pure NVFP4 speedups: the BF16 and W4A16 models sampled different token counts. The D24-versus-D16 speed difference is primarily the larger block size, not FP4 alone. Complete numeric records and caveats are in benchmark_results.json.

Automated quality was close but is not proof of perceptual identity:

• D16 BF16/NVFP4 means were 94.125/94.625; both transcribed 8/8 exactly with zero hard failures. Order-controlled pairwise scoring resolved to seven ties and one BF16 win.
• D24 BF16/NVFP4 means were 95.0/93.125; both transcribed 8/8 exactly with zero hard failures. Order control resolved to six ties, one BF16 win, and one NVFP4 win.

This was not human MOS, a standard Seed-TTS evaluation, or a multi-speaker release study.

Artifact and load behavior

• t3_nvfp4.safetensors: 341,439,040 bytes, 447 tensors, SHA-256 c5ea2d1d1441f657b383d8dad54bbae3778447d675822fb15918cb99b3587f1e
• 360 canonical NVFP4 tensors cover 30 layers × 4 fused projection kinds × FP4 codes/E4M3 scales/FP32 alpha.
• 82 non-projection T3 tensors remain BF16; three fixed priors and two rotary buffers support source-free construction.
• Original T3 checkpoint: 2,129,657,904 bytes. The derivative T3 file is 83.97% smaller.
• Derivative T3 plus the three unchanged downloaded base assets totals 1,412,035,402 bytes versus 3,200,254,266 bytes upstream: 55.88% less model asset storage/download, excluding small runtime files and cache.

In the final source-free release smoke, fresh-process pipeline load with base hash verification and warm JIT state took 2.675 seconds and settled at 1.444 GB PyTorch allocated / 1.478 GB reserved before the first generation. It prepared 120 projections into 283,116,000 bytes, without materializing BF16 projection weights or calling the quantizer. The short generation peaked at 2.548 GB allocated. These are one-machine diagnostics; first-generation setup is reported separately above.

The canonical D16 loader regression reproduced the accepted stripped runtime bit-for-bit: token history, waveform tensor SHA-256, and saved WAV SHA-256 all matched. A longer request also rebuilt the engine successfully while retaining the same source-free projection provider.

Limitations and responsible use

• Only RTX 5060 Ti / SM120, batch one, and the documented settings received the complete technical and small automated-audio gate.
• D16 is the only quality-default profile. D24 is optional; D32 is rejected.
• S3Gen and voice-encoder execution are not quantized by this artifact.
• Prefix projections can remain slower than BF16 at irregular shapes; the decode block projections provide the measured NVFP4 gain.
• Streaming, multi-request throughput, RTX 5080 tuning, and human multi-voice listening remain future work.
• Generated length can change after quantization, affecting latency, memory, pauses, and termination even when transcripts are correct.
• Voice cloning can enable impersonation or deception. Obtain the speaker's consent, disclose synthetic audio where appropriate, and comply with local law and platform policies.

No evaluation audio is included because the reference clip's redistribution and voice-consent status was not established for a public model repository.

Attribution and citation

Chatterbox-Flash model weights and vendored runtime code are Copyright (c) 2026 Resemble AI and licensed under MIT. This runtime is based on source commit 74e05baa8ce574bf2cc571702391a21f1b0d48c5; the derivative modifies the FlashInfer projection path and adds the canonical NVFP4 loader. See THIRD_PARTY_NOTICES.md.

@misc{seo2026chatterboxflashpriorcalibratedblockdiffusion,
  title={Chatterbox-Flash: Prior-Calibrated Block Diffusion for Streaming Zero-Shot TTS},
  author={Deokjin Seo and Gangin Park and Kihyun Nam},
  year={2026},
  eprint={2605.30748},
  archivePrefix={arXiv},
  primaryClass={cs.SD},
  url={https://arxiv.org/abs/2605.30748}
}

Repository files

• t3_nvfp4.safetensors: canonical source-free checkpoint
• nvfp4_config.json: complete schema, fingerprints, revisions, and profiles
• runtime/chatterbox-flash/: pinned installable derivative runtime and lock
• example.py: tested local-bundle zero-shot generation entry point
• benchmark_results.json: machine-readable release measurements
• SHA256SUMS: release integrity manifest