aufklarer/FireRedTTS3-Instruct-MLX-fp32

FireRedTTS3 Instruct — MLX Float32

Native Apple Silicon voice design with speech-swift. The model turns a free-form voice instruction into a text plan of voice attributes, then synthesizes the supplied English or Chinese script without a reference recording. The Swift API returns the generated voice plan alongside mono 24 kHz audio.

This export preserves the selected upstream Float32 tensor payloads and their original names. It includes the Instruct generation model and the RedAE decoder. The native Swift runtime supports reference-free voice design. Voice cloning, audio references and speech editing are outside this bundle's supported tasks.

Model

Property	Value
Source checkpoint	FireRedTeam/FireRedTTS3, Instruct variant
Retained size	Approximately 2.36 billion stored Float32 values; 877 tensors including the decoder window buffer
Quantization	None; original Float32 tensor values
Format	MLX-compatible Safetensors, native MLX Swift runtime
Weight files	9,440,678,972 bytes, approximately 9.44 GB
Complete runtime assets	9,455,116,268 bytes, approximately 9.46 GB
Output	Mono Float32 PCM, 24,000 Hz
Supported languages	English and Chinese
Prompt limit	4,096 tokenizer positions in the native runtime
Voice planning limit	200 generated tokens by default
Audio limit	400 acoustic patches by default, up to 64 seconds; limit exhaustion throws
Waveform delivery	Buffered complete waveform
Validated hardware	Apple M5 Pro, 48 GiB unified memory

The model uses a 28-layer Qwen3 backbone, an 8-layer patch encoder, an 11-layer flow transformer and an 18-layer RedAE decoder. Four generated latents form one acoustic patch; the decoder produces 3,840 samples, or 160 ms, per patch. The unused RedAE encoder and decoder token embedding are omitted. Source tensor layouts are retained, including convolution axes.

The official Python tokenizer adds speech and language tokens at load time. This bundle persists that exact ordered registry: 255 additions, with 151,924 total vocabulary entries. The control IDs include <|sosp|> at 151669, <|sot|> at 151676 and <|eot|> at 151677. The export manifest records the source and prepared tokenizer hashes.

The official CUDA wrapper autocasts its backbone to BF16. The initial native runtime computes the backbone in Float32. Preserved source values therefore do not imply identical sampled speech across frameworks or hardware.

Measured performance

Local validation on 29 October 2026 used an Apple M5 Pro with 48 GiB unified memory, a release build, compiled MLX Metal shaders and MLX_ENABLE_TF32=0. Each language used one fresh process, one first request and three warm repeats of one short neutral script. Filesystem caches were warm. Checksums ran before the timed load using bounded temporary buffers.

Both scripts used seed 2, ten flow steps, guidance 1.2 and a 150-patch audio limit. English used “The quick brown fox jumps over the lazy dog.” with the instruction “A clear, calm English narrator.” Chinese used “你好，欢迎使用语音合成。今天天气很好，我们一起去散步吧。” with the instruction “一位声音清晰、语气亲切的普通话女声。”

Script	Audio duration	Median warm complete PCM	Median warm real-time factor	Peak process physical footprint
English	2.72 s	4.50 s	1.65	10.82 GB
Chinese	5.28 s	11.22 s	2.13	11.70 GB

Real-time factor is generation time divided by audio duration; lower is faster. The English value of 1.65 means 1.65 seconds of generation per second of audio. Complete-PCM latency includes voice planning, acoustic generation and waveform decoding. It is the first available audio for this buffered implementation. Model loading took 0.60 seconds for English and 0.57 seconds for Chinese and is excluded from request timings.

Physical memory is the kernel process high-water measurement across loading and all four requests, expressed in decimal GB. It includes more than weight memory. MLX active weight memory returned to approximately zero after explicit unload; allocator buffers and operating-system residency are separate values.

Validation passed:

• Five numerical reference tests covering the backbone and cached steps, patch/flow computation, decoder sliding attention, inverse FFT boundaries and nonfinite planning safeguards.
• Two real-bundle tests covering English/Chinese synthesis, voice planning, cancellation recovery and explicit unload.
• One CLI test with independent Parakeet transcription matching every word of the five-word script, “The weather is beautiful today.”
• Zero normalized transcription error on the fixed English and Chinese measurement scripts. Four seeded WAV repetitions per language were identical.

These measurements cover one short script per language. They do not establish broad language coverage or instruction-following quality. No InstructTTSEval score is claimed for this export.

The integration audit found all 59 existing inference files unchanged and all 16 existing-model before/after seeded WAV pairs identical. Existing-model speed comparisons remain inconclusive after all 24 planned timing runs completed. Some latency results exceeded the comparison thresholds. This card makes no claim of no performance regression.

Usage

Build speech-swift from main on Apple Silicon before running the examples below. These engines are not yet included in a tagged release or the Homebrew package.

git clone --branch main https://github.com/soniqo/speech-swift.git
cd speech-swift
make build
export PATH="$PWD/.build/release:$PATH"

make build compiles the CLI and the MLX Metal shaders required for inference.

Add the FireRedTTS3TTS SwiftPM product from speech-swift and load the converted bundle directory. Use the main branch for the SwiftPM dependency. Inference runs locally in Swift.

import AudioCommon
import FireRedTTS3TTS
import Foundation

let directory = URL(fileURLWithPath: "./fireredtts3-instruct-mlx")
let model = try await FireRedTTS3Model.fromBundle(directory)
let result = try await model.generateWithPlan(
    text: "The weather is beautiful today.",
    instruction: "A calm, clear female voice, speaking warmly.")

print(result.voiceDescription)
try WAVWriter.write(
    samples: result.samples,
    sampleRate: model.sampleRate,
    to: URL(fileURLWithPath: "speech.wav"))
model.unload()

CLI:

MLX_ENABLE_TF32=0 speech speak "The weather is beautiful today." \
  --engine fireredtts3 \
  --fireredtts3-bundle-dir ./fireredtts3-instruct-mlx \
  --instruct "A calm, clear female voice, speaking warmly." \
  --seed 2 -o speech.wav

Use --fireredtts3-flow-steps to change the default ten flow steps and --fireredtts3-max-patches to change the audio limit. The CLI returns complete audio and rejects --stream. The API's generateStream emits one buffered final chunk. One model instance permits one generation at a time.

Text and instruction are preserved exactly. The runtime does not normalize, split or rewrite the script. Use English or Chinese and avoid reserved control tokens in user text. Cancellation and planning/audio limit errors propagate instead of returning a truncated completed utterance.

Python can verify the two weight files without loading a model:

import hashlib
import json
from pathlib import Path

bundle = Path("./fireredtts3-instruct-mlx")
manifest = json.loads((bundle / "export_manifest.json").read_text())
for item in manifest["files"]:
    digest = hashlib.sha256()
    with (bundle / item["name"]).open("rb") as stream:
        for chunk in iter(lambda: stream.read(8 * 1024 * 1024), b""):
            digest.update(chunk)
    assert digest.hexdigest() == item["sha256"], item["name"]
print("Weight checksums verified")

Files and integrity

The core file contains 673 tensors. The decoder file contains 204 tensors. Both store Float32 values. The manifest includes every selected source tensor key, source revisions, payload checksums and prepared-tokenizer hashes.

File	Bytes	Purpose
model.safetensors	8,475,259,796	Instruct backbone, patch encoder, flow and stop heads
decoder.safetensors	965,419,176	RedAE decoder and stored periodic window
config.json	1,436	Native architecture and supported task
tokenizer.json	11,472,162	Text tokenizer with persisted speech-token registry
tokenizer_config.json	63,101	Tokenizer configuration and special-token IDs
vocab.json	2,776,833	Original vocabulary asset
LICENSE	11,357	Upstream Apache-2.0 license
export_manifest.json	67,273	Pinned source and tensor/file provenance
model.safetensors.verified.json	33,551	Core conversion checksum receipt
decoder.safetensors.verified.json	11,583	Decoder conversion checksum receipt

SHA-256 inventory:

0424eae12ddde6071d7696ee1104d7863e8dbf878d9b4d81c025f8efd867ef42  model.safetensors
6f1eaf97c761f5a708421086e949ee205dd075489013b453a38253a02a3f348c  decoder.safetensors
8cb5fd1c48201d821ab3be417e4ee7b40d7840528a96682708ecb454e39f7413  config.json
4180d5fdbcc4f78a07afad0ac00f64e456501d93cf6535cf0df89b58aab348f5  tokenizer.json
09972575f85999d29dd11237e2a221a6bcf929532598a4025174561cc3d766ad  tokenizer_config.json
ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910  vocab.json
c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4  LICENSE
6488d56326789de6866556ba70148b1d3d927f5f66476095c91af4d860971427  export_manifest.json
806c665c760f33091833fd92108f642352c8a62fc10f5eb9055b19f94318399a  model.safetensors.verified.json
240ad9f71782d5f29b8e3ef579646d941c9c9a0cac5708c3122b13e76e3ddab0  decoder.safetensors.verified.json

Sources and license

The official checkpoint card declares Apache-2.0. The pinned source license is Apache-2.0 and is preserved in this bundle.

Source	Pinned revision
FireRedTeam/FireRedTTS3 checkpoint	dcf1bdcd1b8b25b382fa84c3e34eb82e3054a610
Official implementation	7a1f3a7282ff184cc1c7f070556baaf5f08b5216

Links

• speech-swift — Apple SDK
• Documentation
• Speech Studio
• soniqo.audio
• Blog