FireRedTTS3-Instruct for SGLang Omni

We provide our FireRedTTS3-Instruct checkpoint for inference with SGLang Omni. It supports reference-audio voice cloning, instruction-guided voice design, semantic editing, and acoustic editing. Audio output is 24 kHz mono, with WAV responses and streaming PCM available through the /v1/audio/speech API.

Our model code and original checkpoint release:

• FireRedTTS3 source code
• Original FireRedTTS3 weights and model card

This card documents the SGLang Omni loading and serving workflow.

Install SGLang Omni

Use Linux with an NVIDIA CUDA GPU and Python 3.12. Follow the SGLang Omni installation guide for the CUDA, UCX, and FlashAttention prerequisites, then install from source:

git clone https://github.com/sgl-project/sglang-omni.git
cd sglang-omni
pip install uv
uv venv .venv -p 3.12
source .venv/bin/activate
uv pip install --prerelease=allow -e '.[firered-tts3]'

Use a revision containing the FireRedTTS3 integration. Before that integration is merged, check out the corresponding PR revision before installing; an older PyPI release or checkout does not provide this model. Run the following commands from the sglang-omni repository root.

Download this checkpoint

For PR testing, use our temporary checkpoint at CKermit/FireRedTTS3-Instruct. After this PR, we plan to publish the official checkpoint at FireRedTeam/FireRedTTS3-Instruct.

hf download CKermit/FireRedTTS3-Instruct \
  --local-dir ./models/FireRedTTS3-Instruct

Download the complete repository, including configuration, tokenizer, and redae/ assets. Loading only the root model.safetensors file is insufficient. If you already have the complete checkpoint locally, use that directory directly.

Start the server

CUDA_VISIBLE_DEVICES=0 sgl-omni serve \
  --model-path ./models/FireRedTTS3-Instruct \
  --config examples/configs/firered_tts3_instruct.yaml \
  --allowed-local-media-path "$PWD/docs/_static/audio" \
  --port 8000

Wait for /health to return HTTP 200 before sending requests. Initial model loading, compilation, and warmup can take several minutes.

The reference clips below are included in the SGLang Omni repository. Local paths are resolved on the server, relative to its working directory. For your own clips, allow their directory with --allowed-local-media-path and supply the matching transcript for cloning.

Voice cloning

Provide exactly one reference recording and its actual transcript. The example uses the repository's ref_voice.wav:

curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "FireRedTTS3-Instruct",
    "input": "Hello, welcome to SGLang Omni.",
    "language": "English",
    "references": [{
      "audio_path": "docs/_static/audio/ref_voice.wav",
      "text": "It was the night before my birthday. Hooray! It’s almost here! It may not be a holiday, but it’s the best day of the year."
    }],
    "seed": 42
  }' --output clone.wav

Omit task_type for cloning, or set it to Base. The value tts is not an accepted serving task name.

Voice design

Generate a voice from a description, without a reference recording:

curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "FireRedTTS3-Instruct",
    "input": "今天天气很好，我们一起去公园散步吧。",
    "task_type": "VoiceDesign",
    "instructions": "一个年轻女性的温柔嗓音，语速稍慢，带一点俏皮。",
    "seed": 42
  }' --output voice_design.wav

Semantic editing

Use the reference recording as the source to edit. Its content supplies the input, so input may be empty; no reference transcript is required:

curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  --data-binary @- --output semantic_edit.wav <<'JSON'
{
  "model": "FireRedTTS3-Instruct",
  "input": "",
  "task_type": "SemanticEdit",
  "instructions": "Replace 'birthday' with 'anniversary'.",
  "ref_audio": "docs/_static/audio/ref_voice.wav",
  "seed": 42
}
JSON

For substitutions, use the quoted form Replace 'source' with 'target'. The quoted example above was verified by transcribing the generated audio; omitting the quotes did not perform the intended replacement in this test. The x-gen-text response header contains text with edit masks, so the target word may appear only in the audio. Validate the rendered audio rather than requiring the target word to appear in that header.

Acoustic editing

Adjust the source recording's speed, pitch, or volume. This example requests 0.8x speed through the model's native acoustic-edit task:

curl --fail-with-body -X POST http://localhost:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "FireRedTTS3-Instruct",
    "input": "",
    "task_type": "AcousticEdit",
    "ref_audio": "docs/_static/audio/ref_voice.wav",
    "speed": 0.8,
    "seed": 42
  }' --output acoustic_edit.wav

Raw acoustic instructions must follow the trained templates rather than free-form descriptions. For pitch or volume, use, for example, "instructions": "shift the pitch by 2 steps" or "instructions": "adjust the volume to 0.8". An explicit instructions string takes precedence over the composed speed instruction. The speed field applies to AcousticEdit; it does not change cloning or voice design. Edit sources are capped at 20 seconds. VoiceDesign and SemanticEdit generate text before audio, so their first-audio latency differs from cloning.

Streaming

To stream audio, add "stream": true and "response_format": "pcm" to the request and use curl -N. The response contains raw 24 kHz, signed 16-bit little-endian, mono PCM. Save it as output.pcm and convert it with:

ffmpeg -f s16le -ar 24000 -ac 1 -i output.pcm output.wav

The server also exposes /v1/audio/speech/stream over WebSocket. See the WebSocket protocol for session.config, input.text, input.commit, and input.done events.

Offline Python SDK

Save this as offline_tts.py in the SGLang Omni repository root and run CUDA_VISIBLE_DEVICES=0 python offline_tts.py. This starts the pipeline locally without an HTTP server.

import asyncio
import multiprocessing as mp
from pathlib import Path

from sglang_omni.client import Client, GenerateRequest
from sglang_omni.models.firered_tts3.config import FireRedTTS3PipelineConfig
from sglang_omni.pipeline.mp_runner import MultiProcessPipelineRunner


async def main() -> None:
    config = FireRedTTS3PipelineConfig(
        model_path="./models/FireRedTTS3-Instruct",
    )
    config.stage_named("talker").factory.flow_mode = "fp32"
    runner = MultiProcessPipelineRunner(config)
    await runner.start(timeout=600)
    try:
        client = Client(runner.coordinator)
        result = await client.speech(
            GenerateRequest(
                prompt={
                    "input": "今天天气很好，我们一起去公园散步吧。",
                    "task_type": "VoiceDesign",
                    "instructions": "一个年轻女性的温柔嗓音，语速稍慢，带一点俏皮。",
                    "seed": 42,
                },
                stream=False,
            ),
            request_id="demo-1",
        )
        Path("offline.wav").write_bytes(result.audio_bytes)
    finally:
        await runner.stop()


if __name__ == "__main__":
    mp.set_start_method("spawn", force=True)
    asyncio.run(main())

Serving notes

• The current deployment supports a single GPU (tp_size=1).
• HTTP input is limited to 400 characters. Editing requests may use an empty input as shown above.
• Use an explicit language: Chinese (default), English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, or Italian. Automatic language detection (Auto) is unsupported. These are the stock serving API's accepted language values; FireRedTTS3 supports additional languages and dialects through our standalone inference pipeline.
• Submit spoken-form text when normalization is needed. The FastText and LLM text-normalization setup in our standalone pipeline is not used in this quick start.
• Put flow-solver settings under stage_params.talker, for example {"n_timesteps": 10, "inference_cfg": 2.0, "max_patches": 200}. Unknown top-level fields are not forwarded as model settings.
• seed controls the per-request flow sampler. Different precisions, hardware, or batching schedules can produce different results with the same seed.
• For our published model-quality results, see the FireRedTTS3 model card. SGLang Omni runtime accuracy and performance are evaluated separately.
• Keep the example Instruct configuration's flow_mode: fp32 for the supported default. The offline example sets it explicitly because it does not load the example YAML.
• temperature, top_p, and top_k affect the VoiceDesign/SemanticEdit text phase. They do not control continuous-latent audio generation.

See the FireRedTTS3 cookbook for the full request parameters and deployment configuration.

Usage Disclaimer

• The project incorporates zero-shot voice cloning functionality; Please note that this capability is intended solely for academic research purposes.
• DO NOT use this model for ANY illegal activities❗️❗️
• The developers assume no liability for any misuse of this model.
• If you identify any instances of abuse, misuse, or fraudulent activities related to this project, please report them to our team immediately.

Citation

@article{fireredtts3,
  title   = {FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations},
  author  = {FireRed Team},
  journal = {arXiv preprint},
  year    = {2026},
}

Acknowledgements

• Qwen3 and Qwen2-Audio for the language model and audio understanding foundations
• DiTAR for the patch-level diffusion autoregressive formulation
• X-Codec for the discriminator design used in RedAE training
• CAM++ for speaker embedding extraction
• fastText for automatic language identification
• WeTextProcessing (wetext) for the Chinese / English text normalization front-end

License

FireRedTTS3 is released under Apache-2.0. See our original model card for usage guidance and model evaluation details.