Sarashina2.2-TTS-ONNX

Built with Sarashina.

ONNX Runtime export of sbintuitions/sarashina2.2-tts, a Japanese-centric zero-shot voice-cloning TTS model built on a 0.5B-parameter Llama backbone plus a CosyVoice-style flow-matching decoder and HiFT vocoder.

This derivative runs the entire inference pipeline in ONNX Runtime, including zero-shot voice cloning — no PyTorch, no transformers, no CUDA required — making it light enough for CPU-only local and server deployment. The LLM stage runs at fp32. It was originally exported at int4 for a smaller/faster package, but that was found to measurably degrade content accuracy — a phrase that mispronounced consistently across every tested sampling seed at int4 came out correct in most seeds at fp32, using an otherwise identical pipeline. fp32 trades about 3x the LLM download size and ~2x slower generation for that accuracy. (fp16 would be a smaller middle ground, but onnxruntime-genai only supports it with CUDA/DML — not CPU, which is what this runtime targets — so it isn't an option here.)

⚠️ License: NonCommercial. This is a derivative of a model released under the Sarashina Model NonCommercial License Agreement. Commercial use is not permitted. See LICENSE and the attribution in NOTICE. s3_tokenizer.onnx is a third-party file under a separate Apache 2.0 license (see NOTICE) — not covered by the NonCommercial terms.

Pipeline

text ─▶ LLM (onnxruntime-genai, fp32) ─▶ semantic tokens
     ─▶ flow_encoder.onnx    ─▶ mu / mask / speaker / cond
     ─▶ flow_estimator.onnx  ─▶ (Euler ODE loop, 10 steps) ─▶ mel
     ─▶ hift.onnx            ─▶ waveform (24000 Hz)

--audio-prompt reference.wav (zero-shot voice cloning)
     ─▶ numpy DSP ─▶ s3_tokenizer.onnx  ─▶ semantic tokens (LLM few-shot prime)
                  └▶ campplus.onnx      ─▶ speaker embedding

torch.stft / torch.istft in the vocoder are replaced with equivalent real-valued conv/matmul implementations, since the ONNX exporter cannot handle complex tensors. The reference-audio feature extraction for voice cloning (mel/fbank computation) is reimplemented in pure numpy. Outputs were validated against the original PyTorch model on real speech: vocoder mean abs diff ~7e-4, flow stages ~1e-6, speaker embedding cosine similarity ~0.998, semantic tokens match exactly in the large majority of cases (a resampling-algorithm difference can very occasionally flip one token to an acoustically adjacent codebook entry).

Files

Path	Description
`llm/`	onnxruntime-genai model: text → semantic speech tokens
`flow_encoder.onnx`	tokens + prompt → conditioning tensors
`flow_estimator.onnx`	one flow-matching velocity step (driven by an Euler loop)
`hift.onnx`	mel spectrogram → waveform
`campplus.onnx`	speaker encoder, for zero-shot voice cloning
`s3_tokenizer.onnx`	semantic tokenizer for the reference wav, for zero-shot voice cloning (third-party, Apache 2.0 — see NOTICE)
`s3_mel_filters.npz`	mel filterbank asset needed by `s3_tokenizer.onnx`
`flow_rand_noise.npy`	fixed flow-matching ODE initial noise, extracted from the reference model (see below)
`default_prompt.wav`	bundled reference voice used when no `--audio-prompt` is given (see below; third-party, same NonCommercial license — see NOTICE)
`meta.json`	sample rate, mel channels, ODE steps, and other runtime constants

Usage

This model is designed to run through the jpn-sarashina-onnx backend of sherox:

# use uv pip or pip
uv pip install git+https://github.com/bagustris/sherox[tts-ja-sarashina-onnx]


# download this repo into models/sarashina-onnx/, then:
sherox.tts --lang jpn-sarashina-onnx --text "こんにちは。" --output out.wav

# zero-shot voice cloning — also torch-free, no extra install needed
sherox.tts --lang jpn-sarashina-onnx \
    --text "明日は友達と映画を見に行きます。" \
    --audio-prompt reference.wav --audio-prompt-text "参照音声の書き起こし。" \
    --output cloned.wav

# --seed lets you work around a bad sample for a specific phrase (see below)
sherox.tts --lang jpn-sarashina-onnx --text "お元気ですか。" --output out.wav --seed 2

The runtime uses repetition_penalty=1.3 by default on the LLM stage; without it the model tends to get stuck repeating a single semantic token at the start of generation (a property of the base model, independent of the ONNX export).

Default voice (no `--audio-prompt`)

The reference model's own prompting guide states that generation quality depends heavily on having a real audio prompt — it describes no supported "no prompt" mode. A zero speaker-embedding fallback is therefore an unsupported configuration, not just a degraded one (verified: the original PyTorch backend is also unstable there — e.g. 5.7s of near-silence for a 5-character greeting with certain seeds).

When --audio-prompt is omitted, this model bundles default_prompt.wav (sourced from sbintuitions/sarashina2.2-tts's own official demo samples, under the same NonCommercial license — see NOTICE) and uses it automatically instead of a zero embedding.

This is a real but partial improvement, not a full fix for content accuracy. Some phrases that were wrong with the old zero-embedding fallback now decode correctly; others that were fine with it can come out wrong with the bundled prompt instead (a longer reference can suppress/truncate the LLM's generated token count for the target text, sometimes severely — shorter references were empirically better here). If a specific phrase comes out wrong, try a different --seed; there is no evidence of one seed that's reliably correct across every phrase.

Separately, content accuracy overall is meaningfully better now that the LLM runs at fp32 instead of int4 — see above. A phrase ("お元気ですか") that mispronounced across every tested seed (0–3) at int4 was correct in most seeds at fp32, on an otherwise identical pipeline, indicating that was a quantization-precision issue rather than ordinary sampling variance. --seed remains the practical lever for whatever residual sampling variance is left.

Regenerating these artifacts

pip install 'sherox[tts-ja-sarashina-onnx-export]'
python -m sherox.sarashina_onnx_export \
    --model-dir <sarashina2.2-tts checkpoint> --out-dir models/sarashina-onnx
    # add --precision int4 for a smaller/faster but less accurate LLM

Attribution & License

s3_tokenizer.onnx is redistributed from FunAudioLLM/CosyVoice2-0.5B (https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B), licensed under Apache License 2.0, Copyright FunAudioLLM. Not covered by the Sarashina NonCommercial License above.

default_prompt.wav is redistributed from sbintuitions/sarashina2.2-tts's own official demo samples (samples/samples/prompt_C.wav), under the same Sarashina Model NonCommercial License above.

Original model: sbintuitions/sarashina2.2-tts by SB Intuitions Corp. Redistributed under the Sarashina Model NonCommercial License Agreement — see LICENSE. Commercial use requires a separate agreement with SB Intuitions.