WaveGen [NeurIPS 2026]

WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning
Sang-Hoon Lee and Ha-Yeong Choi

Ajou University and KT

Paper · Source Code · Live Demo · Demo Page · Benchmark Demo · Spectral Forcing Demo

WaveGen generates speech directly from text and reference audio using Spectral Forcing. The models are trained on English and Mandarin Chinese speech from Emilia, using 24 kHz audio and 50 Hz waveform patches (480 samples per patch).

Models

This repository contains the pre-trained, V-Spec, and four-step Flash checkpoints in separate folders.

Model	Checkpoint	Folder	Inference steps
WaveGen (Pre-trained)	1300k EMA	WaveGen/	50
WaveGen (Fine-tuned with v-spec losses) — Recommended	1400k EMA	WaveGen_VSpec/	50
WaveGen-Flash	48k EMA	WaveGen_Flash/	4 (fixed)

WaveGen (Fine-tuned with v-spec losses) is the recommended model for inference and the default in the source repository. All checkpoints contain FP32 EMA weights only, in Safetensors format. Optimizer state and other training state are excluded; these files are intended for inference, not for resuming a training run.

WaveGen-Flash is optimized for four-step speech generation. Its complete merged weights load from one Safetensors file, together with its YAML and vocabulary. The Flash release includes inference code and model weights; its training recipe is not included.

Seed-TTS evaluation

Results on the complete standard Seed-TTS test sets for the EMA weights released here. The pre-trained and V-Spec models use Prompt Anchor, Scheduled K3 (base CFG 1.25), 50 Euler steps, Mel temperature 0, inference bottleneck dropout 0.2, and seed 0, with FP32 model weights and BF16 CUDA inference. The original test prompts are used.

Flash uses four Euler steps, the time grid [0, 0.1, 0.25, 0.5, 1.0], CFG 0, inference bottleneck dropout 0, waveform temperature 1, initial Mel temperature 0, and seed 0. It uses the same original test prompts and FP32-weight/BF16-inference precision. Prompt states interpolate from their initial noise toward the reference at each step; the initial Mel noise is scaled by the Mel temperature. The Flash results below use the new default temperature 0. WER/CER and UTMOS are truncated to two decimal places, and SIM to three decimal places (not rounded).

The complete test sets contain 1,088 English and 2,020 Chinese utterances.

Model	EN			ZH
	WER (%) ↓	SIM ↑	UTMOS ↑	CER (%) ↓	SIM ↑	UTMOS ↑
WaveGen Pre-trained · 1300k EMA	1.47	0.755	3.68	1.75	0.805	2.87
WaveGen V-Spec 1400k EMA · Recommended	1.39	0.743	3.59	1.65	0.783	2.79
WaveGen-Flash 48k EMA · 4 steps	1.49	0.727	3.53	1.59	0.768	2.74

WaveGen achieves state-of-the-art performance among TTS models trained on the Emilia dataset.

WER is reported for English and CER for Chinese. Error rates are percentages; all metrics are macro-averaged over utterances. The Safetensors exports preserve the FP32 EMA inference tensor values.

Download

Download the checkpoint, YAML, and vocabulary from the desired model folder and place that folder under ckpts/ in the WaveGen source repository:

ckpts/
├── WaveGen/
│   ├── model_1300000.safetensors
│   ├── WaveGen_SF_Emilia_24k_50Hz_IBDrop02.yaml
│   └── vocab.txt
├── WaveGen_VSpec/
│   ├── model_1400000.safetensors
│   ├── WaveGen_VSpec.yaml
│   └── vocab.txt
└── WaveGen_Flash/
    ├── model.safetensors
    ├── WaveGen_Flash.yaml
    └── vocab.txt

Keep each checkpoint together with its matching configuration and vocabulary. This Hub repository uses WaveGen/, WaveGen_VSpec/, and WaveGen_Flash/ at its root; ckpts/ is their destination in your local source checkout.

Usage

Follow the installation instructions, then run from the WaveGen source repository root. Supply a reference recording, its exact transcript, and the text to synthesize. Automatic transcription is not required.

Fine-tuned model (recommended default)

python -m wavegen_v_spec.infer.infer_cli \
  --ckpt_file ckpts/WaveGen_VSpec/model_1400000.safetensors \
  --model_cfg ckpts/WaveGen_VSpec/WaveGen_VSpec.yaml \
  --vocab_file ckpts/WaveGen_VSpec/vocab.txt \
  --ref_audio reference.wav \
  --ref_text "The transcript of the reference recording." \
  --gen_text "The text to synthesize."

Pre-trained model

python -m wavegen.infer.infer_cli \
  --ckpt_file ckpts/WaveGen/model_1300000.safetensors \
  --model_cfg ckpts/WaveGen/WaveGen_SF_Emilia_24k_50Hz_IBDrop02.yaml \
  --vocab_file ckpts/WaveGen/vocab.txt \
  --cfg_schedule scheduled_k3 \
  --cfg_strength 1.25 \
  --ref_audio reference.wav \
  --ref_text "The transcript of the reference recording." \
  --gen_text "The text to synthesize."

Both examples use EMA weights, Prompt Anchor (Prompt_xt_Anchor), Scheduled K3 (base CFG 1.25), 50 steps, Mel temperature 0, and seed 0. The supplied models use inference bottleneck dropout 0.2. The pre-trained module uses constant CFG 3 when no schedule is specified, so its example selects Scheduled K3 explicitly. CUDA inference uses BF16 autocast with FP32 model weights; CPU inference uses FP32.

Load the fine-tuned model from Hugging Face

The subfolder paths below select the 1400k model from LeeSangHoon/WaveGen. The loader resolves main to one commit before downloading the three assets; use a full commit SHA instead of main to pin a specific release.

import soundfile as sf
import torch
from wavegen_v_spec.infer.utils_infer import WaveGen

torch.set_num_threads(1)
model = WaveGen.from_pretrained(
    "LeeSangHoon/WaveGen",
    revision="main",
    checkpoint_filename="WaveGen_VSpec/model_1400000.safetensors",
    config_filename="WaveGen_VSpec/WaveGen_VSpec.yaml",
    vocab_filename="WaveGen_VSpec/vocab.txt",
    use_ema=True,
)
sample_rate, audio = model.synthesize(
    "reference.wav",
    "The transcript of the reference recording.",
    "The text to synthesize.",
)
sf.write("generated.wav", audio, sample_rate, subtype="PCM_16")

For the pre-trained model, download the WaveGen/ folder and use the pre-trained CLI above. See the source repository for the local Python API, Gradio demo, and batch evaluation.

The source checkout defaults to the V-Spec 1400k Safetensors checkpoint. The base and V-Spec loaders support existing local .pt checkpoints through an explicit checkpoint-path override.

WaveGen-Flash (4 Steps)

python -m wavegen_flash.infer.infer_cli \
  --checkpoint ckpts/WaveGen_Flash/model.safetensors \
  --model-config ckpts/WaveGen_Flash/WaveGen_Flash.yaml \
  --vocab-file ckpts/WaveGen_Flash/vocab.txt \
  --ref-audio reference.wav \
  --ref-text "The transcript of the reference recording." \
  --gen-text "The text to synthesize." \
  --output generated_flash.wav

Or load the same bundle directly from this repository:

import soundfile as sf
import torch
from wavegen_flash import WaveGenFlash

torch.set_num_threads(1)
model = WaveGenFlash.from_pretrained("LeeSangHoon/WaveGen", revision="main")
sample_rate, audio = model.synthesize(
    "reference.wav",
    "The transcript of the reference recording.",
    "The text to synthesize.",
)
sf.write("generated_flash.wav", audio, sample_rate, subtype="PCM_16")

The Flash loader accepts Safetensors only and requires no additional base weights. Its YAML fixes four Euler calls at [0, 0.1, 0.25, 0.5], ending at 1.0, with CFG 0 and IB dropout 0. The grid already includes time shift 3; it is not shifted again during inference. The final Mel output head is skipped by default; --compute-final-mel or compute_final_mel=True restores its computation. The Mel input and shared network remain active on the final call.

The YAML sets mel_temperature: 0.0 by default, while waveform temperature remains 1. Mel temperature scales only the initial Gaussian Mel noise; the clean reference Mel and later predicted Mel states are not scaled. Both noise draws are preserved even at temperature 0. Use --mel-temperature 1 or model.synthesize(..., mel_temperature=1.0) to reproduce the previous initialization with the same weights. Compared with temperature 1, temperature 0 improves WER/CER, SIM, and UTMOS on both complete Seed-TTS test sets.

The default duration limit is 30 seconds for prompt plus generated speech. Split longer text or explicitly increase max_duration_seconds; the sampler does not automatically split text. See the Flash guide for the complete inference interface.

Safeguards and limitations

• Use reference speech with the speaker's informed consent and permission to use the recording.
• Do not use generated speech for deceptive impersonation, fraud, harassment, or fabricated statements presented as authentic speech.
• Clearly identify shared or published outputs as AI-generated audio, and protect reference recordings and personal information.
• Reference quality and transcript accuracy affect the generated speech. Pronunciation errors, omissions, repetitions, and speaker-similarity failures may occur.
• For long passages, split the text into shorter segments; the single-utterance examples above do not automatically split long text.

These usage guidelines are not automatic safety filters. Public-facing services should add appropriate access controls, abuse reporting, and review of suspected misuse.

License

The model checkpoints are released under CC BY-NC 4.0.

WaveGen source-code contributions are released under the MIT License. Reused source components retain their original licenses and notices; see the source-code third-party notices.