Vyvo/drifting-tts-tr-voxcpm2

Drifting TTS Turkish: VoxCPM2 latents (experimental)

Experimental. For quality, use the released mel model Vyvo/drifting-tts-tr (v3.1).

This is the one-step Turkish TTS of drifting-tts, trained in the latent space of a pretrained audio VAE: the paper's latent setting (Deng et al., 2026).

• Output. The DriftDiT generator writes VoxCPM2 AudioVAE latents in one network evaluation (64 channels, 25 Hz).
• Decoding. VoxCPM2's own decoder turns the latents into 48 kHz audio, which is then resampled to 24 kHz. No separate vocoder is needed.
• Training. The model uses the same recipe and the same training data as v3.1. Only the target space differs.

Compare the two models side by side: Vyvo/drifting-tts-tr-compare.

Checkpoints

file	training steps
drifting_tts_voxcpm2_10k.pt	10k (the better one)
drifting_tts_voxcpm2_50k.pt	50k, continued from 10k at a constant learning rate

Results

Freya-TR-Eval, first 100 sentences. Speaker 722, T = 0.3, α = 2. Whisper large-v3 on audio band-matched to 8 kHz; UTMOSv2 on the full band.

model	WER	CER	UTMOSv2
v3.1: mels + BigVGAN-v2-ft (released)	0.66%	0.14%	2.934
VoxCPM2 latent, 10k steps + fine-tuned decoder	1.87%	0.43%	2.530
VoxCPM2 latent, 10k steps + released decoder	4.94%	1.39%	2.052
VoxCPM2 latent, 50k steps + released decoder	7.57%	2.50%	1.904
mel model, 10k steps (same recipe, for reference)	18.77%	5.26%	2.746

• At equal training, latents learn to speak much faster. At 10k steps the latent model makes a quarter of the mel model's word errors. It also trains 2.5× and synthesises 3× faster.
• The released VoxCPM2 decoder was the main source of noise. Fine-tuned on the model's generated latents, it brings WER from 4.94% to 1.87% and DNSMOS OVRL from 1.49 to 3.25.
• Naturalness still lags v3.1 (UTMOSv2 2.53 against 2.93).
• Longer generator training did not help. From 10k to 50k steps, WER and UTMOSv2 got worse.
• The analysis and the next steps are in docs/LATENTS.md.

Usage

pip install "drifting-tts[voxcpm] @ git+https://github.com/kadirnar/drifting-tts"

from huggingface_hub import hf_hub_download
from drifting_tts.synthesize import Synthesizer

repo = "Vyvo/drifting-tts-tr-voxcpm2"
synth = Synthesizer(hf_hub_download(repo, "drifting_tts_voxcpm2_10k.pt"), "cuda",
                    vocoder=hf_hub_download(repo, "voxcpm2_decoder_ft.pt"))  # the fine-tuned decoder (below)
wav, info = synth("Merhaba! Bu ses tek bir adımda üretildi.", speaker="studio", seed=0)  # 24 kHz

• The checkpoint says which audio VAE it uses (voxcpm2). Synthesizer decodes with it: the VoxCPM2 AudioVAE weights are downloaded from openbmb/VoxCPM2 (Apache-2.0) on first use. vocoder= swaps in the fine-tuned decoder; without it, the released decoder sounds noisy on generated latents.
• The voices are the same as v3.1's: studio, male, female, or any integer speaker ID.

License

• The weights in this repository: CC BY 4.0, as v3.1.
• The VoxCPM2 AudioVAE: Apache-2.0, from its own repository.

Fine-tuned decoder

voxcpm2_decoder_ft.pt is the VoxCPM2 AudioVAE decoder, fine-tuned for 35k steps on this model's generated latents (#27, vocoder.arch: vae_decoder of drifting-tts finetune-vocoder).

• Why it is needed. The released decoder renders the model's latents, which are smoother than real ones, as noise.
• Results. Freya-TR-Eval, first 100 sentences, 10k-step model:

decoder	WER	CER	UTMOSv2	DNSMOS OVRL
released VoxCPM2	4.94%	1.39%	2.052	1.49
fine-tuned, 20k steps	2.31%	0.50%	2.340	3.21
fine-tuned, 35k steps (voxcpm2_decoder_ft.pt)	1.87%	0.43%	2.530	3.25
fine-tuned, 40k steps	1.76%	0.42%	2.410	–
v3.1 (mels), for reference	0.66%	0.14%	2.934	3.33

• Output rate. Use it at 24 kHz, the LatentVocoder default. It was trained against 24 kHz recordings.

synth = Synthesizer(hf_hub_download("Vyvo/drifting-tts-tr-voxcpm2", "drifting_tts_voxcpm2_10k.pt"), "cuda",
                    vocoder=hf_hub_download("Vyvo/drifting-tts-tr-voxcpm2", "voxcpm2_decoder_ft.pt"))