FranciscoCarlosErra/ttslibre-poc-en

TTSLibre v0.1.2 – English proof of concept (experiment 012)

Proof-of-concept checkpoint from TTSLibre, an open recipe for a small zero-shot TTS (Supertonic-lineage: character input, flow matching over a compressed autoencoder latent, Vocos vocoder). ~19.8M parameters. Not a usable product: mean Whisper WER 0.23 on held-out LibriTTS-R sentences, 13% of sentences above 0.5.

Files

file	what
ttl.pt	TTS model, experiment 012 checkpoint at 6h21m / step 138000 (best on whole-val)
ae.pt	mel autoencoder (24-d latent, 6x compression), experiment 001
vocab.json, latent_stats.pt	character vocab and latent normalization used by the checkpoint
config.json	effective training config of the run
ref_default.wav	default reference clip (LibriTTS-R, CC BY 4.0)
voices/*.pt	voicepacks: AE latents of a 6-8 s reference clip (speakers 6209, 4137 of train-clean-100)

Vocoder: charactr/vocos-mel-24khz (loaded from the Hub at runtime).

Training

LibriTTS-R dev-clean + train-clean-100 + train-clean-360 (177 h after a 1-12 s clip filter, ~1150 speakers), 1x RTX 3090, ~14 h total across experiments 006 → 010 → 011 → 012 (see the repo's experiments/ READMEs and RESULTS for the full chain and numbers).

Whole-val Whisper WER (99 sentences x 3 draws):

checkpoint	mean	median	frac > 0.5
006 (5 h data)	0.386	0.350	0.28
011 (44 h data)	0.291	0.200	0.18
012 this file (177 h)	0.232	0.136	0.13

Inference from a cold start

Tested with Python 3.12, torch 2.10 (CUDA 12.8), GPU or CPU (CPU is ~1-2 s per sentence).

git clone https://github.com/franciscocarloserra/ttslibre && cd ttslibre
python3 -m venv venv && . venv/bin/activate
pip install torch torchaudio vocos soundfile numpy num2words einops huggingface_hub
hf download FranciscoCarlosErra/ttslibre-poc-en --local-dir hfckpt
# put the files where experiments/012-english-235h-overnight/config.json expects them
E=experiments/012-english-235h-overnight
mkdir -p $E/runs/overnight $E/runs/ae datasets/libritts-r/prep-clean100 datasets/libritts-r/prep
cp hfckpt/ttl.pt $E/runs/overnight/ttl.pt
cp hfckpt/ae.pt $E/runs/ae/ae.pt
cp hfckpt/latent_stats.pt datasets/libritts-r/prep-clean100/
cp hfckpt/ref_default.wav datasets/libritts-r/prep/
cp hfckpt/voices/*.pt experiments/voices/
cd $E
python synth.py "Hello from a tiny open model." hello.wav --run runs/overnight/ttl.pt --ref ../voices/6209.pt
# add --cpu for CPU; --ref can also be any 5-10 s wav of a new speaker

Vocos weights (charactr/vocos-mel-24khz) are fetched from the Hub on first run.

In-memory use (load once, generate many)

import sys, soundfile as sf
sys.path.insert(0, "experiments/012-english-235h-overnight")
from common import load_config
from synth import Synth
c = load_config("experiments/012-english-235h-overnight/config.json")
S = Synth(c, run="runs/overnight/ttl.pt")          # ~2 s on GPU, keeps model + AE + Vocos resident
zref, rmask = S.style("../voices/6209.pt")          # voicepack, or a wav path for a new speaker
for i, t in enumerate(["First sentence.", "Second sentence."]):
    wav, dur = S(t, zref, rmask, steps=16, cfg=2.0, duration_scale=1.0)
    sf.write(f"out{i}.wav", wav, c["data"]["sample_rate"])

Paths in Synth are resolved relative to the experiment dir. steps (flow steps) and cfg are the two sampling knobs; 16 / 2.0 are the defaults used for the numbers above. Generation is ~0.3 s per sentence on an RTX 3090 (RTF ~0.1).

To make a voicepack from a clip: python ../voice.py make clip.wav myvoice --exp 012-english-235h-overnight --run runs/overnight/ttl.pt (from experiments/), writes voices/myvoice.pt.

License

Weights and voicepacks CC0-1.0. Training data LibriTTS-R is CC BY 4.0 (attribution: LibriTTS-R, Koizumi et al. 2023; LibriVox readers).