VantoraLabs/Vocetta2-276k

Vocetta2 276K

A complete English text-to-speech system in 276,765 parameters — three tiny networks and a dictionary-based grapheme-to-phoneme front end, all running on CPU in real time.

Params	276,765 (duration 5,344 + acoustic 65,299 + decoder 206,122)
Audio	24 kHz mono
Speed	~80x real time on CPU (RTF 0.0125)
Intelligibility	WER 0.10 on a 24-sentence diverse held-out set (9/24 exact)
Naturalness	SCOREQ 2.04, DNSMOS-OVRL 3.18, DNSMOS-SIG 3.41
License	MIT (runtime and weights); bundled G2P data is Apache-2.0

Listen to samples/ first — those eight files were rendered by this exact checkpoint.

How it works

Text becomes audio through four stages, all trained by distillation from a larger teacher TTS:

text ──► G2P ──► phoneme ids
                  │
      duration.pt │  ids ──────────────► frame count per phoneme
                  │
      acoustic.pt │  ids + durations ───► mel spectrogram [100 bands, T frames]
                  │
      decoder.pt  │  mel + noise ───────► complex spectrum ──► iSTFT ──► audio

G2P (microtts/g2p/). A dictionary-first English front end: words are looked up in bundled pronunciation dictionaries (gold and silver), and words the dictionaries miss go through a small bundled neural fallback model. NumPy only, no espeak, no torch, no network access. The output is a string of IPA phonemes, mapped to a frozen 62-symbol vocabulary (<bos> and <eos> bracket each utterance).

Duration student (duration.pt, 5,344 params). A 3-layer 1D convolutional network over the phoneme sequence. It predicts how many mel frames each phoneme occupies, using learned position, sequence-length and duration features, with residual blocks around each conv pair. The output is exponentiated log-duration, rounded and clamped to at least one frame per phoneme.

Acoustic student (acoustic.pt, 65,299 params). Embeds the phoneme ids, refines them with token-context convolutions, expands them to the frame grid by repeating each phoneme its predicted number of frames, then runs a second stack of convolutions over frames and projects to 100 mel bands.

Decoder (decoder.pt, 206,122 params). Mel to waveform. A ConvNeXt1D stack (depthwise conv, LayerNorm, two pointwise layers with a GELU between them, residual) maps the mel to a complex spectrum of 513 bins, which the iSTFT turns into audio. The magnitude head is exponential with bin 0 and the Nyquist bin zeroed, and a DC-blocking filter removes the remaining offset. The decoder is noise-fed: a 4-channel noise input is projected and added to the mel embedding. At inference, zero noise is the best choice.

The training recipe

Every student is trained by distillation: a larger teacher TTS renders a text corpus once, and the students learn to reproduce the teacher's intermediate representations.

Stage 0 — build the pack

Pick a teacher TTS and a text corpus (thousands of sentences of varied, spoken-style text). For each line store: phoneme ids, teacher audio, per-phoneme durations, and the mel spectrogram of the teacher audio (100 bands, n_fft 1024, hop 256). One .npz per line (train/build_pack.py). Watch the duration units when the teacher's frame rate differs from the mel hop — the script shows the conversion.

Stage 1 — duration student

Train ids → frame counts against the teacher's durations. Loss: smooth-L1 on log-duration plus a term on the total length (weight 0.35). Learning rate 2e-3 with AdamW, ~4k steps at batch 32. Capacity matters and not monotonically: hidden 12 is the sweet spot this model uses; sweep a few sizes if you change the corpus.

Stage 2 — acoustic student

Train ids + durations → mel against the teacher's mel. Loss: L1 plus normalized L1, temporal-delta L1 (weight 0.10 — this anti-smoothing term is load-bearing), channel-statistics L1, and a hinge-loss PatchGAN critic on the mel from step 1500 with weight 0.1. The learning rate matters most: 2e-3, constant. ~25k steps at batch 8.

Stage 3 — decoder

1. Initialize, do not train from scratch. Slice the first N channels of a pretrained neural vocoder's decoder (train/init_decoder.py; dim 62, pw 186, 4 blocks for the shipped model). Training a decoder this small from scratch does not reach intelligibility.
2. Recovery. Fine-tune the slice on teacher mels (--mix-prob 0.0) with waveform L1 + multi-resolution spectral loss + a hinge PatchGAN discriminator + a cosine-gram temporal-structure loss (weight 0.4 — this supervises temporal texture and is what keeps the output from sounding mushy), plus high-band-excess and quiet-ceiling terms. ~20k steps at batch 4, constant lr 2e-4.
3. Z-mix. Continue with a 50/50 mix of teacher mels and acoustic-student mels (--mix-prob 0.5), another ~20k steps. Load-bearing for the handoff.
4. Joint fine-tune. Co-train the acoustic and decoder together end-to-end against audio targets for ~50k steps. This closes the acoustic→decoder mismatch that no separately-trained pair fully avoids; it was worth +0.35 SCOREQ in this model's development.

Two traps measured during development, worth stating plainly:

• Use a constant learning rate. Decaying the rate (e.g. ×0.9998 per step) silently starves these small models: the loss looks fine while the output stays muddy. Removing the decay was the single largest fix in this model's history.
• Do not over-trust one metric. A decoder tuned only against SCOREQ learned to emit muffled audio that scored 3.07 SCOREQ while being fully unintelligible (WER 1.0). Every candidate in this model's development was checked with WER, DNSMOS and listening.

Use it from scratch

pip install -r requirements.txt          # numpy, torch, soundfile

from microtts import MicroTTS

tts = MicroTTS.load(".")                  # reads duration.pt / acoustic.pt / decoder.pt
wav = tts.synthesize("Hello world.")      # float32 numpy array, 24 kHz
tts.save("out.wav", wav)                  # saves RMS-normalized to -26 dBFS

MicroTTS.load accepts device="cpu" (default) or "cuda". The full pipeline loads in ~0.1 s and needs no network access. If you already have phoneme ids, call tts.synthesize_ids(ids) directly and skip the G2P.

Two practical notes:

• Loudness. synthesize returns the raw output; its loudness is not normalized. MicroTTS.normalize(wav) applies RMS normalization (0.05, about -26 dBFS), and MicroTTS.save does it for you.
• Noise. synthesize(..., noise_scale=0.0) is the default and gives the best measured quality. The decoder still accepts noise_scale=1.0 if you want variation across renders, but it costs quality on every metric.

Python 3.10+, CPU is enough.

Benchmarks

Measured on this exact checkpoint. All word-error numbers use Whisper small as the judge; naturalness scores use the standard open models (SCOREQ, DNSMOS), each evaluated on the raw rendered audio.

Intelligibility (word error rate, lower is better)

set	WER	sentences exactly right
24-sentence diverse held-out set	0.101	9 / 24
128-sentence templated eval set	0.000	128 / 128

The templated set (near-identical sentence frames) is fully memorized; the diverse set is the honest generalisation measure and the number to compare against other models.

Naturalness / audio quality (higher is better, 24-sentence diverse set)

setting	SCOREQ	DNSMOS-OVRL	DNSMOS-SIG
noise_scale = 0.0 (recommended)	2.04	3.18	3.41

DNSMOS-SIG catches metallic distortion; 3.41 at zero noise says the output is not buzzy.

Speed (24 diverse sentences, warm-up excluded, zero noise, CPU)

device	RTF	real-time factor
CPU	0.0125	~80x faster than real time

RTF includes the duration, acoustic and decoder forwards; the G2P adds ~1 ms per sentence on top.

Reproducing the scores. Word error: transcribe the rendered wavs with openai/whisper-small and compute WER against the input text (scripts in benchmark/). Naturalness: pip install scoreq speechmos, then score each wav with the library's own defaults.

What is in this folder

duration.pt, acoustic.pt, decoder.pt   the weights (~1.1 MB total)
model.safetensors                       same weights, prefixed keys, fp32
                                        (auto-detected by HF Hub so the params
                                        count shows on the repo card)
microtts/                              runtime package (frontend, models, g2p)
  g2p/g2p_data/                        dictionaries + fallback model, Apache-2.0
samples/                               eight rendered examples
train/                                 the training scripts (see the recipe above)
benchmark/                             RTF + WER measurement scripts and results
README.md, LICENSE, requirements.txt

Limits, stated plainly

• English only. The G2P dictionaries are US English.
• One voice. This is a single-voice model; there is no speaker conditioning.
• Utterances cap at 207 phoneme tokens and 2400 mel frames (25 s).
• Free-form conversational text is harder than templated text; expect word errors there (see the benchmark table).
• No text normalization beyond the front end's number handling. Unusual punctuation or markup should be stripped before synthesis.

Credits

Runtime, weights and training scripts: MIT (this release). The bundled grapheme-to-phoneme dictionaries and the fallback model are Apache-2.0; see microtts/g2p/g2p_data/NOTICE.md.