VantoraLabs/Vocetta-181K

MicroTTS 181K

A complete English text-to-speech system in 181,189 parameters — three tiny networks and a dictionary-based grapheme-to-phoneme front end, all running on CPU in real time.

Params	181,189 (duration 13,864 + acoustic 65,299 + decoder 102,026)
Audio	24 kHz mono
Speed	~135x real time on CPU, ~85x on a 2014 budget GPU (RTF 0.0074)
Intelligibility	WER 0.007 on the templated eval set (119/128 sentences exact)
Naturalness	SCOREQ 1.03, UTMOS 1.33, DNSMOS-OVRL 2.57 on a 24-sentence diverse set
License	MIT (runtime and weights); bundled G2P data is Apache-2.0

Listen to samples/ first — those eight files were rendered by this exact checkpoint.

How it works

Text becomes audio through four stages, all trained by distillation from a larger teacher TTS:

text ──► G2P ──► phoneme ids
                  │
      duration.pt │  ids ──────────────► frame count per phoneme
                  │
      acoustic.pt │  ids + durations ───► mel spectrogram [100 bands, T frames]
                  │
      decoder.pt  │  mel + noise ───────► complex spectrum ──► iSTFT ──► audio

G2P (microtts/g2p/). A dictionary-first English front end: words are looked up in bundled pronunciation dictionaries (gold and silver), and words the dictionaries miss go through a small neural fallback model. NumPy only, no espeak, no torch, no network access. The output is a string of IPA phonemes, mapped to a frozen 62-symbol vocabulary (<bos> and <eos> bracket each utterance).

Duration student (duration.pt). A 3-layer 1D convolutional network over the phoneme sequence. It predicts how many mel frames each phoneme occupies, using learned position, sequence-length and duration features, with residual blocks around each conv pair. The output is exponentiated log-duration, rounded and clamped to at least one frame per phoneme.

Acoustic student (acoustic.pt). The largest predictor. It embeds the phoneme ids, refines them with token-context convolutions, expands them to the frame grid by repeating each phoneme its predicted number of frames, then runs a second stack of convolutions over frames and projects to 100 mel bands. A mel spectrogram carries less information than the waveform, which is exactly why the next stage exists.

Decoder (decoder.pt). Mel to waveform. A ConvNeXt1D stack (depthwise conv, LayerNorm, two pointwise layers with a GELU between them, residual) maps the mel to a complex spectrum of 513 bins, which the iSTFT turns into audio. The magnitude head is exponential with bin 0 and the Nyquist bin zeroed, and a DC-blocking filter removes the remaining offset. The decoder is noise-fed: a 4-channel noise input is projected and added to the mel embedding, which gives the model a source of variation. At inference, zero noise is the best choice — see Benchmarks.

The training recipe

Every student is trained by distillation: a larger teacher TTS renders a text corpus once, and the three students learn to reproduce the teacher's intermediate representations. The recipe is short enough to describe exactly.

Stage 0 — build the pack

Pick a teacher TTS and a text corpus (thousands of sentences of varied, spoken-style text). For each line, store: the phoneme ids, the teacher's audio, the teacher's per-phoneme durations, and the mel spectrogram of the teacher's audio (100 bands, n_fft 1024, hop 256). One .npz per line. train/build_pack.py does this with Kokoro-82M as the teacher. Watch the duration units when the teacher's frame rate differs from the mel hop — the script shows the conversion.

Stage 1 — duration student

Train ids → frame counts against the teacher's durations. Loss: smooth-L1 on log-duration plus a term on the total length (weight 0.35) so the model gets the overall timing right, not just per-phoneme averages. Learning rate 2e-3 with AdamW, about 4k steps at batch 32. This stage trains in ~10 minutes.

Capacity matters, and not monotonically: hidden size 20 is the sweet spot this model uses. Sizes 14, 18 and 22 all produced worse end-to-end speech, so try a few sizes before committing. Training a second seed of the champion config and averaging the two models' durations is a cheap way to reduce seed variance.

Stage 2 — acoustic student

Train ids + durations → mel against the teacher's mel. Loss: L1 plus spectral convergence, plus an optional PatchGAN critic on the mel (hinge loss, dense patch scores over time) from step 1000 with weight 0.1. The critic matters: a plain L1 regressor produces a mel that is too smooth frame-to-frame, and the output stays muddy even when the L1 looks converged.

The learning rate is the single most important setting here: 2e-3. At lower rates this model underfits in a way that is not obvious from the loss — the mel is plausible but temporally over-smoothed, and garbled audio is the result. ~50k steps at batch 8.

Stage 3 — decoder

1. Initialize, do not train from scratch. Slice the first N channels of a pretrained neural vocoder's decoder (train/init_decoder.py slices Vocos: 512-dim backbone, 4 ConvNeXt blocks, pw 1536). Training a decoder this small from scratch does not reach intelligibility. A fresh slice renders noise — that is normal, the next two steps fix it.
2. Recovery. Fine-tune the slice on teacher mels only (--mix-prob 0.0), with a waveform L1 + multi-resolution spectral loss + loudness matching, ~20k steps at batch 4. This brings the slice back to clear speech.
3. Z-mix. Continue with a 50/50 mix of teacher mels and acoustic-student mels (--mix-prob 0.5), another ~20k steps. This is the load-bearing stage for the handoff: a decoder that has only seen teacher mels is brittle to the acoustic student's smoother output at synthesis time. Do not skip it.

Decoder capacity dominates final quality more than any other single choice. In this model's sweep, widening the decoder from dim 32 to dim 40 (adding ~25k params) cut word error on the eval set by 46%. If you have parameters to spend, spend them on the decoder. Note that decoder hyperparameters interact with the learning rate used in recovery/z-mix: the rate that worked for one width can be too aggressive for a wider one, so A/B the pair together.

Use it from scratch

pip install -r requirements.txt          # numpy, torch, soundfile

from microtts import MicroTTS

tts = MicroTTS.load(".")                  # reads duration.pt / acoustic.pt / decoder.pt
wav = tts.synthesize("Hello world")       # float32 numpy array, 24 kHz
tts.save("out.wav", wav)                  # saves RMS-normalized to -26 dBFS

MicroTTS.load accepts device="cpu" (default) or "cuda". The full pipeline loads in ~0.1 s and needs no network access. If you already have phoneme ids, call tts.synthesize_ids(ids) directly and skip the G2P.

Two practical notes:

• Loudness. synthesize returns the raw output; its loudness is not normalized. MicroTTS.normalize(wav) applies RMS normalization (0.05, about -26 dBFS), and MicroTTS.save does it for you. Measure loudness before normalizing if it matters to your application.
• Noise. synthesize(..., noise_scale=0.0) is the default and gives the best measured quality. The decoder still accepts noise_scale=1.0 if you want variation across renders, but it costs quality on every metric.

Python 3.10+, CPU is enough. On this machine the model runs 135x faster than real time on CPU and 85x on an old GPU — the models are small enough that kernel-launch overhead eats most of the GPU's advantage.

Benchmarks

Measured on this exact checkpoint. All word-error numbers use Whisper small as the judge; naturalness scores use the standard open models (SCOREQ, UTMOS, DNSMOS), each evaluated on the raw rendered audio.

Intelligibility (word error rate, lower is better)

set	WER	sentences exactly right
128-sentence templated English eval set	0.007	119 / 128
24-sentence diverse held-out set (templated + conversational)	0.23	1 / 24

The two rows are honest about scope: on templated, spoken-style text the model is near-perfect, while free-form conversational text is clearly harder. The diverse set is hard for everyone at this size, and closing that gap would take the decoder capacity described above.

Naturalness / audio quality (higher is better, 24-sentence diverse set)

setting	SCOREQ	UTMOS	DNSMOS-OVRL	DNSMOS-SIG
noise_scale = 0.0 (recommended)	1.03	1.33	2.57	2.86
noise_scale = 1.0	0.91	1.26	1.90	2.18

Intelligibility and naturalness agree here: zero noise wins both, and the margin on DNSMOS is large. This is a free inference-setting change, no retraining required.

DNSMOS-SIG catches metallic distortion; the 2.86 at zero noise says the output is not buzzy. The SCOREQ number is the honest ceiling of a 181K model: clean and intelligible, but not as natural as a large voice.

Speed (24 diverse sentences, warm-up excluded, zero noise)

device	RTF	real-time factor
CPU	0.0074	~135x faster than real time
GTX 750 (2014, 4 GB)	0.011	~85x faster than real time

RTF includes the duration, acoustic and decoder forwards; the G2P adds ~1 ms per sentence on top.

Reproducing the scores. Word error: transcribe the rendered wavs with openai/whisper-small and compute WER against the input text (benchmark scripts in benchmark/). Naturalness: pip install scoreq speechmos plus torch.hub.load("tarepan/SpeechMOS:v1.2.0", "utmos22_strong"), then score each wav file with the library's own defaults.

What is in this folder

duration.pt, acoustic.pt, decoder.pt   the weights (~750 KB total)
model.safetensors                       same weights, prefixed keys, fp32 (auto-detected by HF Hub so the params count shows on the repo card)
microtts/                              runtime package (frontend, models, g2p)
  g2p/g2p_data/                        dictionaries + OOV model, licenses in NOTICE.md
samples/                               eight rendered examples
train/                                 the training scripts (see the recipe above)
benchmark/                             RTF + WER measurement scripts and results
README.md, LICENSE, requirements.txt

Limits, stated plainly

• English only. The G2P dictionaries are US English.
• One voice. This is a single-voice model; there is no speaker conditioning.
• Utterances cap at 207 phoneme tokens and 2400 mel frames (25 s).
• Conversational text is much harder than templated text (see the benchmark table); expect errors there.
• No text normalization beyond the front end's number handling. Unusual punctuation or markup should be stripped before synthesis.

Credits

Runtime, weights and training scripts: MIT (this release). The bundled grapheme-to-phoneme dictionaries and the OOV model are Apache-2.0; see microtts/g2p/g2p_data/NOTICE.md for provenance.