speecht5_tts-pld-ceb-solo

Splintir/speecht5_tts-pld-ceb-v2 continue-finetuned on one Cebuano speaker alone — CEB_0348, the most-recorded speaker in sapinsapin/pld, 110 clips of 15.5 minutes.

Not more data — less, deliberately. Its parent already trained on all 14,007 usable Cebuano clips and still collapsed on 13 of 16 native speaker embeddings. That is not a volume problem: those 14,007 clips are 139 speakers averaging ~8 minutes each, so the model learns a Cebuano-in-general voice and no individual one. Adding more of the same broad, shallow data sharpens the average; it never produces a person.

This run removes the variable instead of feeding it. Filter to a single speaker, train briefly on top of a checkpoint that already speaks the language, and stop. 15.5 minutes is a voice-lock budget, not a voice-building one — hence 300 steps at lr 5e-06 rather than anything from scratch.

speaker	CEB_0348 — 110 clips, 15.5 min
training clips	99 train + 11 eval
steps	300
batch	4 × 4 accumulation
learning rate	5e-06
precision	fp32 + gradient checkpointing (fp16 NaNs SpeechT5's mel loss)
eval loss	0.3060 — see the warning below before comparing it to anything

What it bought

A synthetic voice cannot be scored directly, so the test is indirect: play each clip to a speech recognizer that never saw the original sentence, and count how much of the text it fails to recover. Character error rate (CER) is the share of characters it gets wrong; lower is better. It measures intelligibility to a machine listener — not naturalness, which still needs human ears.

All three checkpoints benched in one run per voice, ten held-out pld test lines, whisper-large-v3-turbo as judge:

	base v1	v2	solo
CER, arctic:slt	0.146	0.123	0.114
WER, arctic:slt	0.539	0.450	0.412
CER, native CEB_0348	0.769	0.177	0.101
WER, native CEB_0348	0.974	0.499	0.343
rms, native	0.013	0.035	0.024

The 0.769 is the argument in one number. Handed this speaker's own voice, the published checkpoint produces 6.7 seconds of near-silence at rms 0.013 and gets 77% of characters wrong. The solo model, for which that voice is native, is the best result across every configuration tried in this work.

Four listeners, not one

One judge is one opinion, and every recognizer is deaf in its own way. Whisper has no Cebuano in its training data, so part of every score above is the judge's failure rather than the model's — a real Cebuano speaker reading these lines scores 0.081, not 0. The same audio, rescored by four different recognizers:

judge	v1	v2	solo	what it is
sapinsapin/whisper-small-pld-ceb	0.139	0.065	0.059	small Whisper, finetuned on Cebuano
whisper-large-v3-turbo	0.146	0.123	0.114	large, zero-shot, no Cebuano
whisper-turbo + Filipino LoRA	0.126	0.092	0.083	large, nudged toward Philippine speech
facebook/mms-1b-all (ceb adapter)	0.125	0.067	0.064	CTC model, 491k hours, ceb head

CER, arctic:slt, ten lines, one bench run.

Two things to take from it. All four agree on the ranking — solo ahead of v2 ahead of v1, every time. A result that survives four different kinds of deafness is not an artifact of one judge's blind spot. And the gap between v1 and the later checkpoints is far wider for judges that know Cebuano (0.139 → 0.059, down 58%) than for the zero-shot one (0.146 → 0.114, down 22%). A recognizer that cannot hear the language well is partly measuring its own ignorance, and that noise flattens the distance between a good voice and a poor one.

Read down a column, never across: a CTC model and a sequence-to-sequence model make different classes of mistake, so their absolute values are not on one scale.

One caveat on the in-domain judge: whisper-small-pld-ceb was finetuned on the same corpus these models trained on. It is the best-informed listener available and also the one most likely to reward speech that sounds specifically like PLD. Sharpest instrument, not the neutral one.

What it does not fix

Pace and loudness. At rms 0.024 and 6.4 s per line — against a healthy 0.08 and 3.9 s — it is intelligible but quiet and dragging. 15.5 minutes locks a timbre without buying rhythm. That needs a real recording session, not another finetune on this corpus.

The eval loss is not comparable to its parent's. 0.3060 here is measured on 11 held-out clips of the same single speaker; v2's 0.3687 is on 200 clips spanning many speakers. One-voice prediction is an easier task. The lower number is not evidence that this model is better in general — the bench table above is.

This checkpoint is a good default when you want one dependable Cebuano voice. For an intelligibility-first model to pair with an arctic:* speaker, or as a base to continue-train, use v2.

Usage

SpeechT5 holds no voice of its own — every call needs a 512-d x-vector. speaker.npy in this repo is CEB_0348, the voice this model was locked onto. Use it; a foreign embedding gives back what the table above shows.

import numpy as np, torch, soundfile as sf
from huggingface_hub import hf_hub_download
from transformers import SpeechT5ForTextToSpeech, SpeechT5HifiGan, SpeechT5Processor

REPO = "Splintir/speecht5_tts-pld-ceb-solo"

processor = SpeechT5Processor.from_pretrained(REPO)
model = SpeechT5ForTextToSpeech.from_pretrained(REPO).eval()
vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan").eval()

# speaker.npy is stored as (512,); the model wants (1, 512). Without the
# unsqueeze this raises "The first dimension of speaker_embeddings must be
# either 1 or the same as batch size".
speaker = torch.from_numpy(np.load(hf_hub_download(REPO, "speaker.npy"))).float().unsqueeze(0)

ids = processor(text="Maayong buntag sa imong tanan.", return_tensors="pt")["input_ids"]
with torch.inference_mode():
    speech = model.generate_speech(ids, speaker, vocoder=vocoder)

sf.write("out.wav", speech.numpy(), 16000)   # 16 kHz mono

# In a notebook (Colab, Jupyter), play it inline instead of saving:
from IPython.display import Audio, display
display(Audio(speech.numpy(), rate=16000))

Runs unmodified on a stock Colab CPU runtime — every dependency is preinstalled, including the sentencepiece the tokenizer needs — at roughly 1.8× real time. For GPU, .to("cuda") the model, the vocoder and ids/speaker, then .cpu() the result before sf.write.

The display(Audio(...)) line renders a play button in a notebook cell. It produces no sound over a terminal or SSH session; there, write the wav and fetch it (from google.colab import files; files.download("out.wav")).

Spell numbers out: the tokenizer is character-level Latin and drops digits silently.

Generation is not deterministic. SpeechT5 keeps decoder-prenet dropout active during inference, by design. Five renders of one line gave five different durations and rms values, and one checkpoint scored CER 0.093 and 0.123 in two identical bench runs — so compare systems within a run, never across runs, and cache production audio by hash(text + voice).

Trained with scripts/train_tts.py (--speaker-id top); preprocessing matches finetune_tts.py from the halohalo pipeline — one x-vector per clip, never averaged.