kokoro-inno-clone-tuner

Zero-shot voice tuner for Kokoro-82M. Intended as an exploration and customization tool, not a strong identity cloner.

Outputs base Kokoro compatible voice packs @ [510, 1, 256].

Samples

Same passage for every voice, enrolled from the reference next to it. LibriTTS-R speakers are dev-clean, held out from training.

voice	reference	tuned	--fmax
libritts 1272
libritts 5338
libritts 6241
david attenborough			140
amelia earhart
jane goodall			220
vincent price

The packs are in voices/.

Usage

pip install -e . # kokoro==0.9.4, transformers, safetensors, scipy, soundfile

Enrolling a voice:

python -m inno_ref.enroll my_ref.wav me # -> voices/me.pt and voices/me_test.wav

If enrollment warns on a deep voice, try re-running with --fmax 140 (audio artifacts can trigger the warning in some cases, so use your ear as the judge)

import soundfile as sf
from kokoro import KPipeline

pipe = KPipeline(lang_code="a")
wav = next(pipe("Hello from a tuned voice.", voice="voices/me.pt")).audio
sf.write("out.wav", wav.cpu().numpy(), 24000)

Architecture

Enrollment embeds an input audio sample via UniSpeech-SAT-sv and maps through model.safetensors to build the encoder side; prosody aspects are measured to build the predictor half:

• style head: the decoder (timbre) half of the pack comes from the speaker embedding:
  • UniSpeech-SAT-sv embeds the reference (512) to Kokoro style (256). Shifts along a learned spectral-tilt direction by the reference's measured tilt.
• prosody blend: the predictor (prosody) half uses the stock Kokoro packs as a foundation:
  • Blended by nonnegative least squares to the reference F0 mean, F0 spread and syllable rate.
  • Packs more than 4 st from the reference's pitch are excluded.
  • Higher graded packs are weighted as preferred during selection.

Metrics

Measured on the shipped weights through the stock KPipeline, on the seven sample voices above unless noted.

metric	value
speaker cosine (WavLM-sv), LibriTTS-R dev-clean held out	0.932 (nearest stock voice 0.857)
F0 mean, tuned minus reference	0.8 st mean abs
F0 spread, tuned minus reference	0.6 st mean abs

Limits

• English trained (LibriTTS-R) currently
• Some texture/identity is traded to maintain audio quality and stability within Kokoro's range.
• Rythym is a best-effort match to the speaker.
  • Prosody is a convex blend of stock prosody by pitch, spread and pace. Outside of that range, it will only pin to the nearest edge
• Reference clip requirements:
  • 3 second minimum, up to a max of 30s
  • Reasonably free of audio artifacts
  • Single speaker

Training notes

All stages: Kokoro frozen; using LibriTTS-R train-clean-100.

Trained via HF Jobs a10g-small; about $7 in GPU time (~20 min per 3k steps)

• Stage 1: mapped speaker embedding to timbre:
  • Style head on a speaker-embedding cycle loss: clone, render, re-embed, cosine to the input audio embedding. [10k steps: dev cosine ~0.92; nearest stock voice baseline 0.857]
• Stage 2: the embedding alone missed texture and pitch; adjustments:
  • Augmented pitch and speed. Added losses via a WavLM layer-statistics L1, a small log-F0 error, and spectral-tilt. [11k steps: dev cosine 0.934, DNSMOS 3.30]
• Stage 3: tuned the encoder side to match the blended predictors:
  • Predictor half fixed to the blended rows, only the decoder half trained. [3k steps: dev cosine 0.932, DNSMOS 3.48]

Licenses and credits

Adapter weights and code: Apache-2.0.

• Built on hexgrad/Kokoro-82M (Apache-2.0, StyleTTS2 architecture).
• Enrollment uses microsoft/unispeech-sat-base-plus-sv (Microsoft UniSpeech, CC BY-SA 3.0 on GitHub).
• Trained on LibriTTS-R (CC BY 4.0); the LibriTTS-R sample references above are CC BY 4.0 as well.
• Sample references:
  • David Attenborough from a White House interview (public domain, US federal work);
  • Amelia Earhart from "Speech by Amelia Earhart", Library of Congress AFC 1986/022;
  • Jane Goodall from a BBC Woman's Hour excerpt (BBC, CC BY 3.0);
  • Vincent Price public domain from archive.org "The House On Haunted Hill"

Only clone voices you have permission to clone, even the shallow cloning offered via this model and technique.