Byrne-Speech (female voice)

Model will be ungated for open download once I am done with the base..

A tiny (~12M-parameter) English text-to-speech model, built from scratch and trained on a single 12 GB GPU.

text → characters → acoustic transformer → 80-bin mel @ 24 kHz → HiFi-GAN → waveform

Stage	Params
Acoustic (char FastSpeech2-style transformer)	8.4M
Vocoder (HiFi-GAN generator + Snake)	3.6M
Total	~12M

Architecture

The acoustic model is a non-autoregressive FastSpeech2-style transformer, but the blocks use "SpikeWhale-v2 family" primitives rather than vanilla FastSpeech2:

• RMSNorm everywhere (including the variance predictors), replacing LayerNorm.
• RoPE + QK-Norm attention — rotary positional embeddings with query/key normalization (Gemma2 / OLMo2-style logit stability), replacing sinusoidal PE + vanilla attention.
• Conv-SwiGLU FFN — gated silu(gate) * up → down with a depthwise conv, keeping FastSpeech's local context.
• DERF-gated value-embedding residual (DERFValueGate) — every block re-injects the character embedding through an erf gate (use_value_embed: true). The gate init is tuned (bias=-2) so it actually opens and contributes during training.
• HRM refinement (HRMRefinement) — an iterative refinement head that polishes the mel decoder output (hrm_refine_steps: 3, hrm_refine_dim: 192). Its gate-init is fixed — only the scalar gate is zeroed, not the output projection — so it receives gradient and learns a real residual.

Pipeline: char embedding → 3 encoder FFT blocks → variance adaptor (duration / pitch / energy predictors, 256-bin pitch & energy) → length regulation with forced-aligned durations → 3 decoder FFT blocks → HRM refinement → 80-bin mel. Dims: hidden 256, 2 heads, FFN 320, conv kernel 3.

Active refinement (v2). Earlier checkpoints shipped with the two gated add-ons effectively inert — an over-conservative initialization left both the HRM refinement and the DERF value-embed gate without gradient, so they stayed at zero and all quality came from the core blocks. This checkpoint fixes the init (zero only one side of each gate): the HRM gate trains to a meaningful strength and the DERF gate opens, measurably sharpening the mel — higher spectral detail and a cleaner, less buzzy voice.

Vocoder: HiFi-GAN generator with Snake activation (periodic inductive bias), upsampling 256× to 24 kHz, trained with MPD + MSD discriminators and then fine-tuned on the acoustic model's predicted mels to remove train/inference mismatch.

The voice is clear and naturally paced (durations come from a forced aligner, not a uniform heuristic), then distilled from a VITS teacher to sharpen the spectral detail — all while staying a 12M model.

Usage

pip install -r requirements.txt
python -m src.infer --text "Hello, this is the Byrne Speech model." \
    --config config.yaml --acoustic acoustic.pt --vocoder vocoder.pt --out out.wav

Or in Python:

import torch
from src.utils import load_config
from src.infer import load_acoustic, load_vocoder, synthesize
cfg = load_config("config.yaml")
dev = "cuda" if torch.cuda.is_available() else "cpu"
ac = load_acoustic("acoustic.pt", cfg, dev)
voc = load_vocoder("vocoder.pt", cfg, dev)
wav = synthesize("Text to speak.", ac, voc, cfg, dev)   # float32 @ 24 kHz

Inference is lightweight — torch, numpy, soundfile, pyyaml, inflect, scipy. No phonemizer, no g2p, no torchaudio. Runs comfortably on CPU.

Samples

See samples/ — sample_1_fox.wav, sample_2_welcome.wav, sample_3_weather.wav, sample_4_thanks.wav. Try it live in the companion Space.

Fine-tuning to your own voice

The model adapts to a new single speaker from a few hours of clean audio — see FINETUNE.md (--finetune --lr 1e-4).

Notes & provenance

• Character input (no g2p), so any English text works directly.
• Durations are forced-aligned (torchaudio MMS) for natural rhythm.
• Distilled from kakao-enterprise/vits-ljs (a VITS model trained on the public-domain LJSpeech dataset). Quality is bounded by a 12M model + a mel L1 objective, so a slight soft/even texture remains by design.
• A male voice is in progress and will ship as a future update.

Built as a "small model, done from scratch" exercise — intelligible, fast, and tiny.

Citation

@misc{byrne_speech_2026,
  title        = {Byrne-Speech: a tiny (~12M) from-scratch English text-to-speech model},
  author       = {Byrne, Dean},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Quazim0t0/Byrne-Speech}}
}

Built on the public-domain LJSpeech corpus and distilled from the kakao-enterprise/vits-ljs teacher.