Imbatmann/matcha-tts-nepali-multispeaker-v3

Matcha-TTS Nepali — v3, multi-speaker (pitch-filtered male)

Self-contained release of the 2-speaker Nepali Matcha-TTS model: acoustic checkpoint, vocoder, exact training configs, the matcha package, a seeded synthesis script and reference audio. Everything needed to reproduce the shipped samples bit-for-bit is in this directory.

• Model: https://huggingface.co/Imbatmann/matcha-tts-nepali-multispeaker-v3
• Source / training pipeline: https://github.com/45Harry/matcha-tts-code (branch feature/finetune-english-male-voice, bundle under release/version3-multispeaker-pitchfiltered/)

The GitHub copy gitignores the checkpoint and vocoder, so it is code and samples only — this Hugging Face repo is the runnable one. Previous versions: v2, v1.

What changed vs v2: v2 trained on the raw chunks/001 corpus whose audio is 1/3 female-range pitch; both speakers converged to a compromise ~185 Hz voice that flipped registers on words absent from the male data. For v3 the male corpus was filtered by median F0 (≤155 Hz) — 479 of 732 chunks kept — and training warm-started from the pure female model instead of the male-biased single-speaker one, so no contradictory pitch enters the shared weights. Result: a stable male register (100–115 Hz) including on out-of- vocabulary words, and an unchanged clean female voice.

Run	logs/train/male_female_2spk_v2/runs/2026-08-21_17-10-56
Branch	feature/finetune-english-male-voice
Trained	2026-08-21, 17:10 → 19:00 (~1 h 50 min), single NVIDIA GB10
Sample rate	22 050 Hz, 80-band mel, hop 256
Speakers	2
Vocabulary	union symbol set, n_vocab = 264
Params	20.9 M

Speaker ids

spk	Voice	Training data
0	male (pitch-filtered podcast chunks)	455 train / 24 val utterances
1	female (chatterbox corpus)	1441 train / 75 val utterances

Keep this ordering everywhere — the API voice field maps "male" -> 0, "female" -> 1.

Contents

version3-multispeaker-pitchfiltered/
├── README.md                                  this file
├── synthesize.py                              seeded, standalone synthesis (v3 recipe)
├── synth_samples.sh                           regenerates every wav in samples/
├── requirements.txt                           exact pinned runtime
├── MANIFEST.sha256                            checksums for every shipped file
├── checkpoints/
│   └── matcha_male_female_2spk_v3.ckpt        epoch 199 / step 5999 (final = last.ckpt)
├── vocoder/
│   └── g_02500000                             HiFi-GAN universal v1 generator
├── configs/
│   ├── experiment_male_female_2spk_v2.yaml    hydra experiment config
│   ├── data_male_female_2spk.yaml             hydra data config
│   ├── train_config_resolved.yaml             fully-resolved config as trained
│   ├── train_overrides.yaml                   CLI overrides for the run
│   └── train.log                              training log for the run
├── matcha/                                    the model/text/vocoder package
└── samples/
    ├── texts.txt                              6 source sentences
    ├── texts_medium.txt                       6 two-clause sentences
    ├── texts_long.txt                         4 paragraph-length passages
    ├── phonemes.txt                           their G2P output + token counts
    ├── epoch199/{spk0_male,spk1_female}/       000–005.wav
    ├── tests_medium/{spk0_male,spk1_female}/   000–005.wav  (4–9 s)
    └── tests_long/{spk0_male,spk1_female}/     000–003.wav  (14–25 s)

Fixed 2026-09-02 — mel-stats correction was silently disabled. synthesize.py guarded the correction with isinstance(stats, dict), but model.hparams.data_statistics is an omegaconf DictConfig, so the test was always false and the correction the recipe below depends on never ran. Effect: ~15–20 dB of extra broadband hiss (noise floor −63 dB instead of −87 dB), audible as a metallic background, worst on the female voice and on long utterances. Only this bundle was affected — scripts/synthesize.py and deploy/server.py index the stats directly and were always correct. Everything under samples/tests_* is rendered with the fixed script; the samples/epoch199/ wavs predate the bug and were always correct.

Checkpoint

matcha_male_female_2spk_v3.ckpt is the final epoch (199). Validation loss: best 3.1268 @ step ~1500, final 3.204 — flat-ish across the run like v2, but unlike v2 late epochs are what you want here: speaker separation and the male register keep sharpening to the end (ep49 male median was 142 Hz with register flips; ep199 is 111 Hz stable). Use it as shipped.

Synthesis

pip install -r requirements.txt          # see the note about `real_nepali`

# single sentence
python synthesize.py --spk 0 --text "नमस्ते! हाम्रो सेवा केन्द्रमा स्वागत छ।" --out male.wav
python synthesize.py --spk 1 --text "नमस्ते! हाम्रो सेवा केन्द्रमा स्वागत छ।" --out female.wav

# a file of sentences, one per line
python synthesize.py --spk 0 --file samples/texts.txt --out out_dir/

Defaults ARE the verified v3 recipe: --steps 50 --temperature 0 --denoiser_strength 0.02, mel-stats correction applied automatically, HiFi-GAN universal v1, seed 1234. This combination is what fixed the metallic "pause-and-play" artefact of earlier attempts (denoiser 0.001 + mel smoothing thinned the signal; temperature 0.667 added register randomness). Useful extras: --speaking_rate 1.1 (>1 is slower), --lowpass_khz 9, --print-phonemes.

Text frontend

Devanagari is phonemized by the hand-written real_nepali G2P (profile real_nepali_v0.2), not espeak-ng — from the nepali_frontend repo pinned at commit b7395f28, installed editable. Latin runs inside mixed text go through espeak-ng en-us, so espeak-ng must be on PATH for code-switched input.

samples/phonemes.txt records the token string and id count per sample sentence (--print-phonemes to compare).

Note on spk0_male/000.wav: this is the human-approved master take (pinned byte-for-byte, also shipped as 000_master.wav and restored by synth_samples.sh after a re-render). The seeded re-render of the same sentence is acoustically equivalent (same duration, ~95–100 Hz male) but not byte-identical — at temperature 0 the flow-matching prior still draws its initial latent from the RNG, so the seed shifts micro-timing only.

Reproducibility

synthesize.py seeds the RNG per utterance (--seed, default 1234); with temperature 0 output is deterministic anyway, but seeding keeps batch order independent too.

PYTHON=python bash synth_samples.sh      # re-render everything under samples/
sha256sum -c MANIFEST.sha256             # confirm byte-for-byte match

Bit-exact only on the same GPU/library stack (NVIDIA GB10, CUDA 13.0, torch 2.13.0+cu130); elsewhere perceptually identical but different checksums.

Training recipe

Warm-started from the pure female model (checkpoints/matcha_nepali_final.ckpt) widened to 2 speakers by scripts/adapt_1spk_to_2spk.py (old weights embedded top-left, new speaker channels zeroed, fresh random spk_emb). The male voice is therefore learned only from pitch-consistent data — no female-range segments pull the shared weights around.

Lineage: English LJSpeech base → Nepali (union symbols) → 2-speaker male(pitch-filtered)+female (this release).

Data pipeline (reproduce before training):

.venv/bin/python scripts/filter_chunks_by_pitch.py \
    --chunks .../data/chunks/001 --out .../data/chunks/001_male --max-f0 155
.venv/bin/python scripts/prepare_chunks.py \
    --chunks .../data/chunks/001_male --name male_001m --spk-id 0
bash scripts/compute_stats.sh male_001m data/filelists_male_001m
.venv/bin/python scripts/build_multispk_filelists.py
bash scripts/compute_stats.sh male_female_2spk data/filelists_multispk
.venv/bin/python scripts/adapt_1spk_to_2spk.py \
    --src checkpoints/matcha_nepali_final.ckpt \
    --out checkpoints/male_female_2spk_adapted_v2.ckpt
bash scripts/train.sh male_female_2spk_v2

Init	checkpoints/male_female_2spk_adapted_v2.ckpt
Optimizer	Adam, lr 1e-4, no scheduler, weight decay 0
Epochs	200 (ran to completion), ~30 steps/epoch
Batch	32, accumulate_grad_batches: 2
Precision	16-mixed, gradient_clip_val: 5.0
Encoder	RoPE, 6 layers, 192 ch, 2 heads, spk_emb_dim: 64
Decoder	CFM/euler, sigma_min 1e-4, channels [256, 256], snakebeta
Mel stats	mel_mean -5.840723, mel_std 2.485387 (merged filelist)
Seed	1234

Full settings in configs/train_config_resolved.yaml.

Deploying

deploy/server.py in the main repo already serves this model: CHECKPOINT → matcha_nepali_v3.ckpt, DEFAULT_STEPS=50, DEFAULT_TEMPERATURE=0.0, mel-stats correction on by default, voice mapped via VOICE_TO_SPK (male/am_male -> 0, female/af_kore -> 1).

curl -s http://localhost:8002/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"input": "नमस्ते", "voice": "male",   "response_format": "wav"}' -o /tmp/male.wav
curl -s http://localhost:8002/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"input": "नमस्ते", "voice": "female", "response_format": "wav"}' -o /tmp/female.wav

Known limitations

• Male corpus is 479 utterances after filtering (was 732); rare words still have thin coverage — pronunciation is inherited from the shared network and is good, but prosody on unseen words is generic.
• Opening phrase of an utterance can sit slightly high (~165 Hz) before settling into the male register within ~1 s.
• Occasional whole-sentence register flip on loanword-dense male input: samples/tests_medium/spk0_male/002.wav ("इन्टरनेट राउटर पुनः सुरु") renders at ~159 Hz median rather than the usual ~105 Hz. The same content in tests_long/spk0_male/001.wav is correct, so it is sentence-specific, not a length effect.
• No code-switched utterances in either corpus; mixed Nepali+English in one sentence garbles at the language transition.