You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Matcha-TTS Nepali — v2, multi-speaker

Self-contained release of the 2-speaker Nepali Matcha-TTS model: acoustic checkpoints, vocoder, the exact training configs, the matcha package, a seeded synthesis script and reference audio. Everything needed to reproduce the shipped samples bit-for-bit is in this directory.


Run	`logs/train/male_female_2spk/runs/2026-08-20_16-55-15`
Repo commit	`e154b4b7f23c25010f3951b594ae310d7c5d1b9b` (branch `feature/finetune-english-male-voice`)
Trained	2026-08-20, 16:55 → 19:00 (~2 h 5 min), single NVIDIA GB10
Sample rate	22 050 Hz, 80-band mel, hop 256
Speakers	2
Vocabulary	`union` symbol set, `n_vocab = 264`
Params	20.9 M

Speaker ids

spk	Voice	Training data
`0`	male_001 (Rajesh Pandey podcast chunks)	695 train / 37 val utterances
`1`	female (chatterbox corpus)	1441 train / 75 val utterances

Keep this ordering everywhere — the API voice field maps "male" -> 0, "female" -> 1.

version2-multispeaker/
├── README.md                                  this file
├── synthesize.py                              seeded, standalone synthesis
├── synth_samples.sh                           regenerates every wav in samples/
├── requirements.txt                           exact pinned runtime
├── MANIFEST.sha256                            checksums for every shipped file
├── checkpoints/
│   ├── matcha_male_female_2spk.ckpt           epoch 199 / step 6800 (final)
│   └── matcha_male_female_2spk_epoch126.ckpt  epoch 126 / step 4300
├── vocoder/
│   └── g_02500000                             HiFi-GAN universal v1 generator
├── configs/
│   ├── experiment_male_female_2spk.yaml       hydra experiment config
│   ├── data_male_female_2spk.yaml             hydra data config
│   ├── train_config_resolved.yaml             fully-resolved config as trained
│   ├── train_overrides.yaml                   CLI overrides for the run
│   └── train.log                              training log for the run
├── matcha/                                    the model/text/vocoder package
└── samples/
    ├── texts.txt                              6 source sentences
    ├── phonemes.txt                           their G2P output + token counts
    ├── epoch199/{spk0_male,spk1_female}/       000–005.wav
    └── epoch126/{spk0_male,spk1_female}/       000–005.wav

Which checkpoint to use

Both are shipped so they can be compared by ear; epoch 199 (matcha_male_female_2spk.ckpt) is the default and what synthesize.py loads unless --ckpt says otherwise.

Validation loss is flat across the whole run, so it does not separate them:

checkpoint	epoch	step	`loss/val_epoch` at save
(not shipped, in run dir)	8	300	2.9875 ← best monitored
(not shipped, in run dir)	41	1400	2.9965
`matcha_male_female_2spk_epoch126.ckpt`	126	4300	2.9908
`matcha_male_female_2spk.ckpt`	199	6800	3.057 (final epoch)

The spread is ~0.07 nats — within run-to-run noise, and the checkpoint callback's best-scoring epoch (8) is far too early to have learned the speaker split. Pick by listening, not by the table. Level statistics for the shipped samples are equivalent (RMS ≈ 0.060–0.065, peak 0.53–0.69, no clipping in either).

Synthesis

pip install -r requirements.txt          # see the note about `real_nepali`

# single sentence
python synthesize.py --spk 0 --text "नमस्कार, तपाईंलाई कस्तो छ?" --out male.wav
python synthesize.py --spk 1 --text "नमस्कार, तपाईंलाई कस्तो छ?" --out female.wav

# a file of sentences, one per line
python synthesize.py --spk 0 --file samples/texts.txt --out out_dir/

# the mid-run checkpoint
python synthesize.py --ckpt checkpoints/matcha_male_female_2spk_epoch126.ckpt \
    --spk 0 --file samples/texts.txt --out out_ep126/

Defaults match what deploy/server.py serves: --steps 25 --temperature 0.667 --denoiser_strength 0.02 --symbols union, HiFi-GAN universal v1. Useful extras: --speaking_rate 1.1 (>1 is slower), --lowpass_khz 9 (tames HF hiss), --print-phonemes (dump the token string fed to the model).

Text frontend

Devanagari is phonemized by the hand-written real_nepali G2P (profile real_nepali_v0.2), not espeak-ng — it comes from the nepali_frontend repo, pinned at commit b7395f28, installed editable. Latin runs inside mixed text go through espeak-ng en-us and are kept as native English IPA, so espeak-ng must be on PATH for code-switched input. Pure Devanagari input needs only real_nepali.

samples/phonemes.txt records the exact token string and id count for each sample sentence — diff against it (or --print-phonemes) to confirm your frontend matches this release before blaming the acoustic model.

Reproducibility

synthesize.py seeds the RNG per utterance (--seed, default 1234). This matters: the flow-matching prior is torch.randn_like(mu) * temperature, and the repo's scripts/synthesize.py leaves it unseeded, so it produces slightly different audio on every run. Seeding is why the shipped wavs have stable checksums.

PYTHON=python bash synth_samples.sh      # re-render everything under samples/
sha256sum -c MANIFEST.sha256             # confirm byte-for-byte match

Verified bit-exact on re-run on the training machine (NVIDIA GB10, CUDA 13.0, torch 2.13.0+cu130). On different GPU/CUDA/library versions the audio is perceptually identical but checksums will differ — cuDNN kernel selection and reduction order are not portable. Pass --seed -1 for varied output.

Training recipe

Warm-started from the converged single-speaker male model, widened to 2 speakers by scripts/adapt_1spk_to_2spk.py (old weights embedded top-left, new speaker-conditioning channels zeroed, fresh random spk_emb), so training starts behaving exactly like the 1-speaker model plus a zero speaker path.

Lineage: English LJSpeech base → Nepali (union symbols) → male_001 single-speaker → 2-speaker male+female (this release).


Init	`checkpoints/male_female_2spk_adapted.ckpt`
Optimizer	Adam, lr `1e-4`, no scheduler, weight decay 0
Epochs	200 (ran to completion), 67 steps/epoch
Batch	32, `accumulate_grad_batches: 2`
Precision	`16-mixed`, `gradient_clip_val: 5.0`
Encoder	RoPE, 6 layers, 192 ch, 2 heads, `spk_emb_dim: 64`
Decoder	CFM/euler, `sigma_min 1e-4`, channels [256, 256], snakebeta
Mel stats	`mel_mean -5.780489`, `mel_std 2.456294`
Seed	1234

Full settings in configs/train_config_resolved.yaml. To retrain from the main repo:

.venv/bin/python scripts/build_multispk_filelists.py
bash scripts/compute_stats.sh male_female_2spk data/filelists_multispk
.venv/bin/python scripts/adapt_1spk_to_2spk.py \
    --src checkpoints/matcha_male_voice_001.ckpt \
    --out checkpoints/male_female_2spk_adapted.ckpt
bash scripts/train.sh male_female_2spk

Deploying

deploy/server.py in the main repo is still single-speaker (spks=None, the voice field ignored). To serve this model, point CHECKPOINT at matcha_male_female_2spk.ckpt and pass the speaker through in _synth_chunk:

spks = torch.tensor([spk_id], device=DEVICE).long()   # was spks=None

routing voice: "af_kore" | "female" -> 1, "male" | "am_male" -> 0, default 0.

Known limitations

Training data contains no code-switched utterances (pure-Nepali and pure-English sentences only). Mixed Nepali+English in one sentence garbles at the language transition; use the repo's scripts/synthesize_mixed.py, which splits on script runs and concatenates, for that case.
Validation loss plateaued by epoch ~8 and never improved meaningfully, so the extra 190 epochs bought little measurable gain.
The female speaker has ~2× the male's data; male coverage is 695 utterances.

Downloads last month: -; Downloads are not tracked for this model. How to track

Imbatmann
/

matcha-tts-nepali-multispeaker-v2