Imbatmann/nepali-tts-multilingual

Nepali TTS Multilingual

By Imbatmann.

This model speaks Nepali and English in one female voice (Khushi), including both in the same sentence (for example "मेरो laptop बिग्रियो"). Voice cloning from a reference clip of 5 seconds or more also works.

How it works

• Nepali text is turned into phonemes (sound symbols) by the real_nepali G2P, and the phones are fed as their own tokens.
• English text uses a standard BPE tokenizer, so one sentence can mix both.
• Training data: about 1.8 hours of Khushi's Nepali speech, plus about 47 minutes of English in her voice. About 10% of samples splice Nepali and English clips together, to teach code switching.

Checkpoints

Folder	Step	Nepali val loss	English val loss	Notes
root	1,250	3.96	3.51	best Nepali val loss (default)
step_2750/	2,750	4.00	3.74	more Nepali training, English drifted more

Usage

This needs the matching inference code (the T3 loader with the Nepali text frontend) and the real_nepali G2P package installed.

import os
from huggingface_hub import snapshot_download

path = snapshot_download("Imbatmann/nepali-tts-multilingual")
model = TTS.from_local(path, "cuda", nano=True)                      # TTS = loader class from the inference code
# model = TTS.from_local(os.path.join(path, "step_2750"), "cuda", nano=True)

wav = model.generate("नमस्ते! मेरो laptop बिग्रियो, कृपया मद्दत गर्नुहोस्।")       # Khushi's voice by default
wav = model.generate("Hello, how are you?", audio_prompt_path="someone_10s.wav")  # cloning

• Text is split into sentences at ।, ॥, ?, !, and ., and joined with 250 ms pauses.
• Digits in a mostly Nepali sentence are read in Nepali.

Files

File	What it is
t3_nano_v1.safetensors	Text to speech token model (text vocab 50,379)
tokenizer.json, tokenizer_config.json	BPE tokenizer plus Nepali phone tokens
nepali_frontend.json	Phone to id map and G2P profile (real_nepali_v0.2)
conds.pt	Khushi's default voice conditioning
s3gen_meanflow.safetensors, ve.safetensors	Speech decoder and voice encoder
LICENSE	MIT license notice

Limits

• Nepali pronunciation depends on the G2P. Decimal points are not read (३.५ is read as "तिन पाँच").
• English sounds a little less like Khushi than Nepali does.
• Long texts can occasionally run on or add words that aren't in the text. Generating again with another seed usually fixes it.
• Outputs carry an imperceptible audio watermark.