Chatterbox Nepali TTS

Zero-shot Nepali text-to-speech with voice cloning, built by fine-tuning ResembleAI/Chatterbox-Multilingual-hi (Hindi v3) with LoRA on a Nepali speech dataset.

Model Details


Base model	ResembleAI/Chatterbox-Multilingual-hi (v3)
Fine-tuning method	LoRA (rank 32, alpha 64, dropout 0.05)
LoRA target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training data	~~3,938 Nepali utterances (~~4,138 total with test split)
Checkpoint	Epoch 14 (merged)
Language	Nepali (ne) — Devanagari script
Sample rate	24,000 Hz

Files

File	Description
`t3_mtl23ls_v2.safetensors`	T3 transformer (text → speech tokens), LoRA merged
`s3gen.pt`	S3Gen vocoder (speech tokens → waveform)
`ve.pt`	Voice encoder (extracts speaker embedding from reference audio)
`grapheme_mtl_merged_expanded_v1.json`	Multilingual grapheme tokenizer config

Usage

Install

git clone https://github.com/your-org/chatterbox-streaming-v2
cd chatterbox-streaming-v2
conda activate chatterbox-v3

Inference

from chatterbox.mtl_tts import ChatterboxMultilingualTTS

model = ChatterboxMultilingualTTS.from_local("path/to/model", device="cuda")

wav = model.generate(
    "नमस्ते, तपाईंलाई कस्तो छ?",
    language_id="ne",
    audio_prompt_path="reference_voice.wav",  # any ~5–10s clean speech clip
    exaggeration=0.5,
    cfg_weight=0.5,
    temperature=0.8,
)

import soundfile as sf
sf.write("output.wav", wav.squeeze(0).cpu().numpy(), model.sr)

Long-form inference with sentence chunking

Use the provided infer_ne_hindi.py script, which handles:

Automatic sentence chunking at । boundaries
Sub-chunking at , for sentences estimated > 40 seconds
Decimal expansion (१२.५ → १२ दशमलव ५) so dots are never mistaken for sentence boundaries

python infer_ne_hindi.py \
  --model path/to/model \
  --ref reference_voice.wav \
  --text "नेपाल दक्षिण एसियामा अवस्थित एउटा सुन्दर देश हो। यहाँ हिमाल, पहाड र तराई छन्।" \
  --out output.wav

Training Details

Optimizer: AdamW with cosine annealing LR scheduler
LoRA rank: 32, alpha: 64, dropout: 0.05
Dataset: Internal Nepali speech corpus (ground-truth Devanagari transcripts)
Training split: ~3,938 utterances train / ~200 utterances test
Base model weights frozen; only LoRA adapter weights were trained, then merged

Intended Use

Nepali TTS for news reading, voice assistants, accessibility tools
Zero-shot voice cloning: provide any reference audio clip to clone a speaker's voice
Long-form synthesis of Nepali paragraphs

Limitations

Quality depends on the reference audio (clean, 5–10s clips work best)
Primarily optimized for standard Nepali (Khas Kura); dialectal variation may reduce quality
Numbers and special characters should be pre-expanded to their spoken Nepali form

Citation

If you use this model, please cite the original Chatterbox paper and acknowledge Oshara's fine-tuning work.

License

Apache 2.0 — see LICENSE. The base model weights (ResembleAI/chatterbox) retain their original license.

Downloads last month: -