Sahithinethi/speecht5-marathi-tts

SpeechT5 Marathi TTS (undertrained — experimental)

microsoft/speecht5_tts fine-tuned to speak Marathi (Devanagari).

⚠️ Status: undertrained. Read before using.

This checkpoint comes from a 300-step CPU run (~1,200 samples seen) and has not converged. Each length below was generated 5 times:

Input	Tokens	Audible runs	Best amplitude
short — नमस्कार आपले स्वागत आहे	24	0 / 5	0.00002
medium — महाराष्ट्र राज्यात आज पाऊस पडत आहे	35	1 / 5	0.62
long — 12-word sentence	73	5 / 5	0.84

Amplitude below 0.02 is inaudible; healthy generation is 0.2–0.8.

Why it varies run to run: SpeechT5 keeps its speech-decoder prenet dropout active during generation (Tacotron-style — .eval() does not disable it). So the same sentence can be silent on one call and fine on the next. Reliability falls off sharply as input gets shorter than the training clips (which were 2.9–5.9 s, 28–57 tokens).

Practical guidance: use long sentences, check np.abs(wav).max() on every call, and retry on silence — that is a legitimate workaround here, not a hack.

The real fix is more training, not different inference code: the same notebook on a T4 with ~3,000 steps (20× the data) resolves this. Treat this repo as a working pipeline artifact, not a finished voice.

The base checkpoint has an English-only tokenizer that maps every Devanagari character to <unk>, so it cannot pronounce Marathi at all. This model extends the vocabulary with 66 tokens — 63 Devanagari characters plus a <sp> word-boundary token — growing the vocabulary from 81 to 145.

⚠️ Spaces must become <sp>

SpeechT5 marks word starts with the SentencePiece ▁ character. Once Devanagari characters are added tokens, the tokenizer consumes the spaces between them and ▁ is never emitted — the model would receive each sentence as one unbroken string with no word boundaries at all.

This model therefore uses an explicit <sp> token (id 144) in place of spaces, initialized from the pretrained ▁ embedding so it starts out already meaning "word break".

Passing raw spaces produces silence or run-on speech. Always use prepare() below. (Verified: the same sentence with <sp> gave amplitude 0.47; without it, 0.00002.)

Usage

import re, torch, numpy as np, soundfile as sf
from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech, SpeechT5HifiGan
from huggingface_hub import hf_hub_download

REPO = "Sahithinethi/speecht5-marathi-tts"
processor = SpeechT5Processor.from_pretrained(REPO)
model     = SpeechT5ForTextToSpeech.from_pretrained(REPO).eval()
vocoder   = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan").eval()

# This repo ships the exact speaker vector used in training.
spk = torch.tensor(np.load(hf_hub_download(REPO, "speaker_embedding.npy"))).unsqueeze(0)

DEVA = re.compile(r"[^ऀ-ॿ\s]")

def prepare(text):
    # Must match training: Devanagari only, and spaces become <sp>.
    text = str(text).replace("।", " ").replace("॥", " ")   # danda
    text = DEVA.sub(" ", text)
    return re.sub(r"\s+", " ", text).strip().replace(" ", "<sp>")

# Use a LONG sentence - short ones come out silent (see Status above).
text = "तुमचे काम खूप छान झाले आहे मला मराठी भाषा खूप आवडते आणि मी रोज सराव करतो"
ids = processor(text=prepare(text), return_tensors="pt").input_ids

# Generation is non-deterministic (prenet dropout stays on), so retry on silence.
for attempt in range(5):
    with torch.no_grad():
        wav = model.generate_speech(ids, spk, vocoder=vocoder, maxlenratio=15.0).numpy()
    if np.abs(wav).max() > 0.02:
        break
    print("silent, retrying (%d)" % (attempt + 1))
else:
    print("still silent after 5 tries - use a longer sentence")

sf.write("out.wav", wav, 16000)

CPU is fine — a few seconds of speech takes a second or two, no GPU required.

Input must be Devanagari. Latin characters, digits and punctuation are stripped by prepare() because they were stripped in training.

Speaker embedding

SpeechT5 is speaker-conditioned. This model was trained on a single speaker with one fixed 512-d x-vector (cmu-arctic validation[7306]), shipped here as speaker_embedding.npy. A different embedding will not give you a different voice in any controlled way — it mostly just degrades this one.

Training

Marathi read speech (SPRINGLab/IndicTTS_Marathi, single female speaker, 16 kHz), clips capped at 6 s. Hyperparameters read directly from the run's training_args.bin:

Base model	microsoft/speecht5_tts
Steps	300, warmup 40
Batch size	4, gradient accumulation 1
Learning rate	1e-4, linear schedule
Precision	fp32, CPU, no gradient checkpointing
Weight decay	0.01, max grad norm 1.0
Seed	42
Tokens added	66 (63 Devanagari + <sp> + 2 digits)
Vocabulary	81 → 145

Roughly 1,200 samples seen. For comparison, the same notebook's GPU preset does 3,000 steps at batch 8 (24,000 samples) — about 20× more, which is what this checkpoint is missing.

Limitations

• Undertrained: silent on short inputs, intermittently silent on long ones. See Status.
• Single speaker, single fixed x-vector — one voice only.
• Read studio speech. Expressive, conversational or code-mixed input degrades further.
• Devanagari only. No Latin, digits, or punctuation.
• Always check np.abs(wav).max() before using the output.

Evaluation

The training run measured before/after ASR round-trip CER/WER and a DTW-aligned mel-cepstral distance against ground truth, with facebook/mms-tts-mar as an Indic reference. Those metrics and the before/after audio samples are not in this repo — they remain in the training session's report/ and samples/ folders.

Independently verified post-hoc on this checkpoint: mel std 3.3–4.2 and waveform amplitude 0.47–0.79 on long inputs (healthy), versus mel std ~0.07 and amplitude 0.00002 on short inputs (silence). Base-model reference for scale: mel std 0.73, amplitude 0.26.

Licence

Weights inherit MIT from microsoft/speecht5_tts. The training corpus SPRINGLab/IndicTTS_Marathi declares no licence on the Hub — check with the dataset authors before commercial use.