AKIlukas/speecht5-german-lora

SpeechT5 German (LoRA fine-tune)

German text-to-speech model: microsoft/speecht5_tts fine-tuned with LoRA (PEFT) on German data from facebook/voxpopuli (config de, CC-0 license).

Demo: demo_german.wav — "Hallo! Ich bin ein deutschsprachiges Sprachmodell. Danke, dass Sie zuhoeren."

Results

• Final eval loss: 0.4998 (held-out 10% split; community full fine-tunes of SpeechT5 on this data reach ≈ 0.34–0.50)
• Training loss: 3.49 (step 1) → 2.10 (step ~2050), eval loss 2.82 → 0.50, stable grad norms, no divergence
• 2,500 steps, effective batch 32 (batch 8 × grad accum 4), lr 1e-4 (500-step warmup + linear decay), bf16, gradient checkpointing

Training data

• 3 parquet shards of VoxPopuli de train (~20k raw utterances), CC-0 license
• Filtered to speakers with 100–400 utterances, clips ≤ 25 s, token length < 200, spectrogram frames < 1500
• German umlauts/ß romanized (ä→ae, ö→oe, ü→ue, ß→ss): SpeechT5's character tokenizer was trained on English text and has no umlaut tokens
• 16 kHz mono, normalized text, 90/10 train/test split

Method

• LoRA r=16, alpha=32, dropout=0.05 on q_proj/k_proj/v_proj/out_proj/fc1/fc2 → 2.36M trainable params (1.6% of 146.8M)
• Merged into the full model before upload — this repo is a standard SpeechT5ForTextToSpeech checkpoint, no adapter needed
• Speaker conditioning via x-vector speaker embeddings (speechbrain/spkrec-xvect-voxceleb, Apache-2.0)

Licenses

Model MIT · Dataset CC-0-1.0 · Speaker model Apache-2.0 · Vocoder microsoft/speecht5_hifigan MIT

Usage

import io, torch, soundfile as sf
from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech, SpeechT5HifiGan

device = "cuda" if torch.cuda.is_available() else "cpu"
processor = SpeechT5Processor.from_pretrained("AKIlukas/speecht5-german-lora")
model = SpeechT5ForTextToSpeech.from_pretrained("AKIlukas/speecht5-german-lora").to(device)
vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan").to(device)

# 512-d x-vector speaker embedding from any German speaker clip:
#   EncoderClassifier.from_hparams("speechbrain/spkrec-xvect-voxceleb")
#   emb = normalize(model.encode_batch(waveform), dim=2).reshape(1, 512).to(device)
speaker_embedding = torch.load("speaker_embedding.pt", map_location=device)  # [1, 512]

text = "Hallo! Ich bin ein deutschsprachiges Sprachmodell."
inputs = processor(text=text, return_tensors="pt").to(device)
with torch.no_grad():
    speech = model.generate_speech(inputs["input_ids"], speaker_embedding, vocoder=vocoder)
sf.write("out.wav", speech.cpu().numpy(), samplerate=16000)

Notes:

• Romanize umlauts/ß as above before passing text (the tokenizer has no umlaut characters).
• The voice is controlled entirely by the speaker embedding — use one extracted from a real German speaker's audio for the best result.