VoxCPM2 — Swedish LoRA (Sturmark audiobook)

A LoRA adapter for openbmb/VoxCPM2 fine-tuned on a professionally narrated Swedish audiobook (~7.5 hours, 5,139 (clip, transcript) pairs, single speaker) to make the base model sound natural in Swedish — correct alveolar R, Swedish prosody and compound-word stress that the base model (trained mostly on other languages) does not produce.

The base model was evaluated against Zonos 3 Mini, KugelAudio, Qwen3-TTS and Piper on Swedish — none produced natural Swedish speech. This adapter closes that gap for one voice. Read the full evaluation story on the Berget AI blog.

Rights

The audiobook narration was contributed for ML training by the narrator without compensation (agreement on file with Berget AI, 2026-11). Do not use the adapter to generate speech intended to impersonate the narrator.

Usage

import json, torch, soundfile as sf
from huggingface_hub import snapshot_download
from voxcpm import VoxCPM
from voxcpm.model.voxcpm2 import LoRAConfig

ckpt = snapshot_download("bergetai/sten-lora")

# NOTE: lora_config.json is nested: {"base_model": ..., "lora_config": {...}}
raw = json.load(open(f"{ckpt}/lora_config.json"))
lora = LoRAConfig(**{k: v for k, v in raw["lora_config"].items()
                     if k in LoRAConfig.model_fields.keys()})

model = VoxCPM.from_pretrained(
    "openbmb/VoxCPM2", load_denoiser=False,
    device="mps",  # cuda / cpu also work
    optimize=False,
    lora_config=lora,
    lora_weights_path=ckpt,
)

wav = model.generate(
    text="Hej! Det här är en svensk talsyntes tränad på öppen data.",
    cfg_value=2.0, inference_timesteps=10)
sf.write("out.wav", wav, model.tts_model.sample_rate)

pip install voxcpm soundfile (Python 3.12, torch ≥ 2.5).

Training details

  • Base: openbmb/VoxCPM2 (2B, tokenizer-free diffusion AR)
  • Data: 5,139 segments (4,883 train / 256 val), 2–15 s each, 24 kHz source resampled to the AudioVAE encoder rate (16 kHz), transcripts from forced alignment via KBLab/kb-whisper-large
  • Method: LoRA r=16, alpha=32, enable_lm + enable_dit, lr 1e-4, batch 1 × grad-accum 4, 4,000 steps (~3.3 epochs), fp32
  • Hardware: 1× AMD MI300X (ROCm 7.2) — see ROCm notes below

ROCm notes (if training/serving on AMD)

  • Install torch + torchaudio as a matched pair from the ROCm index: pip install torch==2.11.0+rocm7.2 torchaudio==2.11.0+rocm7.2 --index-url https://download.pytorch.org/whl/rocm7.2
  • Mixed bf16/fp32 crashes in forward pass — force fp32 (patch _dtype() / cast the model; amp autocast does not save you on ROCm here)
  • train_voxcpm_finetune.py takes --config_path (argbind YAML), not inline kwargs

Limitations

  • Numbers are best spelled out in the source text ("fyrtiotre" rather than "43")
  • Sentence-level line breaks in the source text produce natural pauses
  • Long texts: raise max_len (default 4096) or synthesize per sentence
  • The adapter carries the narrator's timbre and Swedish prosody — it does not make the base model multilingual
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for bergetai/sten-lora

Base model

openbmb/VoxCPM2
Adapter
(43)
this model