Tohirju/chatterbox-mtl-kazakh

Chatterbox Multilingual — Kazakh (t3 fine-tune)

Chatterbox Multilingual v3 fine-tuned for Kazakh (Cyrillic) by Saidzoda Lab.

Only the t3 text-to-speech-token transformer was fine-tuned; s3gen and ve (voice encoder) are the frozen upstream weights. The model does voice cloning: a reference wav is required at inference time. Outputs are Perth-watermarked (the upstream Chatterbox perceptual watermarker is applied to generated audio).

Training

Setting	Value
Base	Chatterbox Multilingual v3 (t3_mtl23ls_v2, stock 2454-token vocab)
Trained modules	t3 only (s3gen / ve frozen)
Precision	fp32
Learning rate	1e-5
Steps trained	12,500
Shipped checkpoint	step 6,000 (winner of a checkpoint sweep)
Data	100k utterances / 183.7 h from the Kazakh Speech Corpus 2 (ksc2)
Held out	FLEURS-kk was held out of training (used only for evaluation)
Language token	ru (Kazakh is out-of-vocab for Chatterbox; the Russian token fits Kazakh Cyrillic best)

Results

We report CER (character error rate), not WER. Kazakh is highly agglutinative, so word-level error rate is unfairly harsh — a single differing suffix flips a whole word to wrong. In addition, our Central-Asian fine-tuned Whisper scorer is broken for Kazakh (it drifts to Latin script and hallucinates), so it cannot be used here. Evaluation therefore uses base whisper-large-v3 with language="kk" as the transcriber.

Round-trip protocol on FLEURS-kk (n=60): synthesize the reference text, transcribe with base whisper-large-v3 (kk), then compute folded (case/space-normalized) CER against the reference.

Model	Median CER	Mean CER
This model (Chatterbox-mtl-kazakh, step 6000)	7.4%	10.2%

Per-utterance evidence is in eval/per_utt.jsonl; the CER summary is in eval/scores_cer.txt.

Usage

Download the repo files into a directory, then load with the chatterbox-tts package (pip install chatterbox-tts, or add a source checkout to sys.path):

import torchaudio
from huggingface_hub import snapshot_download
from chatterbox.mtl_tts import ChatterboxMultilingualTTS

ckpt_dir = snapshot_download("Tohirju/chatterbox-mtl-kazakh")

# Workaround for newer transformers: the loader expects eager attention.
# If loading fails with an attention-implementation error, patch the t3 config
# before/while loading: cfg._attn_implementation = "eager"

model = ChatterboxMultilingualTTS.from_local(ckpt_dir, device="cuda")

wav = model.generate(
    "Сәлеметсіз бе! Бұл қазақ тіліндегі дауыстық үлгі.",
    language_id="ru",                  # Kazakh uses the "ru" language token
    audio_prompt_path="ref.wav",       # REQUIRED: reference speaker wav (voice cloning)
)
torchaudio.save("out.wav", wav, model.sr)

Notes:

• audio_prompt_path (a reference wav of the target speaker) is required — this is a voice-cloning model.
• Pass language_id="ru" — the model was fine-tuned under the Russian language token (Kazakh Cyrillic).
• With newer transformers versions the loader needs eager attention (cfg._attn_implementation = "eager").
• Generated audio is Perth-watermarked by the upstream Chatterbox pipeline.

Files

File	Role
t3_mtl23ls_v2.safetensors	Fine-tuned t3 (Kazakh)
s3gen.pt	Upstream speech-token-to-wav generator (frozen)
ve.pt	Upstream voice encoder (frozen)
conds.pt	Default conditioning
grapheme_mtl_merged_expanded_v1.json	Grapheme tokenizer

License and data

• Model / code: MIT (upstream Chatterbox is MIT).
• Training data: Kazakh Speech Corpus 2 (ksc2) — Khassanov et al., ISSAI, Nazarbayev University. Downstream users should observe the ksc2 license/attribution for the training data source.

Citation

Saidzoda Lab, Central-Asian speech program (Tajik / Uzbek / Kazakh / Kyrgyz), 2026.