Rabe3/voxcpm2-full-youtube

VoxCPM2 — Saudi Dialectal Arabic (full fine-tune)

Full fine-tune of openbmb/VoxCPM2 on Saudi dialectal Arabic from Rabe3/youtube-300.

All 2.29B non-VAE parameters were trained (no LoRA). Arabic is already one of VoxCPM2's 30 supported languages, so this is dialect and style adaptation, not new-language training.

Checkpoints

Seven checkpoints are published, one per 100 optimizer steps:

folder	step	epoch	val loss
step_0000600	600	1.37	0.7881
step_0000700	700	1.60	0.7868
step_0000800	800	1.83	0.7836
step_0000900	900	2.06	0.7775
step_0001000	1000	2.29	0.7805
step_0001100	1100	2.52	0.7842
step_0001200	1200	2.75	0.7828

Validation loss reached its minimum at step 500 (0.7738), before the earliest published checkpoint; step 1200 is 1.2% above that minimum. The differences across this range are small and within run-to-run noise — pick by listening, not by val loss. In our measurements steps 600 and 1200 tied for the best ASR CER.

Earlier checkpoints (steps 100–500) were not retained by the run's keep_last: 8 policy. Optimizer state is not published, so these cannot be used to resume training.

Training data

Important: this run used 4 of the dataset's 47 shards — the number uploaded at the time the data was prepared. The full dataset is roughly 132 h.

clips	6,993 train / 286 val
audio	10.45 h train / 0.47 h val
speakers	18 train / 3 val (speaker IDs are per-video)
videos	21
clip length	2.0–12.4 s (mean 5.38 s)
sample rate	24 kHz source, resampled to 16 kHz for the AudioVAE encoder
prompted rows	2,853 (40.8%) carry a same-speaker ref_audio

Validation is held out by whole video, so val speakers are unseen in training. Transcripts are YouTube auto-captions: only 4.3% contain any punctuation and 0% contain parentheses. This matters at inference — see Limitations.

Training configuration

learning rate	1e-5, cosine to 0, 120 warmup steps
effective batch	16 (batch_size 8 × grad_accum_steps 2)
steps	1200 (2.75 epochs)
weight decay	0.01
grad clipping	max norm 1.0
precision	fp32 master weights, bf16 autocast
loss weights	loss/diff 1.0, loss/stop 1.0
hardware	single H100, ~28 min wall clock

Usage

from voxcpm import VoxCPM
import soundfile as sf

model = VoxCPM(voxcpm_model_path="Rabe3/voxcpm2-full-youtube/step_0001200")

wav = model.generate(
    text="يا زين طلعة الصبح بدري يوم يكون الجو هادي والهواء بارد شوي",
    reference_wav_path="reference.wav",
    cfg_value=1.8,
    inference_timesteps=16,
)
sf.write("out.wav", wav, model.tts_model.sample_rate)

cfg_value=1.8 measured better than the upstream-recommended 1.6 on this checkpoint (CER 0.039 vs 0.053).

Limitations

Strip punctuation from your input. With only 4.3% of training transcripts carrying any punctuation and none carrying parentheses, punctuation is out of distribution and the model answers with hallucinated syllables at those positions. Removing it before synthesis moved mean CER from 0.051 to 0.008 on a long-form test.

Chunk anything longer than ~12 s. The longest training clip is 12.4 s and the stop head is now confident about that duration distribution (stop loss fell from 0.099 to 0.0001). A single 47 s pass drifts badly: SNR fell from ~37 dB in the first 10 s to ~10 dB by the end, with the speaking rate accelerating. Synthesize sentence by sentence against the same reference and concatenate. The cost is a prosody discontinuity of roughly 4.9 semitones at chunk boundaries — present in base VoxCPM2 too (5.3 st), so it is inherent to stitching independent generations, not caused by this fine-tune.

Style control instructions do not work reliably. The (instruction)text control prefix produces effects (±0.3–1.2 st of pitch variance) no larger than seed-to-seed noise on this model. Use the reference clip to control delivery.

Prosody is flatter than base. Pitch variance drops from 3.65 st (base) to 2.27 st here, measured over 3 seeds. This appeared by step 600 and did not deepen with further training, so it reflects the narrow training distribution — 21 videos from one channel — rather than overfitting. The train/val gap is only 2.4%.

Dialect coverage is limited by the 4-shard subset. A run over all 47 shards would be expected to generalize considerably better.

Reproducing

Data prep, manifest building and training scripts are in the VoxCPM fork used for this run:

python scripts/prepare_youtube300.py --workers 8
python scripts/build_youtube300_manifests.py
python scripts/train_voxcpm_finetune.py \
  --config_path conf/voxcpm_v2/voxcpm_full_youtube300.yaml

prepare_youtube300.py reads the shard list from the dataset repo at runtime, so re-running it now picks up all 47 shards. Note that num_iters: 1200 in the config was sized for 6,993 rows (437 steps/epoch); at full scale one epoch is ~5,300 steps and the schedule needs resizing accordingly.