ampixa/voxcpm2-nepali

VoxCPM2 Nepali LoRA (ampixa)

Two LoRA adapters that fine-tune openbmb/VoxCPM2 (revision 32279eff) for Nepali zero-shot voice cloning, plus a small inference script and samples.

Headline: this does NOT beat Swar. On the public Swar head-to-head bench (492 jobs, unchanged scorer), our best systems improve stock VoxCPM2 a lot, but they are behind Swar NFE16 on every suite (table below). Two of the three suites use FLEURS and Rasa prompts, and Swar was trained on FLEURS and Rasa while we excluded both; clean cloning is the fairest comparison, and we are behind there too (6.49 vs 5.06 CER). We publish this as a documented baseline, not as a state-of-the-art claim.

file	what
adapters/run1_ivr_step3500/	main adapter. LoRA r=32 on LM + DiT attention, trained 3,500 steps on 46.8 h of IndicVoices-R Nepali
adapters/run2_d2_step2500/	variant. Warm-started from the main adapter's run, 2,500 more steps on 151.9 h (IndicVoices-R + extra public Nepali, see Data)
infer.py	cloning script with the settings used for the bench runs (cfg 3.0, reference VAD trim, optional best-of-N)
configs/	training configs (official VoxCPM LoRA recipe values) and the frozen inference settings of both bench runs
samples/	eight single-take samples from our own sentences (six fine-tuned, two stock VoxCPM2 for comparison; none from any benchmark)

Listen

Full listening page: ampixa/voxcpm2-nepali-listen: NepTTS-Bench (studio + Maya), our held-out NepTTS-Fresh v1 set (with real recordings), a fixed excerpt of the Swar bench, each clip next to Swar on the same input, with its score.

Samples in this repo (single take, default infer.py settings, our own sentences, our own synthetic voices as prompts). Two sentences also have a stock VoxCPM2 take with the same voice and seed: on easy sentences like these, stock is already intelligible; the gap shows on the benchmarks (see the listening page).

text	voice	model	audio	internal-ASR CER %
नमस्ते, म तपाईंलाई कसरी सहयोग गर्न सक्छु?	Pooja	main (run 1)		2.6
नमस्ते, म तपाईंलाई कसरी सहयोग गर्न सक्छु?	Pooja	stock VoxCPM2		2.6
काठमाडौं उपत्यकामा आज दिउँसो हल्का वर्षा हुने सम्भावना छ।	Pooja	main (run 1)		1.8
काठमाडौं उपत्यकामा आज दिउँसो हल्का वर्षा हुने सम्भावना छ।	Pooja	stock VoxCPM2		1.8
भोलि बिहान दस बजे तपाईंको भेटघाट तय भएको छ।	Sunita	main (run 1)		9.5
कृपया आफ्नो खाता नम्बर फेरि एकपटक भन्नुहोस्।	Sunita	main (run 1)		2.3
हिमालको काखमा बसेको यो सानो गाउँ साँच्चै सुन्दर छ।	Pooja	main (run 1)		4.1
हिमालको काखमा बसेको यो सानो गाउँ साँच्चै सुन्दर छ।	Pooja	variant (run 2)		0.0

Use

pip install "git+https://github.com/OpenBMB/VoxCPM@cce58a5b59303c9bd63d12f23afe6a49a4a80c59" huggingface_hub soundfile
python infer.py --text "नमस्ते, म तपाईंलाई कसरी सहयोग गर्न सक्छु?" \
    --ref_wav ref.wav --ref_text "<exact transcript of ref.wav>" --out out.wav
# --adapter run2 for the variant, --adapter none for stock VoxCPM2, --n 4 for best-of-4 MMS rerank (user convenience; needs transformers + librosa)

Cloning quality depends on the reference: use 4-12 s of clean speech and its exact transcript. A CUDA GPU with about 8 GB is enough. On an NVIDIA L40S (bf16, batch 1) a single take runs at RTF about 0.40, with first streamed audio after about 0.12 s. Bad-case retries (VoxCPM's own) can add takes.

Every bench number in this card is either a single take or best-of-8 reranked by our internal ASR (LONG1C, not public). None used MMS for selection. MMS (facebook/mms-1b-all) is the bench's own scoring ASR, so reranking with it would grade the takes with the same model that scores them.

--n > 1 in infer.py reranks with MMS (Nepali adapter). This is a user convenience; do not use it when scoring with MMS. If you rerank with MMS and then score with an MMS-based benchmark, the numbers will look better than they are.

Results on the Swar head-to-head bench

Bench: 205 NepTTS sentences with a FLEURS-style noisy prompt, the same 205 with a Rasa studio prompt, and 82 clean cloning items from the Nepali Voice Cloning Benchmark (4 FLEURS ne test speakers). Scorer unchanged: MMS CER (NepTTS: mean sentence CER; cloning: corpus CER) and ECAPA voice match. Lower CER is better, higher voice match is better. Brackets are 95 % bootstrap intervals over sentence clusters.

System	FLEURS-prompt CER	Rasa-prompt CER	clean-cloning CER	voice match
Stock VoxCPM2, cloning, single take	50.52 [39.83-63.46]	15.98 [13.59-18.63]	9.59 [7.97-11.30]	.685 [.668-.703]
Stock VoxCPM2, voice design only (no reference)	69.49	69.49	17.30	n/a
+ main adapter (run 1), native settings, single take	33.58 [29.49-37.70]	12.10 [10.01-14.41]	6.94 [5.61-8.35]	.675 [.657-.692]
+ main adapter (run 1), cfg 3.0 + ref trim, best-of-8 (bench run 1)	20.87 [17.72-24.16]	9.34 [7.63-11.21]	6.49 [5.31-7.76]	.672 [.657-.688]
+ variant adapter (run 2), same settings, best-of-8 (bench run 2)	19.13 [15.87-22.60]	11.00 [8.70-13.53]	6.57 [5.12-8.16]	.678 [.665-.691]
Swar NFE16, single take (the bar)	17.24 [13.91-20.87]	6.72 [5.35-8.24]	5.06 [3.90-6.40]	.687 [.670-.704]
Swar best-of-4 (its own reranker)	16.19 [13.14-19.40]	6.96 [5.48-8.62]	5.13 [4.01-6.40]	.69 [.67-.71]
Real recordings (calibration, not a floor)			9.77 [7.90-11.87]	.740 [.72-.76]

NepTTS-Bench with Swar's own metric (Ampixa/neptts-bench, 205 scored sentences; Naamche-Labs/swar @4d2a7406 tools/rescore_deva.py + tools/pron_cer.py). Our harness gives 6.21 / 7.45 on Swar's published audio (Swar card: 6.19 / 7.45), so it reproduces the published numbers.

System (studio voice prompt)	standard CER	pronunciation CER
Swar, published audio	6.21	2.97
Stock VoxCPM2, single take	15.98	13.72
+ main adapter, best-of-8 (bench run 1 audio)	9.34	7.28
+ variant, best-of-8 (bench run 2 audio)	11.00	8.49
Swar, Maya voice, published audio	7.45	4.52

Our studio rows reuse the audio of the bench's Rasa-prompt suite (the same 205 sentences and prompt). We did not run VoxCPM2 with the Maya voice: the GPU machine was stopped first.

NepTTS-Fresh v1 (our held-out set: 12 speakers from public Nepali YouTube videos, audio no system trained on). VoxCPM2 finished the 96 real_yt items: a speaker's prompt, then a different sentence that speaker really said. CER from our internal ASR, paired over 96 items: real recordings 8.71, Swar 8.81, ours single take 11.43, ours best-of-8 8.03. The best-of-8 take is picked by that same ASR, so its number is optimistic; the single take is the fair comparison. All clips are on the listening page.

The comparison is tilted toward Swar on two of the three suites. Swar's own model card (Aarjan/swar, revision 8ca6f1bb) lists its fine-tuning data as AI4Bharat Rasa, IndicVoices-R, Google FLEURS, OpenSLR 43 and 54, LibriTTS-R and private podcast speech. The FLEURS-prompt suite uses FLEURS audio as the prompt, and the Rasa-prompt suite uses a reference clip from the Rasa training split. So Swar has probably heard that material, or audio very like it, in training. We removed every FLEURS sentence (all splits) and every Rasa sentence from our training data. Clean cloning (4 FLEURS-test voices that Swar's card calls unseen) is the fairest of the three suites. Swar's card does not say which FLEURS splits it trained on.

Our harness reproduces Swar's published numbers closely:

Swar	published (Swar card)	measured here
Studio-voice prompt (NepTTS-Bench, 205 sentences, one take)	6.19 % character error	6.72 (Rasa-prompt suite)
"Maya" voice prompt	7.45 %	not run
Clean cloning	4.72 % at voice match .697 (bench v1.1, 64 prompt + sentence pairs)	5.06 at .687 (82 items; intervals include the published values)

Read with care:

• Not equal compute. The best-of-8 rows generate 8 takes per job and pick the one with the lowest character error rate of our internal ASR against the input text. Measured on run 1: 16.7 s per job (11.0 s generation + 5.6 s ASR), RTF 4.8 versus 0.41 for a single take. Swar's row is one take.
• Not a blind test. Settings were tuned on a separate DEV split (FLEURS dev + IndicVoices-R test text + held-out speakers), but the bench had already been seen for the stock, single-take and run-1 rows before run 2 was chosen. This is a fixed-benchmark result, not evidence of population-level ranking.
• The FLEURS-prompt suite is the weak spot. With short target sentences after a long noisy prompt, the model sometimes keeps talking (continues the prompt's own text) or babbles. Best-of-8 removes most but not all of these.
• Voice match is where the adapters lose ground: the LoRA lowers it slightly versus stock (.685 -> .675). The variant (run 2) recovers it to .678 and is better on FLEURS-prompt (19.13 vs 20.87) but worse on Rasa-prompt (11.00 vs 9.34) and clean cloning (6.57 vs 6.49), so run 1 stays the main adapter. Run 2 cost 16.4 s per job (RTF 4.3).

Data

• Main adapter (run 1): ai4bharat/IndicVoices-R Nepali train split (revision 5f4495c9, CC BY 4.0), filtered to 20,096 clips / 46.8 h / 675 speakers: 3-15 s, upstream CER <= 5 %, SNR >= 40 dB and C50 >= 45 dB, Devanagari-only verbatim transcripts, 6-20 characters/s, at most 40 clips per speaker. Half of the rows carry another clip of the same speaker as the isolated cloning reference.
• Variant (run 2): the run-1 data plus 55,144 clips (~105 h) from voidash/nepali-tts-ready (revision 26011e0f): asr_base 53.1 h and tts_silver 48.6 h (single-speaker segments of public YouTube/podcast audio with ASR-derived transcripts), Jenny038/nepali-training-data 2.3 h, Bijay13 podcast 0.7 h, unique_promotion 0.3 h. Kept only clips whose transcript our ASR reproduced at CER <= 15 %, SNR >= 25 dB, deduplicated, capped per channel. Licence caution: the YouTube/podcast-derived lanes carry no explicit licence from their original creators. If that matters for your use, use the main adapter (run 1), which is trained only on CC BY 4.0 data.
• No call-centre or customer audio was used anywhere.
• Exclusions: every bench sentence, every FLEURS ne_np sentence (all splits) and every Rasa Nepali sentence was removed from training by normalized-text hash and long-substring match, and bench/FLEURS/Rasa sources were excluded by source id. Two public sources (shunyalabs, pujanpaudel) contained FLEURS sentences and were dropped entirely. Bench reference voices were screened against training speakers with ECAPA (max cosine .572 for run 1).

Limitations

• Does not beat Swar on any suite (see table).
• Split audit incomplete. An external review (Codex) asked for a fuller split audit before any "uncontaminated" claim: Rasa substring overlap (only exact Rasa matches were checked), DEV short targets against training-text prefixes, and a calibrated speaker check between DEV, bench and run-2 speakers. That audit was not completed. Treat the exclusion statement above as what was checked, not as proof of no overlap. The pretraining data of VoxCPM2 itself is unknown to us.
• Nepali only was evaluated. Code-mixed English words are read with mixed quality.
• Short targets with long noisy references can run on past the end of the sentence.
• Voice cloning can be misused. Only clone voices you have permission to use.

What is next

• Close the split audit above before anyone cites these numbers as clean.
• The remaining gap to Swar is pronunciation on short sentences (FLEURS-prompt suite) and voice match. Candidates: better text normalization (numbers, English words), more clean read Nepali with verified licences, and a reranker with a speaker term, all tuned on DEV only.
• We stopped further work on this line for now. Questions and issues welcome on the repo.

Licence

Base model VoxCPM2: Apache-2.0. Adapters, script and samples here: Apache-2.0. Training data keeps its own licences (IndicVoices-R CC BY 4.0 with attribution to AI4Bharat; see the Data section for the run-2 sources). Samples use our own synthetic agent voices as references.

Provenance

How these adapters were made is recorded in a private internal repository, so that any copy of them can later be matched to its training data, settings and evaluation receipts. It is not public: it holds internal plans, ids of training clips and host paths. It holds no audio, no transcript text and no model weights.

item	value
private record	voidash/voxcpm2-nepali-provenance (Hugging Face dataset, private)
record commit	341bc670721db91697bfe70c3ddd9aeaa2cad043 (first release: 24f7e2c29c0413d08475ce9321648400534fef6f)
provenance fingerprint (sha256 of its MANIFEST.sha256, 343 files)	ba3e5897a5b4c7d022a189c1889ca88519728cb8cd8d3f25d6381322e40f896f (first release: 61683e2e727444a80c8b648abcd4c262cda91a874b6782ee158d7afa0e0fd1dd)
adapters/run1_ivr_step3500/lora_weights.safetensors sha256	ec5eb16fda2781bd410a21dbd063d944b203f9218e07937ebf7d807fad03e287
adapters/run2_d2_step2500/lora_weights.safetensors sha256	21193c16dd1de8c5794450ba33adcad2277e337f3a2734711fece75093383e55
listening page	Space ampixa/voxcpm2-nepali-listen, commit 72197b2d20f6a5bbf2965e0400ac2f66fbeae544

To check a copy, compare its lora_weights.safetensors sha256 with the table above.