TTS-AGI/acestep-vocals-lora

ACE-Step 1.5 — vocals-only (a cappella) LoRA

A LoRA for ACE-Step 1.5 acestep-v15-base (2B) that makes the model sing unaccompanied when nothing in the prompt told it to. The un-adapted base model, given a prompt that describes only a voice, still builds a backing track. This adapter removes most of it without touching diction.

Recommended	root of this repo — variant A, rank 128 / alpha 256, merge scale 1.0
Base model	ACE-Step/acestep-v15-base (2B). Not XL, not turbo, not SFT.
Effect	accompaniment fraction 0.188 → 0.084 (−55%), 28/40 clips improved, sign-test p = 0.017, n = 40
Cost	1.59 GPU-hours on one GH200 (13.93 across all 8 variants)
Demo	side-by-side: base ACE vs this LoRA vs LeVo 2 — 26 samples, three arms each. The original 120-player base-vs-LoRA grid is linked from the top of that page.

---

⚠️ Read this before you generate anything

ACE-Step's dcw_enabled parameter defaults to True, and that default destroys the CFG checkpoints — including the one this LoRA attaches to.

With the default left alone, a 30-second clip comes back transcribing as the single word "so". Measured against a Whisper transcript:

dcw_enabled	Whisper F1
True (library default)	0.021
False	0.911

Pass dcw_enabled=False in every generation call. infer.py in this repo forces it. If you wire the adapter into your own pipeline and get gibberish, this is why — the adapter is not the problem.

---

Usage

infer.py in this repo runs as-is and pulls the adapter from the Hub on first use.

pip install acestep                # or: git clone ACE-Step-1.5 && pip install -e .
wget https://huggingface.co/TTS-AGI/acestep-vocals-lora/resolve/main/infer.py

python infer.py \
  --caption "A solo female voice sings an Andean huayno, high and keening, with a bright nasal placement, ornamented turns and a piercing sustained upper register." \
  --lyrics "[verse]. The quarry filled with rain the year I left. I watched it take the ladder and the crane." \
  --duration 30 --seed 770001 --out out.flac

Useful flags: --adapter DIR (local copy instead of the Hub), --scale 0.5 (weaker), and --no-lora (the un-adapted baseline, for A/B).

Settings — exactly what the numbers on this page were produced with

base checkpoint	acestep-v15-base (2B)
LoRA merge scale	1.0
inference steps	50
guidance scale (CFG)	7.0
dcw_enabled	False
lyric LM	acestep-5Hz-lm-4B, lm_temperature 0.85
thinking / use_cot_metas	True / True
use_cot_caption / use_cot_language	False / False
vocal_language	en
duration	30 s

50 steps at guidance 7.0 is the CFG-model pairing. The 8-step / guidance-1.0 pairing you will see in ACE-Step examples belongs to the turbo checkpoints and produces mush on base.

Attaching it yourself

from acestep.handler import AceStepHandler
dit = AceStepHandler()
dit.initialize_service(project_root=".", config_path="acestep-v15-base", device="cuda")
dit.load_lora("path/to/adapter")     # directory basename must contain NO dots (see below)
dit.set_use_lora(True)
dit.set_lora_scale(1.0)
assert dit.get_lora_status()["active"]

load_lora() uses the adapter directory basename as the PEFT adapter name, and PEFT rejects names containing dots. Stage the adapter into a dot-free directory first — infer.py does this for you. This is what makes raw training checkpoints (epoch_2_loss_0.7641) unloadable as written.

get_lora_status() returns {'loaded', 'active', 'scale'} — not lora_loaded / use_lora. Asserting the wrong keys silently disables the check, not the adapter.

---

Files

Path	Variant	Base	Rank/α	Notes
adapter_model.safetensors (root)	A	acestep-v15-base	128/256	✅ Recommended. Use this one, at scale 1.0.
variants/B_base2b_r64/	B	acestep-v15-base	64/128	Secondary. Half the rank, statistically indistinguishable at scale 0.5 — rank 128 buys nothing.
variants/H_base2b_gr90/	H	acestep-v15-base	128/256	Secondary. genre_ratio 90. Most consistent variant (30/40, p = 0.002) but a smaller effect; never measured at scale 1.0.
variants/G_base2b_mlp/	G	acestep-v15-base	128/256	Secondary. Also targets the SwiGLU MLP. Nearly 2× the file size, no gain.
variants/F_base2b_lr5e5/	F	acestep-v15-base	128/256	Secondary. LR 5e-5. Weakest variant (p = 0.43).
variants/C_turbo2b_r128/	C	acestep-v15-turbo	128/256	Secondary, different base. Only for turbo. Largest effect of any variant, and it also lifts turbo's lyric recall 0.644 → 0.781.
variants/D_sft2b_r128/	D	acestep-v15-sft	128/256	Secondary, different base. Only for SFT.

Each variant is bound to the checkpoint it was trained on — C and D will not do anything sensible on acestep-v15-base. Variant E (XL/4B base) is absent: it produced NaNs at step 70, inside warmup, and no usable weights. That says XL needs a lower LR, not that XL cannot work.

---

Training

Data — 1,314 clips, 60 genres

Selected from a pool of 8,582 captioned singing clips by a deliberately strict filter:

Filter	Remaining
starting pool	8,582
solo_voice ∧ ¬backing_voices ∧ ¬instruments_audible	2,544
∧ duration ≥ 25 s	1,973
∧ lyrics present	1,314

The last cut is the large one and it is not arbitrary: only one of the source runs carries lyrics, and training a vocal adapter on empty lyric conditioning teaches wordless singing, so those 659 rows were dropped rather than padded. The result is 1,314 clips over 60 genre labels (min 3 / median 20 / max 48 per genre), 25–30 s each, 48 kHz stereo, with [verse]/[chorus] markers and long prose captions describing the voice — register, vibrato, attack, mic distance — rather than tag lists.

genre_ratio = 0: captions are used in full. The Side-Step docs suggest genre_ratio 90 (swap the caption for a bare genre tag on 90% of samples) for style LoRAs, but the prompts this adapter is meant to fire on are ~300-character prose. Variant H tests that choice and does not overturn it. No trigger word — the adapter should improve vocals-only rendering unconditionally.

Synthetic-source caveat. The training audio is generated, not recorded: all 1,314 clips are LeVo 2 output, captioned by an audio LLM. This adapter distils LeVo 2's a cappella behaviour into ACE-Step. It learns the vocal character of a synthetic singer and inherits that generator's artefacts and biases. It is not trained on, and does not clone, any real performer.

Hyperparameters (variant A)

adapter	LoRA, rank 128 / alpha 256, dropout 0.1
target modules	q_proj k_proj v_proj o_proj (attention only, self + cross)
optimizer	adamw, LR 1e-4, cosine schedule, 100 warmup steps
weight decay / grad clip	0.01 / 1.0
batch	1 × gradient accumulation 4
schedule	12 epochs = 3,940 optimizer steps
timestep sampling	fixed (continuous logit-normal, μ −0.4 / σ 1.0, from the model's own config.json)
CFG dropout	0.15
seed	42
cost	1.59 GPU-hours, one GH200

python train.py --plain -y fixed \
    --checkpoint-dir <ckpts> --model-variant acestep-v15-base \
    --dataset-dir <preprocessed-tensors> --output-dir <out> \
    --batch-size 1 --gradient-accumulation 4 \
    --optimizer-type adamw --scheduler-type cosine --warmup-steps 100 \
    --weight-decay 0.01 --max-grad-norm 1.0 --seed 42 --save-every 2 \
    --rank 128 --alpha 256 --lr 1e-4 --epochs 12 --dropout 0.1 \
    --cfg-ratio 0.15 --adapter-type lora --attention-type both

All eight variants were trained as independent single-GPU processes, four to a node — Lightning Fabric was not installed and the trainer degrades to a single-device loop with only a warning, so four different variants per node was a better use of whole-node billing than one faster run. Total 13.93 GPU-hours for all eight.

What was held out

No training clips were held out. All 1,314 went into training. Evaluation is instead on 40 prompts written specifically for it — 20 style captions × 2 lyric takes — and those captions and lyrics appear nowhere in the training corpus (verified: 0 of 20 captions and 0 of 2 lyric texts occur in the training set). So the evaluation measures generalisation to unseen prompts, not held-out audio, and it cannot detect memorisation of training clips.

---

Evaluation

Metric. accomp_frac = RMS(instrumental stem) / (RMS(instrumental) + RMS(vocal)) after BS-Roformer separation (ep.317, SDR 12.98). 0 = pure a cappella, 0.5 = instruments as loud as the voice. Separation artefacts are real but identical across both arms of a pair, so the trustworthy quantity is the within-pair difference, not the absolute level.

Design. 40 prompts (n = 40), each generated twice with identical caption, lyrics and seed in both arms, so every comparison is paired. p-values are two-sided exact sign tests over the 40 paired differences; CIs are 10k-resample bootstraps of the mean difference.

These prompts describe only the voice and never mention instruments. That condition matters: on the other grid, whose captions all say "no instruments are present", the un-adapted model already scores 0.02 and there is no headroom for any adapter to show anything (22/40, p = 0.64). Building the neutral grid was the difference between a null result and a measured one.

Variant A (recommended)

arm	accomp. fraction	Δ paired	improved	p (sign)	95% CI	lyric recall
un-adapted base	0.1880	—	—	—	—	0.8614
LoRA @ 1.0	0.0838	−0.1042	28/40	0.017	[−0.153, −0.057]	0.8585
LoRA @ 0.7	0.1264	−0.0616	25/40	0.154	[−0.109, −0.017]	0.8690
LoRA @ 0.5	0.1361	−0.0519	25/40	0.154	[−0.096, −0.011]	0.8760

At scale 1.0 that is a 55% relative reduction in accompaniment energy. The effect grows monotonically with scale over the range tested. Lyric word recall (Parakeet ASR) is flat — 0.861 → 0.859 — with 0 collapsed clips out of 160, so the a cappella behaviour is not bought by degrading diction.

All variants

Each is paired against the un-adapted arm of its own base checkpoint, n = 40:

variant	scale	un-adapted	with LoRA	Δ	improved	p	recall
A r128	1.0	0.1880	0.0838	−0.1042	28/40	0.017	0.861 → 0.859
A r128	0.7	0.1880	0.1264	−0.0616	25/40	0.154	→ 0.869
A r128	0.5	0.1880	0.1361	−0.0519	25/40	0.154	→ 0.876
B r64	0.5	0.1880	0.1318	−0.0562	28/40	0.017	→ 0.875
G +MLP	0.5	0.1880	0.1242	−0.0638	26/40	0.081	→ 0.874
H gr90	0.5	0.1880	0.1284	−0.0597	30/40	0.002	→ 0.857
F LR 5e-5	0.5	0.1880	0.1477	−0.0404	23/40	0.430	→ 0.867
C turbo	0.5	0.4652	0.3769	−0.0883	33/40	<1e-4	0.644 → 0.781
D SFT	0.5	0.2336	0.2064	−0.0272	28/40	0.017	0.879 → 0.870

Why A at 1.0 is the shipped choice: it has the largest paired effect of any arm on acestep-v15-base (−0.104, roughly double every scale-0.5 arm) at p = 0.017. The honest qualification is that only A was evaluated at scale 1.0 — H is the most consistent variant at 0.5 (30/40, p = 0.002) with a smaller effect, and it was never tested at 1.0, so "A is best" is a statement about the arms that were actually measured, not a proof that A dominates H.

Secondary findings: rank 128 buys nothing over 64; LR 5e-5 is the weakest setting; genre_ratio 90 did not hurt, contradicting the prediction made when the dataset was built; and un-prompted a cappella compliance runs base < SFT < turbo, so the choice of base checkpoint matters more than the adapter does.

---

Limitations

• Style flattening at scale 1.0 is unmeasured. It is the documented failure mode of timbre LoRAs at high merge scale, and an energy-ratio metric is structurally blind to it. This is the main open risk on the "ship at 1.0" recommendation — judge it by ear on the demo page, which carries scale 1.0 next to the un-adapted baseline.
• The evaluation uses 2 distinct lyric texts across its 40 clips (20 captions × 2 takes). Prompt diversity is real; lyric diversity is not.
• No human listening study. Every number here is automatic (source separation + ASR).
• 2B only. Transfer to the XL/4B checkpoints is untested; the one XL run NaN'd in warmup.
• Nothing else improved. The adapter does not improve pitch, musicality or diction — it removes accompaniment nobody asked for, and that is the whole claim.
• Prompts that already say "no instruments are present" get no measurable benefit: the base model already complies there.

Provenance

Trained and evaluated on JUWELS/JUPITER GH200 nodes. Training audio is LeVo 2 output (see the synthetic-source caveat above). Apache-2.0, following the base model.