VoxCPM2 serving engine patches (dq_engine)
Python code, no weights: a VoxCPM2 serving engine (dq_engine) built on top of nano-vllm-voxcpm and tuned for many
concurrent voice-cloning calls. The Arabic dialects project uses it as the baseline that its own engine is
compared against.
At a glance
| Content | 4 Python files and 1 patch, no model weights |
| Runs on | stock nano-vllm-voxcpm 2.0.4 (with a small compatibility patch, see Usage) |
| Model | any VoxCPM2 checkpoint, e.g. VoxCPM2-Arabic-Dialects |
| Used by | arabic_voxcpm.eval.generate_dq in voxcpm2-arabic-training |
Files
| File | What it does |
|---|---|
engine.py |
DqCoreEngine and Request: the engine runs in its own thread; the audio VAE is decoded outside the model step, batched across requests; no per-step host syncs; one batched noise draw per step |
dit_graph.py |
CUDA-graphs the flow-matching decoder (LocDiT) per batch-size bucket, so it is also fast in the prefill path |
prefill_graph.py |
CUDA-graphs the prefill: pads each prefill to a fixed token bucket and replays one graph per (batch, tokens) bucket; falls back to eager when it does not fit |
chunked_prefill.py |
Optional: caps the prompt tokens prefilled per step so a burst of new requests cannot stall running streams |
compat-nanovllm-2.0.4.patch |
Renames the noise argument to z_noise and casts VAE input for the released nano-vllm-voxcpm 2.0.4; apply it only on that version |
Measured on an H100 (from the code's own notes): a prefill step went from ~147 ms to ~85 ms with the graphed LocDiT.
Settings (environment variables)
| Variable | Default | Effect |
|---|---|---|
DQ_ENGINE_PREFILL_GRAPH |
1 |
CUDA-graph the prefill |
DQ_ENGINE_PREFILL_GRAPH_LORA |
0 |
Also graph prefills that use a LoRA adapter |
DQ_ENGINE_PREFILL_BUCKETS |
1:320,2:640,4:1152,8:2176 |
Prefill buckets, requests:tokens |
DQ_ENGINE_MAX_PREFILL_TOKENS |
0 (off) |
Chunked prefill token budget per step (suggested 1024–2048) |
DQ_ENGINE_MAX_PREFILL_SEQS |
0 (off) |
Cap on requests prefilled per step |
DQ_ENGINE_GPU_LATENTS |
1 |
Hand the GPU latents to the VAE thread (no re-upload) |
DQ_ENGINE_VAE_COMPILE |
1 |
torch.compile the VAE decoder |
DQ_ENGINE_VAE_WEIGHT_NORM_FOLD |
1 |
Fold weight norm in the VAE |
DQ_ENGINE_TIMING |
0 |
Print per-step timings |
Usage
hf download VoxCPM-Kernel/VoxCPM2-Serving-Engine-Patches --local-dir <dir>/dq_engine && touch <dir>/dq_engine/__init__.py
cd <dir> && patch -p1 < dq_engine/compat-nanovllm-2.0.4.patch # only on stock nano-vllm-voxcpm 2.0.4
DQ_ENGINE_DIR=<dir> python -m arabic_voxcpm.eval.run --engine dq ... # from voxcpm2-arabic-training
import sys; sys.path.insert(0, "<dir>")
from dq_engine.engine import DqCoreEngine, Request
License
No license was supplied with this code. Ask the repository owners before reusing it outside this project.