⚡LFM3-1.7B-DSpark: Up to 3.1× faster LFM3-1.7B inference with a DSpark draft model (vLLM + llama.cpp)
LFM3-1.7B-DSpark
DSpark Model: https://huggingface.co/rasyosef/LFM3-1.7B-DSpark
GGUFs: https://huggingface.co/rasyosef/LFM3-1.7B-DSpark-GGUF
A lossless DSpark draft model for LiquidAI/LFM3-1.7B, trained with speculators (training code).
- Up to 3.14× faster on math and 2.63× on code, 2.88× on average (default generation params)
- Mean acceptance length 4.10 on math and 3.24 on code, 2.83 overall (block size 8)
Usage
vLLM
Requires vLLM from PR #66323 (Python-only changes, so precompiled kernels can be reused):
VLLM_USE_PRECOMPILED=1 uv pip install \
"vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/66323/head"
Then serve:
vllm serve LiquidAI/LFM3-1.7B --port 8000 \
--speculative-config '{"model": "rasyosef/LFM3-1.7B-DSpark", "num_speculative_tokens": 8}'
llama.cpp (GGUF)
A GGUF build of this drafter is available at rasyosef/LFM3-1.7B-DSpark-GGUF. Pair it with a LFM3-1.7B GGUF target:
llama-server -hf LiquidAI/LFM3-1.7B-GGUF:Q8_0 \
-hfd rasyosef/LFM3-1.7B-DSpark-GGUF:Q8_0 \
--spec-type draft-dspark --spec-draft-n-max 8 --spec-draft-n-min 0 \
-fa on -ngl 99
Details
- ~0.3B params, bfloat16, block size 8
- Draft vocab 32,000; aux hidden states from verifier layers 3/11/19/23
- 100,000 Open PerfectBlend prompts (filtered) with verifier-generated responses, sample length 2560, packed to 8192
- 4 epochs, lr 4e-4, cosine (4% warmup), 98/2 train/val split, loss
{"ce": 0.1, "tv": 0.9}
Evaluation
RedHatAI/speculator_benchmarks, default generation params (temperature 0.1, top_k 50, repetition_penalty 1.1 from the model's generation_config.json), single A100 40GB, concurrency 1.
Throughput (tokens/s)
Mean output throughput. Baseline is LFM3-1.7B without speculative decoding.
| subset | baseline (no drafter) | DSpark (this model) | speedup |
|---|---|---|---|
| math_reasoning | 220.8 | 692.9 | 3.14× |
| HumanEval | 221.0 | 580.2 | 2.63× |
| average | 220.9 | 636.6 | 2.88× |
Acceptance at default params
acceptance_length is mean tokens committed per verification step, including the bonus token (max 9.0 at block size 8).
| subset | acceptance_length | pos_0 | pos_1 | pos_2 | pos_3 | pos_4 | pos_5 | pos_6 | pos_7 |
|---|---|---|---|---|---|---|---|---|---|
| math_reasoning | 4.101 | 78.5% | 61.1% | 47.7% | 37.4% | 29.3% | 23.3% | 18.4% | 14.5% |
| HumanEval | 3.243 | 71.4% | 49.1% | 34.4% | 24.3% | 17.4% | 12.4% | 8.9% | 6.3% |
| rag | 2.971 | 68.7% | 46.4% | 31.1% | 20.7% | 13.4% | 8.4% | 5.2% | 3.1% |
| tool_call | 2.684 | 64.4% | 39.7% | 24.9% | 15.8% | 10.2% | 6.5% | 4.1% | 2.7% |
| translation | 2.630 | 62.6% | 39.4% | 24.4% | 15.1% | 9.4% | 6.0% | 3.7% | 2.3% |
| writing | 2.586 | 61.4% | 36.7% | 22.7% | 14.7% | 9.7% | 6.3% | 4.2% | 2.9% |
| question | 2.573 | 61.6% | 36.7% | 22.4% | 14.2% | 9.4% | 6.1% | 4.2% | 2.8% |
| summarization | 2.467 | 63.1% | 37.4% | 21.4% | 12.4% | 6.7% | 3.4% | 1.6% | 0.7% |
| qa | 2.446 | 61.6% | 36.2% | 21.0% | 12.0% | 6.8% | 3.8% | 2.1% | 1.2% |
Weighted across all subsets: 2.832 over 485,518 verification steps.
Limitations
Only works with LFM3-1.7B. Throughput was measured only on math_reasoning and HumanEval. Temperature 0, concurrent load, and contexts beyond the 2,560-token training samples were not evaluated.
License
Apache 2.0
Nice work. LFM3-1.7B is the default local model in our desktop app (node-llama-cpp, CPU and Vulkan), mostly for short French voice turns. We’ll try the drafter in that setup and report back here if it holds up.
One thing we’re curious about: the draft vocabulary is 32,000 tokens. Does acceptance drop on non-English text, where more target tokens may fall outside that vocab? A per-language acceptance figure (FR/ES/DE) would help anyone serving LFM3 multilingually decide whether it pays off.
Thanks @YAKA001 ! The drafter was trained on ~100k mostly English math, code, and chat prompts, and the 32k reduced draft vocab was selected from that same data, so I'd expect acceptance to drop significantly on French and other non-English text.
While I don't have per-language numbers yet, the closest proxy is the translation subset at 2.63 mean acceptance length (vs. 2.83 overall).
Thanks, that matches what we see. In your metric (generated tokens ÷ verification rounds) we get about 2.3 on English and 1.8 on French, vs your 2.83 overall and 2.63 on translation. French drops as you expected, but English is also below your numbers, so our Q4_K_M target likely plays a part. At 2.3 the draft is still slower than LFM3 alone on our RTX 4050 (≈51 vs ≈73 tok/s): with a 1.7B target the drafting overhead weighs a lot.
I’ll rerun with a Q8_0 target to separate the quantization effect. If you ever train a multilingual variant (FR/ES in the training mix and the draft vocab), we’d be glad to run it through the same bench.
Script and raw results: https://gist.github.com/yaka0007/8a4ab4b7e882a962cf7af9ea209dd2a6
Thanks @YAKA001
One note: your prompts are all writing/explanation/advice, and in my benchmark those subsets were around 2.47–2.59, 2.83 overall includes math and code (humaneval), which pull up the average. So the English gap is smaller than it looks, and the Q4_K_M target could plausibly cover the rest. Curious to see the Q8_0 run.
With acceptance this low, it's also worth trying --spec-draft-n-max 2–4 instead of 8.
On the RTX 4050, I'd also recommend trying vLLM if you can. That's where I measured the speedups, with the bf16 target the drafter was trained against.
And thanks for the multilingual offer. I'll ping you here if I train an FR/ES variant.
Thanks, good call on n-max. With the Q8_0 target, English acceptance length is 2.4, in line with your 2.47 to 2.59 for these prompt types, so the English gap was mostly the prompt mix. French stays lower (1.7 to 1.8).
n-max made the difference on the RTX 4050 (Vulkan, b12211): 87 tok/s on English with n-max 4, against 66 tok/s without the draft at best (the laptop GPU baseline moved between 53 and 66 across runs), so about +30%. French breaks even (67 vs 65). n-max 2 gives 78 / 66, n-max 8 was the slowdown we saw. No gain on CPU at any setting.
All tables and raw results are in the gist. vLLM will have to wait: Windows box, but noted.
Notebook to compare Baseline (no drafter) vs DSpark with vllm (T4 GPU): https://colab.research.google.com/drive/1fsDsUO2YoN9Rlzyn9orSo51BgwBaHNkO
The 3x speedup holds here, especially for math and code