[BUG] DSpark slower than EAGLE + severe cold-start penalty
Setup
- Backend: NVIDIA GPU B200
- Serving framework: SGLang 0.5.17
- Model:
Qwen3.8-27B-FP8 - Workload: streaming
/v1/completions,temperature=0, prompt ~512 tokens,max_tokens∈ {128, 512, 1024}
Observation
After switching from EAGLE to DSpark, the first ~5 requests after (re)start exhibit a ~12× throughput collapse, while steady-state performance is slightly below EAGLE rather than above it.
| Stage | EAGLE | DSpark (cold) | DSpark (warm) |
|---|---|---|---|
| TTFT | ~0.29 s | ~0.95 s | ~0.30 s |
| Decode throughput (1024 tok) | ~96 tok/s | ~7 tok/s | ~84 tok/s |
| Stability across runs | low variance | high variance | low variance |
Issues
- Cold-start penalty — DSpark's first requests run at 7–16 tok/s (vs ~90 tok/s warm). EAGLE showed no such warmup sensitivity under the same protocol. This suggests DSpark performs on-the-fly graph compilation / cache population on the first requests instead of pre-warming at server startup.
- Steady-state regression — even after warmup, DSpark decode (~84 tok/s) is ~12% slower than EAGLE (~96 tok/s), which contradicts the expected speedup from a more advanced spec-decoding method.
- Variance — during the cold phase, identical requests produce wildly different timings (e.g. 1024-tok run: 21 s vs 65 s vs 9 s), indicating nondeterministic compilation/lookup behavior.
Expected
- DSpark warmup should happen at server boot (or first idle request), not penalize real user traffic.
- Steady-state throughput should match or exceed EAGLE; if it doesn't, this is a regression worth investigating (draft acceptance rate, target length, or overhead per step may be misconfigured for this model).
Drafter is trained on hiden states of a fp8 model. It does not really work for different models.
A bf16 drafter against a 4-bit target is structurally uncompetitive — it would need 91.7 % sustained across seven positions. The better the target is quantized, the more a fat drafter costs relatively. This is the reason recommendations tuned on FP8 targets do not transfer.
Am I better off switching from Qwen-27B-FP8 to the RadixArk NVFP4 variant? Thank you.
No, there is no DFlash for nvfp4. This artifact is strictly fp8.
The data I have provided is for FP8 version.
Tested on vLLM, and no faster than with MTP
vllm: #50851 seams to break everything: dflash does not get hidden states and starts to guess token. Currently retesting with fixed vllm.
Testing finished. Drafter is like it is, works at optimum
│ MTP n=3 │ DSpark k=7 ││
│ NVFP4-Target │ 87,2 % │ 62,4 % │ −24,8 pp │
│ FP8-Target │ 83,4 % │ 65,0 % │ −18,4 pp │
The high acceptance rate in some posts are caused by simple prompts. Breaks down with real data.