chutesai/Qwen3-32B-FP8-DFLASH

Qwen3-32B-FP8 DFlash Draft Model

DFlash speculative decoding draft model for Qwen/Qwen3-32B-FP8. Trained using the DFlash (Block Diffusion for Flash Speculative Decoding) method from Z-Lab.

Architecture

Parameter	Value
Draft layers	5
Hidden size	5120
Attention heads	32 (8 KV heads, GQA)
Head dim	128
Intermediate size	9728
Block size	16
Target layers captured	[1, 16, 31, 46, 61]
Parameters (draft-only)	~3.2B (bf16)
Tied embeddings	Yes (shared with target)

The draft model takes concatenated hidden states from 5 target model layers as input and predicts a block of 16 tokens in parallel via iterative denoising. Attention is non-causal: queries attend to both target hidden states (context) and noise embeddings (draft tokens).

Training

Detail	Value
Target model	Qwen/Qwen3-32B-FP8
Dataset	~50k multi-turn conversations (ShareGPT, OpenHermes, WildChat)
Max sequence length	2048
Effective batch size	8 sequences/step (DDP across 8 GPUs)
Training steps	~103k (1 epoch)
Hardware	8x NVIDIA H200 141GB
Optimizer	AdamW, lr=4.8e-3, cosine schedule
Loss	Focal cross-entropy (gamma=7.0)
Precision	bf16 (draft), FP8 (target, frozen)

Benchmarks

All benchmarks on a single NVIDIA RTX PRO 6000 Blackwell (98GB VRAM) using SGLang v0.5.13.post1.

ShareGPT (200 prompts, concurrency 8, max 1024 output tokens)

Metric	DFlash	Vanilla	Speedup
Output throughput (tok/s)	423.0	229.1	1.85x
Median TTFT (ms)	88.0	313.2	3.56x
Median ITL (ms)	14.7	34.6	2.35x
Median TPOT (ms)	19.5	34.5	1.77x
Accept length	2.46	—	—

Synthetic (random tokens, 100 prompts per config)

Input/Output	Concurrency	DFlash tok/s	Vanilla tok/s	Speedup	Accept len
128/128	1	54.5	20.7	2.64x	2.15
128/128	8	299.0	205.4	1.46x	2.13
128/128	32	578.3	605.8	0.96x	2.13
512/512	1	71.0	20.9	3.40x	2.47
512/512	8	438.6	215.5	2.04x	2.58
512/512	32	814.3	631.2	1.29x	2.63
1024/1024	1	77.8	—	—	2.77
1024/1024	8	478.3	—	—	2.83
1024/1024	32	869.0	—	—	2.87
2048/256	1	100.9	—	—	2.92
2048/256	8	555.0	—	—	2.96
2048/256	32	945.5	—	—	2.99

Accept length increases with context length (2.13 at 128 tokens to 2.99 at 2048 tokens).

Usage (SGLang)

python -m sglang.launch_server \
    --model-path Qwen/Qwen3-32B-FP8 \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path chutesai/Qwen3-32B-FP8-DFLASH \
    --speculative-num-draft-tokens 16 \
    --speculative-draft-attention-backend triton \
    --trust-remote-code \
    --mem-fraction-static 0.85 \
    --host 0.0.0.0 --port 30000

Requires SGLang >= v0.5.13 with DFlash support.