RedHatAI/Qwen3-8B-speculator.dflash2

This is a DFlash2 speculator model for Qwen/Qwen3-8B, trained without reasoning (enable_thinking=False).

Training Details

This model was trained using the Speculators library on a mix of Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered, the train_sft split of HuggingFaceH4/ultrachat_200k, and nvidia/Nemotron-Personas data. Responses were regenerated by Qwen3-8B (without reasoning). Training uses D-PACE loss weighting and the Muon optimizer, with a re-warm cosine learning rate schedule for continued improvement after initial convergence.

Commands

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

python scripts/prepare_data.py \
  --model Qwen/Qwen3-8B \
  --data ./regenerated_data.jsonl \
  --output ./output \
  --seq-length 16384

Launch vLLM

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm_venv/bin/python scripts/launch_vllm.py \
  Qwen/Qwen3-8B \
  --target-layer-ids 3 11 19 27 35 43 \
  -- --port 8000 \
  --gpu-memory-utilization 0.85 \
  --tensor-parallel-size 4 \
  --disable-uvicorn-access-log

Launch training (resumed with re-warm cosine LR)

CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \
  --standalone \
  --nproc_per_node 4 \
  scripts/train.py \
  --verifier-name-or-path Qwen/Qwen3-8B \
  --speculator-type dflash2 \
  --block-size 16 \
  --num-layers 8 \
  --data-path ./output \
  --vllm-endpoint http://localhost:8000/v1 \
  --save-path ./output/checkpoints \
  --epochs 7 \
  --lr 6e-4 \
  --total-seq-len 16384 \
  --max-anchors 768 \
  --target-layer-ids 3 11 19 27 35 43 \
  --on-missing generate \
  --on-generate delete \
  --loss-fn ce \
  --per-position-loss-weight dpace \
  --dpace-alpha 0.5 \
  --optimizer muon \
  --muon-lr 0.003 \
  --muon-momentum 0.95 \
  --muon-weight-decay 0.1 \
  --muon-ns-steps 5 \
  --weight-decay 0.01 \
  --scheduler-type cosine \
  --scheduler-total-steps 68922 \
  --prefetch-factor 2 \
  --num-workers 8 \
  --log-freq 100

Model Specifications

Base Model	Qwen/Qwen3-8B
Chat Template	Qwen/Qwen3-8B (use /chat/completions endpoint)
Reasoning	Disabled (enable_thinking=False)
Format	Safetensors
License	Apache 2.0
Validation Hardware	8× Nvidia H100

Deployment

pip install vllm

vllm serve Qwen/Qwen3-8B \
    --tensor-parallel-size 1 \
    --max-model-len 16384 \
    --speculative-config '{
        "model": "RedHatAI/Qwen3-8B-speculator.dflash2",
        "num_speculative_tokens": 7,
        "method": "dflash2"
    }'

Evaluations

Per-position token acceptance rates on the RedHatAI speculative decoding benchmark (native 7 draft tokens, enable_thinking=False, max_tokens=512, temperature=0):

Dataset	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Pos 7	Avg Length
HumanEval	88.0%	75.8%	64.4%	54.5%	46.5%	39.4%	33.4%	5.018
math_reasoning	92.1%	83.3%	75.0%	66.8%	58.8%	52.1%	46.5%	5.747
qa	67.9%	45.3%	29.2%	19.4%	12.4%	8.4%	5.0%	2.876
question	74.3%	53.4%	37.1%	26.9%	20.0%	15.5%	12.0%	3.390
rag	73.6%	52.9%	37.3%	26.1%	18.4%	12.5%	8.5%	3.293
summarization	70.8%	46.3%	28.8%	17.2%	10.4%	6.5%	3.8%	2.838
tool_call	77.0%	57.5%	42.6%	32.7%	23.7%	17.8%	13.6%	3.649
translation	79.6%	57.7%	39.9%	20.9%	10.8%	6.1%	3.8%	3.188
writing	74.9%	53.6%	37.6%	27.3%	20.5%	16.0%	12.4%	3.424

References

Paper: DFlash2: Block Diffusion for Improved Speculative Decoding