RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.dflash

This is a DFlash speculator model for Qwen/Qwen3-30B-A3B-Instruct-2507.

Training Details

This model was trained using the Speculators library on a subset of Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered and the train_sft split of HuggingFaceH4/ultrachat_200k. Responses were regenerated by Qwen/Qwen3-235B-A22B-Instruct-2507. and stored at Dataset-Qwen3-235B-Instruct

Commands

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

# In virtual environment with speculators installed
python scripts/prepare_data.py \
  --model Qwen/Qwen3-30B-A3B-Instruct-2507
  --data ./regenerated_data.jsonl \
  --assistant-pattern "<\|im_start\|>assistant\s*([\s\S]*?)<\|im_end\|>" \
  --output ./output \
  --seq-length 16384

Launch vLLM

# In (separate) virtual environment with vllm installed
CUDA_VISIBLE_DEVICES=0,1 vllm_venv/bin/python scripts/launch_vllm.py \
  Qwen/Qwen3-30B-A3B-Instruct-2507 \
  --target-layer-ids 1 12 23 34 45 \
  --max-model-len  32768 \
  --max-num-batched-tokens 32768\
  --tensor-parallel-size 2 \
  --no-enable-chunked-prefill

Launch training

Must be run once vLLM has finished launching and is running in the background.

# In virtual environment with speculators installed
CUDA_VISIBLE_DEVICES=2,3 torchrun \
  --standalone \
  --nproc_per_node 2 \
  scripts/train.py \
  --verifier-name-or-path Qwen/Qwen3-30B-A3B-Instruct-2507 \     
  --data-path ./output \    
  --on-missing generate \    
  --on-generate delete \    
  --scheduler-type cosine \    
  --draft-vocab-size 32000 \    
  --max-anchors 1024 \    
  --target-layer-ids 1 12 23 34 45 \
  --speculator-type dflash \    
  --num-layers 5 \    
  --logger trackio  \    
  --lr 0.0006 \    
  --epochs 5 \    
  --sliding-window 2048 \    
  --sliding-window-indices 0 1 2 3 4 \    
  --draft-hidden-act silu

Model Specifications


Base Model	Qwen/Qwen3-30B-A3B-Instruct-2507
Chat Template	Qwen/Qwen3-30B-A3B-Instruct-2507 (use `/chat/completions` endpoint)
Format	Safetensors
License	Apache 2.0
Validation Hardware	Nvidia A100

Deployment

# Install vLLM from the required PR
pip install git+https://github.com/vllm-project/vllm.git     
                                                                                                                                                                                                                                                                                                          
# Deploy with speculative decoding                                                                                                                                                                                                                                                                        
vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \                                                                                                                                                                                                                                                                                
    --tensor-parallel-size 2 \                                                                                                                                                                                                                                                                            
    --max-num-batched-tokens 32768 \
    --attention-backend FLASH_ATTN \ 
    --speculative-config '{                                                                                                                                                                                                                                                                               
        "model": "RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.dflash",                                                                                                                                                                                                                                                   
        "num_speculative_tokens": 15,                                                                                                                                                                                                                                                                      
        "method": "dflash"                                                                                                                                                                                                                                                                                
    }'

Preliminary Evaluations

Per-position token acceptance rates across datasets:
(with reasoning enabled)

Dataset	Pos 0	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Avg. Length
HumanEval	83.2%	66.0%	51.6%	40.2%	31.3%	24.3%	18.1%	4.15
math_reasoning	87.1%	72.5%	59.0%	47.6%	37.1%	28.6%	21.3%	4.53
qa	58.5%	30.4%	14.7%	7.1%	3.3%	1.6%	0.8%	2.16
question	67.0%	40.3%	24.6%	15.9%	10.7%	7.3%	5.0%	2.71
rag	64.8%	37.1%	20.5%	11.0%	5.6%	2.7%	1.3%	2.43
summarization	60.6%	30.9%	15.0%	7.4%	3.4%	1.5%	0.6%	2.19
tool_call	65.7%	42.0%	27.9%	19.6%	12.8%	8.3%	5.3%	2.82
translation	70.2%	36.3%	15.6%	6.3%	2.4%	0.9%	0.4%	2.32
writing	67.0%	40.3%	24.6%	15.9%	10.7%	7.3%	5.0%	2.71