RedHatAI/gemma-4-31B-it-speculator.dflash2

A DFlash2 speculative decoding draft model for google/gemma-4-31B-it, trained with Speculators.

It was trained using the Speculators library on the mlabonne/open-perfectblend dataset. Training data used responses from this dataset for the gemma-4-31B-it model (with reasoning).

Model configuration

Setting	Value
Speculator	DFlash2
Verifier	`google/gemma-4-31B-it`
Draft transformer layers	5
Auxiliary verifier hidden-state layers	`1, 17, 29, 47, 59`
Block size	8
Default proposed tokens	7
Sliding window	2,048
Proposal method	Greedy
Weight dtype	bfloat16

vLLM Serving

Use a vLLM version or build that supports both DFlash2 and Gemma 4. The draft model is used with the matching Gemma-4 verifier:

vllm serve google/gemma-4-31B-it \
  --tensor-parallel-size 2 \
  --speculative-config '{
    "model": "RedHatAI/gemma-4-31B-it-speculator.dflash2",
    "num_speculative_tokens": 7,
    "method": "dflash"
  }'

Training details

The checkpoint’s recorded training command used the following DFlash2 configuration (paths are shown as placeholders).

torchrun --standalone --nproc_per_node=4 \
  /path/to/speculators/train/__main__.py \
  --draft-vocab-size 262144 \
  --verifier-name-or-path google/gemma-4-31b-it \
  --data-path /path/to/prepared-open-perfectblend-data \
  --save-path /path/to/checkpoints \
  --hidden-states-path /path/to/hidden-states \
  --log-dir /path/to/training-logs \
  --run-name dflash_5layers_5epochs \
  --speculator-type dflash \
  --block-size 8 \
  --max-anchors 1536 \
  --num-layers 5 \
  --target-layer-ids 1 17 29 47 59 \
  --sliding-window 2048 \
  --epochs 3 \
  --lr 6e-4 \
  --weight-decay 0.01 \
  --scheduler-type cosine \
  --optimizer muon \
  --muon-lr 0.003 \
  --dpace-alpha 0.5 \
  --per-position-loss-weight dpace \
  --loss-fn ce \
  --checkpoint-freq 0.1 \
  --log-freq 100 \
  --num-workers 12 \
  --prefetch-factor 2 \
  --fsdp-shard \
  --vllm-endpoint http://localhost:8906/v1 \
  --on-missing generate \
  --on-generate delete \
  --request-timeout 180 \
  --total-seq-len 8192 \
  --max-retries 20

The training run used 4 processes, Speculators 0.8.0, vLLM 0.29.0, Transformers 5.14.1, and PyTorch 2.13.0. The data-path is the prepared tokenized data derived from mlabonne/open-perfectblend. The recorded run name contains 5epochs, but the actual --epochs setting was 3.

Evaluation

Acceptance results by evaluation subset.

Subset	Acceptance length	Pos. 0	Pos. 1	Pos. 2	Pos. 3	Pos. 4	Pos. 5	Pos. 6
HumanEval	6.21	93.33%	86.27%	79.71%	73.61%	67.95%	62.67%	57.78%
math_reasoning	6.88	96.23%	92.25%	88.38%	84.43%	79.99%	75.53%	71.46%
qa	4.02	78.86%	61.26%	47.40%	37.58%	30.73%	25.17%	20.59%
question	4.35	82.31%	65.10%	52.27%	42.67%	35.90%	30.65%	26.26%
rag	5.10	88.08%	76.18%	64.92%	55.80%	48.09%	41.28%	35.15%
summarization	4.16	82.13%	65.81%	51.62%	40.81%	31.57%	24.49%	19.10%
tool_call	4.56	84.07%	69.21%	56.50%	46.84%	38.87%	32.89%	27.51%
translation	5.03	86.28%	74.86%	63.85%	55.11%	47.51%	40.88%	34.96%
writing	4.35	82.26%	65.08%	52.14%	42.53%	35.80%	30.60%	26.22%