RedHatAI/GLM-5.2-speculator.dspark

GLM-5.2 DSpark speculator

Overview

A DSpark speculator model for the zai-org/GLM-5.2-FP8 base model, enabling faster inference through speculative decoding. DSpark extends the DFlash parallel draft backbone with two lightweight heads: a Markov logit-bias head (low-rank intra-block token dependency) and a per-position confidence head (accept-rate prediction). Trained with the speculators library.

Model Specifications

• Base Model: zai-org/GLM-5.2-FP8
• Chat Template: GLM-5.2 (compatible with /chat/completions)
• Format: Safetensors
• License: MIT
• Draft: 5 layers, block_size=8, full vocabulary (154,880), aux layers [8, 23, 39, 55, 70]
• Training/Validation Hardware: NVIDIA B300 donated by Verda Cloud

Checkpoint series

This repo publishes per-epoch checkpoints of a single 3-epoch run. main tracks the latest available epoch; each epoch is also a permanent revision.

revision	epoch	status
epoch-1	1 / 3	✅ this checkpoint
epoch-2	2 / 3	training
epoch-3	3 / 3	training

Training Details

The model was trained using the Speculators library on prompts from Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered and HuggingFaceH4/ultrachat_200k, with responses regenerated by GLM-5.2-FP8 itself (published as mgoin/GLM-5.2-FP8-magpie-ultrachat).

Training is online: the draft consumes hidden states streamed on-the-fly from a live GLM-5.2-FP8 vLLM server, with the trainer running FSDP data-parallel on separate GPUs. The three commands below (data prep → server → trainer) reproduce the run. Install speculators and vLLM from main. GPU indices/parallelism are examples — adjust to your hardware.

Data Preparation

python scripts/prepare_data.py \
  --model zai-org/GLM-5.2-FP8 \
  --trust-remote-code \
  --data ./regenerated_data.jsonl \
  --output ./output \
  --seq-length 8192 \
  --assistant-pattern '<\|assistant\|>((?:(?!<\|user\|>|<\|assistant\|>).)*)'

--assistant-pattern is currently needed for GLM-5.2's inline-reasoning chat format (the <think>...</think> trace is kept inside the assistant turn); it may be auto-detected by future speculators versions.

vLLM Server Launch (hidden-states server)

CUDA_VISIBLE_DEVICES=0,1,2,3 python scripts/launch_vllm.py \
  zai-org/GLM-5.2-FP8 \
  --target-layer-ids 8 23 39 55 70 \
  -- --port 8000 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --trust-remote-code

Training Command

CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \
  --standalone \
  --nproc_per_node 4 \
  scripts/train.py \
  --verifier-name-or-path zai-org/GLM-5.2-FP8 \
  --speculator-type dspark \
  --num-layers 5 \
  --block-size 8 \
  --data-path ./output \
  --vllm-endpoint http://localhost:8000/v1 \
  --save-path ./output/checkpoints \
  --epochs 3 \
  --lr 0.0006 \
  --scheduler-type cosine \
  --total-seq-len 4096 \
  --draft-arch qwen3 \
  --draft-hidden-act silu \
  --target-layer-ids 8 23 39 55 70 \
  --max-anchors 1024 \
  --markov-rank 256 \
  --enable-confidence-head \
  --confidence-head-with-markov \
  --loss-fn '{"ce": 0.1, "tv": 0.9}' \
  --confidence-head-alpha 1.0 \
  --checkpoint-freq 0.2 \
  --on-missing generate \
  --on-generate delete \
  --seed 42 \
  --log-freq 100 \
  --prefetch-factor 2 \
  --num-workers 8 \
  --trust-remote-code

Notes:

• Omitting --draft-vocab-size trains on the full vocabulary; pass --draft-vocab-size 32000 for a reduced draft vocab.
• DSpark-specific flags: --markov-rank, --enable-confidence-head, --confidence-head-with-markov, --confidence-head-alpha. Dropping them (and using --speculator-type dflash) recovers a plain DFlash draft.
• --checkpoint-freq 0.2 writes sub-epoch checkpoints (~5 per epoch) for resumability.

Deployment

DSpark inference support is available on the vLLM nightly (uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly):

vllm serve zai-org/GLM-5.2-FP8 \
    --tensor-parallel-size 4 \
    --max-model-len 16384 \
    --trust-remote-code \
    --speculative-config '{
        "model": "RedHatAI/GLM-5.2-speculator.dspark",
        "num_speculative_tokens": 7,
        "method": "dspark",
        "draft_sample_method": "probabilistic"
    }'

Evaluation Results

Train-set metrics at the end of epoch 1 (the epoch-1 validation pass did not complete due to a server restart; per-dataset acceptance evaluation will accompany later epochs):

metric	value
mean accepted length	3.376
full accuracy	0.532
mean acceptance rate	0.497
confidence abs error	0.107

Per-position accuracy (positions 1-7): 0.780 / 0.649 / 0.557 / 0.494 / 0.454 / 0.409 / 0.381

Acceptance length in vLLM (revision epoch-1)

Measured end-to-end in vLLM speculative decoding (nightly 0.23.1rc1.dev709+g2b753ad20), serving zai-org/GLM-5.2-FP8 with this speculator on 4xB300, compared against the earlier GLM-5.2-speculator.dspark-preview under the identical setup: num_speculative_tokens=7, greedy sampling, batch size 1, 16 single-turn chat prompts (8 HumanEval + 8 math_reasoning from RedHatAI/speculator_benchmarks), 1024 output tokens each. Acceptance is computed from vLLM's spec_decode_num_accepted_tokens_per_pos / num_drafts counter deltas.

Results are for revision epoch-1 (what main currently points to); this table will be updated as later epochs are published.

Checkpoint	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Pos 7	Accept Len	Decode tok/s
epoch-1 (this repo)	75.8%	55.2%	40.3%	29.3%	21.5%	15.5%	11.1%	3.49	219
dspark-preview	57.5%	31.7%	15.4%	7.4%	3.3%	1.5%	0.7%	2.18	139

The full-vocabulary draft and regenerated Magpie+UltraChat data substantially improve deep-position acceptance, where the preview fell off after position 3. For reference, the same server without speculative decoding decodes at 102 tok/s (2.15x speedup for epoch-1).

References

• DFlash: Block Diffusion for Flash Speculative Decoding (arXiv:2602.06036) — the parallel draft backbone DSpark builds on.
• DSpark (DeepSeek) — the Markov + confidence-head additions replicated here.
• speculators — training library.

AI assistance was used to build the training pipeline and run these experiments.