GLM-5.3-speculator.dflash2-preview

Model Overview

• Model Architecture: GlmMoeDsaForCausalLM
  • Input: Text
  • Output: Text
• Model Optimizations:
  • Speculative Decoding Algorithm: DFlash2
  • Draft Model: 5-layer qwen3 backbone (DFlash2DraftModel)

This is a DFlash2 speculator for zai-org/GLM-5.3. The acceptance tables use eight draft tokens with adaptive verification disabled; performance graphs also include adaptive verification enabled and native MTP. The draft proposes tokens for verification by GLM-5.3; deploy both models together.

Model Specifications

Property	Value
Verifier	zai-org/GLM-5.3
Draft	RedHatAI/GLM-5.3-speculator.dflash2-preview
Draft format / precision	Safetensors / BF16
Verifier precision	Native FP8 weights; BF16 compute and KV cache
Chat template	Pinned GLM-5.3 chat template; /v1/chat/completions
Proposal configuration	Greedy proposal, 8 speculative tokens
Evaluated serving context	131,072 tokens
Validation hardware	Speedup curves: 4 × NVIDIA B300, TP4; dedicated temperature-1 acceptance: 8 × NVIDIA B200, TP8

The serialized draft config records Speculators 0.8.0.dev222. This identifies the config's recorded library version; it does not establish the full training environment. The config's maximum-position value is 1,048,576; evaluation used the smaller serving context above.

Deployment

vllm serve zai-org/GLM-5.3 \
    --served-model-name zai-org/GLM-5.3 \
    --host 127.0.0.1 --port 9155 \
    --tensor-parallel-size 4 \
    --dtype bfloat16 --kv-cache-dtype bfloat16 \
    --max-model-len 131072 \
    --max-num-seqs 128 --max-num-batched-tokens 16384 \
    --gpu-memory-utilization 0.9 --no-enable-prefix-caching \
    --speculative-config "{\"method\":\"dflash\",\"model\":\"RedHatAI/GLM-5.3-speculator.dflash2-preview\",\"num_speculative_tokens\":7,\"enable_adaptive_verification\":true}" \
    --per-request-spec-decode-metrics summary

Evaluation

Results were collected with AIPerf 0.13.0 and frozen Speculators evaluation scripts, with thinking enabled and a maximum of 32,768 output tokens. All results below use the same pinned draft and verifier. Failed requests remain included in outcome counts; acceptance metrics use successful responses with complete counters.

Definitions: acceptance length = 1 + accepted draft tokens / speculative steps; acceptance rate = 100 × accepted / proposed draft tokens; position k = 100 × steps accepting at least k drafts / speculative steps. Counters are pooled before division, and positions are numbered 1–8. These are decoding statistics, not answer-accuracy scores or measured speedup factors.

Basic datasets — temperature 0

Dataset / category	Acceptance length	Acceptance rate	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Pos 7	Pos 8
HumanEval	4.8074	47.59%	83.49%	68.30%	56.05%	46.44%	39.08%	33.36%	28.84%	25.18%
GSM8K	5.8453	60.57%	89.11%	78.43%	69.43%	61.57%	54.92%	48.93%	43.54%	38.61%
MATH500	5.3157	53.95%	86.43%	73.84%	63.21%	54.37%	47.03%	40.77%	35.33%	30.57%
MT-Bench	3.6381	32.98%	74.96%	54.31%	39.48%	29.28%	22.36%	17.57%	14.18%	11.67%

Basic datasets — temperature 1

Dataset / category	Acceptance length	Acceptance rate	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Pos 7	Pos 8
HumanEval	4.2941	41.18%	78.18%	61.05%	48.52%	39.08%	32.31%	27.16%	23.16%	19.96%
GSM8K	4.6623	45.78%	79.89%	64.58%	53.30%	44.73%	37.98%	32.71%	28.35%	24.70%
MATH500	4.6329	45.41%	80.61%	65.35%	53.61%	44.63%	37.59%	31.78%	26.94%	22.76%
MT-Bench	3.0959	26.20%	67.74%	45.31%	30.91%	21.63%	15.70%	11.83%	9.21%	7.25%

SPEED-Bench qualitative — temperature 0

Dataset / category	Acceptance length	Acceptance rate	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Pos 7	Pos 8
coding	4.0971	38.71%	78.33%	60.15%	46.26%	36.22%	28.99%	23.66%	19.59%	16.51%
humanities	3.8826	36.03%	76.53%	57.73%	43.58%	33.12%	25.87%	20.46%	16.82%	14.16%
math	3.9028	36.28%	77.84%	59.14%	44.81%	34.08%	26.16%	20.21%	15.73%	12.30%
qa	4.2508	40.64%	80.65%	63.71%	49.73%	38.84%	30.85%	24.68%	20.06%	16.56%
rag	4.9694	49.62%	84.57%	70.38%	58.39%	48.39%	41.03%	35.35%	31.23%	27.59%
reasoning	4.5232	44.04%	81.24%	65.24%	51.80%	41.39%	34.17%	29.37%	25.84%	23.27%
stem	3.8932	36.17%	77.98%	59.32%	44.71%	33.55%	25.57%	19.78%	15.73%	12.67%
writing	3.1636	27.04%	70.81%	48.53%	32.90%	22.57%	15.76%	11.23%	8.27%	6.27%
multilingual	3.9722	37.15%	75.38%	57.57%	44.40%	35.04%	28.12%	22.69%	18.64%	15.38%
summarization	4.6290	45.36%	85.01%	69.37%	55.91%	45.30%	36.43%	28.88%	23.32%	18.67%
roleplay	2.9572	24.47%	67.65%	43.17%	27.69%	18.57%	13.21%	10.11%	8.30%	7.01%

SPEED-Bench qualitative — temperature 1

Dataset / category	Acceptance length	Acceptance rate	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Pos 7	Pos 8
coding	3.7523	34.40%	73.84%	54.36%	40.78%	31.31%	24.80%	19.98%	16.46%	13.70%
humanities	2.8186	22.73%	63.61%	40.74%	26.60%	17.83%	12.42%	8.95%	6.64%	5.05%
math	3.4813	31.02%	72.08%	51.41%	37.67%	27.85%	21.03%	16.06%	12.42%	9.61%
multilingual	3.4985	31.23%	69.54%	50.08%	37.05%	28.23%	21.87%	17.45%	14.14%	11.48%
qa	3.1041	26.30%	69.17%	47.03%	32.16%	22.07%	15.29%	11.00%	7.90%	5.80%
rag	3.8383	35.48%	72.85%	54.41%	42.06%	33.02%	26.83%	21.84%	17.96%	14.87%
reasoning	2.8581	23.23%	63.28%	40.63%	26.91%	18.54%	13.24%	9.87%	7.52%	5.82%
roleplay	2.4505	18.13%	61.08%	35.24%	20.09%	11.66%	7.14%	4.56%	3.08%	2.20%
stem	3.0027	25.03%	66.86%	44.85%	30.35%	20.85%	14.38%	10.17%	7.41%	5.40%
summarization	4.2974	41.22%	81.47%	64.73%	51.15%	40.21%	31.52%	24.83%	19.86%	15.97%
writing	2.7726	22.16%	64.72%	41.07%	26.21%	16.99%	11.30%	7.69%	5.42%	3.86%

MRCR eight-needle long-context acceptance — temperature 0

Dataset / category	Acceptance length	Acceptance rate	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Pos 7	Pos 8
8 needles	6.2071	65.09%	92.19%	84.61%	75.13%	65.38%	57.71%	52.20%	48.23%	45.26%

Native prompt-length bin	Acceptance length	Acceptance rate
0-8192	6.9190	73.99%
8193-16384	5.8916	61.14%
16385-32768	5.9500	61.88%
32769-65536	5.8348	60.44%
65537-131072	5.6294	57.87%

Throughput versus interactivity

Each graph compares DFlash2 (8 tokens, adaptive off), DFlash2 (8 tokens, adaptive on), and native MTP (3 tokens) on four NVIDIA B300 GPUs with TP4. Thinking is enabled, the output cap is 32,768 tokens, and prefix caching is disabled. Point labels show concurrency: 1, 4, 8, 16, 32, 64, and 128.

X is native AIPerf output_token_throughput_per_user.avg (mean per-request reciprocal inter-token latency, tokens/s/user); Y is output_token_throughput.avg (aggregate tokens/s). Native per-run means are averaged equally across repeats. With server token counts and zero grace, unfinished requests' partial output is excluded. All figures use AIPerf's native plotting API; confidence ovals and error bars are hidden.

Temperature 0

Adaptive-on curves use two repeats with 180-second measurement windows on vLLM 0.32.0rc1.dev281+g49cb2bbf7. Adaptive-off and MTP curves are historical 600-second measurements on the earlier recorded vLLM build, with three repeats for HumanEval/GSM8K/MATH500 and two for MT-Bench. All points have 30-second warmup and zero grace. These overlays do not isolate adaptive verification from the server-version and window differences.

HumanEval

GSM8K

MATH500

MT-Bench

Reproducibility

Performance commands, native summaries, checkpoint/build identities, and plotting provenance accompany the updated graphs.

Component	Recorded identity
Draft revision	f1da82107865449c6f5a830f4b8a3d60e0ab55e3
Draft weights SHA256	2d8c661655c43e528d63c3de16fdcd7cc7cc58962fb223ca18e330df0f6f9267
Verifier revision	30333038ada1f1dacb294a93270305a890b50c14
Evaluation scripts commit	21192d6ff51bb2aaa2ab0aa1d72e6f0b8f6a7d84
AIPerf	0.13.0
SPEED-Bench preparation	NVIDIA NeMo Skills commit 5ac8609a56ac941540b10c92e68d556e6343cd4c
Preserved qualitative JSONL SHA256	e9bbbe998472b79501ed79ba2bf126e58b63e153a6deef70ce2b63ea7a4de845
MRCR source	openai/mrcr revision f4c69fae7cf81f7ca26b9fee34b392a50f6b8a1d; 8needle_0.parquet and 8needle_1.parquet, 400 rows each