alice1001/dspark_qwen3_4b_block7

Qwen3-4B DSpark Block-7 — Open-PerfectBlend Regen, Step 18,000

This repository contains a DSpark draft model for speculative decoding with Qwen/Qwen3-4B as the target model. It is not a standalone chat or text-generation model. The checkpoint is loaded by the DeepSpec Qwen3DSparkModel implementation and must be used together with the Qwen3-4B target model.

The published weights are the training checkpoint's original BF16 model.safetensors from step_18000. They were copied byte-for-byte into this Hugging Face repository; optimizer, RNG, and per-rank training states are not included.

Model architecture

Item	Value
Draft architecture	Qwen3DSparkModel
Target model	Qwen/Qwen3-4B
DSpark block size	7
Draft transformer layers	5
Target feature layers	[1, 9, 17, 25, 33]
Hidden size	2,560
Anchors	512
Markov head	vanilla, rank 256
Confidence head	Enabled; Markov features enabled
Weight dtype	BF16
Maximum position configuration	40,960

The legacy internal experiment name contains dspark_block8, but the saved checkpoint configuration and the actual model both use block_size=7. The repository name follows the checkpoint's authoritative configuration.

Training details

The model was trained with DeepSpec's Qwen3DSparkTrainer on 1,339,649 Open-PerfectBlend conversations. Answers were regenerated by Qwen3-4B with thinking disabled. The public prompt/answer dataset is alice1001/open-perfectblend-qwen3-4b-regen. Training target caches were generated from each complete prompt + regenerated answer sequence, truncated to at most 4,090 cached tokens.

Setting	Value
Hardware	38 × NVIDIA A800 GPUs (19 two-GPU logical workers)
Precision	BF16
Local batch size	1 per GPU
Global batch size	494
Gradient accumulation	13
Planned epochs / steps	10 / 27,110
Published checkpoint	step_18000 (66.40% of planned steps; about 6.64 data epochs)
Training sequence length	4,096 (target-cache content up to 4,090)
Warmup ratio	0.04
LR schedule	warmup to approximately 6e-4, then cosine decay
Weight decay	0.0
Max gradient norm	1.0
Loss weights	CE 0.1, L1 0.9, decay gamma 4.0
Confidence-head alpha	1.0
Torch compile	Enabled

At the saved step, the recorded training metrics were loss=0.5449, accept_ratio@0=0.9180, and learning_rate=0.00016383. The training accept_ratio@0 is an in-distribution training signal and is not interchangeable with the runtime acceptance metrics below.

Evaluation results

The step_18000 checkpoint was evaluated with the DeepSpec runtime using four GPUs, max_new_tokens=2048, temperature=1.0, confidence_threshold=0.0, and seed=980406. The macro average over the nine datasets is accept length 4.74 and verify rate 0.5945.

These values measure speculative-decoding behavior, not benchmark answer accuracy or pass rate. Task-quality scores were not reported by this evaluation run, so none are claimed here.

Dataset	Samples	Proposals	Accept length	Verify rate	AR@0	AR@1	AR@2	AR@3	AR@4	AR@5	AR@6
GSM8K	500	6.95+1	6.12	0.7696	0.9321	0.8638	0.7964	0.7317	0.6711	0.6134	0.5599
MATH-500	500	6.98+1	5.67	0.7095	0.9075	0.8144	0.7286	0.6528	0.5856	0.5243	0.4662
AIME 2025	30	7.00+1	4.99	0.6240	0.8761	0.7508	0.6401	0.5465	0.4624	0.3892	0.3270
HumanEval	164	6.96+1	5.40	0.6777	0.8981	0.7960	0.6987	0.6164	0.5406	0.4697	0.4073
MBPP	256	6.94+1	5.12	0.6441	0.8844	0.7726	0.6672	0.5727	0.4913	0.4162	0.3541
LiveCodeBench	500	6.95+1	4.91	0.6177	0.8692	0.7497	0.6365	0.5388	0.4557	0.3806	0.3176
MT-Bench	80	6.97+1	3.64	0.4566	0.7687	0.5759	0.4248	0.3169	0.2364	0.1854	0.1462
Alpaca	500	6.97+1	3.50	0.4391	0.7655	0.5668	0.4100	0.2951	0.2128	0.1543	0.1100
Arena-Hard-v2	500	6.99+1	3.29	0.4118	0.6678	0.4828	0.3588	0.2739	0.2128	0.1672	0.1315

AR@n is the acceptance rate at proposal position n. Verify rate is the accepted-token fraction under this DeepSpec evaluation configuration.

Confidence-head calibration

Confidence metrics were collected with the early-stop threshold disabled (confidence_threshold=0.0). ECE and Brier score are lower-is-better; AUC is higher-is-better. Pred mean and target mean show mean predicted confidence and empirical acceptance respectively.

Dataset	Samples	Proposals	ECE	AUC	Brier	Pred mean	Target mean
GSM8K	500	24,560	0.0659	0.8850	0.1179	0.8048	0.7390
MATH-500	500	71,813	0.0325	0.8888	0.1175	0.7004	0.6687
AIME 2025	30	11,065	0.0175	0.8677	0.1292	0.5757	0.5703
HumanEval	164	13,325	0.0761	0.8772	0.1323	0.7083	0.6330
MBPP	256	16,119	0.0584	0.8624	0.1383	0.6523	0.5950
LiveCodeBench	500	29,063	0.0682	0.8677	0.1395	0.6322	0.5649
MT-Bench	80	10,378	0.0214	0.8710	0.1200	0.3970	0.3797
Alpaca	500	52,345	0.0271	0.8484	0.1254	0.3868	0.3599
Arena-Hard-v2	500	155,497	0.0203	0.9103	0.1011	0.3465	0.3280

Usage with DeepSpec

Install the current DeepSpec implementation and use this repository as the draft checkpoint. The tokenizer is taken from the target model and is therefore not duplicated in this repository.

git clone https://github.com/deepseek-ai/DeepSpec.git
cd DeepSpec
# Install the dependencies described by DeepSpec, then run:
CUDA_VISIBLE_DEVICES=0,1,2,3 python eval.py \
  --target_name_or_path Qwen/Qwen3-4B \
  --draft_name_or_path alice1001/dspark_qwen3_4b_block7 \
  --max-new-tokens 2048 \
  --temperature 1.0 \
  --confidence-threshold 0.0 \
  --seed 980406

Limitations

• This is a draft model for Qwen3-4B speculative decoding, not a replacement for the target language model.
• It was trained on non-thinking Qwen3-4B generations; behavior may degrade for thinking-mode or strongly shifted domains.
• The checkpoint is from step 18,000 of a planned 27,110-step run, not the final checkpoint.
• Reported results depend on the exact target model, sampling settings, DeepSpec revision, hardware, and benchmark prompts.

License and attribution

The base Qwen3-4B model is released under Apache-2.0. DeepSpec code is released under MIT. Users should review and comply with both upstream projects and the licenses of the evaluation datasets.