dspark-block8-qwen25-7b (step_2080)

A Qwen2DSparkModel block-parallel draft model trained online against a frozen Qwen/Qwen2.5-7B target, for block-wise speculative decoding.

This is a research checkpoint, archived as-is. The training line was discontinued after step 2080 (2026-07-15).

What is in this repo

file	what
model.safetensors	draft model weights, bf16 (~4.5 GB)
config.json	Qwen2DSparkModel architecture config
train_config.py	the exact training config this checkpoint was produced with
training_state.rank{0..3}.pt	full optimizer / trainer state (4 x ~15 GB), for resuming training

There is no tokenizer in this repo — use the target model tokenizer: AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B").

Architecture

• 5 draft layers, hidden size 3584, 28 heads / 4 KV heads (Qwen2.5-7B geometry)
• block_size = 8 (predicts 8 tokens per block)
• target layer ids [1, 7, 14, 21, 26] — deliberately stops below the final decoder layer 27, so that raw forward-hook hiddens captured in training match output_hidden_states at eval
• Markov head: vanilla, rank 256; num_anchors = 512; mask_token_id = 151665 (first unused embedding row of the padded 152064 vocab)
• confidence head disabled (alpha = 0) and L1/distillation disabled — the online-hiddens path provides no aligned target logits

Training

• online hiddens: target hiddens computed each step by forwarding the frozen Qwen2.5-7B, no cached hidden dataset
• corpus: perfectblend token trajectories regenerated by the base Qwen2.5-7B, max length 4096
• lr 6e-4, warmup ratio 0.04, weight decay 0, bf16, global batch 480, grad clip 1.0
• CE-only loss with loss_decay_gamma = 4.0
• 2080 steps

Caveat

training_state.rank*.pt are pickled PyTorch objects, saved for 4 ranks with sharding_strategy="no_shard". Load them only if you trust this repo.