TIE-Pilot/deepspec-drafter-ablations

DeepSpec drafter ablations — budget-matched one-epoch arms

Training checkpoints for the ablation arms behind TIE-Pilot/dspark-attnconv-block7-qwen3-4b, the drafter reported in arXiv:2608.27339.

Every arm here ran exactly one epoch — 2,616 optimizer steps — at lr 6e-4 with 512 anchors on the same data, against the same frozen Qwen/Qwen3-4B target. The arms differ only along the axis each is named for, so the budget is matched by construction and the comparison is not confounded by training length.

What each arm changes

Directory	Order-1 head	Short conv	Slot embed	Objective (CE / L1)
dspark_b7_qwen3_4b_1ep	vanilla, rank 256	—	—	0.1 / 0.9
dspark_b7_qwen3_4b_1ep_shortconv	vanilla, rank 256	✔	—	0.1 / 0.9
slotembed_b7_qwen3_4b	vanilla, rank 256	—	✔	0.1 / 0.9
attnhead_b7_qwen3_4b	attention, rank 512	—	—	0.1 / 0.9
attnconv_b7_qwen3_4b	attention, rank 512	✔	✔	0.1 / 0.9
v3a_condhead_b7_qwen3_4b	conditional, rank 256	—	—	0.1 / 0.9
v3b_slotsplit_b7_qwen3_4b	conditional, rank 256	—	✔ (+ cross/self split)	0.1 / 0.9
xg_v21_b7_qwen3_4b	XG v2.1	—	—	0.1 / 0.9
dflash_b7_qwen3_4b_1ep	none	—	—	1.0 / 0.0 (CE only)
dflash_b7_qwen3_4b_1ep_tv	none	—	—	0.1 / 0.9
dflash_b7_qwen3_4b_1ep_cemix_serve	none	—	—	0.1 / 0.0
official_dflash2_b8_qwen3_4b_1ep	none, block 8	official wiring	—	Speculators recipe

Measured at this budget

Accepted length on 430 temperature-zero rows, per-task cap 50, maximum generation length 2048:

One-epoch configuration	Directory	Accepted length	vs vanilla
Vanilla DSpark	dspark_b7_qwen3_4b_1ep	4.5646	—
DFlash2 reproduction	official_dflash2_b8_qwen3_4b_1ep	4.4409	−0.1237
Vanilla + our short convolution	dspark_b7_qwen3_4b_1ep_shortconv	4.6133	+0.0488
Vanilla + slot embeddings	slotembed_b7_qwen3_4b	4.5552	−0.0093
Prefix-attention head	attnhead_b7_qwen3_4b	4.6767	+0.1122
Head + convolution + slot embeddings	attnconv_b7_qwen3_4b	4.7655	+0.2009

The reproduced DFlash2 is below vanilla at this budget. The head and the short convolution both help, and the combined variant is best. Slot embeddings alone are essentially unchanged; they are kept because they cost nothing at inference and appeared to speed up convergence.

These rows do not isolate final component contributions. The head arm changes the head architecture and adds the nomination objective, and its initial output scale is ~0.0426 while the combined arm uses 0.35. Read them as design comparisons, not as an additive decomposition.

One epoch is not enough to judge an architecture. The same recipe reaches 87% of its converged accepted length after a single epoch, so a change that only pays off late looks like noise here. The per-epoch trajectory in the flagship repository shows how much moves after this point.

The anchor column

The head and combined arms both carry a separate anchor key/value column (markov_head.anchor_type, markov_head.anchor_norm.weight). Removing it at evaluation does not affect acceptance, and keeping it made longer runs collapse — the first attempt at the 10-epoch run diverged with that column's gradient scale 13.7x a healthy control's after median normalisation, against a 2.45x ceiling across the other 118 tensors. The released flagship sets markov_anchor_kv=False. If you train from these checkpoints for more than an epoch, turn it off.

Layout and loading

<arm>/step_2616/config.json, model.safetensors, train_config.py, training_metadata.json

train_config.py is the exact configuration the trainer wrote for that arm — diff two of them to see precisely what an axis changed.

Optimizer state is not included: every arm here ran its cosine schedule to the end and its scheduler records _last_lr == 0, so resuming as-is would take zero-sized steps. Continuing any of these arms means defining a new schedule regardless.

Qwen3DSparkModel is a custom block-parallel drafter, not a transformers architecture, so AutoModel.from_pretrained will not resolve it; loading needs the DeepSpec code. The safetensors files are otherwise standard.

Citation

This work is described in Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting (arXiv:2608.27339). The paper separates a block drafter's rejection into an information floor — the minimum expected rejection at a given conditioning order — and the model gap above it, which is the part a better proposal can actually recover. The prefix-attention head in this repository is the design that distinction points to: it reads committed context conditional on the predecessor, rather than applying a context-independent predecessor correction.

@misc{qiang2026parallelblindnessinformationfloors,
      title={Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting}, 
      author={Xinwei Qiang and Xiang Fang and Chang Chen and Zaifeng Pan and Yue Guan and Yufei Ding},
      year={2026},
      eprint={2608.27339},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2608.27339}, 
}