CHOI0126/minsik-repro

Qwen3.6-27B DFlash draft — reproduction in progress

This is an unfinished training run, published as it goes. Checkpoints appear here every half epoch of a planned six. Nothing here is a final result, and the numbers below will move.

A DFlash draft head for Qwen/Qwen3.6-27B, trained as a reproduction of DFlash (arXiv:2602.06036) on a model the paper does not cover. Full method, scripts and measurement records: ms-choi-126/specforge-dflash-repro.

What a DFlash draft is, and what it is not

It is not a standalone model. It cannot generate text by itself. It writes a 16-token block in one forward pass, conditioned on the target's hidden states at layers 1/16/31/46/61, and the target then verifies that block in a single pass and commits the leading run that matches.

Greedy verification means the committed tokens are exactly what the target would have produced alone. A worse draft makes generation slower; it never makes the output different. Measured directly during this reproduction: a draft with τ 2.41 and one with τ 7.08 produced byte-identical output on all 32 prompts of a control set, differing only in verify cycles (5,567 vs 1,895).

Layout

path	contents	size
epoch-<E>-step-<N>/	exported draft, config.json + model.safetensors — load this	3.46 GB
epoch-<E>-step-<N>-full/	full training state: model + optimizer + scheduler + RNG	24.22 GB

One checkpoint every half epoch; 96,117 optimizer steps make an epoch.

The first two were uploaded before the naming settled and keep a step-only name. Nothing else differs:

folder	epoch
step-48058, step-48058-full	0.50
step-96116, step-96116-full	1.00
epoch-1.50-step-144174 …	1.50 onward

"epoch" here counts optimizer steps, not passes over the corpus. Training was restarted from the step-96,116 checkpoint on 2026-09-14 after the capture servers were killed by operator error, and the prompt producer holds no checkpoint state, so it restarted its loop at epoch 0 while the trainer continued from step 96,116. Same corpus, different order: some rows are seen seven times and some five. The optimizer-step budget is unchanged at 576,704.

step-<N>-full/ is FSDP1, DP4 rank-sharded — four training_state_rank*.pt files. Restoring it needs the same rank count.

It warm-starts a new attempt; it cannot be dropped back into the run that wrote it. Two separate things prevent that, and both were established by doing it on 2026-09-14 rather than by reading alone. A producer is refused on a channel path that already exists, so the new attempt needs a fresh control directory; and the original ledger's unacknowledged tail would be requeued against Mooncake objects that no longer exist once the store went down. What the resume actually requires is a ledger whose durable marker equals the checkpoint's step — and that ledger can be created empty. RESTORE.json in each directory carries the exact procedure, and scripts/seed_resume_ledger.py in the GitHub repository does it.

Keep num_epochs and total_steps identical to the original when resuming: both sit in the checkpoint's resume contract and a mismatch is refused.

Measured so far

τ (block efficiency) = tokens committed per target verify pass, computed as (completion_tokens - 1) / spec_verify_ct averaged over requests. Compared against z-lab/Qwen3.6-27B-DFlash, the released draft, measured on the same harness with identical prompt sets (prompt_set_sha256 verified equal on all five datasets).

step 48,058 — half an epoch of six:

dataset	ours	released draft	achieved
gsm8k	5.9494	7.3204	81.3%
math500	6.2513	7.8262	79.9%
humaneval	7.7830	11.2378	69.3%
mbpp	5.6396	7.5507	74.7%
mt-bench	3.9949	4.8305	82.7%

Conditions: thinking off, greedy (T=0, top_p 1, top_k 1), concurrency 1, block size 16, max 2048 new tokens, SGLang 0.5.18, target snapshot 6a9e13bd.

Only τ is reported. That sweep ran co-tenant with the training job, and τ is a count that survives a busy neighbour while throughput and speedup are wall clock and do not. Speedup gets measured on an exclusive card when training ends.

The two code suites sit lowest, which was predicted before measuring: the training corpus is 6.8% code against an intended 14.7%, because the Nemotron code split holds 175,000 rows but only 33,884 unique prompts and deduplication removes 73% of what was requested. Whether that is the cause or code simply converges slower is not yet decided — later checkpoints separate the two.

Training setup

• Corpus: 775,164 rows, Nemotron Post-Training Dataset V2 (English splits) + CodeAlpaca, every response regenerated by Qwen3.6-27B at T=0
• 6 epochs, global batch 8, lr 6e-4 cosine, warmup 0.04, max length 3072, block size 16, anchors 512 — the paper's stated settings; batch size is not in the paper and follows upstream's Qwen3.6 recipe
• 6×H200: two capture servers holding the target, four trainer ranks
• 576,704 optimizer steps, checkpoint every 48,058

Provenance and licensing

Derived from Qwen/Qwen3.6-27B (target weights, frozen embedding, and every training label is its greedy output) and NVIDIA's Nemotron Post-Training Dataset V2 plus CodeAlpaca. Both upstream licenses apply to anything derived from them; check them before redistribution or commercial use.

Known contamination

Eight GSM8K test items appear in the training corpus (2 exact, 6 contained), 0.0010% of 775,164 rows. None of the eight is among the 128 items the evaluation sweeps measure, so the τ above was not measured on anything trained on. The item indices are recorded in the GitHub repository; raising the GSM8K sample size in a future evaluation would pull them into the measured set and that run would have to report a figure excluding them.