ByteResearch/Cygnus-12B-20260820-ck600-FP8Dynamic-AutoRound-DSpark

Cygnus-12B-20260820-ck600 · FP8-Dynamic (W8A8) + AutoRound

The deployed 0820 model (Cygnus-12B-20260820-W8A16-AutoRound-DSpark) re-quantised from its bf16 mother with FP8 per-channel weights + per-token dynamic activations and AutoRound learned rounding. Same weights, same lineage, different number format — and that alone is worth +31% to +61% aggregate throughput on the business workload.

Lineage

item	value
bf16 mother	gemma-4-12B-dpo2400-merged + ck600 LoRA (r=64, α=128), merged
checkpoint	ck600 — the same batch as ck500, step 600
recovered from	the production serving log (/workspace/frameworks/online6-probe-models/0820) plus opd_eval/ck600/adapter_config.json
deployed counterpart	Cygnus-12B-20260820-W8A16-AutoRound-DSpark (W8A16, INT8 weight-only, 19 GB)
this artifact	FP8 W8A8, compressed-tensors / float-quantized, 12.13 GiB of weights

Quantisation recipe

CUDA_VISIBLE_DEVICES=0 python quantize.py \
  --model_path <bf16 mother> \
  --quant_path <out> \
  --data_path /workspace/opd_eval/calib_31b_full_msgs.jsonl \
  --nsamples 50 --seqlen 8192 --iters 500

• Scheme: INT8_W8A8 preset with data_type/act_data_type swapped to fp. AutoRound has no "FP8 per-channel + per-token dynamic" preset; the INT8 one is the correct template because it already exports per-channel weights plus per-token dynamic activations. Export format must be llm_compressor — auto_round cannot carry the activation config and silently falls back to a fake format.
• Calibration is the part that is easy to get silently wrong. It must be teacher-side complete dialogues, 50 samples, seqlen covering the whole conversation:

recipe	calibration	seqlen	iters	deployment tax
old (❌)	student's own generations	4096 (still truncated)	200	−3.4
this run (✅)	50 teacher dialogues	8192	500	−0.1

The deployed W8A16 artifact used nsamples=192, seqlen=1024, i.e. the older side of that table. Recipe self-check from this build's log:

Loading dataset from /workspace/opd_eval/calib_31b_full_msgs.jsonl...
Prepared 50 samples. Shape: torch.Size([1, 8192])
被 seqlen 截断的样本: 0/50 (0.0%)

Throughput

The business yardstick: input 3600 / output 400 tokens, prefix caching OFF, 16 and 32-way concurrency, metric = aggregate output tok/s, single H100, vLLM 0.27.1 with DSpark (7 draft tokens, greedy).

prod_cfg = the deployed serving values (max_num_batched_tokens=4096, max_num_seqs=16, gpu_memory_utilization=0.7). tuned_cfg = 4096 / 128 / 0.9.

All arms below were taken in one session with the two models interleaved, because cross-session readings on this box drift by about 10% for an unchanged config (the same setting measured 1962.8 and 2163.1 tok/s an hour apart). The production pair was run twice (r1 and r3) to make the drift visible.

build	serving cfg	16-way	32-way
deployed W8A16	prod_cfg	1528.9 (r1 1552.7 / r3 1505.0)	1596.6 (r1 1601.4 / r3 1591.8)
this model	prod_cfg	2195.7 (+43.6%)	2240.6 (+40.3%)
deployed W8A16	tuned_cfg	1559.9	1742.3
this model	tuned_cfg	2161.7 (+38.6%)	2421.3 (+39.0%)

Best combination, this model at tuned_cfg 32-way, is +51.7% over the deployed artifact (2421.3 vs 1596.6). Quantisation alone, at the deployed serving config, is worth +40% to +44%.

Latency (median / p90, seconds), 32-way, from the same interleaved session:

build	serving cfg	median	p90
deployed W8A16	prod_cfg	7.18–7.26	8.76–9.08
this model	prod_cfg	5.12–5.17	6.35–6.51
this model	tuned_cfg	4.47	7.41

Where the gain comes from

W8A16 quantises only the weights and keeps activations in bf16, so every GEMM has to dequantise the weights back and run a bf16 matmul — it never touches the FP8 tensor cores. It saves memory bandwidth, not compute, which is why it helps at low concurrency and does nothing once the workload turns compute bound. W8A8 quantises both sides and runs on FP8 tensor cores directly.

On this shape the wall clock is ~65% prefill (isolated with an output=1 probe: 4.3–4.9 s of a ~7.0 s turn at 32-way), and prefill runs at roughly 65% of the H100 dense FP8 peak. It is compute bound, so the number format is the lever.

The tuned_cfg column is a second, independent win: the deployed max_num_seqs=16 caps concurrency at 16, so a 32-way test was really running at 16. Raising it admits the load that was always offered.

Quality

Perplexity in the vLLM harness, wikitext-2 test, 512 chunks × 512 tokens:

model	PPL	vs bf16	vs W8A16
bf16 mother	79.7336	—	−7.6%
deployed W8A16	86.2890	+8.2%	—
this model	79.6588	−0.09%	−7.7%

The FP8 build is at parity with the unquantised bf16 mother and 7.7% better than the W8A16 artifact it replaces — while being 31–61% faster. The learned rounding plus the corrected calibration recipe recovered everything the format would otherwise have cost, and then some.

512 chunks rather than 24: the 24-chunk setting has a standard error around ±6% on this corpus, which cannot resolve differences of this size at all.

What we did not use, and why

This build does not use RCO (per-tensor bit allocation) or the code-space GSQ refinement, and that is a deliberate, measured decision rather than an omission:

• RCO's variable is "how many bits does each tensor get". In this scheme every Linear layer is FP8, so there is no allocation to search.
• Both techniques only buy anything if a cheaper compute format exists to move to. On Hopper it does not: there are no FP4 tensor cores, and FP8 is the compute floor. We verified the FP4 route directly — a mixed FP8+NVFP4 checkpoint loads in vLLM on sm_90 but as bf16 (22.83 GiB for an 11.26 GiB file) and delivers nothing.
• They become relevant again below FP8 (W4A8 / pruned FFN), where quality has to be recovered. That is the next track, not this artifact.

DSpark: what we verified on the draft in this repo

dspark_draft/ (6.4 GB) ships with the model and is not optional: its config has enable_confidence_head: true and confidence_head_with_markov: true, i.e. the draft carries a trained per-token acceptance head.

Static schedule table — verified equivalent, not a gain, for this shape. The ck500 card recommends num_speculative_tokens_per_batch_size = [[1,16,7],[17,32,3],[33,4096,1]]. At batch ≤ 16 the table specifies 7 drafts, which is exactly what the plain num_speculative_tokens: 7 config already does, so the two should be identical — and an interleaved A/B/A/B/A/B at 16-way says they are:

arm	16-way, aggregate
plain, 7	2163.1 / 2166.1 / 2157.0
with table	2164.4 / 2164.8 / 2160.5

Mean difference 0.06%. An earlier non-interleaved reading suggested +9% for the table at 16-way; that was machine drift, not the table. At 32-way the table switches to 3 drafts and measured neutral to slightly negative here.

Adaptive verification — cannot be enabled on the installed vLLM. Two prerequisites are needed, and only the first one is satisfiable locally:

1. Model side: upstream gemma4_dspark.py drops the head's weights (gemma4_dspark.py:299-300, if "confidence_head" in name: continue) and never builds the module. The ck500 repo's patches/gemma4_dspark_confidence_head.patch fixes exactly that — it adds DSparkConfidenceHead and deletes those two lines. Confirmed by reading both.

2. Serving side: enable_adaptive_verification does not exist in any vLLM installed on this box. SpeculativeConfig has no such field, and the string appears in zero files across all four vLLM environments (0.25.1 and 0.27.1). The model patch does not add it; it needs a scheduler that implements adaptive verification, which is a different build.

   The ck500 card reaches the same practical conclusion from the other end — its adaptive gains are same-backend paired numbers, and the TRITON_ATTN backend it requires costs back the whole gain (its own guidance: keep the static table, zero patch, FA2 default).

So: keep the default FA2 attention path and the plain DSpark config.

Known negatives

Measured on the same rig, so they do not have to be rediscovered:

knob	effect
--kv-cache-dtype fp8	−25% throughput (1191 / 1362 vs 1596 / 1769). Do not use.
raising max_num_batched_tokens 4096 → 16384/32768	no gain; at 16-way it is slightly negative
DSpark num_speculative_tokens 7 → 3	no gain
DSpark static schedule table	no gain for this shape (see above)
DSpark adaptive verification	not enable-able on the installed vLLM (see above)
comparing throughput across sessions	this box drifts ~10%; always interleave arms

Serving

vllm serve <this dir> \
  --trust-remote-code --max-model-len 65536 \
  --max-num-seqs 128 --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.9 --no-enable-prefix-caching \
  --speculative-config '{"method":"dspark","model":"<abs path>/dspark_draft","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

Model loading takes 15.42 GiB including the DSpark draft.