GB10 tile values confirmed for DFLASH verify shape + a memory-config note for spec decoding
Thanks for this quant and the unusually careful card — it carried our deploy. Two findings from running it with the incoai DFlash2 drafter on 2x GB10 (SGLang TP=2, PR #36507 branch):
Your GB10 tile (block_I=32, num_stages=1, threads=128) is also what fixes the DSA tilelang smem overflow at the DFLASH verify shape (num_tokens_per_req=8 requests 169,984 B > 101,376 B). Plain decode fits at stock tiles, so drafter-less boots never hit it — worth a card note for anyone adding spec decoding.
Your tested memory envelope (mem-fraction 0.84) does not survive a drafter: with D=8 the hybrid state pool needs per_req*(1+D) (~633 MB) plus ~ratio=5 per-request amplification, and fails with max_mamba_cache_size<=0. Working config: mem-fraction 0.88, --max-total-tokens 131072, --mamba-full-memory-ratio 2.
Measured with the drafter: 27.6 tok/s code decode (1.88x vs no-spec same stack). Full recipe + patches: https://github.com/beastllama/GLM-5.3-Flash-DFlash2-SGLang-2x-DGX-Spark
Thank you — both findings are now in the card, credited to you:
- The GB10 tile is documented as required once you add a drafter, with the point that plain decode fits at stock tiles so a drafter-less boot never surfaces it, and the DFLASH verify shape (
num_tokens_per_req=8) requests the same 169,984 B against the same 101,376 B limit. - The mem-fraction table is now explicitly labelled drafter-less, with your working configuration (
--mem-fraction-static 0.88,--max-total-tokens 131072,--mamba-full-memory-ratio 2) and theper_req * (1 + D)reason it is needed.
Separately, there is now a compressed-tensors branch (--revision compressed-tensors) — the same 4-bit expert tensors re-laid out to match RedHat's NVFP4 parameters, with the MTP layer at FP8 instead of FP4. Two honest caveats about how much that means for you specifically:
- Its main benefit is to vLLM users: it sidesteps the
ModelOptNvFp4FusedMoEalpha-fold bug that forces--moe-backend marlin. Your stack is SGLang, which is not affected by that. - The FP8 MTP layer helps the checkpoint's built-in MTP. You are drafting with the external incoai DFlash2 model, so layer 45 is not in your path and this most likely changes nothing for your 27.6 tok/s.
And a caution rather than a recommendation: stay on main for now. The SGLang path is known to work with the ModelOpt layout here — including the ignore-list fix that made it load — and nobody has yet loaded the branch under SGLang or vLLM. It is verified numerically only (round-trip cosine 0.998 vs BF16 source, tensor-name parity with RedHat's index), with no task evals.