Optimized SGLang for Qwen3.8 Flash-Next NVFP4 on 1x RTX PRO 6000: 171 tok/s, 524K, and HiCache/NIXL
I wanted to run this checkpoint properly on one 96 GB RTX PRO 6000, so I built an optimized SGLang runtime around it. I then qualified the entire stack I actually use and published the source, launch recipe, results, provenance, and validation suite in case anyone else wants to play with it.
The runtime is Pennyroyal. It is one patched SGLang source tree with two qualified TP=1 configurations:
RadixArk/Qwen3.8-Flash-Next-NVFP4 with native NEXTN MTP
Qwen3.8-27B FP8 with DFlash2
The Flash-Next configuration is not just a collection of launch flags. The optimized path combines:
FlashInfer GDN decode and prefill with BF16 recurrent state
FlashInfer CUTLASS MoE for the target and native-MTP layer
Triton QSA sparse prefill
FlashInfer's QSA wrapper resolving to XQA for sparse decode on exact SM120
Native NEXTN MTP with QSA index sharing
RecoverSSM with target, draft, verify, prefill, and accepted-state recovery CUDA graphs
Correct multimodal three-axis mRoPE
524,288-token factor-2 YaRN context
FP8 target/native-MTP KV with an automatically sized 824,384-token GPU pool
The QSA wording is deliberate. This runtime does not use TRTLLM-Gen on SM120. TRTLLM-Gen is an SM100-targeted path. Forced selection fails its architecture check, and directly loading its cubin on SM120 returns CUDA_ERROR_NO_BINARY_FOR_GPU (209). Pennyroyal enters the FlashInfer QSA wrapper, which resolves to XQA on exact SM120.
HiCache/NIXL is also part of the optimized runtime, not an unrelated cache benchmark. HiCache writes pages through host RAM using kernel I/O and page-first layout. NIXL persists the representation to POSIX FILE storage on NVMe using io_uring and O_DIRECT. For Flash-Next, that persistent representation includes packed target/native-MTP KV plus GDN, PLE, and compressed QSA state. Restoring KV alone would not restore this hybrid model coherently.
After a service restart, the approximately 490K request restored 489,856 of 489,879 input tokens and recomputed only 23. All three needles remained exact. The resulting 62,040.60 tok/s is effective restored-prefix throughput, not cold model prefill.
Canonical TP=1 results on one RTX PRO 6000 at a 450 W power cap:
64K cold prefill: 10,103.70 tok/s
Approximately 490K cold prefill: 7,872.15 tok/s, with 3/3 needles exact
C1 decode: 171.09 tok/s
C4 decode: 427.54 tok/s aggregate
Native MTP: 2.58 mean accepted length and 52.74% acceptance
Reasoning: 97.49/100 across 139,863 completion tokens
Tools: 30/30 exact and semantically correct
Complete vision validation passed
The real workload mattered more to me than another synthetic ceiling. Across 96 completed agentic requests and 100,666 generated tokens, Pennyroyal averaged 139.5 tok/s token-weighted with a 153.5 tok/s per-request median. The 90K to 279K input lane remained at 138.7 tok/s weighted and 148.4 tok/s median.
Pennyroyal v2.1 is a bounded correctness update to that optimized runtime. It adopts SGLang PR #36806's exact-SM120 QSA gate, explicitly excluding SM121/GB10, and adapts PR #35821 to prevent empty Mamba radix ghost nodes and bound accepted-state tracking across the eager, fused CUDA, and KDA paths. Seventy-two focused QSA, Mamba, and CUDA tests passed. Flash-Next and 27B/DFlash2 both retained graph capture, correct KV capacity, prefix reuse, and NIXL restart restoration.
Another operator independently validated the runtime with the official Flash-Next FP8 checkpoint, FP8 KV, native MTP, and TP=2/EP=2. Removing the optional overlap plan-stream setting allowed native MTP to operate normally. Their durable results were successful TP=2 FP8 MTP operation, decode scaling through C6, and a 3,182,848-token KV pool. That is separate external FP8 evidence, not a comparison with my TP=1 NVFP4 campaign.
Optimized runtime, recipes, backend documentation, and results: https://github.com/jpezzulli/sglang-rtxpro6000
Pennyroyal v2.1 Release: https://github.com/jpezzulli/sglang-rtxpro6000/releases/tag/pennyroyal-v2.1.0
Public validation suite and result catalog: https://github.com/jpezzulli/pennyroyal-validation
Flash-Next is still early and system-specific. These are measured results for one exact SM120 workstation, checkpoint family, and software stack, not universal SM120, SM121, or GB10 guarantees. I built it for myself, published the evidence, and figured this checkpoint's users were the people most likely to find it useful.