Soomin33/Qwen3.8-Flash-Next-FP6

Qwen3.8-Flash-Next — FP6 (e2m3) experts + FP8 n-gram sidecar

Community quantization of Qwen3.8-Flash-Next targeting 2×64 GB GPUs with the 51B-parameter n-gram/PLE embedding table served from host RAM.

Provenance: this model was quantized from the official FP8 release (Qwen/Qwen3.8-Flash-Next-FP8, fine-grained fp8, block size 128) — not from the original BF16 weights. Tensors listed as bf16 below are bf16 containers holding values inherited from that FP8 checkpoint. The n-gram table ships in fp8 e4m3 in the official release; this repo only re-lays it out into a flat sidecar file — its values are unchanged.

Precision map

Verified from the safetensors headers of the released checkpoint.

Component	Params	Stored as	Size
MoE experts (gate/up/down, incl. the MTP layer experts)	123.31 B	fp6 e2m3, group 64, fp16 scales, hi4/lo2 uint8 planes (6.25 bits/weight)	89.7 GiB
GDN linear attention (36 layers)	2.09 B	bf16	3.9 GiB
Full attention + QSA indexer (12 layers)	0.81 B	bf16	1.5 GiB
Gated residual (hc) + misc	0.64 B	bf16	1.2 GiB
Token embeddings	0.64 B	bf16	1.2 GiB
lm_head (untied)	0.64 B	bf16	1.2 GiB
Vision encoder	0.31 B	bf16	0.6 GiB
Shared experts (not quantized)	0.24 B	bf16	0.45 GiB
MoE routers/gates	0.06 B	bf16	0.12 GiB
MTP draft, non-expert part	0.03 B	bf16	0.06 GiB
Sidecar: n-gram / PLE table (ple/ngram.bin)	51.2 B	raw fp8 e4m3, one global scale	47.7 GiB

Totals: 180 B parameters, ≈147 GiB on disk (≈6.6 bits/param checkpoint, ≈7.0 bits/param overall). No KV-cache or activation quantization anywhere; runtime uses bf16 KV and fp32 GDN/mamba state.

See FORMAT.md for the exact fp6 packing and sidecar layout.

Quantization quality

• The Triton fp6 dequant kernel is bit-exact against the reference encoder for all 64 codebook values.
• Expert-weight relative RMS error vs. the FP8 source: 0.0244 (predicted 0.0245 by the code-selection probe; e2m3 was chosen over e3m2 and int6 on measured GEMM-space error, not weight-space error).
• Fused MoE kernels validated against a torch reference on the same dequantized weights (max rel. err 3.2e-3 / 6.3e-3).

Benchmarks

GPQA Diamond, 45-question seeded subset (simple-evals prompt format, temperature 0, thinking enabled), served by this exact checkpoint on 2× CMP 170HX:

• Token budget set at 16,384. Of the questions it did answer, 100% were correct — 38/38. The remaining 7 exceeded the thinking budget and went unanswered rather than wrong.
• Official reference for the unquantized model: 91.7 (larger output budget).
• This is a quantization health check, NOT a leaderboard entry: n=45, single greedy run, one prompt format.

We are currently working on expanding the benchmarks: a full 198-question GPQA Diamond run, with DeepSWE 1.1 planned later.

Hardware support

Built for and validated on NVIDIA Ampere (sm80, GA100 die) — the entire kernel set (fp6 dequant + fused MoE, PLE gather, QSA paths) is Triton with an sm80-safe fp6/fp8 decode written as integer bit manipulation, no fp8 tensor cores or Hopper/Blackwell intrinsics required. Anything an A100 can run, this runs; the validation hardware was the GA100-based CMP 170HX. Newer architectures (sm89/sm90/sm120) should work via the same Triton kernels but are untested here. Minimum footprint: 2× 64 GB GPUs (or ≥110 GB on one GPU) plus ≈50 GiB host RAM for the pinned n-gram sidecar.

Serving

Reference stack: SGLang (fork with the patches in patches/), 2× 64 GB sm80 GPUs (tested on CMP 170HX, PCIe gen2 x4), TP=2, plus ≈50 GiB of pinned host RAM for the sidecar (64 GiB system RAM minimum; the table is cudaHostAlloc pinned, not /dev/shm).

Serving architecture — what lives where

                 ┌───────────────────────────────────┐
                 │ Disk (≈147 GiB)                   │
                 │ 25 safetensors shards (≈99 GiB)   │
                 │ ple/ngram.bin       (≈48 GiB)     │
                 └─────────────────┬─────────────────┘
                                   │ read once at startup
          ┌────────────────────────┴────────────────────────┐
          ▼                                                 ▼
┌───────────────────────┐  2 all-reduces per layer  ┌───────────────────────┐
│ GPU 0 (64 GB, ≈58 in  │◄════════ PCIe ══════════► │ GPU 1 (64 GB, ≈58 in  │
│ use)                  │   (TP=2: each GPU holds   │ use)                  │
│  fp6 experts (half)   │    half of every tensor)  │  fp6 experts (half)   │
│  bf16 attn/GDN (half) │                           │  bf16 attn/GDN (half) │
│  KV cache, bf16       │                           │  KV cache, bf16       │
│  GDN state, fp32      │                           │  GDN state, fp32      │
└──────────┬────────────┘                           └──────────┬────────────┘
           │  zero-copy gather, ≈2.5 KB/token (16 rows × 160 B)│
           ▼                                                   ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ Host RAM: n-gram / PLE table — fp8, 47.7 GiB, cudaHostAlloc-pinned      │
│ (vocab-sharded across the two ranks under TP; never occupies VRAM)      │
└─────────────────────────────────────────────────────────────────────────┘

Key properties of this layout:

• The 51B-parameter n-gram table never touches VRAM. It is a pure embedding lookup — never part of a matmul — so a GPU kernel gathers the ≈16 needed rows per token directly from pinned host memory over PCIe: ≈2.5 KB/token, microseconds even on a slow link. This is what lets a 180B-parameter model serve from 128 GB of VRAM.
• Compute weights (experts + attention) stay resident in VRAM at all times; KV cache is bf16 and GDN/mamba state fp32 — nothing about the cache or activations is quantized.
• Under PP (--pp-size 2 --tp-size 1) the split is by layers (0–23 / 24–47) instead of by tensor halves; the n-gram table then lives entirely with the stage that owns the PLE layer, and the inter-GPU traffic drops to one ≈20 KB hidden-state hop per token.
• Host RAM sizing: the pinned table is unreclaimable, so a 64 GiB box runs tight during weight loading — the reference launcher gates on MemAvailable and can wrap the server in a cgroup memory cap to keep weight-streaming page cache from stalling the machine.

Measured on the test box (CMP 170HX ×2, sm80, PCIe gen2 x4): ≈17 tok/s single-stream plain decode (TP=2), 23–33 tok/s with the built-in MTP head (EAGLE, steps 3, topk 1), 33–39 tok/s aggregate at 6 concurrent requests, ≈800–1,100 tok/s prefill. Pipeline parallelism (--pp-size 2 --tp-size 1) is supported by the included patches and verified output-identical; on this box it measures 13 tok/s single / ≈36 aggregate.

Where the decode latency comes from (instrumented)

The single-stream decode latency is not interconnect. Phase-level timers in the pipeline-parallel scheduler loop (included in the patches, env-gated behind SGLANG_PP_STEP_TIMING=1) itemize a batch-1 token at ≈77 ms as:

• Scheduler loop, batch prep, proxy/metadata transfers between stages, PP machinery: ≈1–2 ms combined.
• N-gram sidecar gather from pinned host RAM: microseconds (16 rows × 160 B/token; the PLE-owning stage measures the same as the PLE-free one).
• The GPU kernel chain itself: ≈73 ms — ≈36–38 ms per pipeline stage, spread across roughly two thousand small kernels per token: GDN state updates, the QSA indexer/top-k/gather sequence, and the multi-stage fp6 MoE dispatch. Each kernel is individually tiny; the cost is their serial execution floor. CUDA graphs do not help — measured identical with graphs on and off — because launch overhead is not the problem, execution is.

A control run of a mature dense-ish model (Qwen3.8-27B) through the same PP machinery on the same PCIe gen2 x4 link pays <1 ms of overhead per token — the interconnect and scheduler are exonerated; the cost is specific to this architecture's day-one kernels.

Implication for buyers and optimizers: this kernel chain, not bandwidth or interconnect, is currently the biggest hindrance to reaching 50 tok/s single-stream — even on 2× A100 with NVLink. Better links move ≈1–2 ms of a ≈77 ms token. The remaining speed lives in kernel fusion on the GDN/QSA/MoE paths, which is ordinary post-release kernel maturation — as those land upstream, this checkpoint gets faster on unchanged hardware.

Known limits of the reference stack (upstream issues, not weight issues):

• Batched (bs>1) MTP speculative decoding deadlocks under TP=2 — run speculation with max-running-requests 1, or disable speculation for concurrent serving. PP and speculation are mutually exclusive in SGLang.
• Speculative CUDA-graph capture has a flaky stream race on sm80.
• flash-attn-4 >= b28 required (b19 has a PackGQA compile bug on sm80).

Repository contents

• 25 safetensors shards + index, config.json, tokenizer files.
• ple/ngram.bin + ple/ngram-manifest.json — the sidecar and its layout.
• patches/ — SGLang fork patches (fp6 quant method, fused MoE kernels, PLE sidecar loader, qwen4_exp pipeline-parallel support, QSA indexer sync fix, env-gated PP step timers).
• tools/ — fp6_encode.py (encoder), ple_extract.py (sidecar builder), fp6_probe.py (code-selection probe), test scripts.
• launch-example.sh — reference SGLang launcher (parameterized version of the launcher this model was validated with).
• LICENSE — Qwen Community License 1.0, inherited from the base model.
• FORMAT.md — byte-level format spec for third-party loader authors.