Soomin33/Qwen3.8-Flash-Next-FP6-INT8

Qwen3.8-Flash-Next FP6-INT8

Qwen3.8-Flash-Next (180 B parameters including the 51 B n-gram/PLE embedding table) quantized for 2×64 GB GPUs: fp6 e2m3 routed experts, int8 dense projections, bf16 remainder, and the n-gram table as an fp8 sidecar served from pinned host RAM. Full 262,144-token context; KV cache, activations and recurrent state are not quantized.

Provenance: quantized from the official FP8 release (Qwen/Qwen3.8-Flash-Next-FP8, fine-grained fp8, block size 128), not from bf16 weights. Tensors listed as bf16 hold values inherited from that checkpoint. The n-gram table is fp8 e4m3 in the official release; the sidecar re-lays it out without changing values.

This is the second generation of the format. The first (v1, the FP6 repository: fp6 experts, bf16 everything else) remains available; this repository (v2) adds the int8 tier. v2 is produced from v1 with tools/int8_encode.py (CPU, ≈15 min); w8_manifest.json lists every int8 tensor with shape and measured error. The serving stack in patches/ loads either and keys on quantization_config.dense_int8.

	v1 (FP6)	v2 (this repository)
Routed + MTP experts	fp6 e2m3, group 64	same
Embeddings, lm_head, GDN out_proj, QSA q/k/v/o, shared experts	bf16	int8 symmetric, group 32, fp16 scales
Everything else	bf16	bf16
Shards on disk	99.97 GiB	97.63 GiB
KV pool, 2×64 GB, reference settings	292,288 tokens	425,024 tokens
Single-stream decode, reference stack	54–56 tok/s	61 tok/s median, 63 best

Why this build

The previous generation (v1 with its 2026-08 launcher: mem-fraction 0.945–0.955, chunked prefill 8,192, GPU image preprocessing, no input-logprob limit) crashed intermittently under agent traffic. Each cause was isolated and closed (STABILITY.md): MMU faults at the top of VRAM during long prefills, a poisoned CUDA context in the tokenizer process after large images, and out-of-memory on input-logprob requests. This build has run without a crash since those fixes, decodes 61 tok/s single stream against 54–56, and holds a 45 % larger KV pool. As of 2026-09-03 it is the fastest known public build of this model for sm80/CMP-class hardware.

Spec sheet

Hardware profile

GPUs	2× NVIDIA CMP 170HX: GA100, sm80, 70 SMs, 64 GB HBM2e at ≈1.49 TB/s, 1,410 MHz. No NVLink, no P2P, CUPTI disabled, no fp8/fp6/int8 tensor-core formats
Interconnect	PCIe gen2 (a CMP hardware limit), x4 link in this build; card-to-card traffic passes through host RAM; measured ≈1.1–1.3 GB/s
Host	AMD Ryzen 5 9600X (6C/12T), 96 GB DDR5 (92 GiB usable; ≈48 GiB pinned by the sidecar), NVMe
Software	NVIDIA driver 610.43, CUDA 13.3, torch 2.13, Triton 3.7.1, NCCL 2.29.7, SGLang fork fp6-stable (patches/)
Cost	≈ €5,000, 2026 second-hand prices

Measured (2026-09-02, this checkpoint, reference configuration)

Single-stream decode, steady state	60.9 tok/s median, 63.3 best (MTP accept length 1.78)
Single-stream incl. TTFT, 400-token replies	56–58 tok/s
Aggregate decode, steady state, 1 / 2 / 3 streams	≈61 / 85 / 101 tok/s (a 4th stream queues)
Prefill	1,126 tok/s end to end, 53k prompt (47.5 s); ≈1,270 tok/s per chunk. PCIe-bound on this hardware, see below
TTFT, short prompt	≈0.3 s
Context	262,144 per request; 425,024-token KV pool over 3 slots. Pool deliberately limited: mem-fraction-static 0.93 for memory safety, against 0.945–0.955 in earlier builds that crashed (STABILITY.md §2)
Long-context cost	none measurable: same decode rate at 209k as at 500 tokens
VRAM	61 of 64 GB per card in use; 4.8 GB free after graph capture
Load to ready	≈9 min

Step budget and hardware floors

Decode step at batch 1: ≈30.6 ms per card. Components:

	ms	Bound
Weight reads, ≈7.2 GiB per card per step at 1.49 TB/s	5.2	hardware
101 TP all-reduces, latency-bound (42 µs LL / 55 µs default, no P2P)	4.3–5.4	hardware, given TP=2
fp6 expert GEMV (GEMV3, tensor cores)	≈6	kernel
Attention / GDN / norm / hyper-connection chain	≈10	kernel, unfused
int8 dense GEMMs	2.0	kernel
Sampling, glue	remainder

Prefill is interconnect-bound on this hardware: 101 all-reduces × 21 MB per 4,096-token chunk over ≈1.2 GB/s is ≈2 s of each ≈3.2 s chunk, a ceiling of ≈2,000 tok/s at TP=2 on a gen2 x4 link. The CMP 170HX is limited to PCIe gen2; the same stack on cards with a PCIe 4.0 x16 link (≈25 GB/s practical, e.g. A100 PCIe) cuts the bus time per chunk to ≈0.1 s. Estimated chunk time then ≈1.3 s, ≈3,000 tok/s, bounded next by the expert GEMM. Per added decode stream the step grows ≈15 ms: ≈6 ms expert GEMV (rows), ≈9 ms unattributed (suspects: per-token sparse-attention verify, per-sequence graph nodes).

A 20–24 ms step (85–110 tok/s single stream at the present accept length) is reachable with known techniques. The figures here represent about one week of kernel work and are the limit of this release, not of the hardware. Contributions that fuse the attention/GDN chain, improve the all-reduce schedule, or batch the sparse-attention verify path are welcome; measurements taken with the harnesses in tools/ pair directly with the tables above and can be reported in the repository discussions.

Precision map

From the safetensors headers. Parameters in billions; rel-RMS is the relative RMS error of the stored weights against the FP8-derived bf16 source.

Component	Params	v1	v2 (this repository)	Size, v2
Routed MoE experts, 48 × 512, gate/up/down	120.80	fp6 e2m3 g64, fp16 scales, hi4/lo2 planes (6.25 bit/w), rel-RMS 2.44 %	same	87.89 GiB
MTP draft-layer experts	2.52	fp6	same	1.83 GiB
GDN in-projections, gates, conv (36 layers)	1.52	bf16	bf16	2.83 GiB
GDN out_proj (36)	0.57	bf16	int8 g32, 0.57–0.65 %	0.56 GiB
QSA q/k/v/o_proj (12 layers)	0.60	bf16	int8 g32; q 0.58–0.67, k 0.65–0.86, v 0.66–0.84, o 0.57–0.61 %	0.59 GiB
QSA indexer (12)	0.02	bf16	bf16	0.04 GiB
Shared expert gate/up/down (48)	0.24	bf16	int8 g32, 0.56–0.79 %	0.23 GiB
MoE routers	0.06	bf16	bf16	0.12 GiB
Hyper-connection mixers	0.63	bf16	bf16	1.19 GiB
Token embeddings	0.64	bf16	int8 g32, 0.53 %	0.63 GiB
lm_head (untied)	0.64	bf16	int8 g32, 0.54 %	0.63 GiB
Vision encoder + projector	0.45	bf16	bf16	0.84 GiB
MTP draft, non-expert	0.09	bf16	bf16	0.17 GiB
Norms, PLE projection, misc	0.04	bf16	bf16	0.07 GiB
Sidecar ple/ngram.bin	51.20	fp8 e4m3, one global scale	same file	47.68 GiB

Totals: 128.8 B parameters in shards + 51.2 B in the sidecar. v2 ≈ 6.5 bits/parameter (shards), ≈ 6.9 overall. KV cache bf16, GDN state fp32, activations bf16, accumulation fp32.

Selection rule for the int8 tier: dense projections with large input dimension and well-conditioned ranges, whose int8-g32 error (0.5–0.9 %) is below the expert error (2.44 %). Kept in bf16: routers and the QSA indexer (top-k selection boundaries; a 0.5 % logit perturbation changes routing on near-ties), GDN in-projections and gates (feed a recurrent state that integrates error over the sequence), hyper-connection mixers (scale the residual stream across 48 layers), norms, vision tower, MTP draft layers. QSA q/k/v are a single fused qkv_proj parameter in the serving model and are quantized together. Format details: FORMAT.md; procedure: QUANTIZING.md.

Quantization quality

All checks paired: same prompts, seeds, temperature 0 and server settings on both sides. Decoding is not byte-reproducible at temperature 0 (see Known limits), so single greedy diffs are not used as evidence.

Kernel level:

• fp6 decode bit-exact for all 64 e2m3 codes in the tiled, GEMV2 and GEMV3 kernels; bf16 cast of every decoded value exact.
• GEMV3 vs fp64: rel-RMS 1.66e-3 (bf16 output rounding), equal to GEMV2; element-wise ≤2 bf16 ulps from GEMV2 on the per-card shard shapes with real expert planes (layers 0/7/23/47, both cards); 224 adversarial cases per card (plane patterns × fp16 scale extremes × activation extremes), 0 failures, no inf/nan where fp64 is finite.
• int8 kernel: every code −127…127 in every group position × scale extremes × activation extremes, GEMV (M ≤ 16), dequant to cuBLAS (M > 16) and embedding paths vs fp64: 1,730 checks, 0 failures (75 skipped at fp32 overflow).

Model level:

• Per-layer output error of the int8 layers vs bf16 on real activations: 0.6–0.8 %.
• Teacher-forced log-probabilities, 6,131-token code/documentation corpus, 512-token windows: perplexity 1.6337 (v2). v1 side not yet captured.
• Long-context recall: one planted code in random-word haystacks of 33k / 104k / 209k tokens at depths 0.1 / 0.5 / 0.9: 9/9 (v2), with two concurrent requests, prefill 1,050–1,110 tok/s.
• Deterministic smoke set (arithmetic, bat-and-ball, capital/year, iterative code, list recall, prose): identical correct answers on v1, v2, v2+GEMV3.

Benchmarks

Served by this checkpoint on the hardware above, thinking mode, temperature 0 (the official protocol samples at T = 1.0, top-p 0.95, top-k 20).

GPQA Diamond (198 questions, simple-evals prompt, single greedy run)

Run	Checkpoint, kernel	Output budget	Result
Official reference (unquantized, sampled)	n/a	n/a	91.7
45-question seeded subset	v1	16,384	38/38 answered correct, 7 hit the budget
Partial full run	v1	65,536	66/71 = 93.0 % (1 truncation)
Full run, stopped at 160	v2, GEMV2	65,536	86.9 % raw; 90.8 % excluding 7 truncations
262k-budget run, 79 questions	v2, GEMV2	260,000	75/79 = 94.9 %, 0 truncations
262k-budget run, in progress	v2, GEMV3	260,000	20/21; same letter as GEMV2 on 20/21

Paired:

• v2 vs v1, 71 common questions: 68 vs 66; same letter 68/71; v2 alone correct on 2, v1 alone on 0, both wrong with different letters on 1. Within binomial noise (±3 %) at n = 71.
• v2 at 262k vs 65k budget, 79 questions: 75 vs 70; same letter 72/79; 262k alone correct on 6 (all 3 questions truncated at 65k included), 65k alone on 1. Completion tokens at 262k: median 5.5k, mean 11.5k, max 76k.
• Control: Qwen3.8-27B (dense, unquantized, same harness, 65k budget): 173/198 = 87.4 % (official 89.2), 8 truncations; on the 45-question subset 39/45 vs Flash-Next-FP6 38/45, with 4 wrong completed answers vs 0.

SWE-bench Pro

Official reference 62.5. Seeded 50-task subset, mini-swe-agent 2.4.6, temperature 0, 16,384 max tokens per turn, 75 steps, 40-min wall clock per task, local Docker evaluation: 9/9 resolved (7 Python, 2 JavaScript; mean 40 steps, 14.5 min per task) on v2 + GEMV2, stopped there to keep the baseline on one kernel. Small n; establishes multi-file repository work through tool calls, not a leaderboard number.

Serving

Reference stack: SGLang fork (branch fp6-stable, 16 files changed against upstream 7c66045d71; diff in patches/sglang-fp6.patch), Triton 3.7.1, torch 2.13 / CUDA 13.3, NCCL 2.29, 2× 64 GB sm80 GPUs at TP=2, ≈50 GiB pinned host RAM for the sidecar (cudaHostAlloc, not /dev/shm; 64 GiB system RAM minimum). All kernels are Triton with integer bit-manipulation decode; no fp8 tensor cores or Hopper/Blackwell intrinsics. sm89/sm90/sm120 untested.

                 ┌───────────────────────────────────┐
                 │ Disk (≈145 GiB)                   │
                 │ 25 safetensors shards (≈98 GiB)   │
                 │ ple/ngram.bin       (≈48 GiB)     │
                 └─────────────────┬─────────────────┘
                                   │ read once at startup (≈9 min incl. graph capture)
          ┌────────────────────────┴────────────────────────┐
          ▼                                                 ▼
┌───────────────────────┐  101 all-reduces / step   ┌───────────────────────┐
│ GPU 0 (64 GB, ≈61 in  │◄════════ PCIe ══════════► │ GPU 1 (64 GB, ≈61 in  │
│ use)                  │   (TP=2: each GPU holds   │ use)                  │
│  fp6 experts (half)   │    half of every tensor)  │  fp6 experts (half)   │
│  int8 dense (half)    │                           │  int8 dense (half)    │
│  bf16 GDN/hc/vision   │                           │  bf16 GDN/hc/vision   │
│  MTP draft head       │                           │  MTP draft head       │
│  KV cache, bf16       │  425,024-token pool       │  KV cache, bf16       │
│  GDN state, fp32      │  (24 mamba slots)         │  GDN state, fp32      │
│  decode CUDA graphs   │                           │  decode CUDA graphs   │
└──────────┬────────────┘                           └──────────┬────────────┘
           │  zero-copy gather, ≈2.5 KB/token (16 rows × 160 B)│
           ▼                                                   ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ Host RAM: n-gram / PLE table, fp8, 47.7 GiB, cudaHostAlloc-pinned       │
│ (vocab-sharded across the two ranks under TP; never occupies VRAM)      │
└─────────────────────────────────────────────────────────────────────────┘

The n-gram table is an embedding lookup, never a matmul operand: ≈16 rows (2.5 KB) per token are gathered from pinned host memory over PCIe. Compute weights stay resident in VRAM.

Reference configuration (launch-example.sh)

Setting	Value	Basis
Speculative decoding	built-in MTP head (NEXTN), 1 step, 1 draft token	accept length ≈1.8; two drafts measured no faster
Decode CUDA graphs	on, batch sizes 1–3	decode was ≈99 % CPU-issue-bound without them
Prefill CUDA graphs	off	capture fails on sm80
--max-running-requests	3	fp6 GEMV path requires tokens × top_k ≤ 64 rows; 3 sequences × 2 MTP tokens × top-10 = 60. A 4th sequence falls to the 2.8× slower tiled GEMM. Without speculation: 6
Chunked prefill	4,096	halves prefill workspace vs 8,192 at −11 % prefill (STABILITY.md §2)
--mem-fraction-static	0.93	deliberately below the 0.945–0.955 of earlier builds; ≈4.8 GB free per card after capture; rule: ≥1 GB free after a long prefill
--max-mamba-cache-size	24	pins the GDN state pool; freed VRAM becomes KV pool
Image processor	pil	the torchvision processor allocates GPU memory in the tokenizer process, outside the memory budget (STABILITY.md §3)
NCCL	NCCL_P2P_DISABLE=1, SHM transport	decode all-reduce 55 µs SHM vs 93 µs socket
Expert decode kernel	GEMV3 (SGLANG_FP6_GEMV3=1), fp16-placement decode (SGLANG_FP6_DECODE_FP16=1)	kill-switches revert to GEMV2
Request logging	off	level-3 capture records every prompt and output

Decode speed by change

Date	Change	Single stream
2026-08-27	TP=2, no speculation	13.6–16.9 tok/s
2026-08-28	MoE tile fix, GEMV2 decode kernel, Triton packed-decode attention	27–28, flat to 47k
2026-08-29	MTP head (1 draft), chunk 8,192, prefill tile 128	39–44
2026-09-01	decode CUDA graphs, blocking copy events, GEMV2 retune	54
2026-09-02	fp16-placement fp6 decode	56
2026-09-02	int8 dense (KV pool 292k to 425k)	56
2026-09-02	GEMV3 tensor-core expert kernel	61

Details, methodology and negative results: OPTIMIZATION-REPORT.md.

Stability

Four failure modes found under agent traffic and closed; incident timelines, root causes and verification in STABILITY.md.

1. Input-logprob requests over long prompts crashed the server. A /generate request with return_logprob and logprob_start_len=0 on a ≈5k-token prompt (the shape of echo=True, logprobs=N) materializes [positions × vocab/TP] fp32 logits per prefill chunk: 2.0 GB per tensor per card at chunk 4,096 with the 248k vocabulary, plus the log-softmax, into ≈0.8 GB of headroom. Both TP ranks died. Fix: a cap on scored prompt positions per request, SGLANG_MAX_INPUT_LOGPROB_TOKENS (default 1,024), enforced in the tokenizer manager as HTTP 400 with a message stating the limit and the windowed alternative, and again in the scheduler as an abort for callers that bypass validation. Verified after relaunch: 5,000 positions gives 400 with the server up; 500 positions gives 499 logprobs. 1,024 positions cost ≈1 GB per card, a ≈4× margin below the ≈4,000 that exhaust an idle card. tools/tf_logprobs.py scores long texts in 512-token windows over the prefix cache.
2. MMU faults (Xid 31) during long prefills: the allocator grows ≈3.4 GB at chunk 8,192 (≈1.7 GB at 4,096) with context; on these cards a first-touch write into the top ≈140 MiB of VRAM faults instead of raising OOM. Reproduced with a standalone allocation. Rule: ≥1 GB free at steady state after a long prefill, hence chunk 4,096 and mem-fraction 0.93; validated with a 209k-token prefill.
3. Image requests failing after one large image while text served: the fast image processor runs on the GPU in the tokenizer process, which has no memory-budget share; its context was poisoned by an Xid 13. --image-processor-backend pil; the tokenizer process now holds no VRAM; 16.7 MP images pass.
4. Host-RAM livelock during weight loading on a 64 GiB box (48 GiB pinned table + 100 GiB streaming through the page cache). Launcher cgroup cap (MemoryHigh=54G, MemoryMax=57G) and a MemAvailable ≥ 52 GiB gate.

Known limits

• Temperature-0 decoding is not byte-reproducible run to run: decode logits carry allocator-layout-dependent noise (KV page and GDN slot assignment) that flips tokens where the top-2 gap is small (≈1 % of steps below 0.01 nats). Two runs of the same kernel diverged at the same token as a GEMV3-vs-GEMV2 comparison (gap 0.125 nats). No outcome-level effect measured (30/30 repeated greedy runs on a reasoning battery; paired GPQA flips symmetric). All custom kernels verified bitwise deterministic in isolation. Consequence: compare configurations with paired benchmarks and element-wise kernel tests, not text diffs.
• Concurrency ceiling 3 with speculation (GEMV row cutoff); 6 without.
• Speculation and pipeline parallelism are mutually exclusive in SGLang; PP works with the patches (output-identical) but is slower than TP=2 on PCIe-only hosts.
• Prefill CUDA graphs cannot be captured on sm80; the torch profiler stalls the scheduler; CUPTI is unavailable on CMP dies. All timings are CUDA events and in-graph microbenchmarks.
• A prefilling stream pauses the decoding streams (no mixed chunking in the reference configuration).

Repository contents

• 25 safetensors shards (97.6 GiB) + model.safetensors.index.json, config.json (fp6 quantization_config with the dense_int8 block), w8_manifest.json, tokenizer files, chat template, generation and preprocessor configs.
• ple/ngram.bin + ple/ngram-manifest.json: the n-gram sidecar (byte-identical to v1).
• patches/: sglang-fp6.patch (the fork as one diff against upstream 7c66045d71) and the new files whole: fp6.py (quant method, MoE dispatch, int8 wiring), fp6_moe_kernels.py (tiled, GEMV2, GEMV3, split-K fp6 kernels), w8_dense.py (int8 skinny GEMM, dequant path, embedding gather), qsa_packed_decode.py (Triton packed-decode attention). The diff also carries the input-logprob guards, the sidecar loader, qwen4_exp pipeline-parallel support, the QSA indexer sync fix, CPU image decode and env-gated phase timers.
• tools/: encoders (fp6_encode.py, int8_encode.py, ple_extract.py), code-selection probe, fp6 kernel correctness battery and soak, GEMV3 and int8 full-range and real-tensor tests, input-logprob guard unit test, and the measurement harnesses behind every number above (decode_bench.py, prefill_bench.py, concurrency_bench.py, needle_test.py, tf_logprobs.py, smoke_prompts.py, gpqa_diamond_eval.py).
• launch-example.sh: reference launcher, parameterized.
• OPTIMIZATION-REPORT.md, STABILITY.md, QUANTIZING.md, FORMAT.md, LICENSE (Qwen Community License 1.0, inherited from the base model).