Qwen3.8-Flash-Next-E384-FP4-Local

A compact build of nvidia/Qwen3.8-Flash-Next-FP4-Local found by Neural Architecture Search (NAS) over the routed-expert configuration: it keeps 384 of the 512 routed experts in every decoder MoE layer and in the MTP layer.

Unchanged from the original:

• top-10 routing and the shared experts;
• attention, the n-gram PLE table, the vision tower and the tokenizer.

This is an unofficial derivative, not produced or endorsed by Alibaba or NVIDIA.

	original (512 experts)	E384
weights	102.14 GB = 95.12 GiB	85.66 GB = 79.78 GiB (−15.35 GiB, −16.1 %)
routed experts / layer (decoder + MTP)	512	384
active experts / token	10 + shared	10 + shared

Checkpoint format. The checkpoint keeps NVIDIA's exact format:

• Every kept tensor is a byte-for-byte copy: packed NVFP4 weights, CSF-coded E4M3 block scales, global and input scales, MXFP8 linears and the NVFP4 PLE table.
• Only the 49 router matrices change: they keep the rows of the experts chosen by the search.
• text_config.num_experts is 384.

It runs in the same containers as the original (see the base model card).

Benchmarks (one B200, vLLM 0.31.0, same harness and flags for both models)

Scoring is strict:

• A response that hits the token budget before its final answer counts as wrong.
• Answers are read only from the final content (after </think>).

Sampling: temperature=1.0, top_p=0.95, top_k=20, except HumanEval (greedy) and BFCL (bfcl-eval default 0.001). These are single runs; treat ±2 points as noise on the small sets.

capability	benchmark	setting	original	E384	Δ
reasoning (science)	GPQA-Diamond (198)	effort=low, 32k	85.9	77.3	−8.6
English knowledge	MMLU test (14,042)	effort=low, 16k	92.0	86.9	−5.1
Chinese knowledge	C-Eval val (1,346)	effort=low, 16k	93.5	89.0	−4.5
math	AIME 2026 (30 × 4, pass@1)	effort=xhigh, 64k	95.0	93.3	−1.7
vision	MMMU val (900)	effort=low, 16k	81.1	79.4	−1.7
code	HumanEval (164)	greedy, effort=low	97.0	97.6	+0.6
tool use	BFCL v4 Non-Live AST	effort=low	88.2	88.6	+0.4
	BFCL v4 Live AST		79.3	78.9	−0.4
	BFCL v4 Multi-Turn Base (200)		62.5	71.5	+9.0
	BFCL v4 relevance / irrelevance		81.3 / 80.1	81.3 / 78.6	0 / −1.5

Code, tool use, math and vision are preserved. The cost of the 25 % smaller expert set lands on knowledge-heavy multiple choice (GPQA, MMLU, C-Eval). GPQA in the table is a quick self-test at low effort with a 32k budget.

GPQA-Diamond at higher reasoning effort

Same strict scoring and sampling (temperature=1.0, top_p=0.95, top_k=20, min_p=0); single run, no truncations at the full budget.

setting	original (512)	E384	Δ
effort=low, 32k budget	85.9	77.3	−8.6
effort=xhigh, 62,500-token budget (max-model-len 65,536)	—	76.8 (25 of 198 hit the budget)	—
effort=xhigh, 259,000-token budget (max-model-len 262,144)	90.9 (180/198)	85.9 (170/198)	−5.1

Notes:

• The original's 90.9 is within 0.6 points of NVIDIA's published 91.5 for the FP4 checkpoint, which validates the harness.
• With enough thinking budget there are no truncated answers on either model, and the gap narrows from −8.6 to −5.1.
• E384 thinks longer: median reasoning 16,817 vs 9,458 tokens (mean 25,832 vs 17,657; longest 126,083). Give it a large context: at a 62,500-token budget, 25 answers ran out of tokens and counted as wrong (13 of them were correct at 259k).

Both models were evaluated with the same vLLM 0.31.0 build, loader and serving flags.

Faster MTP speculative decoding (lossless): mtp_draft/

The MTP layer uses the same 384-expert architecture. Native MTP acceptance changes little: τ = 4.40 vs 4.49 for the 512-expert model at k=5.

mtp_draft/ applies the recipe of autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK to the Flash-Next MTP layer, so drafting reads far fewer bytes per step:

1. 64k draft vocabulary (artifacts/draft_vocab_64k.pt: 64,026 rows, special tokens forced). It uses the same tokenizer as Qwen3.8-27B and covers 97–99.8 % of the tokens this model generates on C-Eval, MMLU, GPQA, AIME, HumanEval and MMMU.
2. Draft lm_head. The draft head is those rows of the BF16 target head, re-quantized to NVFP4 W4A16 (Marlin). Greedy drafting takes the local argmax and maps it back to target ids (use_local_argmax_reduction).
3. MTP BF16 linears → NVFP4 W4A16: fc_embedding, fc_hidden, attention qkv_proj / o_proj, and the shared expert. The routed MTP experts were already NVFP4. The router, hyper-connection mixers and QSA indexer stay BF16.

Lossless. The target model and the lm_head it verifies with are untouched. At temperature 0, outputs are identical (24/24) to native MTP at the same k. Both differ from no-MTP decoding on some prompts: verifying k+1 tokens per forward changes the kernel numerics, and that happens with native MTP too.

Byte ledger per draft step at batch 1:

	native drafter	this recipe
draft lm_head	1,271 MB (BF16, 248,320 rows)	92 MB (NVFP4, 64,064 rows)
MTP BF16 linears	136 MB	38 MB
MTP routed experts (10 active), router, HC, indexer	~70 MB	~70 MB
total per draft step	~1.48 GB	~0.20 GB (7×)
target decode step (for reference)	~7.0 GB	~7.0 GB

Measured on one B200 (single stream, 24 prompts × 1,024 tokens, temperature 0, effort=low)

config	k	tok/s	τ	vs no MTP
no MTP	0	92.6	—	1.00×
native MTP	3	241.3	3.39	2.61×
native MTP	5	278.6	4.40	3.01×
native MTP	7	300.1	5.14	3.24×
this recipe	3	248.4	3.31	2.68×
this recipe	5	282.0	4.30	3.05×
this recipe	7	308.9	4.93	3.34×
this recipe	9	297.4	5.31	3.21×

B200. A draft step costs little next to the target step, so the recipe adds only +2.9 % at the best k (the 27B recipe also gained about 4 % on B200).

DGX Spark (estimated, not measured). The gain is meant for bandwidth-bound machines. On a DGX Spark (~260 GB/s) the native drafter reads about 21 % of a target step per drafted token; this recipe reads about 3 %. Benchmark it on your own machine and workload.

Choosing k:

• Temperature 0, single stream: k=7.
• Sampling (temperature 1.0) or high concurrency: acceptance drops (per-position about 0.58 / 0.36 / 0.23), so use k = 2–4.

Use with vllm serve

pip install -e mtp_draft/plugin            # registers a vllm.general_plugins entry point
export Q38_DRAFT_HOME=$PWD/mtp_draft Q38_DRAFT_VOCAB=$PWD/mtp_draft/artifacts/draft_vocab_64k.pt \
       Q38_DRAFT_HEAD_Q=marlin Q38_DRAFT_MTP_Q=marlin
vllm serve <this repo> ... \
  --speculative-config '{"method":"mtp","num_speculative_tokens":7,"use_local_argmax_reduction":true}' \
  --block-size 48

Notes:

• The plugin is inactive unless Q38_DRAFT_VOCAB is set.
• use_local_argmax_reduction requires greedy drafting, so don't combine it with draft_sample_method=probabilistic.
• --block-size must be divisible by the QSA ring capacity 4·ceil((4+k)/4); 48 works for every k ≤ 12.
• The patch hooks vLLM 0.31 Model Runner V2 (MTPSpeculator.load_draft_model) and was validated only with stock vLLM 0.31.0 on B200. The community containers (b12x) have their own MTP options (VLLM_MTP_NVFP4_LM_HEAD, …); check that their speculator path matches before combining.

Files

config.json, model.safetensors.index.json, tensors/*.safetensors   checkpoint (num_experts=384)
mtp_draft/                 cheap MTP drafter: q38_draft.py + nvfp4_ops.py (patch), plugin/ (vLLM entry point),
                           artifacts/draft_vocab_64k.pt (draft vocabulary)

Limitations

• Knowledge-heavy multiple choice drops by 4.5–8.6 points (see the table). Prefer the 512-expert original for those workloads.
• Like the base model, it can produce inaccurate or unsafe content. Evaluate it for your use case.

License: NVIDIA Open Model Agreement (inherited); the base model is under the Qwen Community License.