Qwen3.8-Flash-Next-W4A16-PLE8 (2× AMD Strix Halo Edition)

This checkpoint is a re-quantization of aixiaoma/Qwen3.8-Flash-Next-W4A16: the W4A16 (INT4, group-128, symmetric) quantization in compressed-tensors pack-quantized format is inherited unchanged from that release. We added the one step it was missing — converting the n-gram (PLE) embedding table from BF16 to W8A16 (group-32), "PLE8" — and tuned the whole setup for 2× AMD Ryzen AI Max+ 395 "Strix Halo" (gfx1151) over Thunderbolt in a two-node vLLM cluster. No new quantization method is claimed here.

This build is not a drop-in for NVIDIA GPUs. It is tuned for the AMD Strix Halo APU on gfx1151 — a platform where the official FP8 checkpoint and the standard ROCm vLLM stack both fail unless the ROCm/RCCL and transformers dependencies are correctly pinned. See "Deployment for 2× AMD Strix Halo" below.

This checkpoint runs the model on AMD Strix Halo (gfx1151) in a 2-node Thunderbolt cluster. Consumer NVIDIA Ampere (4×RTX 3090 — 8×RTX 3090) also works, but requires different flags — see the W4A16 release aixiaoma/Qwen3.8-Flash-Next-W4A16 for the NVIDIA build.

• 335 GB (BF16) → 168 GB (W4A16) → 122.75 GB with PLE8 on disk — the n-gram/PLE tables are served from W8A16 instead of BF16 and no longer require host-RAM offload (critical on Strix Halo where host and device share the same 128 GiB UMA).
• Runs on 2× Strix Halo over Thunderbolt in TP2 + EP, native 262k context, MTP speculative decoding (nst=3; A/B 05.09.: answer quality identical, 1.9–3.2× faster than MTP off).
• Quality on our greedy eval suite matches the base-family reference (see Benchmarks).
• MTP speculative-decoding head preserved in BF16 — speculative decoding works; on ROCm this requires a 2-line rocm_types allow-list patch (see Deployment notes) so num_speculative_tokens ≥ 2 works at all.

What was quantized (and what deliberately was not)

Only GEMM-heavy weights are quantized to INT4 group-128 symmetric:

Component	Precision	Rationale
Routed experts (512/layer, fused gate_up_proj/down_proj, unpacked to per-expert tensors)	INT4 g128	120B of the 125B backbone; the entire point of the exercise
Attention q/k/v/o_proj (QSA layers)	INT4 g128	Standard, well-conditioned GEMMs
N-gram embedding (*ngram_embedding*)	W8A16 g32 (PLE8)	51B lookup table; W8A16 group-32 via re:.*ngram_embedding.*, saves 44.7 GiB vs BF16; critical on UMA (no host-RAM offload budget)
Shared expert (all 48 layers)	BF16	Its intermediate dim 640 is TP-sharded to 640/TP columns, which no Marlin group size divides at TP=8 (see "The 640/128 problem" below). ~470 MB total — not worth the risk
MTP module (mtp.*)	BF16	Speculative decoding head; quantizing draft heads measurably hurts acceptance; losing it disables MTP entirely
MoE router (*.gate)	BF16	A wrong expert choice costs more than the bytes saved; the authors themselves single out the router as fragile (they exclude it from Muon in training)
QSA indexer (*indexer*)	BF16	Selects which context blocks attention sees; quantization noise here degrades long-context retrieval
GDN / linear attention (*.linear_attn.*)	BF16	Recurrent state dynamics; follows the reference practice of the RedHatAI Qwen3-Next W4A16 quant
Embeddings, lm_head, gated-residual (hyper_connection) branches, vision encoder	BF16	Small and sensitive

Method: RTN (round-to-nearest), symmetric, per-group scales along the input axis, no calibration data. AWQ was attempted first but the 335 GB BF16 model cannot be linearized by llm-compressor within 251 GB host RAM (meta-tensor offload breaks expert linearization). RTN quality turned out to match the family reference on our eval suite, so AWQ was deferred. (Inherited from the upstream aixiaoma/Qwen3.8-Flash-Next-W4A16 decision — not a fresh choice by this re-target.)

The 640/128 problem — why Expert Parallelism is mandatory

The experts of this architecture have moe_intermediate_size = 640. With group-128 quantization (ours, and equally the official FP8 checkpoint's block-128), tensor parallelism shards the expert down_proj input dimension into 640/TP columns: 320 at TP=2, 160 at TP=4, 80 at TP=8 — none divisible by 128. The upstream vLLM recipe states this for the FP8 checkpoint verbatim: "plain TP8 is incompatible with its 128-wide quantization blocks" and prescribes TEP8 (TP8 + expert parallelism).

The same applies to this checkpoint: always launch with --enable-expert-parallel. With EP, experts are distributed whole (no intra-expert sharding), and the group math is exact. The shared expert is not covered by EP (it is always TP-sharded), which is why it ships in BF16 here.

Note for 2× Strix Halo (TP2): EP is doubly important — at TP2 you have only 2-way tensor parallelism and no intra-node headroom. Without EP the expert GEMMs become 320 % 128 ≠ 0 and vLLM refuses to load (or silently corrupts). Use --enable-expert-parallel --enable-ep-weight-filter so each rank does not re-read the full 122 GB checkpoint.

Deployment for 2× AMD Strix Halo

The model is not in any generic ROCm image (standard vllm/vllm-openai-rocm:latest does not know qwen3_8_flash_next). You need a dedicated ROCm image with all of:

1. Dedicated ROCm image — tested rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0 (or the newer vllm/vllm-openai-rocm:qwen38-flash-next vendor build if available). The upstream recipe for this model asks for vLLM ≥ 0.29.0 — that does not apply here: this build is a PR16 fork (Inferact/wef HEAD b0731de1c) whose arch registry already carries the model. The registry must contain Qwen3_8FlashNextForCausalLM, Qwen3_8FlashNextForConditionalGeneration, and Qwen3_8FlashNextMTP.
2. transformers ≥ 5.16.0.dev0 from github.com/huggingface/transformers — older versions do not know model_type: qwen3_8_flash_next → config parse fails with model_type qwen4_exp.
3. ray[default] 2.58.0 installed — TP2 = 2 nodes = Ray cluster. Without it, no multi-node.
4. RCCL with gfx1151 target — non-negotiable. RCCL 2.27.7 does not have a gfx1151 build target (empty GPU_TARGETS → gfx906 fallback → SIGSEGV at 2 ranks). Use RCCL rocm-7.2.3 built for gfx1151 (a 2-line GPU_TARGETS patch), fed in via LD_LIBRARY_PATH=/opt/rccl-gfx1151/lib (reversible). Do not overwrite the image's own RCCL library — this keeps the rollback trivial.
5. The PLE fix (quant_config pass-through, 2 lines in ple_layer.py) and the Mamba pointer fix (_reinterpret_u64_as_i64, upstream #48109) must be in the image. Both are already committed in PR16 and the vllm-rocm-qwen38:ed tag; do not revert.

The config.json shipped here is already patched: architectures: ["Qwen3_8FlashNextForConditionalGeneration"], model_type: "qwen3_8_flash_next". Do not rename the model directory or use --served-model-name; keep /models as the model key — this is the key that the local Telegram agent and opencode client expect.

Container invocation (Docker, ROCm host)

docker run -d --entrypoint bash \
  --net=host --ipc=host --shm-size=64g \
  --device=/dev/kfd --device=/dev/dri --group-add 44 --group-add 992 \
  --security-opt seccomp=unconfined --security-opt label=disable \
  -v /path/to/Qwen38-W4A16-PLE8:/models:ro \
  vllm-rocm-qwen38:ed -c "sleep infinity"

--entrypoint bash is mandatory (the image entrypoint collides with bash -lc).

Startup sequence (two nodes)

Start the containers on both nodes, then bring up Ray inside them. All environment variables must be present in the Raylet and the worker actors (see the env table below) — a variable that only reaches one side is the classic "cluster looks fine, server hangs" failure.

# ── on BOTH nodes ─────────────────────────────────────────────────────────
export GLOO_SOCKET_IFNAME=thunderbolt0
export NCCL_SOCKET_IFNAME=thunderbolt0
export RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES=1
export LD_LIBRARY_PATH=/opt/rccl-gfx1151/lib:${LD_LIBRARY_PATH:-}
export RAY_num_heartbeats_timeout=600 RAY_raylet_heartbeat_period_milliseconds=1000
export RAY_health_check_failure_threshold=600
export HSA_ENABLE_INTERRUPT=1 HSA_TOOLS_DISABLE_REGISTER=1

# ── node 1 (head) ─────────────────────────────────────────────────────────
ray start --head --port=6384 --node-ip-address=<HEAD_IP> --include-dashboard=false

# ── node 2 (worker) ───────────────────────────────────────────────────────
ray start --address=<HEAD_IP>:6384 --node-ip-address=<WORKER_IP>

# ── verify BEFORE starting the server ────────────────────────────────────
python -c "import ray; ray.init(address='auto'); print(ray.nodes())"
# must show 2 nodes, each with Resources: GPU=1.0

<HEAD_IP>/<WORKER_IP> are the Thunderbolt addresses (thunderbolt0). Verify before every server start: a zombie worker Raylet can leave "2 nodes alive" while the GPU count has silently dropped to 1.0 — TP2 then hangs forever on the first request. Also clean up leftover ray::RayWorker processes after a crashed engine (pkill -9 -f "[r]ay::RayWorker") — they hold 60+ GiB of GTT each and the next start will OOM.

Two-node serve (canonical)

vllm serve /models \
  --distributed-executor-backend ray \
  --tensor-parallel-size 2 \
  --enable-expert-parallel --enable-ep-weight-filter \
  --max-model-len 262144 \
  --max-num-batched-tokens 1024 \
  --kv-cache-memory-bytes 8589934592 \
  --gpu-memory-utilization 0.75 \
  --enforce-eager \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --tool-call-parser qwen3_xml \
  --reasoning-config '{"reasoning_parser":"qwen3","reasoning_start_str":"<think>","reasoning_end_str":"\n\nBudget reached — pausing here, briefly summarizing the state, and giving the best available answer.</think>"}' \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs '{"reasoning_effort": "medium", "preserve_thinking": true}' \
  --port 8011 --host 0.0.0.0

Per-flag rationale:

• TP2 + EP, never PP2 — PP2 is locked for this architecture (N-gram PLE does not survive a pipeline split). EP is mandatory: 640 % 128 = 0 only with EP; without EP the shared-expert shard math is 320 % 128 = 2.5 and the loader rejects. --enable-ep-weight-filter so each worker reads only its shard (saves ~60 GB per node at TP2).
• --enforce-eager — CUDA graphs are not affordable at 262k context (~27 GiB of graph state).
• --max-num-batched-tokens 1024 — GPU-fault workaround. 2048 causes EngineCore to die in the last prefill chunk at ~90k tokens.
• --kv-cache-memory-bytes 8589934592 (8 GiB) — pool sized for ~560,900 tokens. Flag is bytes in this build, not "8G". More KV buys concurrency, not speed: when the pool is exhausted vLLM preempts (drops and later recomputes) — slow, but not an error.
• --speculative-config … "num_speculative_tokens": 3 — requires the ROCm rocm_types allow-list patch (vllm#53983) in llm_base_proposer.py; without it nst ≥ 2 fails with ValueError: Unsupported attention metadata type → EngineDeadError → HTTP 500. A/B (05.09., 5 fixed greedy prompts): answer quality identical with n3 vs. MTP off, 1.9–3.2× faster. Caveat: this server is run-to-run non-deterministic even at temperature 0 (3/5 prompts diverged n3-vs-n3), so byte-exact losslessness is unprovable on this stack — evaluate quality, not bytes.
• --default-chat-template-kwargs '{"reasoning_effort": "medium", "preserve_thinking": true}' — two dials. reasoning_effort: medium replaces the template default xhigh, which on hard prompts reasons until max_tokens is exhausted. preserve_thinking controls whether past assistant turns re-render their reasoning into every new prompt (chat_template.jinja):
  • false keeps only the current tool-loop's reasoning — a 6,000-char reasoning block drops the prompt from 831 to 82 tokens, answers unchanged. Safe default for long agent loops.
  • true (shipped default since 05.09., operator decision) re-sends prior reasoning so the model stays consistent with its own earlier deliberation; measured cost ≈ 0.33 prompt tokens per reasoning character per follow-up turn. Watch long sessions for context bloat — rollback is false + restart.
  • Fairness note: "the model thinks too much" turned out to be a framing artifact — cloud agents reason just as long, it is simply not visible in their stream. Judge reasoning by answer quality, not by visible token count.
• --tool-call-parser qwen3_xml — hermes and other generic parsers do not recognize the Qwen XML tool call and silently fall through to finish_reason: stop with no tool_calls. --enable-auto-tool-choice is required so tool_choice: "auto" resolves.
• --reasoning-config (supersedes bare --reasoning-parser qwen3) — parser behavior is unchanged (thinking tokens land in reasoning_content; without any parser they land in content and surface as TTFT 0.00 s / 0 tok/s). The addition is reasoning_end_str: when the thinking-token budget is exhausted, the sampler no longer forces a bare </think> but a wrap-up sentence + </think> — the model receives its own "summarize now" signal in-stream and answers cleanly instead of leaking reasoning into content. Natural ends are still detected per token, so the longer string only affects the forced end. The wrap-up text is free-form — translate it freely (our production runs a German variant). Known risk: forcing mid-tool-call that began inside the think block can corrupt JSON (vllm#44676/#44690, PR open) — rare with large budgets.
• Sampling defaults via the shipped generation_config.json — temperature: 0.6, top_k 20, top_p 0.95. Deliberate deviation from the model card (1.0 for thinking mode): the 4-bit experts at 1.0 produce word-form errors ("Schaufte" instead of "Schaufel"); 0.6 is stable in daily operation since 05.09. with noticeably more consistent reasoning. Applies to every client that does not send its own temperature; explicit client values override (standard behavior).
• thunderbolt0 (not eno1, not virbr0) for GLOO_SOCKET_IFNAME and NCCL_SOCKET_IFNAME in the env of both head Raylet and worker actors (see table below).

Required environment variables (in both the head's Raylet and every worker actor)

Variable	Value	Why
VLLM_HOST_IP	<head-ip>	Thunderbolt address of the head node (thunderbolt0).
GLOO_SOCKET_IFNAME	thunderbolt0	Gloo falls back to 127.0.1.1 → connectFullMesh fails.
NCCL_SOCKET_IFNAME	thunderbolt0	RCCL falls back to virbr0 (identical on both hosts).
RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES	1	Otherwise Ray masks the APU (saw only 1.0 GPU).
RAY_num_heartbeats_timeout	600	Long 94k prefill (181 s) makes Ray think the node is dead.
RAY_raylet_heartbeat_period_milliseconds	1000	Ditto.
RAY_health_check_failure_threshold	600	Ditto.
HSA_TOOLS_DISABLE_REGISTER	1	ROCr hot-loop (85k → 112 ioctl/s) → idle burn.

Verify before starting: ray.nodes() must show 2 nodes / 2.0 GPU. "2 nodes alive" alone is not sufficient — a zombie M2 Raylet silently drops to 1 GPU and TP2 hangs on the first request.

What is different on AMD vs NVIDIA

Concern	NVIDIA (Ampere)	AMD Strix Halo
FP8	not supported on SM86 → we use W4A16	FP8 not supported on gfx1151 either → W4A16 is the only path
Context length	native 262k works; YaRN ×4 to 1M optional	native 262k; YaRN ×2 auf 524k getestet und zurückgestellt (kein Durchsatzgewinn) — ausgeliefert ist 262k
MTP nst ≥ 2	works out of the box	blocked by ROCm rocm_types allow-list; 2-line patch in llm_base_proposer.py (vllm#53983) required
PLE table	host-RAM offload (VLLM_PLE_CPU_OFFLOAD=1)	do not — UMA shares host + device memory; W8A16 PLE8 (this build) is the correct path
CUDA graphs	FULL_DECODE_ONLY at mode: 0	--enforce-eager (inductor hangs; cudagraph_mode flag is unsupported here)
Tool-call parser	qwen3_coder works	qwen3_xml is mandatory (generic parsers miss the Qwen XML tool call)
Reasoning parser	optional in practice	--reasoning-config (qwen3 + reasoning_end_str wrap-up) mandatory — without the parser reasoning_content lands in the wrong field → TTFT 0.00 s / 0 tok/s
Sampling defaults	card values (1.0 thinking)	temperature: 0.6 shipped via generation_config.json — 4-bit quant compensation; explicit client values override
Host interface	any single homed NIC	must pin thunderbolt0 — dual 127.0.1.1 / 192.168.122.1 on this multi-homed host
--gpus flag	required	not used — AMD ROCm uses --device=/dev/kfd + --ipc=host

Benchmarks (2× Strix Halo, 262k context, greedy, temperature 0)

Metric	Value	Source
Ray cluster	2 nodes / 2.0 GPU	ray status
Weights per node	~60 GiB (62.98 GiB with MTP)	gpu_model_runner.py
Startup	Application startup complete.	vLLM log
KV cache	~560,900 tokens (8 GiB) — ~2.1× full-length (262k) concurrency	kv_cache_utils
MTP (nst=3)	quality identical, 1.9–3.2× vs. MTP off (A/B, 5 greedy prompts); n1-era acceptance ~83 % (2.6 tok/step)	A/B 05.09. + /metrics
Prefill (cold, 6k prompt)	~530 tok/s (TTFT ~11.5 s)	bench_vllm.py
Prefill (cache warm)	~1000 tok/s	measured
Generation (real operation, agent/chat workload)	≈15–18 tok/s	measured
Idle temp	M2 ≤ ~31 °C, no spinning thread	measured

Use bench_vllm.py for benchmarking (streaming, temperature 0, bump SEED per row — otherwise prefix cache produces fantasy rates). usage.prompt_tokens counts all tokens incl. cache — report cold/warm separately.

Decision log (engineering, 03.09.2026)

1. W4A16 (not FP8) — FP8 has no kernel on gfx1151 (refuses to load); the official FP8 checkpoint's block-128 also assumes EP. W4A16 + EP is the only combination that boots.
2. RTN (no AWQ) — inherited from the upstream W4A16 release (the upstream author's choice; host RAM (335 GB) exceeds the host's 251 GB ceiling for linearization — the host is 256 GB nominal, 251 GB usable after kernel/agent reservations; validated post-hoc by quality parity).
3. PLE8 (not BF16) — BF16 PLE forces host-RAM offload (which competes with the APU's UMA). W8A16 group-32 fits on the APU and is 44.7 GiB lighter.
4. EP mandatory — at TP2 it is doubly so (see above).
5. MTP with nst=3 — works only with the rocm_types allow-list patch. A/B vs. MTP off (05.09., 5 fixed greedy prompts): answer quality identical, 1.9–3.2× faster; the server is run-to-run non-deterministic at temperature 0 (3/5 prompts diverged even n3-vs-n3), so byte-exactness is unprovable on this stack. parallel_drafting is not usable (arch has 1 draft layer). Realistic end-to-end generation: ≈15–18 tok/s — synthetic short-prompt runs can exceed that; do not quote them as production speed.
6. qwen3_xml + qwen3 parsers — hermes does not recognize the Qwen XML tool call and falls through to finish_reason: stop.
7. thunderbolt0 for *_SOCKET_IFNAME — multi-homed host with duplicate 127.0.1.1 / 192.168.122.1 on both ends will silently hang sessions otherwise.
8. ray[default] 2.58.0 — without ray, no multi-node. The 6k-prefill timeout is not a RAM problem: it is solved by RAY_num_heartbeats_timeout=600 + companion flags (see §8 of the internal runbook), not by host-RAM headroom.
9. Context capped at native 262k, no YaRN in shipped config — MRCR (multi-needle) scores collapse past 256k (93 → 30–40 at 512k even for full attention). Base supports YaRN ×4 to 1M if you accept the trade. Measured 2026-09-06: we ran YaRN ×2 → 524k live for a day and reverted. A/B at temperature=0 (n=3 × 6 tasks, short prompts): correctness identical — but operator sessions showed circle-reasoning, stumbling over token-like words (spelling out "vLLM"), and repeating already-settled points; output volume grew ~+30 % reasoning / +11 % answer tokens. 2× context minus +41 % token spend left a net win of ~59 % — not worth it. Stayed at native 262k.

What is new vs the upstream W4A16 release

• The upstream aixiaoma/Qwen3.8-Flash-Next-W4A16 ships a generic NVIDIA-targeted quant with mode: 0 + cudagraph_mode: FULL_DECODE_ONLY. On Strix Halo (gfx1151) the cudagraph flag is unsupported, the FP8 blocksize assumption is wrong, and the qwen4_exp arch name in config.json is unknown to transformers ≥ 5.16.0.dev0 without the patch.
• This fork re-targets the quant to 2× AMD Ryzen AI Max+ 395 over Thunderbolt in TP2 + EP, ships the patched config.json (qwen3_8_flash_next, Qwen3_8FlashNextForConditionalGeneration), and documents the two ROCm-specific code patches that are required (rocm_types MTP allow-list + _reinterpret_u64_as_i64 Mamba pointer fix).
• All NVIDIA-specific flags (--gpus all, PYTORCH_CUDA_ALLOC_CONF, CUDA graph mode) replaced with --ipc=host + Ray + thunderbolt0 interface pins; all FP8 / AWQ / qwen3_coder flags dropped.

Known limitations (do not "fix")

1. TP2-only — PP2 is blocked (N-gram PLE); TP2 + EP is the only configuration that runs.
2. No CUDA graphs at 262k context.
3. nst ≥ 2 only with the rocm_types allow-list patch — without it: ValueError: Unsupported attention metadata type → EngineDeadError → HTTP 500.
4. GPU fault on long prompts (~90k) in the last prefill chunk → chunk=1024 workaround. The "prefill halving" is a measurement artefact, not a kernel bug — real prefill is ~530 cold / ~1000 warm (see §16.8 of the internal runbook). Root cause open (suspected: prefill→decode transition / MTP draft).
5. AWQ group_size=128 without EP is structurally TP2-incompatible → EP is mandatory.
6. --reasoning-parser qwen3 → thinking tokens land in reasoning, not content. Surfacing them in content produces 0.00 s TTFT / 0 tok/s.

Tested on

• 2× AMD Ryzen AI Max+ 395 (124 GiB UMA each), gfx1151, Thunderbolt (thunderbolt0).
• vllm-rocm-qwen38:ed (M1 14c5b837fd17, M2 979a14376fc3).
• vLLM 0.27.0+ (PR16 basis, b0731de1c), transformers ≥ 5.16.0.dev0, ray[default] 2.58.0, RCCL rocm-7.2.3 (custom gfx1151 build), amd_iommu=off on the host kernel.

License

This is a derivative work of Qwen/Qwen3.8-Flash-Next and is distributed under the same Qwen Community License 1.0 (see LICENSE). Upstream terms apply: attribution required at >100M MAU / $20M monthly revenue; Model-as-a-Service and commercial AI work-assistant offerings require a separate license from Qwen.

Citation

If you use this checkpoint, cite the base model:

@techreport{qwen2026design,
  title = {On the Design of {Qwen3.8-Next} Architecture: Evaluation, Efficiency, and Training Stability},
  author = {{Qwen Team}},
  institution = {Alibaba Group},
  month = {August},
  year = {2026}
}

This AMD Strix Halo build by arnomatic, 2026-09-04 — server settings updated 2026-09-05 (--reasoning-config wrap-up end string, preserve_thinking: true shipped default, temperature: 0.6 via generation_config.json, MTP n3 A/B documented), derived from aixiaoma/Qwen3.8-Flash-Next-W4A16 (which is itself a derivative of Qwen/Qwen3.8-Flash-Next).