Israeli-AI/Qwen3.8-Flash-Next-NVFP4-FP4-Local-QAD-VOLTA

Qwen3.8-Flash-Next-NVFP4-QAD-VOLTA

ITAI have a treat for V100 comunity! Enjoy!

What this is: NVIDIA's best-quality 4-bit version of Qwen3.8-Flash-Next (nvidia/Qwen3.8-Flash-Next-FP4-Local), repacked so it runs on old V100 GPUs. NVIDIA's original only runs on their newest (Blackwell) cards.

What it's meant for: 4× NVIDIA V100 32GB (NVLink, one model split across all four) serving the full 262,144-token context window, with MTP speculative decoding, tool calling, thinking and vision. It also needs about 64 GB of system RAM. A 27 GB word table (the "n-gram"/PLE table) lives in RAM, not on the GPUs. Our box uses about 50 GB of RAM in total while serving.

Size: 105 GiB (113 GB) download.

Not abliterated. These are NVIDIA's original weights, with no refusal edits.

What you need to do (3 steps)

1. Install 1Cat-vLLM 1.5.0. This is a vLLM fork that supports V100s. Get the wheel from the v1.5.0 release (Python 3.12). Do not use 1.5.1 yet, because the patch below doesn't fit it.
2. Run our one-file patch. Stock 1Cat can only read the word table in a bigger 8-bit form. Our patch teaches it to read NVIDIA's original 4-bit table, which saves about 21 GB of RAM and loses nothing. It edits one Python file inside the installed vLLM. It checks that the file is the exact expected version first, keeps a backup next to it (.bak-pre-ple4), and is safe to run twice.

   huggingface-cli download Israeli-AI/Qwen3.8-Flash-Next-NVFP4-QAD-VOLTA --local-dir qwen38-fn-qad
   python qwen38-fn-qad/conversion/ple4/patch_ple_nvfp4.py \
     $VENV/lib/python3.12/site-packages/vllm/models/qwen4_exp/nvidia/ple_layer.py
   

   If you ever reinstall or upgrade vLLM, run the patch again.
3. Start the server with the command in the Serving section below.

Technical background

The NVIDIA release uses a compressed NVFP4 scale format (CSF), MXFP8 attention and an NVFP4 n-gram (PLE) table. None of these load on sm_70 or on stock vLLM. This repo stores the same weights in the layout of the RadixArk NVFP4 checkpoint that the 1Cat SM70 lane already serves. Routed-expert FP4 values and scales are copied losslessly. The PLE table is NVIDIA's original NVFP4, bit-exact and row-packed: 80 bytes of E2M1 nibbles + 10 E4M3 block scales per row, plus one FP32 global scale. Config ple_embedding_dtype: "nvfp4_rowpacked" turns it on.

The patch adds a Triton kernel that gathers those rows straight from pinned host memory. On 4096 random rows its output is bit-identical to a CPU reference (conversion/ple4/test_kernel.py). Other checkpoints load exactly as before. It requires VLLM_PLE_CPU_OFFLOAD=0 (pinned-host/UVA path). Pinned PLE is 6.7 GiB per GPU worker (26.8 GiB total), versus 11.9 / 47.7 GiB for the 8-bit form stock 1Cat expects.

Comparison

Hardware: 4× V100-SXM2-32GB, NVLink, TP4, FP16, MTP4 speculative decoding, 262K context, max-num-seqs 8, 1Cat-vLLM 1.5.0rc1 (its PLE code is byte-identical to the 1.5.0 release). GSM8K is the first 150 test questions with temperature 0, thinking off, and an exact numeric answer match. The tool-call test is 12 single-turn get_weather requests. Decode speed is the single-stream median of 6 runs (1024-token code answer, temperature 0).

checkpoint	GSM8K-150	tool calls	decode tok/s
RadixArk/Qwen3.8-Flash-Next-NVFP4 (PTQ, stock)	146/150	12/12	92.8
this repo (QAD)	147/150	12/12	96.4
ref: QAD + abliteration, FP8 PLE	148/150	12/12	93.5
ref: QAD + abliteration, NVFP4 PLE	146/150	12/12	93.1
ref: RadixArk PTQ + abliteration (dealignai)	145/150	12/12	~95

The rows in italics are local abliterated builds, listed only for reference. They are not published here. With thinking on, tool calls were 24/24 for both QAD builds. Vision (OCR on a test image) was verified on both abliterated QAD builds. Their vision weights are identical to this repo's. A ±2 difference on 150 questions is within noise. On this quick check the QAD conversion is on par with the PTQ checkpoint at the same speed; GSM8K-150 is too easy to separate them.

KV cache: ≈317K tokens, which covers the full 262K window. The PLE table lives in host RAM, not VRAM.

Tensor-level fidelity (conversion/check_pairs.log)

Cosine similarity of each conversion product against NVIDIA's dequantized source was ≈1.0 by construction (lossless or exact dequant). Against the stock RadixArk PTQ weights, it shows how far QAD moved the weights:

tensor class	conversion	cos vs RadixArk PTQ
routed experts (NVFP4)	nibbles copied, CSF scales decoded losslessly to E4M3	0.996–0.998
attention / linear-attn / shared experts (MXFP8)	exact dequant → BF16	0.9996
vision MLP + merger (NVFP4 W4A16)	dequant → BF16	0.995
MTP experts (NVFP4)	dequant → fused BF16 gate_up_proj (gate‖up, verified by swap test)	0.9955
embeddings, lm_head, MTP attention, PLE projections	copied	1.0000
PLE n-gram table	NVIDIA original NVFP4, row-packed (bit-exact)	0.995

validate_out.py: headers matched the RadixArk layout across all 206 shards with 0 mismatches. There were 0 non-finite values.

Serving (4× V100 32GB, tested)

export VLLM_PLE_CPU_OFFLOAD=0 VLLM_USE_V2_MODEL_RUNNER=1 VLLM_WORKER_MULTIPROC_METHOD=spawn
export CUDA_DEVICE_MAX_CONNECTIONS=32 VLLM_SM70_QWEN4_EXP_ONLINE_QPN8=0
export VLLM_SM70_QWEN4_EXP_VISION=1          # optional: keep the vision tower
vllm serve Israeli-AI/Qwen3.8-Flash-Next-NVFP4-QAD-VOLTA \
  --tensor-parallel-size 4 --dtype float16 \
  --mamba-cache-mode align --max-model-len 262144 --gpu-memory-utilization 0.90 \
  --enable-prefix-caching --max-num-seqs 8 --disable-custom-all-reduce \
  --limit-mm-per-prompt '{"image": 16, "video": 0}' \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
  --speculative-config '{"method":"qwen4_exp_mtp","num_speculative_tokens":4}'

Use --language-model-only instead of the vision flags for text-only serving. Thinking is on by default in the chat template. Send "chat_template_kwargs": {"enable_thinking": false} for direct answers. With MTP4 we see ~74 % draft acceptance. At concurrency 8, throughput is ~150 tok/s aggregate.

How it was made

conversion/convert.py takes the target schema from the RadixArk checkpoint headers (names, dtypes, shapes, file placement) and produces every tensor from the NVIDIA source:

• CSF decode (codec 0, 16-row slabs): per-row base byte + packed 4-bit offsets, then u32 exception records (24-bit row-major position, 8-bit value) override. The output is raw E4M3 block-scale bytes, identical to what ModelOpt would have stored uncompressed.
• MXFP8: e4m3 × 2^(e8m0 − 127), exact in BF16.
• NVFP4 → BF16 where the SM70 lane expects BF16: E2M1 × E4M3 block scale × FP32 global.
• conversion/ple4/build_ple4.py writes the PLE table in NVIDIA's original NVFP4, row-packed (the FP8 path in convert.py is only for unpatched 1Cat).
• Chat template, tokenizer and generation config are NVIDIA's, unchanged.

License

Use is governed by the NVIDIA Open Model Agreement. The Qwen3.8-Flash-Next base model is governed by the Qwen Community License 1.0 (LICENSE). This is a format conversion of NVIDIA's checkpoint. All credit for the QAD weights goes to NVIDIA and the Qwen team.

Converted and served by the Israeli-AI homelab stack. Same lineage as Qwen3.8-27B-MTP-W4A16-VOLTA-Ampere.