Startup issues with vllm-openai:qwen38-flash-next
Has anyone been able to get this to startup ok with the vllm-openai:qwen38-flash-next docker image?
ValueError: There is no module or parameter named 'ngram_embedding.weight_scale' in Qwen3_8FlashNextNGramEmbedding. The available parameters belonging to ngram_embedding (VocabParallelEmbedding) are: {'ngram_embedding.weight'}
--model RadixArk/Qwen3.8-Flash-Next-NVFP4
--max-num-seqs 12
--tensor-parallel-size 2
--gpu-memory-utilization 0.96
--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--safetensors-load-strategy lazy
--enable-prefix-caching
--max-model-len auto
--enable-chunked-prefill
--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'
I don't have this error. I'm using VLLM_PLE_CPU_OFFLOAD=1 env var to offload the ngram to the cpu (cf https://recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next )
however I get an OOM when the weights are loaded into a single RTX 6000 Pro
I don't have this error. I'm using VLLM_PLE_CPU_OFFLOAD=1 env var to offload the ngram to the cpu (cf https://recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next )
however I get an OOM when the weights are loaded into a single RTX 6000 Pro
I have not tried this one - But I got the Inferact version working on a single RTX Pro 6000. It is very fast and runs well. Switching over to this model from DSv4-Flash for now.
Qwen3.8-Flash-Next NVFP4 on single RTX PRO 6000 (TP=1): fixed the weight_scale + pidfd_getfd crashes
Got RadixArk/Qwen3.8-Flash-Next-NVFP4 running on a single RTX PRO 6000 Blackwell (96GB, TP=1, MTP enabled, 262K ctx) after hitting three different startup crashes. Sharing the fixes:
- ValueError: There is no module or parameter named 'ngram_embedding.weight_scale'
Root cause: the NVFP4 checkpoint stores the PLE shards as global-scale FP8 (ngram_embedding.shard_N.weight F8_E4M3 + ngram_embedding.weight_scale) while listing .ple. under ignore in the quant config. The stock _get_ple_embedding_quant_method() gate in ple_layer.py requires an Fp8Config and returns None for the ModelOpt/NVFP4 config — so the weight loader never gets the FP8 embedding method and dies on weight_scale. Fix: small patch that returns Qwen3_8FlashNextPLEFp8EmbeddingMethod() when VLLM_PLE_FP8_CHECKPOINT=1 is set (stock behaviour otherwise). Self-contained Dockerfile, no extra files:
FROM vllm/vllm-openai:qwen38-flash-next
RUN python3 - <<'EOF'
p = "/usr/local/lib/python3.12/dist-packages/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py"
s = open(p).read()
old_import = "import math\n"
new_import = "import os\nimport math\n"
assert s.count(old_import) == 1
s = s.replace(old_import, new_import, 1)
old_gate = (
" if not isinstance(quant_config, Fp8Config):\n"
" return None\n"
)
new_gate = (
" if not isinstance(quant_config, Fp8Config):\n"
' if os.environ.get("VLLM_PLE_FP8_CHECKPOINT", "0").lower() in ("1", "true"):\n'
" return Qwen3_8FlashNextPLEFp8EmbeddingMethod()\n"
" return None\n"
)
assert s.count(old_gate) == 1
s = s.replace(old_gate, new_gate, 1)
open(p, "w").write(s)
print("ple_layer.py patched")
EOF
- RuntimeError: pidfd_getfd: Operation not permitted
Only happens with VLLM_PLE_CPU_OFFLOAD=1 (which you need on 96GB to keep the 51B ngram table out of VRAM). The PLE offload worker is a separate process that receives CUDA tensors from the worker via torch IPC, which uses pidfd_getfd to dup the fd across processes — that syscall requires CAP_SYS_PTRACE. Fix: --cap-add=SYS_PTRACE for docker, or in k8s:
securityContext:
capabilities:
add:- SYS_PTRACE
- RuntimeError: Insufficient space in /dev/shm: 160 MiB required, 64 MiB free
k8s default /dev/shm is 64MiB. Fix: --shm-size=1g for docker, or an emptyDir: {medium: Memory} volume mounted at /dev/shm in k8s.
Working run command (docker):
VLLM_PLE_CPU_OFFLOAD=1 VLLM_PLE_FP8_CHECKPOINT=1
docker run --gpus all --shm-size=1g --cap-add=SYS_PTRACE
-v /path/to/model:/model-data -p 8000:8000
vllm-qwen38-flash-next-nvfp4
--model /model-data --served-model-name Qwen3.8-Flash-Next
--max-model-len 262144 --gpu-memory-utilization 0.96
--max-num-seqs 2 --max-num-batched-tokens 8192
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
--enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3
Model loads at ~79GB, PLE table offloaded to host RAM, MTP drafting works. Runs, though I'd benchmark before comparing tg speeds.
FYI I finally used this SGLANG fork : https://old.reddit.com/r/BlackwellPerformance/comments/1w04xb7/qwen38_flashnext_on_1x_rtx_pro_6000_171_ts_c1_428/