LibertAIDAI

/

GLM-5.3-Flash-NVFP4

like 55

Follow

LibertAI 45

tl;dr: Got GLM-5.3-Flash (320B/18B active MoE) running across 4 DGX Sparks using tensor parallelism over Ethernet (no NVLink). Decode throughput: 5.3–7.0 tok/s, TTFT ~200ms, with 131K max context. Required fixing 5 vLLM bugs along the way. Full reproduction steps below.

Hardware

Model & Config

Performance Results

Throughput (post-warmup, JIT compiled)

Memory

Quality Spot-Check

The multilingual degeneration is likely an NVFP4 quantization artifact at temperature=0. Math and code quality are solid.

Bugs Fixed (5 total)

Getting TP4 working on DGX Spark required fixing several vLLM issues:

Bug 1: FLASHINFER_CUTLASS MoE backend produces garbage on GB10

The default NvFp4 MoE backend selection picks FLASHINFER_CUTLASS, which silently produces garbage output on GB10 (sm_121). No error, no crash — just wrong answers.

Fix: Force moe_backend="marlin". The Marlin backend does FP4 weight-only decompression correctly on GB10, though with a performance penalty ("Your GPU does not have native support for FP4 computation").

Bug 2: external_launcher backend required for multi-node

vLLM's standard mp (multiprocessing) executor doesn't work across separate Docker containers on different hosts. The external_launcher backend reads RANK, LOCAL_RANK, WORLD_SIZE, MASTER_ADDR, MASTER_PORT from environment and uses torch.distributed for coordination.

Fix: Use distributed_executor_backend="external_launcher" with torchrun-compatible environment variables set per container.

Bug 3: vllm serve incompatible with external_launcher

The vllm serve CLI and AsyncLLM engine don't work with the external_launcher backend — they hang waiting for HTTP server initialization that never happens on non-zero ranks.

Fix: Use the synchronous LLM class directly in a Python script. All ranks must call llm.generate() with identical prompts for deterministic TP coordination.

Bug 4: Docker image has vllm CLI as entrypoint

The Docker image glm53:sm121-v8 has ENTRYPOINT ["vllm"], so docker run ... image script.py tries to parse the script as a vLLM CLI argument.

Fix: --entrypoint python3 in the docker run command.

Bug 5: SHM Broadcast MessageQueue deadlock (the tricky one)

vLLM's parallel_state.py creates a GroupCoordinator with use_message_queue_broadcaster=True for the TP group. This creates ZMQ XPUB/SUB sockets for fast intra-process communication. The socket binds to get_ip(), which inside Docker with --network host returns the WiFi IP (10.168.168.x) instead of the cluster subnet (192.168.88.x). Remote readers on other Sparks can't connect, causing a deadlock in wait_until_ready().

Fix: Monkey-patch GroupCoordinator.__init__ to force use_message_queue_broadcaster=False:

import vllm.distributed.parallel_state as ps
_orig = ps.GroupCoordinator.__init__
def _patched(self, *a, **kw):
    kw["use_message_queue_broadcaster"] = False
    _orig(self, *a, **kw)
ps.GroupCoordinator.__init__ = _patched

This falls back to torch.distributed.broadcast_object_list via CPU/Gloo, which correctly uses GLOO_SOCKET_IFNAME=enp1s0f0np0 (the cluster interface).

Full Reproduction

Prerequisites

• 4× DGX Spark on the same subnet
• Docker with --gpus all support
• vLLM Docker image with GLM-5.3-Flash support and sm_121 (GB10) CUDA kernels
• Model weights downloaded to one node, shared via NFS to others

Docker command (per node)

# Adjust RANK (0-3) and container name per node
sudo docker run -d \
  --name glm53-node${RANK} \
  --gpus all --network host --ipc host --cap-add SYS_PTRACE \
  --entrypoint python3 \
  -e VLLM_ENABLE_V1_MULTIPROCESSING=0 \
  -e RANK=${RANK} -e LOCAL_RANK=0 -e WORLD_SIZE=4 \
  -e MASTER_ADDR=192.168.88.11 -e MASTER_PORT=29500 \
  -e NCCL_SOCKET_IFNAME=enp1s0f0np0 -e NCCL_DEBUG=WARN \
  -e GLOO_SOCKET_IFNAME=enp1s0f0np0 \
  -v /var/tmp/glm-5.3-flash-nvfp4:/var/tmp/glm-5.3-flash-nvfp4:ro \
  -v /var/tmp/glm_test.py:/workspace/glm_test.py:ro \
  glm53:sm121-v8 /workspace/glm_test.py

Important: Do NOT use --runtime nvidia on DGX Spark — only --gpus all works.

Inference script (glm_test.py)

"""GLM-5.3-Flash TP4 — multi-node inference on DGX Spark."""
import os, time
RANK = int(os.environ.get("RANK", 0))
WORLD_SIZE = int(os.environ.get("WORLD_SIZE", 1))

# Disable ZMQ MessageQueue — deadlocks multi-node external_launcher
import vllm.distributed.parallel_state as ps
_orig = ps.GroupCoordinator.__init__
def _patched(self, *a, **kw):
    kw["use_message_queue_broadcaster"] = False
    _orig(self, *a, **kw)
ps.GroupCoordinator.__init__ = _patched

from vllm import LLM, SamplingParams

llm = LLM(
    model="/var/tmp/glm-5.3-flash-nvfp4",
    tensor_parallel_size=WORLD_SIZE,
    distributed_executor_backend="external_launcher",
    max_model_len=131072,
    gpu_memory_utilization=0.78,
    kv_cache_dtype="fp8",
    block_size=2304,
    trust_remote_code=True,
    enforce_eager=True,
    moe_backend="marlin",
)

# All ranks must call generate() with identical prompts
prompt = "What is 2+2? Answer in one word:"
outputs = llm.generate([prompt], SamplingParams(temperature=0, max_tokens=10))
print(f"[RANK={RANK}] {outputs[0].outputs[0].text!r}")

Limitations & Notes

• No NVLink: DGX Spark nodes communicate via Socket NCCL over 1 GbE. This is the primary throughput bottleneck — each decode step requires an all-reduce across all 4 nodes.
• Marlin MoE backend: GB10 doesn't have native FP4 compute, so the Marlin backend does FP4→FP16 decompression. A native FP4 backend would be faster.
• No CUDA graphs: enforce_eager=True because CUDA graph capture isn't supported with this architecture on GB10 yet. CUDA graphs would significantly improve decode throughput.
• Quantization artifacts: NVFP4 quantization shows degeneration on some tasks (multilingual, complex reasoning). FP8 or unquantized weights would produce better quality.
• JIT compilation on first run: The first inference takes ~15s due to TileLang and Triton kernel compilation. Subsequent runs are ~0.19s TTFT.

What's Next

• Test with longer context lengths (32K, 65K, 131K)
• Try FP8 quantization for quality comparison
• Benchmark with CUDA graphs once supported
• Compare with single-node TP1 (18B active fits in one 128GB node)

Tested 2026-08-27 on 4× DGX Spark (GB10, 128GB). Built image from vLLM GLM-5.3-Flash branch with sm_121 CUDA support.