Qwen

/

Qwen3.6-27B-FP8

like 347

Follow

Qwen 99.6k

Claude came up with this, which does work for me.
Also make sure you have enough RAM

# vLLM stack config — Qwen3.6-27B-FP8 on RTX PRO 6000 Blackwell Max-Q (96 GB), WSL2.
# Tuned 2026-06-17 from a full BF16/FP8/NVFP4 benchmark matrix (short-context, long-context
# 8K/32K/128K, decode-max). FP8 chosen: bit-identical quality to BF16 (objective 0.967 both),
# best high-concurrency aggregate decode, official checkpoint. See README.md for all numbers.
VLLM_IMAGE=vllm/vllm-openai:v0.23.0-cu129-ubuntu2404
VLLM_PORT=8000
HF_HOME_HOST=/home/ml/ml/hf-cache

# Model + served name (consumed by docker-compose.yml command:)
MODEL=Qwen/Qwen3.6-27B-FP8
SERVED_NAME=qwen3.6-27b

# Full native context. fp8 KV cache (in compose) is what makes 262144 fit at high concurrency.
MAX_MODEL_LEN=262144


CUDAGRAPH_SIZES=[1,2,4,8,16,32]
MAX_NUM_BATCHED_TOKENS=4096


# --- GPU memory utilization ---
# CORRECTION (2026-06-17): the old "~76 GiB single-process WSL wall" was STALE/WRONG for this
# driver (596.72). A direct allocation probe reached 94 GiB in one CUDA process (0.98 of the
# 95.6 GiB total) before OOM. So we run near the card, not at 0.74.
# Measured FP8 @ 0.90, vision ON: 1,729,547 KV tokens (6.60x @262k), boots + captures cleanly,
# peak 301 W, max 87 C, zero thermal throttle. 0.90 leaves ~9.6 GiB for CUDA context + capture
# transients + cuBLAS/vision buffers (the probe proved the process can reach 94 GiB, so ample).
# 2026-06-22: lowered 0.90 -> 0.80. At 0.90 (only ~9.6 GiB headroom) vLLM crash-looped on
# WSL2 with a fragmentation OOM during KV-cache alloc (torch.zeros 3.43 GiB failed with
# 33 GiB free). expandable_segments (the usual fix) is unsupported on WSL2 (see compose
# note), so the fix is more headroom + smaller KV cache. 0.80 still gives >=256 Mamba
# blocks for MAX_NUM_SEQS=256. Revert to 0.90 only if fragmentation no longer recurs.
GPU_MEM_UTIL=0.80

# --- Concurrent sequences ---
# Hybrid model (Gated-DeltaNet + attention): each decode seq needs one Mamba state-cache block,
# so vLLM REQUIRES max_num_seqs <= the Mamba block ceiling or CUDA-graph capture crashes
# ("exceeds available Mamba cache blocks"). Measured ceiling @ 0.90 = 897.
# BUT 256 is the SWEET SPOT, not a compromise: decode-max showed aggregate decode peaks at
# batch 256 (2,022 tok/s) and DROPS at 768 (1,789 tok/s, -12%) — bandwidth-saturated. And at
# 256 the KV cache is LARGER (1,729,547 tok) than at 897 (1,678,324) since fewer Mamba blocks
# are reserved. So 256 wins on BOTH throughput and long-context capacity.
# Raise toward 897 ONLY if you must ADMIT >256 simultaneous requests (deeper queue) and accept
# lower per-request throughput. Never exceed 897 at this util.
MAX_NUM_SEQS=256

services:
  vllm:
    image: ${VLLM_IMAGE}
    container_name: vllm-qwen36
    restart: ${RESTART:-unless-stopped}
    ports:
      - "${VLLM_PORT}:8000"
    volumes:
      # ext4 HF cache mounted read-only-ish; vLLM only reads weights from here
      - ${HF_HOME_HOST}:/hf-cache
    environment:
      - HF_HOME=/hf-cache
      # Model is fully present locally -> never hit the network on boot
      - HF_HUB_OFFLINE=1
      - TRANSFORMERS_OFFLINE=1
      # RAM-bound trick: stream safetensors straight to GPU, bypass host-RAM staging
      - SAFETENSORS_FAST_GPU=1
      # NOTE: Do NOT set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True on WSL2 —
      # it relies on CUDA VMM (cuMemMap) which WSL2 doesn't support and causes
      # "CUDA driver error: unknown error" at first allocation. Fine on bare-metal Linux.
      - OMP_NUM_THREADS=8
      - VLLM_NO_USAGE_STATS=1
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
    # vLLM needs large shared memory; host IPC is the recommended setting
    ipc: host
    ulimits:
      memlock: -1
      stack: 67108864
    # WSL2: `gpus: all` (== `docker run --gpus all`) correctly injects /dev/dxg and
    # the WSL driver libs. The deploy.reservations.devices form does NOT in WSL and
    # yields "CUDA driver error: unknown error" at context creation.
    gpus: all
    # List form: each element is one literal argv entry (no shell word-splitting),
    # so JSON args like --compilation-config pass through cleanly.
    command:
      - --model
      - ${MODEL:-Qwen/Qwen3.6-27B-FP8}
      - --served-model-name
      - ${SERVED_NAME:-qwen3.6-27b}
      - --max-model-len
      - "${MAX_MODEL_LEN}"
      - --gpu-memory-utilization
      - "${GPU_MEM_UTIL}"
      - --kv-cache-dtype
      - bfloat16
      # CONCURRENCY BOUND (hybrid-model finding): Qwen3.6-27B is a Gated-DeltaNet+attn
      # HYBRID. Each decode seq needs one Mamba/GDN state-cache block. A literally
      # uncapped run (vLLM default max_num_seqs=1024) CRASHES at CUDA-graph capture:
      #   "max_num_seqs (1024) exceeds available Mamba cache blocks (235)".
      # So the cache (Mamba blocks, which scale with gpu-mem-util) IS the real bound.
      # MAX_NUM_SEQS is therefore set to that auto cache ceiling (NOT the old artificial
      # 8). Measured ceilings: util 0.70 -> 235 blocks; 0.72 -> >=256; 0.74 -> >=272.
      - --max-num-seqs
      - "${MAX_NUM_SEQS}"
      - --enable-chunked-prefill
      - --max-num-batched-tokens
      - "${MAX_NUM_BATCHED_TOKENS:-8192}"
      - --enable-prefix-caching
      - --reasoning-parser
      - qwen3
      # Tool/function calling for OpenAI-compatible clients (the Hermes agent + MCP tools).
      # Qwen3.6 emits Qwen3-Coder XML tool calls (<function=name><parameter=k>v</parameter>),
      # NOT Hermes JSON — so the qwen3_coder parser is required (hermes parser leaves the
      # call as plain text -> no tool_calls). Verified against this checkpoint.
      - --enable-auto-tool-choice
      - --tool-call-parser
      - qwen3_coder
      # CUDA graphs ON (big single-stream decode speedup). Capture set [1,2,4,8] is what
      # was benchmarked; larger batches run eager (fine). With util 0.90 there's plenty of
      # VRAM headroom now, so capturing more sizes (e.g. up to 32/64) is a plausible further
      # decode win at mid batch sizes — test before shipping.
      - --compilation-config
      - '{"cudagraph_capture_sizes":${CUDAGRAPH_SIZES:-[1,2,4,8]}}'
      - --host
      - 0.0.0.0
      - --port
      - "8000"
    healthcheck:
      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
      interval: 30s
      timeout: 10s
      retries: 30
      # Model load from ext4 can take a few minutes; don't mark unhealthy too early
      start_period: 900s