Qwen3.6-35B-A3B-TextOnly-FP4

This is a text-only, hardware-accelerated 4-bit FP4 quantization of Qwen/Qwen3.6-35B-A3B (a Mamba-Attention Hybrid MoE model with 35B active parameters and 1M context length).

This optimized checkpoint was created by pruning the multimodal vision encoder and MTP heads to drastically reduce the VRAM footprint (saving ~3.7 GB of VRAM), and then quantizing it to the W4A16_NVFP4 format using Nvidia ModelOpt. It is designed to be served natively under vLLM V1 with FP8 KV cache support.

Key Work Done & Optimizations

1. Multimodal Pruning: Removed the visual tower and MTP heads, lowering the model's baseline VRAM occupancy from ~13.7 GB to ~9.8 GB per GPU (under TP=2). This frees up VRAM to host a significantly larger KV cache context window.
2. M-RoPE Causal Patching: Patched the text-only causal LM base class (Qwen3_5ForCausalLMBase) in vLLM to natively support M-RoPE positional scaling. This retains 100% of the text model's context reasoning capabilities without needing to load or execute the heavy vision class wrappers.
3. Nvidia ModelOpt Quantization: Calibrated and quantized the weights to Nvidia's W4A16_NVFP4 block-wise schema (block size 16), reducing model size on disk by 50% (from 41.7 GB to 20.6 GB).
4. vLLM V1 Integration: Serves fully in Compiled Graph Mode with AOT graph compilation and CUDA Graph captures active, bypassing all Triton/Dynamo shape validation bugs under large context lengths.

Hardware & Serving Recommendations

GPU Requirements

• Ideal Dual-GPU Setup: Two RTX 5060 Ti GPUs (8GB VRAM each) for cost-efficient local deployment, or any Blackwell GPU pair (e.g., RTX 5080 / 5090) for larger KV cache headroom.
• Ideal Single-GPU Setup: One RTX 5090 (24GB/32GB VRAM).
• VRAM Budget: The quantized FP4 model weights occupy 20.6 GB (10.3 GB per GPU under TP=2). Operating with FP8 KV cache enables massive contexts to scale comfortably.

vLLM Serving Command

vllm serve Cadododoom/Qwen3.6-35B-A3B-TextOnly-FP4 \
  --tensor-parallel-size 2 \
  --quantization modelopt_fp4 \
  --kv-cache-dtype fp8 \
  --max-model-len 65536 \
  --trust-remote-code

Performance Metrics (TP=2, Dual GPU)

• Time to First Token (TTFT): 1.72s (1 stream) / 3.58s (32 concurrent streams).
• Decode Throughput: 102.7 tokens/s (1 stream) / 774.29 tokens/s (cumulative system throughput under 32-concurrency load).
• Driver Stability: 100% stable under maximum GPU load, with GSP compute heartbeat checks active.