Qwen3.6-35B-A3B-DSV4Pro-TextOnly-FP4

This is a text-only, hardware-accelerated 4-bit FP4 quantization of nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill (the thinking-distilled reasoning variant of Qwen 3.6 35B).

Like the non-distilled model, it has been pruned of its vision tower and MTP heads to minimize VRAM usage, and quantized to the W4A16_NVFP4 format using Nvidia ModelOpt. It supports structured reasoning traces in a text-only causal context.

Key Work Done & Optimizations

1. Multimodal Pruning: Removed the visual tower and MTP heads, lowering the baseline VRAM footprint to ~9.8 GB per GPU (under TP=2) and freeing up ~3.7 GB of VRAM to allocate to the KV cache.
2. M-RoPE Causal Patching: Re-engineered the base causal LM class inside vLLM (Qwen3_5ForCausalLMBase) to support M-RoPE positional scaling, retaining 100% of the reasoning trace accuracy without visual overhead.
3. Nvidia ModelOpt Quantization: Calibrated and quantized the weights to W4A16_NVFP4 block-wise schema, reducing size on disk from 41.7 GB to 20.6 GB.
4. vLLM V1 Integration: Serves fully in Compiled Graph Mode with AOT graph compilation and CUDA Graph captures active.

Hardware & Serving Recommendations

GPU Requirements

• Ideal Dual-GPU Setup: Two RTX 5060 Ti GPUs (8GB VRAM each) for budget deployment, or any Blackwell GPU pair for larger KV cache headroom.
• Ideal Single-GPU Setup: One RTX 5090 (24GB/32GB VRAM).
• VRAM Budget: The quantized FP4 model weights occupy 20.6 GB (10.3 GB per GPU under TP=2). Operating with FP8 KV cache enables massive contexts to scale comfortably.

vLLM Serving Command

vllm serve Cadododoom/Qwen3.6-35B-A3B-DSV4Pro-TextOnly-FP4 \
  --tensor-parallel-size 2 \
  --quantization modelopt_fp4 \
  --kv-cache-dtype fp8 \
  --max-model-len 65536 \
  --trust-remote-code

Performance & Accuracy Metrics (TP=2, Dual GPU)

• GSM8K Accuracy: 95.00% (when evaluated with a max_tokens=4096 budget to allow the model to fully complete its thinking logs).
• Time to First Token (TTFT): 2.006s (32 concurrent streams).
• Decode Throughput: 40.50 tokens/s per stream (cumulative system throughput of 1,296 tokens/sec under 32-concurrency load).
• Driver Stability: 100% stable under maximum GPU load, with GSP compute heartbeat checks active.