Vishva007/Qwen3.5-9B-NVFP4

This is an NVFP4 (W4A16: 4-bit weight, 16-bit activation) quantized version of Qwen/Qwen3.5-9B, produced using AutoRound — Intel's sign gradient descent based quantization method designed for production-grade accuracy retention. Exported via compressed-tensors (llm_compressor format) with MTP layers preserved.

Quantization Details

Parameter	Value
Method	AutoRound (NVFP4 Scheme)
Format	`llm_compressor` / `compressed-tensors`
Group Size	16
Symmetric	Yes
Iterations	1000
Calibration Samples	512
Batch Size	8
Sequence Length	2048
Torch Compile	Disabled

Key Notes

NVFP4 Quantization — Weights are quantized to FP4 micro-scaled formats using group size 16; activations remain in 16-bit for stable numerical fidelity.
Vision Tower (visual / quant_nontext_module): Kept in BF16 to preserve visual reasoning and OCR precision.
Special Modules (ignore_layers & layer_config): linear_attn, mtp, and mtp.fc are strictly retained in native bfloat16 to ensure zero regression in speculative decoding and attention mechanisms.

vLLM Serving

Serve the model directly via vLLM with tool-calling, MTP speculative decoding, and multimodal options enabled:

vllm serve Vishva007/Qwen3.5-9B-NVFP4 \
  --port 8000 \
  --host 0.0.0.0 \
  --max-model-len 18000 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 18000 \
  --max-num-seqs 128 \
  --trust-remote-code \
  --dtype bfloat16 \
  --mm-processor-kwargs '{"max_pixels": 10485760, "max_length": 65536, "truncation": false}' \
  --mm-processor-cache-type shm \
  --limit-mm-per-prompt '{"image": 50}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{"num_speculative_tokens": 3, "method": "mtp"}'

MTP / Speculative Decoding

This model supports Multi-Token Prediction (MTP) for accelerated inference throughput using speculative decoding.

When serving with compatible backends like vLLM, enable MTP using:

--speculative-config '{"num_speculative_tokens": 3, "method": "mtp"}'

Notes

Native draft weights for MTP were preserved during quantization to ensure seamless compatibility without drafting degradation.
You can tune num_speculative_tokens (e.g., between 1 and 3) to optimize speed and latency depending on context length and hardware.

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.14

Template	CUDA Version	Docker Image	Template ID
PyTorch 2.14 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.14-runpod`	`d7lxsa4w9m`
PyTorch 2.14 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.14-runpod`	`yk0y6j6rpg`
PyTorch 2.14 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.14-runpod`	`gsp4gwx0nw`

PyTorch 2.13

Template	CUDA Version	Docker Image	Template ID
PyTorch 2.13 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.13-runpod`	`gmlupxnxfk`
PyTorch 2.13 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.13-runpod`	`y3j8xvk4f4`
PyTorch 2.13 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.13-runpod`	`vigpissn5w`

PyTorch 2.12

Template	CUDA Version	Docker Image	Template ID
PyTorch 2.12 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.12-runpod`	`ctmz86zmf0`
PyTorch 2.12 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.12-runpod`	`qjko5yiwzi`
PyTorch 2.12 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.12-runpod`	`ifg6xmye0f`