Qwen3.5-9B-NVFP4

Model Overview

• Model Architecture: Qwen/Qwen3.5-9B (Qwen3_5ForConditionalGeneration)
  • Input: Text / Image
  • Output: Text
• Model Optimizations:
  • Weight quantization: FP4 (NVFP4)
  • Activation quantization: FP4 (NVFP4)
• Release Date: 11/06/2026
• Version: 1.1
• Model Developers: RedHatAI

This model is a quantized version of Qwen/Qwen3.5-9B. It was evaluated on several tasks to assess its quality.

Model Optimizations

This model was obtained by quantizing the weights and activations of the language-model linear operators of Qwen/Qwen3.5-9B to FP4 (NVFP4) data type, ready for inference with vLLM. The vision encoder and the MTP layer are kept in BF16, matching the source checkpoint.

This optimization reduces the number of bits per parameter in the quantized layers from 16 to 4, roughly halving the checkpoint size (9.4 GB vs 19.3 GB BF16).

Quantization was performed with LLM Compressor using GPTQ with the NVFP4 scheme and nvfp4_expanded_imatrix weight observer, block size 128, static actorder and dampening fraction 0.01; recipe.yaml in this repository holds the exact recipe. The model is exported in compressed-tensors format.

Deployment

Use with vLLM

The model runs on a single GPU. Following the official vLLM recipe:

vllm serve RedHatAI/Qwen3.5-9B-NVFP4 \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --reasoning-parser qwen3

Tool use:

vllm serve RedHatAI/Qwen3.5-9B-NVFP4 \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

The checkpoint ships the MTP weights, so vLLM's MTP speculative decoding can be enabled with --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'.

Evaluation

This model and the unquantized Qwen/Qwen3.5-9B were evaluated on GSM8K, GPQA Diamond, MathVista (testmini split) and MMLU-Pro using Inspect with the inspect_evals task suite, served with vLLM (OpenAI-compatible API, single B200 GPU, vLLM nightly). Both models ran through the identical harness, serving stack, sampling and limits — single epoch (pass@1), thinking mode enabled, sampling following the Qwen3.5 card's thinking-mode recommendation (temperature 1.0, top_p 0.95, top_k 20, presence_penalty 1.5) — so the recovery column below is a controlled like-for-like comparison.

Accuracy

Category	Benchmark	Qwen/Qwen3.5-9B (BF16)	RedHatAI/Qwen3.5-9B-NVFP4	Recovery
Reasoning	GSM8K (10-shot CoT, pass@1)	95.9%	95.7%	99.8%
	GPQA Diamond (0-shot, pass@1)	82.3%	79.3%	96.3%
	MathVista testmini (pass@1)	85.6%	84.4%	98.6%
	MMLU-Pro (0-shot, pass@1)	82.9%	81.2%	98.1%

Standard errors of the measured scores: GSM8K ±0.6 (BF16) / ±0.6 (NVFP4), GPQA Diamond ±2.7 / ±2.9, MathVista ±1.1 / ±1.1, MMLU-Pro ±0.3 / ±0.4 percentage points. The quantized checkpoint reasons longer than the BF16 reference (mean output on MMLU-Pro 6,455 vs 5,029 tokens; 158 of 12,032 samples reached the 32,768-token generation cap, scoring as failures), which accounts for most of its remaining losses. For reference, the Qwen3.5-9B model card reports 81.7 (GPQA Diamond), 85.7 (MathVista mini) and 82.5 (MMLU-Pro) under Qwen's own protocol, consistent with the BF16 measurements above.

Reproduction

Both checkpoints (this model and Qwen/Qwen3.5-9B) were served with vLLM and evaluated for one epoch per benchmark with identical settings (only the model id differs):

vllm serve RedHatAI/Qwen3.5-9B-NVFP4 \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": true}'

COMMON="--model vllm/RedHatAI/Qwen3.5-9B-NVFP4 --temperature 1.0 --top-p 0.95 --presence-penalty 1.5 --generate-config generate_config.json"
# generate_config.json: {"extra_body": {"top_k": 20, "chat_template_kwargs": {"enable_thinking": true}}}

inspect eval inspect_evals/gsm8k       $COMMON --max-tokens 32768
inspect eval inspect_evals/mmlu_pro    $COMMON --max-tokens 32768
inspect eval inspect_evals/gpqa_diamond $COMMON --max-tokens 81920
inspect eval inspect_evals/mathvista   $COMMON --max-tokens 81920