Qwen3.5-9B-NVFP4

Model Overview

• Model Architecture: Qwen/Qwen3.5-9B (Qwen3_5ForConditionalGeneration)
  • Input: Text / Image
  • Output: Text
• Model Optimizations:
  • Weight quantization: NVFP4 (FP4, group size 16 with FP8 group scales and a per-tensor global scale)
  • Activation quantization: NVFP4 (FP4 dynamic per-group-16, calibrated per-tensor global scale)
  • Model size: 11.7 GB (reduced from 19.3 GB in BF16)
• Release Date: 2026-08-27
• Version: 1.0
• Quantized by: RishabhSinha (community contribution)

This model is an NVFP4 (W4A4) quantized version of Qwen/Qwen3.5-9B, produced for the community checkpoint drive in vllm-project/llm-compressor#3088 ("NVFP4" slot for Qwen3.5-9B). NVFP4 is optimized for NVIDIA Blackwell GPUs.

Model Optimizations

This model was obtained by quantizing the weights and activations of Qwen/Qwen3.5-9B to the NVFP4 format: FP4 values in groups of 16 sharing an FP8 scale, with a calibrated per-tensor global scale for activations.

Only the linear operators of the transformer decoder blocks are quantized, following llm-compressor's canonical Qwen3.5 NVFP4 recipe (examples/quantization_w4a4_fp4/qwen3_5_example.py, dense variant): lm_head, embeddings, the vision tower, and the hybrid linear-attention (gated delta net) blocks are kept in BF16, and the MTP (multi-token prediction) layers are carried over unquantized via compressed_tensors.utils.save_mtp_tensors_to_checkpoint.

Calibration: 256 samples from garage-bAInd/Open-Platypus, max sequence length 4096, formatted with the model's chat template.

Quantization was performed with LLM Compressor.

Deployment

Use with vLLM

1. Initialize vLLM server:

vllm serve RishabhSinha/Qwen3.5-9B-NVFP4 \
  --reasoning-parser qwen3 \
  --max-model-len 262144

For text-only serving (lower memory), add --language-model-only.

2. Send requests to the server:

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

outputs = client.chat.completions.create(
    model="RishabhSinha/Qwen3.5-9B-NVFP4",
    messages=[
        {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
    ],
)
print(outputs.choices[0].message.content)

Use with Transformers

from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration

model_id = "RishabhSinha/Qwen3.5-9B-NVFP4"
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    model_id, dtype="auto", device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

messages = [{"role": "user", "content": "Explain quantum mechanics clearly and concisely."}]
input_ids = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
output = model.generate(input_ids, max_new_tokens=256)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Creation

This model was created with LLM Compressor using the NVFP4 preset scheme with Open-Platypus calibration, as shown below.

Creation script

import torch
from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from datasets import load_dataset
from transformers import AutoProcessor, AutoTokenizer, Qwen3_5ForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.utils import load_context

MODEL_ID = "Qwen/Qwen3.5-9B"
SAVE_DIR = "Qwen3.5-9B-NVFP4"

NUM_CALIBRATION_SAMPLES = 256
MAX_SEQUENCE_LENGTH = 4096

IGNORE_LAYERS = [
    "re:.*lm_head",
    "re:visual.*",
    "re:model.visual.*",
    "re:.*embed_tokens$",
    "re:.*linear_attn.*",
]

with load_context(Qwen3_5ForConditionalGeneration):
    model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained(MODEL_ID)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=IGNORE_LAYERS,
)

ds = load_dataset("garage-bAInd/Open-Platypus", split=f"train[:{NUM_CALIBRATION_SAMPLES}]")
ds = ds.shuffle(seed=42)


def preprocess_function(example):
    user_text = example["instruction"]
    if example.get("input"):
        user_text = f"{user_text}\n\n{example['input']}"
    messages = [
        {"role": "user", "content": [{"type": "text", "text": user_text}]},
        {"role": "assistant", "content": [{"type": "text", "text": example["output"]}]},
    ]
    return processor.apply_chat_template(
        messages,
        return_tensors="pt",
        padding=False,
        truncation=True,
        max_length=MAX_SEQUENCE_LENGTH,
        tokenize=True,
        add_special_tokens=False,
        return_dict=True,
        add_generation_prompt=False,
    )


ds = ds.map(preprocess_function, batched=False, remove_columns=ds.column_names)


def data_collator(batch):
    assert len(batch) == 1
    return {key: torch.tensor(value) for key, value in batch[0].items()}


oneshot(
    model=model,
    recipe=recipe,
    dataset=ds,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
    data_collator=data_collator,
)

model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)
processor.save_pretrained(SAVE_DIR)
save_mtp_tensors_to_checkpoint(source_model=MODEL_ID, dest_dir=SAVE_DIR)

Package versions

• llm-compressor==0.13.1.dev51+g50d0a1c75 (main)
• compressed-tensors==0.18.1.dev21+g8c0fa69 (main)
• transformers==5.16.1
• torch==2.11.0+cu128

Hardware: 1x NVIDIA L40S (48 GB).

Evaluation

Sanity checks only so far — lm-eval results pending.

The checkpoint was reloaded fresh from disk with transformers + compressed-tensors on an NVIDIA L40S; greedy generations are coherent and logits are finite:

PROMPT: 'The capital of France is'
OUTPUT: 'The capital of France is Paris.\nThe capital of France is Paris.\n...'

PROMPT: 'def fibonacci(n):'
OUTPUT: 'def fibonacci(n):\n    if n == 0:\n        return 0\n    elif n == 1:\n        return 1\n    else:\n       '

PROMPT: 'Water boils at'
OUTPUT: 'Water boils at 100°C at sea level. At what temperature does water boil at 10,000 feet above sea level?\n\nA. '

The checkpoint has not yet been validated in vLLM on this hardware (native NVFP4 execution targets NVIDIA Blackwell; vLLM falls back to emulated kernels on older GPUs). Formal benchmark results (lm-evaluation-harness) have not yet been run for this checkpoint. If you run them, contributions to this model card are welcome.