Qwen3-VL-8B-Instruct-w4a16-llmcompressor-v0.12.0

Model Overview

• Model Architecture: Qwen3VLForConditionalGeneration
  • Input: Text, Image
  • Output: Text
• Source Model: Qwen3-VL-8B-Instruct
• Supported Hardware: AMD EPYC (CPU inference)
• Preferred Operating System: Linux
• Inference Engine: vLLM v0.26.0
• Quantization Framework: LLM Compressor v0.12.0
• Quantization Method: 4-bit Weight-Only Quantization (W4A16)
• Compatible Stack:
  • ZenDNN v6.1.0
  • ZenTorch v2.11.0.3
  • PyTorch v2.11.0
  • LLM Compressor v0.12.0
  • vLLM v0.26.0

This is a quantized version of Qwen3-VL-8B-Instruct created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from Qwen3-VL-8B-Instruct using LLM Compressor via the GPTQ algorithm. This reduces the model weights from 16.3 GiB to 6.7 GiB on disk (~59% reduction).

• Method: 4-bit Weight-Only Quantization (W4A16)
• Config: compressed-tensors, num_bits=4, type=int, symmetric=true, group_size=128
• Weights: INT4, symmetric, group-wise (group_size=128, actorder=static), stored as pack-quantized
• Activations: BF16 (unquantized)
• Group Size: 128
• Calibration: 128 text-only examples from HuggingFaceH4/ultrachat_200k at a max sequence length of 2048
• Kept in BF16: the vision tower and vision-to-text projector (model.visual.*) and lm_head. Only the language-model Linear layers are quantized, so a text-only calibration set exercises exactly the modules being quantized.

import torch
from transformers import AutoProcessor, AutoTokenizer, Qwen3VLForConditionalGeneration
from datasets import load_dataset

from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier

model_id = "Qwen/Qwen3-VL-8B-Instruct"
output_dir = "./Qwen3-VL-8B-Instruct-w4a16-llmcompressor-v0.12.0"

NUM_CALIBRATION_SAMPLES = 128
MAX_SEQUENCE_LENGTH = 2048

# Step 1: Load the BF16 model and tokenizer.
# Load the top-level Qwen3VLForConditionalGeneration rather than AutoModelForCausalLM,
# which would demote config.json to the inner text-only LM and produce a checkpoint
# vLLM rejects.
model = Qwen3VLForConditionalGeneration.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="cpu",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Step 2: Load calibration data. GPTQ is data-driven: it needs real activations to
# build the per-layer Hessians used to compensate the rounding error.
ds = load_dataset(
    "HuggingFaceH4/ultrachat_200k",
    split=f"train_sft[:{NUM_CALIBRATION_SAMPLES}]",
)
ds = ds.map(
    lambda example: {"text": "\n".join(m["content"] for m in example["messages"] if m["content"])},
    remove_columns=ds.column_names,
)

# Step 3: Define the W4A16 GPTQ recipe. The vision tower lives under model.visual.*
# and is kept in BF16 along with lm_head.
recipe = GPTQModifier(
    scheme="W4A16",
    targets="Linear",
    ignore=[r"re:.*lm_head", r"re:.*visual.*"],
)

# Step 4: One-shot quantize with calibration and save in compressed-tensors format
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    tokenizer=tokenizer,
    output_dir=output_dir,
    trust_remote_code_model=True,
)

# oneshot does not save the processor; multimodal checkpoints need it for vLLM.
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
processor.save_pretrained(output_dir)

# Smoke test (text-only)
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Quick Start

Use with vLLM

from vllm import LLM, SamplingParams

model = LLM(
    model="amd/Qwen3-VL-8B-Instruct-w4a16-llmcompressor-v0.12.0",
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

torch==2.11.0
zentorch==2.11.0.3
vllm==0.26.0
llmcompressor==0.12.0

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on multimodal benchmarks using lm-evaluation-harness with the vLLM vision-language engine.

Benchmark	BF16 Baseline	W4A16 (this model)	Recovery
ChartQA	0.5544	0.5884	106.13%
MMMU (val, tech & engineering)	0.3952	0.3476	87.96%

Evaluation Command

lm_eval \
    --model vllm-vlm \
    --model_args pretrained=amd/Qwen3-VL-8B-Instruct-w4a16-llmcompressor-v0.12.0,dtype=bfloat16 \
    --tasks chartqa,mmmu_val_tech_and_engineering \
    --batch_size auto \
    --trust_remote_code \
    --apply_chat_template \
    --log_samples \
    --output_path .

Limitations

• Version Lock: This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.11.0.3 / PyTorch v2.11.0. It may not load correctly on other versions.
• CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.
• Vision Path Unquantized: The vision tower remains in BF16, so the memory saving is smaller than for text-only W4A16 models and image preprocessing cost is unchanged.
• Accuracy Trade-off: 4-bit weight-only quantization is more aggressive than INT8. Knowledge-heavy multimodal reasoning is the most affected: MMMU drops to about 88% of the BF16 baseline, while chart reading is unaffected.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.