Instructions to use CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8", trust_remote_code=True)
model = AutoModelForMultimodalLM.from_pretrained("CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8", trust_remote_code=True, device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8

SGLang

How to use CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8 with Docker Model Runner:
```
docker model run hf.co/CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8
```

Qwen3-VL-30B-A3B-Instruct — SmoothQuant-W8A8

Quantized derivative of Qwen/Qwen3-VL-30B-A3B-Instruct, created using a local llm-compressor checkout. SmoothQuant with smoothing strength 0.8, followed by INT8 rounding. All expert gate/up projections and the unquantized router are balanced against each post-attention norm to preserve routing under smoothing.

Setting	Value
Weights	INT8, symmetric, per output channel
Activations	INT8, dynamic per token (W8A8)
Checkpoint format	compressed-tensors
Quantized linear modules	18,624
Weight shard size	30.08 GiB in 9 shards
Base revision	`9c4b90e1e4ba969fd3b5378b57d966d725f1b86c`

Language attention and all MoE expert linear weights are quantized. Vision weights, embeddings, router gates, normalization layers, and lm_head use BF16. The MoE stores all 30B-class model weights even though only a subset of experts is active for each token.

Calibration

128 shuffled Flickr30k test-split image/first-caption pairs, seed 42, with every expert receiving calibration inputs. Each example uses up to 256 image tokens and 1024 total tokens. Dataset revision: 765d117f3eec816f2bfdc2d73ebb50a6f77b86a4. Each quantization method starts independently from the original BF16 weights. The recipe and package versions are in recipe.yaml and quantization_run.json.

Load with Transformers

The source model's fused 3D expert parameters were converted to separate 2D linear modules. Use AutoModelForImageTextToText with trust_remote_code=True to load the bundled modeling_qwen3_vl_moe_quantized.py implementation. Loading directly with the stock Qwen3-VL MoE class does not reproduce this layout.

The validated environment used Torch 2.7.0, Transformers 4.57.1, compressed-tensors 0.13.0, and Accelerate. Transformers can decompress weights to BF16 while loading; allow memory for the full BF16 model plus working space. Saved checkpoint size is not a measurement of runtime GPU memory.

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="sdpa",
).eval()

This is a compressed-tensors export. Compatibility with AutoGPTQ serialization, vLLM serving, and accelerated low-bit kernels has not been validated.

Validation

On September 13, 2026, the local export passed a fresh Transformers reload with no missing, unexpected, or mismatched weights and no loader errors. Greedy generation of 48 tokens for the next held-out Flickr30k image produced a nonempty image description with no NaN generation scores. See validation.json. artifact_audit.json records quantization coverage, storage dtypes, shard index, and file completeness checks for all 18,624 quantized linears.

These checks establish reload and generation functionality. Benchmark accuracy, accuracy relative to the original model, and low-bit serving speed were not measured in this validation. The calibration and validation images are not redistributed in this model repository.

License and attribution

The original model is from the Qwen team and is released under Apache-2.0. This repository provides a quantized derivative under the same license. See the original model card, LICENSE, and NOTICE.

Downloads last month: 29

Safetensors

Model size

31B params

Tensor type

BF16

Model tree for CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8

Base model

Qwen/Qwen3-VL-30B-A3B-Instruct

Quantized

(63)

this model

CompressedMichael
/

Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8