Instructions to use amd/gemma-4-26B-A4B-it-w4a16-llmcompressor with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use amd/gemma-4-26B-A4B-it-w4a16-llmcompressor with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="amd/gemma-4-26B-A4B-it-w4a16-llmcompressor")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("amd/gemma-4-26B-A4B-it-w4a16-llmcompressor")
model = AutoModelForMultimodalLM.from_pretrained("amd/gemma-4-26B-A4B-it-w4a16-llmcompressor", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use amd/gemma-4-26B-A4B-it-w4a16-llmcompressor with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "amd/gemma-4-26B-A4B-it-w4a16-llmcompressor"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/gemma-4-26B-A4B-it-w4a16-llmcompressor",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/amd/gemma-4-26B-A4B-it-w4a16-llmcompressor

SGLang

How to use amd/gemma-4-26B-A4B-it-w4a16-llmcompressor with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "amd/gemma-4-26B-A4B-it-w4a16-llmcompressor" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/gemma-4-26B-A4B-it-w4a16-llmcompressor",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "amd/gemma-4-26B-A4B-it-w4a16-llmcompressor" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/gemma-4-26B-A4B-it-w4a16-llmcompressor",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use amd/gemma-4-26B-A4B-it-w4a16-llmcompressor with Docker Model Runner:
```
docker model run hf.co/amd/gemma-4-26B-A4B-it-w4a16-llmcompressor
```

gemma-4-26B-A4B-it-w4a16-llmcompressor

Model Overview

Model Architecture: Gemma4ForConditionalGeneration
- Input: Text
- Output: Text
Source Model: gemma-4-26B-A4B-it
Supported Hardware: AMD EPYC (CPU inference)
Preferred Operating System: Linux
Inference Engine: vLLM v0.26.0
Quantization Framework: LLM Compressor v0.12.0
Quantization Method: 4-bit Weight-Only Quantization (W4A16)
Compatible Stack:
- ZenDNN v6.1.0
- ZenTorch v2.11.0.3
- PyTorch v2.11.0
- LLM Compressor v0.12.0
- vLLM v0.26.0
Published with: LLM Compressor v0.12.0

This is a quantized version of gemma-4-26B-A4B-it created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from gemma-4-26B-A4B-it using LLM Compressor via the GPTQ algorithm. This reduces the model weights from 48.1 GiB to 14.6 GiB on disk (~70% reduction).

Method: 4-bit Weight-Only Quantization (W4A16)
Config: compressed-tensors, num_bits=4, type=int, symmetric=true, group_size=64
Weights: INT4, symmetric, group-wise (group_size=64, actorder=static), stored as pack-quantized
Activations: BF16 (unquantized)
Group Size: 64
Calibration: 128 examples from HuggingFaceH4/ultrachat_200k at a max sequence length of 2048
Kept in BF16: the MoE router (router.proj), the vision tower (model.vision_tower), the vision-to-text projector (model.embed_vision), lm_head, and the layer norms. The text-tower attention projections, the dense shared MLP, and all 128 routed experts are quantized.

The group size is 64, not the usual 128. Gemma-4's down_proj input dimensions are 2112 in the dense MLP and 704 in each expert, neither of which is divisible by 128, so the stock W4A16 preset fails validation. 64 divides all of them (704/64=11, 2112/64=33, 2816/64=44), which is why the recipe builds an explicit config_groups entry instead of naming the preset.

import torch
from transformers import AutoProcessor, AutoTokenizer, Gemma4ForConditionalGeneration
from datasets import load_dataset

from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier
from compressed_tensors.quantization import (
    QuantizationArgs,
    QuantizationScheme,
    QuantizationStrategy,
    QuantizationType,
)

model_id = "RedHatAI/gemma-4-26B-A4B-it"
output_dir = "./gemma-4-26B-A4B-it-w4a16-llmcompressor"

CALIB_SIZE = 128
MAX_SEQ_LENGTH = 2048

# Step 1: Load the BF16 model and tokenizer.
# Load the top-level Gemma4ForConditionalGeneration rather than AutoModelForCausalLM,
# which would demote config.json to the text-only Gemma4TextConfig and produce a
# checkpoint vLLM rejects.
model = Gemma4ForConditionalGeneration.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="cpu",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Step 2: Load calibration data. GPTQ is data-driven: it needs real activations to
# build the per-layer Hessians used to compensate the rounding error.
ds = load_dataset("HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{CALIB_SIZE}]")
ds = ds.map(
    lambda example: {"text": "\n".join(m["content"] for m in example["messages"] if m["content"])},
    remove_columns=ds.column_names,
)

# Step 3: Define the W4A16 GPTQ recipe with an explicit group_size=64. The default
# 128 fails validation: the down_proj input dims (2112 dense, 704 expert) are not
# divisible by 128, while 64 divides every quantized weight's column count.
w4a16_g64 = QuantizationScheme(
    targets=["Linear"],
    weights=QuantizationArgs(
        num_bits=4,
        type=QuantizationType.INT,
        symmetric=True,
        strategy=QuantizationStrategy.GROUP,
        group_size=64,
    ),
)

recipe = GPTQModifier(
    config_groups={"group_0": w4a16_g64},
    ignore=[
        "lm_head",
        r"re:.*lm_head",
        # MoE router — the single most important layer to skip.
        r"re:.*router\.proj$",
        # Vision tower + multimodal projector must stay BF16: their activation
        # statistics are not represented in a text-only calibration set.
        r"re:.*vision_tower.*",
        r"re:.*embed_vision.*",
        r"re:.*embed_audio.*",
    ],
)

# Step 4: One-shot quantize with calibration and save in compressed-tensors format
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQ_LENGTH,
    tokenizer=tokenizer,
    output_dir=output_dir,
    trust_remote_code_model=True,
)

# oneshot does not save the processor; multimodal checkpoints need it for vLLM.
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
processor.save_pretrained(output_dir)

# Smoke test
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Quick Start

Use with vLLM

from vllm import LLM, SamplingParams

model = LLM(
    model="amd/gemma-4-26B-A4B-it-w4a16-llmcompressor",
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

torch==2.11.0
zentorch==2.11.0.3
vllm==0.26.0
llmcompressor==0.12.0

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.

Benchmark	BF16 Baseline	W4A16 (this model)	Recovery
GSM8K (5-shot)	0.9469	0.9325	98.48%

Evaluation Command

lm_eval \
    --model vllm \
    --model_args pretrained=amd/gemma-4-26B-A4B-it-w4a16-llmcompressor,tokenizer=RedHatAI/gemma-4-26B-A4B-it,dtype=bfloat16,max_model_len=4096,language_model_only=True \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --apply_chat_template \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --output_path .

Limitations

Version Lock: This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.11.0.3 / PyTorch v2.11.0. It may not load correctly on other versions.
CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.
Non-Standard Group Size: This checkpoint uses group_size=64 rather than the more common 128. Tooling that assumes a 128-wide group will not read it correctly.
Vision Path Unquantized: The vision tower and projector stay in BF16. Evaluation was run with language_model_only=True.
Accuracy Trade-off: 4-bit weight-only quantization is more aggressive than INT8. On GSM8K the model retains 98.48% of the BF16 baseline for a ~70% smaller memory footprint.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

Downloads last month: 713

Safetensors

Model size

26B params

Tensor type

I32

BF16

Model tree for amd/gemma-4-26B-A4B-it-w4a16-llmcompressor

Base model

google/gemma-4-26B-A4B

Finetuned

RedHatAI/gemma-4-26B-A4B-it

Quantized

(2)

this model