aleada/Gemma-3-12B-it-W4A16

⚠️ Update (2026-07-05) — vLLM loading fixed

An earlier revision of these weights failed to load on vLLM (≤ 0.23.x), raising:

KeyError: embeddings.patch_embedding.bias

Cause. transformers > 4.52.2 changed the multimodal weight-naming convention (to model.vision_tower.* / model.language_model.*), and GPTQModel followed it during quantization — but vLLM still expects the previous naming (vision_tower.vision_model.* / language_model.model.*), so the SigLIP vision tower failed to map (upstream: llm-compressor #1546).

Fix. The tensor keys are remapped to the vLLM-expected convention. The quantized weights are byte-identical — no re-quantization, nothing else changed. Just re-download the safetensors (or re-pull the repo) and it loads.

Verified: loads + runs on vLLM v0.23.0 (TP = 2), and still loads under transformers. Details in the Community tab.

gemma-3-12b-it-qat-q4_0-unquantized — W4A16 (GPTQModel)

Standard W4A16 GPTQ quantization of Lightricks/gemma-3-12b-it-qat-q4_0-unquantized, produced with GPTQModel 7.1.0 inside a reproducible Docker container. The artifact is drop-in loadable by vLLM (≤ 0.23.x, tensor-parallel) — vLLM auto-detects the GPTQ method from the embedded quantization_config at load time. (See the update note above: the vision-tower key naming is now aligned to vLLM's expectation.)

This release fills a gap where the canonical llm-compressor path fails — specifically hybrid SSM + Attention architectures like GraniteMoeHybrid that llm-compressor 0.12.0's sequential GPTQ pipeline can't handle (KeyError 'mamba' in causal mask lookup). GPTQModel ships a model-specific adapter that monkey-patches the Mamba forward path so only the attention + MoE Linears get quantized.

Maintained by Alex Adamopoulos at assert.gr as part of an ongoing series of vLLM-friendly quantized packs targeting underserved 2026 models.

Reproducibility

Parameter	Value
Source model	Lightricks/gemma-3-12b-it-qat-q4_0-unquantized
Quantization tool	GPTQModel 7.1.0 (ModelCloud)
bits	4
group_size	128
desc_act	False
sym	True
Calibration dataset	allenai/c4
Calibration samples	128
Quantized size	7.87 GiB

License

Inherits the license of the base model. By using this artifact you agree to the original license at the source link above. Atlas / assert.gr adds no additional restrictions on the quantized weights.

Usage with vLLM

docker run --runtime=nvidia --gpus all \
    -p 8000:8000 \
    -e HF_TOKEN=hf_XXX \
    vllm/vllm-openai:latest \
    --model aleada/Gemma-3-12B-it-W4A16 \
    --limit-mm-per-prompt 'image=1' \
    --gpu-memory-utilization 0.92 \
    --enable-prefix-caching

vLLM auto-detects the GPTQ format from the embedded quantization_config.quant_method. Passing --quantization gptq is allowed but redundant. vLLM also picks the model's full native context window from config.json. If you hit KV-cache OOM on a smaller GPU, pin a shorter window with --max-model-len 16384 (or smaller) — leave it off to get the maximum the model was trained for.

Vision encoder + multimodal projector remain BF16 by design — quantizing them gives negligible memory benefit relative to accuracy cost. Only the language tower's Linear weights are W4 GPTQ-quantized. Once vLLM is running, hit it with any OpenAI client:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="aleada/Gemma-3-12B-it-W4A16",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Hardware target

Requires CUDA compute-capability ≥ 7.5 (Turing+). Verified on NVIDIA RTX 3090 (compute 8.6) where the W4A16 path runs the quantized Linears at INT4 weights / BF16 activations.

About the maintainer

Alex Adamopoulos is the founder of assert.gr and the engineer behind the atlas self-evolving AI agent platform.

Connect:

• HuggingFace: @aleada
• Company: assert.gr