Instructions to use Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ

SGLang

How to use Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ with Docker Model Runner:
```
docker model run hf.co/Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ
```

Gemma 4 E4B Instruct — W4A16 Quantized AWQ

This repository hosts W4A16 INT4-quantized versions of google/gemma-4-E4B-it, a multimodal mixture-of-experts model supporting text, vision, and audio inputs. Two quantized variants are available:

Note on MTP / Speculative Decoding:
If you want to use the official speculative decoding assistant model (google/gemma-4-E4B-it-assistant) for MTP support, it is recommended to use the vllm/vllm-openai:gemma4-0505-cu129 Docker image, which includes newer Gemma 4 support and decoding patches.

Due to INT4 quantization, the assistant acceptance rate may be lower compared to the original unquantized google/gemma-4-E4B-it model.

Variant	Method	Repo
AutoRound (RTN)	intel/auto-round	`Vishva007/gemma-4-E4B-it-W4A16-AutoRound`
AWQ	AutoAWQ	`Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ`

Note: Only the language model (LM) layers are quantized to INT4. The vision tower, audio tower, and multimodal projectors are kept at full precision (BF16) to preserve multimodal quality.

Quantization Details

Parameter	Value
Base model	`google/gemma-4-E4B-it`
Quantization scheme	W4A16 (INT4 weights, BF16 activations)
Group size	128
Symmetric	Yes
Calibration samples	256
Sequence length	2048
Non-LM modules	Kept at FP32 (vision, audio, projectors)
Quantized layers	All LM linear layers (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`, `per_layer_input_gate`, `per_layer_projection`)
AutoRound mode	RTN (`iters=0`) — required for Gemma 4 compatibility
Hardware used	NVIDIA A100 80GB PCIe
Framework	PyTorch 2.10.0 + CUDA 12.8

Model Architecture

Gemma 4 E4B is a multimodal MoE model (Gemma4ForConditionalGeneration) with:

Text backbone: 42-layer Gemma4TextModel with 2560 hidden dim, mixed local/global attention
Vision tower: 16-layer Gemma4VisionModel (768-dim, unquantized)
Audio tower: 12-layer Gemma4AudioModel with conformer-style layers (unquantized)
Vocabulary size: 262,144 tokens

Usage

vLLM Inference

The recommended way to serve this model is via the official vllm/vllm-openai:gemma4 Docker image, which ships vLLM v0.19.1 with the latest Transformers patches required for Gemma 4.

Serve with Docker (recommended)

docker run --gpus all --rm -p 8000:8000 \
  vllm/vllm-openai:gemma4 \
  vllm serve Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ \
    --served-model-name Gemma-4-E4B-it \
    --quantization autoround \
    --kv-cache-dtype auto \
    --max-num-batched-tokens 16384 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --dtype bfloat16 \
    --max-model-len 18432 \
    --enable-auto-tool-choice \
    --tool-call-parser gemma4 \
    --port 8000 \
    --default-chat-template-kwargs '{"enable_thinking": false}' \
    --mm-processor-kwargs '{"max_soft_tokens": 560}'

Direct `vllm serve` (vLLM ≥ 0.19.0)

vllm serve Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ \
  --served-model-name Gemma-4-E4B-it \
  --quantization autoround \
  --kv-cache-dtype auto \
  --max-num-batched-tokens 16384 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --dtype bfloat16 \
  --max-model-len 18432 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --port 8000 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --mm-processor-kwargs '{"max_soft_tokens": 560}'

`max_soft_tokens` — Image Token Budget

The max_soft_tokens parameter controls how many visual tokens are allocated per image. Higher values give richer image representations at the cost of context length and throughput.

`max_soft_tokens`	Detail level	Recommended use
`70`	Minimal	Fast throughput, simple images
`140`	Low	Charts, diagrams
`280`	Medium (default)	General-purpose
`560`	High	Dense scenes, documents
`1120`	Maximum	Fine-grained visual detail

Pass it via --mm-processor-kwargs '{"max_soft_tokens": <value>}'.

OpenAI-compatible API call

Once the server is running, query it like any OpenAI-compatible endpoint:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="Gemma-4-E4B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe what you see."},
                {"type": "image_url", "image_url": {"url": "https://..."}},
            ],
        }
    ],
    max_tokens=512,
)
print(response.choices.message.content)

The quantized model was then exported in both AutoRound and AWQ formats and pushed to Hugging Face Hub.

Limitations & Notes

RTN mode (iters=0) is used instead of full AutoRound optimization due to Gemma 4's architecture constraints.
Some layers with shapes not divisible by 32 are skipped during quantization (minor precision impact).
Multimodal (vision/audio) capabilities are fully preserved as those towers are not quantized.

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.14

Template	CUDA Version	Docker Image	Template ID
PyTorch 2.14 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.14-runpod`	`d7lxsa4w9m`
PyTorch 2.14 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.14-runpod`	`yk0y6j6rpg`
PyTorch 2.14 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.14-runpod`	`gsp4gwx0nw`

PyTorch 2.13

Template	CUDA Version	Docker Image	Template ID
PyTorch 2.13 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.13-runpod`	`gmlupxnxfk`
PyTorch 2.13 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.13-runpod`	`y3j8xvk4f4`
PyTorch 2.13 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.13-runpod`	`vigpissn5w`

PyTorch 2.12

Template	CUDA Version	Docker Image	Template ID
PyTorch 2.12 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.12-runpod`	`ctmz86zmf0`
PyTorch 2.12 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.12-runpod`	`qjko5yiwzi`
PyTorch 2.12 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.12-runpod`	`ifg6xmye0f`

Acknowledgements

Special thanks to OLAF-OSS and the contributors of the gemma4-vllm project for their excellent work in enabling Gemma 4 support with vLLM. Their QUANTIZE.md guide was extremely helpful in understanding the correct quantization approach for gemma-4-E2B-it.

At the time of writing, the original repository appears to be unavailable or removed. To ensure reproducibility, I have documented the full quantization workflow used for this model in my own repository.

The full quantization process used to produce these models is documented here: 📓 auto_round_Gemma4-E4B.ipynb

License

This quantized model is derived from google/gemma-4-E4B-it and is subject to the Gemma Terms of Use.

Citation

If you use this quantized model, please also cite the original Gemma 4 work:

@misc{gemma4_2026,
  title  = {Gemma 4},
  author = {Google DeepMind},
  year   = {2026},
  url    = {https://huggingface.co/google/gemma-4-E4B-it}
}

Quantized by Vishva007

Downloads last month: -; Downloads are not tracked for this model. How to track

Model tree for Vishva007/gemma-4-E4B-it-W4A16-AutoRound-AWQ

Base model

google/gemma-4-E4B

Finetuned

google/gemma-4-E4B-it