gemma-4-E2B-GGUF

GGUF-format, Q6_K quantization of Google's google/gemma-4-E2B base (non-instruction-tuned) checkpoint, for use with llama.cpp.

This is not an official Google release. "Gemma" is a trademark of Google LLC; this repository is not affiliated with, endorsed by, or sponsored by Google.

Files

File	Quantization	Size	SHA-256
gemma-4-E2B.Q6_K.gguf	Q6_K, importance-weighted	3.85 GB	ee715622018952e81a01308bc1307b55e142f60748a19f5fd7c509abe5e8f73c
imatrix.gguf	The importance matrix the Q6_K file was built with	2.8 MB	c5ac5efb6c8b6b6d5d3bbb3d861ab24db7016e785be192648d2cb66f36264edd

With imatrix.gguf and the same llama.cpp builds (below), the Q6_K file can be rebuilt byte for byte from a bf16 conversion of google/gemma-4-E2B:

llama-quantize --imatrix imatrix.gguf gemma-4-E2B-bf16.gguf gemma-4-E2B.Q6_K.gguf Q6_K

What was changed from upstream

Converted from google/gemma-4-E2B to GGUF (bf16) with llama.cpp's convert_hf_to_gguf.py (build b11115), then quantized to Q6_K with llama-quantize, using an importance matrix from llama-imatrix (both build b10210). The importance matrix was computed from about 250 KB of English, Polish and German text: random Wikipedia articles and excerpts from public-domain novels. That text is not included here.

The last 20 layers reuse the keys and values of earlier layers, so their own key and value projections never run and receive no importance data. llama-quantize reports "did not find weights" for them and quantizes them plainly. They are unused at inference, so this does not affect output.

The audio and vision encoders are not included; this is the text model only. No other modification was made. Architecture, tokenizer, and generation config are unchanged from upstream.

Why Q6_K

Measured against the bf16 conversion on about 22,000 tokens of held-out text in 18 languages (llama-perplexity --kl-divergence):

	Q4_K_M	Q5_K_M	Q6_K
Mean KL divergence	0.036	0.011	0.004
Same top token	90.0%	94.5%	96.6%
Perplexity vs bf16	+4.1%	+2.0%	+0.9%

On a text-continuation task over 8,769 positions in 18 languages, Q6_K scored the same as bf16, while Q4_K_M scored measurably lower.

License

Distributed under the Apache License 2.0 (see LICENSE in this repository), matching the license Google publishes google/gemma-4-E2B under.

Google additionally applies use restrictions to the Gemma family that are not waived by the Apache License tag and continue to apply to these weights regardless of who redistributes them:

• Gemma Terms of Use
• Gemma Prohibited Use Policy
• Gemma Intended Use Statement

By downloading or using this checkpoint you agree to comply with the Prohibited Use Policy.

See NOTICE for the upstream attribution this repository carries forward.

Usage

brew install llama.cpp

llama-cli -hf Animateus/gemma-4-E2B-GGUF:Q6_K -p "The quick brown fox" -n 32

This is a base (non-instruction-tuned) checkpoint. Prompt it with raw text to continue, not with chat-formatted instructions.