Cactus-Compute/gemma-4-e2b-it-hybrid-GGUF

Cactus Hybrid — Gemma 4 E2B (GGUF)

A small, on-device model is fast and private, but sometimes wrong. At Cactus we post-train models to know when they are wrong: we ship probes inside the checkpoint that score every answer with a confidence between 0 and 1, returned as structured data (never parsed out of the answer text). Answer on-device when confidence is high; re-route to a bigger model when it's low:

if confidence < 0.85:
    answer = ask_a_bigger_model(prompt)

This repo holds GGUF builds of Cactus-Compute/gemma-4-e2b-it-hybrid for llama.cpp.

Benchmarks

Gemma 4 E2B Hybrid, the smallest Gemma model, matches Gemini 3.1 Flash-Lite on most benchmarks by routing only 15–35% of queries to Flash-Lite and running the rest itself:

Benchmark	Handoff to match Flash-Lite (FP16)	At 4-bit	At 3-bit
ChartQA	15–20%	25–30%	40–50%
MMBench	30–35%	40–45%	50–55%
LibriSpeech	25–30%	35–40%	55–65%
GigaSpeech	30–35%	40–45%	50–55%
MMAU	30–35%	35–40%	50–55%
MMLU-Pro	45–55%	~90%	n/a

Quantisation quality is measured on Cactus Quants, which performs well at uniform quantization; developers are encouraged to benchmark Unsloth, GGUF, and MLX quantization independently.

Quickstart

The gemma-4-e2b-it-hybrid architecture is not yet in upstream llama.cpp. Run these files with a build that includes the Cactus patch series — on unpatched llama.cpp they fail to load with "unknown model architecture" by design. Build the patched server once:

git clone https://github.com/cactus-compute/cactus-hybrid && cd cactus-hybrid
./patches/llama.cpp/install.sh && rehash   # clones the pinned tag, applies the patches, builds

Then serve and query it like any llama-server:

llama-server -hf Cactus-Compute/gemma-4-e2b-it-hybrid-GGUF:Q4_K_M --jinja

curl -s http://localhost:8080/v1/chat/completions \
  -d '{"messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":512}' \
  | jq '{answer: .choices[0].message.content, confidence}'

Chat-completions responses (and the final SSE chunk when streaming) carry a top-level "confidence" field.

Files

file	quant	size	notes
gemma-4-e2b-it-hybrid-f16.gguf	F16	9.31 GB	closest to the bf16 reference
gemma-4-e2b-it-hybrid-Q4_K_M.gguf	Q4_K_M	3.43 GB	recommended for consumer hardware

The probe head (11 probe.* tensors) is stored in F32 in all quants — only the trunk is quantized.

Calibration note

Quantized trunks shift the layer-28 activations the probe reads, moving confidences downward relative to the bf16 reference (measured mean drift: F16 −0.07, Q4_K_M −0.10; easy-vs-hard ordering fully preserved). If you use aggressive thresholds, calibrate per quant; the 0.85 default remains conservative (it hands off more, never less).

Routing quality (AUROC)

AUROC measures how well the probe separates wrong answers from right ones (higher = better, 0.5 is random, 1.0 is perfect):

Hold-out	Modality	Cactus Hybrid	Token Entropy
MMLU	text MCQ	0.770	0.697
MMLU-Pro	text MCQ	0.771	0.692
ARC-Easy	text MCQ	0.888	0.655
ARC-Challenge	text MCQ	0.834	0.646
GSM8K (3-shot)	text gen	0.782	0.731
MMBench-EN-Dev	vision MCQ	0.840	0.435
ChartQA	vision QA	0.779	0.615
DocVQA	vision QA	0.781	0.512
MMAU	audio MCQ	0.789	0.517
GigaSpeech	audio	0.876	0.343
Earnings-22	audio	0.839	0.323
LibriSpeech	audio	0.822	0.427
Mean		0.814	0.549

The strongest result: the probe was trained on zero audio data, yet achieves 0.79–0.88 AUROC on four audio benchmarks (two transcription, one audio MCQ, one out-of-domain transcription). This rules out surface-level explanations: the probe is reading a modality-independent correctness signal from the hidden state, not memorizing patterns from training data.

All formats

All Cactus Hybrid builds live in the Cactus Hybrid collection: Transformers · GGUF / llama.cpp · MLX · Cactus engine. Copy-paste quickstarts for every engine: github.com/cactus-compute/cactus-hybrid.

License

Gemma is provided under and subject to the Gemma Terms of Use. This derivative includes the Cactus handoff probe head.