Instructions to use ByteOtter/gemma-4-12B-it-CADA-Q3_K_S with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ByteOtter/gemma-4-12B-it-CADA-Q3_K_S with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S
# Run inference directly in the terminal:
llama cli -hf ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S
# Run inference directly in the terminal:
llama cli -hf ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S
# Run inference directly in the terminal:
./llama-cli -hf ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S

Use Docker

docker model run hf.co/ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S

LM Studio
Jan
Ollama
How to use ByteOtter/gemma-4-12B-it-CADA-Q3_K_S with Ollama:
```
ollama run hf.co/ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S
```

Unsloth Studio

How to use ByteOtter/gemma-4-12B-it-CADA-Q3_K_S with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ByteOtter/gemma-4-12B-it-CADA-Q3_K_S to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ByteOtter/gemma-4-12B-it-CADA-Q3_K_S to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for ByteOtter/gemma-4-12B-it-CADA-Q3_K_S to start chatting

How to use ByteOtter/gemma-4-12B-it-CADA-Q3_K_S with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use ByteOtter/gemma-4-12B-it-CADA-Q3_K_S with Docker Model Runner:
```
docker model run hf.co/ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S
```

Lemonade

How to use ByteOtter/gemma-4-12B-it-CADA-Q3_K_S with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S

Run and chat with the model

lemonade run user.gemma-4-12B-it-CADA-Q3_K_S-Q3_K_S

List all available models

lemonade list

Hermes Agent

How to use ByteOtter/gemma-4-12B-it-CADA-Q3_K_S with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use ByteOtter/gemma-4-12B-it-CADA-Q3_K_S with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ByteOtter/gemma-4-12B-it-CADA-Q3_K_S:Q3_K_S" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

base_model: google/gemma-4-12B-it language: - multilingual pipeline_tag: text-generation tags: - gemma4 - gguf - llama.cpp - mixed-precision - task-aware-quantization - reasoning - q3-k-s

Gemma 4 12B IT — reasoning-directed mixed-precision Q3_K_S GGUF

This is an experimental, reasoning-directed mixed-precision GGUF derived from Google's instruction-tuned Gemma 4 12B model. QLAB produced the artifact from an Unsloth BF16 GGUF and selected it under a Q3_K_S-sized byte budget. MLAB then independently evaluated the exact provenance-stamped export against both ordinary stock Q3_K_S and stock Q3_K_S made with the same reasoning calibration matrix.

The principal independent result is a +18.750-point reasoning product effect over ordinary stock Q3_K_S, with a 95% bootstrap interval of +4.225 to +40.196 points on 128 paired items. The decomposition matters: the reasoning imatrix accounted for a measured +17.188 points, while the per-tensor allocation added +1.562 points over same-imatrix stock, with a 95% interval of -4.348 to +6.667. The independent evaluation therefore supports the complete artifact over ordinary stock, but does not isolate a statistically resolved allocation-only advantage.

Model details

Property	Value
Architecture	Gemma 4 Unified 12B, instruction-tuned
Format	GGUF v3
Quantization	Per-tensor mixed precision around a Q3_K_S byte budget
File size	5,534,805,216 bytes (5.155 GiB)
Embedded context length	262,144 tokens
Tensor count	667
Source artifact	`gemma-4-12b-it-BF16.gguf`
Source size	23,832,066,656 bytes (22.195 GiB)
Size reduction from source	76.78%
Exported SHA-256	`6691cfc4540dea356e19cee4c2658ff1e6ba3c08806aeb5396b7511fe844d01b`
Measured pre-stamp SHA-256	`393ec50dfcbef8609ecf0f697a9318cb431edfc50bcaa18dac2a1da1dd7aa6ae`
Tensor payload SHA-256	`8f3509abe06f10cfe01de0ea4b97aefe35b67cd18cccf99e9c20b2d638f4be37`
Source SHA-256	`5a5eefea73350705c6753105b725a51301d653e8d6646173db29a7e8da8e6efd`
Target category	Reasoning

The exported file is 288 bytes larger than the originally measured candidate because QLAB added provenance metadata. Its tensor payload is unchanged. MLAB evaluated the provenance-stamped export and verified its exported SHA-256, so the MLAB results below apply directly to this file.

The GGUF embeds the Gemma 4 chat template and the upstream sampling defaults: temperature 1.0, top-k 64, and top-p 0.95.

Tensor allocation

This is not a uniform Q3_K_S quantization. QLAB retained the stock-like type for most quantized tensors and reassigned 22 tensors under the matched-size budget.

Usage

Use a recent build of llama.cpp with Jinja template support:

llama-cli \
  -m candidate.allocation-win.gguf \
  --jinja \
  -cnv

Or start an OpenAI-compatible server:

llama-server \
  -m candidate.allocation-win.gguf \
  --jinja \
  --host 127.0.0.1 \
  --port 8080

This release was evaluated for text generation. Multimodal projectors are not included here, and image/audio behavior was not evaluated for this artifact.

Independent MLAB evaluation

MLAB evaluated four same-source artifacts:

The supplied BF16 source.
Ordinary stock Q3_K_S without an importance matrix.
Stock Q3_K_S made with the reasoning-category importance matrix.
This mixed-precision allocation, made with that same reasoning matrix.

This design separates the complete product effect from the imatrix effect and the incremental allocation effect.

Target-suite result

Artifact	Bytes	Reasoning score	Valid outputs
BF16 supplied source	23,832,066,656	75.000	128/128
Stock Q3_K_S, no imatrix	5,528,230,496	58.594	128/128
Stock Q3_K_S, reasoning imatrix	5,528,230,848	75.781	128/128
This mixed-precision allocation	5,534,805,216	77.344	128/128

All three quantized artifacts were inside the prespecified ±0.5% symmetric byte window. This artifact was 0.1189% larger than each stock comparator by the symmetric measure.

Reasoning comparison	Estimate	95% bootstrap interval	Paired items
Product: this model − ordinary stock	+18.750	+4.225 to +40.196	128
Imatrix: same-imatrix stock − ordinary stock	+17.188	+2.113 to +40.196	128
Allocation: this model − same-imatrix stock	+1.562	-4.348 to +6.667	128
Source reference: this model − BF16	+2.344	-13.462 to +24.359	128

The candidate's reasoning point estimate was 103.12% of the BF16 source score. Because the paired source-reference interval spans both negative and positive values, this is not evidence that the quantized model is superior to BF16.

Across the pooled coding, math, and reasoning suites, the candidate's secondary product effect over ordinary stock was +15.625 points with a 95% interval of +5.263 to +23.980 on 384 paired items.

Other measured suites

These are secondary diagnostics, not additional optimization targets.

Suite	BF16	Ordinary stock	Same-imatrix stock	This model	Product delta (95% interval)	Allocation delta (95% interval)
Coding	46.875	13.281	32.812	35.938	+22.656 (+15.625, +30.469)	+3.125 (-3.125, +9.375)
Math	45.312	28.906	35.156	34.375	+5.469 (-1.587, +12.500)	-0.781 (-5.217, +3.269)
Knowledge QA	71.875	64.062	71.875	73.438	+9.375 (-5.882, +27.273)	+1.562 (-4.545, +18.182)
Instruction following	35.938	21.875	23.438	25.000	+3.125 (-6.250, +12.500)	+1.562 (-4.688, +7.812)
Summarization / extraction	60.938	42.188	53.125	59.375	+17.188 (+6.250, +29.688)	+6.250 (0.000, +14.062)
Structured output	62.500	62.500	64.062	64.062	+1.562 (-3.125, +7.812)	0.000 (0.000, 0.000)
General fidelity	42.366	36.654	39.292	39.689	+3.035 (+2.801, +3.271)	+0.397 (+0.307, +0.487)
Coherence	100.000	100.000	100.000	100.000	0.000 (0.000, 0.000)	0.000 (0.000, 0.000)
Stability	56.871	43.352	51.895	47.975	+4.623 (-1.993, +11.687)	-3.920 (-13.225, +3.642)
Context	98.958	87.500	100.000	100.000	+12.500 (+2.083, +26.042)	0.000 (0.000, 0.000)

Runtime snapshot

MLAB ran eight position-balanced trials with full GPU offload, flash attention, 8 CPU threads, batch size 512, ubatch size 128, and a 65,536-token runtime context setting.

Artifact	Mean load	Mean prompt speed	Mean generation speed
BF16 source	23,235 ms	84.7 tok/s	20.6 tok/s
Ordinary stock Q3_K_S	43,459 ms	137.1 tok/s	44.7 tok/s
Same-imatrix stock Q3_K_S	29,634 ms	138.0 tok/s	45.3 tok/s
This model	30,217 ms	137.5 tok/s	44.7 tok/s

These measurements are specific to the test system, backend, prompt, and configuration. Peak VRAM was not measured, and the load-time differences should not be generalized to other storage or caching conditions.

Evaluation protocol

Evaluation pack: mlab-v1
MLAB run: v1-gemma4-12b-reasoning-20260812-r2
Bootstrap resamples: 10,000
Bootstrap seed: 5569898590874107905
MLAB report SHA-256: 6604d853510c7946358c63463aa9fff9d59c1787694eb340c24a93b5ad83565b
llama.cpp commit: 73618f27a801c0b8614ceaf3547d3c2a99baae14

Intervals use MLAB's paired, provenance-cluster bootstrap. MLAB reports measurements and availability states; it does not issue a model verdict.

Intended use

This artifact is intended for research and local experimentation with task-directed GGUF quantization where reasoning performance and a roughly 5.2 GiB language-model footprint are useful.

It should not be used as the sole source of truth in medical, legal, financial, safety-critical, or other high-stakes settings. Review generated content and evaluate the model on your own workload before deployment.

License and attribution

This model is licensed under the Apache License 2.0. See:

Gemma 4 is provided by Google DeepMind. The source BF16 GGUF identifies Unsloth as its quantizer. QLAB produced this mixed-precision allocation; MLAB performed the independent measurement described above.

Citation

@article{gemma4_2026,
  title   = {Gemma 4 Technical Report},
  author  = {{Gemma Team}},
  year    = {2026},
  journal = {arXiv preprint arXiv:2607.02770},
  url     = {https://arxiv.org/abs/2607.02770}
}

Support further QLAB experiments: buymeacoffee.com/byteotter

Downloads last month: 345

GGUF

Model size

12B params

Architecture

gemma4

Hardware compatibility

3-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Paper for ByteOtter/gemma-4-12B-it-CADA-Q3_K_S

Gemma 4 Technical Report

Paper • 2607.02770 • Published Jul 2 • 77