Instructions to use SoAIHQ/gemma-4-31B-it-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use SoAIHQ/gemma-4-31B-it-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M

Use Docker

docker model run hf.co/SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use SoAIHQ/gemma-4-31B-it-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SoAIHQ/gemma-4-31B-it-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SoAIHQ/gemma-4-31B-it-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M

Ollama
How to use SoAIHQ/gemma-4-31B-it-GGUF with Ollama:
```
ollama run hf.co/SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M
```
Unsloth Desktop

How to use SoAIHQ/gemma-4-31B-it-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use SoAIHQ/gemma-4-31B-it-GGUF with Docker Model Runner:
```
docker model run hf.co/SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M
```

Lemonade

How to use SoAIHQ/gemma-4-31B-it-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.gemma-4-31B-it-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use SoAIHQ/gemma-4-31B-it-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use SoAIHQ/gemma-4-31B-it-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

gemma-4-31B-it · GGUF

The largest Gemma 4, a dense 31B model with image input and a 256K context.

GGUF builds of gemma-4-31B-it by Google for llama.cpp and SoAI, calibrated on real chat, code and tool calls.

At a glance

Text	Images	Audio	Reasoning	Tool calling	Context
Yes	Yes	No	Yes	Yes	256K

Specification	Details
Parameters	30.7B
Architecture	Dense
Context window	262,144 tokens
Reasoning	Configurable thinking, switched per request
Languages	35+ languages (pre-trained on 140+)
Made by	Google

Choosing a file

Download	Size	Quality	Best for
Q4_K_M	18.7 GB	High	Most machines, with the best quality for its size
Q8_0	32.6 GB	Near-lossless	Output closest to the original, when memory allows
mmproj (F16)	1.2 GB	Add-on	Needed only for image input

Q4_K_M suits most machines. It is about a third of the size of the original 16-bit weights and stays close to them in quality. If Q8_0 fits in your memory, it keeps output closest to the original. SoAI does not publish quants below 4 bits because their quality loss is rarely worth the saved space.

The model needs memory for the file and for the context, and the context part grows with the context length you set. llama.cpp can run the model on the GPU, on the CPU, or with its layers split between them.

Images need the mmproj-gemma-4-31B-it-f16.gguf add-on as well, loaded with --mmproj. llama-server -hf downloads it automatically.

Quick start

With llama.cpp, this command starts a local server with a built-in chat UI and an OpenAI-compatible API:

llama-server -hf SoAIHQ/gemma-4-31B-it-GGUF:Q4_K_M

Open http://localhost:8080 to chat, or send requests to the API:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role": "user", "content": "Explain GGUF in one sentence."}],
  "chat_template_kwargs": {"enable_thinking": true}
}'

Download the files to a folder first

hf download SoAIHQ/gemma-4-31B-it-GGUF --include "gemma-4-31B-it-Q4_K_M.gguf" "mmproj-*" --local-dir gemma-4-31B-it-GGUF
llama-server -m gemma-4-31B-it-GGUF/gemma-4-31B-it-Q4_K_M.gguf \
  --mmproj gemma-4-31B-it-GGUF/mmproj-gemma-4-31B-it-f16.gguf

Recommended settings

Mode	`temperature`	`top_p`	`top_k`
All tasks	1.0	0.95	64

Google recommends these values in the original model card. Google uses the same sampling settings for every task, with thinking on or off.

In llama-server, you set reasoning per request with "chat_template_kwargs": {"enable_thinking": true}.

How these files are made

Q4_K_M is quantized with an importance matrix (imatrix), which tells llama.cpp which weights affect the output most so it can store them with more precision. SoAI computes the imatrix from its own calibration corpus instead of generic web text, and formats every conversation in it with this model's chat template, including its native tool-call syntax. The model is therefore measured on the token patterns it sees in use: multi-turn chat in 21 languages, code edits in 22 programming languages, tool calls and their results, step-by-step math, and web prose.

Q8_0 is quantized without the imatrix, because llama.cpp's Q8_0 format does not use one.

Before any quantization, the build checks that the chat, thinking and tool-call markers are stored as special tokens and that the embedded chat template matches the original. A conversion that imports those markers as plain text breaks turns, thinking and tool calls without any error message, so the build stops instead of publishing the file.

The imatrix file is published in this repository, and the provenance table below pins the original revision and the llama.cpp commit, so you can rebuild these files yourself.

Calibration corpus sources

Source	Revision	License	Records	Used for
HuggingFaceFW/fineweb-2	`af9c13333eb9`	ODC-By 1.0	60	Multilingual web prose in the 21 SoAI UI languages
HuggingFaceFW/fineweb-edu	`87f09149ef47`	ODC-By 1.0	20	English educational prose
OpenAssistant/oasst2	`179dd21fc551`	Apache-2.0	70	Multi-turn, multilingual human conversations
CohereLabs/aya_dataset	`f9ea04583f02`	Apache-2.0	60	Human-written instructions and answers in SoAI UI languages
glaiveai/glaive-function-calling-v2	`8862406ff857`	Apache-2.0	60	Tool definitions, tool calls and tool results
bigcode/commitpackft	`fc56fe33c030`	MIT (dataset); permissively licensed source files	88	Code edits in 22 programming and config languages
openai/gsm8k	`740312add88f`	MIT	40	Step-by-step math reasoning

All files and SHA-256 checksums

File	Type	Size	SHA-256
gemma-4-31B-it-Q8_0.gguf	Q8_0	32.6 GB	`aed6d3f5657b785d…`
gemma-4-31B-it-Q4_K_M.gguf	Q4_K_M	18.7 GB	`6e5b6e6d35df85d4…`
mmproj-gemma-4-31B-it-f16.gguf	mmproj (F16)	1.2 GB	`5f52c97e30bb5d56…`
gemma-4-31B-it.imatrix	imatrix	13.7 MB	`5e84a508e008ff5a…`

The full checksums are in SHA256SUMS.

Provenance

Item	Details
Original model	google/gemma-4-31B-it by Google, revision `842da3794e`
Converted with	llama.cpp `v0.5.0`, commit `7fe450e19`
Importance matrix	SoAI calibration corpus v1, rendered with this model's chat template and computed on the Q8_0 weights
Weights	The original release, with no fine-tuning, merging or retraining. Only the storage format and numerical precision differ.
Built	2026-09-26

License and credits

These files are distributed under the original model's license (apache-2.0), and its full terms continue to apply. The license and the original model card also describe the model's capabilities, training data, benchmarks and limitations.

Google designed and trained the model, and SoAI only repackages the published weights. The conversion uses llama.cpp by Georgi Gerganov and the ggml contributors.

Run it with SoAI

Your models, your hardware, one software suite. Download SoAI for Windows, macOS, Linux or Android.

_{Packaged by SoAI.}

Downloads last month: -

GGUF

Model size

31B params

Architecture

gemma4

Hardware compatibility

4-bit

8-bit

Model tree for SoAIHQ/gemma-4-31B-it-GGUF

Base model

google/gemma-4-31B

Finetuned

google/gemma-4-31B-it

Quantized

(319)

this model