Instructions to use CQSystems/Google-Gemma-4-31B-It-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use CQSystems/Google-Gemma-4-31B-It-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0
# Run inference directly in the terminal:
llama cli -hf CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0
# Run inference directly in the terminal:
llama cli -hf CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0
# Run inference directly in the terminal:
./llama-cli -hf CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0

Use Docker

docker model run hf.co/CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0

Llama new
LM Studio
Jan
Ollama
How to use CQSystems/Google-Gemma-4-31B-It-GGUF with Ollama:
```
ollama run hf.co/CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0
```
Unsloth Desktop

How to use CQSystems/Google-Gemma-4-31B-It-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use CQSystems/Google-Gemma-4-31B-It-GGUF with Docker Model Runner:
```
docker model run hf.co/CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0
```

Lemonade

How to use CQSystems/Google-Gemma-4-31B-It-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0

Run and chat with the model

lemonade run user.Google-Gemma-4-31B-It-GGUF-Q8_0

List all available models

lemonade list

Hermes Agent

How to use CQSystems/Google-Gemma-4-31B-It-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use CQSystems/Google-Gemma-4-31B-It-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "CQSystems/Google-Gemma-4-31B-It-GGUF:Q8_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

This repository contains quantized GGUF formats of the Google Gemma 4 31B Instruction-Tuned model, published and heavily optimized by CQ Systems.

⚡ Mac Neo & Apple Silicon Compatibility

These files are optimized for local inference on professional hardware. Due to the 31-billion parameter size of this model, it is recommended to run these on machines with substantial unified memory (such as high-end MacBook Pros, Mac Studios, or Mac Pros).

By utilizing Apple Silicon's advanced unified memory matrix and native processing capabilities via Metal, these quantizations deliver fast, on-device reasoning and frontier-level cognition without relying on cloud infrastructure.

📦 Available Files

We provide multiple versions of the model, allowing you to scale based on your available unified memory (RAM/VRAM) constraints and latency requirements.

Filename	Size	Quantization	Recommendation / Notes
CQ-Google-Gemma-4-31B-f16-Q4_K.gguf	18.7 GB	Q4_K	🏆 Recommended. Excellent balance of reasoning quality and speed. Requires a machine with at least 32GB of total unified memory to run comfortably with context.
CQ-Gemma-4-31B-q8_0.gguf	32.6 GB	Q8_0	High-fidelity 8-bit integer quantization. Near-uncompressed reasoning quality. Requires 64GB+ of unified memory.
CQ-Google-Gemma-4-31B-f16.gguf	61.4 GB	F16	Baseline uncompressed 16-bit float. Highest absolute fidelity but immense memory footprint. Useful for research, fine-tuning, or requantizing. Requires 96GB+ RAM.

🚀 How to Use

Using llama.cpp

You can run this model directly in your terminal using the compiled llama.cpp CLI tool. Ensure you have built llama.cpp with Metal support for Macs.

# Basic chat launch with the recommended Q4_K model
./llama-cli -m CQ-Google-Gemma-4-31B-f16-Q4_K.gguf \
-c 8192 \
-n 1024 \
-p "<|think|>\nAnalyze the system architecture.\n<|channel>thought\n"

Using LM Studio / Ollama

Download the .gguf file matching your system's capabilities.
LM Studio: Drag and drop the downloaded file into your LM Studio model folder, or use the local import feature.
Ollama: Create a Modelfile with the line FROM ./CQ-Google-Gemma-4-31B-f16-Q4_K.gguf, then run ollama create cq-gemma-31b -f Modelfile.

(Note: While the base Gemma 4 31B supports audio and image inputs natively, standard GGUF text-generation primarily supports text out-of-the-box. Multimodal capabilities in GGUF require an accompanying mmproj encoder file).

🧠 About the Base Model: Gemma 4 31B

Gemma 4 is a family of open models built by Google DeepMind. The 31B variant is the flagship heavyweight model in the family, designed to handle highly complex reasoning, advanced coding, and nuanced logic comparable to frontier closed-source models.

Total Parameters: 31B Dense
Context Length: 128K tokens
Supported Modalities (Base Model): Text, Image, Audio
System Prompts: Gemma 4 uses standard system, assistant, and user roles natively.

Thinking Mode Configuration

Gemma 4 models are designed as highly capable reasoners. To enable the built-in reasoning capabilities, include the <|think|> token at the start of your system prompt.

When thinking is enabled, the model will output its internal reasoning followed by the final answer using this structural formatting:

<|channel>thought\n[Internal reasoning]<channel|>[Final answer]

📜 License & Acknowledgements

Base Model: Google DeepMind
License: Gemma License
Quantization & Optimization: CQ Systems Team

For full benchmark details, evaluations, ethical considerations, and safety metrics, please refer to the original Google Gemma 4 Documentation.

Downloads last month: 95

GGUF

Model size

31B params

Architecture

gemma4

Hardware compatibility

8-bit

16-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for CQSystems/Google-Gemma-4-31B-It-GGUF

Base model

google/gemma-4-31B

Finetuned

google/gemma-4-31B-it

Quantized

(346)

this model