Instructions to use Omega-Factory/granite-4.2-30b-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Omega-Factory/granite-4.2-30b-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use Omega-Factory/granite-4.2-30b-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Omega-Factory/granite-4.2-30b-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Omega-Factory/granite-4.2-30b-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M

Ollama
How to use Omega-Factory/granite-4.2-30b-GGUF with Ollama:
```
ollama run hf.co/Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M
```
Unsloth Desktop

How to use Omega-Factory/granite-4.2-30b-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Omega-Factory/granite-4.2-30b-GGUF with Docker Model Runner:
```
docker model run hf.co/Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M
```

Lemonade

How to use Omega-Factory/granite-4.2-30b-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.granite-4.2-30b-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use Omega-Factory/granite-4.2-30b-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Omega-Factory/granite-4.2-30b-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Omega-Factory/granite-4.2-30b-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Granite 4.2 30B GGUF Precision Ladder

Official Release & Multi-Quant Benchmarks:
View full precision comparison tables (Q8_0, Q4_K_M, Q2_K vs. BF16 GGUF Base), item-level McNemar discordance flips, and verified cryptographic receipts on the Omega Factory Granite 4.2 30B GGUF Precision Ladder Model Page.

Measured dense GGUF precision quantization ladder. Delivered artifacts support text generation and tool calling.

Measured quant ladder

Every rung below is reported on its own exact bytes (sha256-bound). A verdict is never transferred between formats or rungs, and a rung that did not clear its bands is published with its real verdict rather than omitted.

⚠️ Quality Warning: Quantization degradation detected for: Q2_K (DEGRADED). Review capability retention before deploying.

Rung	Verdict	Seeds	Size	SHA256	Receipt
`Q8_0`	SCREEN	1	31.11 GB	`5b14fe1d83c2`	`receipts/cert_Q8_0.json`
`Q4_K_M`	SCREEN	1	17.72 GB	`cd7fc4faec09`	`receipts/cert_Q4_K_M.json`
`Q2_K`	DEGRADED	1	10.86 GB	`db24b9204e91`	`receipts/cert_Q2_K.json`

SCREEN means the result did not use the full seed set and therefore is not a certification.

Delivered modality: text-only GGUF. Base-model vision support does not establish multimodal delivery.

BF16 GGUF Baseline vs. Quantization Precision Comparison

To evaluate quantization precision loss and capability retention, quant variants were benchmarked against the unquantized base model (ibm-granite/granite-4.2-30b) under controlled deterministic evaluation settings.

Precision & Accuracy Matrix vs. BF16 GGUF Baseline

Variant / Format	Size (Disk/VRAM)	ARC-MC Accuracy	GSM8K Accuracy	MATH-500 Accuracy	Babble Rate	Status
BF16 GGUF Baseline	58.56 GB	92.0% (184/200)	94.0% (188/200)	54.5% (109/200)	0.0017	Baseline Ground Truth
`Q8_0`	31.11 GB	92.0% (Δ +0.0 pp)	94.0% (Δ +0.0 pp)	54.5% (Δ +0.0 pp)	0.0017	SCREEN
`Q4_K_M`	17.72 GB	91.0% (Δ -1.0 pp)	92.5% (Δ -1.5 pp)	54.5% (Δ +0.0 pp)	0.0000	SCREEN
`Q2_K`	10.86 GB	86.5% (Δ -5.5 pp)	68.0% (Δ -26.0 pp)	38.0% (Δ -16.5 pp)	0.0000	DEGRADED

Detailed Item-Level Contingency & Flips vs. BF16 GGUF

Per-item McNemar discordance analysis measuring questions that changed answer status relative to the unquantized BF16 GGUF baseline:

Quantization Variant	Category	Quant Correct	BF16 GGUF Base Correct	Flips Broke (BF16 GGUF ✓ → Quant ✗)	Flips Fixed (BF16 GGUF ✗ → Quant ✓)	Net Delta (pp)
`Q8_0`	GSM8K	188 / 200	188 / 200	2	2	+0.0 pp
`Q8_0`	MATH-500	109 / 200	109 / 200	9	9	+0.0 pp
`Q8_0`	ARC-MC	184 / 200	184 / 200	0	0	+0.0 pp
`Q4_K_M`	GSM8K	185 / 200	188 / 200	7	4	-1.5 pp
`Q4_K_M`	MATH-500	109 / 200	109 / 200	16	16	+0.0 pp
`Q4_K_M`	ARC-MC	182 / 200	184 / 200	3	1	-1.0 pp
`Q2_K`	GSM8K	136 / 200	188 / 200	57	5	-26.0 pp
`Q2_K`	MATH-500	76 / 200	109 / 200	43	10	-16.5 pp
`Q2_K`	ARC-MC	173 / 200	184 / 200	16	5	-5.5 pp

Hardware & VRAM Allocation Recommendations

Artifact size is disk usage, not a measured VRAM requirement. Runtime memory and the KV cache depend on architecture, backend, batch settings, and context length. Validate actual deployment with peak-memory profiling.

Reference capacity bands only:

Artifact size (disk)	Reference capacity band
Up to 8 GB	6–8 GB
Over 8 GB through 16 GB	12–16 GB
Over 16 GB through 24 GB	20–24 GB
Over 24 GB through 48 GB	32–48 GB
Over 48 GB	Multi-GPU or CPU offload; no single listed tier

Quant Variant	Artifact Size (disk)	Deployment sizing notes
`Q8_0`	31.11 GB	32–48 GB reference capacity. SCREEN evaluation status; deployment is not recommended.
`Q4_K_M`	17.72 GB	20–24 GB reference capacity. SCREEN evaluation status; deployment is not recommended.
`Q2_K`	10.86 GB	12–16 GB reference capacity. Candidate only, not a measured fit guarantee.

Recommended Sampling Parameters

Measured greedy mode:
- temperature = 0.0
- top_k = 1
- seed = 42

Model Architecture Specifications

Model Type: dense
Parameters: 29.3 Billion
Layers: 64
Context Length: 131072
Attention: Grouped-Query Attention (32 Q heads, 8 KV heads)

Chat template

{%- macro tool_to_json(tool) -%}
    {%- set ns_tool = namespace(first=true) -%}
    {{- '{' -}}
    {%- for k, v in tool.items() -%}
        {%- if k != 'defer_loading' and k != 'strict' -%}
            {%- if not ns_tool.first -%}{{- ', ' -}}{%- endif -%}
            {%- set ns_tool.first = false -%}
            {{- '"' ~ k ~ '": ' ~ (v | tojson(ensure_ascii=False)) -}}
        {%- endif -%}
    {%- endfor -%}
    {{- '}' -}}
{%- endmacro -%}
{%- set enable_thinking = enable_thinking if enable_thinking is defined else True %}
{%- set low_effort = low_effort if low_effort is defined else False %}
{%- if reasoning_effort is defined and reasoning_effort is not none %}
    {%- set low_effort = reasoning_effort == "low" %}
{%- endif %}
{%- set truncate_history_thinking = truncate_history_thinking if truncate_history_thinking is defined else True %}

{%- set ns = namespace(last_user_idx = -1) %}
{%- set loop_messages = messages %}
{%- for m in loop_messages %}
  {%- if m["role"] == "user" %}
    {%- set ns.last_user_idx = loop.index0 %}
  {%- endif %}
{%- endfor %}

{%- if messages[0]["role"] == "system" %}
    {%- set system_message = messages[0]["content"] %}
    {%- set loop_messages = messages[1:] %}
{%- else %}
    {%- set system_message = "" %}
    {%- set loop_messages = messages %}
{%- endif %}
{%- if not tools is defined %}
    {%- set tools = [] %}
{%- endif %}
{# Recompute last_user_idx relative to loop_messages after handling system #}
{%- set ns = namespace(last_user_idx = -1) %}
{%- for m in loop_messages %}
  {%- if m["role"] == "user" %}
    {%- set ns.last_user_idx = loop.index0 %}
  {%- endif %}
{%- endfor %}
{%- if system_message is defined %}
    {{- "<|im_start|>system\n" + system_message }}
{%- else %}
    {%- if tools is iterable and tools | length > 0 %}
        {{- "<|im_start|>system\n" }}
    {%- endif %}
{%- endif %}
{%- if tools is iterable and tools | length > 0 %}
    {%- if system_message is defined and system_message | length > 0 %}
        {{- "\n\n" }}
    {%- endif %}
    {{- "# Tools\n\nYou have access to the following functions:\n\n" }}
    {{- "<tools>" }}
    {%- for tool in tools %}
        {%- if tool.function is defined %}
            {%- set tool = tool.function %}
        {%- endif %}
        {%- if tool.defer_loading is not defined or not tool.defer_loading %}
            {{- '\n' ~ tool_to_json(tool) }}
        {%- endif %}
    {%- endfor %}
    {{- "\n</tools>" }}

    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
{%- endif %}


{%- if system_message is defined %}
    {{- '<|im_end|>\n' }}
{%- else %}
    {%- if tools is iterable and tools | length > 0 %}
        {{- '<|im_end|>\n' }}
    {%- endif %}
{%- endif %}

{%- for message in loop_messages %}
    {%- if message.role == "assistant" %}
        {# Add reasoning content in to content field for unified processing below. #}
        {%- if message.reasoning_content is defined and message.reasoning_content is string and message.reasoning_content | trim | length > 0 %}
            {%- set content = "<think>\n" ~ message.reasoning_content ~ "\n</think>\n" ~ (message.content | default('', true)) %}
        {%- else %}
            {%- set content = message.content | default('', true) %}
            {%- if content is string -%}
                {# Allow downstream logic to to take care of broken thought, only handle coherent reasoning here. #}
                {%- if '<think>' not in content and '</think>' not in content -%}
                    {%- set content = "<think></think>" ~ content -%}
                {%- endif -%}
            {%- else -%}
                {%- set content = content -%}
            {%- endif -%}
        {%- endif %}
        {%- if message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %}
            {# Assistant message has tool calls. #}
            {{- '<|im_start|>assistant\n' }}
                {%- set include_content = not (truncate_history_thinking and loop.index0 < ns.last_user_idx) %}
                {%- if content is string and content | trim | length > 0 %}
                    {%- if include_content %}
                        {{- (content | trim) ~ '\n' -}}
                    {%- else %}
                        {%- set c = (content | string) %}
                        {%- if '</think>' in c %}
                            {# Keep only content after the last closing think. Also generation prompt causes this. #}
                            {%- set c = c.split('</think>')[-1] %}
                        {%- elif '<think>' in c %}
                            {# If <think> was opened but never closed, drop the trailing think segment #}
                            {%- set c = c.split('<think>')[0] %}
                        {%- endif %}
                        {%- set c = "<think></think>" ~ c | trim %}
                        {%- if c | length > 0 %}
                            {{- c ~ '\n' -}}
                        {%- endif %}
                    {%- endif %}
                {%- else %}
                    {{- "<think></think>" -}}
                {%- endif %}
                {%- for tool_call in message.tool_calls %}
                    {%- if tool_call.function is defined %}
                        {%- set tool_call = tool_call.function %}
                    {%- endif %}
                    {{- '<tool_call>\n<function=' ~ tool_call.name ~ '>\n' -}}
                        {%- if tool_call.arguments is defined %}
                            {%- for args_name, args_value in tool_call.arguments|items %}
                                {{- '<parameter=' ~ args_name ~ '>\n' -}}
                                    {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
                                {{- args_value ~ '\n</parameter>\n' -}}
                            {%- endfor %}
                        {%- endif %}
                    {{- '</function>\n</tool_call>\n' -}}
                {%- endfor %}
                {{- '<|im_end|>\n' }}
        {%- else %}
            {# Assistant message doesn't have tool calls. #}
            {%- if not (truncate_history_thinking and loop.index0 < ns.last_user_idx) %}
                {{- '<|im_start|>assistant\n' ~ (content | default('', true) | string | trim) ~ '<|im_end|>\n' }}
            {%- else %}
                {%- set c = (content | default('', true) | string) %}
                {%- if '<think>' in c and '</think>' in c %}
                    {%- set c = "<think></think>" ~ c.split('</think>')[-1] %}
                {%- endif %}
                {%- set c = c | trim %}
                {%- if c | length > 0 %}
                    {{- '<|im_start|>assistant\n' ~ c ~ '<|im_end|>\n' }}
                {%- else %}
                    {{- '<|im_start|>assistant\n<|im_end|>\n' }}
                {%- endif %}
            {%- endif %}
        {%- endif %}
    {%- elif message.role == "user" or message.role == "system" %}
        {{- '<|im_start|>' + message.role + '\n' }}
        {%- set content = message.content | string %}
        {%- if message.role == "user" and loop.index0 == ns.last_user_idx and low_effort %}
            {{- content + '\n\n{reasoning effort: low}' }}
        {%- else %}
            {{- content }}
        {%- endif %}
        {{- '<|im_end|>\n' }}
    {%- elif message.role == "tool" %}
        {%- if loop.previtem and loop.previtem.role != "tool" %}
            {{- '<|im_start|>user\n' }}
        {%- endif %}
        {{- '<tool_response>\n' }}
        {{- message.content }}
        {{- '\n</tool_response>\n' }}
        {%- if not loop.last and loop.nextitem.role != "tool" %}
            {{- '<|im_end|>\n' }}
        {%- elif loop.last %}
            {{- '<|im_end|>\n' }}
        {%- endif %}
    {%- else %}
        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>\n' }}
    {%- endif %}
{%- endfor %}

{%- if add_generation_prompt %}
    {%- if enable_thinking %}
        {{- '<|im_start|>assistant\n<think>\n' }}
    {%- else %}
        {{- '<|im_start|>assistant\n<think></think>' }}
    {%- endif %}
{%- endif %}

License and attribution

License: Apache-2.0 (inherited from the base model).
Base model: ibm-granite/granite-4.2-30b -- all credit for the underlying model belongs to its authors.
This repository publishes a compressed derivative: experts were pruned and the result re-quantised. It is not a new base model.
Commercial use is permitted under the terms of the license above.

Quantized GGUF preparation and evaluation card: Omega Factory. Base weights remain attributed to the upstream authors.

Contact / reporting an issue

Found a problem with these weights or this card? Open a discussion on this repository, or report a security concern to the maintainers via the organisation page at https://huggingface.co/Omega-Factory. We publish corrections and, where warranted, retract a release rather than leave a wrong card standing.

Official Model Page: https://omega-factory.ai/models/granite-4.2-30b-gguf-v1
Organization Website: https://omega-factory.ai

Downloads last month: 197

GGUF

Model size

29B params

Architecture

granite

Hardware compatibility

2-bit

4-bit

8-bit

Model tree for Omega-Factory/granite-4.2-30b-GGUF

Base model

ibm-granite/granite-4.1-30b-base

Finetuned

ibm-granite/granite-4.2-30b

Quantized

(41)

this model