Instructions to use batiai/Hy3-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use batiai/Hy3-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="batiai/Hy3-GGUF",
	filename="Hy3-IQ3_XXS-00001-of-00003.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use batiai/Hy3-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf batiai/Hy3-GGUF:IQ3_XXS
# Run inference directly in the terminal:
llama cli -hf batiai/Hy3-GGUF:IQ3_XXS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf batiai/Hy3-GGUF:IQ3_XXS
# Run inference directly in the terminal:
llama cli -hf batiai/Hy3-GGUF:IQ3_XXS

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf batiai/Hy3-GGUF:IQ3_XXS
# Run inference directly in the terminal:
./llama-cli -hf batiai/Hy3-GGUF:IQ3_XXS

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf batiai/Hy3-GGUF:IQ3_XXS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf batiai/Hy3-GGUF:IQ3_XXS

Use Docker

docker model run hf.co/batiai/Hy3-GGUF:IQ3_XXS

LM Studio
Jan

vLLM

How to use batiai/Hy3-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "batiai/Hy3-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "batiai/Hy3-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/batiai/Hy3-GGUF:IQ3_XXS

Ollama
How to use batiai/Hy3-GGUF with Ollama:
```
ollama run hf.co/batiai/Hy3-GGUF:IQ3_XXS
```

Unsloth Studio

How to use batiai/Hy3-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for batiai/Hy3-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for batiai/Hy3-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for batiai/Hy3-GGUF to start chatting

How to use batiai/Hy3-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf batiai/Hy3-GGUF:IQ3_XXS

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "batiai/Hy3-GGUF:IQ3_XXS"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use batiai/Hy3-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf batiai/Hy3-GGUF:IQ3_XXS

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default batiai/Hy3-GGUF:IQ3_XXS

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use batiai/Hy3-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf batiai/Hy3-GGUF:IQ3_XXS

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "batiai/Hy3-GGUF:IQ3_XXS" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use batiai/Hy3-GGUF with Docker Model Runner:
```
docker model run hf.co/batiai/Hy3-GGUF:IQ3_XXS
```

Lemonade

How to use batiai/Hy3-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull batiai/Hy3-GGUF:IQ3_XXS

Run and chat with the model

lemonade run user.Hy3-GGUF-IQ3_XXS

List all available models

lemonade list

Hy3 (Hunyuan 3.0) GGUF — Quantized by BatiAI

A frontier coding & agent model that runs on your desk. Q4_K_M / IQ3_XXS GGUF of tencent/Hy3 (Hunyuan 3.0 — 295B total, 21B active). Quantized directly from official Tencent BF16 weights by BatiAI — code+multilingual‑calibrated imatrix, MTP‑pruned, BatiAI‑signed.

⚡ Why Hy3?

The smallest of the 2026 frontier MoEs — 295B that thinks like a giant but runs at 21B speed.

	Hy3	GLM‑5.2	DeepSeek‑V4
Total params	295B	753B	~1.6T
Active / token	21B	40B	~37B
Fits a 128GB Mac?	✅ (IQ3_XXS)	✗	✗

Benchmarks — competitive with models 2–5× its size:

SWE‑Bench Verified	SWE‑Bench Pro	GPQA Diamond	BrowseComp
78.0	57.9	90.4	84.2

_{Source: Tencent Hunyuan 3.0 official release (295B‑A21B base). These are base‑model (BF16) figures; the IQ3_XXS / Q4_K_M quants in this repo were not separately benchmarked, so expect some low‑bit degradation from these numbers.}

🛠️ Production‑grade tool‑calling — dedicated parsers, <4% variance across agent scaffolds. Built for agent pipelines.
🧠 256K context, 192 experts (top‑8) + shared expert, 80 layers, GQA, reasoning‑effort modes.
🔓 Apache 2.0 — and the official 3.0 release dropped the geo‑restriction (Korea / EU / UK now cleared). Commercial use, fine‑tune, redistribute freely.

📦 Quantizations

Quant	Size	Min RAM	Best for	Quality
Q4_K_M	166 GB (4 shards)	192 GB	256GB Mac Studio / server	⭐ Cleanest — recommended when RAM allows
IQ3_XXS	106 GB (3 shards)	128 GB	128GB Mac Studio	✅ Great — fits a 128GB Mac (raise the Metal wired limit; ~106 GiB of weights leaves modest context room)

Both are built directly from the official BF16, quantized with a diverse code + EN + KO + ZH imatrix, and have the MTP (multi‑token‑prediction) head pruned (--prune-layers 80) — the speculative head gives no benefit on Apple Metal and isn't imatrix‑covered, so a clean 80‑layer text model is the right target.

✅ Verified (this build). A captured greedy Q4_K_M run produced this exact, correct binary_search (verify log hy3-q4-verify.log shipped in this repo):

# prompt: def binary_search(arr, target):
    lo, hi = 0, len(arr) - 1
    while lo <= hi:
        mid = (lo + hi) // 2
        if arr[mid] == target:
            return mid
        elif arr[mid] < target: lo = mid + 1
        else: hi = mid - 1
    return -1

_{The captured run also appended a correct test harness, but with a Chinese code comment (# 测试) — exactly the low‑bit zh mixing flagged below. Logic was correct; use Q4_K_M for the cleanest output.}

⚠️ Positioning — read this. Hy3's strength is frontier coding / reasoning / agentic tool‑calling (EN/ZH). It is not a Korean‑specialized model (Tencent origin, no published Korean benchmark); lower‑bit quants can show occasional zh/en token mixing on Korean — use Q4_K_M for the cleanest Korean. For Korean‑first chat/STT on 16GB Macs, use batiai/qwen3.6‑27b. Hy3 is a frontier / high‑RAM tier model (like Kimi K2.6, GLM‑5.1, DeepSeek‑V4) — 128GB+ Apple Silicon or a workstation/server only.

🚀 Usage (llama.cpp)

⚙️ Build: Hy3 (hy_v3 arch) needs hy_v3 support — mainline merge pending (ggml‑org/llama.cpp#25395); build from that PR for now. Ollama support follows the mainline merge.

⚠️ Chat template: the stock Hy3 Jinja template uses .format() calls llama.cpp rejects. This repo ships a fixed template (Hy3-chat_template.jinja) — pass it with --jinja.

# 1) download — sharded GGUF (llama.cpp auto‑loads all shards from the first one)
#    128GB Mac → IQ3_XXS   |   256GB / server → Q4_K_M
hf download batiai/Hy3-GGUF \
  "Hy3-IQ3_XXS-*.gguf" Hy3-chat_template.jinja --local-dir ./hy3

# 2) chat (Apple Silicon Metal)
./llama-cli -m ./hy3/Hy3-IQ3_XXS-00001-of-00003.gguf -ngl 99 -c 8192 \
  --jinja --chat-template-file ./hy3/Hy3-chat_template.jinja \
  -p "Refactor this function and explain the change."

# raw completion (no chat template): add -no-cnv

Hy3-imatrix.dat (the calibration matrix used) is included for transparency / re‑quantization.

✨ What BatiAI did

Direct from official Tencent BF16 — never a re‑quant of someone else's GGUF.
Diverse imatrix (code + English + Korean + Chinese) for balanced multilingual + coding fidelity.
MTP head pruned + chat template fixed so it actually runs in llama.cpp.
Verified: load ✅ · coding ✅ · Korean ✅ · MoE routing ✅ — BatiAI metadata‑signed.

📜 License — Apache 2.0

Fully permissive: commercial use, modification, redistribution — no geographic restriction (Korea / EU / UK cleared in the official Hunyuan 3.0 release). Base model © Tencent; quantized weights redistributed under Apache 2.0.

🔗 Source & citation

Base: tencent/Hy3 (Hunyuan 3.0)
Quantized by: BatiAI · https://flow.bati.ai

@misc{batiai-hy3-gguf-2026,
  title  = {Hy3 (Hunyuan 3.0) GGUF — code+multilingual calibrated quantization},
  author = {BatiAI},
  year   = {2026},
  publisher = {Hugging Face},
  url    = {https://huggingface.co/batiai/Hy3-GGUF}
}

— BatiAI · on‑device frontier AI · https://flow.bati.ai

Downloads last month: 227

GGUF

Model size

295B params

Architecture

hy_v3

Hardware compatibility

3-bit

4-bit

Model tree for batiai/Hy3-GGUF

Base model

tencent/Hy3

Quantized

(60)

this model