Instructions to use PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF
# Run inference directly in the terminal:
llama cli -hf PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF
# Run inference directly in the terminal:
llama cli -hf PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF
# Run inference directly in the terminal:
./llama-cli -hf PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF

Use Docker

docker model run hf.co/PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF

LM Studio
Jan
Ollama
How to use PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF with Ollama:
```
ollama run hf.co/PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF
```
Unsloth Desktop

How to use PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF with Docker Model Runner:
```
docker model run hf.co/PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF
```

Lemonade

How to use PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF

Run and chat with the model

lemonade run user.GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "PaoAI/GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP-GGUF

The pure-FP4 pool quant of GLM-5.3-Flash for AMD Strix Halo fleets. The whole model on the chip's native 4-bit format (159.2 GiB), designed to be split across two Strix Halo boxes over a direct link when one box isn't enough. The MTP draft head is preserved in the file — but note: on a 2-box pool, speculative decoding is OFF (pool round-trips eat the draft's gains; measured in our fleet).

Brand: PaoAI · community quant, not affiliated with Z.AI.

Who is this for?

You have two or more Strix Halo boxes connected directly (USB4/10GbE) and want the full GLM-5.3-Flash in native FP4 split across them. If you have ONE box, use our STRIX-BALANCED instead — a pool of this model loses to a solo smaller quant on both speed and simplicity.

The recipe

What	Format
Everything (attention, experts, draft head)	Q4_0_ROCMFP4_FAST (native FP4, 4.26 bpw)
Router, norms, glue	FP32 (keeper list)
MTP draft head (blk.45)	preserved (baked into FP4)

Measured on real hardware

Metric	Result
File size	159.2 GiB (single file)
2-box pool decode	~5.4 t/s (stable ×3, both boxes ~80GB each)
Solo on one box	❌ will wedge the gfx1151 driver (98GB+ allocation wall — see BALANCED/HYBRID for solo options)
Quality	not battery-scored (pool-only use case; solo score would not represent pool serving)

Build source (required)

Both binaries need the glm5next arch and the FP4 type. Both live in guevae2/ROCmFPX, branch glm5next, commit 3345156 (glm5next arch + FP4 type merged; head of that branch — also buildable from kingjones30/ROCmFPX main once glm5next is upstreamed). Build with GGML_VULKAN=ON on Strix Halo (gfx1151).

How to run (2-box pool, direct link)

# halo-2 (worker):
ggml-rpc-server --host 0.0.0.0 --port 50052
# BG1 (head):
llama-server -m GLM-5.3-Flash-PaoAI-ROCmFP4-STRIX-MTP.gguf \
  --host 0.0.0.0 --port 8081 -ngl 99 -c 32768 --parallel 1 \
  --rpc <worker-ip>:50052 -fa on

⚠️ Pool endpoint rule (learned the hard way): use the DIRECT box-to-box link addresses, never routes through a third machine — RPC traffic via a middle box deadlocks after load (head stuck in send, worker in recv).

Honest position

This is the largest and least convenient of our three models: it needs two boxes, delivers ~5.4 t/s, and can't run solo. We publish it because it completes the recipe set for fleet owners — and because the measurement behind it (pools reward lean bytes-per-token over small file size) is one of the core findings of our work.

Credits & license

Z.AI — GLM-5.3-Flash base model (MIT license; included in repo)
Unsloth — BF16 source + imatrix
kingjones30 / charlie12345 — ROCmFPX format & recipes
ggml-org — llama.cpp
PaoAI — glm5next port, FP4 recipe, fleet benchmarking

MIT, same as the base model. Not affiliated with Z.AI.

Downloads last month: 175

GGUF

Model size

321B params

Architecture

glm5next

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support