Instructions to use SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF
# Run inference directly in the terminal:
llama cli -hf SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF
# Run inference directly in the terminal:
llama cli -hf SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF
# Run inference directly in the terminal:
./build/bin/llama cli -hf SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF

Use Docker

docker model run hf.co/SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF

Llama new
LM Studio
Jan

vLLM

How to use SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF

Ollama
How to use SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF with Ollama:
```
ollama run hf.co/SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF
```
Unsloth Desktop

How to use SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF with Docker Model Runner:
```
docker model run hf.co/SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF
```

Lemonade

How to use SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF

Run and chat with the model

lemonade run user.GLM-5.3-Flash-RANMA-EXL3-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "SaturnHeaven/GLM-5.3-Flash-RANMA-EXL3-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

GLM-5.3-Flash RANMA EXL3 GGUF

Requires ranma.cpp ranma_20261007 or newer (github.com/saturnsky/ranma.cpp). These files use fork-specific tensor type numbers (256 and above). They will not open in upstream llama.cpp, LM Studio, Ollama or other llama.cpp-based tools.

No benchmark numbers are published for this model.

What this is

This repository repackages the EXL3 quantizations made by turboderp of zai-org/GLM-5.3-Flash into GGUF files that ranma.cpp can read.

The weights were not re-quantized. The EXL3 trellis-coded tensors are carried over bit for bit; they are only wrapped in the GGUF container (with the fork's EXL3 tensor types) and split into shards. The fused KDA qkv_proj (and its conv1d) of the checkpoint is stored as the separate q, k and v tensors that the GGUF graph reads, cut at 128-row boundaries without changing the trellis bytes. All credit for the quantization goes to turboderp and the ExLlamaV3 project. Thank you.

Sources (EXL3, by turboderp), one branch per quantization. Each source branch maps to one folder in this repository:

Source branch (turboderp)	Folder in this repo
`2.05bpw`	`EXL3-2.05bpw-h5`
`3.05bpw`	`EXL3-3.05bpw-h6`

Both were produced with ExLlamaV3 quantization version 1.4.4 (codebook mul1, calibration 250 rows x 2048). h5/h6 are the head bits (output layer).

The multi-token-prediction (MTP) layer of the checkpoint is included in the same file; see Running.

Not included

Part	Size in the source	Status
Vision encoder	0.6 GB	Not converted: the converter does not handle quantized vision towers yet. Image and video input is not available with these files; no mmproj file is provided.

In llama.cpp the vision encoder is a separate mmproj file, so it can be added later without changing the files here.

Files

Each quantization lives in its own folder. The first shard holds only metadata, the following shards hold the tensors (llama-gguf-split --split --split-max-size 50G --no-tensor-first-split). Pass only the first shard to ranma; the rest are found automatically. manifest.json lists the size and SHA-256 of every file.

Folder	bpw (head)	Size	Files
`EXL3-2.05bpw-h5`	2.05 (5)	84.7 GB (78.9 GiB)	3
`EXL3-3.05bpw-h6`	3.05 (6)	124.7 GB (116.1 GiB)	4

File names: GLM-5.3-Flash-EXL3-<bpw>-<head>-0000K-of-0000N.gguf.

Running

ranma.cpp keeps the routed experts of a MoE model in host RAM and reads them over PCIe from the GPU kernels. A VRAM expert cache (L1) holds the experts that the router picks most often, and a host-RAM tier (L2) bounds how much RAM the experts may use; whatever fits in neither is read from the GGUF file on the SSD. You choose L1 and L2 for your own machine. Start from the example below and adjust.

Needed in every process: the environment variables GGML_CUDA_HOST_DIRECT=1 and GGML_CUDA_HOST_DIRECT_MAX_BATCH=512 (host-direct expert reads, HIP only; there is no CLI flag), --load-mode none on the command line, and no -ncmoe/--cpu-moe (the cache budget decides the placement). The first shard is enough for -m; the other shards are found by name.

Option	Default	Meaning	How to choose
`--expert-l1-mib N`	0 (cache off)	VRAM budget in MiB of the expert cache, L1. It also decides which experts go to host memory.	VRAM - (dense weights + KV cache + compute buffers) - margin; see below.
`--expert-l2-mib N`	-1 (unlimited)	Host-RAM budget in MiB of the expert cache, L2. `-1`: every expert that is not in L1 lives in RAM. A finite value: experts that fit in neither L1 nor L2 stay in the GGUF file on the SSD and are read on demand (Windows + HIP).	The RAM you can spare for the experts, minus what your other programs need; the SSD serves the rest, at some cost in speed.
`--expert-cache-mode inclusive\|exclusive`	inclusive	`exclusive` keeps one copy of each expert (in VRAM or in RAM); `inclusive` keeps a RAM copy of every VRAM resident as well.	`exclusive`. Windows.
`--expert-profile-dir DIR`	none	Directory where the server stores which experts each request used. Without it the placement stays a fixed random one.	Any writable directory. The placement adapts at request boundaries; about ten requests rebuild a profile.
`--spec-type draft-mtp`	none	Speculative decoding with the MTP layer in this file. No `-md`: the MTP layer is read from the same GGUF, and its experts share the expert cache with the main model.	Optional. It needs about 1.5 GiB more VRAM, so lower L1 by that much.
`--spec-smart` (default on), `--spec-smart-store PATH`	on, none	Chooses the draft length at every step from measured times. The store file keeps the estimates across restarts.	With MTP: give a store path so a restart is not cold.
`-np N`	4 in the server	Number of parallel slots. Always give `-np 1` for this example: a finite L2 (`--expert-l2-mib` positive) is refused when the slot count is not fixed (`finite L2 needs positive batch/parallel bounds`).	`-np 1` unless you serve several users.
`--gpu-heartbeat-seconds S`	5.0 (server)	Pulses the GPU while the server is idle so that the OS does not evict VRAM.	Leave the default.

Sizing L1 and L2

L1. L1 = VRAM - (memory the cache does not cover) - margin. What the cache does not cover is the dense weights (about 5.5 GiB in VRAM for 3.05 bpw), the KV cache and the compute buffers. Most layers of this model are linear-attention (KDA) layers with a fixed-size state, so the KV cache is small: about 1.2 GiB at -c 65536 (f16). With L1 = 20480 at -c 65536 the dedicated VRAM peaked at about 28.6 GiB on a 32 GB R9700. Keep a margin of a few GiB, more if the GPU also drives a display. The L1 budget includes the cache's own tables and overhead, so it is not exactly the expert bytes.
L2. The routed experts take 71.3 GiB (2.05 bpw) and 106.7 GiB (3.05 bpw). In exclusive mode L1 and L2 hold different experts, so L1 + L2 at or above that size reads nothing from the SSD. The physical RAM in use is about L2 plus a few GiB; Windows reports a higher private memory, because it also counts the VRAM and the mapped host memory. The server refuses to load if the host requirement is above the physical memory installed, but it does not look at how much is free at that moment.

The command and values below are an example only. Pick them for your own use and system: the context length, the split between L1 and L2, whether to use MTP, and where the expert profile and the smart store are kept (--expert-profile-dir, --spec-smart-store; any writable location, one per model and workload works best).

Example command (64K context)

32 GB VRAM + 128 GB RAM:

set GGML_CUDA_HOST_DIRECT=1
set GGML_CUDA_HOST_DIRECT_MAX_BATCH=512
llama-server -m EXL3-3.05bpw-h6\GLM-5.3-Flash-EXL3-3.05bpw-h6-00001-of-00004.gguf ^
  -c 65536 -ngl 999 -fa on -b 512 -ub 512 -np 1 --load-mode none ^
  --expert-cache-mode exclusive --expert-l1-mib 20480 --expert-l2-mib 92160 ^
  --expert-profile-dir .\expert-profiles

For 2.05 bpw use EXL3-2.05bpw-h5\GLM-5.3-Flash-EXL3-2.05bpw-h5-00001-of-00003.gguf; its experts fit in L1 + L2 entirely, so a smaller L2 is enough. For MTP add --spec-type draft-mtp --spec-smart-store .\smart-store.json and lower L1 to --expert-l1-mib 18944.

The chat template of the original model is in the file. Its prompts start with [gMASK]<sop>; use the chat endpoint (or add that prefix yourself for raw completion).

Details of every option: expert-cache.md, expert-cache-exclusive.md, expert-cache-l2.md, expert-cache-joint.md, host-direct-moe.md, spec-smart.md, gpu-heartbeat.md, exl3.md.