Instructions to use SC117/MiniCPM5-2B-abliterated-FIT-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use SC117/MiniCPM5-2B-abliterated-FIT-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M

Use Docker

docker model run hf.co/SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use SC117/MiniCPM5-2B-abliterated-FIT-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SC117/MiniCPM5-2B-abliterated-FIT-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SC117/MiniCPM5-2B-abliterated-FIT-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M

Ollama
How to use SC117/MiniCPM5-2B-abliterated-FIT-GGUF with Ollama:
```
ollama run hf.co/SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M
```
Unsloth Desktop

How to use SC117/MiniCPM5-2B-abliterated-FIT-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use SC117/MiniCPM5-2B-abliterated-FIT-GGUF with Docker Model Runner:
```
docker model run hf.co/SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M
```

Lemonade

How to use SC117/MiniCPM5-2B-abliterated-FIT-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.MiniCPM5-2B-abliterated-FIT-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use SC117/MiniCPM5-2B-abliterated-FIT-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use SC117/MiniCPM5-2B-abliterated-FIT-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "SC117/MiniCPM5-2B-abliterated-FIT-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

FIT-GGUF v0.3.3GATE-VERIFIED TIERS4 TIERS · 1.14–1.46 GiBABLITERATED (TRIAL #68)MEASURED KL + SAME-TOPAPACHE-2.0

MiniCPM5-2B-abliterated · FIT-GGUF

Four fidelity tiers of a 128K-context 2.5B on-device model — every shipped file re-verified against its own BF16, byte-exact and reproducible.

1.14 GiBverified minimum at each fidelity1.46 GiB

English · 简体中文 📖

🧭 About FIT-GGUF — the tool behind these files

Every file in this repository was planned, executed and verified by FIT-GGUF, an open-source, deterministic tensor-level planning layer on top of llama.cpp quantization. Standard GGUF quantization asks you to pick one of a handful of presets; FIT-GGUF instead asks what quality do you want, then finds and verifies the smallest GGUF that demonstrably meets it.

Traditional GGUF gives you presets. FIT gives you a fidelity contract: macro KL ≤ tier anchor ∧ same-top ≥ model-calibrated floor.

Deterministic size prediction & byte-exact delivery	✅ Validated (G2 gate, delta = 0)
Universally optimal tensor allocation	⚠️ Not established — FIT claims verified fidelity contracts, not a universal quality optimum

Method, pre-registered research record and the fit CLI are all open source: github.com/Scorp1o117/FIT-GGUF

⚠️ Safety notice / 安全提示

The source model is an abliterated, refusal-removed model with no meaningful built-in guardrails, and may comply with harmful, illegal or unsafe requests. Use it only where you can provide appropriate moderation, access control and legal review. Do not deploy it to end users without your own safety layer.

源模型经过拒答方向消融，不具备可靠的内置安全护栏；请仅在合法、受控、具备审核与滥用防护的环境中使用，使用者自行承担部署责任。

🧬 Abliteration — trial #68, documented

The refusal-direction ablation was performed locally with the abliterix pipeline: mean-LoRA steering — rank-3 full-norm LoRA adapters built from per-layer mean refusal directions, with projected abliteration, a linear decay kernel over depth, winsorized direction vectors (q = 0.995), and per-layer vector scope. Only attn.o_proj and mlp.down_proj are steered; attn.q/k/v are disabled. The exported weights are Optuna trial #68 of an 80-trial study (checkpoints_minicpm5_2b_full), selected from the measured Pareto front.

Metric	Trial #68
Refusals (100 held-out harmful prompts)	7
3-token full-distribution KL vs original (nats/token)	0.0628
Validation KL, 100 held-out prompts (mean / p95)	0.0633 / 0.1959
Top-1 disagreement rate	3.0%
Generation health (repetition / collapse check)	PASSED

Why #68. It is the lowest-perturbation member of the best refusal bucket: every trial that reaches ≤7 refusals costs at least KL 0.0628, and the two trials that reach 6 refusals sit at KL 0.227 and 0.255 — 3.6× and 4× the damage. #68 is the front's knee, not a hand-picked favourite.

📦 Pick a tier

File size ≠ RAM/VRAM usage. KV cache, compute buffers and runtime overhead are separate — at the full 128K context a 2.5B model still needs several GiB of KV. Pick a sane -c. Naming: minicpm5-2b-abliterated-FIT-<tier>-<size>GiB-<type>.gguf, where <type> is always a nameable GGUF preset — the native preset when the recipe overrides no tensor, the element-weighted dominant type when that is itself a preset name, and the recipe's base preset otherwise. The three bare tensor types (Q4_K, Q3_K, Q5_K) are deliberately never used as suffixes: llama.cpp only ships the _S/_M/_L variants, and a filename with no recognisable quantisation token is dropped from Hugging Face's own variant panel.

Tier	GiB	Recipe	Macro KL ↓	Same-top ↑	Gates (KL ≤ / top ≥)	Measured positioning
QUALITY	1.459	FIT recipe on Q4_K_M (67 tensor overrides)	0.0493	89.90%	0.05 / 89.52%	At Q4_K_M's size, slightly cleaner than it (0.0526)
BALANCED	1.283	FIT recipe, dominant IQ4_XS (174 tensor overrides over Q3_K_M)	0.0929	86.27%	0.10 / 85.48%	Fills the Q3_K_M (0.163) → IQ4_XS (0.068) gap
COMPACT	1.212	FIT recipe on Q3_K_M (94 tensor overrides)	0.1495	83.02%	0.15 / 82.42%	Same size class as Q3_K_M, markedly cleaner
MINI	1.142	native IQ3_M preset (no tensor overrides)	0.1855	81.13%	0.20 / 79.55%	Native presets win at this size; MINI is the verified one

The four tiers are all KL-bound — KL, not the Same-top floor, is what separates PASS from FAIL on this model, so the recorded margins are the KL ones above.

Also shipped	GiB	Macro KL ↓	Same-top ↑	Why it is here
Q8_0	2.496	0.0013	98.25%	Highest fidelity that is still a quantized file
Q6_K	1.928	0.0079	95.54%	Best quality-per-byte on this model's ladder
BF16	4.693	0	100%	The reference every KL / Same-top number here is measured against
imatrix	0.003	—	—	500×512 chunks of `APEX-imatrix-Small.txt`, so you can re-quantize this model yourself

Read the curve, not just the tiers. On a 2.5B model the fidelity tiers are deliberately capped at KL 0.05–0.20, so the two big native presets are cleaner than every tier — Q6_K at 1.93 GiB is 6× cleaner than QUALITY at 1.46 GiB. If you want maximum fidelity rather than minimum size, take Q8_0 or Q6_K.

📈 Measured quality

Macro KL against main GGUF size, with every native preset and the four shipped tiers labelled

Quality improves monotonically across the healthy preset ladder. The 2-bit region is reported as measured: IQ2_XXS / IQ2_XS / IQ2_M / IQ3_XXS collapse (macro KL 3.92 / 1.93 / 0.81 / 0.40) and are excluded by design. COMPACT and BALANCED fill the Q3_K_M → IQ4_XS gap where no native preset exists; MINI is the native IQ3_M preset, which is why its diamond and its tier label sit on the same point.

Same-top is the share of positions where the quantized model and the aligned BF16 reference pick the same highest-logit token. On this model it tracks KL closely and never becomes the binding gate in the tier range.

Measurement basis, stated plainly. These charts plot the numbers recorded by the calibration and delivery run. In that run the 12-preset ladder was evaluated on CPU, while the four shipped tiers were verified on CUDA — the calibration was started before a CUDA-runtime-loading fix and the tier searches ran after it, and fit calibrate resumes by skipping already-recorded presets, so both measurement eras ended up in one bundle. Re-measuring the shipped files on CUDA reproduces their verdicts with modestly different numbers (macro KL 0.0493 / 0.0918 / 0.1484 / 0.1831 for QUALITY / BALANCED / COMPACT / MINI). Every one of the 20 per-domain pairs moved the same way — the CPU-basis figures are slightly conservative (higher KL); the largest difference is MINI · Chinese, 0.1723 → 0.1689 (−2.0%), and no domain got worse. All four tiers PASS either way. Within one backend the evaluation is bit-reproducible: a fresh CUDA re-run returned the delivery values to the last digit.

Full-size EN · 中文大图 · raw results JSON

🚀 Run it

Good news: this is plain llama architecture. Unlike models that need a patched llama.cpp, these files load in any reasonably recent upstream build — no PR branch, no custom fork. The measurements here used b10690.

llama.cpp

./llama-server \ -m minicpm5-2b-abliterated-FIT-BALANCED-1.28GiB-IQ4_XS.gguf \ -ngl 99 -c 8192

./llama-cli
-m minicpm5-2b-abliterated-FIT-QUALITY-1.46GiB-Q4_K_M.gguf
-ngl 99 -c 4096 -p "Hello" -n 256

The chat template is embedded in every file — including the tool-calling and reasoning-content handling — so LM Studio, KoboldCpp, Jan and friends load them without extra configuration. This is a dense 2.5B text model: no vision projector, no MTP head, no MoE. The full 131,072-token context is available, but KV cache grows with it; start at -c 8192 and work up.

🔬 Evaluation protocol & honest scope

Runtime	llama.cpp b10690 · Windows x86_64 · CUDA 13.3 (RTX 5090-class)
Command shape	`llama-perplexity -ngl 99 -t 16 -c 512 -b 512 --kl-divergence …`
Reference	This model's own BF16 logits (the BF16 GGUF shipped here)
Domains	wiki_test · wiki_valid · Chinese · code · agent_chat (five fixed 64 KiB slices, macro mean)
Calibration	12-point standard preset ladder + 10 gap probes; guard profile `guard-minicpm5-2b-abliterated-exact-v1`, scope `exact_model`, floors by `empirical_p5` (n = 3)
Weights binding	`source_weights_sha256 = f161afcf…c6e12` — the registry entry is keyed to these exact weights

Tier verification: a tier ships only if macro KL ≤ its anchor and Same-top ≥ a model-calibrated guard floor. Floors are exact_model-scoped and are deliberately not transferable to other models. Each shipped file was re-evaluated on its own bytes before delivery (G2 delta +0 on all four — the delivered byte count equals the re-finalized prediction exactly).

Allocator scope: the stock balanced v0.3 policy with the APEX-imatrix-Small importance matrix; no model-specific refine profile, because this model did not get one. What is claimed is deterministic size planning plus measured, double-gate verification of these specific artifacts — not a universally optimal allocation.

Sizes in the file names are measured bytes, not budgets. The tool refuses to emit an artifact whose size it cannot predict to the byte.

🧩 Included — and not included

✅ 4 gate-verified tier GGUFs · ✅ BF16 reference GGUF (the exact weights every measurement above is taken against) · ✅ Q6_K and Q8_0 native presets · ✅ the importance matrix used for quantization · ✅ chat template embedded in every file · ✅ checksums (SHA256SUMS.txt) · ✅ labelled quality curves (results/) · ✅ tier manifest (FIT-TIERS.md) · ✅ the exact numbers as JSON (results/four-tier-results.json)

❌ 2-bit quantization classes — measured collapse on this model (macro KL 0.40–3.92), excluded by design · ❌ no vision projector / no MTP head (dense text model) · ❌ no GGUF above Q8_0; take the BF16 file if you need the exact weights

The abliteration (trial #68) was performed locally from openbmb/MiniCPM5-2B; this repository contributes the abliterated weights, the FIT quantization plans and the artifacts.

🔍 Verify & reproduce

sha256sum -c SHA256SUMS.txt

The evaluation slices are public in the FIT-GGUF repository; the calibration ladder, gap probes, search audit and guard profile for this release are retained in the FIT-GGUF experiment record experiments/2026-09-10-minicpm5-2b-4tier/. The five-domain .kld reference logits are not stored there (9.6 GiB) — they regenerate from the BF16 GGUF shipped here and are checked against reference-manifest.json, which pins each domain's hash and byte count. Because the importance-matrix path is embedded in the GGUF metadata, the byte count of a re-quantization depends on where you put the imatrix file — the weights do not. Exact-size behaviour is scoped to the recorded source metadata and the recorded llama.cpp build; changing the converter, runtime, source layout or metadata requires revalidation.

📄 License & credits

Apache-2.0, inherited from the base model — follow the upstream license and model-card requirements.

OpenBMB — the MiniCPM5 family · llama.cpp — quantization and the KL/perplexity evaluator · abliterix — the local ablation pipeline (trial #68) · FIT-GGUF — verified size-exact quantization. FIT-GGUF is an independent project, not affiliated with OpenBMB or llama.cpp.

⭐ FIT-GGUF on GitHub — the tool, the method, the full research record

English · 简体中文