OsaurusAI/MiniCPM5-2B-JANG_8M

MiniCPM5-2B — JANG_8M

8-bit MLX quantization of openbmb/MiniCPM5-2B, built with the full calibrated pipeline: imatrix + AWQ + GPTQ on 1.85M calibration tokens.

2.49 GiB · 8.501 bits/weight · median KL 0.0055 bits vs bf16 · 97.25% top-1 agreement

Loads with stock mlx_lm — no custom model code, no trust_remote_code.

---

JANG_8M vs stock MLX 8-bit

Both quantized from the same bf16 source at the same width (affine 8-bit, group_size 64), measured on the same held-out prompts on an M5 Max.

	JANG_8M	MLX 8-bit (mlx_lm.convert -q)	bf16 source
size	2.49 GiB	2.49 GiB	4.69 GiB
bits/weight	8.501	8.501	16
median KL vs bf16	0.005533 bits	0.010297 bits	—
mean KL vs bf16	0.013609 bits	0.021629 bits	—
top-1 agreement	97.25%	96.66%	—
decode	147.9 tok/s	147.8 tok/s	93.9 tok/s
prefill	9,124 tok/s	9,123 tok/s	10,447 tok/s
misaligned tensors	0 / 973	296 / 973	—
tool-call parser	included	none (silently unparsed)	—

At identical size and identical speed, JANG_8M is 1.86× closer to the source distribution. That gap is what the calibration buys — it is not a size-versus-quality trade.

Speed is a tie with stock MLX, as it should be: same width, same group size, same kernels. The 57% decode gain over bf16 is the quantization itself.

Alignment

Stock mlx_lm.convert leaves 296 of 973 tensor payloads at offsets that do not match their dtype's natural alignment. MLX/Metal cannot expose those as zero-copy typed buffers and allocates a resident aligned copy of each at load. Every shard here is written aligned — 0 misaligned — verified as a hard gate before publishing.

---

How it was built

One calibration sweep over 1,849,995 tokens (1.68M prompt + 173K of the model's own greedy generations, so its thinking-mode distribution is in-calibration) drawn from a coding-weighted domain mix — coding 35%, agentic 20%, academic 15%, general 12%, Chinese 10%, long-context 4%, science 2%, security 2% — rendered through the model's real chat template across the thinking, non-thinking and tool-framed presets.

That one pass produces all three calibration signals:

	what it is	where it goes
imatrix	E[x_c²] per input channel	activation-weighted error metric
Hessian	H = E[x xᵀ], full, per activation group	GPTQ
AWQ	max|x| per input channel	fold into the producing RMSNorm

• AWQ (α=0.25) folds into input_layernorm → q/k/v and post_attention_layernorm → gate/up — 84/84 fold sites, each proven function-preserving and proven to have actually moved the norm. o_proj and down_proj are deliberately not folded: o_proj's input channels are 4× tiles of v-rows under 16q/2kv GQA, and down_proj's input is the SwiGLU product whose gate side is nonlinear.
• GPTQ error-compensated rounding onto the fixed grid, float64 factorization with escalating damping, per-tensor best-of-RTN guard. Shipped on 295/296 tensors; embed_tokens stays RTN by design (a lookup table has no input Hessian). The down_proj Hessian is overdetermined 301× over its 6144 input dimension.
• bfloat16 scales. MLX computes affine quantized_matmul at promote(x.dtype, result_type(scales, biases)). This is a bf16 model, so float16 scales would promote every matmul to float32 and disqualify the fused bf16 kernels downstream. Verified: logits come back bfloat16.

Margin-conditioned flip curve

Where the quantization disagrees with bf16, conditioned on how decided the source model was (top1 − top2 logit margin):

source margin	tokens	top-1 flips
0.0 – 0.5	3,179	15.10%
0.5 – 1.0	2,738	1.94%
1.0 – 2.0	3,754	0.27%
2.0 – 4.0	4,353	0.16%
4.0 – 8.0	3,916	0.05%
8.0+	2,175	0.09%

A 300× drop from the undecided bin to the confident ones: the disagreements land where the source model was already a coin flip, not where it was sure. (The last two bins are 2 flips each — the 0.05→0.09% step is counting noise, not a rise.)

---

Usage

from mlx_lm import load, generate

model, tokenizer = load("OsaurusAI/MiniCPM5-2B-JANG_8M")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is 84 * 3 / 2?"}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512, verbose=True))

Sampling

The source model card's recommendation, declared in both generation_config.json and jang_config.json:

temperature = 1.0
top_p       = 0.95

Reasoning

The chat template takes enable_thinking:

value	behavior
omitted	the model decides
True	pre-opens <think> — always reasons
False	pre-closes <think></think> — answers directly

prompt = tokenizer.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=True)

Tool calling — read this

MiniCPM5 emits XML tool calls, not JSON:

<function name="get_weather"><param name="city">Seoul</param></function>

Values containing <, & or newlines are wrapped in <![CDATA[...]]>.

⚠️ mlx_lm cannot infer a parser for this dialect. Its _infer_tool_parser matches this template against none of its built-in literals, returns None, and every tool call is passed through as raw assistant text with no error. A parser is required, and one ships with this bundle's tooling:

from jang_tools.minicpm5.tool_parser import install, parse_tool_calls
install()                       # must run BEFORE mlx_lm.load()
model, tokenizer = load("OsaurusAI/MiniCPM5-2B-JANG_8M")

Or parse directly:

parse_tool_calls(output, tools)
# [{'name': 'get_weather', 'arguments': {'city': 'Seoul'}}]

jang_config.json → tool_calling carries the dialect, the delimiters and the parser name so a non-Python runtime can wire its own.

---

Verified before publishing

gate	result
shard alignment	0 / 973 misaligned
per-module quantization entries	296 / 296, keyed by live module path
scale dtype	bfloat16 (no float32 promotion)
sampling contract	generation_config.json ≡ jang_config.json
token ids	bos 0, eos [1, 130073], pad 1 — config ≡ generation_config
reasoning template	on / off / unset render distinctly
eos stop	terminates on its own
thinking off	direct correct answer, no trace
thinking on	reasoning trace + correct answer
tool call	emitted and parsed correctly
tool call with CDATA	code round-trips, no wrapper leak
Chinese	answered in Chinese
grounding	uses in-prompt facts over parametric memory

---

Model

architecture	Llama (model_type: llama) — stock mlx_lm
layers	42
hidden / intermediate	2048 / 6144 (SwiGLU)
attention	GQA 16q / 2kv × head_dim 128
norm	RMSNorm, no +1 shift
vocab	130,560 (untied embeddings)
RoPE	θ = 5×10⁶
context	131,072
languages	English, Chinese

---

Quantized by Jinho Jang — eric@osaurus.ai · osaurus.ai

Base model © OpenBMB, Apache-2.0.