Manprot/MP-02-16B-Preview

MP-02 16B Preview

A pruned and healed version of GLM-5.3-Flash, made by the Manprot team

Made by	The Manprot team
Family	MP-02 (see also MP-01 100M Preview)
Status	Preview: it works, but it's rough (see Limitations)
Type	Chat model with thinking, healed on GLM-5.3 conversations
Parameters	16.06B in the transformer layers; 18.28B in total with embeddings and LM head (1.27B), MTP head (0.39B) and vision encoder (0.56B)
Architecture	GLM-5.3-Flash (glm5_next) cut from 288 to 8 routed experts per MoE layer
Built from	zai-org/GLM-5.3-Flash (MIT)
Tokens trained	22.9M (healing only; no pretraining of our own)
7-task average	51.29, about 3 points above MP-01 100M Preview
License	MIT, same as the base model (Z.ai copyright notice kept in LICENSE)

Note: this is a preview. It is somewhat decent at chat and roleplay, but cutting 97% of the experts cost it much of GLM-5.3-Flash's knowledge and reasoning. It is not a replacement for GLM-5.3-Flash. Expect wrong facts, broken arithmetic, over-long thinking and the occasional loop.

Contents

• Training lineage
• Datasets
• Architecture
• Evaluation
• Compute
• Running it
• Limitations
• Repository contents
• Citation

Training lineage

1. Expert pruning (REAP-style). Each of the 42 MoE layers of GLM-5.3-Flash (FP8) kept the 8 of its 288 routed experts with the highest router-weighted output norm (gate × ‖expert output‖), measured on a 102k-token calibration set. Router rows were cut to match. The shared expert, attention, the 3 dense layers, the MTP head and the vision encoder are unchanged. Held-out loss right after pruning: 3.725.
2. Healing, stage 1. LoRA (rank 64, alpha 128, on every attention and MLP projection, routed experts included) plus fully trained routers. AdamW, learning rate 1e-4, warmup and cosine decay, 2,048-token chunks, loss on assistant turns only. 174 steps, 10.9M tokens. Held-out loss: 3.725 → 2.105.
3. Healing, stage 2. Continued from stage 1 with a fresh optimizer on 12.1M new tokens (no conversation from stage 1 repeated), learning rate 8e-5, 193 steps. Held-out loss on a new held-out set: 2.449 → 2.218. This is the checkpoint released here.

The two held-out sets differ, so their losses are not comparable with each other.

Datasets

In total: 13 healing datasets and 3 calibration datasets (one used for both). Every dataset that shaped these weights is listed here and in this page's metadata. Healing data is only text written by GLM-5.3 or GLM-5.3-Flash, with one exception noted below. Training data is not re-released.

Healing

Dataset	What it is	Stage 1	Stage 2
Zek-Takai/glm53-flash-harvest	Prose, chat, multi-turn, reasoning, code, structured output, agentic (GLM-5.3-Flash)	65%	69%
shanjiaz/glm-5.3-flash-open-perfectblend-regen-25pct	Mixed instructions	15%	2%
best-distill/glm-5.3-flash-distillation-chat	Instructions with reasoning	7%	3%
open-athena/GLM-5.3-RLVR1-Training-Rollouts-2026.09.25	Mixed tasks (GLM-5.3)	5%	10%
sollamon/ox-alpha-glm-5.3-flash-distillation-coding-17k-raw	Coding	3%	3%
open-athena/GLM-5.3-tmax-swesmith-k8-traces-2026.10	Terminal and bug-fix agent runs, passed only (GLM-5.3)	1%	7%
liangzhidanta/claude-code-glm53-swesmith-trajectories and its compact-aware version	Coding-agent runs, some with context compaction (GLM-5.3)	–	3%
kaivoss/UltraData-SFT-Agent-2609-GLM53-20k	Agent runs in GLM-5.3's tool format. Exception: the text was written by the upstream models, not GLM	2%	–
Rallex3, closestfriend, FranckAbgrall, flaviojoshua, Offlin33er function-calling sets	Tool calls (GLM-5.3-Flash)	2%	3%

Shares are of each stage's tokens. Stage 1 drew 10.9M tokens from a 40M-token mix; stage 2 used 12.1M tokens from an 18M-token mix (the time budget ended the run first). In both stages, conversations where the model refused as an AI were dropped, and any sexual content involving minors was removed.

Calibration for pruning (stage 1 only)

These decided which experts were kept. The model was not trained on them.

Dataset	Share
Gryphe/Sonnet3.5-Charcard-Roleplay	30%
anthracite-org/c2_logs_32k_llama3_qwen2_v1.3	15%
Zek-Takai/glm53-flash-harvest (multi-turn, chat, prose, reasoning)	55%

Evaluation benchmarks

Benchmark	Dataset
ARC-Challenge, ARC-Easy	allenai/ai2_arc
HellaSwag	Rowan/hellaswag
OpenBookQA	allenai/openbookqa
PIQA	baber/piqa
SciQ	allenai/sciq
WinoGrande	allenai/winogrande

Architecture

Component	Value
Layers	45 (3 dense + 42 MoE)
Width (hidden size)	4096
Experts	8 routed + 1 shared per MoE layer, all 8 routed experts active on every token (from 288, top-8 of 288 in the base)
Expert hidden	2048
Attention	GLM-5.3-Flash hybrid sparse / linear attention (unchanged)
Extra heads	1 MTP (multi-token prediction) layer, kept
Vision	GLM-5.3-Flash vision encoder (unchanged, not evaluated after pruning)
Vocabulary	154,880 (GLM-5.3 tokenizer)
Context length	healed at 2,048 tokens; the base supports far more, but long context is untested after pruning
Parameters	18.28B total: 16.06B layers, 1.27B embeddings and LM head, 0.39B MTP, 0.56B vision

Because all 8 kept experts fire on every token, MP-02 runs at the speed of a dense 16B model, not a sparse MoE.

Evaluation

Scored with lm-evaluation-harness: zero-shot, the first 500 questions of each task, bf16 on one H100. The metric per task is the same as MP-01's harness: accuracy for ARC-Easy and WinoGrande, length-normalised accuracy for the rest. With 500 questions a task, differences under about 2 points are within noise. MP-01 was scored by its own harness on the same tasks, metrics and sample size, so treat small differences between the two as noise.

Comparison with MP-01

Rank	Model	Params	ARC-C	ARC-E	HellaSwag	OBQA	PIQA	SciQ	Wino	Avg (7)
1	MP-02 16B Preview (this model)	18.28B	27.0	48.4	47.0	29.2	63.2	89.0	55.2	51.29
2	MP-01 100M Preview	100.6M	29.4	56.4	31.8	30.2	64.4	73.0	53.8	48.43

Bold is the best score in each column.

Where this model stands

• 7-task average: about 3 points above MP-01, mostly from HellaSwag (+15.2) and SciQ (+16.0). It is behind MP-01 on ARC-Easy (−8.0) and slightly on ARC-Challenge, OpenBookQA and PIQA.
• That is a low result for its size. ARC-Challenge is near chance (25% for 4 choices). GLM-5.3-Flash itself was not run on this setup, but its published scores are far higher, so the pruning loss is large.
• These are multiple-choice log-likelihood tests. They don't measure chat, roleplay or thinking, which is what the healing focused on.

Progress across checkpoints of this model

Checkpoint	Held-out loss	ARC-C	ARC-E	HellaSwag	OBQA	PIQA	SciQ	Wino	Avg (7)
Stage 1, step 174	2.105 (set A)	27.0	48.0	47.2	27.4	64.4	89.4	56.8	51.46
Stage 2, step 193 (this release)	2.218 (set B)	27.0	48.4	47.0	29.2	63.2	89.0	55.2	51.29

Stage 2 left the benchmarks where they were (all changes are within noise) and lowered loss on its own held-out set from 2.449 to 2.218.

Compute

Stage	Hardware	GPU hours	Tokens
Expert selection and pruning	Local consumer GPU (8 GB)	about 1.9	102k (calibration)
Healing, stage 1 (3 runs)	Cloud B200	2.85	10.9M
Healing, stage 2 (this release)	Cloud B200	2.60	12.1M
Benchmarks	Cloud H100	0.12 per checkpoint

Hours are wall-clock GPU time from the run logs, including model loading. Costs in credits are not listed.

Running it

llama.cpp (GGUF)

hf download Manprot/MP-02-16B-Preview MP-02-16B-Preview-Q4_K_M.gguf --local-dir MP-02
llama-server -m MP-02/MP-02-16B-Preview-Q4_K_M.gguf -ngl 99 --n-cpu-moe 99 -c 8192

Needs a llama.cpp build with glm5-next support. On a desktop with an 8 GB GPU and DDR5 memory, keeping the experts on the CPU (--n-cpu-moe 99), it runs at about 6–7 tokens/s. On a single 24 GB GPU with everything on the GPU, it runs at about 28 tokens/s. For images, add --mmproj mmproj-MP-02-16B-Preview-BF16.gguf.

transformers (safetensors)

import torch
from transformers import AutoTokenizer, Glm5NextForConditionalGeneration

tok = AutoTokenizer.from_pretrained("Manprot/MP-02-16B-Preview")
model = Glm5NextForConditionalGeneration.from_pretrained("Manprot/MP-02-16B-Preview", dtype=torch.bfloat16, device_map="auto")
msgs = [{"role": "user", "content": "Write a short poem about the sea."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, reasoning_effort="low", return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=400)[0][ids.shape[1]:]))

Needs transformers 5.17 or newer and about 37 GB of GPU memory in bf16.

Thinking and the chat template

MP-02 uses GLM-5.3's chat template, which always starts a thinking block and, with no reasoning_effort, sends Reasoning Effort: Max. All healing data was rendered at Max, so by default it thinks long on everything, even simple requests. For normal chat, pass reasoning_effort: "low" (for example "chat_template_kwargs": {"reasoning_effort": "low"} on llama-server). A thinking on/off switch in a chat UI has no effect with this template.

Suggested sampling: temperature 0.7, top-p 0.8, repeat penalty 1.05.

Limitations

• Preview quality. Most of GLM-5.3-Flash's knowledge is gone. It invents facts, gets simple arithmetic wrong and can talk confidently about things it does not know.
• Over-thinking. It thinks at maximum effort by default (see above) and can spend its whole token budget thinking before it answers.
• Agent habits. Agent traces in the healing data sometimes leak into chat: invented commands, sandbox paths such as /home/user or /testbed, and fake tool output.
• Loops. It can repeat a phrase until the token limit. A repeat penalty helps.
• Identity. It was trained on GLM's own conversations and may say it is GLM, made by Z.ai.
• Few refusals. AI refusals were removed from the healing data, so it follows most requests. It has not been checked for safety; filter outputs yourself.
• Untested after pruning: vision, tool calling, long context and languages other than English.

Repository contents

• model-*.safetensors, model.safetensors.index.json: merged weights in bf16 (norms in fp32)
• config.json, generation_config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja: model configuration and tokenizer
• MP-02-16B-Preview-Q4_K_M.gguf: 4-bit GGUF for llama.cpp (12.9 GB)
• mmproj-MP-02-16B-Preview-BF16.gguf: vision projector for llama.cpp (optional)
• eval_results.json: full evaluation output for this checkpoint
• LICENSE: MIT license with Z.ai's copyright notice
• manprot-logo.png: logo

Citation

Made by the Manprot team, based on GLM-5.3-Flash by Z.ai. All credit for the original architecture and weights goes to Z.ai.

@misc{manprot2026mp02,
  title        = {MP-02 16B Preview},
  author       = {{Manprot team}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Manprot/MP-02-16B-Preview}}
}

The datasets listed above belong to their authors and keep their own licenses.