Ttimms

/

KAT-Coder-V2.5-Dev-REAP-50-GGUF

like 2

Thank you, glad it's running well on the 4080 Super. 16 GB is the budget this build is designed around, so that's good to hear.

On screenshots: the base model this is built from doesn't ship with working vision, just the scaffolding for it with no trained weights, so it's not a switch I can flip here. Real vision support would mean either the upstream team releasing a vision-capable version, or training a vision adapter onto it. The second is doable, but it's a separate project with its own data and evaluation work, so I can't promise a timeline.

For now, the practical route is to run a small vision model (3–7B) alongside this one: let it read the screenshot and describe the UI, then pass that to KAT-Coder for the fix. Fits on your card and works today.

Appreciate the comment and the follow!

really good job literally and also are there any way to add MTP head to this?

Thanks, glad it's useful!

On MTP: the short answer is that this build didn't drop an MTP head. The base model doesn't ship usable MTP weights to begin with. Kwaipilot/KAT-Coder-V2.5-Dev sets mtp_num_hidden_layers: 0 in its config, so there
was never a head to carry through the prune.

So "adding it back" isn't really restoration, because there's nothing to restore. A speculative-decoding head has to be trained against a specific model's hidden states, and REAP pruning shifts those, so a donor head from a sibling checkpoint would be miscalibrated even if one existed.

The path that would actually work is training a draft module against this pruned model, either EAGLE-3 or an MTP-style head. NVIDIA's Model-Optimizer supports exactly this (EAGLE3_DEFAULT_CFG / EAGLE_MTP_DEFAULT_CFG), so it's tractable, but it's a training run rather than a config flag, and you'd also need your serving stack to accept the resulting head. It's on my list of things worth trying.

Thanks, that makes sense. I’m running the REAP-50 GGUF on a 12 GB + 8 GB dual-GPU setup and it is already exceptionally efficient for me (~34 tok/s generation at very long context), so a properly trained draft head would be especially interesting.
My current Unsloth llama.cpp build already exposes draft-eagle3, draft-mtp, --model-draft, draft GPU-layer controls and separate draft KV types. If you do train one, would you consider publishing a llama.cpp-compatible GGUF draft/head as well?
For this pruned model, do you currently lean toward EAGLE-3 or an MTP-style head? I’d be very interested in testing acceptance rate and net tok/s on REAP-50.

People in the discussion threads of KAT Coder V2.5 reported the model can use base model's mmproj and MTP head. Several quants are doing that; according to this one https://huggingface.co/gbuzhf/KAT-Coder-V2.5-Dev-MTP-GGUF/tree/main, that MTP head's draft acceptance is >70% in agentic task.

Thank you, this is useful.

On the MTP question first: KAT-Coder-V2.5-Dev ships mtp_num_hidden_layers: 0, so there is no native draft head in the
base weights. The community MTP GGUFs, gbuzhf's write-up in particular, is thorough, grafting Qwen3.6-35B-A3B's MTP head
onto KAT's trunk. Those acceptance numbers, roughly 73 percent on agentic and higher on copy-heavy, are measured on
the full unpruned trunk, where gbuzhf found essentially no penalty versus Qwen's own co-trained head. Worth noting,
they also fine-tuned that head on KAT rollouts twice, and both attempts made acceptance worse.

The open question for this repo is whether that donor head survives 50 percent expert pruning. REAP-50 shifts the
trunk's output distribution more than a straight Qwen-to-KAT swap does, so the graft may or may not hold. I plan
to test the grafted donor head on the pruned trunk first, since that is nearly free and gives a real baseline, before
training anything.

On EAGLE-3 versus MTP for the pruned model: I lean toward trying the donor MTP graft first, then EAGLE-3 trained
directly against the pruned checkpoint if the graft degrades too far. EAGLE-3 is a different operation from
fine-tuning an MTP head; it takes multiple hidden-state layers as input and trains end-to-end, so gbuzhf's negative
fine-tune result does not rule it out, but it is a reason not to assume a trained head beats a good donor graft.

Yes, if I train or graft one, I will publish a llama. cpp-compatible draft in GGUF alongside the safetensors, plus the
head on its own for re-grafts

dallaspage, if you are set up for it, I would welcome acceptance-rate and net tok/s numbers on the REAP-50 GGUF with a
grafted head. gbuzhf's serving recipe is a good starting point, especially --spec-draft-p-min 0.75 and running
draft-mtp together with ngram-mod rather than either alone.

Timeline: this is on the near-term list, but behind a quantization pass I am wrapping up now.

Gemma4 has MTP heads shipped separately (like the MTP assistant ones google provided). But Qwen's are all embedded. I find the separate MTP ggufs more convenient. For example, in this case we can load it and test it right away without downloading another GB. It would be nice, if you can provide a few separate MTP ggufs as well.