AceTheUnfazed/KAT-Coder-V2.5-Dev-APEX-MTP-GGUF

KAT-Coder V2.5 Dev APEX I-Compact MTP GGUF

Community MTP-enabled GGUF derivative of KAT-Coder V2.5 Dev, created by grafting the Qwen3.6-35B-A3B MTP/NextN block onto Mudler's KAT-Coder V2.5 Dev APEX I-Compact GGUF.

This is a community derivative. The underlying models and Mudler's APEX quantized source files remain the work of their respective authors. This README documents the grafting process and local testing performed for this derivative.

---

At a glance

Recommended llama.cpp configuration

For the merged model, the preferred configuration from testing is:

PS C:\Users\acetheunfazed\llama-cpp-turboquant\build\bin\Release> .\llama-server.exe --host 0.0.0.0 --port 8080 -m "C:\Users\acetheunfazed\Models\KAT-Coder-V2.5-Dev-APEX-I-Compact-MTP.gguf" --mlock --no-mmap -ctk turbo4 -ctv turbo2 --cache-prompt --cache-reuse 256 -c 98304 -b 2048 -ub 2048 -t 6 --n-cpu-moe 18 -np 1 --tools all -fa on --spec-type draft-mtp --spec-draft-n-max 2

Why --spec-draft-n-max 2?

Local testing showed that 2 was substantially better than 3 on this system.

MTP value	Long-run decode	Reported draft acceptance
n-max=3	39.22 t/s	74.13%
n-max=2	46.61 t/s	82.59%

For n-max=2, llama.cpp also reported:

#gen tokens = 4772
#acc tokens = 3835

or approximately 80.35% accepted draft tokens for that statistics interval.

For n-max=3, the tested run reported:

#gen tokens = 9036
#acc tokens = 6554

The practical result was clear: the third speculative token added enough overhead that it reduced overall throughput instead of improving it. Therefore, --spec-draft-n-max 2 is the recommended setting for the tested hardware.

These are local measurements, not universal guarantees. Different hardware or llama.cpp builds may prefer a different MTP depth.

---

Hardware tested

• CPU: AMD Ryzen 5 4500 — 6 cores / 12 threads
• RAM: 32 GB DDR4
• GPU: NVIDIA GeForce RTX 3060 12 GB
• OS: Windows 11 Pro (25H2)

The GGUF graft itself was performed locally on this system with approximately 150 GB of free disk space.

---

llama.cpp build

Testing was performed with TheTom's llama.cpp TurboQuant Fork:

https://github.com/TheTom/llama-cpp-turboquant

Version: 9914 (0c8fcfe73)
Compiler: MSVC 19.44.35227.0
Target: x64

The build used here supports the required Qwen3.6/NextN MTP path and draft-mtp.

---

Benchmark comparison

These are local observations from the same general hardware/environment. The exact server parameters were tuned slightly per model, so they should be treated as practical reference results rather than a controlled benchmark.

Model	Peak decode	Long-run observed decode	Recorded output
KAT-Coder V2.5 Dev APEX I-Compact	~44 t/s	~40 t/s	~7.8k tokens
Qwen3.6-35B-A3B APEX MTP I-Compact	~55 t/s	~48.5 t/s	~10.8k tokens
KAT-Coder V2.5 Dev APEX I-Compact MTP	~55 t/s	~46.6 t/s	~6k tokens

KAT-Coder V2.5 Dev APEX I-Compact

PS C:\Users\acetheunfazed\llama-cpp-turboquant\build\bin\Release> .\llama-server.exe --host 0.0.0.0 --port 8080 -m "C:\Users\acetheunfazed\Models\KAT-Coder-V2.5-Dev-APEX-I-Compact.gguf" --mlock --no-mmap -ctk turbo4 -ctv turbo2 --cache-prompt --cache-reuse 256 -c 98304 -b 1024 -ub 1024 -t 6 --n-cpu-moe 15 -np 1 --tools all -fa on

Observed:

• Peak: ~44 t/s
• Long-run: ~40 t/s
• ~7.8k output tokens
• ~3 min 15 sec
• Relatively gradual decode-speed decline

Qwen3.6-35B-A3B APEX MTP I-Compact

PS C:\Users\acetheunfazed\llama-cpp-turboquant\build\bin\Release> .\llama-server.exe --host 0.0.0.0 --port 8080 -m "C:\Users\acetheunfazed\Models\Qwen3.6-35B-A3B-APEX-MTP-I-Compact.gguf" --mlock --no-mmap -ctk turbo4 -ctv turbo2 --cache-prompt --cache-reuse 256 -c 98304 -b 1024 -ub 1024 -t 6 --n-cpu-moe 18 -np 1 --tools all -fa on --spec-type draft-mtp --spec-draft-n-max 3

Observed:

• Peak: ~55 t/s
• Long-run: ~48.5 t/s
• ~10.8k output tokens
• ~3 min 37 sec
• Noticeable decode-speed decline as generation progressed

KAT-Coder V2.5 Dev APEX I-Compact MTP

Recommended:

PS C:\Users\acetheunfazed\llama-cpp-turboquant\build\bin\Release> .\llama-server.exe --host 0.0.0.0 --port 8080 -m "C:\Users\acetheunfazed\Models\KAT-Coder-V2.5-Dev-APEX-I-Compact-MTP.gguf" --mlock --no-mmap -ctk turbo4 -ctv turbo2 --cache-prompt --cache-reuse 256 -c 98304 -b 2048 -ub 2048 -t 6 --n-cpu-moe 18 -np 1 --tools all -fa on --spec-type draft-mtp --spec-draft-n-max 2

Observed:

• Peak: ~55 t/s
• Long-run: ~46–47 t/s
• ~6k output tokens
• ~2 min 10 sec
• Strong decode-speed stability over the tested run

---

Final result

The resulting file is:

KAT-Coder-V2.5-Dev-APEX-I-Compact-MTP.gguf

Validated characteristics:

GGUF.tensor_count                  = 753
qwen35moe.block_count              = 41
qwen35moe.nextn_predict_layers     = 1
general.architecture               = qwen35moe

The final GGUF size is:

17,436,808,896 bytes
≈ 17.44 GB

The original KAT-Coder trunk contains 733 tensors. The graft adds exactly 20 Qwen3.6 blk.40.* tensors, resulting in 753 tensors total.

The main KAT-Coder tensors retain their original APEX/I-Compact quantization. The MTP block is copied from the Qwen3.6 APEX MTP donor using its existing Q8_0 / F32 / BF16 representation.

---

Context-length / decode-stability observation

One of the most notable observations was sustained decode stability.

With the merged model and --spec-draft-n-max 2, one long run reported approximately:

100 tokens    → 39.38 t/s
399 tokens    → 48.44 t/s
997 tokens    → 49.00 t/s
1587 tokens   → 48.89 t/s
2306 tokens   → 48.51 t/s
3030 tokens   → 48.34 t/s
4122 tokens   → 47.42 t/s
5082 tokens   → 47.00 t/s
5751 tokens   → 46.67 t/s

Final measured decode:

5870 tokens
46.61 t/s

This indicates relatively shallow throughput degradation across the tested generation range.

It does not prove that the model eliminates context-related slowdown at the full configured 98k context. A controlled test at 20k, 30k, 50k, 70k and 90k+ context would be required to establish that.

---

Practical advantages observed

• Faster responses: MTP substantially improves decode throughput over the non-MTP KAT-Coder APEX build on the tested system.
• Higher utilization of the context budget: KAT-Coder's comparatively concise reasoning/output can leave more room for additional turns.
• KAT-Coder behavior retained: The main model remains KAT-Coder rather than being replaced by Qwen.
• High MTP acceptance: n-max=2 produced approximately 78–83% draft acceptance in the longer tests shown above.
• Efficient speculative depth: On the tested system, two speculative tokens provided a better throughput/acceptance tradeoff than three.

---

How it was done

1. Obtain the two Mudler source GGUFs

KAT-Coder source

https://huggingface.co/mudler/KAT-Coder-V2.5-Dev-APEX-GGUF/blob/main/KAT-Coder-V2.5-Dev-APEX-I-Compact.gguf

Qwen3.6 MTP donor

https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/blob/main/Qwen3.6-35B-A3B-APEX-MTP-I-Compact.gguf

Copy both files into a working directory. Keep the originals untouched.

---

2. Install the GGUF Python tooling

python -m pip install -U gguf

Confirm the package and APIs:

python -c "import gguf; print(gguf.__file__)"
python -c "import gguf; print(hasattr(gguf,'GGUFReader')); print(hasattr(gguf,'GGUFWriter'))"

---

3. Inspect the source files

python -m gguf.scripts.gguf_dump --no-tensors "KAT-Coder-V2.5-Dev-APEX-I-Compact.gguf"
python -m gguf.scripts.gguf_dump --no-tensors "Qwen3.6-35B-A3B-APEX-MTP-I-Compact.gguf"

The important values were:

KAT:
733 tensors
40 blocks

Qwen MTP:
753 tensors
41 blocks
nextn_predict_layers = 1

---

4. Identify the MTP tensors

On Windows, force UTF-8 before filtering:

$env:PYTHONIOENCODING="utf-8"

Then:

python -m gguf.scripts.gguf_dump "Qwen3.6-35B-A3B-APEX-MTP-I-Compact.gguf" | Select-String "blk.40|nextn|mtp"

The Qwen file contains exactly 20 blk.40.* tensors.

KAT-Coder contains no blk.40.* tensors.

---

5. Verify the shared trunk

Compare the final existing KAT block:

python -m gguf.scripts.gguf_dump "KAT-Coder-V2.5-Dev-APEX-I-Compact.gguf" | Select-String "blk.39"

and Qwen:

python -m gguf.scripts.gguf_dump "Qwen3.6-35B-A3B-APEX-MTP-I-Compact.gguf" | Select-String "blk.39"

The tensor names and logical shapes match across the shared trunk.

The quantization types do not need to match; KAT's existing APEX/I-Compact tensors must remain untouched.

---

6. Graft the 20 MTP tensors

Create graft_mtp.py.

The essential operations are:

1. Read KAT metadata and tensors.
2. Read only Qwen's blk.40.* tensors.
3. Preserve all 733 KAT tensors.
4. Append the 20 Qwen MTP tensors.
5. Change qwen35moe.block_count from 40 to 41.
6. Add qwen35moe.nextn_predict_layers = 1.
7. Preserve raw quantized tensor bytes with:

   raw_shape=tensor.data.shape
   raw_dtype=tensor.tensor_type
   

8. Write a new GGUF without modifying either source.

Validate syntax before execution:

python -m py_compile graft_mtp.py

Run:

python graft_mtp.py

The resulting file should contain:

733 KAT tensors
+
20 MTP tensors
=
753 tensors

---

7. Validate the generated GGUF

python -m gguf.scripts.gguf_dump --no-tensors "KAT-Coder-V2.5-Dev-APEX-I-Compact-MTP.gguf"

Confirm:

tensor_count = 753
block_count = 41
nextn_predict_layers = 1

Then confirm all 20 MTP tensors:

python -m gguf.scripts.gguf_dump "KAT-Coder-V2.5-Dev-APEX-I-Compact-MTP.gguf" | Select-String "blk.40"

Finally confirm KAT's existing quantization remains intact:

python -m gguf.scripts.gguf_dump "KAT-Coder-V2.5-Dev-APEX-I-Compact-MTP.gguf" | Select-String "blk.39.ffn_down_exps.weight|blk.39.ffn_gate_exps.weight|blk.39.ffn_up_exps.weight"

Expected examples:

blk.39.ffn_down_exps.weight = Q4_K
blk.39.ffn_gate_exps.weight = Q4_K
blk.39.ffn_up_exps.weight   = Q4_K

---

8. Test with llama.cpp

Use the recommended n-max=2 command shown at the top of this README.

To verify MTP is actually active, look for:

creating MTP draft context
adding speculative implementation 'draft-mtp'
speculative decoding context initialized

During generation, llama.cpp reports:

draft acceptance = ...
statistics draft-mtp:

These lines provide the acceptance and speculative-token statistics.

---

Troubleshooting and failed attempts

1. Windows encoding error

gguf_dump initially failed while printing tokenizer text:

UnicodeEncodeError: 'charmap' codec can't encode character '\u0120'

Fix:

$env:PYTHONIOENCODING="utf-8"

---

2. get_tensor() expected an index

Attempting:

reader.get_tensor("blk.40.attn_k.weight")

failed because the installed API expects a numeric tensor index.

Fix: find the tensor index first using enumerate(reader.tensors).

---

3. Overly strict quantization check

The first graft validation rejected:

blk.35.ffn_down_exps.weight
KAT : Q4_K
Qwen: Q3_K

This check was wrong for the intended graft.

Fix: validate common tensor names and logical shapes, but preserve KAT's own quantization instead of requiring the Qwen donor's quantization to match.

---

4. Incorrect raw shape

For quantized tensors, the logical shape and raw byte-storage shape differ.

Example:

logical shape = [2048, 512]
raw data shape = (512, 1152)

Using the logical shape as raw_shape would be incorrect.

Fix:

raw_shape=tensor.data.shape

with:

raw_dtype=tensor.tensor_type

---

5. GGUF writer state error

The first full graft reached the final write stage but failed with:

ValueError:
Expected output file to contain KV data,
got WriterState.HEADER

The installed writer requires:

HEADER → KV_DATA → WEIGHTS

Fix:

writer.write_header_to_file()
writer.write_kv_data_to_file()
writer.write_tensors_to_file(progress=True)

The corrected run completed successfully.

---

Formal project motivation and observed behavior

KAT-Coder V2.5 Dev is a coding/agent model derived from the Qwen3.6 family. The purpose of this derivative is to combine KAT-Coder's behavior with Qwen3.6's MTP/NextN capability while retaining Mudler's APEX I-Compact quantization for the KAT-Coder trunk.

In local use, KAT-Coder was observed to be more concise in its reasoning and output than Qwen3.6 in some workflows, reducing unnecessary token generation. I also found its code-generation quality to be roughly equal to, and frequently better than, base Qwen3.6 for my own coding tasks.

The resulting derivative is therefore intended to provide:

• KAT-Coder's coding and agent behavior.
• Reduced unnecessary reasoning/output.
• The APEX I-Compact quantization of the KAT-Coder trunk.
• Qwen3.6 MTP/NextN speculative generation.
• More usable conversation turns within a fixed context budget.
• Faster response generation on compatible llama.cpp builds.

These quality and efficiency observations are personal/local findings rather than standardized evaluation results.

---

Important limitations

• This is a community derivative, not an official KAT-Coder, Qwen, or Mudler release.
• The underlying model weights and APEX quantized source files belong to their respective creators.
• The derivative is a GGUF tensor graft; it is not newly trained or fine-tuned.
• The main KAT-Coder trunk retains Mudler's APEX I-Compact quantization.
• The MTP/NextN block is inherited from Mudler's Qwen3.6 APEX MTP donor and is not re-quantized.
• Performance results are from one system and may differ significantly on other hardware.
• The observed decode stability was tested only over the reported generation/context range and should not be extrapolated automatically to the full 98k configured context.
• The MTP path requires a compatible llama.cpp build with the relevant Qwen3.6/NextN and draft-mtp support.
• CLI flags and MTP implementation details can change between llama.cpp revisions.

---

Benchmark disclaimer

All reported "peak", "long-run", "stable", and "acceptance rate" values are measurements from local testing using the documented hardware, model files, and llama.cpp build.

The three model tests used slightly different server settings, particularly batch size, CPU MoE offload, and MTP depth. Therefore these numbers are best interpreted as practical reference measurements, not controlled scientific benchmarks.

The reported output quality observations are also subjective and should not be interpreted as standardized model evaluations.

---

Credits

This derivative builds on the work of:

• Qwen / Tongyi Lab — Qwen3.6-35B-A3B
• Kwaipilot — KAT-Coder V2.5 Dev
• Mudler — APEX quantization and the APEX GGUF source files used for this graft
  • APEX Quantization project: https://github.com/localai-org/apex-quant
• ggml-org / llama.cpp contributors — GGUF tooling and MTP/speculative decoding infrastructure
• TheTom — llama-cpp-turboquant fork used for local testing

Source model links

Qwen3.6-35B-A3B:
https://huggingface.co/Qwen/Qwen3.6-35B-A3B

KAT-Coder V2.5 Dev:
https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev

Mudler Qwen3.6 APEX MTP:
https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/blob/main/Qwen3.6-35B-A3B-APEX-MTP-I-Compact.gguf

Mudler KAT-Coder APEX:
https://huggingface.co/mudler/KAT-Coder-V2.5-Dev-APEX-GGUF/blob/main/KAT-Coder-V2.5-Dev-APEX-I-Compact.gguf

TheTom's llama.cpp TurboQuant fork:
https://github.com/TheTom/llama-cpp-turboquant

---

License

This community derivative is released under the Apache License 2.0.

The model lineage used for this derivative is based on four Apache-2.0-licensed source components:

• Qwen3.6-35B-A3B — Qwen / Tongyi Lab
• KAT-Coder V2.5 Dev — Kwaipilot
• Qwen3.6-35B-A3B APEX MTP I-Compact — Mudler
• KAT-Coder V2.5 Dev APEX I-Compact — Mudler

The derivative is provided under Apache 2.0, subject to the applicable upstream licenses, notices, attribution, and redistribution requirements.

By the community, for the community!

---