Myric/KAT-Coder-V2.5-Dev-APEX-GGUF

KAT-Coder-V2.5-Dev — APEX GGUF (torch imatrix)

MoE-aware, mixed-precision APEX quantization of Kwaipilot/KAT-Coder-V2.5-Dev — an agentic-coding fine-tune built on the Qwen3.6-35B-A3B backbone (qwen3_5_moe: 40 layers, 256 routed + 1 shared expert, hybrid GatedDeltaNet linear-attention + periodic full attention). Kwaipilot reports SWE-bench Verified 69.40 for this size class (vs Qwen3.6-35B-A3B's 64.40).

Two architectural notes vs. the stock Qwen3.6 base this is built on:

• No MTP/NextN head — this checkpoint ships mtp_num_hidden_layers: 0, unlike Qwen3.6's fused MoE MTP layer. Nothing to quantize or skip; the head simply isn't there.
• Text-only release — the upstream config carries a vision_config block (a Qwen3-VL-style SigLIP tower) with no corresponding weights in the checkpoint; Kwaipilot's own README confirms the vision/multimodal components are not included in this release. Conversion here uses the text-only path (Qwen3_5MoeForCausalLM), which only ever reads the model's text_config — irrelevant to output correctness either way, but avoids routing through vision-aware conversion code for a component that has no weights.

Checkpoint stores MoE experts as fully separate per-expert tensors (mlp.experts.{e}.{gate,up,down}_proj.weight × 256), fused into the gate_up_proj/down_proj layout during imatrix generation and GGUF conversion.

The -torch imatrix — what it is

Same rationale as this project's Qwen3.6 quants (identical qwen3_5_moe architecture): llama-imatrix is impractical here because the GatedDeltaNet linear-attention is a serial state-space recurrence — the imatrix collection callback breaks the GPU path, falling back to a single CPU thread. kat-coder.imatrix is a standard GGUF-format importance matrix (in_sum2 + counts per tensor), generated with a band-serialized PyTorch forward pass over a general text/code corpus, bit-compatible with llama-quantize --imatrix.

Sizes

tier	precision	file size	bits/weight	wikitext PPL	vs bf16
bf16 (reference)	full	64.61 GiB	16.01	7.028	—
APEX-i-quality	Q6_K/Q4_K/Q3_K by role+band	19.23 GiB	4.65	7.117	+1.27%
APEX-dynamic (16GB-card)	Q6_K/Q3_K/IQ3_XXS/IQ2_XXS by role+band	12.13 GiB	2.94	7.543	+7.33%
IQ3_XXS (stock, imatrix-guided)	uniform	12.68 GB	~2.98	7.881	+12.13%

All three tiers gated coherent with 3/3 chained tool-calling (see TOOL_CALLING.md).

APEX-dynamic beats the stock IQ3_XXS baseline on both axes at once here — smaller (12.13 vs 12.68 GiB) and lower PPL (7.543 vs 7.881) — a cleaner result than this toolkit's Laguna-XS run, where the same method traded size for a small PPL cost. Whether role/depth-aware allocation beats a well-tuned stock quant appears to depend on the specific architecture and calibration, not something to assume either way without measuring — this is the honest result for this model.

i-quality is the pick if you're not VRAM-constrained to 16GB — within 1.3% of full bf16 quality at under a third of the size.

Structural allocation (dynamic-APEX)

Both APEX tiers use dynamic_apex.py's structural mode: routed experts are bucketed into edge / near / middle depth-bands (edge = first/last layers, generally most load-bearing) and allocated decreasing precision toward the middle:

• i-quality (19.23 GiB, 4.65 bpw): routed edge → Q6_K, near → Q4_K, middle → Q3_K.
• APEX-dynamic (12.13 GiB, 2.94 bpw): routed edge → Q3_K, near → IQ3_XXS, middle → IQ2_XXS.

Attention, shared-expert, and router tensors stay at higher fixed precision (Q6_K–Q8_0) in both tiers.

Files

• kat-coder.imatrix — the imatrix used for every quant below.
• KAT-Coder-V2.5-Dev-APEX-i-quality.gguf (19.23 GiB) — recommended default; best quality-per-byte of the tiers here.
• KAT-Coder-V2.5-Dev-APEX-dynamic.gguf (12.13 GiB) — role-aware allocation targeting a 16GB-card budget; also the best-quality option at this size point.
• KAT-Coder-V2.5-Dev-IQ3_XXS.gguf (12.68 GB) — stock imatrix-guided baseline, included for the size/PPL comparison above.
• TOOL_CALLING.md — reproducible chained tool-calling transcript.

Attribution

• Base model: Kwaipilot — Kwaipilot/KAT-Coder-V2.5-Dev.
• APEX recipe & toolkit: LocalAI — localai-org/apex-quant.
• Quantization engine: llama.cpp (ggml-org).

Unofficial community quantization; not affiliated with or endorsed by Kwaipilot.