KAT-Coder-V2.5-Dev-W4A16-GPTQ

Production W4A16 GPTQ (compressed-tensors pack-quantized: symmetric INT4, group size 128, column nibble-packed int32, bfloat16 scales) quantization of Kwaipilot/KAT-Coder-V2.5-Dev (35B-A3B MoE: 40 decoder layers, 256 routed experts / 8 active, 2,048 hidden dimension, linear attention GDN + full attention hybrid, MTP draft block).

Built specifically for high-efficiency inference on the vLLM XPU backend (CompressedTensorsWNA16MoEMethod and XPUExpertsWNA16 on Intel Arc / Xe2 architectures, e.g. Arc 140V Lunar Lake and Arc Pro B-Series), as well as compatible standard vLLM backends.

---

Model Architecture & Quantization Recipe

• Trunk:
  • 256 Routed Experts per Layer (40 Layers): Quantized to symmetric INT4 with group size 128 using activation-aware GPTQ (calibrated on 65,536 tokens from pile-val). Fallback to RTN with optimal scale clipping is applied for any singular/unvisited channels to preserve 100% expert weight integrity.
  • Attention & Shared Experts: Kept in native BFloat16 to preserve delicate multi-step quantitative reasoning and eliminate activation quantization noise.
  • Embeddings, Norms, Router Gates, and Heads: Preserved in BFloat16.
• Speculative Decoding Draft Head (MTP $K=2$):
  • Grafted 19-tensor BFloat16 Multi-Token Prediction (MTP) draft model block (model-mtp.safetensors).
  • text_config.mtp_num_hidden_layers is set to 1, enabling high-speed speculative decoding (--speculative-config '{"method":"mtp","num_speculative_tokens":2}').
• Total Tensors: 93,412 tensors across 42 shards (~20.0 GiB resident). SHA256SUMS included.

---

Benchmark Results (Measured on Intel Arc 140V, Lunar Lake 17W Handheld)

All benchmarks were evaluated under greedy decoding (temperature=0.0, batch size 1) on vLLM 0.31.0+xpu with MTP $K=2$ speculative decoding and FP8 KV cache:

Benchmark / Test Suite	Score / Accuracy	Codegen Duration	Decode Throughput
HumanEval (Base)	86.59% (142 / 164)	42m 12s (Fastest) ⚡	~23.4 tok/s
HumanEval+ (Plus Fuzzing)	82.93% (136 / 164)	—	—
16-Prompt Quality Suite	15 / 16 (93.8%) 🏆	165.52s (24% faster) ⚡	21.5–25.2 tok/s
10-Task Hard Benchmark Suite	8 / 10 (80.0%)	303.26s	80.5 chars/s
Peak Decode Throughput (TG)	29.10 tok/s ⚡	—	Burst decode
Peak Chunked Prefill (PP)	1,247.1 tok/s	—	FlashAttention-2 (4k batch)
MTP Draft Acceptance ($K=2$)	77.3% avg (up to 100%)	—	2.55–2.71 tokens / step

Key Evaluation Highlights

• HumanEval+: Achieved 82.93% pass@1, outperforming Tiel-Coder-35B-A3B-W4A16-GPTQ (81.10%) while completing all 164 tasks in 42m 12s (~15.4s / task).
• Quantitative Math: Successfully solved the Chinese Remainder Theorem Olympiad problem (math_chinese_remainder) and flashlight bridge crossing (17 min).
• Algorithmic Coding: Solved all three LeetCode Hard problems in the benchmark suite (Trapping Rain Water, LRU Cache, and Minimum Window Substring) with 100% automated test pass.
• Long Context: Successfully completed multi-hop synthesis across 12,500 tokens of infrastructure audit logs with 128k context ceiling.

---

Serving with vLLM

Production Launch Command (Intel Arc / XPU)

vllm serve alpha2beta/KAT-Coder-V2.5-Dev-W4A16-GPTQ \
  --served-model-name KAT-Coder-V2.5-Dev-W4A16-GPTQ \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 131072 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.74 \
  --language-model-only \
  --enforce-eager \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --kv-cache-memory-bytes 2500000000 \
  --kv-cache-dtype fp8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

---

Provenance & Attribution

• Base Model: Kwaipilot/KAT-Coder-V2.5-Dev (Apache-2.0).
• MTP Draft Head: SpectreSystems/KAT-Coder-V2.5-Dev-MTP.
• Quantizer & Reproduction Pipeline: Developed and maintained in alpha2beta/vllm.xpu.