KAT-Coder-V2.5-Dev-AWQ-W4A16-ASYM

INT4 (W4A16, asymmetric) AWQ quantization of Kwaipilot/KAT-Coder-V2.5-Dev, produced with llm-compressor 0.12.0.

	Original	This model
Weights on disk	69.33 GB	21.65 GB (3.20× smaller)
Format	BF16	compressed-tensors, pack-quantized

Verified on 2×RTX 3090 (24 GB) with vLLM 0.26.0: weights load in 27 s, and at --tensor-parallel-size 2 --max-model-len 8192 --gpu-memory-utilization 0.85 the engine settles at 20.9 GB per card including KV cache. The BF16 original needs roughly three times the weight memory.

Requirements

vLLM ≥ 0.24.0 is required. Asymmetric W4A16 for MoE layers was blocked in vLLM by an explicit Only symmetric quantization is supported for MoE assertion until vllm#44025 (merged 2026-06-02). On older versions this checkpoint fails to load rather than degrading gracefully. If you are pinned to an older vLLM, use a symmetric W4A16 quantization instead.

vllm serve Ar4ikov/KAT-Coder-V2.5-Dev-AWQ-W4A16-ASYM \
    --tensor-parallel-size 2 \
    --max-model-len 32768

What was quantized

KAT-Coder-V2.5-Dev is built on Qwen3.5-35B-A3B: 40 decoder layers, 30 of which use gated-deltanet linear attention and 10 full attention, with 256 routed experts per layer plus one shared expert.

Only the routed experts are quantized — 30 720 modules (mlp.experts.*.{gate,up,down}_proj), about 32.2B of the 34.7B parameters. Everything else stays BF16:

Component	Precision	Why
Routed experts (256/layer)	INT4	92% of the weights; sparsely activated
linear_attn.*	BF16	Gated-deltanet recurrence, numerically delicate
self_attn.*	BF16	Only 10 layers, active for every token
mlp.shared_expert.*	BF16	Active for every token
mlp.gate (router)	BF16	INT4 here shifts expert selection
Embeddings, lm_head	BF16	Standard practice

This mirrors the layer coverage of cyankiwi/Qwen3.5-35B-A3B-AWQ-4bit on the same base, which is symmetric where this one is asymmetric.

Recipe

• Scheme: W4A16_ASYM — 4-bit integer weights, asymmetric, group_size=128, group strategy
• Algorithm: AWQ, duo_scaling="both", n_grid=20
• Calibration: 256 conversations (~223k tokens, max 2048 each) — 128 from m-a-p/CodeFeedback-Filtered-Instruction and 128 from HuggingFaceH4/ultrachat_200k, rendered through the model's chat template. Code-weighted on purpose, since this is an agentic coding model.
• Expert calibration: natural top-8-of-256 routing (not all-experts), so each expert sees roughly 7k calibration tokens.

recipe.yaml in this repo is the exact recipe llm-compressor recorded.

Two things this architecture needed

Experts must be linearized first. In current transformers the 256 experts are stored as fused 3D parameters (experts.gate_up_proj, shape (256, 1024, 2048)), not as nn.Linear modules. A plain targets=["Linear"] recipe therefore matches nothing inside the MoE and produces a checkpoint that is not smaller at all. llm-compressor's llmcompressor.modeling.moe linearizes them into per-expert modules for calibration.

The router has to be compensated. AWQ folds its smoothing scale into post_attention_layernorm, whose output feeds four consumers: the router, the routed experts, the shared expert and the shared-expert gate. Any consumer left out of the balance layers gets rescaled inputs with nothing to compensate — for the router that silently changes which experts are selected, which is invisible in loss curves and shows up as degraded quality. All four are listed explicitly in the AWQ mapping, so the router is compensated while staying in BF16:

AWQMapping(
    "re:.*post_attention_layernorm$",
    [
        "re:.*mlp.gate$",                    # router — not quantized, still compensated
        "re:.*mlp.shared_expert_gate$",
        "re:.*mlp.shared_expert.gate_proj$",
        "re:.*mlp.shared_expert.up_proj$",
        "re:.*mlp.experts.*.gate_proj$",
        "re:.*mlp.experts.*.up_proj$",
    ],
)

Prompt format

Uses the base model's chat template unchanged. See the original model card for the agentic-coding prompt conventions.

Caveats

• No benchmark evaluation was run. Only a generation smoke test (coherent Python, an explanation of binary-search complexity, and a Russian prompt). The quality claims are those of AWQ in general, not measured for this checkpoint.
• Loading with transformers does not work. Its loader expects fused 3D expert parameters and cannot rebuild them from per-expert packed tensors, so the MoE is silently randomly initialized and the model emits gibberish. Use vLLM, or a runtime with a compressed-tensors MoE loader. The same applies to the reference quantization of this base, which uses the identical tensor layout.
• The base repo declares a vision_config but ships no visual weights, so the model is text-only. transformers randomly initializes that tower on load; those tensors were removed from this checkpoint, matching the source.
• Loading with transformers decompresses the weights back to BF16, so it uses as much memory as the original. The saving is real only in runtimes with native INT4 kernels.