Kimi-K3-W4A16-RTN

Kimi K3 on a single NVIDIA A100 80GB.

A weight-only quantisation of Moonshot AI's Kimi K3 (2.8T total / 104B activated parameters) that loads and generates on one A100 80GB GPU, with the routed experts held in host RAM. No Ampere-targeted K3 build existed for vLLM, so this was made to fix that gap.

On one A100 80GB GPU this model gets 1 token/second for a single request and a ~100-minute model load. Useful if you have one A100 in a machine with well over 1.3 TB of RAM and want to run K3 at all. This also works with more than one GPU, but ~1tok/s is the base speed on one GPU.

What was changed

Routed experts and attention re-encoded from MXFP4/BF16 into compressed-tensors pack-quantized, served by vLLM's Marlin kernels. Activations stay BF16 (W4A16 / W8A16). Round-to-nearest only — no calibration data, so no dataset is baked into these weights.

component	scheme	size	added RMS error
routed experts (896 × 92 MoE layers)	int4, channelwise, symmetric, RTN + clip search	1,267.88 GiB	8.07% vs the released MXFP4 values
attention (every self_attn.*proj)	int4, group size 128, symmetric, RTN	16.85 GiB	10.32% vs BF16
shared experts	int8, channelwise, symmetric, RTN + clip search	11.32 GiB	0.95% vs BF16
embeddings, lm_head, router gates, norms, vision tower	unchanged BF16	~16.5 GiB	—
total		~1,314.6 GiB

Errors were measured by round-tripping each tensor through compressed-tensors' compress()/decompress(). Weight error is a proxy, not a quality measurement — see Evaluation for the real numbers.

Why channelwise: during post-load preparation vLLM's Marlin MoE path materialises the quantisation scales on the GPU, where the CPU offloader cannot reach them, so the scale set must fit in whatever VRAM is left. For K3's experts that is around 158 GB for MXFP4 as published, around 39.6 GB for int4 g128, and around 1.5 GB channelwise. Marlin supports only group sizes {-1, 32, 64, 128}, so channelwise was the only scheme that fits. It costs ~1.1 points of added weight error versus g128. Shared experts run on every token and are GPU-resident, so int8 there cuts their error from 10.15% to 0.95% for +5.4 GiB and no measurable slowdown.

Requirements

GPU	1 × A100-SXM4-80GB (compute capability 8.0), PCIe gen4 x16
host RAM	~1.3 TB consumed by the offloaded experts (tested on a 2 TB machine)
disk	~1.3 TiB
software	vLLM 0.28.0, compressed-tensors 0.17.0, PyTorch 2.13.0

Other configurations are untested.

Usage

# Required. Without it vLLM pins the offloaded weights twice and the load fails partway.
export VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY=1

# Do NOT set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True — incompatible with UVA
# offload; fails with "cudaHostAlloc failed".

from vllm import LLM, SamplingParams

llm = LLM(
    model="Fluffy/Kimi-K3-W4A16-RTN",
    tensor_parallel_size=1,
    max_model_len=8192,
    gpu_memory_utilization=0.90,
    kv_cache_dtype="bfloat16",       # fp8 KV is refused on Ampere for MLA
    cpu_offload_gb=1400,
    cpu_offload_params=["experts"],  # routed experts live in host RAM
    linear_backend="marlin",
    trust_remote_code=True,          # K3's tokenizer is custom code
    enforce_eager=True,
)

out = llm.generate(["The three branches of government are"],
                   SamplingParams(max_tokens=32, temperature=0))
print(out[0].outputs[0].text)

K3's chat template lives in tokenization_kimi.py, not a Jinja chat_template, and defaults to thinking on at thinking_effort="max". Moonshot's published evaluation sampling is temperature=1.0, top_p=0.95. Image input is verified through LLM.chat() with chat_template_kwargs={"thinking": True, "thinking_effort": "low"}.

Performance

Measured on one A100 80GB:

	value
model load	98–114 min from network storage
GPU memory	46.15 GiB of weights; 69.1 of 79.3 GB used at gpu_memory_utilization=0.90
KV cache	108,544 tokens (BF16) at max_model_len=8192
decode, single request	0.91–0.96 tok/s
decode, 32 concurrent	~1.6 tok/s aggregate — only ~1.7× single
prefill	~49 s per forward pass + ~11 ms/token (512–4,096 tokens)

Decode is bandwidth-bound: ~28 GiB of expert weights cross PCIe per generated token, at 28.7 GB/s (91% of gen4 x16). That is also why batching buys so little — concurrent requests share the expert transfer. Prefill behaves differently: from 512 tokens up, a forward pass costs a roughly fixed ~49 s (one full sweep of the expert set) plus ~11 ms/token, so a 4,096-token prompt takes ~94 s — about 44 tok/s.

Evaluation

Everything below is at thinking_effort="low". Moonshot's official scores are at max effort, and no low-effort reference is published. We ran low effort because generation here is ~1 tok/s. So these are not like-for-like: each gap mixes quantisation loss with the effect of reduced reasoning, and we cannot separate the two. They are shown side by side so you can judge for yourself.

benchmark	this build (int4, low effort)	Kimi K3 official (full precision, max effort)
OCRBench (1,000 samples)	0.879 ± 0.010	0.89
GPQA Diamond (198 questions, 1 epoch)	0.843 ± 0.026	0.935

Both were run through Moonshot's own Kimi-Vendor-Verifier (commit 3dad65a), which Moonshot uses to check that third-party deployments serve an undegraded model, with its published K3 parameters (temperature 1.0, top_p 0.95).

OCRBench: all 1,000 samples, max_tokens 16384, no truncation, no errors. Per category from 98% (regular text) to 53% (handwritten mathematical expressions). The verifier's list of correctly-serving max-effort deployments spans 0.878–0.897, so this build at low effort lands inside that band.

GPQA Diamond: inspect_evals/gpqa_diamond (the port of OpenAI simple-evals) driven through the verifier's opensource provider, all 198 questions, 0-shot chain-of-thought, max_tokens 16384, 1 epoch — the harness averages 4 by default, so the ±0.026 is wider than a standard run's. 198/198 completed, 0 errors, nothing truncated. By domain: physics 93.0%, biology 78.9%, chemistry 77.4%. For reference, the benchmark's authors measured 65% for PhD experts answering within their own field.

BEAM (1M-token context) cannot run on this hardware: the KV cache holds ~108K tokens.

Known limitations

• Long context is limited in practice by the KV cache (~108K tokens at BF16) and by prefill cost. An fp8 KV cache would extend it, but vLLM refuses one on Ampere for K3's MLA layers: the check is written as kv_cache_dtype.startswith("fp8"), so it gates e5m2 as collateral even though Triton supports e5m2 on every NVIDIA GPU. This is a software limit, not a hardware one — a patched vLLM should lift it.
• Needs ~1.3 TB of host RAM, and loading it will fill your page cache — on a busy machine, host memory fragmentation can make the final cudaHostAlloc pin fail.

License

Kimi K3 is released by Moonshot AI under the Kimi K3 License. This quantised derivative is distributed under the same license, with the original LICENSE and copyright notice included unchanged. Note its conditions: a Model-as-a-Service business exceeding $20M in revenue over any consecutive 12 months requires a separate agreement with Moonshot AI, and products with more than 100M monthly active users or more than $20M in monthly revenue must prominently display "Kimi K3". See LICENSE.

Acknowledgements

All model capability is Moonshot AI's; this build only changes how the weights are stored.

Quantised and evaluated on the MLCore cluster at Beth Israel Deaconess Medical Center, whose compute made this possible. This is an independent side project and is not a BIDMC product or an endorsement by BIDMC.

Thanks to the vLLM and compressed-tensors projects, whose UVA offloader and Marlin kernels this build depends on.