GrEarl/Kimi-K3-GGUF

Author Note

The author does not own enough hardware to run this 864.81 GiB model. Full-model runtime results below were contributed by independent users and have not been reproduced by the author. They are reported with their environment and limitations rather than presented as author-run benchmarks.

Kimi-K3 GGUF — Q2_K experts / Q4_K dense (2.673 bpw)

864.81 GiB in 94 parts. Text model only. Regenerated 2026-07-28 to the llama.cpp PR #26185 (pwilkin/kimi-k3-text) tensor contract.

Source: moonshotai/Kimi-K3 at revision 9f62e4e9fffbd0a83ddd60e1c209d828994b3569 (pinned for every download).

Parts	94 (Kimi-K3-Q2_K-000NN-of-00094.gguf)
Size	864.81 GiB (source 1,453.8 GiB → 0.595x)
Tensors	2,573 (matches the contract exactly)
Effective rate	2.673 bpw
Architecture	kimi-k3
general.file_type	10 (MOSTLY_Q2_K)
Chat template	embedded, validated byte-exact against K3's own builder
Sub-block scale search	6 candidates, unweighted squared error
Previous build	superseded; pin revision d76240360965
imatrix	no

Quantization layout

ggml type	tensors	size	bpw	what
Q2_K	276	832.04 GiB	2.6250	routed expert stacks (ffn_{gate,up,down}_exps)
Q4_K	1,067	28.58 GiB	4.5000	dense 2-D weights, token_embd
F32	1,112	2.25 GiB	32	norms, router, ssm_a, ssm_conv1d, fused AttnRes scores
Q8_0	1	1.16 GiB	8.5	output.weight
F16	117	0.76 GiB	16	attn_k_b / attn_v_b (row length not a multiple of 256)

The experts hold 2.72 T of the 2.78 T parameters, so they set the overall rate. Dense weights are kept at Q4_K rather than Q2_K: they are 3.3% of the bytes, and spending 4.5 bpw there is cheap insurance for the layers that every token passes through. output.weight is promoted further, to Q8_0, because its error lands directly on the logits with nothing downstream to average it out. The Q8_0 tensor is 1.16 GiB and adds 0.54 GiB versus keeping it at Q4_K, or 0.06% of the artifact; upstream llama.cpp promotes the output head for low-bit file types as well.

Chat template

K3's tokenizer_config.json ships no chat template — conversation assembly lives in encoding_k3.py as code, so nothing could simply be copied across. The template embedded here is Xenova's Jinja port, patched for llama.cpp's Minja engine and validated against K3's own build_chat_segments(): 28 of 28 cases byte-exact. The cases cover normal and multi-turn messages, thinking_effort, tool declarations, assistant tool calls, tool results, and a nested response_schema.

The exact template extracted back from this GGUF was also rendered by llama.cpp's Minja implementation (commit 91f8c9c5): a tools/tool-result case was 1,218 bytes and a nested-schema case was 621 bytes, both byte-identical to K3's renderer. Images and batched conversations remain untested.

Provenance and independent conformance. The file in this artifact began with Xenova's port and was expanded and patched locally; it is not claimed to be byte-identical to the separate upstream implementation in Moonshot PR #66. PR #66 and ChatLint's K3 findings provide independent, oracle-backed conformance work. The Minja namespace(items=[]) collision reproduced here was accepted and fixed upstream in ChatLint commit ecb1a727 and on the PR branch. Against this artifact's 15,053-byte file, ChatLint pinned to that commit reports 294/294 checks, 0 errors, 0 warnings. ChatLint renders with transformers-style Jinja2 rather than Minja and checks structural properties, so it supplements rather than replaces the 28/28 oracle comparison and direct Minja runs above.

One limitation is shared with PR #66: sandboxed Jinja cannot parse a JSON string inside tool_calls[].function.arguments. This template emits a valid <|open|>json type="object" fallback for a non-empty string; callers that require the reference renderer's per-argument XTML must parse the string into an object before applying the template.

Measured quantization error

MXFP4 source dequantized to F32, requantized to Q2_K, then dequantized again and compared against the source. 92 expert tensors sampled, one per routed layer:

metric	value
relative RMSE, mean	0.2769
cosine similarity, mean	0.961095

The superseded min/max build measured 0.3309 relative RMSE / 0.951450 cosine on the same 92-tensor comparison. Those values are retained here as historical baseline numbers, not as measurements of the current files.

For reference, the same measurement on the IQ1_S sibling current top32/refit2 payload is 0.455936 / 0.890050 at 528.0293 GiB. This repo is 1.638x larger by file size and noticeably closer to the MXFP4 source in this weight-space measurement. Historical IQ1_S figures of 0.4626 / 0.886692 for the top8/refit1 payload, and 0.5375 / 0.848230 for the earlier ternary-space snap, are superseded and do not describe its current files.

The Q2_K encoder is a numpy implementation of block_q2_K (84 B / 256 elements). Its output bytes are fed to gguf-py's own dequantizer and compared against our dequantizer: max absolute difference 0.0, 0 mismatching elements, on both synthetic data and a real K3 expert tensor. The current encoder uses the 6-candidate unweighted squared-error search described below. The previous build used direct min/max; neither statement changes the standard block_q2_K byte format.

Sub-block scale search (applied 2026-07-28)

The sub-block scale is chosen by a 6-candidate search minimising unweighted squared error, not by taking the min/max of each 16-element sub-block. Measured over 30 expert tensors spanning layers 1, 11, 24, 36, 48, 60, 72, 84 and 92 (plus experts 100, 500 and 895 of layer 48), aggregated by element count:

sub-block scale search	rel_rmse
min/max — the previous build	0.3309
6 candidates, unweighted squared error — this build	0.2769 (−16.4%)
16 candidates	0.2703
16 candidates + super-block coordinate descent	0.2620

How much did it improve

One expert tensor was measured in every one of the 92 rebuilt parts, before and after, on identical inputs:

	previous build	this build
relative RMSE, mean	0.3309	0.2769
relative RMSE, range over 92 parts	0.3292 – 0.3323	0.2760 – 0.2776
cosine similarity, mean	0.951450	0.961095
cosine, range over 92 parts	0.950904 – 0.952143	0.960898 – 0.961344

• 16.3% less error (relative RMSE), equivalently 30.0% of the squared error removed — reconstruction error energy is 70.0% of what it was.
• Cosine similarity to the source weights rises by 0.965 percentage points.
• 0 of 92 parts got worse. Per-part reduction spans 16.2% to 16.5%, so the gain is uniform rather than concentrated in a few layers.
• File size, tensor count, tensor names, types and runtime-relevant metadata are identical. The one intentional metadata change is the provenance KV kimi-k3.conversion.q2k_subblock_search: minmax in the previous build and unweighted-sse nstep=5 in this build. Apart from that provenance field, only the routed-expert 2-bit payload differs; nothing about how the files load changes.

These are weight-space numbers. They say the quantised weights sit closer to the MXFP4 source; they do not say by how much generation improves, because no perplexity or benchmark comparison exists for either build.

Dropping ggml's weights=|x| helps because quantisation error propagates as Δy = ΔW·x, so E‖Δy‖² = Σ ΔW²·E[x²]; the correct weighting is the input second moment — an importance matrix — and the weight's own magnitude has no standing in that expansion. Absent an imatrix, an isotropic-input assumption makes plain ‖ΔW‖_F the consistent objective. Note this optimises exactly the metric reported, so read it as a bound on L2 reconstruction, not as downstream quality. No perplexity comparison exists, for this build or the previous one.

The two rows below this build were measured and not applied: 16 candidates costs more compute for a further 2.4%, and coordinate descent — which re-solves the fp16 d/dmin by least squares instead of pinning d = max_scale/15 — reaches 0.2620 but costs several times more. Per-tensor improvement is uniform (16.2%–16.9% across all 30 samples), so no subset of tensors carries the gain and selective effort allocation buys nothing.

The requantisation ran on NVIDIA A10/A10G hardware with CuPy and was verified byte-identical to the NumPy reference before use. The 92-part run took 38.5 minutes. Its result records incorrectly labelled each part as L4 and therefore reported $6.107 using the L4 rate; applying the code's A10 rate to the same recorded duration gives a corrected estimate of $7.660.

Previous build

The earlier min/max build was staged under v1/ during the switchover and has since been removed from the file listing to keep the repository to one copy. Its bytes remain reachable in the commit history: pin revision d76240360965, at which the repository root still held that build.

If you already downloaded the earlier files, they are still valid and still load. The difference is weight-space error only, with no measured effect on output quality, so re-downloading 864 GiB is a judgement call rather than a fix.

Runtime status: full 94-part load and generation reported on 8×B200

kimi-k3 is not merged into released llama.cpp; support exists on PR #26185 (pwilkin/llama.cpp, branch kimi-k3-text).

A detailed independent report in Discussion #4 (2026-07-28 UTC) exercised the current 94-part Q2_K root. The repository root was at d8aebda6c891e203284fc4ddef5483f2177ee841 when the report was posted; the reporter did not provide per-file hashes, so this attribution is based on the repository timeline rather than a cryptographic pin.

Reported environment and results:

Item	Third-party report
Hardware	AWS p6-b200.48xlarge: 8× NVIDIA B200 / 1.43 TiB aggregate HBM, sm_100
Runtime	pwilkin/llama.cpp kimi-k3-text at 06eec9f5
Load	All 94 parts loaded cleanly, fully VRAM-resident; 869 GiB resident, 97–116 GB per GPU
Startup	About 110 seconds from launch to serving from local NVMe
Decode	17.5–17.8 tokens/s, reported stable across tasks
Prompt processing	Up to about 330 tokens/s with -ub 4096
Generation checks	At temperature 0, the reporter observed coherent factual answers, instruction following, and correct handling of a simple trap question
Tools	The model emitted well-formed tool calls with correctly typed arguments
Context	A 262K context allocation ran with little VRAM change and unchanged decode speed; needle retrieval was tested at 26K tokens, not 262K
Input rendering	The reporter confirmed that the 28/28-tested prompt template rendered correctly in practice

These are third-party observations, not author-run measurements. They establish full load, serving, generation, raw tool-call emission, and a 26K retrieval check on the named setup. They do not establish perplexity, benchmark scores, source-model equivalence, or 262K retrieval quality. The statement that no quantization damage was visible is a qualitative user observation, not a formal quality result.

Known runtime limitations from the same report

• CPU-only currently crashes while loading K3 at GGML_ASSERT(*cur_backend_id != -1) in ggml_backend_sched_split_graph. At present use a GPU or GPU+CPU hybrid configuration; the traditional all-CPU large-RAM/mmap route is not validated.
• The tested branch lacked a K3 output parser. Without the reporter's parser PR, /v1/chat/completions can return empty content, place output in reasoning_content, and leak raw XTML markers. This is an output-parsing issue, not evidence that the embedded input template is wrong.
• special_eos_id is not in special_eog_ids was reported as a benign warning; raw /completion stopped with stop_type: eos. The metadata has not been changed solely to silence that warning.
• Released/stock llama.cpp, CPU-only execution, Metal, Vulkan, and other hardware configurations remain unverified for this artifact.

What has been verified, and by whom:

Level	Status
GGUF structure, split metadata, tensor names/types, hparams, vocab	Verified by the author on the published files via HTTP Range reads
Numeric agreement of custom quant bytes with gguf-py's dequantizer	Verified by the author, exact
Chat input template vs K3 build_chat_segments()	Verified by the author, 28/28 byte-exact; tools/schema rendered directly with Minja
Full 94-part load, serving, generation, tool emission on 8×B200	Reported by an independent user in Discussion #4; not reproduced by the author
262K context allocation / 26K needle retrieval	Reported by the same user; retrieval was not tested at 262K
Perplexity, standardized benchmark, source-model output equivalence	Not measured

Template compatibility fix

The first template published here used Jinja's namespace({...}) dict-literal form. Minja accepts only namespace(key=value, ...). Direct Minja execution then found two more tool/schema-path issues: tojson(sort_keys=true) is not implemented, and a namespace field named items collides with the object method. The current 15,053-byte template uses recursive dictsort JSON rendering and a non-conflicting field name. It is embedded in part 1 and also published as chat_template.jinja. Thanks to the user who caught the original incompatibility.

What was fixed relative to the previous 96-part upload

The earlier Q2_K release in this repo (96 parts, 938.59 GiB) could not have been loaded even with K3 support present. All four defects required regenerating every part, so this upload replaces it entirely.

1. Tensor names were raw Hugging Face names. 668 names across 92 of the 96 parts exceeded GGML_MAX_NAME (64). Now every name follows the PR contract; longest is 29 characters.
2. Part 1 had no model metadata. No hparams, no vocabulary. Now part 1 carries 66 KV entries: 25 required hparams plus the full vocabulary (163,840 tokens / 163,328 merges, gpt2 / pre kimi-k2, chkhsh verified).
3. split.tensors.count was UINT16 and per-part. llama-model-loader.cpp reads it as a required key and compares it against weights_map.size(), throwing corrupted model on mismatch. Now INT32 / 2573 in all 94 parts, equal to the actual sum of per-part tensor counts.
4. Vision tensors were mixed into the text model. HF shards 95 and 96 hold 168 vision_tower / mm_projector tensors. done_getting_tensors() is called with partial=false, so a single extra tensor triggers wrong number of tensors. Those two shards are excluded, which is why there are 94 parts and not 96.

Conversion details

Recorded in the file itself under kimi-k3.conversion.*:

• a_log_transform = -exp(A_log[:n_head]) — K3 stores A_log as 128 elements but only the first num_attention_heads = 96 are used
• kv_b_split = k_b(transposed)+v_b — kv_b_proj split into attn_k_b / attn_v_b
• attn_res_fused = res_norm*res_proj[0] in float32 — the AttnRes norm/proj pair is fused into one score vector, matching the reference _apply_attn_res()
• expert_stack_dim = 0 — 896 experts stacked on dim 0
• source_quant = compressed-tensors/mxfp4-pack-quantized
• defaults_used = rope_theta — the only value not present in config.json, taken from the configuration_kimi_k3.py class default (10000.0)

Usage on the PR branch

Pass the first part; llama.cpp follows the split metadata to the rest.

llama-cli -m Kimi-K3-Q2_K-00001-of-00094.gguf -p "..." 

All 94 parts must be present in the same directory.