Kuberwastaken/Kimi-K3-GGUF

---

What this is

GGUF conversion assets for moonshotai/Kimi-K3 — Moonshot AI's 2.8T-parameter (104B active) multimodal MoE.

This repo hosts the complete vision encoder, the full tokenizer and architecture metadata, and a converter proven lossless across the entire 1.419 TiB model. It does not re-host the expert weights: those already live in Moonshot's repo, and the converter streams from there so you get identical bytes without a second terabyte sitting on the Hub.

Nothing here runs yet. llama.cpp has no kimi_k3 architecture, so no GGUF of this model — from anyone — will load in llama-cli today. These files are correct, verified and ready for the moment support lands. See What's blocking it; the gap is smaller than the model's size suggests.

Contents

File	Size	What it is
mmproj-Kimi-K3-BF16.gguf	853 MiB	Complete 27-layer MoonViT vision tower + patchmergerv2 projector, lossless BF16
Kimi-K3-tokenizer.gguf	6.6 MiB	163,840-token vocab, 163,328 merges, all kimi-k3.* hyperparameters — no tensors
MANIFEST.json	458 KiB	SHA-256 of all 2,760 output tensors, so you can verify your own conversion
convert_kimi_k3.py	—	Streaming, resumable text-model converter
convert_vision.py	—	Vision tower → mmproj
convert_tokenizer.py	—	tiktoken → GGUF vocab
modal_run.py	—	Cloud fan-out (94 containers, ~5 min wall-clock)
k3lib.py · k3meta.py · k3vocab.py	—	MXFP4 repack, arch metadata, vocab reconstruction
TENSOR_MAP.md	—	Every tensor: HF name → GGUF name, with transforms

---

Setup

pip install numpy gguf huggingface_hub
hf download Kuberwastaken/Kimi-K3-GGUF --local-dir k3gguf
cd k3gguf

That's enough to use the vision encoder, tokenizer and manifest. To build the full model you additionally need the metadata files from the source repo:

hf download moonshotai/Kimi-K3 --local-dir meta \
  config.json model.safetensors.index.json tokenizer_config.json tiktoken.model

Requirements for a full conversion

Disk (working)	~32 GiB — one input shard + one output shard
Disk (final)	~1.42 TiB for the finished GGUF
RAM	~6 MiB of tensor data at a time (experts stream through a memmap)
Time	~4 min per shard; 94 shards, fully parallelisable
GPU	none — this is a byte-level repack, not a compute job

There is no fp16 intermediate at any point. A dequantise-then-requantise pipeline would need ~5.6 TB of scratch; this needs one shard.

---

Operating guide

Convert the whole model

python convert_kimi_k3.py --meta ./meta --out-dir ./out --shards 1-94

Output is Kimi-K3-MXFP4-000NN-of-00094.gguf. Weights stream from the Hub automatically; pass --src ./local-copy if you already have the safetensors.

Convert a piece at a time

Source shard k holds exactly layer k−1, and no tensor spans two shards. The 94 units are therefore fully independent — interrupt, resume, or spread them across machines freely.

python convert_kimi_k3.py --meta ./meta --out-dir ./out --shards 2      # layer 1
python convert_kimi_k3.py --meta ./meta --out-dir ./out --shards 2,5,9  # a few
python convert_kimi_k3.py --meta ./meta --out-dir ./out --shards 40-60  # a range

Shard	Holds
1	layer 0 (dense MLP) + all model KV metadata + tokenizer
2–93	layers 1–92, one per shard, 896 experts each
94	token_embd, output, output_norm, AttnRes output tensors

Options

Flag	Default	Meaning
--meta	src	Directory with config.json, index, tiktoken.model
--src	(stream)	Local safetensors copy; omit to pull from the Hub
--out-dir	out	Where GGUF shards are written
--shards	1-94	N, A-B, or a comma-separated mix
--scratch	(out-dir)	Expert memmap scratch; point at a fast disk

In the cloud

modal_run.py fans the same work across 94 containers (~$4 of Modal credits, ~5 min wall-clock). Create a huggingface secret holding HF_TOKEN first — 94 unauthenticated containers will hit Hub rate limits.

modal secret create huggingface HF_TOKEN=hf_...
modal run modal_run.py::pilot            # one layer, verified, no upload
modal run modal_run.py::convert_all      # all 94
modal run modal_run.py::build_manifest   # all 94 + SHA-256 manifest

Rebuild the small artifacts

python convert_vision.py    --src ./meta --out mmproj-Kimi-K3-BF16.gguf
python convert_tokenizer.py --src ./meta --out Kimi-K3-tokenizer.gguf

convert_vision.py needs shards 95 and 96 (the projector and vision tower).

---

Why the conversion is lossless

Kimi-K3 ships natively in MXFP4 (compressed-tensors, mxfp4-pack-quantized, group size 32, E8M0 uint8 scales). GGML's block_mxfp4 stores exactly the same information:

source:  packed nibbles [K/2 bytes]  +  E8M0 scale [K/32 bytes]
ggml:    block_mxfp4 { uint8 e; uint8 qs[16]; }  ×  K/32

For one expert tensor that is 5,505,024 + 344,064 = 5,849,088 bytes on both sides — identical. The conversion is a nibble permutation (the source packs consecutive pairs 2k/2k+1; GGML packs j/j+16) with the scale byte copied verbatim. No dequantisation, no intermediate, no precision loss.

The scale byte transfers untouched because GGML's half-scale × doubled-kvalues convention is algebraically identical to the OCP MX interpretation: E8M0_TO_FP32_HALF(e) × (2·e2m1) == 2^(e-127) × e2m1.

One tensor's values do change. A_log → ssm_a is stored pre-transformed as -exp(A_log), matching llama.cpp's kimi_linear.py. Every other tensor in the model is bit-preserved.

Verification

Nothing above is asserted without a check:

What	Result
MXFP4 repack vs. gguf-py's own dequantizer, real K3 tensor	0 mismatches / 11,010,048 elements, max abs diff 0.0
Source vs. GGML byte counts, expert tensor	5,849,088 == 5,849,088
mmproj vs. source safetensors	168/168 tensors byte-identical, shapes correct
Layer 0 shard vs. source	22/22 byte-identical (2,232 MiB), ssm_a == -exp(A_log) exact
Layer 1 shard, 896 stacked experts	0 mismatches, 15.82 GiB, 31 tensors
Full model — 94 shards, 1.419 TiB streamed	0 mismatches, 2,760/2,760 tensors

Two independent implementations are compared for the MXFP4 path: a NumPy reference written from the compressed-tensors semantics, and llama.cpp's own gguf.quants.MXFP4 dequantizer. They agree bit-for-bit.

Verify your own conversion

MANIFEST.json records the SHA-256 of the canonical GGML bytes for all 2,760 tensors, with ggml type and ne:

"blk.1.ffn_down_exps.weight": { "sha256": "…", "type": 39, "ne": [3072, 3584, 896] }

So you never have to take this on faith:

import hashlib, json
from gguf import GGUFReader

man = json.load(open("MANIFEST.json"))["tensors"]
for t in GGUFReader("out/Kimi-K3-MXFP4-00002-of-00094.gguf").tensors:
    assert hashlib.sha256(t.data.tobytes()).hexdigest() == man[t.name]["sha256"], t.name
print("match")

Composition: 1,978 BF16 + 506 F32 + 276 MXFP4 (stacked experts, 92 layers × 3) = 2,760 tensors, 1,559,972,708,032 bytes. The ~846 MiB difference from the source index is the vision tower, which lives in the mmproj file instead.

---

Warnings and gotchas

It will not load in llama.cpp yet. general.architecture is kimi-k3, which yields an honest "unknown architecture" error. Declaring kimi-linear would make llama.cpp attempt a graph that cannot represent this model.

• You still need the disk. GGUF has no remote-pointer mechanism — tensor bytes live inside the file. Streaming avoids re-hosting, not downloading.
• Don't run 94 containers unauthenticated. Set HF_TOKEN or you'll hit Hub rate limits partway through.
• --scratch on a slow disk will dominate runtime. Each layer stages a ~5.2 GiB expert memmap.
• Tensor names are a considered guess for K3-only tensors. AttnRes and LatentMoE have no upstream precedent yet; whoever lands the kimi_k3 PR picks the final names. Everything with a kimi-linear equivalent already uses it. Full table in TENSOR_MAP.md.
• gguf-py shape trap. Passing a uint8 buffer with raw_dtype makes gguf-py treat raw_shape as a byte shape and divide by the type size, silently halving the declared row length. Pass a 2-byte view for BF16.

What's blocking it

llama.cpp has no kimi_k3 architecture — but it already ships LLM_ARCH_KIMI_LINEAR with a full llama_model_kimi_linear implementation, and K3's text_config.architectures is literally ["KimiLinearForCausalLM"]. Kimi Delta Attention and MLA are already implemented upstream.

The remaining delta is narrower than 2.8T parameters suggests:

• AttnRes — extra residual pathway (attn_res_*, ffn_res_*, output_attn_res_*), attn_res_block_size = 12
• LatentMoE — routed experts run in a 3,584-dim latent space, not model space
• situ activation — β = 4.0, linear β = 25.0
• 2 shared experts (Kimi-Linear has 1) and 896 routed, 16 active
• output gate on every layer (mla_use_output_gate)
• NoPE — mla_use_nope = true, no rope_theta anywhere in the config
• the multimodal wrapper (KimiK3ForConditionalGeneration)

Tensor names here deliberately follow the existing kimi-linear conventions so that delta stays as small as possible.

Model facts

Parameters	2.8 T total / 104 B active
Layers	93 — 24 MLA + 69 KDA
Hidden size	7,168
Experts	896 routed (16 active) + 2 shared
Expert latent dim	3,584
Vocabulary	163,840 (tiktoken, kimi-k2 pre-tokenizer)
Context	1,048,576
Vision	27-layer MoonViT, patch 14, sd2_tpool merge
Native precision	MXFP4 experts, BF16 everything else

Credits

Model and logo by Moonshot AI — the weights this repo converts live at moonshotai/Kimi-K3. Conversion conventions follow llama.cpp's kimi_linear.py and gguf-py. Licensed under the upstream Kimi-K3 License.

---

Built with ♥ by Kuber Mehta (kuberwastaken)