Kimi-K3-Tiny-L13

A reduced-depth Kimi-K3 built for systems work on a single GPU. 13 of the original 93 decoder layers, 365.8 B params, 195.6 GiB — it fits on one B300 with ~73 GiB left for state and activations.

⚠️ This model does not produce meaningful text

The network is truncated, so the logits are garbage (typical output: "of of of of"). There is no point running accuracy benchmarks on it, and it should not be used to judge anything about Kimi-K3's quality. It exists so that serving-infrastructure work — kernel profiling, expert-parallel sharding, attention-backend selection, memory accounting — can be done without 6 GPUs and 1.56 TB of disk.

What is preserved

Depth is the only knob that changed. Every other dimension is held at its real value, because each one determines GEMM tile efficiency, TP sharding, all-to-all message size, or cache layout:

	value
hidden size	7168
experts / top-k	896 / 16 (unchanged)
expert intermediate	3072 × 3584, MXFP4 group 32
KDA heads	96 × 128
MLA	1536 / 512
latent	3584
vocab	163840
attn_res_block_size	12

Expert count matters most: reducing it would change EP sharding and all-to-all behaviour, which is exactly what this checkpoint is meant to study.

Four structural invariants of the architecture survive the reduction:

1. KDA : MLA = 3 : 1 by the real rule — full attention every 4th layer plus the last layer. Here: full attention at 0-indexed 3, 7, 11, 12 (9 KDA / 4 MLA).
2. Layer 0 dense, all remaining layers MoE.
3. At least 2 residual blocks — otherwise _apply_attn_res degenerates to one slot and the softmax-mixing path never runs. Residual blocks land at 0, 12.
4. MXFP4 packed/scale layout unchanged (weight_packed u8 + E8M0 weight_scale u8, group 32).

Invariant 3 sets a hard floor of L ≥ 13, which is why 13 layers: it is the minimum depth that keeps all four properties, and it also reproduces the real model's "tail quirk" (layers 91 and 92 are both full attention).

The weights are real

This is a partial download, not a synthesis job. The release stores one decoder layer per shard, so a shallow variant can be assembled from real shards:

Source shard	→	Contents
model-00001 … model-00012	shards 1–12	real layers 0–11, whose types are already correct
model-00093	shard 13	real layer 92, renamed to layer 12
model-00094	shard 14	embed_tokens, lm_head, norm, output_attn_res_*
model-00095, model-00096	shards 15–16	mm_projector, vision_tower

Shard 13 is real layer 92, not real layer 12 — layer 12 is KDA and would break invariant 1, while layer 92 is full attention and the real model's final layer, making it the closest substitute. Renaming layers.92. → layers.12. was done by patching the safetensors header in place: both indices are two digits, so the JSON header keeps its byte length and every data_offsets entry stays valid — 5,398 tensor names rewritten without touching the 16.6 GB payload.

Why real weights matter here: the router weights and e_score_correction_bias are real, so expert load imbalance is real. Random routing gives a uniform expert load, which is precisely the wrong baseline for EPLB / all-to-all work — it would make any load-balancing optimization look better than it is. Real MXFP4 E8M0 scales and real KDA A_log / dt_bias also remove any risk of inf/NaN or degenerate decay dynamics from synthetic initialization.

Known fidelity gaps

Read results off this checkpoint with these in mind:

• KDA : MLA is 2.25, not 2.88 (30.8% vs 25.8% full-attention layers). The "+ last layer" special case is proportionally heavier in a shallow stack, which skews the ratio between KDA recurrent state and MLA KV cache in per-sequence memory.
• The block residual only reaches K = 2, and only 2 _apply_attn_res calls see K ≥ 2, versus 162 in the real model. The code path is exercised; the magnitude is not. No shallow model can fix this — K = 8 needs L ≥ 85. If block-residual activation memory is itself the target, the knob is attn_res_block_size, not depth.
• Layer 12 sees off-distribution input: it receives hidden states from layer 11 rather than layer 91, so its router inputs — and therefore its expert load — are less representative than the other 11 MoE layers'.

Serving

Fits one B300 / B200-class GPU at TP=1. With SGLang:

python -m sglang.launch_server \
  --model-path Qinghao/Kimi-K3-Tiny-L13 \
  --trust-remote-code \
  --tp-size 1 \
  --prefill-attention-backend trtllm_mla \
  --decode-attention-backend cutedsl_mla

trtllm_mla decode does not work at TP=1 here: with 96 Q-heads on a single rank the MQA-mode group size is unsupported ("numHeadsQ/numHeadsKv is not supported" during autotune). That shape is an artifact of running a tiny model without head splitting; at TP=8 trtllm_mla decode works fine. Hence cutedsl_mla for decode above.

Measured on 1× B300: ~230 tok/s single-stream decode, ~3.1k tok/s at batch 43, 234 GB resident. Concurrency is capped near 43 by the KDA/mamba state pool, not by the KV cache (which sits ~0% used) — relevant knobs are --mamba-full-memory-ratio, --max-mamba-cache-size, --mamba-ssm-dtype.

Other presets

Preset	Layers	KDA / MLA	K:M	Blocks	Tail quirk	Size	Params	Fits
K3 (real)	93	69 / 24	2.88	8	✅	1560.8 GB	2779.9 B	6 × B300
tiny-L13	13	9 / 4	2.25	2	✅	210.0 GB	365.8 B	1 × B300
tiny-L16	16	12 / 4	3.00	2	✗	261.0 GB	456.5 B	1 × B300 (tight)
tiny-L25	25	18 / 7	2.57	3	✅	412.6 GB	727.9 B	2 × B300

"Tail quirk" = the last two layers are both full attention, as in the real model. It appears whenever L % 4 != 0.

License

Inherits the Kimi K3 License, © 2026 Moonshot AI, which permits distribution of derivative works provided the copyright and permission notice is retained. All weights here are unmodified real weights from moonshotai/Kimi-K3; the only change is which layers are present and the renaming of one layer index.