DiogenesLab/kimi-k3-research

Kimi K3 — Research Notes & Open-Source Capability Inventory

Study notes on Kimi K3: Open Frontier Intelligence (Moonshot AI, July 2026) — the first open 3T-class model — plus a verified inventory of everything the K3 release actually open-sourced. Maintained by Diogenes / Oratis.

Kimi K3 is a 2.78T-parameter / 104.2B-activated natively multimodal MoE with a 1M-token context window. It is interesting well beyond its size: it solves three problems that normally appear separately — sequence-length scaling, extreme MoE width, and depth-wise information flow — and each solution detaches cleanly from the rest of the model.

These notes are written to be learned from, not just skimmed: the emphasis is on why each design exists and what failure mode it removes.

Contents

File	What it is
k3_architecture_notes.md	Architecture deep dive — Kimi Delta Attention and the lower-bounded decay trick, Attention Residuals, Stable LatentMoE (Normalized LatentMoE + SiTU-GLU + Quantile Balancing), MoonViT-V2 trained from scratch, Per-Head Muon. Includes the K2 → K3 spec diff.
k3_training_and_infra_notes.md	Pre-training (data, scaling law methodology, the four-stage 8K→1M context curriculum), post-training (SFT → 9 domain × effort RL experts → multi-teacher on-policy distillation, MXFP4 QAT, EAGLE-3 draft + LK loss), RL environments and task synthesis, and the infrastructure (FlashKDA, KDA Context Parallelism, MoonEP, AgentENV, KDA-aware prefix caching).
k3_open_source_inventory.md	The practical artifact. Every open-sourced component of the K3 release, each independently verified against its HF/GitHub page: what it does, its license, its hardware requirements, and how it maps back to a section of the technical report. Includes a breakdown of the Kimi K3 License, which is not MIT.
k3_evaluation_summary.md	Where K3 actually lands — public benchmarks, in-house suites, the cyber-capability evaluation, third-party results (Artificial Analysis, Vals AI, LMArena), and cost-efficiency.

Three things worth knowing if you read nothing else

1. A parameterization change bought an order of magnitude of hardware utilization. KDA's chunkwise form needs to rescale keys by the reciprocal cumulative decay 1/Γ, which grows without bound and overflows in low precision — so the predecessor (Kimi Linear) had to compute diagonal tiles with an explicit position-pair path that cannot use Tensor Cores. K3 bounds the log-decay from below with a scaled sigmoid (g_min = −5), which puts the cumulative decay over a 16-token tile in (−80, 0) and the reciprocal below e^80 — inside BF16's dynamic range. Diagonal and off-diagonal tiles now both run as dense Tensor Core matmuls, and the special-case path is deleted. The algorithm did not change; only the range of one quantity did.

2. Contrastive vision pre-training turned out to be unnecessary as an initialization. K3's vision tower, MoonViT-V2, is trained entirely from scratch with next-token prediction rather than initialized from SigLIP. The reported motivation is stability — the SigLIP-initialized baseline shows persistently higher vision-tower gradient norms with frequent spikes throughout joint optimization — and MoonViT-V2 matches it on vision evaluations anyway. This is a direct challenge to a default design choice in multimodal LLMs. (Caveat worth keeping: this is evidence at 2.8T scale with a full multimodal corpus; it does not automatically transfer to small-scale fine-tuning regimes.)

3. The open-source surface is much larger than the weights. Six engineering repositories ship under MIT or Apache-2.0 — including the attention kernels, the expert-parallelism library, and the microVM sandbox platform that powered the agentic RL — while the weights themselves carry a different, more restrictive license. Two of the repos (minitriton, nano-kpu) were written by K3 itself and are explicitly labeled demonstrations rather than products. See k3_open_source_inventory.md.

Notes on method

• Primary source is the 47-page Kimi K3 technical report (Kimi Team, Moonshot AI). Every open-source claim in the inventory was independently re-verified against the live HF or GitHub page rather than transcribed from the report — a few details (exact hardware requirements, merge status of the upstream FLA context-parallel PR, license thresholds) are only available there.
• Claims are marked [report] (stated in the technical report), [verified] (independently checked against a live page), or [analysis] (our inference, not a claim of the original authors).
• This is a curated public subset of a larger internal research effort. Organization-specific roadmap judgments are intentionally not included; the focus here is the general method, the mechanisms, and the reusable artifacts.
• Last refreshed 2026-07-29.

Citation

The underlying work is Moonshot AI's:

@techreport{kimi2026k3,
  title  = {Kimi K3: Open Frontier Intelligence},
  author = {Kimi Team},
  year   = {2026},
  institution = {Moonshot AI},
  url    = {https://www.kimi.com/blog/kimi-k3}
}

License

These notes released under CC BY 4.0. The Kimi K3 model, its technical report, and all cited repositories belong to their respective authors under their own licenses — see k3_open_source_inventory.md for the per-artifact breakdown.