# Code study — KDA background (Kimi Linear)

Paper: **arXiv:2510.26692** — *Kimi Linear: An Expressive, Efficient Attention Architecture* (Kimi Team, 2025).  
Local metadata: `reference/papers/kimi-linear-2510.26692.atom.xml`.

This is **prior art** for the mechanism named in K3, **not** the K3 tech report.

---

## Why it matters for K3

K3’s long-context story is built on **Kimi Delta Attention**. Understanding KDA explains:

- How 1M context can be economically served  
- Why classic **prefix KV cache** assumptions break  
- Why Moonshot invested in **vLLM KDA kernels**  

---

## Core ideas (from abstract / paper framing)

1. **KDA** extends **Gated DeltaNet** with finer-grained gating → better use of finite-state RNN memory  
2. Chunkwise algorithm uses a specialized **Diagonal-Plus-Low-Rank (DPLR)** transition — cheaper than general DPLR, closer to classical delta rule  
3. **Kimi Linear** architecture: layerwise hybrid of **KDA + Multi-Head Latent Attention (MLA)**  
4. Results (paper’s Kimi Linear 3B-active / 48B-total model):  
   - Outperforms full MLA under matched recipes  
   - Up to **~75%** less KV cache  
   - Up to **~6×** decoding throughput at **1M** context  

---

## Hybrid stacking (paper-era)

Community summaries of Kimi Linear often describe **3 KDA : 1 MLA**.  
Treat as **Linear-era**, not automatically K3’s ratio.

K3 blog additionally names **Gated MLA** and **AttnRes** — depth-side mechanisms beyond the Linear paper’s headline.

---

## Open-source artifacts (Linear)

Paper states open-sourcing of:

- KDA kernel  
- vLLM implementations  
- Pretrained + instruction-tuned **Kimi Linear** checkpoints  

Useful for studying kernels before K3 weights drop.

---

## Reading order

1. Abstract + intro of arXiv:2510.26692  
2. KDA module definition + DPLR specialization  
3. Hybrid KDA/MLA experiments  
4. Long-context / decode throughput tables  
5. Re-read K3 blog “Architecture and Infrastructure” with that vocabulary  

---

## What **not** to assume

| Assumption | Status |
|---|---|
| K3 layer count = Linear model | False |
| K3 is 48B total | False (2.8T) |
| K3 interleave is exactly 3:1 | Unconfirmed |
| Active params scale linearly from Linear | Unknown |

When the K3 report ships, add a `code-study/03-tech-report.md` that supersedes speculative mapping.