Kimi Linear: An Expressive, Efficient Attention Architecture
Authors: name: Kimi Team, name: Yu Zhang, name: Zongyu Lin, name: Xingcheng Yao, name: Jiaxi Hu, name: Fanqing Meng, name: Chengyin Liu, name: Xin Men, name: Songlin Yang, name: Zhiyuan Li, name: Wentao Li, name: Enzhe Lu, name: Weizhou Liu, name: Yanru Chen, name: Weixin Xu, name: Longhui Yu, name: Yejie Wang, name: Yu Fan, name: Longguang Zhong, name: Enming Yuan, name: Dehao Zhang, name: Yizhi Zhang, name: T. Y. Liu, name: Haiming Wang, name: Shengjun Fang, name: Weiran He, name: Shaowei Liu, name: Yiwei Li, name: Jianlin Su, name: Jiezhong Qiu, name: Bo Pang, name: Junjie Yan, name: Zhejun Jiang, name: Weixiao Huang, name: Bohong Yin, name: Jiacheng You, name: Chu Wei, name: Zhengtao Wang, name: Chao Hong, name: Yutian Chen, name: Guanduo Chen, name: Yucheng Wang, name: Huabin Zheng, name: Feng Wang, name: Yibo Liu, name: Mengnan Dong, name: Zheng Zhang, name: Siyuan Pan, name: Wenhao Wu, name: Yuhao Wu, name: Longyu Guan, name: Jiawen Tao, name: Guohong Fu, name: Xinran Xu, name: Yuzhi Wang, name: Guokun Lai, name: Yuxin Wu, name: Xinyu Zhou, name: Zhilin Yang, name: Yulun Du
Organization: Moonshot AI

We introduce Kimi Linear, a hybrid linear attention architecture that, for
the first time, outperforms full attention under fair comparisons across
various scenarios -- including short-context, long-context, and reinforcement
learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an
expressive linear attention module that extends Gated DeltaNet with a
finer-grained gating mechanism, enabling more effective use of limited
finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware
efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR)
transition matrices, which substantially reduces computation compared to the
general DPLR formulation while remaining more consistent with the classical
delta rule.
  We pretrain a Kimi Linear model with 3B activated parameters and 48B total
parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention
(MLA). Our experiments show that with an identical training recipe, Kimi Linear
outperforms full MLA with a sizeable margin across all evaluated tasks, while
reducing KV cache usage by up to 75% and achieving up to 6 times decoding
throughput for a 1M context. These results demonstrate that Kimi Linear can be
a drop-in replacement for full attention architectures with superior
performance and efficiency, including tasks with longer input and output
lengths.
  To support further research, we open-source the KDA kernel and vLLM
implementations, and release the pre-trained and instruction-tuned model
checkpoints.