Kimi Linear: An Expressive, Efficient Attention Architecture
Authors: name: Kimi Team; name: Yu Zhang; name: Zongyu Lin; name: Xingcheng Yao; name: Jiaxi Hu; name: Fanqing Meng; name: Chengyin Liu; name: Xin Men; name: Songlin Yang; name: Zhiyuan Li; name: Wentao Li; name: Enzhe Lu; name: Weizhou Liu; name: Yanru Chen; name: Weixin Xu; name: Longhui Yu; name: Yejie Wang; name: Yu Fan; name: Longguang Zhong; name: Enming Yuan; name: Dehao Zhang; name: Yizhi Zhang; name: T. Y. Liu; name: Haiming Wang; name: Shengjun Fang; name: Weiran He; name: Shaowei Liu; name: Yiwei Li; name: Jianlin Su; name: Jiezhong Qiu; name: Bo Pang; name: Junjie Yan; name: Zhejun Jiang; name: Weixiao Huang; name: Bohong Yin; name: Jiacheng You; name: Chu Wei; name: Zhengtao Wang; name: Chao Hong; name: Yutian Chen; name: Guanduo Chen; name: Yucheng Wang; name: Huabin Zheng; name: Feng Wang; name: Yibo Liu; name: Mengnan Dong; name: Zheng Zhang; name: Siyuan Pan; name: Wenhao Wu; name: Yuhao Wu; name: Longyu Guan; name: Jiawen Tao; name: Guohong Fu; name: Xinran Xu; name: Yuzhi Wang; name: Guokun Lai; name: Yuxin Wu; name: Xinyu Zhou; name: Zhilin Yang; name: Yulun Du
Published: 2025-10-30 16:59:43

Abstract: We introduce Kimi Linear, a hybrid linear attention architecture that, for
the first time, outperforms full attention under fair comparisons across
various scenarios -- including short-context, long-context, and reinforcement
learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an
expressive linear attention module that extends Gated DeltaNet with a
finer-grained gating mechanism, enabling more effective use of limited
finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware
efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR)
transition matrices, which substantially reduces computation compared to the
general DPLR formulation while remaining more consistent with the classical
delta rule.
  We pretrain a Kimi Linear model with 3B activated parameters and 48B total
parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention
(MLA). Our experiments show that with an identical training recipe, Kimi Linear
outperforms full MLA with a sizeable margin across all evaluated tasks, while
reducing KV cache usage by up to 75% and achieving up to 6 times decoding
throughput for a 1M context. These results demonstrate that Kimi Linear can be
a drop-in replacement for full attention architectures with superior
performance and efficiency, including tasks with longer input and output
lengths.
  To support further research, we open-source the KDA kernel and vLLM
implementations, and release the pre-trained and instruction-tuned model
checkpoints.