MiniCPM4: Ultra-Efficient LLMs on End Devices
Authors: name: MiniCPM Team; name: Chaojun Xiao; name: Yuxuan Li; name: Xu Han; name: Yuzhuo Bai; name: Jie Cai; name: Haotian Chen; name: Wentong Chen; name: Xin Cong; name: Ganqu Cui; name: Ning Ding; name: Shengdan Fan; name: Yewei Fang; name: Zixuan Fu; name: Wenyu Guan; name: Yitong Guan; name: Junshao Guo; name: Yufeng Han; name: Bingxiang He; name: Yuxiang Huang; name: Cunliang Kong; name: Qiuzuo Li; name: Siyuan Li; name: Wenhao Li; name: Yanghao Li; name: Yishan Li; name: Zhen Li; name: Dan Liu; name: Biyuan Lin; name: Yankai Lin; name: Xiang Long; name: Quanyu Lu; name: Yaxi Lu; name: Peiyan Luo; name: Hongya Lyu; name: Litu Ou; name: Yinxu Pan; name: Zekai Qu; name: Qundong Shi; name: Zijun Song; name: Jiayuan Su; name: Zhou Su; name: Ao Sun; name: Xianghui Sun; name: Peijun Tang; name: Fangzheng Wang; name: Feng Wang; name: Shuo Wang; name: Yudong Wang; name: Yesai Wu; name: Zhenyu Xiao; name: Jie Xie; name: Zihao Xie; name: Yukun Yan; name: Jiarui Yuan; name: Kaihuo Zhang; name: Lei Zhang; name: Linyue Zhang; name: Xueren Zhang; name: Yudi Zhang; name: Hengyu Zhao; name: Weilin Zhao; name: Weilun Zhao; name: Yuanqian Zhao; name: Zhi Zheng; name: Ge Zhou; name: Jie Zhou; name: Wei Zhou; name: Zihan Zhou; name: Zixuan Zhou; name: Zhiyuan Liu; name: Guoyang Zeng; name: Chao Jia; name: Dahai Li; name: Maosong Sun
Published: 2025-06-09 16:16:50

Abstract: This paper introduces MiniCPM4, a highly efficient large language model (LLM)
designed explicitly for end-side devices. We achieve this efficiency through
systematic innovation in four key dimensions: model architecture, training
data, training algorithms, and inference systems. Specifically, in terms of
model architecture, we propose InfLLM v2, a trainable sparse attention
mechanism that accelerates both prefilling and decoding phases for long-context
processing. Regarding training data, we propose UltraClean, an efficient and
accurate pre-training data filtering and generation strategy, and UltraChat v2,
a comprehensive supervised fine-tuning dataset. These datasets enable
satisfactory model performance to be achieved using just 8 trillion training
tokens. Regarding training algorithms, we propose ModelTunnel v2 for efficient
pre-training strategy search, and improve existing post-training methods by
introducing chunk-wise rollout for load-balanced reinforcement learning and
data-efficient tenary LLM, BitCPM. Regarding inference systems, we propose
CPM.cu that integrates sparse attention, model quantization, and speculative
sampling to achieve efficient prefilling and decoding. To meet diverse
on-device requirements, MiniCPM4 is available in two versions, with 0.5B and 8B
parameters, respectively. Sufficient evaluation results show that MiniCPM4
outperforms open-source models of similar size across multiple benchmarks,
highlighting both its efficiency and effectiveness. Notably, MiniCPM4-8B
demonstrates significant speed improvements over Qwen3-8B when processing long
sequences. Through further adaptation, MiniCPM4 successfully powers diverse
applications, including trustworthy survey generation and tool use with model
context protocol, clearly showcasing its broad usability.