MiniCPM4: Ultra-Efficient LLMs on End Devices
Authors: name: MiniCPM Team, name: Chaojun Xiao, name: Yuxuan Li, name: Xu Han, name: Yuzhuo Bai, name: Jie Cai, name: Haotian Chen, name: Wentong Chen, name: Xin Cong, name: Ganqu Cui, name: Ning Ding, name: Shengdan Fan, name: Yewei Fang, name: Zixuan Fu, name: Wenyu Guan, name: Yitong Guan, name: Junshao Guo, name: Yufeng Han, name: Bingxiang He, name: Yuxiang Huang, name: Cunliang Kong, name: Qiuzuo Li, name: Siyuan Li, name: Wenhao Li, name: Yanghao Li, name: Yishan Li, name: Zhen Li, name: Dan Liu, name: Biyuan Lin, name: Yankai Lin, name: Xiang Long, name: Quanyu Lu, name: Yaxi Lu, name: Peiyan Luo, name: Hongya Lyu, name: Litu Ou, name: Yinxu Pan, name: Zekai Qu, name: Qundong Shi, name: Zijun Song, name: Jiayuan Su, name: Zhou Su, name: Ao Sun, name: Xianghui Sun, name: Peijun Tang, name: Fangzheng Wang, name: Feng Wang, name: Shuo Wang, name: Yudong Wang, name: Yesai Wu, name: Zhenyu Xiao, name: Jie Xie, name: Zihao Xie, name: Yukun Yan, name: Jiarui Yuan, name: Kaihuo Zhang, name: Lei Zhang, name: Linyue Zhang, name: Xueren Zhang, name: Yudi Zhang, name: Hengyu Zhao, name: Weilin Zhao, name: Weilun Zhao, name: Yuanqian Zhao, name: Zhi Zheng, name: Ge Zhou, name: Jie Zhou, name: Wei Zhou, name: Zihan Zhou, name: Zixuan Zhou, name: Zhiyuan Liu, name: Guoyang Zeng, name: Chao Jia, name: Dahai Li, name: Maosong Sun
Organization: OpenBMB

This paper introduces MiniCPM4, a highly efficient large language model (LLM)
designed explicitly for end-side devices. We achieve this efficiency through
systematic innovation in four key dimensions: model architecture, training
data, training algorithms, and inference systems. Specifically, in terms of
model architecture, we propose InfLLM v2, a trainable sparse attention
mechanism that accelerates both prefilling and decoding phases for long-context
processing. Regarding training data, we propose UltraClean, an efficient and
accurate pre-training data filtering and generation strategy, and UltraChat v2,
a comprehensive supervised fine-tuning dataset. These datasets enable
satisfactory model performance to be achieved using just 8 trillion training
tokens. Regarding training algorithms, we propose ModelTunnel v2 for efficient
pre-training strategy search, and improve existing post-training methods by
introducing chunk-wise rollout for load-balanced reinforcement learning and
data-efficient tenary LLM, BitCPM. Regarding inference systems, we propose
CPM.cu that integrates sparse attention, model quantization, and speculative
sampling to achieve efficient prefilling and decoding. To meet diverse
on-device requirements, MiniCPM4 is available in two versions, with 0.5B and 8B
parameters, respectively. Sufficient evaluation results show that MiniCPM4
outperforms open-source models of similar size across multiple benchmarks,
highlighting both its efficiency and effectiveness. Notably, MiniCPM4-8B
demonstrates significant speed improvements over Qwen3-8B when processing long
sequences. Through further adaptation, MiniCPM4 successfully powers diverse
applications, including trustworthy survey generation and tool use with model
context protocol, clearly showcasing its broad usability.