MiMo: Unlocking the Reasoning Potential of Language Model -- From
  Pretraining to Posttraining
Authors: name: Xiaomi LLM-Core Team; name: Bingquan Xia; name: Bowen Shen; name: Cici; name: Dawei Zhu; name: Di Zhang; name: Gang Wang; name: Hailin Zhang; name: Huaqiu Liu; name: Jiebao Xiao; name: Jinhao Dong; name: Liang Zhao; name: Peidian Li; name: Peng Wang; name: Shihua Yu; name: Shimao Chen; name: Weikun Wang; name: Wenhan Ma; name: Xiangwei Deng; name: Yi Huang; name: Yifan Song; name: Zihan Jiang; name: Bowen Ye; name: Can Cai; name: Chenhong He; name: Dong Zhang; name: Duo Zhang; name: Guoan Wang; name: Hao Tian; name: Haochen Zhao; name: Heng Qu; name: Hongshen Xu; name: Jun Shi; name: Kainan Bao; name: QingKai Fang; name: Kang Zhou; name: Kangyang Zhou; name: Lei Li; name: Menghang Zhu; name: Nuo Chen; name: Qiantong Wang; name: Shaohui Liu; name: Shicheng Li; name: Shuhao Gu; name: Shuhuai Ren; name: Shuo Liu; name: Sirui Deng; name: Weiji Zhuang; name: Weiwei Lv; name: Wenyu Yang; name: Xin Zhang; name: Xing Yong; name: Xing Zhang; name: Xingchen Song; name: Xinzhe Xu; name: Xu Wang; name: Yihan Yan; name: Yu Tu; name: Yuanyuan Tian; name: Yudong Wang; name: Yue Yu; name: Zhenru Lin; name: Zhichao Song; name: Zihao Yue
Published: 2025-05-12 14:30:11

Abstract: We present MiMo-7B, a large language model born for reasoning tasks, with
optimization across both pre-training and post-training stages. During
pre-training, we enhance the data preprocessing pipeline and employ a
three-stage data mixing strategy to strengthen the base model's reasoning
potential. MiMo-7B-Base is pre-trained on 25 trillion tokens, with additional
Multi-Token Prediction objective for enhanced performance and accelerated
inference speed. During post-training, we curate a dataset of 130K verifiable
mathematics and programming problems for reinforcement learning, integrating a
test-difficulty-driven code-reward scheme to alleviate sparse-reward issues and
employing strategic data resampling to stabilize training. Extensive
evaluations show that MiMo-7B-Base possesses exceptional reasoning potential,
outperforming even much larger 32B models. The final RL-tuned model,
MiMo-7B-RL, achieves superior performance on mathematics, code and general
reasoning tasks, surpassing the performance of OpenAI o1-mini. The model
checkpoints are available at https://github.com/xiaomimimo/MiMo.