MiMo: Unlocking the Reasoning Potential of Language Model -- From
  Pretraining to Posttraining
Authors: name: Xiaomi LLM-Core Team, name: Bingquan Xia, name: Bowen Shen, name: Cici, name: Dawei Zhu, name: Di Zhang, name: Gang Wang, name: Hailin Zhang, name: Huaqiu Liu, name: Jiebao Xiao, name: Jinhao Dong, name: Liang Zhao, name: Peidian Li, name: Peng Wang, name: Shihua Yu, name: Shimao Chen, name: Weikun Wang, name: Wenhan Ma, name: Xiangwei Deng, name: Yi Huang, name: Yifan Song, name: Zihan Jiang, name: Bowen Ye, name: Can Cai, name: Chenhong He, name: Dong Zhang, name: Duo Zhang, name: Guoan Wang, name: Hao Tian, name: Haochen Zhao, name: Heng Qu, name: Hongshen Xu, name: Jun Shi, name: Kainan Bao, name: QingKai Fang, name: Kang Zhou, name: Kangyang Zhou, name: Lei Li, name: Menghang Zhu, name: Nuo Chen, name: Qiantong Wang, name: Shaohui Liu, name: Shicheng Li, name: Shuhao Gu, name: Shuhuai Ren, name: Shuo Liu, name: Sirui Deng, name: Weiji Zhuang, name: Weiwei Lv, name: Wenyu Yang, name: Xin Zhang, name: Xing Yong, name: Xing Zhang, name: Xingchen Song, name: Xinzhe Xu, name: Xu Wang, name: Yihan Yan, name: Yu Tu, name: Yuanyuan Tian, name: Yudong Wang, name: Yue Yu, name: Zhenru Lin, name: Zhichao Song, name: Zihao Yue

We present MiMo-7B, a large language model born for reasoning tasks, with
optimization across both pre-training and post-training stages. During
pre-training, we enhance the data preprocessing pipeline and employ a
three-stage data mixing strategy to strengthen the base model's reasoning
potential. MiMo-7B-Base is pre-trained on 25 trillion tokens, with additional
Multi-Token Prediction objective for enhanced performance and accelerated
inference speed. During post-training, we curate a dataset of 130K verifiable
mathematics and programming problems for reinforcement learning, integrating a
test-difficulty-driven code-reward scheme to alleviate sparse-reward issues and
employing strategic data resampling to stabilize training. Extensive
evaluations show that MiMo-7B-Base possesses exceptional reasoning potential,
outperforming even much larger 32B models. The final RL-tuned model,
MiMo-7B-RL, achieves superior performance on mathematics, code and general
reasoning tasks, surpassing the performance of OpenAI o1-mini. The model
checkpoints are available at https://github.com/xiaomimimo/MiMo.