InternLM-Math: Open Math Large Language Models Toward Verifiable
  Reasoning
Authors: name: Huaiyuan Ying; name: Shuo Zhang; name: Linyang Li; name: Zhejian Zhou; name: Yunfan Shao; name: Zhaoye Fei; name: Yichuan Ma; name: Jiawei Hong; name: Kuikun Liu; name: Ziyi Wang; name: Yudong Wang; name: Zijian Wu; name: Shuaibin Li; name: Fengzhe Zhou; name: Hongwei Liu; name: Songyang Zhang; name: Wenwei Zhang; name: Hang Yan; name: Xipeng Qiu; name: Jiayu Wang; name: Kai Chen; name: Dahua Lin
Published: 2024-02-09 11:22:08

Abstract: The math abilities of large language models can represent their abstract
reasoning ability. In this paper, we introduce and open-source our math
reasoning LLMs InternLM-Math which is continue pre-trained from InternLM2. We
unify chain-of-thought reasoning, reward modeling, formal reasoning, data
augmentation, and code interpreter in a unified seq2seq format and supervise
our model to be a versatile math reasoner, verifier, prover, and augmenter.
These abilities can be used to develop the next math LLMs or self-iteration.
InternLM-Math obtains open-sourced state-of-the-art performance under the
setting of in-context learning, supervised fine-tuning, and code-assisted
reasoning in various informal and formal benchmarks including GSM8K, MATH,
Hungary math exam, MathBench-ZH, and MiniF2F. Our pre-trained model achieves
30.3 on the MiniF2F test set without fine-tuning. We further explore how to use
LEAN to solve math problems and study its performance under the setting of
multi-task learning which shows the possibility of using LEAN as a unified
platform for solving and proving in math. Our models, codes, and data are
released at https://github.com/InternLM/InternLM-Math.