MetaMath: Bootstrap Your Own Mathematical Questions for Large Language
  Models
Authors: name: Longhui Yu; name: Weisen Jiang; name: Han Shi; name: Jincheng Yu; name: Zhengying Liu; name: Yu Zhang; name: James T. Kwok; name: Zhenguo Li; name: Adrian Weller; name: Weiyang Liu
Published: 2023-09-21 17:45:42

Abstract: Large language models (LLMs) have pushed the limits of natural language
understanding and exhibited excellent problem-solving ability. Despite the
great success, most existing open-source LLMs (\eg, LLaMA-2) are still far away
from satisfactory for solving mathematical problem due to the complex reasoning
procedures. To bridge this gap, we propose MetaMath, a fine-tuned
language model that specializes in mathematical reasoning. Specifically, we
start by bootstrapping mathematical questions by rewriting the question from
multiple perspectives without extra knowledge, which results in a new dataset
called {MetaMathQA}. Then we fine-tune the LLaMA-2 models on MetaMathQA.
Experimental results on two popular benchmarks (\ie, GSM8K and MATH) for
mathematical reasoning demonstrate that MetaMath outperforms a suite of
open-source LLMs by a significant margin. Our MetaMath-7B model achieves
66.4% on GSM8K and 19.4% on MATH, exceeding the state-of-the-art models
of the same size by 11.5% and 8.7%. Particularly, {MetaMath-70B} achieves
an accuracy of 82.3% on {GSM8K}, slightly better than {GPT-3.5-Turbo}. We
release the {MetaMathQA} dataset, the {MetaMath} models with different model
sizes and the training code for public use.