OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data

Authors: shubham  toshniwal
wei  du
ivan  moshkov
branislav  kisacanin
alexan  ayrapetyan
igor  gitman
Venue: NeurIPS_cc_2024_Workshop_MATH-AI
Published: 2024-09-18 00:22:17

Abstract: Mathematical reasoning continues to be a critical challenge in large language model (LLM) development, with a significant performance gap between closed-source and open-source efforts, largely due to differences in training data quality and scale. The emergence of frontier open-weight LLMs offers new opportunities to generate high-quality, commercially permissible synthetic data to help bridge this gap. In this paper, we investigate the recently released Llama3.1 family of models to improve open-source math reasoning through synthetically generated supervised finetuning (SFT) data. We conduct ablation studies to optimize design choices for the dataset, such as solution format and teacher model selection, which enhance SFT performance. We also investigate SFT’s robustness to incorrect solutions and find that at large data scales, the model can be robust to as much as 20% noise, suggesting that the simple answer-matching heuristic is sufficient for SFT data selection. Based on these insights, we create the OpenMathInstruct-2 dataset which consists of 14M question-solution pairs (> 600K unique questions), making it nearly eight times larger than any previous such dataset. Finetuning the Llama-3.1-8B-Base using OpenMathInstruct-2 outperforms Llama3.1-8B-Instruct on MATH by an absolute 14.6% (51.9 → 66.5), demonstrating the effectiveness of the dataset. As part of our open-source efforts, we will release the code, the finetuned models, and the OpenMathInstruct-2 dataset under a commercially permissive license.

Main paper: 
 introduction: 
 Synthetic data has emerged as a key technique for building large language models due to its costeffectiveness and scalability [Meta-AI, 2024 , NVIDIA, 2024 , DeepSeek-AI, 2024b] . In particular, synthetic data is well suited for mathematical reasoning where the performance improvements with synthetic data scaling are yet to saturate [Zeng et al., 2024 , Chan et al., 2024 , Yang et al., 2024] . However, access to this progress is limited because the current largest math datasets remain closedsource [Zeng et al., 2024 , Yang et al., 2024] . The closed nature of these datasets introduces two major issues. First, concerns over data leakage erode trust in reported benchmark results [Aiyappa et al., 2023] . For e.g., Zhang et al. [2024] show a drop of more than 10% for popular LLMs on an unpublished test set which is distributionally similar to the popular grade school math benchmark GSM8K [Cobbe et al., 2021] . Second, it prevents practitioners from fully understanding the impact of data composition and algorithmic choices [Soldaini et al., 2024] .Among open-source alternatives, the recent NuminaMath dataset [Li et al., 2024] has the largest collection of questions collected from diverse sources. However, its restrictive license-likely due to the use of GPT-4o in data processing and synthesis-limits its broader use. Similarly, other popular math instruction tuning datasets, such as MetaMathQA [Yu et al., 2024] and MathInstruct [Yue et al., 2024] , have also relied on GPT models for data synthesis, which prohibits their usage in non-commercial settings. A notable exception is the OpenMathInstruct-1 [Toshniwal et al., 2024] dataset, one of the biggest open-source math reasoning datasets, where solutions are synthesized using open-weight models. However, OpenMathInstruct-1 has two key limitations. Firstly, its question diversity is constrained, since all the questions in the dataset are drawn from the training sets of MATH [Hendrycks et al., 2021] and GSM8K [Cobbe et al., 2021] . Secondly, at the time of its release, there was a sizable gap in the math reasoning capabilities of open and closed-source models. As a result, the dataset underrepresents more challenging problems compared to its GPT-based counterparts [Gou et al., 2024] .The emergence of frontier open-weight models [Meta-AI, 2024, DeepSeek-AI, 2024b] has made it possible to create high-quality, commercially permissible math reasoning datasets. In this paper, we use the recently released Llama3.1 family of models to generate synthetic math instruction tuning (SFT) data, and evaluate the quality of the math reasoning data by finetuning the Llama-3.1-8B-Base model. To create the dataset, we conduct careful ablation studies using the MATH dataset to determine design choices that impact the final SFT performance. The highlights of our findings include:• Chain-of-Thought (CoT) Solution Format: Excessive verbosity can be detrimental to the SFT performance. Our proposed CoT format outperforms Llama's CoT format while being 40% shorter in solution length (see Figure 2 for a sample solution).• Choice of Teacher Model: The SFT performance mirrors the teacher model's performance even when controlling for the SFT data size. Specifically, we find that finetuning Llama3.1-8B-Base on solutions generated by Llama3.1-405B-Instruct outperforms solutions generated by the model itself even when controlling for the SFT data size.• Robustness of SFT: At sufficient data scale (≥ 256K), the SFT performance suffers minimal to no degradation with as much as 20% incorrect solutions. Additionally, removing solutions with incorrect intermediate steps, as determined by LLM-as-a-Judge or the Nemotron Reward model [NVIDIA, 2024] , also yields no performance benefit.• Impact of Question Diversity: Controlling for SFT data size, we find that question diversity has a huge positive impact on SFT performance.Based on the above findings, we create OpenMathInstruct-2 with data synthesized using Llama-3. The final answer is: 5The number of darts that hit each board must be a non-negative integer, and these numbers must add to 4. Since the dartboards are identical, we can list the numbers from greatest to least.The possibilities are:4 + 0 + 0 + 0 = 4 + 0 + 0 = 4 + 0 = 4 3 + 1 + 0 + 0 = 4 + 0 + 0 = 4 + 0 = 4 2 + 2 + 0 + 0 = 4 + 0 + 0 = 4 + 0 = 4 2 + 1 + 1 + 0 = 3 + 1 + 0 = 4 + 0 = 4 1 + 1 + 1 + 1 = 2 + 1 + 1 = 3 + 1 = 4Counting these, there are 5 possibilities. synthesized questions, we perform thorough decontamination using the lm-sys pipeline, followed by manual inspection [Yang et al., 2023] . Figure 1 provides an overview of the entire dataset construction pipeline. The final dataset consists of 14M question-solution pairs with 600K unique questions. Thus, OpenMathInstruct-2 is about 8 times bigger than the previous biggest standalone open-source dataset [Toshniwal et al., 2024] .The quality of OpenMathInstruct-2 is illustrated by the strong performance of the finetuned models. The OpenMath2-Llama3.1-8B model, which is the Llama3.1-8B-Base model finetuned with OpenMathInstruct-2, outperforms Llama3.1-8B-Instruct by an absolute 14.6% on MATH with just SFT. With a performance of 66.5 on MATH, OpenMath2-Llama3.1-8B is one of the strongest sub-10B open-source models. We will release all our fine-tuned models, code, the OpenMathInstruct-2 dataset, and a dataset explorer. 
experimental setup: 
 Training Details. For all the experiments, except when training on the full dataset, we train the Llama3.1-8B-Base model for four epochs and save a checkpoint at the end of every epoch. For the full dataset, we train the model for about 2.2 epochs (60K steps) and save six equally spaced checkpoints. The final checkpoint is created by averaging all the saved checkpoints. A global batch size of 512 is used along with the AdamW optimizer [Loshchilov and Hutter, 2019] with a learning rate of 5e-6 and a weight decay of 1e-2. All experiments are performed using the NeMo toolkit [Kuchaiev et al., 2019] .Evaluation Setup. We evaluate the final finetuned model on popular math reasoning benchmarks, namely GSM8K, MATH, AMC 2023, and AIME 2024. The finetuned model is evaluated in the zero-shot setting with greedy decoding.  
results: 
 To understand the impact of data scaling, we downsample the full dataset to 1M, 2M, and 5M-sized instruction tuning datasets using fair downsampling [Toshniwal et al., 2024] . Figure 3 plots the performance on the MATH test set with the increase in SFT data size. With even the 1M downsampled version of OpenMathInstruct-2, the final model easily outperforms Llama3.1-8B-Instruct.Finally, we observe a consistent gain with an increase in data size, and even at 14M dataset size, we see no signs of saturation in performance gains.Table 1 presents the results for top-performing, sub-20B, open-weight and open-source models (without tool use). The OpenMath2-Llama3.1-8B model, which is finetuned on the full OpenMathInstruct-2 dataset, outperforms or matches Llama3.1-8B-Instruct on all the math reasoning benchmarks. Among the open-source models, we outperform the recently released NuminaMath-7B-CoT on all benchmarks as well. Finally, among all the presented models, the OpenMath2-Llama3.1-8B is second only to the Qwen2.5-Math-7B-Instruct, which has been trained on more than a trillion synthetically generated math reasoning tokens, and starts with a base model, Qwen2.5-Math, which is about 35% better than Llama3.1-8B-Base.2  
conclusion: 
 We introduce OpenMathInstruct-2, a math instruction tuning dataset with 14M question-solution pairs and more than 600K unique questions. The dataset is created using Llama3.1-405B-Instruct model and released with a commercially permissive license. Compared to previous work, OpenMathInstruct-2 is about eight times larger than the previous biggest open-source dataset for math reasoning.To support the open-source efforts, we will publicly release all the finetuned models, code, and the OpenMathInstruct-2 dataset. 
 
 Appendix: Data and models are available at https://huggingface.co/collections/nvidia/openmath-2-66fb142317d86400783d2c7b Code is available at https://github.com/Kipok/NeMo-SkillsWe are unsure of the n-gram based data contamination protocol followed by Qwen2.5-Math given its obvious weakness in detecting paraphrases. In our own decontamination setup, which we borrow fromYang et al. [2023], we find paraphrases of test set questions that are identified by our pipeline but which n-gram matching will miss out on. 



Reviews: review: title: title: A new dataset which significantly improves LLM SFT
main_review: main_review: Overview: The necessity for high-quality datasets available for non-commercial research is undeniable. Access to robust datasets not only facilitates a deeper understanding of the domain but also propels forward the advancements within the field. This paper addresses this need effectively.

Strengths:
1. The idea of conducting an ablation study to pinpoint the crucial elements within the data that significantly influence LLM SFT is commendable. The insights derived from this study appear to be well-founded and logically presented.
2. The newly generated dataset has demonstrably enhanced performance metrics, which strongly supports the underlying hypothesis of the study. This advancement is a significant step forward in the application of LLM SFT.
3. The evaluation methodology is lucid and meticulous, effectively showcasing the proposed model's capabilities in comparison to established baselines. This clarity in presentation adds to the credibility of the results.
4. The introduction of the paper is thorough, setting a solid foundation by delineating the problem and summarizing the current landscape of related work. This context is invaluable for readers to appreciate the contributions of the paper.

Weaknesses:
1. Despite the promising results discussed in highlights 2, 3, and 4, there is a noticeable absence of rigorous evaluation studies to substantiate these claims. This gap might lead to skepticism regarding the replicability and validity of the reported improvements.
2. The paper does not provide a rationale for the specific size of the generated dataset—600K entries—nor does it explore the potential impact of scaling the dataset size. A discussion on how varying the dataset size might affect the performance would have been beneficial for understanding the scalability and limitations of the proposed approach.

Conclusion: Overall, the contributions of this paper are significant, particularly in the enhancement of training data for SFT applications. The introduction of a better-quality dataset not only enriches the available resources for researchers but also potentially sets a new benchmark in the field. I believe that the community will find great value in these contributions, despite the mentioned shortcomings.
title: title: This paper create 14M question-solution pairs math dataset for finetuning, and improve the open-source math reasoning Llama3-8B
main_review: main_review: I think this is a good paper for acceptance. only one question come in my mind.

Questions
- Aside from the dataset size, are there any other differences between OpenMathInstruct-1 and OpenMathInstruct-2?

Pros
- Data Generation with Llama 3-405B Model:-The dataset is generated using the Llama 3-405B model, making the process reproducible.
- Large-Scale Dataset:
- It contains more than 14M question-answer pairs, providing a comprehensive training foundation.

Cons: 
See question.
Model Replacement Consideration:
- Could the Llama model be replaced with another similar model for data generation and fine-tuning?
title: title: A good large-scale, open-source dataset designed to improve the performance of mathematical reasoning in large language models (LLMs)
main_review: main_review: The methodology is robust, employing the Llama3.1 family of models to generate high-quality synthetic data. The results demonstrate significant improvements and the experimental setup is thorough, with detailed training protocols and evaluation on multiple benchmarks such as MATH, GSM8K, AMC 2023, and AIME 2024. 
The paper is well-structured and clearly written.
One area where the paper could improve is in providing more comprehensive comparisons to closed-source models, which would give a broader context for interpreting the performance gains.

Pros:
- Large scale dataset (14 mil)
- Significant performance improvement
- Contribution to the open source
- Well written paper

Cons:
- No comparison to the closed state of the art models
- Limited novelty in terms of groundbreaking new methods

Overall, OpenMathInstruct-2 is a commendable advancement. Addressing the minor limitations noted could further enhance its impact and utility. While the paper is not entirely without shortcomings, the strengths far outweigh the weaknesses.
title: title: Official Review of Reviewer NeAd
main_review: main_review: The paper makes a significant contribution to the field of mathematical reasoning in LLMs by introducing a large, high-quality open-source dataset and demonstrating strong performance improvements. The work is thoroughly supported by extensive experiments, including the exploration of data scaling laws for supervised fine-tuning (SFT) and the robustness of SFT.

Overall, OpenMathInstruct-2 represents a major advancement, providing a massive dataset that can serve as a vital resource for open-source LLMs focused on mathematical reasoning. This significantly addresses the limitations of previous small datasets and enhances the potential for open-source AI development in this domain. However, I have the following two suggestions to further improve the manuscript of this paper:

1.	Limited Exploration of Dataset Quality Metrics: While the paper provides robust performance benchmarks, it could benefit from deeper insights into the qualitative aspects of the dataset. For example, analyzing the diversity of questions or the depth of reasoning required would offer a clearer understanding of the dataset’s strengths and potential gaps.

2.	Lack of Stopping Criterion for SFT: This paper does not offer guidance on when to stop fine-tuning in terms of performance gains versus computational costs, which would be valuable for practical applications.