OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data

Authors: Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, Igor Gitman
Primary area: datasets and benchmarks
Keywords: Math Reasoning, Synthetic Data

Abstract:
Mathematical reasoning continues to be a critical challenge in large language model (LLM) development with significant interest. However, most of the cutting-edge progress in mathematical reasoning with LLMs has become closed-source due to lack of access to training data. This lack of data access limits researchers from understanding the impact of different choices for synthesizing and utilizing the data. With the goal of creating a high-quality finetuning (SFT) dataset for math reasoning, we conduct careful ablation experiments on data synthesis using the recently released Llama3.1 family of models. Our experiments show that: (a) solution format matters, with excessively verbose solutions proving detrimental to SFT performance, (b) data generated by a strong teacher outperforms on-policy data generated by a weak student model, (c) SFT is robust to low-quality solutions, allowing for imprecise data filtering, and (d) question diversity is crucial for achieving data scaling gains. Based on these insights, we create the OpenMathInstruct-2 dataset which consists of 14M question-solution pairs (≈ 600K unique questions), making it nearly eight times larger than the previous largest open-source math reasoning dataset. Finetuning the Llama-3.1-8B-Base using OpenMathInstruct-2 outperforms Llama3.1-8B-Instruct on MATH by an absolute 15.9% (51.9% → 67.8%). Finally, to accelerate the open-source efforts, we release the code, the finetuned models, and the OpenMathInstruct-2 dataset under a commercially permissive license.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730434907126

thread_type:
official_review

<thread_object>
creator: reviewer_umxv

<content>
summary:
This paper curate a large scale (14 M qa-pairs) and high-quality math reasoning SFT datasets - OpenMathInstruct-2. At the same time, authors find some insights:

Verbose solutions is detrimental to SFT performance. Data generated by strong model is better. SFT allow some low-quality solutions. Question diveristy is crucial.

SFT the Llama-3.1-8B-Base on OpenMathInstruct-2 dataset achieve 67.8% performance, outperforming Llama3.1-8B-Instuct by 15.9%

strengths:
The paper introduces a large-scale math reasoning dataset, which would be highly beneficial if released.

It is well-written and easy to follow.

The authors offer valuable insights into data curation and supervised fine-tuning, discussing aspects like the format of chain-of-thought and the diversity of questions. Some insights are interesting and can be explored deeper.

weaknesses:
The process of dataset generation is not novel. The main advantage is the larger scale.

Some findings like data generated by strong teacher is better, question diversity matters are very natural.

questions:
How do authors find or select the OpenMath CoT format? The findings that OpenMath CoT is short but performs better is interesting. I think better COT format exploration is an interesting direction.

In section 3.1, aurthors discussed the LLM decontamination. Is the SFT data is processed the decontamnation? Have authors conducted experiments comparing the performance of with and without the decontamination?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730647350416

thread_type:
official_review

<thread_object>
creator: reviewer_vtsb

<content>
summary:
The paper introduces OpenMathInstruct-2, a large open-source dataset for mathematical reasoning containing 14M question-solution pairs (approximately 600K unique questions), making it nearly eight times larger than previous open-source datasets. The authors conducted systematic experiments to understand how different dataset design choices affect model performance, finding that:

1. Solution format matters - excessive verbosity can hurt performance. 
2. Teacher model quality is important
3. SFT is robust to low-quality solutions
4. Question diversity is crucial.

Using these insights, they created OpenMathInstruct-2 and demonstrated its effectiveness through model fine-tuning. Their OpenMath2-Llama3.1-8B model achieved 67.8% accuracy on MATH, outperforming Llama3.1-8B-Instruct by 15.9%. The authors also introduced a thorough LLM-based decontamination pipeline to ensure test set integrity. The paper concludes by releasing all code, models, and the dataset under a permissive license to support open-source development in mathematical reasoning.

strengths:
* **Ablation Studies & Methodology** - The paper excels in its systematic investigation of dataset design choices. The ablation studies on solution format, teacher model quality, and robustness to low-quality solutions are thorough and well-controlled.

* **Open Science & Reproducibility** - The authors make significant contributions to open-source development by releasing their dataset, code, models, and detailed methodology. Their comprehensive documentation of prompts (in appendices D.1-D.5) and data generation pipelines enable full reproducibility.

* **Math Synthetic Data Insights** - The paper provides several interesting findings that advance our understanding of LLM synthetic data generation in math reasoning. For instance, the discovery that shorter solution formats can outperform more verbose ones (OpenMath CoT vs Llama CoT) challenges conventional wisdom about detailed explanations. Similarly, their finding that SFT is robust to up to 20% incorrect solutions provides valuable guidance for dataset creation, suggesting that aggressive filtering may be unnecessary.

weaknesses:
**Insufficient Exploration of Question Generation Quality** - While the paper presents a thorough decontamination pipeline for synthetic questions, it lacks a detailed analysis of the quality and diversity distribution of generated questions. The authors don't evaluate whether their synthetic questions maintain: 1) the same quality as the original questions; 2) the same difficulty level as the original dataset 3) the same diversity (or more diverse) compared with the seed data. This might be crucial. The authors' ablation studies also show that even though the model is error-tolerant to wrong answers to some extent (e.g., 20%), the accuracy would downgrade more if there were too many error examples in the training. It would be necessary to examine the quality of the 14M generated samples. 


**Potential Knowledge Distillation Bottleneck** - There's a fundamental question about the approach. Though the performance is promising on a small student model (i.e., Llama-3.1-8B) with the 14M synthetic data, it is unclear whether improvement comes from primarily distilling the teacher's knowledge or improving genuine mathematical reasoning capabilities. The diminishing returns with larger student models (70B vs 8B) are consistent with distillation behavior, where the student model's capacity approaches needed to mimic the teacher's surface patterns rather than learning deeper reasoning. This limitation could be particularly important for scaling mathematical reasoning capabilities beyond the current teacher model's abilities. I'd suggest either of the experiments:
- Further fine-tune the teacher model (340B) on the 14M data (which might be costly)
- Generate a relatively larger-scale training set (>1M) using an 8B model and fine-tune the model on the generated data. 

**Incomplete Investigation of Teacher Model Selection** - The authors demonstrate that Llama3.1-405B-Instruct outperforms Llama3.1-8B-Base as a teacher model, but they don't explore the space between these two extremes. Would a 70B model be sufficient for a teacher? Is there a clear relationship between teacher model size and generated solution quality? Given the computational cost implications of using a 405B parameter model for data generation, understanding the minimum viable teacher model size would be practically valuable.

questions:
- The authors acknowledge but don't fully investigate why their OpenMath2-Llama3.1-70B model shows less dramatic improvements compared to their 8B model. They hypothesize that "our data blend or solution format might be more suited for weaker models" but don't test this hypothesis.

- Have you tried other sizes of teacher models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719815792

thread_type:
official_review

<thread_object>
creator: reviewer_7gq8

<content>
summary:
This paper introduces OpenMathInstruct-2, a large open-source dataset for improving mathematical reasoning in language models. The authors propose a more concise Chain-of-Thought (CoT) format, which enhances performance compared to existing methods. Through extensive experiments, they demonstrate that using a strong teacher model for data generation is more effective than on-policy data from weaker models. The study also highlights the robustness of models to low-quality data and emphasizes the importance of question diversity in fine-tuning. Overall, the paper contributes to open-source resources and offers valuable insights into optimizing mathematical reasoning in language models.

strengths:
1. Open-Source Dataset: This paper released the code, the finetuned models, and the OpenMathInstruct-2 dataset under a commercially permissive license.
2. Extensive Experiments: Not only main experiments, authors did a lot ablation study about: 1. choice of data generation model, 2. robustness of SFT, 3. impact of question diversity. The results are interesting and solid.

weaknesses:
This is a solid work; I don't think there are any obvious weaknesses.

questions:
1. What do you think about a self-improving framework, such as using an 8B Llama model to generate synthetic data and fine-tune it, then repeating the process for 2-3 rounds?
2. What do you think is the reason for the inadequate performance of on-policy data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730737678204

thread_type:
official_review

<thread_object>
creator: reviewer_wcne

<content>
summary:
This paper introduces OpenMathInstruct-2, an SFT recipe for improving the math reasoning of language models. With extensive engineering efforts and experiments, the authors show the importance of solution formats, the need of a strong teacher, question diversity, and the robustness to imperfect solutions. Putting everything together, the authors show that OpenMathInstruct-2 improves Llama 3.1 and performs strongly compared to other open-source models.

strengths:
I think this paper is in general a high-signal paper. Specifically:

- **Well-written with high-density of details**: this paper is well-written. The flow is a naturally interleaved summary, detail, and explanations. Many important details (size of model, size of dataset, specific accuracy numbers .etc) are packed in the writing with clear explanations of the motivation.
- **Well-execution**: the experiments in this paper is clear. I particularly appreciate the effort and discussion about decontamination, which essentially strengthens this paper quite a bit.
- **Sufficient signal from experiments**: the model developed in this paper is not top 1 in the leaderboard (but understandable because Qwen uses more resources). But the signal sent by the experiments (tolerate to imperfect annotation, importance of question diversity .etc) is clear.

weaknesses:
There are two conclusions draw by the authors that I am not so sure: 

- **Format of the data**. In general I agree that excessively verbose CoT may not be good for either performance or user preference, but it is not very clear how exactly the CoT proposed in this work contribute to the improved accuracy of reasoning. The higher numbers reported by the author is a good observation but not the mechanism behind. In addition, existing successful models like O1, in many cases, also offer verbose CoT, yet its performance is remarkable. In general, I tend to believe the level of detail that should be included in the CoT may still need further investigation, and I do see mixed signals favor or not favor verbose CoT.
- **Limited domain**: the authors still use GSM8K and MATH as the seed data — they do generate new questions but the seed is still GSM8K and MATH. The author test the performance on more challenging tasks (AMC and AIME), which is a nice practice. But I do feel that the field is moving forward to wider domains (not just math but also other STEM) and higher level (not just high-school but college). For this work I think it is sufficient to just use GSM8K and MATH, but I would suggest the authors to move forward in their future endeavors — only focusing on these two datasets would limit the authors from making higher impact.

questions:
In Table 4, the authors compare with Qwen 2.5, Mathtral, NuminaMath and DS-Coder. I understand these are strong open-source baselines so comparing with them is sufficient to see the signal, but I would still encourage the authors include more OSS/ proprietary  models for further comparison (even other models are much weaker or stronger).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732731140023

thread_type:
official_comment

<thread_object>
creator: area_chair_a2tq

<content>
comment:
Dear reviewers,

Thank you for your efforts reviewing this paper. If you haven't, can you please check the authors' responses and see if your concerns have been addressed? Please acknowledge you have read their responses. Thank you!
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary:

This paper aims to create a high-quality finetuning dataset for math reasoning, and conduct careful ablation experiments on data synthesis using the recently released Llama 3.1 family of models. The experiments show the importance of solution formats, the need of a strong teacher, question diversity, and the robustness to imperfect solutions. This paper creates the OpenMathInstruct-2 dataset which consists of 14M question-solution pairs (≈ 600K unique questions), which is nearly eight times larger than the previous largest open-source math reasoning dataset. Finetuning the Llama-3.1-8B-Base using OpenMathInstruct-2 outperforms Llama3.1-8B-Instruct on MATH by an absolute 15.9% (51.9% → 67.8%). The paper releases the code, the finetuned models, and the OpenMathInstruct-2 dataset under a commercially permissive license.

Strengths:

1. Well-written with great details and easy to follow
2. Experiments are well-executed. 
3. Valuable findings that advance the understanding of LLM synthetic data generation in math reasoning. 

Weaknesses:

There seems to be no major weakness after the discussion period. Most comments and questions raised by the reviewers have been addressed.