MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code

Authors: Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, Hongsheng Li
Primary area: foundation or frontier models, including LLMs
Keywords: large language model, mathematical reasoning, continued pretraining

Abstract:
Code has been shown to be effective in enhancing the mathematical reasoning abilities of large language models due to its precision and accuracy. Previous works involving continued mathematical pretraining  often include code that utilizes math-related packages, which are primarily designed for fields such as engineering, machine learning, signal processing, or module testing, rather than being directly focused on mathematical reasoning. In this paper, we introduce a novel method for generating mathematical code accompanied with corresponding reasoning steps for continued pretraining. Our approach begins with the construction of a high-quality mathematical continued pretraining dataset by incorporating math-related web data, code using mathematical packages, math textbooks, and synthetic data. Next, we construct reasoning steps by extracting LaTeX expressions, the conditions needed for the expressions, and the results of the expressions from the previously collected dataset. Based on this extracted information, we generate corresponding code to accurately capture the mathematical reasoning process. Appending the generated code to each reasoning step results in data consisting of paired natural language reasoning steps and their corresponding code. Combining this data with the original dataset results in a 19.2B-token high-performing mathematical pretraining corpus, which we name MathCode-Pile. Training several popular base models with this corpus significantly improves their mathematical abilities, leading to the creation of the MathCoder2 family of models. All of our data processing and training code is open-sourced, ensuring full transparency and easy reproducibility of the entire data collection and training pipeline.

Decision: accept (spotlight)
Reviewer scores: 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730594371724

thread_type:
official_review

<thread_object>
creator: reviewer_le5x

<content>
summary:
This paper proposes MathCode-Pile, a 19.2B-token dataset of math text and Python code. The dataset includes high-quality math-related web content, code with mathematical packages, math textbooks, and synthetic data. In addition, they present MathCoder2, a family of large language models with enhanced mathematical reasoning capabilities over MathCode-Pile.

strengths:
1. The author combining symbolic math reasoning with executable code in the dataset, MathCode-Pile, which is noval. This innovative methodology extends prior research, making MathCode-Pile a significant resource for advanced math reasoning tasks.

2. The paper is clearly organized, with a well-structured explanation of each step in the MathCode-Pile creation and model evaluation process. Figures and tables also effectively illustrate the overall data pipeline.

3. This work has great significance in advancing mathematical reasoning within language models. MathCoder2, using MathCode-Pile, achieves superior results on math benchmarks, demonstrating the potential of code-paired reasoning data.

weaknesses:
1. The paper lacks a analysis of potential data leakage between MathCode-Pile and evaluation benchmarks, which could artificially inflate model performance.

questions:
I have interested about whether the MathCode-Pile’s strong focus on mathematical reasoning might impact the model’s performance in non-mathematical domains. For example, whether this dataset would enhance the model’s general coding abilities beyond math-focused tasks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688250578

thread_type:
official_review

<thread_object>
creator: reviewer_g66u

<content>
summary:
The paper "MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code" explores enhancing the mathematical reasoning capabilities of LLMs through continued pretraining on a novel dataset called MathCode-Pile. This dataset is constructed from various sources, including math-related web data, textbooks, and synthetic data. A key contribution is the generation of paired natural language reasoning steps and Python code, aimed at improving the alignment between mathematical reasoning and executable code. The authors demonstrate significant improvements in mathematical reasoning benchmarks such as MATH and GSM8K, using models fine-tuned with MathCode-Pile. The paper also emphasizes the open-source nature of their data processing and training pipeline.

strengths:
* Originality: The paper introduces a novel method of generating and pairing Python code with natural language reasoning steps, enhancing the mathematical reasoning capabilities of large language models.
* Quality of Dataset: The MathCode-Pile dataset, comprising 19.2B tokens, is a significant contribution, demonstrating meticulous curation from diverse sources like web data, math textbooks, and synthetic examples.
* Significant Performance Gains: The use of this dataset leads to notable improvements across various models, including Llama-3-8B, DeepSeekMath-7B, and Code-Llama-7B, especially on benchmarks like MATH and GSM8K.
* Detailed Methodology: The process of extracting LaTeX expressions, conditions, and results to generate corresponding Python code is well-documented, offering transparency and reproducibility.
* Open-Source Commitment: The release of data processing and training code enhances the research community's ability to validate and build upon this work.

weaknesses:
* Generalizability of Code Generation: The method’s applicability to more abstract or advanced mathematical domains is unclear, particularly beyond high-school-level math.
* Evaluation Uncertainty: It is ambiguous whether the generated Python code is executed during benchmark evaluations or merely used for pretraining, leaving questions about its practical impact.
* Scope Limitation: The focus on grade-school-level mathematics is not explicitly emphasized, potentially misleading readers about the dataset’s broader applicability.
* Ablation Study Depth: While the ablation studies show the value of the synthesized code, further exploration into the necessity of aligning reasoning steps with code versus treating them as independent could provide deeper insights.

questions:
* Code Execution in Evaluation: Is the Python code generated and executed during benchmark evaluations? Clarifying this would help to understand the role of Tool-Integrated Reasoning in the observed performance improvements.
* Generalization to Formal Proofs: Can the method be extended to generate formal proofs in languages like Lean or Coq? Specifically, how well does the approach handle abstract reasoning steps that require formal verification, which might be better suited to proof assistants rather than executable Python code?
* Independent Reasoning Steps: Would separating reasoning steps and corresponding code into independent examples still yield significant improvements? Such an ablation could help assess the criticality of their alignment in the dataset.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695569483

thread_type:
official_review

<thread_object>
creator: reviewer_yuwf

<content>
summary:
This paper presents a novel approach for enhancing mathematical reasoning in large language models (LLMs). Unlike previous models that used math-related code without detailed explanations, MathCoder2 generates mathematical code paired with natural language reasoning. This process involves filtering a large math-related dataset from web pages, synthetic sources, code, and textbooks to build a high-quality corpus called MathCode-Pile.

This dataset consists of 19.2 billion tokens and includes LaTeX-extracted mathematical expressions, conditions, results, and Python code to capture the underlying reasoning. MathCoder2 uses this corpus to significantly improve performance on various mathematical benchmarks, achieving results competitive with state-of-the-art models. Moreover, the MathCoder2 framework is fully open-source, which supports reproducibility and transparency in model training and data processing. This work sets a foundation for future research by focusing on reasoning capabilities through detailed code integration.

strengths:
MathCoder2’s MathCode-Pile corpus is rigorously curated and filtered from diverse math-related sources, including web data, synthetic data, specialized code, and textbooks. This ensures relevance, reduces noise, and provides a comprehensive dataset tailored specifically for mathematical reasoning, which is essential for pretraining LLMs in this area.

MathCoder2 demonstrates significant gains on multiple mathematical reasoning benchmarks, outperforming comparable models across different tasks. The improvement underscores the effectiveness of continued pretraining on the structured MathCode-Pile corpus and shows MathCoder2's potential for real-world applications in math-intensive fields.

weaknesses:
There are no major weaknesses.

questions:
Python was chosen for code snippets; it it possible to use specialized math software language instead (e.g., Mathematica)? This is not a direct limitation of this paper, but a possible future direction.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732532654982

thread_type:
official_comment

<thread_object>
creator: area_chair_kzku

<content>
title:
Action Required: Respond to Author Rebuttals - Nov 27

comment:
Dear ICLR Reviewers,

The author discussion phase is ending soon. Please promptly review and respond to author rebuttals for your assigned papers. Your engagement is critical for the decision-making process.

Deadlines:
- November 26: Last day for reviewers to ask questions to authors.
- November 27: Last day for authors to respond to reviewers.
- November 28 - December 10: Reviewer and area chair discussion phase.

Thank you for your timely attention to this matter.

Best,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a novel method for generating paired pretraining data that combines mathematical code with corresponding reasoning steps to improve LLMs' mathematical reasoning capabilities through continued pretraining. Using this approach, the authors introduce MathCode-Pile, which combines their generated data with existing datasets to create a comprehensive 19.2B-token mathematical pretraining corpus. 

Their experimental evaluation, conducted across LLMs ranging from 2-8B parameters that are continued pretrained on MathCode-Pile, demonstrates the effectiveness of their approach in enhancing mathematical reasoning capabilities. The reviewers particularly valued the extensive experiments across various LLMs. There is consensus among reviewers that the proposed data generation method is both effective and yields significant improvements in performance. 

The authors have adequately addressed most reviewer concerns, including questions about potential impacts on LLMs' capabilities in other domains and the method's applicability to other programming languages. While a more detailed analysis of potential data leakage issues would have been beneficial, this limitation is not unique to this work but rather a broader challenge in the current era of LLM pretraining. 

Overall, this is a valuable contribution to the field. I agree with the reviewers that the work's comprehensive evaluation and impressive performance improvements merit publication at ICLR.