MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code
Authors: Zimu Lu
Aojun Zhou
Ke Wang
Houxing Ren
Weikang Shi
Junting Pan
Mingjie Zhan
Hongsheng Li
Venue: ICLR 2025 Spotlight (2025)
Keywords: large language model
mathematical reasoning
continued pretraining
Primary area: foundation or frontier models, including LLMs

Abstract: Code has been shown to be effective in enhancing the mathematical reasoning abilities of large language models due to its precision and accuracy. Previous works involving continued mathematical pretraining  often include code that utilizes math-related packages, which are primarily designed for fields such as engineering, machine learning, signal processing, or module testing, rather than being directly focused on mathematical reasoning. In this paper, we introduce a novel method for generating mathematical code accompanied with corresponding reasoning steps for continued pretraining. Our approach begins with the construction of a high-quality mathematical continued pretraining dataset by incorporating math-related web data, code using mathematical packages, math textbooks, and synthetic data. Next, we construct reasoning steps by extracting LaTeX expressions, the conditions needed for the expressions, and the results of the expressions from the previously collected dataset. Based on this extracted information, we generate corresponding code to accurately capture the mathematical reasoning process. Appending the generated code to each reasoning step results in data consisting of paired natural language reasoning steps and their corresponding code. Combining this data with the original dataset results in a 19.2B-token high-performing mathematical pretraining corpus, which we name MathCode-Pile. Training several popular base models with this corpus significantly improves their mathematical abilities, leading to the creation of the MathCoder2 family of models. All of our data processing and training code is open-sourced, ensuring full transparency and easy reproducibility of the entire data collection and training pipeline.

Published as a conference paper at ICLR 2025
MATHCODER2: BETTER MATH REASONING FROM CONTINUED PRETRAINING ON TRANSLTRA MATHEMATICAL CODE
Zimu Lu Aojun Zhou" Houxing Ren', Ke Wang', Weikang Shi" Junting Pan 12 Mingjie Zhan') Hongsheng Li
MMultimedia Laboratory (MMLab), The Chinese University of Hong Kong CCIII under InnoHK luz mozimu@mail il matccedu... hsliles cuhk. edu.
{aojunzhou, amjall) =mjall]{ggaill @aggglmml.............mm..lammmm com
ABSTRACT
Code has been shown to be effective in enhancing the mathematical reasoning abilities of large language models due precision and accuracy. Previous works involving continued mathematical pretraining often include code that uti- lizes math-related packages, which are primarily designed for fields such as engi- neering, machine learning. signal processing, module testing, rather than being directly focused on mathematical reasoning. this paper, we introduce a novel method for generating mathematical code accompanied with corresponding rea- soning steps for continued pretraining. Our approach begins with the construc- tion of high-quality mathematical continued pretraining dataset by incorporat- ing math-related web data, code using mathematical packages, math textbooks, and synthetic data. Next, we construct reasoning steps by extracting LaTeX ex- pressions, the conditions needed for the expressions, and the results of the ex- pressions from the previously collected dataset. Based on this extracted infor- mation, we generate corresponding code to accurately capture the mathematical reasoning process. Appending the generated code each reasoning step results in data consisting of paired natural language reasoning steps and their correspond- ing code. Combining this data with the original dataset results in a 19. 19.2B-token high-performing mathematical pretraining corpus, which name MathCode- Pile. Training several popular base models with this corpus significantly improves their mathematical abilities, leading to the creation of the MathCoder2 family of models. All our data processing and training code is open-sourced, ensuring full transparency and easy reproducibility of the entire data collection and train- ing pipeline.
INTRODUCTION
Various studies (Azerbayev et al., 2024 Shao et al., 2024) have shown that training on code en- hances the mathematical reasoning abilities of large language models (LLMs). Previous research in continued mathematical pretraining often includes code that utilizes math-related packages (Azer- bayev et al. 2024). This code is typically sourced from GitHub and is primarily designed for fields such as engineering, machine learning, signal processing, module testing, rather than focusing directly on mathematics. Recent models (Zhou al., 2024 Yang al., 20246] Ying et al. 2024 Shao et al. 2024 Wang et al. (223a)) have adopted T Reasoning (TIR) in fine-tuning. They utilize integrated natural language reasoning steps and Python code to improve performance on mathematical reasoning tasks. Reasoning with the help of code is particularly effective for more challenging problems, likely due to its precision and accuracy.
Although utilizing existing open-source code in the pretraining phase can enhance the mathematical reasoning abilities of LLMs, such code often lacks accompanying natural language explanations or context. This might hinder the model's ability to fully understand them. In this paper, propose a novel method for generating large amounts of mathematical code accompanied by corresponding natural language reasoning steps, which are extracted from math-related pretraining texts. Different

---

Published as a conference paper at ICLR 2025
from the existing math-related code, our generated code is paired with natural language reasoning steps, making the code more comprchensible, Also, as our code is generated based on math-related texts, they are all highly related to mathematical reasoning. When used in pretraining, the mathe- matical code paired with reasoning steps facilitates LLMs' understanding of math-related pretraining texts, it effectively captures the underlying reasoning process. Furthermore, this data enhances the model's potential to be finetuned for TIR reasoning.
Our data processing pipeline consists of two key steps: (1) carefully curating robust basic dataset for pretraining, and (2) generating paired reasoning steps and mathematical code by extracting La- TeX expressions and their context, translating the extracted information into Python code snippets, executing the generated code snippets, and verifying their correctness.
First, we gather and carefully filter a wide variety of math-related data sources, including web pages, model-generatee data, math-related code, and textbooks. Through an advanced filtering process, we ensure the dataset both large and highly relevant, minimizing irrelevant content while preserving the mathematical texts necessary for training. This results in 16.5B-token dataset that forms the foundation our pretraining efforts. By conducting experiments with smaller models, we show that this careful curation leads to more efficient training without sacrificing model performance.
Second, we propose novel method for generating large amounts of paired mathematical reason- ing steps and their corresponding Python code. Given piece of text from the pretraining corpus collected above, we wrap it in a carefully designed prompt that instructs a Llama-3. 1-rena-..--0------ -nstruutt model to extract LaTeX expressions along with their relevant context, including the conditions for each expression and the result of its computation. This results in a list of comprehensive mathemati- cal reasoning steps, complete with the necessary conditions, the computations taken, and the results. Then, we prompt the model to translate each reasoning step into a Python code snippet that captures the underlying reasoning process. The generated Python snippets are executed, and only those that run successfully and produce outputs matching the expected results are retained. By pairing the code with the corresponding reasoning step. we create the final data. The process is demonstrated in the lower half of Fig. [ This process yields 2.7B-token corpus of mathematical code snippets accompanied with their corresponding reasoning steps, which we combine with the data generated in the first step create a 19.2B-token pretraining dataset, named
We validate the effectiveness of MathCode-Pile on four popular base models: Llama-3-8B, DeepSeekMath-7B, Mistral-7B, and Code-Llama-7B, significantly improving their performance on five representative mathematical benchmarks. We name the resulting family of pretrained mod- els MathCoder2. In particular, Mathhoder2-LLlmaa-333 MATH and 69.9% on GSM8K, outperforming the baseline of training only on the basic data gener- ated in the first step by 3.1 % and 1%1, respectively. This demonstrates that the data of mathematical code accompanied with reasoning steps effectively enhances LLMs' reasoning abilities.
achieves 4-shot accuracies of 38.4% on
Different from recent works, such DeepSeekMath (Shao et al. 2024). InterLL-- (Ying) et al. 2024), and Qwen2.5-Math (Yang et al., 2024b), which only release their model weights, we offer detailed, open-source framework for data processing and training that achieves performance competitive with these models, fostering further progress in mathematical reasoning for LLMs.
Our contributions include:
A novel and effective method for generating large amounts of mathematical code with cor- responding natural language reasoning steps, significantly enhancing pretraining outcomes.
The creation of MathCode-Pile, a meticulously curated 19.2B-token dataset for contin- ued mathematical pretraining. This dataset includes math-related web data, synthetic data, code, textbooks, and mrans-ttansllted mathematical code.
Full opun-sourcing of all data processing and training code, ensuring transparency and reproducibility to support future research.
2

---

Published as a conference paper at ICLR 2025
(a) Other Metnods
Lemma
Common Crawi
Rule based Models
Texts
Ham- related
Deepseok math
(b) MarthCoder 2.0
Filtered Texts
Finer litered Math Texts:
Canditoon Needed:
The intagral taken 3D region deftred the imiis of imterattion 2 The interrnd is fanction of theee varibiles A.8. and Computaton Expression:
(i) (os(
y ssin(0) sin(P)
sta (8)apddd
Computaton Resul:
caliuulate volume:
Transiation LLM
val
Fython Coce Srippet
Execute Coce and Compare with Result
(in apdedp
From JeCegrace Lnport tolqued
Eampy op
48
der inteanandirfe,, theta, 212):
result, tplauset pA12, James prs: Junode pli, prome (resutt 33 Pulliii wite
Interleaved Reasoning tepps and Code Snippets
The data processing pipeline. (a) shows the pipeline of prior works. (b) demonstrates our method. We first use fastText classifier to filter the Common Crawl corpus, resulting in the initial filtered math texts. Then, we annotate part of the filtered texts to train new fast'Text classifier, and conduct second filtering, resulting in the finer filtered math texts. Then tuned LLM to extract reasoning steps from these math-related texts, and translate the reasoning steps into corresponding code snippets. We execute the code snippets and compare the output with the expected result. If the code executes successfully and the result is as expected, the code is retained.
Figure I:
instruction-
CURATION OF MATHCODE-PILE
We curate our mathematical pretraining dataset, MathCode-Pile, in two steps: first, we collect the basic data Sec. 2.1 and then we use it to generate mathematical code snippets with their corre- sponding natural language reasoning steps in Sec. 2.2
2.1
BASIC DATA
We collect and carefully filter diverse range mathematical data to ensure relevance and quality for continued pretraining of LLMs. The data includes math-related web content, synthetic data, code utilizing mathematical packages, and mathematical textbooks.
Math-related Web Data. Web data offers broad range of real-world mathematical examples. We start with the OpenWehMath (Paster et al.] 2023) dataset, which contains mathematical web pages sourced from Common Crawl. Observing that significant portion of these documents unrelated to mathematics, we instruct the Mixtral-8x7B-Insttree prompt (detailed in Appendix in Appendix D This reduces the dataset from 13.7B tokens to 4.8B tokens (measured using the Llama-3 tokenizer). We call this filtered version filtered-Opent Optrredahpp
model with carefully designed filter irrelevant texts. Examples of irrelevant
shown
To further expand the dataset, we train fastText classifier (Joulin et al., 2016) using filtered- OpenWehMath as positive samples and random Common Crawl data as negative samples (training details are explained in Appendix. [B. This model helps identify additional math-related documents within the Common Crawl data from Matrix (Zhang et al., 2024). a general pretraining dataset. A second round of filtering is performed, where Mixtral-8x Instrruct annotates portion of these documents, and a new fastText classifier trained based on these annotations further refines the data. Finally, combine filtered-
This produces 6.4B tokens, which we label Mitered-CC-En-eeee Open OpenWebMath WebMath and filtered-CC- En-hhahh, resulting in comprehensive 11.2B-toke 2-22-token math-related web dataset.
3

---

Published as a conference paper at ICLR 2025
You will be presented with text related to math. need you to identify all the complex computations in For each complex computation, find the conditions needed for the computation, the LaTeXX expression that conducts the computation, and the result of the computation. Then generate a Python code snippet for each computation that demonstrates how the result is reached. Output each computation in the following format:
Prompt:
Conditions Needed:
|Condition II
[Condition 2]
Computation Expression:
S[LaTeX Expression]S
Computation Result: [Computation Result]
Python Code Snippet:
''python
[Python Code]
There can be more than one complex computation in the text. Output only the computations that requires calculation. Do not include mathematical statements or definitions as a computation. Make sure each snippet can be executed individually. The text is as follows: (TEXT) The computations
Figure The prompt extracting reasoning steps from texts in the pretraining corpus and generat- ing the corresponding Python snippets. (TEXT) is replaced with the text from the dataset collected in Sec.
Synthetic Data. Synthetic data offers structured mathematical texts that complement the web data. We collect synthetic data from various open-source repositories on Hugging Face, including datasets like Caattiin-Coollee-tuuctla
Maths (ooths-Coollee]] and synthetic math books from Matrix (Zhang 2024). To ensure relevance, we apply fastText classifier to filter out non- mon-mathmatiiall documents, refining the dataset to 2.2B tokens of high-quality synthetic math content.
Code Utilizing Mathematical Packages. Code data offers practical examples of how mathematical libraries are used in programming. We collect code from Python and Jupyter files within the Star- CoderData dataset (Li et al.) 2023). retaining only programs that import math- packages such as sympy. fractions, cmath, scipy, or statistics. The widely used numpy package is not used to filter the documents, as it appears frequently in non mothmamaeaatiaal contexts. After filtering, this collection process results in 1.7B tokens of code related to mathematical computations.
Mathematical Textbooks. Textbooks provide formal, structured presentations of mathematical con- cepts, making them valuable source of math knowledge. We gather 8K PDFs of textbooks from online resources by identifying those with titles containing math-related keywords such as algebra, geometry, probability, etc. These PDF files are then converted into markdown format using the Nougat tool for easier integration into our training pipeline.
2.2 MODEL-TRANSLATER MATHEMATICAL CODE
In this section, we propose novel approach for extracting reasoning steps from the basic pretrain- ing data and translating them into corresponding Python code snippets that capture the underlying reasoning processes. This extraction and translation process is performed using strong instruction- tuned model, which is 1--mn---.-77-----ee
this paper.

---

Published as a conference paper at ICLR 2025
Table 1: The components and data statistics of MathCode-Pile.
Components
Size (MB)
Documents
Tokens
Average (Tokens)
Setered-OpeeWebb.eee
16,999
2,824,705
4,826,902.621
1,709
Sitered-CC-En-malle
23,465
7,597.718
6,341,745,645
835
Synthetic data
8,855
2,195,974
2,193,189,314
999
Code using math packages
6,120
513,059
1,703,226,005
3,320
Mathematical textbooks
4,431
8,373
1,390,268,773
166,042
Translated mathematical code
8,235
6,347,823
2,728,740,985
430
Total
68,105
19,487,652
19,184,073,443
984
Our method begins with taking a piece of text from the basic pretraining data and wrapping it in a carefully designed prompt, shown in Fig. 2 This prompt instructs the model to identify LaTeX expressions denoting complex computations, along with the necessary context, including the condi- tions required for the computation and the expected result. By explicitly extracting the conditions of the LaTeX expression, we enhance the model's ability to comprehend the underlying mathematical reasoning behind the usage of the expression. The expected result of the computation can later serve as basis for verifying the correctness of the generated code. A mathematical reasoning step is con- structed by combining the conditions, expression and result. The prompt then directs the model to produce Python code snipper that accurately reflects the underlying reasoning process behind the extracted reasoning step. The model asked to present the conditions, LaTeX expression, result, and Python code snippet in a structured format, ensuring that each part can be easily extracted from the generated text. Examples of generated texts are shown in Appendix C
After the Python code snippets are generated, they are executed, and outputs of the execution compared with the expected results extracted from the generated text. Only the Python code snippets that execute without errors and produce correct outputs are retained. This filtering process ensures a higher quality of generated code, making the resulting dataset more reliable for mathematical pretraining compared to approaches that rely unverified and general- purpose code.
Leveraging the Llama-3.1-20 Instuut model, we initially generated 3.1B tokens of the data. After applying the filtering process, obtain total of 2.7B tokens of high-quality data of mathemati- cal code snippets accompanied with their corresponding reasoning steps. This newly generated data significantly enriches our original pretraining corpus. By combining this data with the basic pretrain- ing data, we create a comprehensive pretraining dataset totaling 19.2B tokens, which we refer to as Detailed statistics of MathCode- are presented in Tab. T This dataset is tai- lored specifically for enhancing the mathematical and coding abilities of LLMs. To avoid benchmark contamination, we follow Yang al. (2024b) to filter out samples that have significant overlaps with any of the questions from benchmark datasets used in evaluation. We use exact match to remove the identical samples and further apply 13-gram deduplication (with condition that the Jaccard similarity should be larger than 0.6) to remove more samples that might cause contamination.
In comparison to traditional methods of curating math-related code, which often draw on general- purpose repositories, our method ensures that the code is not only syntactically correct but also mathematically sound, reflecting a deeper understanding of mathematical reasoning. Our math- ematical code is accompanied with corresponding natural language reasoning steps, which makes understanding the reasoning process easier. This makes MathCode- a superior resource for mod- els aimed performing advanced mathematical reasoning tasks.
3 EXPERIMENTS
To demonstrate the effectiveness of our method, we first train several base models ranging from 7B to 8B parameters using MathCode-Pile and compare them to other best-performing models of the same size. The group of models resulting from the continued mathematical pretraining is named MathCoder2. Next, we train and compare various other open open-source math pretraining datasets against MathCode-Pile using smaller model, DeepSeekCoder- I.33.. To showcase the potential of
5

---

Published as a conference paper at ICLR 2025
Table 2: Performance of various pretrained models five representative mathematical datasets. All results reported are based on greedy decoding. "Code-open" shows whether the code for data- processing and model-training is open-soureed. The red numbers show the improvements compared to the base model from which each MathCoder2 model is trained.
Model
Size
Code-
MATH
GSM8K
SAT
OCW
MMLU-
open
MATH
Qwen2-Math
7B
X
50.4
80.4
87.5
14.0
57.9
Qwen2.5-Math
7B
55.4
91.6
M225
7B
X
34.0
74.8
65.6
8.1
49.6
M2-MaaL
7B
21.5
49.2
Llemma
7B
V
18.0
36,4
53.1
7.7
Llama-2
7B
3.2
11.8
25.0
3.7
Llama-3
8B
21.4
54.8
56.3
10.3
42.8
MathCoder2-Llama-3
8B
V
38.4:-17.0)
69.9(-151)
84.4022.D
18.0c+777
46.5(-377
DeepSeekMath
7B
36.2
64.2
84.4
15.4
47.4
MathCooder-DDeqpSeekathh
7B
38.6(-2.4)
68.8 68.8
90.64462)
16.9(.1.5)
48.3
Mistral
7B
X
13.1
52.2
75.0
8.5
38.3
MathCoder2-Mistraal
7B
36.7 (6.7(62336))
68.2(-16.0)
81.3(66.3)
13.2(447)
42.2(
Code-Llama.
7B
6.7
14.6
25.0
3.7
26.4
MathCodde2-CCod--Llaa
7B
28.8:-22.1)
52.3
71.9(+16.9)
8.5
33.7(-7.))
the MathCoder2 models, we further perform supervised fine-tuning on them. Finally, we conduct ablation studies analyze the impact of each component of the dataset.
3.1 MAIN RESULTS
Benchmark datasets. We evaluate the MathCoder2 models on five representative datasets:
GSM8K (Cobbe et al., (021). MATH (Hendrycks et al. (2021b) SAT-Math (Azerbayev et al., 2024).
OCW (Lewkowycz al. 2022), and MMLU-Math (Hendrycks al. 2021a). GSMSK and MATH
are tested using a 4-shot prompt with MAmmoTH' evaluation framework (Yue et al. 2023). SAT-
Math and OCW are tested using a 4-shot prompt with DeepSeekMath's evaluation framework (Shao
al.) 2024). MMLU-Mate is tested using the Im-evaluation iiaitattionaannsssss harness's (Gao al.
2024) default
zero-shot setting for MMLU, These datasets cover a wide range of mathematical problems across various types and difficulty levels, from primary school math word problems college-level chal- lenges, providing a comprehensive evaluation the models.
Base models and training settings. To demonstrate that pretraining corpus is effective across different base models, we continue pretraining four base models with MathCode-Pila Llama-3- 8B (Dubey al.) 2024), PrepSeekMath-777 (Shao et al., 2024). Mistral-7B (Jiang al.) (023). and Mode-Llama-77 7B (Rozière et al. 2024). MathCoder2-Llaan Llama-3-88 3-8B is trained for 3 epochs with
global batch size of million tokens and 8192 token context length. MathCoder2- Deepeere2-DeppSeeMMtt
are each trained for 3 epochs with global batch size of million tokens and 4096 token context length.
MathCoder2-Mistrll and SouhhCodr2--odell
Baselines. We compare our method with various other base models that possess strong mathemat- ical abilities and are of similar sizes, including Qwen2-Math 7B (Yang al. (024a), Qwen2.5. Math 7B (Yang et al. 2024b). MMternLM2-Matth Base 7B (Ying et 20224) Interni.M .22.5 7B (Cai
let al., 2024), SeepSeekMath
(Shao et al. [024)) Llemma 7B (Azerbayev et al., 2024), Mistral
7B (Jiang al.) 2023). Llama2 7B (Touvron et al. 2023), Llama3 8B (Dubey et al. 2024 and Code-Llama 7B Rozière et al. 2024).
Results: As demonstrated in Tab. continued pretraining MathCode-Pile consistently improves performance across all five benchmark datasets. MathCoder2 models rival the performance of top models like IntemLM2-Mathh 112--aat-BBse, InternLM2.5, and DeepSeckMath. In particular, MathCoder2- DeepSeekMath demonstrates that our method continues to enhance the performance of DeepSeek- Math, model that has already been extensively trained on large amounts of math-related data. How-

---

Published as a conference paper at ICLR 2025
Table 3: Performance of various finetuned models five representative mathematical datasets. All results reported are based on greedy decoding. Model
Size
MATH
GSM8K
OCW
Olympiad Bench
SVAMP
Qwen2-Mann Math--nstrectt
7B
89.9
34.6
38.2
(wen2.5-Math--Inse
7B
83.6
95.2
37.1
41.6
DeepSeekMath- Instah---sttut-Coo
82.9
DeepSeckMatt--nstruee
InstruucttttttT1
7B
57.4
83.7
Internl Interni M2-mathh--as
7B
54.4
84.0
17.3
18.8
MominaMath-7BBCCO
7B
55.2
75,4
19.1
19.9
NuminaMath-7BBI
7B
68.1
84,6
ToRA-Code
7B
44.6
72.6
70.4
MathCoder
7B
30.2
67.8
70.7
MAmmoT112-Pltm
8B
42.8
84.1
Llama-3. 1-mm--3t-Inssruutt -Insttruct
47.2
76.6
21.7
15.4
8B
58.5
83.9
29.4
25.8
92.7
8B
69.7
85.8
37.6
37.6
94.9
Setinodddee-DeepSseaattt----ee InstructtCot
7B
53.2
80.3
30.9
23.0
92.1
MathCoodrr-DDeepScckaabb Instrucctttt
7B
69.6
86.5
41.9
37.9
92.8
ever, there remains performance gap between MathCoder2 and the Qwen2-Math and Qwen2.5- Math models. This gap might be attributed to their superior computational, manual, and financial resources, which enable the scaling of data size and the further improvement of data quality, report- ing a mathemtaical dataset of 700B tokens (Yang et al. 2024b).
In contrast to models like Qwen2-Math, which only open-source their model weights, with much of their data processing and training details undisclosed, MathCoder2 is fully open-sourced, including all data processing pipelines and training code. This openness facilitates transparency, reproducibil- ity, and further research, which is crucial for advancing the field. Compared to Llemma, which also open-sources its code, our method achieves better performance on the five datasets. Particularly, when trained the same base model, Code-Llama, method performs significantly better, which demonstrates the effectiveness of the MathCode- pretraining corpus.
3.2 POST-TRAINING
To further demonstrate the potential of the MathCoder2 models in aligning to mathematical problem- solving tasks, we select the WathCoder2-Llama--
model and Saathooder2-DeepSeee for finetuning on mathematical problem-solutice pairs. We first train the base model on general mathematical instructions following Yue al. (2024) for two epochs. Subsequently, we finetune the model on NuminaMath-Con Coo and NuminaMath-11 TIR datasets for three epochs.
The results are shown Tab, 3 MathCoode22Innttuuct datasets, reaching 69.7% on MATH and 86.5% on GSM8K, outperforming many of the best open- source models of similar size and demonstrating our method's potential to improve performance on downstream mathematical reasoning tasks. As this paper focuses continued mathematical pretraining, the post-training serves only as a validation of the potential our models. We conducted only simple supervised fine-tuning without performing reinforcement learning direct preference optimization, which could further improve performance on downstream tasks.
achieves high performance on all five
3.3 ABLATION STUDIES
In this session, we first analyze the impact of various components of the training data. Next, we compare MathCode- to other open-source mathematical pretraining corpora.
Analysis of the impact of the mathematical code. We analyze the impact of the mathematical code on continued pretraining by comparing the results of adding and not adding the mathematical code. As shown in Tab. the addition of the mathematical code in the pretraining corpus significantly

---

Published as a conference paper at ICLR 2025
Table 4: Analysis of the impact of the mathematical code. The upper half presents the results of using and not using the mathematical code data. The lower half analyzes design of concatenating the reasoning steps and code snippets. "Basic + Reasoning-step-onn conditions, expressions, and results, while "Basic + Trans-code-only" represents only adding the translated code. "Basic + Separated Text&Code" represents seperating corresponding code and represents the concatenated data combining both. "Basic No-code-
represents only adding the
text. Creasoning-Step&&Con
prompt" represents using a prompt that simply instruct Llama-3.1-70B-Ile
rewrite texts to
improve their quality.
Data Composition
Base Model
MATH
GSMSK
SAT
OCW
MMLU- MATH
Basic Basic Beasoning-Step&&CCCo
Llama-3-8B
34.7
65,8
71.9
12.9
45.2
Llama-3-8B
38.4:+3.7)
69 9++1)
84.4x+12.5)
18.0(45.1)
46.5(+1.3)
Basic Reasoning-step-only
DeepSeckCoder-1.BB
16.7
22.7
40.6
4.8
25.9
Basic Trans-cnde-only
1.epSeekCooder1133B
14.6
22.
43.8
5.5
25.5
Basic No-code-prone
DeepSeckCoder-1.B5
15.7
21.3
37.5
4.8
24.4
Busic Separated Text&Code Code
CedpSeekCoder-13BB
17.0
22.0
46.9
4.8
25.3
Basic Reasoning- StininngStee&CCdeee
DeepSeekCoder- Code
17.8
25.5
59.4
5.9
26.1
Table 5: Analysis of the effect of different components in MathCoder2-Piile DeepSeekCoder-I1. 1.3.B.
The base model is
Data Composition
МАTH
GSMSK
SAT
OCW
MMLU-MATH
No Math Training
4,8
4.3
18.8
2.6
24.8
Witered-OpenWehhMathh (4.8B)
11.4
34.4
3.7
25.4
WebMt (12.9B)
9.4
11.2
31.3
2.6
24.4
filtred-CC--n--mile (6.4B)
9.1
12.1
31.3
3.7
25.2
CC-Fa-math (B2.1B)
13.0
25.0
2.9
25.0
Aitered-OpenWebMMaee textbooks
9.4
12.7
50.0
4.0
25.4
Mitered-OpenWebbMaee synthetic data
10.8
12.6
50.0
4.0
25.6
Blered--peenebhMthh code
9.4
12.1
46.9
4.0
25,4
MathCoder2-Pille
17.8
25.5
59.4
5.9
26.1
improves performance across all five datasets. Note that the mathematical code only constitutes 14.1% % of the 19.2B tokens in the MathCode-Pile dataset, yet the improvement in accuracy it brings about compared to the total improvement accuracy actauncaatt ocCaie 66.2%, and 35.1 % on the five benchmark datasets, respectively, demonstrating the effectiveness of the mathematical code. Comparison across different training steps shown in Appendix FF
21.8%, 27.1% 44.5%,
We also analyze the design choice of concatenating the natural language reasoning step with the mathematical code for pretraining. This analysis is conducted by studying the results of adding only the natural language reasoning steps, and separately adding only the translated code. As shown in Tab. Basic rreasoning-step-onn represents adding only the natural language reasoning steps; Basic Trans-code-only represents adding only the translated code; Basic + Separated Toxt&Code represents adding the con- catenated data that combines both. The Basic Reasoning-Stevn Code configuration results in the best performance, demonstrating the importance of including both the natural language reasoning step and the translated mathematical code.
represents seperating code and text; and Basic + -Rasoning-SSepel
To rule out the possibility that the improvement comes from the higher quality texts generated by L.ama-3.1-70B- - text. The details this prompt are provided in Appendix E We present the results replacing the mathematical code with texts generated using this prompt in Tab. labeled "Basic No-code- prompt". Our method of generating mathematical code accompanied with corresponding reasoning steps outperforms this baseline, demonstrating the effectiveness of our approach.
prompt that asks 1.ama-3.1-705B -nS-ruuttt rewrite the given
Analysis of the impact of various parts the basic data. We perform experiments on a smaller model, DeepSeekCoder-11333 using different parts of the basic data. As demonstrated in
8

---

Published as a conference paper at ICLR 2025
Table 6: Comparison between MathCode-Pile and other Mathematical Pretrain datasets.
Pretrain Dataset
Base Model
MATH
GSM8K
SAT
OCW
MMLU-MATH
No Math Training
DeepSeckCoder-1338
4.8
4.3
18.8
2.6
24.8
OpenWehMath
DeepSeekCoder
9.4
11.2
31.3
24.4
Proof-Pile-2
DeepSeckCoder-J...
11.2
50.0
MathPile
DeepSeekCoder-L.33
21.9
24.9
DeepSeekMath Corpus
DeepSeckT1.M-I.355
13.6
23.8
56.3
MathCoder2-Pile
DeepSeckCoder-11.5
17.8
25.5
59.4
5.9
26.1
Table 7: Comparison between finetuning the original Llama-3-8B, MathCoder2-Basic.. 3ara-----aa----aa------------- --B8m and MathCoder2-Lllma-- on NuminaMath-TIIR Wathhoode2-BBaic-Lllaaae is the model resulting from continued pretraining on the basic data.
Base Model
MATH
GSM8K
OCW
Olympiad Bench
SVA MP
Llama-3-8B
56.1
80.1
24.6
28.4
62.9
26.8
32.9
MathCodeer-Ilama-
65.1
84.5
34.6
34.4
87.9
Tab. [5] filtered- -erredd-OpeWWanee WebMath and filtered-CC- Ca----math significantly improve the performance of the model. In comparison, textbooks, synthetic data, and code smaller in data size and play less important role. As each of these parts of data is too small for individual pretraining, we combine them with Open WebMath-filterre to show that they each bring small yet noticeable improvement compared to using only Open Web and the initially filtered CC-En to remove irrelevant data, also compare the performance before and after filtering. We observe that there is no obvious degradation in performance after removing irrelevant content, showing the effectiveness of the filtering.
Since we performed filtering on Open WebMath
Comparison with other open-source mathematical pretraining corpora. We compare MathCode-Pile with various other open-source mathematical pretraining corpora. We train each corpus for 3 epochs with global batch size of 2 million tokens and a 4096 token context length, since we observe that the model's performance usually saturates around 3 epochs. shown in Tab. 66 MathCode- Pile significantly outperforms Open WebMah,, Proof-Pile-2 and MathPile when trained on DeepSeekCoders mance DeepSeekLL.M1133 is taken from Shao al, (2024), which is trained for 150B tokens, more than our MathCode Pile's training of approximately 60B tokens. The 1.3B model trained with MathCode-Pile outperforms the 1.3B model trained with DeepSeekMath Corpus.
The DeepSeekMath Corpus is not open-source, and its perfor-
Analysis of the improvement on the potential of being finetuned for TIR reasoning. To analyze the effect of the model-translated mathematical code LLMs' potential to be finetuned for TIR reasoning, we finetune the original Llama-3-8B, Llama-3-8B on NuminaMath-TII for three epochs, respectively. As shown Tab. 7 the results of finetuning on MathCoder2-Basic. Basic-Llama-3-88 are higher than the results of finetuning on Llama- 3-8B. Finetuning on Matthooer2-LLlmma-3338 results in even higher performance than finetuning on Basss----aii--aaaa---- 3-BB, showing that the addition of mathematical code effectively en- hances the models' potential of being finetuned for TIR reasoning.
and MathCoder2-
RELATED WORK
Continued mathematical pretraining. Several works (Shao al.) 2024 Azerbayev et al. 2024 Ying et al.) 2024 Yang et al. 2024b) have explored the continued pretraining of LLMs on math- ematical data, such as mathematical web content, synthetic data, and code. InternLM-Maath (Ying ct al. 2024) and Query of CC e et al. (2024) use BM25 for data retrieval, while other works such as Seep (Shao et al. 2024 and Qwen2-Math Yaang et al. 2024b) employ fastText (Joulin eet al. 2016) and other meta- ieeta-information to retrieve texts from Common Crawl. Our approach fol- lows these methods by using fastText for data filtering, and we introduce a second iteration of finer filtering to retain relevant data. MathPile (Wang al. 2023b) and phi (Gunasekar et al.

---

Published as a conference paper at ICLR 2025
utilize real synthesized textbooks, while Llemma (Azerbayev et al. 2024 and Qwen2- Math Yang et al.; 2024b) incorporate math-related code in their datasets. However, unlike our method of generating mathematical code with accompanied natural language reasoning, their code mostly has no natural language explanations or context. Our work builds on these prior efforts by collecting and expanding upon these sources of math-related text. Unlike works that only open- source their model weights, we take a more transparent approach by open-sourcing both our data processing and model training code, thereby ensuring reproducibility and facilitating future research in this field. Compared to Llemma (Azerbayev et al. 2024), which also open-source their data and training code, our method results in better performance on mathematical reasoning tasks.
2023
Synthetic data. Numerous finetuning Yu et al. 2024, Wang et al. 2023a Lu et al. 2024a) and pre- training Gunasekar et al. (2023): Wang et al. (2023b); Yang et al. (2024b) studies have explored train- ing on synthetic data generated using language models or predefined templates. MathGI.M (Yang) et al. 2023) and IiternLM-Mail (Ying et al., 2024) use templates to generate synthetic numeri- cal operation data, while phi (Gunasekar al. 2023 produces textbook -quliityy data with models.
EntiGraph Yang et al. 2024c) generates diverse text by drawing connections between sampled enti- ties. Our work proposes novel method for extracting mathematical reasoning steps and generating synthetic code snippets that captures the underlying reasoning processes.
Post-training. There many methods for further improving the mathematical problem-solvingg abilities of L.Mss. Supervised finetuning adjusts pretrained models using math problems and solu- tions in various formats, such Chain-of-Thouglh Yuu et al. [024] Yuan al.)
(023). Program-
of-Thought (Yue et al., (023), and Tool-Integrated Reasoning (Go el al. 2024]] Wang et al. 2023a
Liao al. 2024). Reinforcement learning Lightman et al. (2023); Wang al. (2024) and Direct
Preference Optimization Ratailov et al. (2024);(Xu) (2024)) et al. (2024)) utilize mathemati-
cal preference data adjust the models' outputs. These methods are diverse and reveal the potential of pretrained models. Their performance is often influenced by the quality of the training data used
in the pretraining stage. To explore the potential of finetuning our pretrained models for downstream
tasks, we conduct supervised finetuning with existing open-source data.
5 LIMITATIONS AND FUTURE WORK
One limitation of our work is that our continued pretraining corpus focuses primarily on mathe- matics and does not intentionally include other STEM subjects. Additionally, our pretraining data consists entirely of English texts, without incorporating math-related content in other languages, like Chinese. Due to limitations in computational resources, we only trained models ranging from 1.3B to 8B parameters. Future work could address these limitations by expanding the dataset to include other subjects and languages and by training on larger language models. Also, this paper primarily focuses continued mathematical pretraining, we did not apply reinforcement learning meth- ods like PPO and GRPO, or Direct Preference Optimization in our post-training phase, which can further improve performance on mathematical reasoning tasks. In the future, we could explore these methods on our finetuned models. Also, this work did not discuss theorem proving with formal languages such as Lean and Coq, which is worth investigating in future works.
6 CONCLUSION
In this paper, we present an effective open-source continued mathematical pretraining pipeline for enhancing mathematical reasoning of LLMs. Through the meticulous collection and filtering of di- verse math-related texts, such as mathematical web content, synthetic data, code that uses mathemat- ical packages, and math textbooks, we curate basic dataset for continued mathematical pretraining. We then propose a novel method for extracting mathematical reasoning steps from the previously collected dataset and translating them to code snippets reflecting the underlying reasoning processes. By combining the basic data with the -enerrted mathematical code accompanied with their corresponding reasoning steps, produce 19.2B-toke mathematical pretraining corpus named MathCode- -Pile, which significantly improves the performance of four different base models across five representative mathematical benchmarks. By open- open-soureing the entire data processing pipeline and model training code, we actively promote transparency, reproducibility, and collaboration within the research community, facilitating future research in this area.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
This project is funded part by National Key R&D Program of China Project 20222D0161 100, by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technol- ogy Commission (ITC)'s InnoHK, by NSFC-RGC Project N. N.CCHK498/24, Hongsheng Li is a PI of CPII under the InnoHK.
REFERENCES
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Al- bert Q. Jiang, Jia Deng. Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2024. URL
10631
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piga: Reasoning about physical commonsense in natural language. 2019. /ttps:s::///xx www.w/wwwwww..aa//a/aabaaaaa 11641
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo. Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang. Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong. Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv,, Kai Lv, Ma, Runyuan Ma, 7erun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang. Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang. Bin Wang. Guoteng Wang. Jiaqi Wang. Jiayu Wang, Rui Wang, Yudong Wang. Ziyi Wang, Xingjian Wei, Qizhen Weng. Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang. Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.
brg/abs/2110, 14168
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Zhaoye Fei, Yunfan Shao, Linyang Li, Zhiyuan Zeng, Conghui He, Hang Yan, Dahua Lin, and Xipeng Qiu. Query of cc: Unearthing large scale domain- sppeifc knowledge from public corpora, 2024. URL // wraiia..... orq/ abs 220022411.000 14624
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL httphhttt////eeeeee org/ rewg///cordd///oo
12608602
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: tool-integrated reasoning agent for mathematical problem solving. 2024.
URL htt.hhttp////attae www.:oorggaaaooo.... wtwggbb////////oo// 17452
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmman, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. URL httpp. pttss/// /t////axivvvv/ . abs /230. 11644
11

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding, 2021a. URL https:
03300
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021b. 03874
URL httph: htttppss/aaxxi..orrr/bb/////////
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
//arxiv..2g//ab//2310...o 06825
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient http://wxwi..oggabb///ess/ 2ttgg/ab/1660..11555
text classification, 2016.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk. Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sole Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan- guage models, 2022. URL https:://aaat /arxiv.. org/ abs/ 14858
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Lo- gesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang. Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luc- cioni, Paulo Villegas, Maxim Kumakov, Fedor Zhdanov, Manuel Romero. Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavite Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023. URL https: http:p:://axxii..o/ org/ abs / 200 06161
Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan. Mario: Math reasoning with code interpreter output a reproducible pipeline. 2024. URL 081 90
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023. URL
ftttps://arxt www./ao/////oooooooootooo.o..... 020.//25555500000
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation large language models for code generation, 2023. URL
wwwpiii.org//bb.ee htsp//bb//2/33 01210
Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hong- sheng Li. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of Ilms, 2024a. URL
16352
Zimu Lu, Aojun Zhou, Ke Wang. Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hong- sheng Li. Step-controlled dpo: Leveraging stepwise error for enhanced mathematical reasoning, 2024b.
00782
Keiran Paster, Marco Dos Santos. Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023. URL
abs/
231 0. 06786
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model, 2024. URL URI.https:: /tttts://arxiv..oo http:///wwwww..o///////////////
18290
12

---

Published as a conference paper at ICLR 2025
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Ev- timov, Joanna Bitton, Manish Bhatt. Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défosse7, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Seialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code, 2024. URL htttts:///aatttteeee oww.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale, 2019. URL https://axvii wttp:////ggga////9//////
10641
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL /tatpps//arxiv.. www.ii..rgg//oe.ooomooooo abs/2402.
03300
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov. Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL http:///awwiv...oa/////// www./orgggbbbaaeeaee... 2307. 239.809288888
Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song. Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in 1lms for en- hanced mathematical reasoning, 2023a. URL http:::://axxxv.orogg.oo abs/ 20t./2310... 03731
Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce 11ms step-by-step without human annotations, 2024. htttshtss:///eee. org. atsp//bs/22222 23.2. 08935
Zenghi Wang, Rui Xia, and Pengfei Liu. Generative ai for math: Part mathpile: A billion-token- scale pretraining corpus for math, 2023b. URL http::://aaxxv.oorrg////////2111
Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du Wenyi Zhao, Jie Tang, and Yuxiao Dong. Chatglm-math Improving math problem-solving in large language models with self-critique pipeline, 2024. URL https:
//arxxvvorq abs/////////// 2404. 02893
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10677 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-matt technical report: Toward mathematical ex- pert model via self ceff-improvement, 2024b. URL http::ttps///t /ttp::////axvvvooroo/ org/ abs///b////// 2409. 12122
Zhen Yang, Ming Ding, Qingsong Lv, Zhihuan Jiang, Zehai He, Yuyi Jinfeng Bai, and Jie Tang. Gpt can solve mathematical problems without a calculator, 2023. URL https://arxxt org/abs/20s 2tw.aa////...... 03241
Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candès, and Tatsunori Hashimoto, Synthetic continued pretraining. 2024c. URL htt.::://arxi..or//abb///// 07441
13

---

Published as a conference paper at ICLR 2025
Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen, and Dahua Lin. Internlm-math: Open math large language models toward verifiable reasoning, 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2024. URL
hbtprrgg/b/////00 12284
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou, Scaling relationship on learning mathematical reasoning with large language models, 2023. URL Utthttts::// http/axxiivv. http:/twwwi......aaaaoooo att//3/ss3//0 01825
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning, 2023. URL
htttss://arxiie
httt::///arxx..oorrr/////333333 wwtp://////ab/////oo 05653
Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web, 2024. URL http:: http://axxvvv.oeo hwtp://////// 00los/2400 2405 03548
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence?, 2019. URL
Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang. Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang, Yinghao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kaijing Ma, Minghao Liu, Morry Niu, Noah Wang, Quehry Que, Ruibo Liu, Sine Liu, Shawn Guo, Soren Gao, Wangchunshu Zhou, Xinyue Zhang. Yizhi Zhou, Yubo Wang. Yuelin Bai, Yuhan Zhang, Yuxiang Zhang, Zenith Wang, Zhenzhu Yang, Zijian Zhao, Jiajun Zhang, Wanli Ouyang, Wenhao Huang, and Wenhu Chen. Map-nco: Highly capable and transparent bilingual large language model series, 2024. htthsttt::/ /ttp:////axvv.. org/ abs/ 2405 1 19327
Aojun Zhou, Ke Wang. Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. Solving challenging math word problems using GPT-4 code interpreter with code-based self-verification. In The Twelfth International Confer- ence Learning Representations, 2024. URL httph:: //tp:://////eeeeeeee net font/ffoummiioo 3BMC
A PROMPT FOR ANNOTATION OF MATH WEB DOCUMENTS
In this section, we present the prompt we used for annotation of documents in OpenWeeMaae and the initially filtered CC-En. The prompt, as shown in Fig. asks the model classify the document into one of seven types, which are types of documents that frequently appear in the datasets. We observe that this method helps the model better identify and filter out irrelevant text than using a binary classification of whether the text is related to math.
B TRAINING DETAILS OF FASTTEXT CLASSIFIERS
We employ an open-source library for training, configuring the vector dimension 50, the learning rate to 0.5, the maximum length of word n-grams to 2, and the number of training epochs to 5. For the initial filtering of the Common Crawl corpus, we sample million data points from the seed corpus of filtered-Open Whteee--OpenWebMath WebMath as positive training examples and another 8 million web pages from Common Crawl as negative examples. For finer filtering, we use 2 million data points annotated as math metttrelated by fixtral-8x7B-. Instrucct positive training samples and I million data points annotated as unrelated to math as negative training samples.
s httss//fasttext.cc
14

---

Published as a conference paper at ICLR 2025
You will be provided with block of text. I need you to classify the text into one of the following types:
Prompt:
The text describes a mathematical problem and solution.
The text explains a mathematical concept or mathematical theory.
3. The text explains scientific engineering concept that requires mathematical knowledge.
4. The text describes a programming problem and its solution.
The text explains concept or theory related to programming.
The text explains the usage of programming language or software tool.
7. The text does not belong to any of the types above.
Here's the text I've provided. Kindly analyze and classify it into type 1. 2. 3, 4, 6 or 7. Put your choice behind "The type is:". Please do not generate any unrelated additional comments! The type number must match the type description. Here's one of texts needs to classified: (TEXT) The type is:
Figure 3: The prompt for annotation of Open MehMath and the initially filtered CC-En documents. (TEXT) replaced with the content of the document.
You will be presented with related math. I need you carefully read through the text. you any incorrect statments, erroneous computation steps, spelling mistakes, grammatical errors, or formatting issues, adjust them so that the errrr is corrected. Rewrite the text to make it more accurate and easier to understand. You should only output an adjusted version of the given text. Also, do not change the original language. Please do not generate any unrelated additional comments! The text follows: (TEXT)
Prompt:
You should output:
Figure 4: The prompt asking Llama-3 3.1 1--0B--Inttrne to simply rewrite the text and improve its quality. (TEXT} is replaced with the content of the document.
C EXAMPLES OF THE MODEL-GENERATED MATHEMATICAL CODE
This session shows several examples of the translation from math-related texts mathematical code
acompanied with corresponding reasoning steps. As shown in Tab. [8] Tab. [ Tab, I0 and Tab. [I the model first extract the LaTex expression alone with its conditions and result from the original text, then generates an Python code snippet based on this information.
D EXAMPLES OF REMOVED IRRELEVANT TEXTS
In this section, we present several examples in the original OebMMhhaath dataset that irrelevant to mathematical reasoning and removed in the filtering process. As shown in Tab. 12] Tab. [3] and Tab. 14 the content of these documents are related math, but instead are about subjects such as politics, testing software, or web development. Removing these irrelevant texts have no obvious impact on the mathematical continued pretraining performance.
E PROMPT FOR SIMPLE REWRITING TO IMPROVE QUALITY FOR ABLATION STUDY
To rule out the possibility that the improvement results solely from the enhanced quality of the texts generated by 3lama-3.1-700-Instte we designed prompt asking 1lama-3.1-70BB Inntrutntt to rewrite the text, checking for mistakes in content and format to enhance accuracy and clarity, as shown in Fig.
15

---

Published as a conference paper at ICLR 2025
Original Text:
Bayes and his Theorem
My earlier post on Bayesian probability seems to have generated quite lot of readers, so this lunchtime I thought I'd add little bit of background.
If we ask the question "what is the probability of exactly successes from the possible n?", the answer is given by the binomial distribution:
(x(2)), = C(n, x)p" (1 pm)
Translation Output:
Conditions Needed:
1. The number of trials (n) is positive integer.
2. The probability of success (p) a real number between 0 and
3. The number of successes (x) is non-negative integer less than or equal to n.
Computation Expression:
(a m. C(n, z z)p" (1 p) -)"==
Computation Result:
The probability of exactly successes in independent trials, each with a probability of success
Python Code Snippet:
ppython import math def combinations math. math.comb(ae x)
p):
probability = combinations (p ** x) ((1 - p) ** (n - x))
return probability
Example usage:
= number of trials
= =5 number of successes
0.5 # probability of success
result binommil. x. print(resull)
Table 8: An example translating a piece of text to mathematical code and a corresponding rea- soning step.
F COMPARISON BETWEEN ADDING AND NOT ADDING MATHEMATICAL CODE
In this section, we present the comparison between adding and not adding mathematical code across different training steps. The experiments are conducted on Llama-3 8B. As shown in Fig. [S] and Fig. (6 adding the model-translated mathematical code improves accuracy across different training steps.
G REVISIONEFFECT OF MATHCODE-PILE ON GENERAL CODING AND OTHER ABILITIES
In order explore MathCode-Pile effect general coding abilities, test the MathCoder2 mod- els on HvmanEval and MBPP, two representative coding benchmarks, using the EvalPlug frame-
16

---

Published as a conference paper at ICLR 2025
Original Text:
Probability of getting 2 Aces, 2 Kings and 1 Queen in five card poker hand (Part III So reworked my formula method after getting help with my original question
52
54145
5.540678 * 10
Translation Output:
Conditions Needed:
1. The total number of cards in a deck 52.
2. We are drawing 5 cards from the deck.
3. We want to calculate the probability of getting 2 Aces, 2 Kings, and I Queen. Computation Expression:
62
Computation Result:
5.540678 10-5
Python Code Snippet:
python import math
def combination (n, k):
return math comb(n, k)
numerator = combination (4, 2) combination (4, 2) combination (4, 1)
denominator = combination (52, 5)
result numerator denominator
print (result)
Table 9: An example translating a piece of text to mathematical code and a corresponding rea- soning step.
work. HumanEval+ and MBPP+ are extended versions of HumanEval and MBPP with additional test samples, as introduced in Liu al. (2023). The pass @ accuracies are shown in Tab, [T] MathCode-Pile improves the performance of Llama3-8B, DeehSeekMatt-77, and Mistral-7B on these coding benchmarks. The performance of Maahhooder2-CodeLlaama similar to that of CodeLlama-7BB, which is understandable since CodeLlama is specifically trained for code generation. This demonstrates that MathCode-Pill can enhance general coding abilities beyond math-focused tasks for models not explicitly trained on code.
on these benchmarks
To demonstrate how MathCode- Pile affects LLMs' general abilities, we tested the MathCoder2 models Hellaswag (Zellers et al. 2019). PIQA Bisk al. (2019). and Winogrande Saaaguuhi
et al. (2019) using the in-evaluation-harnee 17] As shown in the table below, training on MathCode- Pile slightly impacts the performance of general-purpose models, such Llama3-8B and Mistral- 7B, on these datasets, likely because MathCode-Pile consists entirely of math-related data. The effect on special-purpose models, such DeepSeekMath-7BB and CodeLlama-7B, is minimal.
17

---

Published as a conference paper at ICLR 2025
Original Text: How do you the product rule differentiate g(x) (x²² +)(22? 2xr)²
and h (x) =² 2x N (a)
2x 2
Translation Output:
Conditions Needed:
1. h(a:) is polynomial function
2. The derivative of x²² is 2x
3. The derivative of 22x -
Computation Expression:
de (=² - 2x) 2r - 2
Computation Result:
(x))) 2x - 2
Python Code Snipper:
python
import sympy as sp
s symbols ( ("x1)
= x**2 2+x
h h-prime sp. diff (h, x) print # Output: 2+x
Table 10: An example of translating piece of text to mathematical code and a corresponding reasoning step.
GSMBK
basic data code-added
70.0
67.5
Accuracy (%)
62.5
60.0
55.0
1000
2000
3000
4000
5000
6000
7000 Steps
8000
9000
10000 11000 12000 13000
Figure 5: Comparison of the accuracy GSM8K between adding and not adding mathematical
code across different training steps.
H N-GRAMS OVERLAP BETWEEN TRAINING DATA AND TESTING DATA
We also apply n-gram testing to check for the overlapping ratio between MathCode- Pille and the test datasets, demonstrate the effectiveness of our decontamination. The overlaps at various n-grams very low. The overlap becomes 0.00% when n-grams is 13.
18

---

Published as a conference paper at ICLR 2025
Original Text: How do you use the product rule to differentiate g(x) = (x² + +)( 2x)²
and h (x) = 2 2x h' (z)
2x
2
Translation Output:
Conditions Needed:
1. h((r)) is polynomial function
2. The derivative of x² is 2x
3. The derivative of -2x is -2
Computation Expression:
4- (x² 2x) 2x 2
Computation Result:
(x))) = 2x 2
Python Code Snippet:
python
import sympy as sp
= sp. symbols ( ("x²)
x**2 2*x
h-prime sp. diff (h. x)
print(h_pring ...
Output: 2+x 2
Table 11: An example of translating a piece of text to mathematical code and a corresponding reasoning step.
## Avoiding Weimar Russia
Matthew Yglesias writes:
Matthew Yglesias: Beyond Economies: Over at Brad DeLong's site you can see fascinat- ing discussion of America's Russia policy in the 1990s between DeLong, Martin Wolf, and Lawrence Summers. One remark would make that an extraordinary extent, all three par- ticipants are willing to accept the premise that the only goal of US policy toward Russia in the 1990s was a good-faith effort to induce Russian prosperity, with such efforts being hampered by political constraints, the objective difficulty of the task, and pure policy errors...
Well, yes. Russia was once superpower and may be one again. One would have thought that the history of 1914-1945 would teach ample lessons about the national security undesirability of trying to keep great powers-like Weimar Germany-pood and weak. One would have thought that the history of 1945-1990 would teach ample lessons about the national security desirability of trying to help great powers-like Japan and West Germany- become prosperous, democratic, and well-integrated into the world economy.
One top of the national-security strategic argument there is the economic argument: the fact that richer trading partners are better trading partners: they make more and more interesting stuff for us to buy.
Table 12: An example of removed text irrelevant to mathematical reasoning in Open MatMath.
19

---

Published as a conference paper at ICLR 2025
MicroEJ Test Suite Engineq ## Introductionq[
The MicroEJ Test Suite Engine a generie tool made for validating any development project using automatic testing.
This section details advanced configuration for users who wish to integrate custom test suites in their build flow.
The MicroEJ Test Suite Engine allows the user to test any kind of projects within the configu- ration of generic Ant file.
The MicroEJ Test Suite Engine is already pre-configured for running test suites on MicroEJ Platform (either on Simulator or on Device).
## Using the MicroFJ Test Suite Ant. Tasks'l
Multiple Ant tasks are available in the testsuite-engineji provided in the Build Kit: testsuite allows the user to run a given test suite and to retrieve an XML report document in JUnit format.
javaTesstsuie is subtask of the testsuite task, used to specialized test suite for Java (will only run Java classes).
htmlReport is task which will generate HTML report from list of JUnit report files.
Table 13: An example of removed text irrelevant to mathematical reasoning in Open WebMath.
By Kimserey Lam with
# Conemu A Better Command Prompt For Windows
Jul 22nd, 2017 written by Kimserey with When developing multiple Web api under multiple Visual Studio solutions, it can become very tedious to maintain, run and debug. Opening multiple instances of Visual Studio is very costly in term of memory and running all once also clutter the screen which rapidly becomes irritat- ing. With the advent dotnet CLI tools, it has been clear that the next step would be to move out of the common "right click/build, F5" of Visual Studio and toward "dotnet run" on a com- mand prompt. Last month was looking for a Windows alternative of the bash terminal which can be found on Mac and found ConEmu. ConEmu provides access to all typical shells via an enhanced UI. Today we will see how we can use ConEmu to ease our development process by leveraging only 2 of its features; the tasks and environment setup. 1. dotnet CLI 2. Setup environment 4. Apply to multiple services
Table 14: An example of removed text irrelevant to mathematical reasoning in Open WatMath.
HumanExal. HumanEval+, MBPP and MBPP+, well as general ability benchmarks: Hellaswag, PIQA and Winogrande.
Table 15: Performance of the MathCoder2 models on general coding benchmarks:
Model
Human- Eval
Human- Eval+
MBPP
MBPP+
Hella- swag
PIQA
Winog- rande
Llama-3-8B
40.2
35.4
61.9
52.1
79.2
81.0
73.4
Mathhooder2-Ilama-3388
51.8
43.3
61.9
52.1
75.9
78.1
71.7
De<pSeckMath-7B
36.0
28.7
64.8
52.9
66.4
74.7
64.6
Mathcoodez-DeepscckMahhhh--
36.6
32.3
66.7
54.8
74.0
63.1
Mistral-7B
29.3
23.8
51.3
40.5
81.1
82.0
73.9
MathCuderr--fistrall777
39.6
34.1
54.5
46.8
78.1
78.0
72.3
Code-Llama-77
37.8
35.4
59.5
46.8
62.9
72.5
64.7
MathCoder2-Code-1 -llma--7
38.4
32.3
58.5
47.4
62.8
72.3
63.7
Table 16: Overlap ratios for different n-grams.
D-grams
4
5
8
13
Overlap Ratio (%)
0.21
0.12
0.06
0.03
0.02
0.01
0.00
20

---

Published as a conference paper at ICLR 2025
MATH
38
basic data
36
code-added
Accuracy (%)
32
28
26
22
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000 12000 13000
Steps
Figure 6: Comparison of the accuracy on MATH between adding and not adding mathematical code different training steps.
21


Reviews:
[{"review_id": "MCK0blSBL8", "reviewer": "Reviewer_Yuwf", "summary": "This paper presents a novel approach for enhancing mathematical reasoning in large language models (LLMs). Unlike previous models that used math-related code without detailed explanations, MathCoder2 generates mathematical code paired with natural language reasoning. This process involves filtering a large math-related dataset from web pages, synthetic sources, code, and textbooks to build a high-quality corpus called MathCode-Pile.\n\nThis dataset consists of 19.2 billion tokens and includes LaTeX-extracted mathematical expressions, conditions, results, and Python code to capture the underlying reasoning. MathCoder2 uses this corpus to significantly improve performance on various mathematical benchmarks, achieving results competitive with state-of-the-art models. Moreover, the MathCoder2 framework is fully open-source, which supports reproducibility and transparency in model training and data processing. This work sets a foundation for future research by focusing on reasoning capabilities through detailed code integration.", "questions": "Python was chosen for code snippets; it it possible to use specialized math software language instead (e.g., Mathematica)? This is not a direct limitation of this paper, but a possible future direction.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "MathCoder2’s MathCode-Pile corpus is rigorously curated and filtered from diverse math-related sources, including web data, synthetic data, specialized code, and textbooks. This ensures relevance, reduces noise, and provides a comprehensive dataset tailored specifically for mathematical reasoning, which is essential for pretraining LLMs in this area.\n\nMathCoder2 demonstrates significant gains on multiple mathematical reasoning benchmarks, outperforming comparable models across different tasks. The improvement underscores the effectiveness of continued pretraining on the structured MathCode-Pile corpus and shows MathCoder2's potential for real-world applications in math-intensive fields.", "weaknesses": "There are no major weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tW5Cren1gB", "reviewer": "Reviewer_G66U", "summary": "The paper \"MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code\" explores enhancing the mathematical reasoning capabilities of LLMs through continued pretraining on a novel dataset called MathCode-Pile. This dataset is constructed from various sources, including math-related web data, textbooks, and synthetic data. A key contribution is the generation of paired natural language reasoning steps and Python code, aimed at improving the alignment between mathematical reasoning and executable code. The authors demonstrate significant improvements in mathematical reasoning benchmarks such as MATH and GSM8K, using models fine-tuned with MathCode-Pile. The paper also emphasizes the open-source nature of their data processing and training pipeline.", "questions": "* Code Execution in Evaluation: Is the Python code generated and executed during benchmark evaluations? Clarifying this would help to understand the role of Tool-Integrated Reasoning in the observed performance improvements.\n* Generalization to Formal Proofs: Can the method be extended to generate formal proofs in languages like Lean or Coq? Specifically, how well does the approach handle abstract reasoning steps that require formal verification, which might be better suited to proof assistants rather than executable Python code?\n* Independent Reasoning Steps: Would separating reasoning steps and corresponding code into independent examples still yield significant improvements? Such an ablation could help assess the criticality of their alignment in the dataset.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Originality: The paper introduces a novel method of generating and pairing Python code with natural language reasoning steps, enhancing the mathematical reasoning capabilities of large language models.\n* Quality of Dataset: The MathCode-Pile dataset, comprising 19.2B tokens, is a significant contribution, demonstrating meticulous curation from diverse sources like web data, math textbooks, and synthetic examples.\n* Significant Performance Gains: The use of this dataset leads to notable improvements across various models, including Llama-3-8B, DeepSeekMath-7B, and Code-Llama-7B, especially on benchmarks like MATH and GSM8K.\n* Detailed Methodology: The process of extracting LaTeX expressions, conditions, and results to generate corresponding Python code is well-documented, offering transparency and reproducibility.\n* Open-Source Commitment: The release of data processing and training code enhances the research community's ability to validate and build upon this work.", "weaknesses": "* Generalizability of Code Generation: The method’s applicability to more abstract or advanced mathematical domains is unclear, particularly beyond high-school-level math.\n* Evaluation Uncertainty: It is ambiguous whether the generated Python code is executed during benchmark evaluations or merely used for pretraining, leaving questions about its practical impact.\n* Scope Limitation: The focus on grade-school-level mathematics is not explicitly emphasized, potentially misleading readers about the dataset’s broader applicability.\n* Ablation Study Depth: While the ablation studies show the value of the synthesized code, further exploration into the necessity of aligning reasoning steps with code versus treating them as independent could provide deeper insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8L4DjC1XWL", "reviewer": "Reviewer_Le5X", "summary": "This paper proposes MathCode-Pile, a 19.2B-token dataset of math text and Python code. The dataset includes high-quality math-related web content, code with mathematical packages, math textbooks, and synthetic data. In addition, they present MathCoder2, a family of large language models with enhanced mathematical reasoning capabilities over MathCode-Pile.", "questions": "I have interested about whether the MathCode-Pile’s strong focus on mathematical reasoning might impact the model’s performance in non-mathematical domains. For example, whether this dataset would enhance the model’s general coding abilities beyond math-focused tasks.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The author combining symbolic math reasoning with executable code in the dataset, MathCode-Pile, which is noval. This innovative methodology extends prior research, making MathCode-Pile a significant resource for advanced math reasoning tasks.\n\n2. The paper is clearly organized, with a well-structured explanation of each step in the MathCode-Pile creation and model evaluation process. Figures and tables also effectively illustrate the overall data pipeline.\n\n3. This work has great significance in advancing mathematical reasoning within language models. MathCoder2, using MathCode-Pile, achieves superior results on math benchmarks, demonstrating the potential of code-paired reasoning data.", "weaknesses": "1. The paper lacks a analysis of potential data leakage between MathCode-Pile and evaluation benchmarks, which could artificially inflate model performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)