MegaMath: Pushing the Limits of Open Math Corpora
Authors: Fan Zhou
Zengzhi Wang
Nikhil Ranjan
Zhoujun Cheng
Liping Tang
Guowei He
Zhengzhong Liu
Eric P. Xing
Venue: COLM 2025 (2025)
Keywords: Pre-training Data
Mathematical Reasoning
Synthetic Data
TL;DR: MegaMath is an open dataset of over 300B tokens from web documents, math-related code, and synthetic sources, designed to enhance language models' mathematical reasoning capabilities.

Abstract: Mathematical reasoning represents a cornerstone of human intelligence, driving problem-solving and innovation, and thus serves as a key indicator of the advanced capabilities of large language models(LLMs). However, the research community still lacks an open, adequate-scaled, high-quality mathematical corpus to match the data requirements of top-grade LLMs. We present MegaMath, an open dataset curated from diverse, mathematics-focused sources, designed to enhance LLMs' proficiency in mathematical reasoning. Specifically, MegaMath is curated via following practices: (1) Revisiting web data: We re-extract all mathematical documents with math-oriented HTML optimizations, fasttext-based filtering and deduplication, all aimed at acquiring higher-quality data specifically for the mathematical domain on the Internet. (2)Recalling Math-related code data: We identify high quality math-related code from large code training corpus, Stack-V2, further enhancing data diversity. (3) Exploring Synthetic data: We conduct various data synthesis practices, resulting in a massive dataset including both synthetic text such as QA-style data, and code. By integrating these strategies and validating their practicality via extensive ablations, MegaMath delivers 371B tokens with largest quantity and top quality among existing open math pre-training datasets.

Published as a conference paper at COLM 2025
MegaMath: Pushing the Limits of Open Math Corpora
Fan Zhou; Zengzhi Wang", Nikhil Ranjan, Zhoujun Cheng, Liping Tang, Guowei He, Zhengzhong Liu, Eric P. Xing MBZUAL, Abu Dhabi, UAE (koala99.2r zf, (owang.nlp]@gman egmmal..ccomm com
Abstract
Mathematical reasoning is cornerstone of human intelligence and key benchmark for advanced capabilities in large language models (LLMs). However, the research community still lacks an open, large-scale, high- quality corpus tailored to the demands of math-centric LLM pre-training. We present MegaMath, an open dataset curated from diverse, math- focused sources through following practices: (1) Revisiting web data: We re-extracted mathematical documents from Common Crawl with math- oriented HTML optimizations, fastText-basee filtering and deduplication, all for acquiring higher-quality data on the Internet. (2) Recalling Math- related code data: We identified high quality math-related code from large code training corpus, Stack V2, further enhancing data diversity. (3) Ex- ploring Synthetic data: We synthesized QA-style text, math-related code, and interleaved text-code blocks from web data or code data. By integrat- ing these strategies and validating their effectiveness through extensive ablations, MegaMath delivers 371B tokens with the largest quantity and top quality among existing open math pre-training datasets.
1 Introduction
Mathematical reasoning is fundamental yet challenging aspect of human intelligence- a persistent difficulty for language models. Recent breakthroughs like ol (OpenAI, 2024) and DeepSeek-RI (Guo et al., 2025) demonstrate that, with sufficient pre-training and large- scale reinforcement learning, models can tackle competition-level math problems. However, the success of such models hinges access to massive high-quality math pre-training datasets-es e-g. DeepSeekMath's 120B tokens (Shao et al., 2024) and Qwen-2.5-Malee 1T tokens (Yang et al., 2024b). Yet open-source dataset currently matches this scale and quality (see Table 9 for comparison), hindering progress on open math models.
A key obstacle lies in the limitations of current math web data pipelines. While web data forms the backbone of modern pre-training corpora (Penedo et al., 2024; Tang et al., 2024), existing math-specific pipelines often suffer from overly aggressive pre-filterngg (egg,, filtering based on HTML math tags (Paster et al., 2024)), which causes many math-relevant documents be missed. Moreover, widely used general-purpose text extraction tools are not optimized for mathematical content- they often strip or discard equations and symbols, severely degrading data quality (Han al., 2024; Lozhkov et al., 2024a). As a result, web- collected math data often lacks both scale and fidelity. Beyond web data, math-related code al., 2023), MathCode-Pie (Lu et al., 2024)) and synthetic datasets (e.g., WebInstruct (Yue et al., 2024)) have shown promising potential, but remain either limited in scale or not fully open-sourced.
corpora (e.g., AlgebraicStack (Azerbayev
To bridge this gap, introduce MegaMath - the largest open-source English math corpus to date, totaling 371.6B .6B tokens. It comprises 279B tokens of web data, 28.1B of code, and 64.5B of synthetic data. During its construction, and optimizations across all domains to ensure both scalability and quality. For the web domain, we designed a two-stage, coarse-to-fine extraction and filtering pipeline, improving
conducted extensive ablation studies
Eqqul contribution.

---

Published as a conference paper at COLM 2025
UM Refwing
Magalauh
Higher quality Larger quantity
MegaMath Web Pro
CoT Performancs
11:10
Data Quality
Score bassd Fiburigg
Maga Math-Wab (Top 50%)
VIN
Mega Web (Top 75%)
Mogalfath
MegaMath
Magultatt 1opp 50%)
MegaMathh (Pull
Mega Math-MatWWeb Full
RriMathh
Open
Training tokens
Figure 1: Comparison with existing open math corpora and MegaMath-Wei subsets.
on the common pipeline. We reformatted math elements in HTML into compatible text representations (i.e., (8Tx)) preserve equations and symbols during extraction. In the first stage, we applied a fast text extractor alongside a fast classifier to filter candidate math documents. After deduplication, we reprocessed the retained HTMLs using a slower, high-quality extractor, followed by second-stage fastText trained on the seed data from the first stage to mitigate distributional shift. This pipeline achieves both scale and fidelity, resulting in Megaa.ath-Web. Based on this foundation, we further developed MegaMath- Web-Pro, a premium subset delivering top quality via LM-based filtering and LLM refining, particularly beneficial for later training stages requiring higher data quality (Hu et al., 2024). In the code domain, we fine-tuned a small language model to filter math-relevant code snippets at scale, yielding MegaMath-Code. For Mega MathaSat--Sytthiiic, refined QA pairs from math web documents, translated non-Python code snippets into Python, and generated interleaved text-code samples from web content. Together, these efforts form diverse and scalable math dataset backed by extensive empirical pre-training.
extracted and
Our contribution can be summarized with following offerings in MegaMath:
1. An open math-focused dataset containing 371B tokens with optimized data curation pipelines, and a variety of data variants to cater to customized demands. (2.1)
2. A comprehensive set of studies and ablation experiments that rigorously evaluate key design choices in the data accumulation process. ($3.1- -53.)
3. Empirical demonstrations including head-to-head comparison with existing math datasets ($3.5, Figure 1), and further training on latest Llama-3 series of models. ($3.6)
MegaMath Data Curation
In this section, will describe MegaMath's whole data processing pipelines, which include three main components: web data (S 2.1), code data (S 2.2), and synthetic data generated from the former two (S 2.3). Our key design choices are validated through downstream benchmarks or split validation sets. For computationally intensive operations like dedupli- cation, we prioritize solutions that balance efficiency and effectiveness.
2.1 Curating MegaMatt-Webb
Web data takes up quite lot of the general pre-training corpora, from which Common Crawl is what has been widely used as pre-training data in many recent LLMs training (Dubey et al., 2024; Yang et al., 2024a; Liu et al., 2024). In MegaMMat, we 99 Common Crawl snapshots (2014-15 to 2024-46) as data source to extract high-quality math documents on the Internet. The overall pipeline for web data is presented in Figure 2, with detailed description in the following subsections. In short, our pipeline contains the following steps: (1) data acquisition;
(2) first round text extraction; (3) dastText-based math filtering; (4) deduplication; (5) second round text extraction; (6) further filtering and post-processing.
2.1.1 Data Acquisition, URL Filtering, and Language Identification
Instead of using WET (WARC Encapsulated Text) data or simply filtering from public pre-training datasets (Penedo et al., 2024; Tang et al., 2024), we re-extracted all the text from
2

---

Published as a conference paper at COLM 2025
MagaMarth Web
Common Craw!
Reformatted html
Teat RedEttoction
Math Text
Math Fiter & others
WARC Files
Files
[Trafiatura]
7.URL
Imprars Readering
Whirensl
Find corresponding himi containing math leed
Ectraction
Text
Language
Math Text
[Recliporinj: Fout but Navey
4. Dedaplication
Figure 2: The pipeline for curating MegaMath-Web from Common Crawl data.
WARC (Web ARChive format) file format where each web page is stored as an HTML file. This practice enables us to optimize text extraction from HTML, enhancing corpora quality specifically for math domain ($ 2.12). We downloaded all available CC dumps and applied URL filtering strategy before text extraction (Penedo al., 2023) to exclude domains related to adult, gambling content, etc. Next, we used an off-the-shelf fastText model (Joulin et al., 2016) for language identification and retained only English documents (score 0.65).
2.1.2 Improved Text Extraction for Math Content
Extracting texts from WARC using common extractors (e.g., Resiliparse and trafilatura) could produce higher-quality corpora over WET extraction (Li et al., 2024a). However, these extractors often fail to preserve math symbols and equations, even omitting them entirely (Lozhkov et al., 2024a). To address this, we introduced several HTML parsing optimizations specifically for optimizing math expressions before extraction. Our approach involved traversing the HTML DOM tree parsed by Resil iparse and modifying math elements-related nodes to obtain an improved HTML. file easy for text extraction including:
1. Math Element Conversion: Converts MathML and KaTeX content into LaTeX by extracting annotation tags or using mathml2latex (with namespace handling) and recursively parsing HTML to accurately extract subscripts and superscripts.
2. LaTeX Standardization and Transformation: Removes unnecessary style commands, fixes symbol formatting issues, and converts HTML tags (such <sup>, <sub>, and "intbl" spans) into appropriate LaTeX constructs.
3. Unicode and Entity Conversion: Maps mathematical Unicode characters and HTML entities to their corresponding LaTeX commands using W3C standards.
Two-stage Extraction Our extraction process consisted of two phases, each serving distance purpose. In practice, Resiliparse and trafilatura are widely used for pre-training corpora construction, but they have trade-offs: Resiliparse is significantly faster and retains HTML. elements more faithfully, while trafilatura, though slower, removes noise more aggressively using various extraction engines and heuristics. Unlike prior works based solely on Resiliparse (Paster al., 2024; Han et al., 2024), our pipeline first applied Resiliparse for rapid extraction and filtering, significantly shrinking the candidate data size. For these candidate data, we then used trafilatur a on their WARC files for a second round HTML optimizations and text extraction, obtaining cleaner mathematical data. This coarse-to-fine approach improves text quality while maintaining development efficiency.
2.1.3 Robust Math Document Recall
Common Crawl (CC) contains a vast array of texts from diverse domains. To effectively filter texts scale, require robust and efficient classifier. We used fastText (Bojanowski et al., 2017), a lightweight n-gram model, to score and identify math-related texts. During development, identified the following key factors to obtain robust fastText classifier:
3

---

Published as a conference paper at COLM 2025
1. Text normalization: Techniques like tokenization, case folding, digit normalization, and Unicode handling while managing whitespace and special characters achieve better training compatibility.
2. Seed data:
Uniform sampling from Common Crawl and adding CoT data helps. 3. Comprehensive evaluation: Expanding beyond web texts to Wikipedia, textbooks, StackExchange and research papers improves recall assessment.
fastText Training We started fastText training with one million positive and negative seed documents from Open-Web-Math and random web documents from CC. Initially, we used single snapshot dump for development, which risked reinforcing biases. To mitigate this, we sampled from all CC dumps and retrained the classifier during the second-round filtering process. We used annotate math relevance scores (see Figure 7 for the prompt) on these filtered documents and CoT data was incorporated into the positive set as well, resulting in two million seed data. We used the same training hyperparameters as DeepSeekMath (Shao et al., 2024).
1-1.a..3-770BBsssttuc (Dubey et al., 2024) to automatically
fastText Evaluation When iterating training strategy, we found evaluation on 20K in- distribution (ID) samples yielded easily over 90% Fl score, masking fastText's 's true per- formance. We thus created an out-of-distributio (OOD) suite by sampling arXiv, Stack- Exchange, Wikipedia, and Textbook data from MathPile (Wang et al., 2024). In the OOD setting, our text normalization and training adjustments boosted the average F1 score from 81.8% to 98.8%, validating the effectiveness of our training strategy.
2.1.4 Data Deduplication
Data deduplication plays a vital role in data curation process, especially for improving training efficiency, stability and reducing data memorization (Lee et al., 2022; Tokpanov et al., 2024). We adopted the Locality Sensitive Hash (LSH) implementation of MinHash (Broder, 2000) for efficiency. Given two documents, the probability that they are assigned to the same hash bucket depends on their Jaccard similarity (Broder, 1997) S and is given by P= - 5 where b denotes the number of hash functions per bucket and represents the number of buckets. Given a fixed hash permutation scheme (b r), which strongly correlated with memory cost, and target Jaccard similarity threshold it is desirable to find the optimal deduplication configuration- one that ensures a rapid decay of for any S < f. Considering our CPU capacity, we evaluated multiple configurations with the number of permutations between 110 and 128 and E {0.70, 0.75, 0.80]. Assisted by training experiments, we determined that the most feasible choice is 11, b 10, and = 0.75.
2.1.5 Curating MegaMath-- A Premium Subset
It is increasingly common practice to filter top-quality data due to its superior impact on model performance (Abdin et al., 2024). High-quality data not only enhances performance but also does so at a lower cost, making it ideal for continual pre-training, mid-training, BrogaMMtt--eeb-Paa
or scenarios with limited budgets. We thus further developed premium subset filtered and refined from MegaMatt--ebb. We employed the FineMath classifier (Lozhkov et al., 2024a) filter out low-quality text. Subsequently, we used LLMs to further refine the text, ultimately delivering 15.1B tokens that significantly surpass all existing math corpora such as FineMath-4plus (Lozhkov Though L.M was involved, we focused primarily on noise removal and text reorganizing thus this is not categorized as pure synthetic data. See $B.2 for full developing strategy.
al., 2024a) (cf. Figure 1).
2.2 Curating MegaMath-Code
Code pre-training has proved to enhance general reasoning (Shao et al., 2024; Aryabumi et al., 2024), and LLMs have also shown great potential to leverage code for problem-solving (Gou et al., 2024; Li et al., 2024b). Thus, we believe blending code in math LLM training is also crucial. We built MegaMath-Code based on the Stack V2 (Lozhkov et al., 2024b), pipeline
and employed multi-step pipeline to recall high-quality code relevant to mathematical reasoning, logic puzzles, and scientific computation. As shown in Figure 3, our pipeline consists of: (1) Programming Language Selection (52.2.1) and (2) SLM-based Code Recall (52.2.2).
4

---

Published as a conference paper at COLM 2025
General Code
Scoring
Seed Data
Selected Code Data
MegaMath Code
Train
Corpus
Longuage
Filtering
Kept
LM
Stockyy2
Filtered
Figure 3: The pipeline for curating MegaMath-Code.
2.2.1 Programming Language Selection
The code pre-training corpus includes hundreds of programming languages; however, many of these languages are primarily associated with domains that are not closely related to mathematics scientific computation (Lozhkov et al., 2024b). In order to reduce the cost of model-based recall, we selected eleven programming languages based on choices made in previous studies (Azerbayev et al., 2023; Xu al., 2024b): C, CHf, C++, Go, Java, DavaScript, Python, R, Rust, Shell, SQL. The selected languages are either extensively used in scientific computing and numerical operations or represent a significant portion of the corpus, which may potentially include mathematics-related snippets.
2.2.2 SLM-based Code Data Recall
We applied small language model (SLM) based recall mechanism identify math-related code snippets from public code pre-training datasets. Inspired by recent works (Penedo et al., 2024; Zhou et al,, 2024; Wei al., 2024), we first used a strong LLM to score code quality (educational value) and mathematical relevance, assigning discrete score from 0 to 5 for each aspect, as applied in several works (Yuan et al., 2024; Penedo et al., 2024). Then, we trained SLM on these data for large-scale filtering, (Please see SC for more details). We also found that: (1) Stricter filtering greatly enhances performance to solve problems using code; (2) Allocating
more than 20% of code data maximizes code-integrated problem-solving ability while maintaining NL reasoning benefits. This aligns with DeepSeekMath's (Shao et al., 2024) training recipe and further reinforces the justification for our filtering strategy, and we also empirically show the reasonability of this choice $3.3.
2.3 Curating MegaMath-Syntheiic Data
Beyond being a high-quality mathematical corpus, MegaMath also serves as a strong foundation for large-scale data synthesis. We explored data synthesis methods to further enhance both the quantity and quality of our dataset. Our synthesis spans three distinct formats: (1) Q&A data, (2) code data, and (3) text & code block data.
MegaMoth-Web
MegaMath-Code
MegaMath-Web
Code
Extraction
18
Pythan date Trandiatio
fillring
A2
Q3
A3
Block
Block
Block
Block
Biock S
Solution Refirmment
Genoroted
Block
Block
Block
Block
Bluck 5
Aule-busd Filleing
Venfied
Systax Enror
Auntine Enor
Asswer Misnolch
Venfied
Trenslatdd
Tode Block
Block
Block
Figure 4: The pipeline for curating synthetic data. Left: QA data generation; Middle: Python code augmentation; Right: text & ode block data curation.
data is inherently well-structured and embod- ies concentrated form of knowledge, making it valuable for problem-solving bench- marks (Maini et al., 2024). Recent work reveal that these data can be found in pre-training data with massive quantity (Yue et al., 2024). We thus integrate and further verify this in
Q&A Extraction Question-and-answe

---

Published as a conference paper at COLM 2025
MegaMath. Our pipeline contains two steps: (1) identify and extract Q&A pairs from the raw documents; (2) refine the Q&A to make up or improve the intermediate reasoning steps. To improve diversity and accumulate quantity, ensembled refined Q&A data from Qwen-2.5-72B-Instree (Yang et al., 2024a) and 1lmma-.3-70B-Insttre (Dubey et al., 2024).
Code Translation To enhance Python code data, we employed LLMs to "translate" code from other programming languages into Python, thereby augmenting the code data. We adopted straightforward zero-shot prompting approach using open-source LLMs or code-specialized LLMs to enhance the volume Python code. Specifically, we experi- mented with two models: Owen2.5-Coder-3 -Coodo-32B--Inst -ns-ruutt (Hui et al., 2024) and Llama-3.1-70B- Instruct (Dubey et al., 2024). Please kindly refer to SC for the translation prompt.
Text & Code Block Generation Recent work by Lu et al. (2024) introduced a synthesis pipeline for obtaining "generated mathematical code", consisting of interleaved text, sym- bolic expressions and code blocks. Such data has been shown to enhance model's ability to generate Python snippets for solving mathematical problems and to leverage execution feedback to refine solution steps. As illustrated in Figure 4, we unify such process into: (1) LLM-based generation: given a document, LLMs generate multiple struc- tured blocks including title, mathematical expression, result, and corresponding code; (2) Verification via execution: ensures that the generated code executes correctly without errors and produces expected outputs; (3) Packing verified blocks: combines validated blocks into a single training sample for downstream use. Besides, we found Lu et al. (2024) did not account for malicious code, handling only basic errors like timeouts. To address these issues, we implemented a pre-filtering mechanism based on Abstract Syntax Tree (AST). Any snippet flagged as risky is excluded from execution, safeguarding a 100% execution success rate without abrupt halts or segmentation faults during our curation.
2.4 Dataset Decontamination
To mitigate benchmark contamination (Xu et al., 2024a), we checked the overlap between MegaMath and 12 downstream benchmarks widely used in evaluating LLM's mathematical reasoning ability such as GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), MMLU (Hendrycks et al., 2020), and AIME 1, We concatenated problems and solutions together whole sample, checked the exact 13-gram match, and ruled out contaminated documents. This further removes about 0.01% of the documents from the dataset.
2.5 The Final Dataset: MegaMath 371B Collection
Combining all previous efforts together, the final collection of MegaMath datasets currently contained a total of 371B to- kens (count by the Llama- tokenizer). We present detailed breakdown statistics about MegaMath in Table Designed for various training stages, training budgets, and base model capability, we offer a collection of MegaMath data
Table 1: The category and statistics of MegaMath.
Category
# Sample(M)
Toks(B)
Avg. Toks)
Web Domain
121.5
279.0
2296.9
Web
106.5
263.9
2478.7
Web-Pro
15.0
15.1
1006.0
Code Domain
13.4
28.1
2102.7
Synthetic Data
80.2
64.5
804.5
Translated Code
7.4
7.2
979.5
Q&A
22.6
7.0
308.3
Text&-Code Block
50.2
50.3
1002.1
Total
215.1
371.6
1727.6
variants including: (1) MegaMath-Web: the complete web datasett consisting of 263.9B tokens, and also MegaMath--eebPro (15.1B), the top-quality subset obtained through LM- based scoring and refining. (2) MegaMath-Code Cade (28.1B): math-related code corpus recalled from Stack-v2. (3) MegaSath-Synth (64.5B): LLM--bssed synthetic data enhancing both the quality and quantity, covering three distinct formats of text and code data.
"https:/huugiiggfacee httpr/gignfacc..oo//attttttt datastts dat.aatts//--OO AI-MO/ /

---

Published as a conference paper at COLM 2025
3 Ablation and Demonstration of MegaMath at Scale with Pre-training
During data curation, we conducted extensive pre-training experiments on MegaMath to ablate each key decision. In this section, we present the experimental details, key results, and finally scale up training to further demonstrate the effectiveness of MegaMath.
3.1 Setup
Proxy LM for Ablation During development, we used small proxy model for ablations each data source and component. We chose Tiny TinyLlama-IB (Zhang et al., 2024) for its small size and transparent training, ensuring it effectively monitors data quality. We trained within a controlled budget, typically set to 5/15/55 B tokens, depending on dataset size and experimental cost, and evaluated performance at 1B token intervals.
Evaluation We used a total of 10 math-related benchmarks, splited into two sets: Core and Extended. The Core set includes five math-focused tasks with stable improvements even under limited training, such as GSM8K and MATH. Building on this, the Extend set further includes five datasets, either indirectly related to math with performance fluctuations, such as MMLU-STEM We employ two prompting-basee base- evaluations: (1) few- shot CoT (Wei et al., 2022) for all benchmarks; (2) PAL (Gao et al., 2023) for the Core set to assess problem-solving via Python code generation. Please check S G for more details.
3.2 Ablation Mega
Importance of optimizing text extraction for math content We conducted continual pre-training experiments within a 15B-token training budget on one dump from 2024. The training corpora consisted of filtered math documents from vanilla trafilatura, and text extracted from the optimized HTML using Resilin Resiliparse and trafilatura, all de- rived from the first-round filtering. During original trafilatura extraction, <math> el- ements in HTML for math-related HTML, the extracted data from trafilatura well-preserved math symbols and clearly improved CoT downstream performance (cf. Table 2). When both extractors operated on our optimized HTML, Resiliparse preserved more noise from the original documents, leading to lower data quality compared to trafilature
Table 2: Ablation on Text Extraction for Math
Text Extractors
w/ HTML
Core Avg.
Ext.
Optimization
Avg.
Base Model
11.2
14.7
trafilatura Resiliparse trafilatura
x
22.0
19.2
22.5
18.6
23.8
20.6
directly discarded. After applying specialized optimizations
Parameters of Deduplication To minimize re- dundancy and reduce the costs associated with follow-up text re-extraction using trafilatura we conducted ablation pre-training experi- ments all 2014 dumps with 55B-token training budget to optimize the parameters for Minhash LSH. Our goal was preserve downstream CoT performance while retaining as many mathematical documents as possible within our cluster capacity. As shown in Table 3, applying r = 11, b 10 provided the optimal balance. We reported the average of the last 5 checkpoints to avoid result fluctuations.
Table 3: Ablation on MinhashLSH Dedup
Tokens Left (B)
Core Avg.
Ext.
(r, b)
Avg.
(14, 9)
0.70
16.0
17.3
(14, 8)
23.5
19.1
17.0
(11, 0.75
26.0
19.4
17.5
(11, 11)
0.75
25.0
19.2
16.0
(9, 12)
0.80
29,0
18.8
16.9
(9, 13)
0.80
30.0
17.6
15.7

---

Published as a conference paper at COLM 2025
Ablation fastText Initially, employed Open- Web-Math as the positive seed data for training fastText used for the first round filtering with loose threshold. While it worked well on the single dump used for initial development, it became less accurate when scaled to all dumps, likely due to shift- ing data distributions. We thus re-trained fastText with LLM-annotated math-related documents from all dumps as the positive seed data. As shown in Fig- ure 5, the re-trained fastText (V2) performed better in the second-round filtering compared to the initial version (V1). Our controlled experiments revealed that balanced sampling seed data from each dump provides slight improvement while incorporating CoT data into positive seed data yielded significant gains. Note that the ablations were conducted the top 10% scoring filtered data from all dumps in 2024 within 5B-token training budget. We further validated our decision through experiments on all dumps yearly shown in Figure
Core Performanoe
22
16
Training Tokens (B)
VI: OWN
-B- V2: Random
V2: Balance - V2: Balance CoT data
Figure 5: Ablation on fastText
3.3 Ablation on MegaMath-Code
We conducted sets of ablation studies using 5B training tokens: (1) evaluating the impact of recall filtering criteria on downstream performance 2, and (2) ensuring the recalled code dataset is sufficiently large for training despite aggressive filtering. As shown in Table 4, stricter filtering (i.e,, Sedu 2 4 and Satth 2 4) selects code data that significantly boosts PAL performance. Moreover, even though such strict filtering excludes many samples, the remaining data appears sufficient for effective mathematical training; in fact, mixing too much code data seems harmful for CoT Performance (e-g. 30%).
Table 4: Ablation of filtering criteria.
Table 5: Ablation of data mixture ratios.
Filter Criteria
CoT Avg.
PAL Avg.
Mix Ratio
CoT Avg.
PAL Avg.
text only
19.0
15.6
text only
19.0
15.6
Sedu 2 3, Smath 2 3
19.4
16.1
code: : text 12.5%
19.3
17.4
Sedu 2 3, S Satth 2
19.8
16.8
code : text 20.0%
19.5
1844
Sedu 2 4, mathh 2 3
19.7
17.5
code text 33.3%
16.4
17.5
Sedu 2 4, Smath 2 4
18.8
19.5
code text 50.0%
17.5
3.4 Ablation MegaMatth-Snnhesis
Our synthesis development proceeds in parallel with web data acquisition. Thus, in web data synthesis, we started with existing public сorpora rather than MegaMath-Web. In particular, we utilized quality-filtered subsets of -pen-Web-Math and mnimm--ebbMatt to lower experimental cost. We focus on: (1) Verify that the generated data can boost performance more effectively. (2) Evaluate the impact of different prompts and models on performance, ultimately guiding better strategies.
QA generation We implemented the two- stage pipeline in WebInstruct (Yue et al., 2024) but using the latest and most capable LLMs. Through several prompting itera- tions, we found: (1) using ELi5-style (Ex- plain like am five) prompt for QA refining produces structured solutions; (2) empha- sizing information completeness enhances data quality further. Table shows the 5B training results on different datasets: using prompt with ELI5 improves performance, and further adding information completeness
Table 6: Ablations on prompt and comparison with other data. FM-4plus: FineMath-4plus
Data
Core Avg.
Ext. Avg.
FM-Iplus
28.3
19.6
WebInstruct
34.6
17.6
Vanilla Prompt
39.2
19.5
ELI5
41.3
19.2
ma FI.1 + IC
48.8
23.6
MegaMath-Synth-QA
49.6
23.8
When testing filtering criteria, we prioritize PAL results, and use Python subset for training.
8

---

Published as a conference paper at COLM 2025
(EL15 + IC) yields the best ablation results, with Core and Extended scores of 48.8 and 23.6. Besides, we also show the performance on our final SaggaMath-Synth-O2 demonstrating overall best results. These results indicate that structured and comprehensive extraction keys to enhancing QA data, and also show that QA style data exhibits superior performance to web documents.
Code Synthesis Our experiments com- pared training on raw code data to meth- ods incorporating code translation and inter- leaved text & code blocks. With controlled training budget of 5B tokens (see Table 7), translated code (trans. code) yields modest downstream improvements over raw code, while adding code block data further hances both CoT and PAL performance, even without mixing text data (see the "full" line results). We also report results on our final MegaMath-TTxt & Code Block corpus. Overall, these results clearly show that synthetic data achieves higher quality.
Table Ablations on Code Synthesis: The default code-to-tex ratio is 1:7, "full": no text
is mixed. We exclude Lu et al. (2024) due to its partial release (Rx0.25 tokens).
Data
CoT Avg.
PAL. Avg.
trans. code
19.0
20.6
text & code block
22.5
28.1
(full)
MegaMath-Texte & Code Block (full)
33.5
46.3
3.5 Comparison with Existing Math Corpora
To assess the data quality of MegaMath, we performed continual pre-training on existing corpora within 55B token budget. We compared MegaMath-Welt with mainstream large- scale corpora, including Open-Web-Mand release. As previously shown in Figure 1, MegaMath--ebb already achieves corpus quality comparable to Infimm--eb--Mrl in downstream tasks but providing substantial more tokens, with performance improving if we higher-scored subsets (top 75% and top 50%). Notably, MegaMath--Wbb delivering the highest-quality corpus date. Furthermore, these MegaMath-Wee variants show the potential to offer flexible options to accommodate different computing budgets.
Infinn-Web-Math, and the latest FineMath
outperforms both FineMath-3+ and FineMath-4+ by 2 4%,
3.6 Putting All Together: Training MegaMath on Cutting-Edgy LMs
We demonstrate the effectiveness of MegaMath by training it on state-of-the- open LLMs the Llama-3.2 series. Given that the Llama-3 mod- els have been extensively trained on 14.8T to- kens (Dubey al., 2024) and exhibit strong per- formance across various benchmarks, we believe they exemplify state-of-the-arr capability and ro- bust performance, making them an ideal vali-
MagaMath
COT Core Porfarmance
50-
1.2m--32-10
tlama 3.233
dation point. For training, we adopt and refine the data mixture configurations from DeepSeek- Math and Llemma to accommodate our diverse data sources, and train 100B tokens for Llama- 3.2-1B and 50B tokens for Llama-3.2-33 We eval- uate all models under CoT and PAL configurations. As shown in Figure 6, the MegaMath series of models achieves a 15% to 20% CoT performance improvement over Llama - for example, reaching 56.2% GSM8K and 25.1% MATH for the 3B model with similar boost observed on PAL. This clearly demonstrates the exceptional quality and effectiveness of MegaMath in advancing mathematical reasoning in state-of-the-at language models. Please refer to S F.2 and G.2 for training configuration and full evaluation results.
30
Ja-1-3,2-18
30
50
65
Core Avg.
Figure 6: Training on Llama-3.2-1B/33
3.7 Pre-training on MegaMath Could also Boost Post-training
A key question is whether the performance boost from pre-training on MegaMath endures after the model undergoes supervised fine-tuning (SFT). To investigate this, we conducted SFT both the standard Llama-3.2-3B- BaBe and our MegaMath-11ama-322555. B.ss....

---

Published as a conference paper at COLM 2025
For this process, we used random 5M subset of the OpenMathInstruutt2 dataset. To ensure a fair comparison, both models were trained for one epoch with identical hyperparameters: learning rate of 5xe-6, global batch size of 256, training for one epoch and the same zero-shot evaluation protocol.
The results in Table 8 show that our MlegaMatt-pretraiinn model consistently outperforms the baseline Llama-3.2-33 1aa--33----Basee after SFT. Notably, also achieves significantly stronger performance than the official Llama-32-3B-Instrrucc model across all math benchmarks. This demonstrates that specialized mathematical pre-training on MegaMath provides a durable advantage that amplified, not overwritten, by subsequent post-training.
Table 8: Performance gains from pre-training on MegaMath persist after SFT. We also report the performance of the official Llama-3.2-Insite for reference.
Model
ASDIV
GSM8K
MATH500
MATH
MAWPS
OCW
SVAMP
TabMWP
78.1
67.5
32.6
3288
90.4
7.7
81.4
37.6
Llama-3.2-3B-BB- SFT
85.8
73.1
41.4
43.6
93.1
7.0
79.8
65.9
MegaaMath-LLama-22-BBas- SFT
91.3
82.7
52.6
55.1
95.1
Related Works
Mathematical Pre-training Corpus and Syntheic Datasets OpenWebMath (Paster al., 2024) curated its data from web pages, with strict filtering which may remove potential docu- ments. Mathl'ile (Wang al., 2024) diversified from web domains and built datasets mostly from arXiv papers and textbooks. Furthermore, InfiMM-Web-Matt (Han et al., 2024) assem- bled a multimodal dataset pairing math text with images. Recently, FineMath (Lozhkov et al., 2024a) was developed by retrieving from FineWeb (Penedo et al., 2024) and using BERT classifier to select clear, step-by-step math explanations. For synthetic math datasets, recent work such as NuminaMath (Li et al., 2024b) converted competition-level problems into chain-of-the oftohuuttthhttttt solutions via tool-assisted reasoning. Meanwhile, Skywork-Math (Zeng et al., 2024), Open.MathInstruct22 (Toshniwal et al., 2024) and WebInstruct (Yue et al., 2024) generated large-scale QA pairs from open benchmarks and web contents. MathCoder2 (Lu et al., 2024) used 19.2B-token MathCode-Pile combining filtered datasets with synthetic code data. In MegaMath, we aim to build a large-scale dataset that matches proprietary corpora via reproducible pipelines, diverse data sources, and thorough sanity checks, finally covering larger quantity and higher quality dataset.
5 Conclusion and Future Work
We introduce MegaMath, the largest training corpus to date tailored for the mathematical domain, comprising 371B tokens from web sources, code corpora, and synthesized data. Comprehensive ablation studies guide us to efficient curation high-quality, domain- specific datasets. Large-scale continual pretraining demonstrates MegaMath's effectiveness by producing strong math base models. We hope the MegaMath dataset, alongside our released artifacts, can foster further research in mathe- matical reasoning and -omain-specific language modeling.
Llama-3 series of model further
10

---

Published as a conference paper at COLM 2025
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arv preprint arXiv:2404.14219, 2024.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlfček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Four- rier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big data-centric URL
training of a small language model, 2025.
org
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski Yejin Choi, and Han- naneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2357-2367, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/ 10153/v1/ND 11/N1911N/1//11 URL https: /taps:///aaaaaiie hat///aaaaaanoooogoooeooo org/N19-1244
Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. To code, or not to code? exploring impact of code in pre-training. ar preprint arXiv:2408.1095 10914, 2024.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q liang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXio:2310.106311 2023.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5:135-146, 2017.
Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pp. 21-29. IEËE, 1997.
Andrei Z Broder. Identifying and filtering near-duplicate documents. In Anmual symposium on combinatorial pattern matching, pp. 1-10. Springer, 2000.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. Xiv preprint arXiv:2110.1416: 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arv preprint ar xrXiv:2407.211 217883 2024.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp. 10764-10799, PMLR, 2023.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. In The Twelfth International Conference on Learning Representations 2024.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. ar Xio preprint arXio:2306.1164, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. Xix preprint ar arXiv:2501 12948, 2025.
11

---

Published as a conference paper at COLM 2025
Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yiqi Wang, Qihang Fan, Yuang Ai, Huaibo Huang, Ran He, Zhenheng Yang, et al. Infimm-webmath-4000 Advancing mul- timodal pre-training for enhanced mathematical reasoning. In The 4th Workshop on R Mathematical Reasoning and Al at NeurlPS'24 24, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ar arxiv preprint arXiv:2009.03300 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arviv preprint ar Xfc:2404.00395 2024.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, al. Qwen2. 5-coder technical report. arx preprint arXiv:2409.12186, 2024.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint ar Xric:1607.01755) 2016.
Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Pp. 1152-1157, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10. (0.18653/ 11/N16-1136. URL https: httb::s///cclanhhooogyy http://oooooo.o.o/66oo// wrggN16-1136.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2022.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-38571 2022.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Cururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-Im: In search of the next generation of training sets for language models. Xrv preprint ar #rXiix2400.11794 2024a.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13, 2024b.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. Xix preprint ar Xix:222412.1 19437, 2024.
12

---

Published as a conference paper at COLM 2025
Anton Lozhkov, Loubna Ben Allal, Elie Bakouch, Leandro von Werra, and Thomas Wolf. Finemath: the finest collection of mathematical content, 2024a. URL https: htthtts:: //wgg//igggggg.c....... h//ddaaaset//uugiiggaaaas TBB f/f Finemath.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. ar preprint arXiv:240 19173, 2024b.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chus Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi- structured mathematical reasoning. In The Eleventh International Conference on Learning Representations, 2023.
Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathcoder2: Better math reasoning from continued pretraining model-translated mathematical code. ar Xiv preprint ar axXim:2410008196 2024.
Pratyush Maini, Skyler Seto, Richard Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly, Rephrasing the web: recipe for compute and data-efficient language modeling. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14044- 14072, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ /v1/ / 221/222222llloogg75555 URL https: hacpss/////lattooogoyyy org/ 2024. acl-long, 757.
Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing English math word problem solvers. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 975-984, Online, July 2020. Association for Computational Linguistics. doi: 10.18653 a20 URL https::/aclaatthone wrgee/.rrr///// org/2000.. 2020. acl-main.92.
OpenAL. Learning to reason with Ilms, 2024. terannin-tt-rrrooon-aaa 11ms/
URL
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. In The Twelfth International Conference on Learning Representations, 2024.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur; Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North Ámerican Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080-2094, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 naacl-main.163 URL htttp:://aet httposs///allahhllogyyyy http:ooooyyoogg221111111 naacl-main. 168.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. Xiv preprint ar Xxx::2306.00116, 2023.
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. ar preprint ar Xix::2446.1114 17557, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer, Journal of machine learning research, 21(140):1667,00
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arxiv preprint arXio:2402.0333 2024.
13

---

Published as a conference paper at COLM 2025
Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-cc: Transforming com- mon crawl into refined long-horizon pretraining dataset. arXiv preprint arXiv:2412.02555, 2024.
Liping Tang, Nikhil Ranjan, Omkar Pangarkar, Xuezhi Liang, Zhen Wang, Li An, Bhaskar Rao, Linghao Jin, Huijuan Wang, Zhoujun Cheng, Suqi Sun, Cun Mu, Victor Miller, Xuezhe Ma, Yue Peng, Zhengzhong Liu, and Eric P. Xing. Txt360: A top-quality Ilm pre- training dataset requires the perfect blend, 2024. URL LLM360/TxT360.
co/ wtt.c////sccee spaces/
Yury Tokpanov, Paolo Glorioso, Quentin Anthony, and Beren Millidge. Zyda-2: 5 trillion token high-quality dataset. arXiv preprint ar arXio:2411.06068, 2024.
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstructtz: Accelerating ai for math with massive open-source instruction data. ar Xiv preprint ar XrXi::2410.01560, 2024.
Zengzhi Wang, Xuefeng Li, Rui Xia, and Pengfei Liu. Mathpile: A billion- token-scale pretraining corpus for math. grave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neu- ral Information Processing Systems, volume 37, pp. 75426.25468. Curran Associates, Inc., 2024. URL https://proceediingg neuriii cc/paper tttes//aper/2224/f1le and Benchmarks Track. pdd.
Globerson, L. Mackey, D. Bel-
2d0be3cd5173c aat
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Yuxiang Wei, Hojae Han, and Rajhans Samdani. Arctic-snowcoder: Demystifying high- quality data in code pretraining. arXiv preprint arXio:2409.02322 2024.
Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models. arx preprint ar Xffo:2244418884, 2024a.
Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, Zhoujun Cheng, Siheng Zhao, Lingpeng Kong, Bailin Wang, Caiming Xiong, and Tao Yu. Lemur: Harmonizing natural language and code for language agents. In The Twelfth International Conference on Learning Representationss ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL /ttps://operrevieww.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arv preprint ar 1r11::2412.15115, 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint ar xXix:2409.121222, 2024b.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024.
Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. arxiv preprint ar Xri::24005035445 2024.
Liang Zeng, Liangjun Zhong, Liang Zhao, Tianwen Wei, Liu Yang, Jujie He, Cheng Cheng, Rui Hu, Yang Liu, Shuicheng Yan, al. Skywork-math: Data scaling laws for mathemati- cal reasoning in large language models-the the story goes on. ar preprint ar Xit:2407.08344 2024.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint ar Stt::2001002885 2024.
14

---

Published as a conference paper at COLM 2025
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. Xiv preprint XIXi:22433.1337, 2024.
Fan Zhou, Zengzhi Wang, Qian Liu, Junlong Li, and Pengfei Liu. Programming every exam- ple: Lifting pre-training data quality like experts at scale. ar Xio preprint arXiv:2409.17115, 2024.
15

---

Published as a conference paper at COLM 2025
A Comparison with Existing Corpora
Table Comparison with existing large-scale math corpora
Corpus Name
Fully Public
# Tokens (B)
Date
Type
Open
14.5
2023 Oct.
Web
AlgebraicStack
11.0
2023 Oct.
Code
ArXiv, Web, Textbooks, StackExchange, Wiki
MathPile
9.5
2023 Dec.
MaepseckMath
X
120.0
2024 Teb.
Web
InfiMMWebMatth
55.0
2024 Sep.
Web
Web, Code snippets, Encyclopedias, Books, Exam questions, Synthetic data
Qwen Math Corpus v2
1000.0
2024 Sep.
MathCode-Pile
19.1
2024 Oct.
Web, Code, Textbooks
MegaMath Collection (Ours)
MegaMathh-WW
263.9
Web
MegaMathh-Web-Pig
15.1
Web
MegaMatth-Codk Coode
28.1
Code
Mega SanMaatt-ynt-oodeee
7.2
2025
Code
MltgaMath-Syntth Santh-Q&&A Q&A
Q&A
Mega Math Math-Synth- Synth-yyett-CCodd
50.3
Interleave text&code
B Details for Curating MegaMath-Web
The scoring prompt for evaluating web documents' relevance to mathematics is presented in Figure?. 7.
Please evaluate the given document for its relevance to mathematics and assign score from 0 to 5. Use the following scoring criteria:
5: The document is entirely about mathematics, containing numerous mathematical concepts, formulas, proofs, advanced mathematical educational content.
The document is primarily about mathematics but may include some applications in other disciplines or content related to mathematics education.
3: The document contains significam mathematical content, but it's not the main focus. It might be mathematical applications in physics, engineering, or similar fields.
2: The document includes some mathematical elements, such as basic calculations, simple statistics, or graphs, but these are not the main content of the document. content, possibly only
1: The document has very little mathematics-relatee mentioning numbers simple calculations in passing. The document has no mathematical content whatsoever.
The document is given as:
<EXAMPLE>.
After examining the document:
Briefly justify your total score, up to 100 words.
Conclude with the score using the format: "Score: <total points>"
Figure 7: Scoring Prompts for evaluating web documents relavance to mathematics.
B.1 Fine-grained Deduplication
We also explored several fine-grained deduplication methods, including exact substring (Lee et al., 2022) and sentence-level deduplication (Raffel et al., 2020). Initially, we found that removing duplicates disrupted text consistency. To mitigate this, attempted trimming only the head and tail portions, but still identified many math expressions and degraded downstream performance. We suspect the effectiveness of these methods depends text
16

---

Published as a conference paper at COLM 2025
extraction techniques and may be more suitable for Resiliparse. future exploration.
We thus leave this for
B.2 Strategy for MegaMath-Pro Subset
Building on Gunasekar et (2023), documents with higher educational values are treated as higher-quality samples-a strategy widely adopted in pre-training works. In MegaMath, create the Math set from MegaMath-Wai
Table 10: Yearly Ablation of Edu scoring strategy.
GSMSK
MATH
ASDiv
SVAMP
MAWPS
AVG
FM-iplus
10.5
6.1
41.9
25.3
57.9
28.3
2014
30.2
17.5
2015
5.0
3.0
21.8
14.4
27.7
14.4
sub-
data using
2017
6.4
34.9
21.8
44.6
22.5
2018
6.2
5.9
34.6
22.6
46.7
23.2
FineMath classifier (Allal et al., 2025) score documents on a 0-5 scale. How-
37.7
2020
8.7
4.6
35.5
24.7
49.3
24.6
2021
39.6
24.6
26.2
ever, we found that document distri-
2022
41.9
bution and relevance to mathematical reasoning vary over time. As Table 10
2023
12.1
5.8
43.2
28.3
63.0
30.9
46.6
indicates, after applying Edu filtering, training 5B tokens some years' data (e.g., 2014)
yields marginal improvements, whereas later years achieve much higher performance than adopted dynamic filtering
Math--u-sp (FM-4plus). Based on these observations, strategy: a more tolerant threshold (Edu score 2 3) for recent years (e.g., 2023-2024) and stricter one (Edu score 4) for earlier periods (e-g., 2014-2017) Similar to Nemontron- to further remove noise, and refine the web text into higher quality. Please see Figure for the detailed prompt.
CC (Su et al., 2024), we also used an LLM (in our case, we use
Task:
Carefully analyze the provided text to extract key facts, concrete details, important numbers, and core concepts.
Remove any irrelevant or noisy information, and reorganize the content into a and concise version that is easy to learn from. Output only the refined text.
logically structured,
Strive to maintain the original length much as possible (avoid excessive shortening).
Text:
<EXAMPLE>
Just output the refined text, no other text.
Figure 8: Rewriting Prompt for constructing MegaMath-We Web-Pro..
B.3 Further Ablation on fastText
We further validated our decision through experiments on all yearly dumps. Specifically, we conducted pre-training on the top 10% highest-scoring filtered data from each yearly dump using different versions of fest As shown in Figure 9, the results confirm the effective- ness of our final version (V2: Balance + CoT data), demonstrating clear improvements over the initial version in second-round filtering. Another interesting observation is that data quality, as indicated by downstream performance, gradually improves over time.
17

---

Published as a conference paper at COLM 2025
30
Core Performence
26
22
18
2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024
Year V2: Balance CoT data
-
V2: Random
Figure 9: Ablation on fastText for each year's 's all dumps within 5B-token training budget
C Details for Curating MegaMath-Code
We used -llmm-3.1-700-Instteee to annotate 25K randomly sampled code data and fine- tuned a Qwen-2.5-0.5D model to judge the code quality and decide whether to filter the code. The scoring prompts are presented in Figure 10 and Figure 11. We only keep code data with Math Score > 4 and Code Score 2 4, and all other code data are treated as negative samples during training.
In Table 11, we list details for our supervised fine-tuning configurations. We use Llamal'ac tory (Zheng et al., 2024) as our code base. Same as ProX (Zhou et al., 2024), we also select the model with highest Fl score as out final recalling models, which achieves 80% on split validation set.
Table 11: Training parameters for SLM.
HyperParams
Setting
LR
1e-5
LR Schedule
cosine
Batch Size
64
Number of Epochs
2
Context Length
2048
18

---

Published as a conference paper at COLM 2025
Below is an extract from resource focused on mathematical reasoning. Evaluate its educational value in effectively teaching concepts in this area, with emphasis on mathematical reasoning. Use additive 5-point scoring system described below. Points accumulate based on each criterion:
Add 1 point if the resource contains valid content in mathematics, reasoning, logic puzzles, scientific computation, even if it's not inherently educational (e.g., configurations or specialized algori
Add another point if the resource addresses practical concepts in these areas, such as solving math problems or reasoning tasks, even without annotations or explanations.
Award third point if the resource is suitable for educational use and introduces key concepts in mathematics or reasoning, with structured format some explanations or annotations.
Give fourth point if the resource is self-contained and directly useful for teaching, resembling structured exercise, tutorial, part of lesson in mathematical reasoning or logic.
Grant a fifth point if the resource is outstanding in educational value and perfectly suited for teaching, with clear, step-by-step explanations and thorough annotations mathematical reasoning concepts.
The extract: <EXAMPLE>
After examining the extract:
Briefly justify your total score, up words. Conclude with the score using the format: "Score: <total points>"
Figure 10: Scoring Prompt for evaluating code snippets' relevance to mathematics.
(CODE_TYPES program. Evaluate whether has high educational value and could help teach coding. Use the additive 5-point scoring system described below. Points are accumulated based on the satisfaction of each criterion:
Below an extract
1 point the program contains valid <CODE_TYPE> code, even if it's not educational, like boilerplate code, configs, and niche concepts. Add another point if the program addresses practical concepts, even if it lacks comments.
Award third point if the program suitable for educational use introduces key concepts in programming, even if the topic is advanced (e.g., deep learning). The code should be well-structured and contain some comments. Give fourth point if the program is self-contained and highly relevant to teaching progr amming. It should be similar CYPEE_TYPE> course section.
school exercise, tutorial,
Grant a fifth point if the program is outstanding in its educational value and is perfectly suited for teaching programming. It should be well-written, easy to understand, and contain step-by-step explanations and comments.
The extract: <EXAMPLE>
After examining the extract:
Briefly justify your total score, up words.
Conclude with the score using the format: "Score: <total points>"
Figure 11: Scoring Prompt for evaluating code snippets' general quality, i.e., educational value.
19

---

Published as a conference paper at COLM 2025
D Details for Curating MegaMath-Synth
Synthetic Text Data
We provide the prompts for extraction and refining Q&A below.
Below is web document extract. Assess whether it contains mathematical iuestion-and-answe pair:
If the web document extract does not contain a mathematical question-and-ansinn pair, return the explicit symbol [[NO QA]"
If mathematical question-and-ansee pair is found, extract it in the following format:
Question: <question text with complete problem statement and all necessary mathematical information>
Answer: <complete solution with all necessary steps and calculations included> (only if answer provided, otherwise extracted pair must self-contained mathematically precise, allowing independent solving without additional context.
generate this line)
###### The extract: <EXAMPLE>
Now process the extract and return the result.
Figure 12: Prompt for QA extraction.
Below is a mathematical iasstion-and-answe pair. Refine the answer based on the following rements::
*xIf the process**:
does not contain any explanation intermediate reasoning
Add only necessary intermediate reasoning process leading to the given answer Ensure the added steps are logical, clear, and provide necessary explanation of the solution process
**If the answer already includes necessary solution process**
Reorganize the solution into a clear and well-structured format for better readability and understanding for simple solutions, there is need
latex format
Maintain the original question text and provide the refined answer in the same format:
Question: <question text>
Answer: <refined solution>
#*## The auestion-and-answe pair: <EXAMPLE>
Suppase you are math teacher, you should explain the solution in way that is easy for a student to understand. Now process the pair and return the refined result.
Figure 13: Prompt for Refined QA.
20

---

Published as a conference paper at COLM 2025
Synthetic Code Data
We provide the prompt for code translation in Figure 14 at below.
Below an extract from a code snippet. Translate the code from other programming languages into Python. Read the code carefully and translate it into Python.
If the original code has quality cannot be converted to Python, return the explicit symbol "[Untranslatable]"
translated Python code should meet the following requit:tntts:
Ensure good code formatting.
Include proper comments or explanations for clarity explaining the logic where needed.
docstrings when necessary to improve readability.
Wrap the generated Python code within python "'yython
Keep good test cases if any.
The extract:
<EXAMPLE>
Do not produce any additional commentary or text beyond Now output the translated Python code:
"python
Figure 14: Prompt for translating non-Python code samples into Python code samples.
Synthetic Code Block Data We used the same prompts as in Lu et al. (2024). Please see Figure 15. Our AST filtering mainly contains the following aspects:
1. Code Parsing and AST Generation: The input code is parsed into an AST using Python's built-in ast module. The system first verifies code length constraints (max 100,000 characters) and handles syntax errors through exception catching.
2. Import Declaration Analysis: A specialized visitor collects all imported modules and their aliases through two-phase inspection:
Direct imports (import x as y) mapping
Selective imports from modules (from import b as
3. Semantic Node Traversal: A secondary visitor examines all function calls and context managers, checking against three prohibition categories:
File Operations: file I/O methods (e.g., open, savefig), path manipulations, and serialization functions
Thread /process creation calls and 5+ restricted modules (e.g., threading, asyncio)
Concurrency Patterns:
network libraries and protocol-specific methods (e-g., requests. get, socket.
Network Communication:
4. Module Dependency Verification:
Cross-references imported modules against prohibited libraries spanning file systems (shutil), parallelism (multiprocessing). and network protocols (ftplib).
5. Context-Specific Checks: Special handling for:
wit th statements containing file open operations
Class instantiations of thread process primitives
Path manipulation methods in object-oriented interfaces
21

---

Published as a conference paper at COLM 2025
You will be presented with text related to math. I need you to identify all the complex computations
For each complex computation that requires a scratchpad, find out the conditions needed for the computation, the latex expression that conducts the computation, and the result of the computation. Then generate Pythan code snippet for each computation that demonstrates how the result is reached. Output each computation in the following format:
Conditions Needed: 1. [Condition 1]
2. [Condition 2]
Computation Expression:
$[Latex Expression]s
Computation Result: [Computation Result]
Python Code Snippet: -" "ppython
[Python Code]
There be than one complex computation in the text. Output only the computations that requires calculation. Do not include mathematical statements or definitions as a computation. Make sure each snippet can be executed individually. The text is as follows:
<EXAMPLE>
The computations are:
Figure 15: Prompt for generating code-block data (Lu et al., 2024).
Table 12: LLM Usage Summary During Data Curation
Purpose / Stage
LLM Used
Description / Notes
Math-related web document classi fication
Illama3.1 3.1-00-..--BB--.- Instruutttttt
Used to annotate seed documents for training FastText
classifiers
Web decument refinement
Llama-3.3-70B-nnsttee
Used to further remove noise and retine web text into
higher quality content, similar Nemotron CC.
Math--reated code classification
Llama-3.1-70B-Ines
Used to annolate randomly sampled code data for fine-
tuning small language model (ie. Qwen2.5-0.50 0.58). pains from
Synthetic generation
Qwen2.5 72B Instruct
Llama-
Used to extract and refine question documents.
Instructt
Code Translation
Qwen2.5-Code 3-Codee:32B-nnstrree
/ Used to translate code from other programming lan-
Llama-3.1 3.1--.0---1-sese-- Instruct
guages into Python.
synthesis
Lamma3.1-708-Instte
to synthesize
code blocks.
22

---

Published as a conference paper at COLM 2025
E Summary of LLM Usage During Data Curation
Training Details
E1
Liama Training
In all ablation experiments, we keep our training hyper-parameter the same except for training steps. We present our full training details in Table 13.
Table 13: Training hyper-parameters.
Hyper-earrmeter
5B 15B 55B Tokens
Context Length
2,048
Batch Size
1,024
Max Steps
2,500
7,500 27,500 0
Warmup Steps
Weight Decay
0.1
Optimizer
AdamW cosine 8e-5 + 8e-6
LR Scheduler
Learning Rate (LR)
Table 14: Training Data Mixture for Llama-3.
Data
Ratio %
DCLM
10
Web
15
Web-pro
35
Code
2.5
QA
10
Trans. code
2.5
Text & code block
25
Total
100
Table 15: Training hyper-parameters
Hyper-parameter
Llama-3.2-1B/33 /BB
Context Length
8,192
Batch Size
512
Max Steps
25,000 25,000 (stop 12,500) 0
Warmup Steps
Weight Decay
0.1
Optimizer
Adam cosine
LR Scheduler
5e-5 + 5e-6
Learning Rate (LR)
3e-5 + 3e-6
F2 Llama-3 Training
The data mixture and for Llama-3 training are presented in Table 14 and Table 15.
23

---

Published as a conference paper at COLM 2025
G Evaluation Details and Full Results
G.1 Full Benchmarks
Our core set of tasks and eval settings are at below. We revised our evaluation from DeepSeekMath (Shao et al., 2024): we fixed one of its prompts 3, and support PAL for more benchmarks.
1. GSM8K (Cobbe et al., 2021), 8-shot
2. MATH (Hendrycks et al., 2021), 4-shot
3. ASDiv (Miao et al., 2020), 8-shot
4. SVAMP (Patel et al., 2021), 8-shot
5. MAWPS (Koncel-Kedziorse))
al., 2016), 8-shot
Our extended set of tasks are:
1. MMLU-STEM (Hendrycks et al., 2020), 4-shot
2. TabMWP (Lu et al., 2023), 8-shot
3. MathQA (Amini et al., 2019), 8-shot
4. SAT (Azerbayev et al., 2023), 4-shot
5. OCW Courses (Lewkowycz et al., 2022), 4-shot
G.2 Full Ablation Results
We present our full results in this section:
1. For ablation on Web Data We provide the full ablation results on math text extraction, Minhash deduplication and fastText in Table 16, Table 17, Table 18.
2. For ablations on Code filtering, please see Table 19, and Table 20.
3. For ablations on synthetic data, please see Table 21, and Table 22.
4. The full comparison results are provided in Table 23.
5. For evaluation results for Llama-3, please see Table 24.
Table 16: Full ablation results on math text extraction within 15B-token training budget
Text Detracion
HTML Optimization
ASDIV
GSMHK
MATH
MATHLSAT
MATHOA
MANYS
MMLU-SIM
OCW
SWAMP
TAEMIN?
Cone Avg.
Dst.
Taytlona
150
trafilatura
27.6
59
43
214
17.9
44.
33.2
22
22.3
21.8
220
19.2
Restiiparse
473
313
22.1
72.7
32.5
186
Table 17: Full ablation results on Minhash LSH within 55B-token training budget
12b0
remaining tokens (65
ASDIV
CSMSK
MATH
MATH-SAT
MATHQA
MAWPS MMLL-STEM
OCW
SWAMP
TABMWP
Coes.
Est. Aug.
29.1
011.01
8.75
26.0
29.8
6.4
23.1
10.2
61.4
19.3
2.5
17.6
22.1
19.4
17.5
(1.11
0.75
25:0
30.1
4.3
10.9
28.9
21.0
21
15.7
21.3
19.2
16:0
(8:23
11.2
40.5
17.5
https:/ http:/gtthhhh..aacoogoo http://////ggttaeoooo/////////// whlob//mmttt revaluation/ shot prompts/ pal shot
24

---

Published as a conference paper at COLM 2025
Table 18: Full ablation results on fastText within 5B-token training budget
TastTeet version
ASDLV
GSMBK
MATH
SAT
MATHQA
MAWPS
MMLU-STEM
OCW
SWAMP
TABMWP
Core. Avg.
Est. Avg.
Oprr-WWbb-Vtth
22.4
278
25.8
28.9
Table 19: Performance comparison of CoT and PAL under different filtering criteria
CoT
Filter Criteria
GSMSK
MATH
ASDiV
MAWPS
SVAMP
Avg.
text only
4.1
39.5
17.7
19.0
S.edu 2 3,S.mahh 3
4.1
4.4
28.9
40.2
19.4
19.4
S.edu 2 3amh 2 4
4.9
4.2
29.8
41.1
19.2
19.8
S_edu 2 m,th 2
4.3
29.8
39.9
19.4
19.7
S.edu 4,8 math
17.3
PAL
Filter Criteria
GSM8K
MATH
ASDIV
MAWPS
SVAMP
Avg.
only
2.9
24.8
30.1
17.6
15.6
S_edu 3,S_math 3
3.7
3.6
25.8
31.7
15.6
16.1
S edu 2 3,S math 2 4
4.4
4.3
27.2
31.4
16.5
16.8
S.edu 4,8 math 3
3.7
27.4
32.3
19.5
17.5
edu math
5.5
29.7
36.4
20.2
19.5
Table 20: Performance comparison of CoT and PAL under different mix ratios.
CoT
Mix Ratio
GSMBK
MATH
ASDiV
MAWPS
SVAMP
Avg.
text only
4.4
4.1
29.3
39.5
17.7
19.0
code:text 1:7
3.8
4.2
30.2
40.0
18.3
19.3
code:text 1:4
4.6
4.1
29.5
40.6
18.9
19.5
code:text 1:2
3.9
4.0
28.3
27.6
18.1
16.4
code:text 1:1
3.6
3.9
26.7
36.7
16.6
17.5
PAL
Mix Ratio
GSM8K
MATH
ASDiV
MAWPS
SVAMP
Avg.
text only
2.8
24.8
30.1
code:text
4.4
27.2
code:text
4.4
4.4
29.2
code:text 1:2
4.3
4.4
27.6
34.1
17.1
code:text 1:1
5.4
4.9
29.5
36.3
17.7
18.8
Table 21: Performance comparison of CoT using different Q&A datasets
Data
ASDIV
GSMSK
MATH
MATH-SAT
MATHQA
MAWPS
FM-iplus
41.9
10.5
6.1
344
14.4
37.9
WebInstruct
49.5
13.1
10.6
25.0
14.7
65.8
Vanilla Prompt
57.4
22.1
10.5
25.0
16.5
65.6
w. ELIS
58.6
25.9
12.3
21.9
18.4
71.6
Data
MMLU-STEM
OCW
SVAMP
TABMWP
Core Avg.
Ext. Avg.
FM-4plus
25.5
WebInstruct
16.0
3.3
34.0
29.2
34.6
17.6
Vanilla Prompt
w. ELIS
15.6
4.0
36.1
35.9
41.3
19.2
EL15
18.3
3.3
46.0
40.1
48.8
23.6
25

---

Published as a conference paper at COLM 2025
Table 22: Performance comparison of CoT and PAL under different mix ratios.
CoT
Data
GSMSK
MATH
ASDiV
MAWPS
SVAMP
Avg.
code
4.3
4.2
29.5
38.5
17.3
18.8
trans. code
3.5
4.3
30.2
39.3
17.8
19.0
text code block
6.7
45.4
21.2
22.5
code block (full)
12.4
31.2
30.8
PAL
Data
GSM8K
MATH
ASDIV
MAWPS
SVAMP
Avg.
code
5.7
5.5
29.7
36.4
20.2
19.5
trans. code
7.0
5.3
31.3
39.4
20.1
20.6
text code block
9.6
10.6
41.1
51.2
27.8
28.1
block (full)
26.9
48.3
46.5
Table 23; Full comparison CoT results with existing corpora within 55B-token training budget
ASDIV
GSMM
MATH
MATILAAT
MAWPS MATHOA
MMLE:STEM
OEW
SWAMM
TARMINI
Curn,
Esl.
MegaMath-Webb 1138, Dun)
MegaMath- Web- 90% (Oun)
53.0
13.5
8.4
08.3
15.7
3.7
33.1
12.2
35.5
28.7
Fiektuth3
171
8.5
664
71.1
4.4
30.7
30.5
35.1
Iwflis--peebloo (130)
66.1
63.
12.3
38.5
MrgaMaan-Nob-- Mobl-aa------llllll (Dun))
46.7
11.6
6.1
750
13.0
6:0
714
32
36.3
30.0
38.0
242
Wpee-WW (14.66)
63
542
727
74.1
76.6
79.8
Table 24: Full results training MegaMath on Llama-3 series of models.
CoT
Model
ASDIV
GSMSK
MATH
MAWPS
SVAMP
Avg.
1lama-322-1B
33,8
4.6
43.3
21.5
22.3
MegaMath Llama-3.2-18
59.8
25.7
9.5
74.6
41.3
42.2
Llama-3,2-BB
MegaMath Llama-3.2-3B
78.8
36.2
25.1
90.2
71.6
64.4
Model
PAL
ASDIV
GSM8K
MATH
MAWPS
SVAMP
Avg.
1lama-3.2-1B
13.4
3.1
16.8
MegaMath Llama-3.2-1B
42.8
16.8
6.3
52.7
29.8
29.7
Llama-3.2-3B
65.1
35.7
0.4
83.3
58.3
48.6
MegaMath Ilama-3.2-3B
78.1
35.7
24.6
93.7
74.4
65.3
26


Reviews:
[{"review_id": "bMv63uj1Tl", "reviewer": "Reviewer_J7PA", "summary": "This paper introduces MegaMath, a large-scale, high-quality corpus targeting LLMs' mathematical reasoning capabilities. The corpus has about 371.6B tokens derived from web data (279B tokens), code (28.1B tokens), and synthetic data (64.5B tokens). The paper details the curation process across three main components: (1) Re-extracting mathematical documents from Common Crawl with math-oriented HTML optimizations, (2) Filtering high-quality math-related code from existing code corpora, and (3) Synthesizing QA-style text, math-related code, and code-interpreter style data. The authors validate their approach through extensive ablation studies and demonstrate significant improvements when training Llama-3.2 models on their dataset, achieving substantial gains in mathematical reasoning benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed resource, i.e., MegaMath, would have a significant benefit to the open-source community. \n- Most of the dataset design choices are backed by ablation studies. \n- The empirical results clearly demonstrate the edge of MegaMath over existing alternatives.\n- The work seems highly reproducible, with detailed methodologies that would allow others to build upon this research.", "reasons_to_reject": "- The paper does a thorough job of execution but lacks novelty.\n- The impact of pretraining for math reasoning is not clear after post-training. Would the gains remain post-alignment as well? \n- Some of the key details are not well explained. What is the CoT data that the authors are talking about? What is its source? Similarly \"information completeness\" is introduced without any context in Section 3.4.", "questions_to_authors": "- Qwen models add the SFT dataset to pretraining/mid-training. Have you considered adding NuminaMath/OpenMathInstruct to pretraining to supplement the synthesized data in Section 3.4? \n- What do you mean by \"information completeness\" in Section 3.4? \n- How does improving math reasoning capabilities in pretraining impact other tasks?\n- How do the different subsets of the dataset - web, code, and synthesized data?", "extra_scores": {}}, {"review_id": "zipWjJ6KvG", "reviewer": "Reviewer_CT6F", "summary": "This paper proposes a systematic methodology for developing a large-scale mathematical pretraining corpus. MegaMath comprises three main types of pretraining data:\n\n- First, the Web math pretraining data optimizes HTML extraction techniques to preserve complete mathematical expressions and conducts filtering and deduplication, providing valuable experimental insights. The authors further refine this data by using classifiers and large language models to extract higher-quality subsets.\n- Second, the Code data employs small language models to recall mathematics-related code and offers empirical guidance on optimal mixing ratios for pretraining.\n- Third, the Synthetic data includes question-answer pairs extracted from raw documents and refined using LLMs, translations of code from other languages into Python, and \"generated mathematical code\" consisting of interleaved text, symbolic expressions, and code blocks.\n\nThe resulting pretraining dataset reaches industrial scale at 371B tokens. The authors conduct comprehensive ablation experiments to validate key design choices and demonstrate considerable performance improvements when training the LLaMA-3 series models with their corpus.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work provides an industrial-scale mathematical pretraining corpus spanning web data, code mixtures, and synthetic data. To my knowledge, this approach closely resembles industrial practices and offers significant value to the research community by providing extensive empirical references and practical guidelines.\n\n- The ablation studies validating each key design choice are comprehensive and methodical. The paper systematically compares different text processing techniques (trafilatura vs. Resiliparse), math-optimized HTML processing methods, and deduplication strategies with various parameters, providing valuable insights for domain-specific dataset construction.\n\n- The performance improvements on the LLaMA-3 series models are substantial and demonstrate the effectiveness of the proposed methods and the quality of the constructed corpus.", "reasons_to_reject": "- The evaluation results show puzzling discrepancies with LLama-3.2's official results. For example, Llama-3.2-1B using CoT 4-shot on MATH achieves only 4.6 in this paper, whereas the official report states that Llama-3.2-1B on MATH (CoT) 0-shot reaches 30.6. Similarly, Llama-3.2-1B using CoT on GSM8K scores only 8.5 in this paper, while officially Llama-3.2-1B on GSM8K (CoT) 8-shot achieves 44.4. These discrepancies require explanation.\n\n- All experiments are conducted solely on LLaMA-3.2, leaving uncertainty about whether the method would yield similar improvements on other advanced open-source pretrained models such as Qwen2.5-Base. This lack of cross-model validation weakens the generalizability claims of the approach.\n\n- For the final training phase, the authors select 100B tokens from the total 371B corpus, but provide insufficient details on the selection criteria. What is the mixture configuration of this data? Did the authors conduct further experiments on different mixture configurations? This information is crucial for understanding the method's practicality and reproducibility.\n\n- Regarding the synthetic data, particularly the QA pairs, the paper lacks thorough analysis comparing their quality to human-written data. Data generated using LLMs may introduce model biases or inaccuracies, and this risk has not been systematically evaluated.\n\n- While the paper introduces synthetic QA pairs, it does not thoroughly explore their implications, such as whether they might exhaust the model's potential. How would the model perform if subsequently fine-tuned on QA pairs in SFT stage? This analysis would be valuable for understanding the interaction between synthetic data in pretraining and fine-tuning stages.\n\n- The paper primarily focuses on model performance on benchmarks like GSM8K and MATH, but lacks evaluation on more complex mathematical problems such as advanced competition questions (e.g., AIME, Omni-Math, CollegeMath, OlympiadBench). This limitation makes it difficult to assess MegaMath's effectiveness in developing deep mathematical reasoning capabilities, particularly for more challenging mathematical problems.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ip8WfYIHwk", "reviewer": "Reviewer_PQwQ", "summary": "This paper presents MegaMath, a carefully curated math corpus for pre-training LLMs.\nThe construction pipeline is generally valid: (i) HTML extraction with fastText filtering & deduplication (ii) Math-related code data recall from Stack-v2 (iii) Data synthesis to create QA style data, translated Python code, and text-code blocks.\nThe final MegaMath comprises 371B tokens, the largest open math pretraining corpus.  \nThe curation pipeline and hyperparameter settings are validated using TinyLLaMA-1B and continual pre-training with LLaMA 3.2 justifies the value of the MegaMath.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The curated MegaMath is of great value for LLM pre-training and could benefit future studies towards better reasoning models.\n- The overall pipeline is sound and well-executed comprehensive validation experiments under academic resources.", "reasons_to_reject": "I did not find a strong reason to reject this paper.", "questions_to_authors": "## Questions\n- I am curious about the utility of MegaMath-Synthetic, but did not find any experiments to investigate this. Sec 3.4 is operated on data synthesized from Open-Web-Math and Infimm-Web-Math which may not transfer to the MegaMath? \n\n- There are various models adopted for scoring, annotation, and synthesis. It would be better to aggregate the LLMs choices into ONE table for better clarity. For example, Line 288 states `with LLM-annotated math-related document` and Line 316 ` using the latest and most capable LLMs.`  without specifying the used LLM.\n\n\n\n\n## Typos\n\nLine337:  Also, These -> Also, these\n\nCaption of Figure 4: ; Right: text & ode block data curation -> ; Right: text & code block data curation", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary:

This paper introduces MegaMath, a large-scale (371B tokens) mathematical pre-training corpus curated from web data, math-related code, and synthetic QA/code blocks. It provides a detailed construction pipeline and validates the dataset through ablation studies and empirical gains on multiple math reasoning benchmarks using LLaMA-3.2 models. The dataset and pipeline are intended as a public resource to enhance LLMs’ math reasoning capabilities.

Strengths: 

All reviewers agree that the paper presents a valuable open resource. Most reviewers note the paper presents a well-documented engineering pipeline, and conducts comprehensive ablations and shows consistent performance gains on math benchmarks.

Weaknesses:

Originally, reviewers had concerns regarding limited novelty, limited evaluation using models beyond LLaMA-3.2 (unclear generalization to other strong models such as QWen), and limited evaluation on hard math benchmarks. 

After the response period, the authors have addressed most concerns. The authors should make sure to include the discussions and new results in the revised paper. I do think that the impact of this paper could be enhanced if the proposed dataset can be used to improve the performance of more models beyond LLaMA-3.2. Nonetheless, I think the paper makes a valuable contribution and would recommend accepting it to the conference, provided that the paper could fully open-source all the resources as promised in the paper and during the author response period. In addition, I believe the authors should talk about the license issue when releasing this dataset. In particular, Stack-v2 has their user agreement terms that the paper may need to follow.
Decision: Accept