MegaMath: Pushing the Limits of Open Math Corpora
Authors: name: Fan Zhou, name: Zengzhi Wang, name: Nikhil Ranjan, name: Zhoujun Cheng, name: Liping Tang, name: Guowei He, name: Zhengzhong Liu, name: Eric P. Xing

Mathematical reasoning is a cornerstone of human intelligence and a key
benchmark for advanced capabilities in large language models (LLMs). However,
the research community still lacks an open, large-scale, high-quality corpus
tailored to the demands of math-centric LLM pre-training. We present MegaMath,
an open dataset curated from diverse, math-focused sources through following
practices: (1) Revisiting web data: We re-extracted mathematical documents from
Common Crawl with math-oriented HTML optimizations, fasttext-based filtering
and deduplication, all for acquiring higher-quality data on the Internet. (2)
Recalling Math-related code data: We identified high quality math-related code
from large code training corpus, Stack-V2, further enhancing data diversity.
(3) Exploring Synthetic data: We synthesized QA-style text, math-related code,
and interleaved text-code blocks from web data or code data. By integrating
these strategies and validating their effectiveness through extensive
ablations, MegaMath delivers 371B tokens with the largest quantity and top
quality among existing open math pre-training datasets.