Title: Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset
Primary area: datasets and benchmarks
Keywords: Mathematical Reasoning, Web-Scale Data Curation, LLM-Based Cleaning, Pretraining datasets, Deduplication

Abstract: Pretraining large language models (LLMs) on high-quality, structured data such as mathematics and code substantially enhances reasoning capabilities. However, existing math-focused datasets built from Common Crawl suffer from degraded quality due to brittle extraction heuristics, lossy HTML-to-text conversion, and the failure to reliably preserve mathematical structure. In this work, we intro-
duce Nemotron-CC-Math, a large-scale, high-quality mathematical corpus constructed from Common Crawl using a novel, domain-agnostic pipeline specifically designed for robust scientific text extraction. Unlike previous efforts, our pipeline recovers math across various formats (e.g., MathJax, KaTeX, MathML) by leveraging layout-aware rendering with lynx and a targeted LLM-based cleaning stage. This approach preserves the structural integrity of equations and code blocks while removing boilerplate, standardizing
notation into L A T EX representation, and correcting inconsistencies. We collected a large, high-quality math corpus, namely Nemotron-CC-Math-3+(133B tokens) and Nemotron-CC-Math-4+ (52B tokens). Notably, Nemotron-CC-Math-4+ not only surpasses all prior open math datasets-including Mega-Math, FineMath, and OpenWebMath-but also contains 5.5× more tokens than FineMath-4+, which was previously the highest-quality math pretraining dataset. When used to pretrain a Nemotron-T 8B model, our corpus yields +4.8 to +12.6. 
gains on MATH and +4.6 to +14.3 gains on MBPP+ over strong baselines, while
also improving general-domain performance on MMLU and MMLU-Stem.
We present the first pipeline to reliably extract scientific content—including
math—from noisy web-scale data, yielding measurable gains in math, code, and
general reasoning, and setting a new state of the art among open math pretraining
corpora. To support open-source efforts, we release our code1 and datasets 2
.

TL;DR: We build Nemotron-CC-Math, a high-quality math corpus from Common Crawl that outperforms prior math corpora, boosting LLM performance on math, code, and reasoning tasks.