DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning
Authors: Zhiwei He
Tian Liang
Jiahao Xu
Qiuzhi Liu
Xingyu Chen
Yue Wang
Linfeng Song
Dian Yu
Zhenwen Liang
Wenxuan Wang
Zhuosheng Zhang
Rui Wang
Zhaopeng Tu
Haitao Mi
Dong Yu
Venue: ICLR 2026 Poster (2026)
Keywords: llm
math
reasoning
Primary area: datasets and benchmarks

Abstract: Reinforcement learning (RL) with large language models shows promise in complex reasoning. However, its progress is hindered by the lack of large-scale training data that is sufficiently challenging, contamination-free and verifiable. To solve this problem, we introduce DeepMath-103K, a large-scale mathematical dataset designed with high difficulty (primarily levels 5-9), rigorous decontamination against numerous benchmarks, and verifiable answers for rule-based RL reward. It further includes three distinct R1 solutions adaptable for diverse training paradigms such as supervised fine-tuning. Spanning a wide range of mathematical topics, DeepMath-103K fosters the development of generalizable and advancing reasoning. Notably, models trained on DeepMath-103K achieve leading results on challenging mathematical benchmarks and demonstrate generalization beyond math such as biology, physics and chemistry, underscoring its broad efficacy.

Published as a conference paper at ICLR 2026
DEEPMATH- 103K: A LARGE-SCALE, CHALLENGING, DECONTAMINATED, AND VERIFIABLE MATHEMATI- CAL DATASET FOR ADVANCING REASONING
Zhiwei H, Tian Liang*.1 Jiahao Xu
Qiuzhi Liu'
Xingyu Chen Yue Wang*
Linfeng Song! Dian Yuu Zhenwen Liang'
Wenxuan Wang! Zhuosheng Zhang
Rui Wang Zhaopeng Tu Haitao Mi Dong Yu!
Tencent Shanghai Jiao Tong University
//ttta://gthhb... wtwettuu.coooeeeeeec ctm/ zwhe 3Deepeatth
attps://bE c0/ hes/aatasetta zwhe
ABSTRACT
Reinforcement learning (RL) with large language models shows promise in com- plex reasoning. However, its progress hindered by the lack of large-scale train- ing data that is sufficiently challenging, contamination-free and verifiable. To solve this problem, we introduce Deep-Math-103K, large-scale mathematical dataset designed with high difficulty (primarily levels 5-9), rigorous decontam- ination against numerous benchmarks, and verifiable answers for rule-based RL reward. It further includes three distinct RI solutions adaptable for diverse training paradigms such as supervised fine-tuning. Spanning a wide range mathematical topics, Deep.att-- 103K fosters the development of generalizable and advancing reasoning. Notably, models trained on DeepMath-1000 103K achieve leading results challenging mathematical benchmarks and demonstrate generalization beyond math such as biology, physics and chemistry, underscoring its broad efficacy.
Luvel
120
103K
80K
Lovel
Livel
20K
Rt
DAPO-1TK
DSR.ihreviw
STILL-S
Open
DRZ-129KK
Doep 103
(a) Difficulty levels of different datasets.
60
Stlidh 55 St-hweeee Ri Diciil Dwer 1.50
Quenn 25.78
2.5
Doot as stchhee
Doopltunry
50.6
Accumay
OR2770
Deephtath Zemo
29.8
30.
23.5
12.5
12
14.4
8.5
10.2
11.7
(b) Pass@1 accuracies on AIME25.
Figure 1: (a) Deep 103K challenging compared existing datasets, (b) Results of DeepMath series models under zero RL and RL. setting using DeepMath-1033 103K.
$ Equal Contribution. The work was done when Zhiwei, Xingyu, and Yue were interning at Tencent. Correspondence Zhaopeng Tu <7ptu@tencent.ccmm and Rui Wang <wangrui 12@ eew...uuuuu........

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Reinforcement learning (RL) with large language models (LLMs) has demonstrated significant po- tential in complex mathematical reasoning (Guo et al., 2025; Hu et al., 2025; Zeng al., 2025a; Liu et al., 2025). Despite this promise, the effective advancement of RL is constrained by existing training data. While numerous datasets are available, they fall short in several key aspects crucial for training advanced reasoning models: (1) insufficient difficulty (Figure la) to push the boundaries of current models (Dang & Ngo. 2025; Yu et al., 2025; Luo al., 2025; Face, 2025; Hu et al., 2025), (2) contamination with standard benchmarks (appendix B), (3) a lack of verifiable answers essen- tial for RL with verifiable rewards (RIVR) (Guo et al., 2025; Cobbe et al., 2021; Hendrycks al., 2021b; Yu et al., 2024), or (4) inadequate combination of these critical aspects scale. Further- more, many of existing datasets represent the recombination and filtration of common sources (such as AIME (MAA, a)) which contain already well well-formatted data, thus lacking substantial influx of novel and diverse problems from more varied but less structured sources (Dang & Ngo, 2025; Yu al., 2025; Luo al., 2025; Face, 2025; Hu et al., 2025).
To bridge this gap, we introduce 10e,Math- a large-scale mathematical dataset tailored for advancing reasoning via RLVR. DeepMath- distinguishes itself through several key features.
DeepMathh 103K features a high concentration of challenging mathe- matical problems, with difficulty distribution skewed towards higher levels (> 5) compared to existing open resources (Figure la).
Challenging Problems:
To ensure trustworthy evaluation, Deep 103K 103K underwent a rigorous decontamination process against a comprehensive suite of benchmarks.
Rigorous Decontamination:
Verifiable Answers and Diverse Solutions: To enable rule-based reward functions in RLVR, every problem in DeepMath-1000 103K includes verifiable final answer that has been validated for easy extraction and verification via rules. Each problem is further enriched with three distinct R1 solutions (Guo et al., 2025), supporting diverse training paradigms such SFT.
Beyond these core features, DeepMath- 103K also differentiates itself in its raw data acquisition methodology. The prevalent trend in existing open datasets often recombines readily available and well-formatted problems from common sources such as AIME (MAA, a). This approach does not create new problems, but re-collect existing ones, which leads significant overlaps among differ- ent datasets. Recognizing the potential limitations and eventual exhaustion of common resources, DeepMath- draws its content from more diverse but less structured sources, notably including discussions from Math StackExchange'. The raw content from these sources is informal discourse and lacking a standard format. After a rigorous curation pipeline that transformed these discussions into well-structured QA format, DeepMath- is characterized by its unique problem variety and diversity compared to existing datasets.
Consequently, models trained on 1033 achieve leading results (Figure 1b):
Zero RL Training: Starting from the Qwen-2.5-(Mathh--7 (Team, 2024), DeepMath-Zero- (Math)-7B shows pass@1 improvements of +12.7 (+23.0) on AIME24 and +12.1 (+19.1) on AIME25, surpassing other basclines.
RL Training: Initialized from instruction-tuned models, DeepMath variants also show substan- tial gains. DeepMath-1155 1.5B, starting from RI RI-Distill-Qwen-I pass@1 accuracy improvements of +7.9 on AIME24 and
(Guo al., 2025), achieves +6.0 AIME25. DeepMath-Omm-1155 built upon OpenMath-Nemotooe 1.5B (Moshkov et al., 2025), reaches pass@1 I accuracies of 64.0 on AIME24 and 57.3 on ALME25, surpassing ol-mini (63.6 on AIME24) and low effort 03-mini (60.0 on AIME24).
Generalizable Reasoning beyond Math: DeepMath series models also generalizes their rea- soning abilities to broader domains, achieving the best GPQA-Diamond (biology. physics and chemistry) (Rein et al., 2024), MMLU-STEM (science and engineering reasoning) (Hendrycks et al., 2021a) and BBH (complex multi-step reasoning) (Suzgun et al., 2022) scores compared to the baselines.
Itttps: htw.:/mwww..ttcccccceegeeecoo
2

---

Published as a conference paper at ICLR 2026
These results underscore the value of DeepMath-105: 1033 as resource for developing advanced rea- soning models with broad upplicability. To foster future research, we have released the DeepMath- 103K dataset, along with the code and model weights, hoping to enable further exploration of ad- vanced reasoning techniques and the development of robust and generalizable machine intelligence.
2 OVERVIEW OF D00PMATH-103K
Question: Calculate the line integral & Pdx+ + Qdy, y' where vector fields 11-22 Determine
Solution 1: Okay, need calculate the line integral Hmm, the problem mentions that Thus, the value of the line integral is: 2x
the ellipse 23 36
given by: P the value of the integral, considering that the vector field undefined the point (0.1) inside the ese.
L 1F
R1 Solution 2: Okay, so need to calculate the line integral .....al...... Hmm, first things first, let me recall what line integrals are abouu Thus, the value of line integral
Final Answer: 2x
Difficulty: 8
Okay, need calculate the line integral So, first, maybe should visualize the ellipse Thus, the value the line integral is: 2z
Solution3:
Topie: Mathernttcs -> Calculus -> Integral Calculus Techniques of Integration -> Multi-aariable
Figure 2: data sample from DeepMath-10333 103K.
Each data sample in DeepMath- 103K is intentionally structured to be comprehensive, supporting variety downstream applications mathematical reasoning research. As illustrated Figure 2, single sample includes the following components:
Question: The mathematical problem statement.
Final Answer: A verifiable final answer, crucial for sale-based reward functions in RLVR.
Difficully: A numerical difficulty score, which facilitates techniques like difficulty-awan training (e.g., curriculum learning) adaptive compute allocation based problem complexity (Wang al., 2025b; Chen al., 2024).
Topic: hierarchical topic classification for the problem, enabling topic-specific analysis. Rl Solutions: Three distinct reasoning paths generated by the DeepSeek-RR model (Guo al., 2025), suitable for diverse training paradigms such as SFT.
DeepMath- 103K possesses several key characteristics that make particularly suitable for advanc- ing mathematical reasoning research:
Higher Difficulty DeepMath-1033 includes mathematical problems spanning difficulty levels 3 through 9. The core of the dataset consists of 95K challenging problems (levels 5-9) specifically curated for this research. To ensure broader difficulty coverage, this dataset is augmented with an additional 8K problems (levels 3-5) sourced from SimpleRL (Zeng et al., 2025b). For compari- son, analyzed and labeled the difficulty levels of several existing datascts commonly used for RLVR training in math domain: Open-RS (Dang & Ngo, 2025), DAPO-17K (Yu et al., 2025), DSR- Preview (Luo al., 2025), SITLL-3-RL (Chen et al., 2025), ORZ-1299 (Hu al., 2025), and Open-RI (Face, 2025). Figure la illustrates the difficulty distributions across these datasets. As depicted, DetpMath-103K exhibits significantly more challenging problem distribution, contain- ing a substantially higher proportion of problems at difficulty level 5 and above compared to the other benchmark datasets. This focus on higher difficulty intended push the reasoning limits of current models.
Rigorous Data Decontamination DeepMath- 103K was constructed exclusively using the training splits of existing open resources, with careful avoidance of any known test materials. However, our preliminary analysis revealed that these source data exhibits alarmingly high levels of contam- ination with problems from commonly used evaluation benchmarks. As illustrated in Figure 3, the
3

---

Published as conference paper at ICLR 2026
100%
81.1%
88.4%
78.0%
70.1%
67.7%
Contamination
60%
40.8%
40%
35.7%
33.9%
58.6%,
5.0%
0%
Olymp. Olymp, Bench
MATIE
83-24
24
25
Odyssey
(MC)
500
(MQA)
Bench
Math
STEM
Arena
Figure 3: Contamination rates of common mathematical and STEM benchmarks detected in the raw data sources before decontamination.
contamination rates (defined as the percentage of benchmark test samples found within our raw data pool) are notably high: reaching 90% for AIME24 and AMC233, 76.6% for MATH500, 35.7% for Minerva Math, and 33.6% for OlympiadBench. Recognizing that these benchmarks are frequently employed for model evaluation, DeepMath-100 103K underwent rigorous decontamination procedure. This process systematically identified and removed problems that overlap with these standard eval- uation sets, ensuring the integrity and reliability of future benchmark results obtained using models trained on DeepMath-10333
Broad Topical Coverage Complementing high difficulty and data integrity, key charac- teristic DeepMath- 103K is its extensive top- ical diversity spanning the mathematical land- scape. We categorized each problem using hierarchical topic structure, following the methodology from Gao et al. (2024). As illus- trated in Figure 4, this classification reveals that DeepMath- 103K draws problems from a mul- titude of core mathematical areas. Its scope ranges from fundamental topics such as Preal- gebra and Plane Geometry to sophisticated do- mains like Abstract Algebra (including Group Theory and Field Theory) and advanced Calcu- lus (covering Differential Equations and Appli- cations of Integrals, among others). This broad and deep topical foundation ensures that mod- els trained on DeepMatth 103K are exposed to rich variety of mathematical concepts and problem-solving paradigms, thereby fostering the development of more robust and widely generalizable reasoning skills.
Group Theerry
Frome Theuty
Calcula
Singlewanade
Hard Thcory
Pundtand
Trwwry
Presakulaa
1RAAN Their
uniks
Figure 4: Hierarchical breakdown of covered mathematical topics in DeepMath- 103K.
Data Novelty and Uniqueness As mentioned in $ I. DeepMath- 103K sources mostly from math forum, rather than common resources frequently adopted by other datasets. To evaluate the data novelty and uniqueness of DeepMath- 103K, we performed the following analysis for the datasets:
1. We first embedded all the samples using paraphrase- hwtp:/ssemmiiiiiiiaiiiiAAAAL.LL.
2. Samples with embedding similarity greater than 0.98 were considered the same samples.
Viewing each dataset as a set embeddings, Figure 5 presents the number unique elements in each set and the corresponding set sizes. DeepMath- -03K contains 82.81K problems that not found in others. This stark contrast highlights the data novelty and uniqueness DeepMath- 103K. We also plot their embedding distribution after t-SNE in Figure 6. ORZ-129K (Hu et al.,

---

Published as a conference paper at ICLR 2026
2025). Open-R1 (Face, 2025), SITLL-3-RL (Chen et al., 2025), -SR-Preview (Luo et al., 2025), and DAPO-178 (Yu et al., 2025), though curated independently, show very similar embedding dis- tribution, while DeepMath- 103K exhibits distinctly different pattem. This observation supports our claim that existing datasets overlap with each other because of using common data sources and further demonstrate the data novelty and uniqueness of DeepMath-103K.
DespMaih-101
82.81K
20.21K
Open-R1
43. 13.66K
50.32K
STILL-3-RL
3K
59.75K
DSR-Preview
11 1.07K
29.25K
DAPO-17K
9.37K 8.03K
Non- Unique Problems
Unique Problens
ORZ-129K
125.04K
OK
35K
70K
105K
140K
Count
Figure 5: Unique and non-unique problem counts in DeepMath-106 103K compared to other datasets.
(a) DeepMath- 103K
(b) ORZ-129K
(c) Open-R1
(d) STILL-3-RL
(c) DSR-Preview
(f) DAPO-17K
Figure 6: Embedding distributions of different datasets after (-SNE.
3 CONSTRUCTION OF DEEPMATH-103K
Verifiable Pool
95K
MMIQC 1,202K
High Difficulty Pool 1,090K
Unveribuble Filtered
995K
Raw Pool 2,869K
Decontomibated Pool 2,070K
WehfnstSub 1,612K
Difficalty
1,580K
MutinuMathhCost
Decontomissaion Filtered 199K
Figure 7: The data curation pipeline for DeepMath- 103k. Starting with an initial pool 2,869K raw questions, successive stages of data decontamination. difficulty filtering (retaining levels 25), and answer verifiability filtering yield 95K problems. These are then combined with 8K problems from SimpleRL (Zeng et al., 2025b) to form the final DeepMath- 103K dataset.
5

---

Published as a conference paper at ICLR 2026
This section details the meticulous data curation process used to construct DeepMath-103k. illus- trated in Figure 7. The process comprises four primary stages:
1. Source Analysis and Collection: Identifying and collecting mathematically challenging problems by analyzing the difficulty distributions of existing open data sources.
2. Data Decontamination: Rigorously decontaminating the collected data to remove poten- tial overlaps with standard evaluation benchmarks, ensuring evaluation integrity.
3. Difficulty Filtering: Filtering the decontaminated problems based on difficulty, retaining only those assessed at level higher to focus challenging content.
4. Answer Verification: Ensuring each curated problem possesses a verifiable final answer, consistently validated across multiple solution paths generated by DeepSeek-RI.
Overall, this curation pipeline ensures that DeepMath- 103K largely free from benchmark con- tamination and concentrates challenging mathematical problems suitable for advanced reasoning model training.
Stage 1: Source Analysis and Collection. To iden- tify data sources rich in challenging problems, be- gan by analyzing the landscape of existing open math- ematical reasoning datasets designed for SFT. These datasets utilize diverse collection methods. For instance, datasets such as MetaMathQA (Yu al., 2024), dart-math- hard (Tong et al., 2024), and OpenMathInstruct-2 (Tosh- niwal et al., 2024a) primarily focus on augmenting prob- lems and solutions derived from established datasets like GSM8K (Cobbe et al., 2021) and MATH (Hendrycks al., 2021b). In contrast, datasets like NuminaMath-Con (LI al., 2024), MMIQC (Liu et al., 2024), and WebInstruct- Sub (Yue et al., 2024) source content more broadly from the web, gathering materials such as exercises and discus- sions from online platforms (e.g., Math Stack Exchange). We follow Gao et al. (2024) to estimate the difficulty distri- butions of these potential source datasets, shown in Fig- ure 8, which reveals distinct patterns: datasets derived from GSM8K and MATH (MetaMathQA, mart-math-hard, Opentath) along with NuminaMath -oT. ex- hibited distributions heavily skewed towards lower difficulty levels (levels 1-5). Conversely, datasets sourced more broadly from web content, specifically MMIQC and WebInstructSub, displayed sig- nificantly flatter distributions with larger proportion of problems in the mid-to-high difficulty range (levels 5-9). Based on this finding, we selected Math StackExchange subsets from MMIQC and We- blnstructSub as our primary data sources due to their higher concentration of challenging problems. We also included NuminaMath-Con to enhance the topical diversity of the initial collection. After applying basic filtering, this selection process yielded a raw pool of 2,869K questions.
L6M
Level
14M
Level
1.2M
Level
Occurrence
L.OM
Level
Level
0.8M
Level
0.6M
Level
Level
0.4M
0.2M
0.0M
Dart-Math- Hard
MetaMathQA
OMI-2
NuminaMath-CCo CoT
MMIQC
WebinstruetSol
Figure 8: Difficulty distributions of various open mathematical datasets considered as potential sources.
Stage 2: Data Decontamination. As indicated by the high contamination rates observed in com- mon benchmarks (Figure 3), a rigorous data decontamination process was crucial for ensuring the integrity DeepMath-103K. We performed decontamination against a comprehensive suite of mathematics and STEM benchmarks, including MATH (Hendrycks al., 2021b), AIME (MAA,
a), AMC (MAA, b), Minerva Math (Lewkowycz et al., 2022), OlympiadBench (He et al., 2024), Omni-MATH (Gao al., 2024), MathOdyssey (Fang al., 2024), GAOKAR AO (Zhong al., 2023), JEEBench (Arora et al., 2023), MMLJ-STEM (Hendrycks et al., 2021a), CMATH (Wei et al., 2023), OlympicArena (Huang al., 2024), GSM8K (Cobbe et al., 2021), and GPQA (Rein et al., 2024). We adopted the decontamination method proposed by Toshniwal et al. (2024a):
For each candidate question in our raw dataset, we employed embedding similarity search (using -astirraaammiiiigggllaaaaa- Bingguaal--Mi--LLA. MiniLL--LL22-22 (Reimers & Gurevych, 2019)) to identify the top-k (k 5) most similar examples from the aggregated test sets of all targeted benchmarks.
Each candidate question was then compared against its top-k retrieved benchmark examples using an L.M-Judge (Llama-3.33-705) Inntsuut (Grattafiori al., 2024)) to determine if they

---

Published as a conference paper at ICLR 2026
Table 1: Examples of contamination detected between the raw data pool and benchmarks using
semantic comparison. Colors highlight conceptual or textual similarities.
Benchmark Raw Question
Benchmark Question
AIME24
How many routes there through from top left corner to bottom right in 20x20 grid"? I'm trying to solve this computer program- ming problem
Consider the paths of length 16 that follow the lines from the lower left corcer to the up- per right corner on 8x8 grid. Find the number of such paths that change direction exactly four times, as in the examples shown below.
Project Euler. I've seen a solution using nCr, where n and = 20. Could somcone explain to me how this work, please?
AMC23
Using only 3 paise, paise, and 9 paise coins, what the largest amount that cannot paid in exact change?
In the state of Coinland, coins have values 6,10, and 15 cents. Suppose is the value in cents of the most expensive item in Coinland that cannot be purchased using these coins with exact change. What is the sum of the digits of x?
constituted identical questions or paraphrases. If any of these comparisons indicated potential paraphrase duplicate, the candidate question was discarded.
Table illustrates the effectiveness semantic decontamination compared to simple lexical match- ing. This approach aims to identify not only exact duplicates but also near-duplicates and para- phrased questions that might otherwise overlap with evaluation sets.
Stage 3: Difficulty Filtering. Zeng et al. (2025a) highlights the importance of aligning RL train- ing data difficulty with the target model's reasoning capabilities, noting that powerful models benefit significantly from exposure to challenging problems. Building for DeepMMtth- 103K focuses on selecting problems that represent a significant reasoning challenge. To quantify difficulty, adopted the approach detailed in Gao et al. (2024). We assigned difficulty level to each decontaminated problem by prompting GPT-4o based on the annotation guidelines pro- vided by the AoPS. To ensure robust estimate, we queried GPT-4o six times for each problem and averaged the resulting ratings to determine its final difficulty level. We validated the consistency be- tween GPT-4o's evaluation and human evaluation in Appendix D. Subsequently, we applied strict filtering criterion, retaining only those problems with an estimated difficulty level of or higher.
this insight. curation process
Stage 4: Answer Verification. The availability of verifiable final answers is crucial for enabling rule-based reward in RLVR. which helps mitigate reward hacking and has been instrumental in training successful reasoning models like DeepSeek-R1 (Guo al., 2025). However, reliably con- structing such answers presents two primary challenges:
Some open- open-ended questions inherently lack a easily verifiable final answer.
2. Certain answers are excessively complex (e.g.. lengthy expressions intricate notation), making them challenging or even infeasible for automated rule-based verification.
To address these issues, we implemented rigorous two-stage verification process:
Question Filtering and Formatting: We used GPT-4o to process the raw questions. Problem types inherently unsuitable for verification were discarded. Questions phrased conversationally were rewritten into a standardized format seeking single numerical or symbolic answer.
2. Answer Verification via Consistency Check:
For questions successfully passing the above step, we generated three distinct solution paths using DeepSeek-RI. A rule-based verifier then extracted the final answer from each of these generated solutions, well as from the origi- nal source solution (when available). We enforced striet consistency: only problems where all extracted final answers were identical were retained in the final dataset.
Question standardization and answer consistency checking ensures that every problem included in DeepMath- 103K possesses final answer that is robustly verifiable using automated rules.
7

---

Published as a conference paper at ICLR 2026
Table 2: Math reasoning performance. "DeepMath" denotes models trained on DeepMath- 103K.
MATH
AMC
Olympiad Beuch
Minerva Math
AIME
AIME
Poly
Model
500
23
24
25
Math
Zero frof Base Model
Qwen-2.5-7B (Team, 2024)
54.8
35.3
27.8
16.2
7.7
5.4
28.1
Bpen-Reasoner-22 Zeto-7
2025)
81.8
58.9
47.9
38.4
15.6
40.7
Qwen-2.5-7B-SRL-Zo0 (Zeng et al., 2025a)
77.0
55.8
41.0
41.2
15.6
8.7
33.1
DeepMath-Zaro-277 (Ours)
85.5
64.7
$1.0
20.4
42.7
Qwen-2.5 Marh-BB (Team, 2024)
46.9
31.9
15.8
15.5
11.2
4.4
22.7
Qwen-2.5-Math--7 SRL-o-ooo (Hu at.l, 2025)
75.8
59.7
37.4
24.0
36.0
Dat-Zero-7B (Liu al., 2025)
80.0
66.7
43.4
40.8
32.7
11.7
40.8
Burus-2-7B-PRIME (Cui et al., 2025)
80.2
64.7
44.9
42.1
19.0
38.9
DeepMath-Zero-Math- (Ours)
86.9
74.7
52.3
49.5
34.2
23.5
46.6
Qwen-3-8B-Bass (Team, 2025)
60.3
42.5
33.1
27.2
11.2
7.9
30.0
DeepMMath-Zooo-Qqee---8 (Ours)
93.2
86.2
66.3
53.4
44.6
33.5
$2.3
RL from Instruct Models
AI-Distill-Qwen--L.B (Guo et al., 2025)
84.7
72.0
53.1
36.6
29.4
24.8
39.9
DeepScaleR- SB-Preview (Luo al., 2025)
89.4
80.3
60.9
42.2
42.3
29.6
46.8
Still-3-1.. Pre-rriiww (Chen et al., 2025)
86.6
75.8
55.7
38.7
30.8
24.6
43.1
DeepMath- (Ours)
37.3
OpenMath-Nemotrro- Nemotron- 1.5B (Moshkov et al., 2025)
91.8
90.5
70.3
61.3
DeepMath-Oman Omn-1.5- (Ours)
93.2
94.2
73.4
28.3
64.0
57.3
58.7
4 DEEPMATH SERIES MODELS
This section presents a comprehensive evaluation of the mathematical and general reasoning capa- bilities our DeepMath series of models, which were trained on DeepMath- 103K.
Training Paradigms We employed two distinct RL training paradigms:
Zero RL: This paradigm involves training LLMs from their base (non-instruction--unee) version using RL (Guo et al., 2025). We used group relative policy optimization (GRPO) (Shao et al., 2024) with fixes from Yu al. (2025), and trained Qwen-2.5-(Math)
with a rule-based reward (+1 for correct final answer, -1 otherwise). Detailed settings are available in Appendix C.
RI: We also performed RI. on instruction-tuned models that already possessing math reason- ing ability. We explored this using RI-DDstill-Qwenn-Be 1.5B (Guo et al., 2025) and OpenMath- Nemotron-1.5B (Moshkov et al., 2025).
Evaluation Following Zeng et al. (2025a;b), we assessed the mathematical performance of our models on: MATH-500 (Hendrycks et al., 2021b), AMC 2023 (MAA, b), OlympiadBench (He et al., 2024), Minerva Math (Lewkowycz et al., 2022), AIME 2024-2025 (MAA, a), and the En- glish subset of PolyMath (Wang et al., 2025a). To investigate the generalization of reasoning abili- ties beyond mathematics, we used the GPQA-Diamond benchmark, which covers biology. physics and chemistry (Rein et al., 2024). For all evaluations, we adopted pass @ accuracy (averaged over 16 samples) as the metric, and fixed the decoding parameters to temperature=0.66 top-p=0.95, and max toKens=32K. To ensure a fair comparison and eliminate variance caused by the evaluation script, we re-evaluated the performance of all baseline models under our evaluation settings.
4.1 MATHEMATICAL REASONING RESULTS
The results presented in Table 2 collectively demonstrate the effectiveness DeepMath- 103K as valuable resource for advancing the state-of-the-art in mathematical reasoning:
Zero RL Training on Base Model DeepMath-Zerie Zero-7B and DeepMath-Zeet -ert--ath----- trained from the base Qwen-2.5-7B and Qwen-2.5-Math- models, demonstrate significant performance gains and surpass all baselines on evaluated benchmarks. These results highlight the effectiveness of DeepMath-103k in enabling the training of powerful reasoners from scratch.
RL Training on Instruction-tumed Models Fine-tuning instruction-tuned models with RLVR DeepMath-103K also yields notable performance enhancements. DeepMath-1. 1.5B, initialized
8

---

Published as a conference paper at ICLR 2026
from R1-Distill-QU Saaill--wen-11.53 achieves strong performance, particularly on AMC23 (82.3%) and OlympiadBench (61.8%). Similarly, 1.5B, attains strongest results among 1.5B- scale models all evaluated benchmarks. The consis- tent improvements observed across different instruction-tunee baselines further validate the utility of DeepMath-1033 in boosting strong models.
CaatMMatt------- starting from OpenMath-Nemotrr
Ablation Table presents head-to- head comparison with the representative open dataset ORZ-129K. Under the same RI recipe, training on DeepMath- 103K alone outperforms training on ORZ-129K, while training on both achieves the high- est performance. These results directly validate the effectiveness of our data and demonstrates that DeepMath- 103K serves as a complement to existing resources (Figure 6). We also ablate the difficulty filtering step to verify its necessity. Re-
Table 3: Ablation results training datasets and dif- ficulty filtering strategy. Mean accuracy is computed across all evaluation in Table 2. Full results listed in Appendix E.
Model
Mean Acc.
Model (Qwen-2.5-Matthh
21.2
+ +ORZ-122 129K
50.7
+ Deeepeath-100 103K
52.5
-Difficulty Filtering
49.1
+ +oth (ORZ + DeepMath)
53.0
moving this filter leads to a performance drop from 52.5% 49.1%.
4.2 GENERALIZABLE REASONING BEYOND MATHEMATICS
Table 4: Performance on the GPQA-Diamond, MMLU-STEM and BBH.
GPQA-Diamond
MMLU- STEM
Model
BBH
Blology
Physies
Chemistry
Overall
fl.from Base Model
Qwen-2.5-7B
33.6
27.8
21.4
25.3
10.8
41.3
L, Open-Reasone--eeoo-TB
50.3
47.8
26.7
38.1
61.1
82.8
Qwean-25-7B-SimpleR..
31.9
37.9
22.6
30.2
$1.6
71.3
L DeepMath-Zero-78 (Ours)
57.2
$3.0
28.2
41.7
72.7
84.8
Qwen-2.5-Math-BB
32.2
26.0
21.1
24.3
19.3
35.9
L, Quen-222-Mah--7-5RRR--
40.1
31.2
22.9
28.2
34.8
58.6
Out-Zero-7B
49.0
36.8
22.0
31.0
48.4
65.4
Buros-2-7B-PRIME
44.1
37.4
24.1
31.8
40.4
58.2
DeepMath Math (Ours)
47.4
56.3
26.0
41.2
71.3
81.9
from
Instruct Models
13.5
36.2
4.4
19.1
17.8
62.1
L, DeepScaleR-1.5 1.55Pclee--.B--reiiii
15.5
46.8
9.1
26.1
19.5
65.9
Still-3-1.5B. Previee
38.4
20.7
L, DeepMath-1 1. 1.5B (Ours)
18.1
47.6
12.2
28.2
33,0
69.1
MoaMaatt-Nemotro---.. 1.5B
12.8
23.5
18.9
20.3
41.3
47.4
L, DeepMath-Omm- 1.5B (Ours)
17.1
28.4
21.5
24.1
48.6
54.0
Table 4 presents the performance of DeepMath models on GPQA- -iia--Dd (biology, physics and chemistry) (Rein et al., 2024). MMLU-STEM (science and enginecring reasoning) (Hendrycks et al., 2021a) and BBH (complex multi-step reasoning) (Suzgun al., 2022) scores. DeepMath models achieve superior performance compared to other baselines, demonstrating a remarkable capacity generalize their reasoning abilities acquired from math broader domains. We attribute this generalization to the data diversity and rigorous curation. By sourcing less structured but more diverse data like Math StackExchange, DeepMath-103K yields a dataset with unique and diverse problems. Furthermore, the rigorous curation pipeline ensures both the challenge and the integrity of the data. This exposure wider variety of problem scenarios and reasoning styles likely equips our models with more robust and transferable reasoning skills.
4.3 ANALYSIS OF ZERO RL USING DEEPMATH--033
Figure analyzes the zero RL training. Specifically, Figure 9a illustrates the trend response length throughout the training process, while Figure 9b tracks the emergence of four cognitive behaviors in Gandhi et al. (2025). The increasing trends both response length and the manifestation of cognitive behaviors suggest reproduction of the "aha moment" phenomenon observed in R1 (Guo

---

Published as a conference paper at ICLR 2026
4K
Desphlch-Zaon Zaro-
700
Subyoal Verifszston Rackincking
6%
Opensilaonnn
Qoemn 3 6-78
Ropane: Lagih
3K
Emirreatiion
Respose Langth
4
Derphtth-7an--
L.Prowaryy (dbagg
380
2.7%
2K
0K
-100
6K
122
250
375
500
Triining Stop
(8) Response length (training)
(b) Change of cognitive behaviors.
(c) Response length (test)
Figure 9: Training dynamics and response length of DeepMath-Zero-7B. (a) Rollout response
length. (b) Cognitive behaviors. (c) Average response length on evaluated benchmarks.
et al., 2025). Furthermore, Figure 9c shows the average response lengths of different models on the evaluated benchmarks. The notably longer response lengths exhibited by DeepMath -erhh--err--B suggest that more challenging problems can elicit deeper reasoning processes from the model.
5 CONCLUSION
In this work, we introduce DeepMath- 103K, a large-scale mathematical dataset specifically designed advance the reasoning capabilities of LLMs through RLVR. DeepMath- 1ee3 103K distinguishes itself through its high concentration of challenging problems, rigorous decontamination against a wide range of benchmarks, and the inclusion of verifiable final answers and multiple diverse solutions for each problem. Our data curation pipeline leverages the richness of less structured mathematical forums, resulting dataset with significant novelty and diversity compared existing resources. Our experiments demonstrate the substantial impact of DeepMath- 10MMMath-133...... 103K. Models trained on this dataset, the MeepMath series, achieve strong results on many mathematical benchmarks and ex- hibit remarkable generalization to domains beyond mathematics. By releasing the DeepMath- 103K dataset, along with our code and model weights, we aim to provide robust platform for the com- munity to further explore and push the boundaries of advanced reasoning.
6 ETHICS STATEMENT
We, the authors of this work, confirm that we have read and adhered to the ICLR Code of Ethics. Our work focuses on the development of a large-scale mathematical dataset, DeepMath-1033, to advance reasoning in AI. We have taken great care in its construction, including rigorous de- contamination process, to ensure fair and trustworthy benchmark evaluations. The dataset and our associated training code will be made publicly available promote open research and reproducibil- ity.
7 REPRODUCIBILITY STATEMENT
To ensure the reproducibility our work, we have made the DeepMath-1033 103K dataset, along with all associated training code, scripts, and model weights, publicly available. Further details experimental setup, including hyperparameters and implementation specifics, are provided in Ap- pendix C, enabling other researchers fully reproduce and build upon our results.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models, 2025. URL https: //aaxiv.crrgg....
173877
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlfčck, Agustín Piqueres Lajarin, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big data-centric training of small language model, 2025. URL https: /tttps://arxiv aawiv. org/ abs 22022 02737.
Daman Arora, Himanshu Singh, and Mausam. Have LLMs advanced enough? a challenging prob- lem solving benchmark for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7527-7543, Singapore, December 2023. Association for Computational Lin- URL / http//aaaaanntog........ org/20223
guistics. doi: 10008665///002233emmmmmmmil/ cmnlp-main. 468.
Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q. Tran, and Mehran Kazemi. Smaller, weaker, yet better: Training LLM reasoners via compute-optinal sampling. In The Thirteenth International Conference Learning Representations, 2025. URL net/forum? Srtrrmmmdd3OOaaQQQu
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang. Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like 1lms, 2024. URL https://arxix: org/abs abs/////// / /24222212222 21187.
Zhipeng Chen, Yinggian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, Lei Fang, Zhongyuan Wang. and Ji-Rong Wen, An empiri- cal study on eliciting and improving rl-like reasoning models. arXiv preprint 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01455, 2025.
Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small Ilms: What works and what doesn't, 't 2025. URL httpp:::/aaxiv..rgg/////// 2553/ 16219.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: /tt.thtb.ccm/////t.t http://///ooggggga//////////////
Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing, and Kai Zou. Mathodyssey: Benchmarking mathe- matical coooblem-solvingg skills in large language models using odyssey math data. arXiv preprint arXiv:2406. 18321, 2024.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cogni- tive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.00 00..5003.130000 0130,, 2025.
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang. Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren. Tianyu Liu, and Baobao Chang. Omni- math: A universal olympiad level mathematic benchmark for large language models, 2024. URL https://arxe http///iiw..gg/ab////////sss www.//g.//////oo////////o.ee/s// 079855

---

Published as a conference paper at ICLR 2026
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXh::240.221783, 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yiqi Wang, Qihang Fan, Yuang Ai, Huaibo Huang, Ran He, Zhenheng Yang, and Quanzeng You. infimm-webmath-406 Advancing mul- timodal pre-training for enhanced mathematical reasoning, 2024. URL a29/2409. 12568.
org//
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual mul- timodal scientific problems, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Breung-Yeung, Shum Xiangyu Zhang. Open- reasoner-zero: An open source approach to scaling reinforcement learning on the base model. https://gitans com/Open- Reere--Rasnnne--eeeeeeee Open Reasoner- Zero, 2025.
Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyu- manshan Yc, Ethan Chern Yixin Ye, Yikai Zhang, Yuging Yang. Ting Wu, Binjie Wang. Shichao Sun, Yang Xiao, Yiyuan Li, Fan Zhou, Steffi Chern, Yiwei Qin, Yan Ma, Jiadi Su, Yixiu Liu, Yux- iang Zheng, Shaoting Zhang. Dahua Lin, Yu Qiao, and Pengfei Liu. Olympicarena: Benchmark- ing multi-discipline cognitive reasoning for superintelligent ai. arXiv preprint arXiv:2406. 12753, 2024. URL https:
/206. 12753.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan- guage models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 3843-3857. Curran Asso- ciates, Inc., 2022. URL https: http:////oooooceeee/eooooo neurips. cc wwwpapprpppppeeeeeeeeeeeeeeeeeee files/ htteee//pppp////2222 file/
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang. Ziju Shen, Zihan Qin. Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. hattps://huggingaaee wttp://aaa.....aaaaeeeeeeeea wt..ao////-/-... /ww.iiaaaiaaataaaaaaaaaa project- numi na no-prrorres -errepriieeieeee 2024.
(https:/ http:::///gttttt com/ no-prooress--priee/ttloo /ob/ http://mannneeerrr//neeee na sa_aaasse..pdf).
Haoxiong Liu, Yifan Zhang, Yifan Luo, and Andrew Chi-Chih Yao. Augmenting math word prob- lems via iterative question composing. 2024. URL http::://aa.ii..ogg//////// /2401. .90033.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding ri-zero-like training: A critical perspective. arXiv preprint arXiv:2503. 2025.
Michael Luo, Sijum Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Surpassing ol or-prewe with 1.5b model by scaling rl, 2025. Notion Blog.
12

---

Published as a conference paper at ICLR 2026
MAA. American invitational mathematics examination (AIME). Mathematics Competition Series,
n.d.a. URL https::/mmaa..eeeeeeeeee
MAA. American mathematics competitions (AMC 10/12). Mathematics Competition Series, n.d.b.
URL atttps://maat http://www..reeaaaaamccmececaaaa
Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-art math- ematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504. 2025.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL http:://axxti org/ abss 9008/1908.11 1000.1008000 10084.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proot q&a benchmark. In First Conference Language Modeling, 2024. URL https: /oppreeviee.....e.e net/forum? 166884498.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of math- ematical reasoning in open language models, 2024. URL https: / //axxiv. org/ abs/2404 03300.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thoon thhtuuhtt can solve them. arXiv preprint 1iXiv:2200.092000 2022.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL WtLLhttss/////e... github.ic/blog. /wwn/./wwnn.m.mmmmm 5/.
Qwen Team. Qwen3 technical report, 2025. URL https://aaxxv..or//bbtt
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. DART-math: Difficulty- aware rejection tuning for mathematical problem-solving. In The Thirty-eighth Annual Con- ference Neural Information Processing Systems, 2024. URL htttps://operrvvieee
forum?id=z102100]DD
Shubham Toshniwal, Wei Du Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstructt2:: Accelerating ai for math with massive open-source instruction data. arXiv preprint 2400v:2410.0/5600 2024a.
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-:: A 1.8 million math instruction tuning dataset. arXiv preprint arXiv: Arxiv- 2402. 2024b.
Yiming Wang, Pei Zhang, Jialong Tang, Haoran Wei, Baosong Yang. Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, Qiqian Cang, Yichang Zhang, Fei Huang, Junyang Lin, Fei Huang, and Jingren Zhou. Polymath: Evaluating mathematical reasoning in multilingual contexts. arXiv preprint 2rXiv:2504.18428, 2025a. URL https:// /ttp:///xxvvvvoo// org/ abs/25C4. 18428.
Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang. Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Thoughts all over the place: On the underthinking of ol-like 1lms, 2025b. URL attps:/ /rxiiv.rg// abs/2501 1858.
13

---

Published as a conference paper at ICLR 2026
Zengzhi Wang, Xuefeng Li, Rui Xia, and Pengfei Liu. Mathpile: A billion-token-seen pretrain- ing corpus for math. In The Tiiht Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track, 2024. URL httpp://ppee.erriww.wa///oorrere
RSvhU69sbG.
Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang. Cmath: Can your language model pass chinese elementary school math test?, 2023.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu, Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfih International Conference on Learning Representations, 2024. URL http::/:////ppeveeerree/ net wttee//orrumm.ceeeeengggg...
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei Ying Ma. Ya- fa-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open- source Ilm reinforcement learning system at scale, 2025. URL htttps://axxii.rrrrttt 2503. 14476.
Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. Advances in Neural Information Processing Systems, 37:90629-90665 2024.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 200: Investigating and taming zero reinforcement learning for open base models in the wild, 2025a. URL https::t
http://wwwvv.o////./o /
abs absaabss///// /2503. 18892.
Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. reason, 2025b. Notion Blog.
htttss//hkust-nn2. not ion. site/ wttte//sssseeeeeeeeeeeeeeeeeeeee -eason,,
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023.
Fan Zhou, Zengzhi Wang, Nikhil Ranjan, Zhoujun Cheng, Liping Tang, Guowei He, Zhengzhong Liu, and Eric P. Xing. Megamath: Pushing the limits of open math corpora. arXiv preprint arXiv:2504.02807, 2025. Preprint.
14

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Datasets for advancing mathematical reasoning of LLM falls into three main lines corresponding to the three stages of LLM post-training: continue pre-training (CPT), SFT and RL. CPT aims to inject fundamental mathematical knowledge into LLMs with representative works like OpenWeb- Math (Paster et al., 2023), MathPile (Wang et al., 2024), Math (Allal et al., 2025), and MegaMath (Zhou et al., 2025). SFT has been a foundational approach, utilizing datasets like MATH (Hendryeks et al., 2021b) and GSM8K (Cobbe al., 2021) which pro- vide problems with step-by-step solutions to teach models reasoning patterns. Subsequent efforts have focused on creating larger, harder and more diverse SFT datasets, such as MetaMathQA (Yu et al., 2024), Open.aathnsstruct (Toshniwal et al., 2024b;a). NuminaMath-Con (LI et al., 2024), MMIQC (Liu et al., 2024), dart-math hard (Tong al., 2024), and OpenMatthResonnng (Moshkov et al., 2025). Recent progress in RLVR catalyzes datasets with verifiable reward, such as Open- RI (Face, 2025), ORZ- 129K (Hu et al., 2025), DSR-Preview (Luo et al., 2025), DAPO-17K (Yu et al., 2025), and BigMath (Albalak et al., 2025). DeepMath-103K distinguishes itself by a unique blend of high difficulty, rigorous decontamination, and verifiable answers.
WRMM--We--Mth (Han et al., 2024), Fine-
B CONTAMINATION ANALYSIS OF EXISTING DATASETS
We performed contamination analysis of
250
223
several existing datasets, including ORZ-
Conntanintted Semples
129K (Hu et al., 2025), DSR-Preview (Luo
200
et al., 2025), DAPO-17K (Yu et al.. 2025), Open-RS (Bansal et al., 2025), Open-R1 (Face, 2025), and NeepMath-103K. Our analysis fo- cused on detecting potential contamination from the MATH500 (Hendrycks et al., 2021b),
160
1500
120
100
50
a commonly used benchmark. We employed string-based comparison method, specifically identifying cases where the normalized in- del similarity between problem in the ana- lyzed dataset and a problem in MATH500 ex- ceeded 90%. This approach is notably more lenient than the rigorous semantic decontam- ination procedure used in the construction of DeepMath- 103K ($ 3). However, the numbers of contaminated samples shown in Figure 10 re- veal that most of the analyzed datasets exhibit some degree of contamination, with the exception of 1eepMath-103K.
24
ORZ- 129K
STILL-3-8 --LL-
DSR- Preview
DAPO-17H 1KK
Open-RS
Open-R)
Deep 103K
Figure 10: Number of contaminated samples
in various datasets when compared against the MATH500 benchmark.
C TRAINING DETAILS
We use verl the training framework Configurations are listed in Table 5,
Table 5: Configurations for training DeepMath series models.
Conlig
DeepMath-Zero-77
DeepMuthZZer-Maul Ma
Deep 1.5B
DeepMath- Omn--1.5
Ir
Ie-6
le-6
Ie-6
le-6
kloooef
Ic-3
1o-3
max prompt length
2K
IK
2K
2K
response length
10K
train butch size
512
512
128
128
ppo mini butch size
32
64
clip Jow
clip ratio high
0.28
0.28
0.27
0.27
temperature
1.0
rolloata
18
overlong, buffer.
2K
512
4K
4K
training. steps
Phttps:/ / com/voom/oo http://vvoceeggicoooooo vere
15

---

Published as a conference paper at ICLR 2026
Table 6: Examples of geometry problems retained by the difficulty filtering process (level 5).
Difficulty Problem
5
Four random points are placed in the plane, with each point's horizontal and vertical coor- dinates uniformly distributed on the interval (0. 1). What is the expected largest size of subset of these points that can form the vertices a convex polygon?
square has one side lying on the line y y x Determine the minimum possible area of the square."
17 two other vertices on parabola
Determinc the sequence s(k, n), which represents the number of sides the intersection of a unit-radius regular polygon Ps with & sides and rotating unit-radius regular polygon P. with i kt sides, the angle of rotation / varies from to 2m. Provide the sequence s(k,n) n) for all >
Consider convex n-gon A2 A,, inscribed in a unit circle. Determine the maximum value of the sum of the squares of all its sides and diagonals
projective planes on w such no two distinct members of C are isomorphic. A set I. S P(X) is projective plane on X * Ø if: 1. For any distinct z, X, there unique [ E L such that z, y e I. For any distinct There exist four distinct elements of such that no member contains more than two of these four elements. Two projective planes L. and M on X are isomorphic if there is bijection y X - X such that if and only if (p()) M.
Determine the maximal cardinality of collection
1.m E I., e m|
D DIFFICULTY ASSESSMENT
As mentioned in $ 3, difficulty ment follows the methodology from Omni- MATH (Gao al., 2024), which relies an automated, model-based evaluation. While the reliability of this approach was rigorously validated in Gao et al. (2024), we further verified the consistency between model and human judgment in Figure 11, which com-
Model-Labeles Difficulty
2.9
3
2.4
pares GPT-4o difficulty ratings with the estab- lished human-annotated difficulty levels from the MATH dataset (Hendrycks et al., 2021b) and shows strong positive correlation. Table 6 also illustrates how increasing difficulty levels often correlate with greater conceptual depth and reasoning complexity, successfully captur- ing the inherent challenge of the problems.
1.9
2
1
2
3
Nluman-Labeleds Difficulty
Figure 11: Correlation between model and human labeled difficulties.
E ABLATION
Table 7: Per-henchmark performance breakdown for Table 3.
MATH
AMC
Olympiad Bench
Minerva Math
AIME
AIME
Poly
Model
500
23
24
25
Math
2wen-2.5-Math-77
46.9
31.9
15.8
15.5
11.2
4.4
22.7
ORZ.-129 129K
83.7
71.7
51.6
46.6
32.7
21.2
47.2
DeepMath-103E
86.9
74.7
52.3
49.5
34.2
23.5
46.6
-Difficulty Filtering
84.5
70.9
48.6
46.8
31.5
16.0
45.2
+Both
87.5
75.0
51.7
50.9
34.2
24.0
48.0
16

---

Published as a conference paper at ICLR 2026
F LLM USAGE STATEMENT
In the preparation of this manuscript, we utilized LLMs as an assistive tool. The primary applications of LLMs were to help polish the writing and enhance the clarity of the text, as well aid in various data processing tasks. The core research ideas, experimental design, and final analyses were conceived and executed by the human authors.
17


Reviews:
[{"review_id": "SFDKkMzDfR", "reviewer": "Reviewer_tfPX", "summary": "This paper introduces a large-scale dataset, DeepMath-103K, with high difficulty (mainly levels 5-9) and verifiable answers for rule-based RL, compared with other datasets (DAPO-17K, DSR-Preview, Open-R1, ORZ-129K). Using DeepMath-103K, the models under zero RL and RL, outperform than others, such as ORZ-7B, Qwen-2.5-Math-7B. It seems a very useful dataset for AI community. The construction of DeepMath-103K is clear. However, more experiments should be conducted, such as Qwen-2.5-7B-DAPO-17K, Qwen-2.5-7B-ORZ-129K, etc.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a useful dataset DeepMath-103k with high difficulty, and the construction of dataset is very clear.\n2. DeepMath-103k performs better than other datasets, with same training method (zero RL from base model, and RL from instruct models)", "weaknesses": "The experiments is not enough. A more comprehensive comparison with existing datasets is needed, including ORZ, DAPO-MATH-17K, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "49jRetTtvp", "reviewer": "Reviewer_JNea", "summary": "The paper proposed DeepMath-103k, a large-scale and challenging mathematical reasoning dataset. The dataset is constructed to emphasize the difficulty of the problems, which is proven to be essential to the improvement of finetuning models on these data. The authors show with data that their dataset is also the most diverse dataset that contains most unique questions. Above all, the extensive experiments show that models finetuned with this new dataset are promised to have significant improvement over existing open-source baselines.", "questions": "1. The table 2 results are promising, but it seems lack of solid baselines with previous existing datasets. I understand that it is not easy to find appropriate baselines sometimes, but I would like the authors to clarify this point with some discussions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and demonstrated.\n2. The authors clearly declared and proved the unique advantage of their dataset, with its number, difficulty and diversity.\n3. The experimental results proved that the improvment brought by the dataset is promising and consistent.", "weaknesses": "There is no significant weakness within the dataset scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aXkGfwWf4T", "reviewer": "Reviewer_8Pxb", "summary": "This paper introduces DeepMath-103K, a large-scale and challenging mathematical reasoning dataset for reinforcement learning with verifiable rewards. It emphasizes benchmark decontamination, difficulty filtering, and verified final answers. The authors also train a series of DeepMath models on this dataset and show performance gains on AIME, AMC, MATH-500, OlympiadBench, and GPQA-Diamond.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1) The work is well-motivated and tackles a meaningful gap in mathematical reasoning datasets, focusing on challenge level, contamination issues, and result verifiability.\n\n(2) A thorough decontamination process is implemented, combining semantic matching and LLM-based assessment to effectively minimize data leakage. \n\n(3) Experiments demonstrate consistent gains over baseline methods across various model sizes and learning settings—both in zero-shot and reinforcement learning scenarios.", "weaknesses": "(1)\tDifficulty annotation relies entirely on GPT-4o without expert calibration, and no inter-rater agreement or error analysis is provided.\n\n(2)\tLack of ablation on the curation pipeline—unclear which components (decontamination, difficulty filtering, answer verification) are most critical to performance gains.\n\n(3)\tR1 solution generation bias – R1-generated reasoning paths may reinforce self-biased reasoning structures, but the paper lacks discussion of potential reasoning homogenization or confirmation bias.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iFyAam3R1C", "reviewer": "Reviewer_3HiP", "summary": "This paper presents a carefully constructed, high‑difficulty, decontaminated, and verifiable dataset for math RL and demonstrates strong small‑model results and promising cross‑domain transfer. The authors curate problems primarily from Math StackExchange‑style web sources and pass them through a four‑stage pipeline. The result is 95K challenging items augmented with 8K level‑3–5 problems, each annotated with a hierarchical topic label and three diverse solutions. Models trained on DeepMath‑103K achieve substantial gains on several existing benchmarks such as MATH‑500 and  AMC23. Also, results on GPQA‑Diamond indicate transfer beyond mathematics.", "questions": "see above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1: This paper builds a high‑difficulty dataset with a clear distribution skew toward level‑5–9 problems, which directly addresses a key bottleneck for training strong reasoners.\n\nS2: The dataset shows strong topical breadth via a hierarchical taxonomy that spans core and advanced areas. Each problem includes a rule‑verifiable final answer and three R1 solution paths, enabling RLVR and SFT without manual cleaning.\n\nS3: The models trained on this dataset achieve state‑of‑the‑art small‑model results on AIME and other math benchmarks and show transfer to GPQA‑Diamond.", "weaknesses": "W1: The paper briefly cites contemporaneous datasets (e.g., BigMath and OpenMathReasoning) but does not provide head‑to‑head dataset ablations or training‑data mixing studies that would establish relative benefits.\n\nW2: The claims of cross‑domain generalization rely mainly on GPQA‑Diamond. It would be more convincing if the authors evaluated on additional scientific or reasoning benchmarks\n\nW3: Several closely related works are missing from the discussion [1–4].\n\nW4: The data curation pipeline heavily depends on LLMs—an LLM-judge (Llama-3.3-70B) for decontamination, GPT-4o for difficulty labeling, and GPT-4o again for rewriting and answer verification. Since LLMs are not perfectly reliable, hallucinations or propagation errors may occur. The authors should clarify whether human verification was performed for each stage.\n\nW5: The experiments are restricted to small and medium-sized models. It would strengthen the paper if the authors included results on larger models (e.g., 72B)\n\n\n\nTypos: \n\n+ In abstract, advancing reasoning  -> advanced reasoning \n\n+ SITLL‑3‑RL or STILL‑3‑RL should be made consistent throughout the paper.\n\n\n\nReference\n\n[1] Albalak et al., Big‑Math: A Large‑Scale, High‑Quality Math Dataset for Reinforcement Learning in Language Models, arXiv:2502.17387, 2025.\n\n[2] Moshkov et al., AIMO‑2 Winning Solution: Building State‑of‑the‑Art Mathematical Reasoning Models with OpenMathReasoning Dataset, arXiv:2504.16891, 2025.\n\n[3] Paster et al., OpenWebMath: An Open Dataset of High‑Quality Mathematical Web Text, NeurIPS 2023.\n\n[4] Zhou et al., MegaMath: Pushing the Limits of Open Math Corpora, COLM 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)