AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning
Authors: Yang Chen
Zhuolin Yang
Zihan Liu
Chankyu Lee
Peng Xu
Mohammad Shoeybi
Bryan Catanzaro
Wei Ping
Venue: NeurIPS 2025 poster ()
Keywords: reasoning
reinforcement learning
code
math
Primary area: deep_learning
TL;DR: scaling math and code rl

Abstract: Despite recent progress in large-scale reinforcement learning (RL) for reasoning, the training recipe for building high-performing reasoning models remains elusive. Key implementation details of frontier models, such as DeepSeek-R1, including data curation strategies and RL training recipe, are often omitted. Moreover, recent research indicates distillation remains more effective than RL for smaller models. In this work, we demonstrate that large-scale RL can significantly enhance the reasoning capabilities of strong, small- and mid-sized models, achieving results that surpass those of state-of-the-art distillation-based models. We systematically study the RL training process through extensive ablations and propose a simple yet effective approach: first training on math-only prompts, then on code-only prompts. 
Notably, we find that math-only RL not only significantly enhances the performance of strong distilled models on math benchmarks (e.g., +14.6\% / +17.2\% on AIME 2025 for the 7B / 14B models), but also code reasoning tasks (e.g., +6.8\% / +5.8\% on LiveCodeBench for the 7B / 14B models).  In addition, extended code-only RL further improves code benchmark performance while causing minimal degradation in math results.  We develop a robust data curation pipeline to collect challenging prompts with high-quality, verifiable answers and test cases to enable verification-based RL across both domains. The dataset will be released to support open research.
Finally, we identify key experimental insights, including curriculum learning with progressively increasing response lengths and the stabilizing effect of on-policy parameter updates.  We find that RL not only elicits the foundational reasoning capabilities acquired during pretraining and supervised fine-tuning (SFT), but also pushes the limits of the model’s reasoning ability, enabling it to solve problems that were previously unsolvable.

AceReason-Nemotron Advancing Math and Code Reasoning through Reinforcement Learning
Yang Chen
Zhuolin Yang
Zihan Liu
Chankyu Lee
Peng Xu
Mohammad Shoeybi
Bryan Catanzaro
Wei Ping
NVIDIA
Abstract
Despite recent progress large-scale reinforcement learning (RL) for reasoning, the training recipe for building high-performing reasoning models remains elusive. Key implementation details of frontier models, such DeepSeek-RI, including data curation strategies and RI. training recipe, are often omitted. Moreover, recent research indicates distillation remains more effective than RI for smaller models. In this work, we demonstrate that large-scale RL can significantly enhance the reasoning capabilities of strong, small- and mid-sized models, achieving results that surpass those of state-of-the-ar distillation-based models. We systematically study the RL training process through extensive ablations and propose simple yet effective approach: first training math-only prompts, then on code-only prompts. Notably, we find that math-only RL not only significantly enhances the performance of strong distilled models math benchmarks (c.g.. + +14.6% +17.2% on AIME 2025 for the 7B 14B models), but also code reasoning tasks (c.g. +6.8% +5.8% Binch for the 7B/ 14B models). In addition, extended code- only RL iterations further improve code benchmark performance with minimal degradation in math results. We develop robust data curation pipeline collect challenging prompts with high-quality, verifiable answers and test cases to enable verification-based RL across both domains. Finally, identify key insights, including curriculum learning with progressively increasing response lengths and the stabilizing effect of on-policy parameter updates. We find that RL not only elicits the foundational reasoning capabilities acquired during pretraining and supervised fine-tuning (e.g., distillation), but also pushes the limits of the model's reasoning ability, enabling it to solve problems that were previously unsolvable."
Introduction
Reasoning capabilities are fundamental component of AI. Since the introduction of OpenAI ol (OpenAI, 2024), building reasoning models using large-scale reinforcement learning (RL) has attracted significant attention. Remarkable progress has followed the open-soureing of DeepSeek- RI (Guo et al., 2025), empowering the open LLM and research communities to develop of the-art reasoning models through RL distillation. However, key technical details necessary for reproduction, such as data curation strategies and the specific RL training recipe, were omitted from the original DeepSeek-RI report (Guo et al., 2025), leaving the community scrambling to replicate its success.
"Equal contribution. Correspondence to: (yachen, zhuoliny, spingi/nvidia... "W release the model, dataset, and full technical report https:: htttpst//huggingffcceee. wttpt/cee..oppppppeepeee 2505. 16400.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Subsequent efforts by different teams explored different model sizes (e.g.. 1.5B (Luo et al., 2025b), 7B (Wen et al., 2025), 14B (Luo et al., 2025a), and 32B-only (Yu al., 2025)), different initial checkpoints (e.g., base models (Yu et al., 2025) and distilled reasoning models (He et al., 2025)), and different target domains (e.g., math (Luo al., 2025b), code (Luo al., 2025a), and physical Al (Azzolini et al., 2025)). Each study shows potential path to success in specific settings but lacks a conclusive or consistent training recipe. Moreover, both DeepSeek-R (Guo et al., 2025) and Llama- Nemotron (Bercovich et al., 2025) report that distillation outperforms RL for small and mid-sized models, recommending RL only for the largest models, such as the SeepSeek--33677 2024a)
(Liu et al., strategy also adopted by the Qwen3 (Qwen, 2025a).
In this work, we demonstrate that large-scale reinforcement learning (RL) can significantly enhance the reasoning capabilities of strong small- and mid-sizee SFT models (DeepSeck-)) Sell-wee--Disttee 7B/14B) achieving performance competitive with state-of sfte-of-the-are distillation -hased results at 7B, and surpassing them at 14B (Moshkov et al., 2025; Ahmad et al., 2025).
Specifically, we make the following contributions:
1. We propose conducting math-only and code-only RL separately: the distilled SFT model is first trained on math-only prompts, followed by training on code-only prompts. This approach was initially motivated by training efficiency considerations, as the average verification time for code is significantly longer than that for math. Subsequently, we found two exciting observations: i) Math-only RI. significantly boosts the performance of strong distilled models not only on math benchmarks (e.g., +1 +14.6% / +17.2% on AIME 2025 for the 7B/ / 14B models), but also on code reasoning tasks (e.g., +6.8% / +5.8% on LiveCodeBench v5 for the 7B / 14B models); see Table for details. ii) Extended iterations of code-only RL lead to minimal or no degradation on math reasoning tasks (e.g.. +1.0% / -0.8% on AIME 2024 / 2025 for the 7B model); see Table 4 for details. These observations contrast with domain- specific supervised fine-tuning (SFT), which can lead to catastrophic forgetting and degraded performance on other domains.
2. We develop and share systematic data curation recipe to collect high quality math problems with verifiable answers, as well as coding descriptions with test cases, ensuring that all data is reliable and testable. We open-source the dataset for the benefit of the community at https: / httpss://uggggggg /wwggghggaccc..eeeeco/oo co/ dawatttss./aaaaeaa.aaaaaaaa hiviiviiii dida AceReason-Mateee
3. To ensure consistent conclusions, we examine the RL training recipe through detailed ablation studies and analysis under state-of-the-art settings. Our findings include: i) Curriculum learning with progressively increasing maximum response length improves both training efficiency and final accuracy on reasoning benchmarks. ii) On-policy parameter updates stabilize the RL process. iii) RL not only elicits the foundational reasoning capabilities acquired during pretraining and supervised fine-tuning (c.g., distillation), as evidenced by significant improvements
pass@1, 1, but also expands the model's capabilities to solve previously unsolvable problems, as demonstrated by substantial and consistent gains from pass@64 to even pass@ 1024.
2 Related Work
Training LLMs to reason has been long-standing research focus (Wei et al., 2022), especially in the domains of code (Chen et al., 2021) and math (Cobbe et al., 2021). In recent years, major development efforts have focused on acquiring reasoning capabilities by training on math and code data during both the pretraining and supervised fine-tuning (SFT) stages (Shao et al., 2024; Guo et al., 2024; Grattafiori et al., 2024; Yang al., 2024b; Liu et al., 2024d). Reinforcement learning (RL) has previously been explored for math reasoning using reward models tailored to the math and code domains (Shao et al., 2024; Yang et al., 2024b). However, the gains have been limited, largely due to the inherent challenges of reward modeling in mathematical and coding domains (Lightman et al., 2023; Liu et al., 2024d.b).
The release of OpenAl ol (OpenAI, 2024), and especially the open-sourcing of DeepSeek-RI (Guo et al., 2025), highlights the effectiveness of large-scale RI. through rule-based verification. In the case of math problems with deterministic answers, models are required to output the final result in a specific format (e.g., boxed), enabling accurate rulc-based verification (c.g., Yang et al., 2024b; Liu

---

et al.. 2024d). For code problems, feedback is provided through compilation and execution against predefined test cases (e.g., Zeng et al., 2025; Luo al., 2025a).
Due to the absence of key implementation details in frontier models, such as RL training recipes and data curation strategies, subsequent works have explored and shared data curation methods (Luo et al., 2025b; He et al., 2025; Luo et al., 2025a), and introduced various techniques to improve and stabilize the widely adopted GRPO training (Shao al., 2024). These include progressively increasing the maximum response length (Luo et al., 2025b,a; He et al., 2025), clip-higher to mitigate entropy collapse (Yu et al., 2025), and overlong filtering to avoid penalties from truncated generations within the maximum response length (Yu et al., 2025). Many of these efforts focus exclusively on either the math domain (Luo et al.. 2025b; Yu al., 2025; Chen al., 2025; RL Lab, 2025) or the code domain (Luo et al., 2025a: Zeng et al., 2025). highlighting the difficulty of handling heterogeneous prompts and inherent complexity of RI. training. Furthermore, the range of reported benchmarks remains limited, typically to AIME 2024 2025, and LiveCodeBench (Jain et al., 2024), compared to broader evaluations frontier reasoning models (Guo et al., 2025; Qwen, 2025a).
Another line of follow-up work focuses on distilling existing frontier reasoning models, which are originally trained via RL (Guo al., 2025; Qwen, 2025b), through strong-to-weak distillation using rejection sampling (Ahmad et al., 2025; Moshkov et al., 2025; Bercovich et al., 2025), as prior studies have found that RL yields suboptimal results for smaller models compared to distillation (Guo et al., 2025; Bercovich et al., 2025). In this work, we initiate RL from strong distilled models, and show that it can achieve results that are competitive with or surpass existing of-t-of-theean distillation-bar approaches on math (Moshkov et al., 2025) and code (Ahmad et al., 2025).
Method
3.1 Framework
We adopt the GRPO algorithm (Shao al., 2024), as used DeepSeek-R1, instead of PPO (Schulman et al., 2017), due to its simplicity and the advantage of not requiring separate value function model. For each question-ansee answer pair (q, a), we sample from policy model Tea to generate a group of G individual rollouts
We assign a reward score S; S(o, a) to each response on, given the oracle answer using rule-based reward function S. We employ the token-level policy gradient loss variant of GRPO, as introduced by Yu al. (2025),
JGRPO(O) (0) E(gaa)-D, (a)))_maa
lod min (rir(0)A..e. clip clip (rp(rr((0)) E, + - - -3DKL (
1
Σ
G
loy|
1=
(1)
therein,
To(0re g. 01.cc) (0)
Â., S: mean (fS)) (td for VI,
Te (0r. g, (i,e)))
where (0) is the teven-level importance weight, and the token-level advantage Å,, within each response uniformly assigned with the value of normalized reward score across the group (S.)
Our experiments strictly adhere to on-policy training by performing a single gradient update after generating a group of G rollouts. This approach ensures stable RL training and helps prevent entropy collapse. Consequently, the policy used for data collection matches the current policy, i.e., Te... (. q)= = T- g). and the importance weight (0) 1. Additionally, we eliminate the KL divergence term by setting 3 = 0, thus the GRPO objective became REINFORCE objective (?) with group-normalized rewards,
G
(0) = B(g,,),,, 1o (e)(-
A
(3)
loi|
i=1
3

---

So, the update rule becomes,
VaJGRPO (, (0))__)=0((1)
Ve log mo(oia q, 0i,0))))
Aid
Σ
i== oii
(4)
We started RL experiments from distilled reasoning models DeepSeek-RIl WeI--wee--Distlllee 7B/14B (Guo et al., 2025), ensuring that the experiments were conducted under controlled conditions without introducing variations in distillation data fine-tuning compute. We the veRL. frame- work (Sheng et al., 2024), which implements token-level loss for GRPO, and employ the vLLM inference engine (v0.7.3) (Kwon al., 2023) for sample generation. Our custom modification includes adding math and code reward functions (verifiers) to the implementation. All experiments are conducted using 128 NVIDIA H100 GPUs.
Reward functions:
For verification of math problems, we employ rule-based Python verification function built on top of sympy, following the approach of AceMath (Liu et al., 2024d). Specifically, it relies on antlr4- python3-runtime (v4.11.1) and sympy (v1.12). This configuration is crucial for ensuring accurate symbolic equivalence. We extract the answer from \ \bboxed )) appearing after the <\think> token and assign rewards strictly based on the correctness of this answer (1 for correct, 0 for incorrect), without applying any cormat-basede rewards or length penalties. Using process pool with 64 workers, the average verification time is approximately 3.9 seconds per 1,024 instances.
For coding problem verification, we utilize a local sandbox verifier, following the code execu- tion tools implemented in the LiveCodeBench repository (Jain et al., 2024). Given the model's code block that follows
output, we extract the code generated within . 'python' [code] < Nthink> token. Binary rewards are then assigned based on code execution outcome on full set of test cases. positive reward will be granted if and only if the extracted code successfully passes all test cases within the specific time limit. Using a process pool with 64 workers, the average verification time for code approximately 552.4 seconds per 1,024 instances.
Given the significant difference in verification time between math and code. we propose conducting math-only and code-only RL separately.
3.2
Math-only RL
3.2.1
Data Curation
We developed a data collection and verification pipeline to generate high-quality mathematical data for RL training. Our dataset combines DeepScaler (Luo et al., 2025b; Gao al., 2024; Min al., 2024) and NuminaMath (Li et al., 2024), covering algebra, combinatory, number theory, and geometry. We apply 9-gram filtering to avoid contamination with common math benchmarks and implement filtering rules to exclude unsuitable data, such as questions involving multiple sub-questions, multiple-choice or true/false questions, overly long or complex answers, proof-based questions, non-English content, references to figures, excessively brief prompts.
Since NuminaMath data originates from online sources processed through OCR and parsing tools. it contains considerable noise due to incorrect questions or answers. To address this, we use the DeepSeck-RI model with up eight attempts per question, retaining only those that achieve correct majority-voted solutions via rule-based verifier. Questions that are consistently unsolvable by DeepSeek-RP RI often exhibit ambiguity or OCR-related errors upon human inspection and are therefore discarded. We further filter out questions requiring fewer than 2,000 R1 response tokens to answer, as we consider these questions to solvable without extensive reasoning, and downsample problems with responses of 2,000-4,000 tokens to balance the dataset based response length. Our final, rigorously verified dataset contains approximately 49,000 high-quality math problems for RL training.
3.2.2 Training Process
RL training can be computationally intensive when involving long CoT reasoning, with around 80% training time spent on generating model outputs. address this challenge, our RL pipeline focuses on enhancing reliability and efficiency through three primary strategics: 1) strict on-policy training to

---

maintain stable training and prevent entropy collapse, 2) stage-wise length extension from 8K to 32K tokens, and 3) curriculum training using inreasingly difficult prompts at later stages.
On-policy training to stabilize entropy loss. The entropy of the policy model serves as a key metric for assessing its ability to explore during RL training. In early experiments, we found applying multiple (2 or 4) gradient updates after model generation with a group of G rollouts per prompt led to rapid entropy collapse around 100 steps (see Figure 2c). In contrast, using exactly one gradient update after model generation, as in original DeepSeek-Mathl GRPO implementation (Shao et al., 2024), consistently prevented collapse. We therefore adopted this strict on-policy approach throughout RL training.
Length extension to accelerate training. Length extension has been shown be effective for smaller models (c.g., the 1.5B DeepScaler (Luo et al., 2025b)), but Wen et al. (2025) reported challenges in scaling larger models, as training an 8K response length initially led to degraded performance. In contrast, we were surprised to observe substantial performance improvements when extending training from 8K to 16K maximum response length. Based on this, we adopted a stage- wise length extension strategy (8K + 16K + 24K - 32K) to enable more efficient training, as directly starting from 16K or 24K resulted in suboptimal results (see Figure 2b).
Harder problems to push the model. We used curriculum learning by introducing more difficult prompts during the 24K and 32K stages. As the model mastered casier examples, their advantage reach 0 in the GRPO objective. We filtered prompts by model pass rate, filtering out those with pass rate > 6/1 16, which significantly improves model performance (Table 3).
Math RL improves code reasoning. In Table I, we show performing math RL not only improves math reasoning on AIME24/25 but also boosts Live- CodeBench v5 score to 44.4% (6.8% 1) for 7B and 58.9% (5.8% t) for 14B, which already outperforms the very recent code RL model DeepCoder-14D (57.9%) (Luo et al., 2025a). Furthermore, we show that math-only RL improves coding performance across all problem topics tasks (see Figure 9 in section A.8). This cross- generalization is a compelling advantage of reinforcement learning. In contrast, domain-specific supervised fine-tuning (SFT) often results in poor performance on other domains.
Table 1: Math-only RL improves code reasoning.
AIME24
AIME25
LCB v5
Models
avg@64
avg
avg88
DeepSeck-RI--istill-oeees
55.5
39.0
37.6
AceReason-7B (Math-only)
69.0
53.6
44.4 (681)
DeepSeck-R1 PrepSeck-RI-Disiiee R-Dissttll--OP Distill-Qwen-14E
69.7
50.2
53.1
AceReason- 14B (Math-only)
78.6
67.4
(581)
not just math-related coding
We used batch size of 1 288, sampling G = 8 responses per prompt for 8K length training and 16 responses otherwise. We adopted learning rate of 1 x 10-6 with AdamW (Kingma, 2014), and set both the entropy loss coefficient and KL loss coefficient 3 to O.
3.3
Code-only RL
3.3.1 Data Curation
We curated our code-only RL training dataset from modern competitive programming platforms using strict selection criteria to ensure high-quality coding problems. The dataset includes both function- calling and standard input/output (stdin/stdout) formats and covers a wide range of algorithmic topics, including graph theory, data structures, number theory, greedy algorithms, and more.
To ensure stability for RL training, we filtered out problems incompatible with standard output comparison (e.g., multi-solution or interactive problems requiring special judges) or those needing platform-specific templates, thereby minimizing potential false negative reward. Furthermore, we curated strong testcases covering tricky edge cases or extreme cases under input limitations, ensuring that incorrect solutions would fail and thereby eliminating potential false positive reward. As discussed in Appendix A.9, both false positive reward and false negative reward can obfuscate RL training by introducing noisy reward signals. To gauge difficulty, we evaluated each problem using DeepSeek-RP RR 16-71B with 8 rollouts, assigning difficulty score from 0 to 8. Problems where the model failed all attempts (level 8) were excluded. Finally, we performed careful benchmark decontamination and problem deduplication across platforms using n-gram context analysis and
5

---

original URL matching (see Appendix A.7 for details). After such aggressive filtering process, 8,520 problems remained, forming our final training set.
3.3.2 Training Process
We apply the two-stage code-only RL pipeline designed to accommodate models of varying scales. The pipeline leverages training sets composed of coding problems within specific difficulty ranges, along with customized settings for maximum response length and sampling temperature.
Stage I initiates the code RL. process, launching after prior math-only RL to ensure training stability,
In Stage 1, training data is constructed by diffieulty: problems with difficulty up level 5 are used for 7B model, while problems up to level 7 are used for 14B model. We set maximum response length as 24,000, temperature 0.6 and number of rollouts as 8 for Stage 1 training.
Stage 2 employs the full set of training problems with 32,768 maximum response length. In this stage, we implement an epoch-wiss wise filtering strategy by filtering out relatively easy problems w.r.t. prior epoch checkpoints and gradually increasing the sampling temperature from 0.6 to 1.0, number of rollouts from 8 to 16 across epochs. This aims to encourage policy convergence while encouraging exploration.
We set batch size to 128 and learning rate to 5 10-6 with Ad continuing training both stages until policy converges. For reward function, we adopt the strict rule- sile-hased reward: positive reward 1 granted if and only if the generated code successfully passes all test cases for the given problem. As for efficient evaluation, we deploy parallelized local verifier to check testcase correctness.
3.4 Summary of Training Curriculum
We use CrepSeek-RI-Distinee RI---iiiill----e2.5--7777 and 14B as our initial SFT models. To integrate math-only and code-only RL, we first perform math-only RL with stage-wise length extension from 8K 24K. Next, we apply code-only RL, extending the length from 24K to 32K. Finally, we conduct math-only RL at 32K. We find that this training curriculum slightly more effective and efficient in practice than first performing math-only RL from 8K to 32K, followed by code-only RI. from 24K to 32K.
4 Evaluation
4.1
Experimental Setup
Our experiments start from strong SFT models, DeepSeek-RR RI-Distill- based on the Qwen2.5 model family (Yang et al., 2024a) and distilled from DeepSeek-RI R1 (Guo et al., 2025). To ensure consistency and reproducibility, we follow the DeepSeek-F R evaluation protocol, using a temperature of 0.6, top-p of 0.95, and maximum output length of 32,768 tokens.
and 14B, which are
4.1.1 Math Evaluation
We use diverse math competition benchmarks, including AIME2024, AIME2025, MATH500 (Hendrycks al., 2021), addition with HMMT2025 Feb and BRUMO2025 from AathArena (Balunović et al., 2025). Due to the high variance in outputs from reasoning models when using sampling, we report pass@1 performance averaged over generations (avg@k). For small-scale benchmarks such as AIME, we use k = 64, following DeepSeek-R1. This choice of k is critical for obtaining a reliable evaluation, as lower values of & lead to a significantly higher standard error of the mean (e.g., AIME2024 @ 16 16/32//44 1.8/1.2/0.7).
To isolate the effects of pre-training, we primarily compare with reasoning models based on ei- ther Qwen2.5 or Llama-3.1 at similar parameter scales. These include SFT models (distilled from much larger frontier models) such Light-R1-7B (Wen et al., 2025), OpenMathhResoonngg (Bercovich et al., 2025),
7/14/32B (Moshkov et al., 2025), and LLaMA-Nemotron- wwwpoottooonnnnnSSuerrrr///eee as well as RL models like AReal-boba-RL-77 (RL Lab, 2025), Wwwwork-ORI-Mathie
(He et al.,
2025), and Light-R1- such as DeepSeek-RI (Guo et al., 2025), QwQ-32B (Qwen, 2025b), -ewor---ettoo-UUUIIi 253B (Bercovich et al., 2025), and 03-mini (OpenAI, 2024).
(Wen et al., 2025). For context, we also include frontier reasoning models

---

Table 2: Math and Code reasoning evaluation. We report pass@1 averaged over gener- ations (avg@k) following the DeepSeek-RI evaluation framework (template, temperature=0.66 top pop__==095,, max response length=32,7688 By default, we report self-reported numbers from model developers if they are available. Otherwise, twe evaluate the model using the same evaluation setting, or +wee collected from MathArena or LiveCodeBench leaderboard.
AIME
MATH
HMMT
BRUMO
LiveCodeBench v6 avg
Codeforees Percentile pasel
Eval
arg061
mgg
avg064
mgg88
QwQ-328
79.5
65.8
96.0
47.55
63.4
1982
97.7
DeepSeck-R R1-671B
79.8
70.01
97.3
41.71
80.8
65.9
2029
98.1
80.8
72.5
97.0
66.3
63-mini (low)
60.0
95.8
28.37
66.7
60.97
1918
03-mini (medium)
79.6
76.7
97.3
53.3F
80.0
67.4
2036
98.1
ARcal-boha RL-7B
61.9
48.3
93.8
29,4'
58.9
34.3
Skywork-ORl- -BB-
69.8
52.3
94,47
31,41
60.6
43.6
OlympicCxier-78
40.7
37,1
79.8
Light-R1-78
27.6'
52.8
40.6
Light-Ri-4B R1-14B
74.0
60.2
94.6'
37.8
67.1
57.91
51.5
DeepCoder-140 (32K Inforence)
71.0
85.31
OpenMath OpenMath-Nemooo--2 Ne
74.8
61.2
OpcnMath Nemotron- 14B
76.3
63.0
OpenMatl--Name Nemotnon-328
76.5
62.5
OpenCaaleeesannnnn Nemotron-7B
51.3
46.1"
83.4
www.a-oeeesnninnNeeeeeeee Nemettron-oBbb
59.4
Llamz. Nemotrco Naso 8B.v1
613
47.1
95.4
46.6
46.2'
$1.21
Llama Nammottennttre-SSSe Super-1988
DeepSeck RI ST--istillQunen--r
58.5
39.0"
92.8
26.3'
51,21
37.6
34,1"
1189
57.4
80.4
DeepSeek-RR
Owen
DeepSeck- Di--siiil-QQen------
72.6
54,91
94.3
33.3
68.31
57.2
1691
93.2
DeepSeck-R1 D-stiill--L Auma-708
55.0f
33.37
BeaRessoo-Nemotree
69.0
53.6
94.1
33.9
62.2
51.8
45.1
1475
84.8
84.6
AceResson Nemootron
78.6
67.4
46.4
72.3
61.1
54,9
2024
98.1
85.7
4.1.2 Code Evaluation
For coding tasks, we evaluate AceReason-Nemotnee models on LiveCode (LCB) (Jain et al., 2024) v5 (20240801 - 20250201) and v6 (20250201 - 20250501) subsets, containing recently released AtCoder, LeetCode problems. We also report Codeforces ELO and percentile number of our models based on LiveCodeBench Pro dataset (Zheng et al., 2025), which contains Codeforces problems from 202407 to 202412. We also include evaluations on EvalPlus (Liu et al., 2024c, 2023).
We compare our model with state-of-the-art open-sourced code-gen LLMs of similar parameter scales, including OlympicCoder-7B (Face, 2025), 1a-ma--.--Neeotroooe CooomottonnNano-gBBB.. -a-v- (Bercovich et al., 2025), opsnnodeRRassoningg-77 7B/14B (Ahmad et al., 2025), DeepCoder- 14B (Luo et al., 2025a). For further context, also include strong frontier reasoning models as titled above.
4.2 Main Results
From the evaluation results Table 2, we summarize the key conclusions as follows:
models show that using RL significantly improves over the initial SFT models (DeepSeek-R) -I-istill-- Qwen-7B/14B) both math and coding tasks. Specifically, for math tasks, our AceReason- Nemotron- achieves remarkable improvements over SFT model by increasing 14.5% accuracy on AIME 2024, and 14.6% accuracy on AIME 2025. For coding tasks, it achieves 14.2% and 8% accuracy improvements over DeepSeek-RI- RI--ittill Qwwatill-ween-BBBB on LiveCodeBench v5 and v6, respec- tively. Meanwhile, on AIME24/25, and 53. 13..% 1 47.9% to 61.1% % on LiveCodeBench v5/v6 from initial SFT model DeepSeek-RI Riikk--R----tl-l----- Qwelll--eene-tte 14B, even surpassing significantly larger SFT models such as DeepSeek- Biieeckk----istlllll RRI-Distil--ue
RL significantly improves reasoning capabilities. Our
14B improves from 69.7%/50 550.2% to 6%/67.7777
Seatill--wen--222 and DeepSeek-RR Biieeeck--R---iiieee -i-istill---lmma 70B. Additional results on 1.5B and 32B further validate the same improvements shown in Appendix Table ??.
AcceeesonnNemottee vs. SOTA RL-based reasoning models. While comparing with of the-art open RL-based reasoning models under the same parameter scale, DeceReason-Nemorre model still remains its superiority. In math reasoning domain, provides competitive results while comparing with strong RL-bbased models (Skywork-OPP MInik-ORIMMA -Math, Light-R1, etc.), while CeRReason-Nemooronn 14B provides the best-in-class results. In code
model
7

---

AIME 2024
LiveCodeBench vS
Aceltesson-78 AccReason 148
35-
Accurary (49g66))
70
Accuracy (avg@a)
Deeponed O3 t
60
66-
50-
Stage (1663
Mesh-B
64-
62-
x5
www
60
58
Ki-Skoi
40-
54
AccReason 78 tuvo Math RL)
AceReeson 140
500
1000
1500
2000
2500
S0on
3500
117
195 234 273 312
Training Steps
Training Steps
Figure 1: Model accuracy AIME2024 and LiveCodeBench v5 during RL.
generation domain, AceReason-Nemotron models outperform open-sourced reasoning models with similar parameter scale. Its Math-Code ability remains competitive even comparing with frontier reasoning models, such as QWQ-32B, 03-mini, etc.
NeeeasoonNemotro vs. SOTA models through distillation. AceReason- 14B shows better math and code performance than the latest SOTA specialized distilled model such as DpenMath-14B/322 by +2. 12/4%% on AIME24/25, OpenCodeReusooing. 14B by +1.7%/+0..89 on LiveCodeBench v5/v6, demonstrating RL leads to higher upper-bound of model performance than distillation. In contrast, at 7B model scale, AceRcason-Nemotrrn 7B performs competi- tively with OpenCodeReassoinng-T2 advantage than RL-trained Nemotron models significantly outperform the general- purpose reasoning model Llama-Nemotron- 8B/49B, which is trained via distillation. This indicates that the effectiveness of distillation versus RL still depends model size and task domain, though RL offers the potential for significantly higher accuracy at the 14B scale and beyond.
coding tasks, while OpenMath-7B appears have bigger on math reasoning. In all cases, AceReason-
4.3 Analysis
Response length grows alongside performance. Figure (left subfigure) and Figure 2a show the evolution of response length on the training set and corresponding AIME24 accuracy (AIME25 in appendix Figure throughout the RL training of AceRRaaoon-Nemotron-T7 model. We analyze two distinct stages in the 8K - 16K length-extension training strategy: 1) Stage (8K) Transition from imitation to RL: During this stage, the model learns compress its reasoning process adapt an 8K token limit, causing an initial drop in
capabilities compared to bascline. However, after approximately 1K-2K RL steps, the model gradually recovers its performance: 2) Stage (16K)
Accelerated learning: Upon extending the token limit to 16K, the model immediately exploits the extra reasoning tokens capacity: within 500 RL steps, its average response length increases from 4K to around 6.5K tokens. At the same time, AIME24 accuracy improves sharply from 55% to 62%, after which both response length and accuracy plateau. Extending the maximum response length further to 24K and 32K demonstrates a similar trend.
Hard prompts drive the largest gains. the 24K response length RL stage, we build [Easy, Medium, HardJ-prompt sets based on difficulty estimated by 7B model's performance over 16 attempts. Easy prompt set includes prompts except those solved more than 14 times, Medium prompt set excludes those solved more than 10, and Hard excludes those solved more than 6. shown Table 3, our abla- tion confirms that training with Hard prompts yields a 2.6% improvement on the AIME24 benchmark com- pared to fullset data and outperforms using Easy/Medium prompts, although it only has 2.2K prompts.
Table 3: Prompt difficulty and its impact on Stage (24K) training.
AIME24
AIME25 avg @ 64
Data
avg @ 64
Starting Checkpoint
62.2
50.2
Full (no filtering, #19K)
$1.1
Easy (410K)
64.4
50.8
Medium (K4.6K)
65.3
51.9
Hard (#2.2K)
65.9
52.5

---

Length Exteen vian Contlips: 9K-16KK-24K
12:993
0.30
Resporse LongteLLnhhhh tokeend)
AINEZOEA Accurny (auyBri))
62
24k
Ixing hard prodlems 24K
0.20
6901
training aanl
4303
inttol decine tradding
0.10
Taining Method:
flp-tigher
Dn-poliy
0.6
12
150 Thaining Steps
Thaining Steps
(a)
(b)
(c)
Figure 2: Analysis of RL training: (a) response length during math-RL training, (b) GPU hours for different length extension configurations, and (c) entropy of output logits using on-policy RL training versus off-policy training with Clip-Higher trick.
Starting from 8K improves later performance. Although training initially at 8K max response length offers faster training per step than starting at 16K 24K, the early decline in AIME24 accuracy raises questions about its necessity for 7B-sized models. Figure 2b presents an ablation study comparing three strategies: 1K-16KK-2447 16K -6KK, and directly 24K. Each strategy utilizes identical full data, with accuracy measured on AIME24 alongside GPU hours consumed. The results indicate that while the 16K strategy initially outperforms the baseline, the 8K~166K strategy quickly catches up during the 16K stage and consistently outperforms other strategies thereafter. Conversely, starting directly 24K results in minimal performance gains and significantly longer training times, suggesting limited effectiveness during RL training for 7B model.
Interplay of math-only RL and code-only RL, In Figure (right subfigure), we observe that initializing code RL training from math-RL checkpoints offers a substantially better starting point and results in significantly higher final accuracy on LCB. We investigate the effects of sequential Math RL and Code RL training stages on developing strong reasoning model in both math and code reasoning domains. Starting from DeepSeek- RI L--Distill-Qwerr -all---wwnnBBBil 7B/14B as our initial models, we first apply Math-RL and then Code-RL, evaluating performance on standard Math (AIME 24/25) and Code (LCB v5/v6) benchmarks. Surprisingly. as shown in Table 4, initial Math-RL RL training not only significantly improves accuracy on math benchmarks, but also improves model performance on coding. Subsequent Code-RI. training further boosts coding scores, with negligible degradation on math performance. This shows that the proposed sequential training strategy is effective developing models with strong reasoning abilities across math and coding domains.
Does RL improve p or pass@1 I over the distilled model? Recent stud- ies (Shao et al.. 2024; Yue et al., 2025) sug- gest that RL primarily improves pass@ ac- curay over SFT model (e.g., DeepSeek-R1- Distilled) without significantly impacting pass@k: k. However, Figure 3 demonstrates that RL consistently improves pass k score (from & 8 to k 64) for both 7B and 14B models on LiveCodeBench v5 and v6. with an improved pass @k scores maintaining a 10% margin from pass 8 to 64. On AIME24/25, we found the 7B and 14B model also show better pass @k compared to the SFT model across all k. For both AIME and BiveCodeBench, we generated 128 responses for each question and randomly sampled k to calculate the pass @ result with an average of 100 runs to reduce variance.
Table 4: Interplay of math-only RL and code-only RL.
AIM
LiveCeeeeBench
Midela
2024
2025
v5 @v
v6
8
2v
DequSeck R3 Diaill Qwee-78
55.5
390
37.6
34.1
24K1
(10-41)
(13
46415311
(5.5))
(1,015
DeepSeck Diaill Rl (24K
75.7 63.9 00.511
61.112.511
SE9
To further validate our conclusion, we extend pass@k evaluation from k= 64 to 1024 on Live- CodeBench v6, where correct answers are difficult to "guess" through limited sampling. For each question, we generated 1,024 responses to compute @ 1024. For smaller values of k (k 1024), we randomly sampled responses from the 1,024 and calculated pass@k by averaging over 100 runs. In Figure 5, we observe that our AceReason-Nemotton: consistently outperforms

---

A.M22024
Lisczeteno10
UwdDaddterch +t
PacceR 2
035
00.4
the candidates
candidates
AcoPessen--eemstton--e
AceReason Nemotron 78
DelpSnakkR--itill--
Figure 3:
The Pass @ of RL (AceReason) and SFT (DeepSeck-RI-Dins models on AIME 2024/2025 and LiveCodeBench v5/v6.
LiveCodeBench v5
LiveCodeBench v6
200
Coepreek-A
After mppresmentt After Degiatation
Accuracy
60-
20-
Prablem Instances
Prablem Instances
AIME2024
AIME2025
Accuraty
60
Ocopsook R1 Ditil 72
After RL
Problem
Problam
Figure 4: Problem-level solving rates comparison between distilled model and after RL training. Accuracy for each problem is calculated average of 64 attempts.
the SFT model (DeepSeek- TrisSeck-RI-Disttee
by approximately 10% across all pass @ k values. Pass @ K=1024 on AIEM25 shows a similar gap in Table 8.
Where does RL improve the distilled model? Fig- ure 4 compares the problem-level accuracies of the initial 7B SFT model with on LCB v5/v6 and AIME 2024/2025, Results for the 14B model are shown Appendix Figure On LCB, we ob- serve RL unlocks long tail of hard coding problems that the distilled model fails to solve in 64 attempts, adding 30 and 23 additional solvable problems to LCB v5 and v6. also significantly improves on challenging problems where the SFT model has lower than 20% accuracy. On AIME, for the most challenging problems with zero solve rate, RL enables the model to solve more problems AIME24, Hence, we find RL not only improves the accu- racy problems with high solve-rate but also extends the boundary to solve hard problems that the SFT model was initially unable to solve.
UuecCoeeeench vz
Ierecsson-eemotron--
after RL
Acilossion www.saow-----------... 70
Partgic Gap
Figure The Pass RL (AceReason) SFT model (DeepSeck Ri-es--lllisdlledd on LCB v6.
5 Conclusion
We demonstrate that large-scale RL can substantially enhance the reasoning capabilitics of strong, small- and mid-sized distilled models. We propose performing RL math-only prompts first, followed by code-only prompts. Notably, math-only RL significantly boosts performance not only math benchmarks but also code reasoning tasks. Crucially, subsequent code-only RL further improves code benchmark performance with minimal to no degradation in math results. To support this process, develop robust data curation pipeline that collects challenging prompts with high-quality, verifiable answers and test cases, enabling verification-bar based RL across both domains.
10

---

References
Ahmad, W. U. Narenthiran, S.. Majumdar, S., Ficek. A., Jain, S., Huang, J., Noroozi. V., and Ginsburg. B. Opencodercasoning: Advancing data distillation for competitive coding. arXiv preprint arXiv25544009943 2025.
Azzolimi, A.. Brandon, H., Chattopadhyay, Chen, H., Chu. J., Cui, Y. Diamond, J., Ding. Y., Ferroni, F., Govindaraju, R., et al. resmon-reasonl: From physical common sense to embodied reasoning. arXiv preprint arXiv:25003. /5555, 2025.
Balunovié, M., Dekoninck, J. Petrov, I. Jovanovié, N., and Vechev, M. Matharena: Evaluating Ilms on uncontaminated math competitions, February 2025, URL
ai/.
Bercovich, A., Levy. L., Golan, I., Dabbah, M., Elnia,iv, R., Puny, O., Galil, 1,, Moshe, Z., Ronen, T., Nabwani, N., ct al. :lmooronmooooo: Efficient Reasoning Models. arXiv preprint arXiv:2505.00000000 2025.
Chen, H., Zheng. K., Zhang. Q. Cui, G., Cui, Y., Ye, H., Lin, T.-.., Liu, M.-Y., Zhu, J., and Wang. H. Bridging supervised learning and reinforcement learning in math reasoning. arXiv preprint, 2025.
Chen, M., Tworek, J., Jun, H., Yuan, Q.. Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., ct al. Evaluating large language models trained on code. arXiv preprint 1idi::2117.03331. 2021.
Cobbe, K., Kosaraju, V., Bavarian. M., Chen, M., Jun, H., Kaiser, L., Plappert, M. Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
Face, H. Open rl: A fully open reproduction of deepseek-rl, January 2025. httptt:: huggingface/ open-ri..
com/
Gao. B., Song, F., Yang, Z.. Cai, Z., Miao, Y., Dong, Q., Li. L., Ma. C., Chen, L.. Xu, R., et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985, 2024.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. Ilama herd of models. arXiv preprint arXiv:2407.21783 2024.
Guo, D., Zhu, Q. Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li. Y., al. Deepseek- coder: When the large language model meets programming the rise of code intelligence, arXiv preprint arXiv:2401.1411 2024.
Guo, D., Yang, D.. Zhang, H. Song, J., Zhang, R., R., Zhu, Q., Ma. S., Wang. P., X., et al. Deepseek-R:: Incentivizing reasoning capability in LLMs via reinforcement learning, arXiv preprint arXiv:2501.12945 2025.
He. J., Liu, J., Liu, C. Y., Yan, R., Wang, C., Cheng, P., Zhang, X., Zhang, F. Xu, J., Shen, W., Li, S., Zeng. L., Wei. T., Cheng, C., An, B., Liu, Y., and Zhou, Y. Skywork open reasoner series, 2025. Notion Blog.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
Jain, N., Han, K., Gu, A.. Li, W.-D., Yan, F., Zhang. T., Wang. S., Solar-Lezama, A., Sen, K., and Stoica. L. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint MrXiv:2403007974 2024.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.69990 2014.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H. Gonzalez, E., Zhang. H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Principles, 2023.
Li, J., Beeching, F.,, Tunstall, I.., Lipkin, B., Soletskyi, R., Huang, S. C., Rasul, K., Yu, L.,, Jiang, A., Shen, Z. Qin. Z., Dong, B., Zhou, L.. Fleureau, Y.. Lample, G., and Polu, S. Numina- math. lhttps: : / http////hggggggggeeeeeeee ///IIMMO /MwminaMattt-ooo (https: htapp::///itie /tttttttthb./////// com/ project-numined http:/aaioooororrreeeeeeee 2024.
www.bo/aaiiiiieeee http://///////////o///tttttttttd dataset.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, L, and Cobbe, K. Let's verify step by step. In The Twelfth International Conference on Learning Representations. 2023.

---

Lin, B. Y. Bras. R. L.,, Richardson, K., Sabharwal. A., Poovendran, R., Clark, P.. and Choi, Y. Zebralogic: On the scaling limits of Ilms for logical reasoning. arXiv preprint arXiv:2502.01100, 2025.
Liu, A., Feng, B., Xue, B., Wang. B., Wu, Lu, C.,, Zhao, C., Deng, C., Zhang. C., Ruan, et Deepseek- V3 technical report. arXiv preprint arXiv:2412. 19437. 2024a.
Liu, C. Y., Zeng, L., Liu, Yan, R., He, J., Wang. C., Yan, S., Liu, Y., and Zhou, Y. Skywork-reward: Bag of tricks for reward modeling in lls.. arXiv preprint arXiv:2410. 18451, 2024b.
Liu, J., Xia, C. S., Wang, Y., and Zhang. L. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: wwppneeeeeeeeeeeee net/ forum? i d- qvx OCu7.
Liu, J. Xie, S., Wang, J. Wei, Y.. Ding, Y., and Zhang, L. Evaluating language models for efficient code generation. In First Conference Language Modeling, 2024c. URL btttps://openrevilie net/ forum? 2ini IBCBMeAhCC
Liu, Chen, Y., Shoeybi, M., Catanzaro, B., and Ping. W. AceMath: Advancing frontier math reasoning with post-training and reward modeling. arXiv preprint arXiv:2412.15084 2024d.
Luo, M., Tan, S., Huang, R., Shi, X., Xin, R., Cai, C.,, Patel, A., Ariyak, A., Wu, Q. Zhang, C., Li, L. E., Popa,
R. A., and Stoica, I. Deepooder: A fully open-source 14b coder at 03-mini level, 2025a. Notion Blog.
Luo, M., Tan, S., Wong, J., Shi, X., Tang, W. Y., Roongta, M., Cai, C., Luo, J., Li, L. E., Popa, R. A., and Stoica, I. DeepScaleR: Surpassing 01 O1-Preview with 1.5B Model by Scaling RI, 2025h. Notion Blog.
Min, Y., Chen, Z., Jiang, J., Chen, J., Deng, J., Hu, Y., Tang, Y., Wang, J., Cheng, X., Song, H., Zhao, W. X., Liu, Z., Wang. Z., and Wen, J.-R. Imitate, explore, and self-improve: A reproduction report slow-thinking reasoning systems, arXiv preprint arXiv:2412.09411 2024.
Moshkov, L., Hanley, D., Sorokin, Toshniwal. S., Henkel, C., Schifferer, B., Du, W., Gitman, Aimo-2 winning solution: Building state-of-the the-art art mathematical reasoning models with openmathreaoning dataset. arXiv preprint arXiv:2504. 16891, 2025.
OpenAl. Learning to reason with LLMs, 2024.
Qwen, T. Qwen3, April 2025a. URL https: /qwnelm.mmmmmmmmmmmm/..mmmmmmmm. githut.thh..ooo wtt.lo////bo//////////ooo
Qwen, T. Qwq-32b: Embracing the power of reinforeement lcarning, March 2025b. URL htttt:////wenn github. 1c/blog///// hooloo//ao hwogg//ww/////0 32b/.
Rein, D., Hou, B. I.., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference Language Modeling, 2024.
RL Lab, A. R. Areal: Ant reasoning https:
2025.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimoy, O. Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.0634 2017.
Shao, Z.. Wang, P. Zhu, Q. Xu. R., Song. J., Bi. X., Zhang, H., Zhang, M., Li, Y., Wu. Y., et DeepseekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. 2024.
Sheng. G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., and Wu, C. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F, Chi, E., Lc, Q V., Zhou, D., et al. Chain-of-thouge prompting elicits reasoning large language models, Advances in neural information processing systems, 35: 24824-24837, 2022.
Wen, I.,, Cai, Y., Xiao, F., He, X., An, Q. Duan, Z., Du, Y., Liu, J., Tang, L.,, Lv, X., et al. Light-RI: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond. arXiv preprint arXiv:2503. 10460, 2025.
Xie, C., Huang, Y., Zhang, C., Yu, D., Chen, X., Lin, B. Y., Li, B., Ghazi, B., and Kumar, R. On memorization of large language models in logical reasoning. arXiv preprint arXiv:2410.2312. 2024.
Yang. A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2.5 technical report. arXiv preprint 4rXiv:2412.15115 2024a.
12

---

Yang. A.. Zhang, B., Hui, B., Gao, B., Yu. B., Li, C., Liu, D., Tu, J. Zhou, J., Lin, J., et al. Qwen2.5-Math technical report: Toward mathematical expert model via sell-improvement. arXiv preprint 1rXiv:2409.12122. 2024b.
Yu. Q. Zhang. Z., Zhu, R., Yuan, Y., Zao, X., Yue, Y., Fan, T., Liu, G., Liu, L., Liu, X., et al. DAPO: An open-source LLM reinforcement learning system scale. arXiv preprint arXiv:2503. 14476, 2025.
Yue, Y. Chen, Z., Lu, R., Zhao, A., Wang, Z., Song, S., and Huang, G. Does reinforcement learning really incentivize reasoning capacity llms beyond the base model? arXiv preprint arXiv:2504. 138177 2025.
Zeng, H., Jiang, D.. Wang, H., Nie, P.. Chen, X., and Chen, W. Acecoder: Acing coder via automated test-case synthesis. arXiv preprint 011101 2025.
Zheng, Z.,, Cheng, Z., Shen, Z., Zhou, S., Liu, K., Wei, S., He, H., Li, D., Hao, H., Yao, J., Sheng. P., Wang. Z.,, Chai, W., Henderson. Korolova, A.. Viswanath, P., Xie. S., and Shang, Livecodebench pro: How olympiad medalists view 11ms in competitive programming?, 2025, URL https: /eeeep//eeeeeedddeeeeppoooo.... com/. Available at LivcCodeBench Pro website.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
we have comprehensive experimental results in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section A.1 in appendix include a limitation section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all the details of the data collection for training and evaluations, which can be found in section 4.1 and appendix A.7. We open-source model weights and data for reproducing our results at https: htt.///ggggggaccceee/e/oo coocoloectionsssec nvi aviiiaa a aaliaaaaceeessss
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: we open-source model weights, training data and use open-source code. Training data are public available data with open access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: /tt//nnsssc.m cc/
public/ gui quides/ Soddes/Cooe CoodeSuomii ssi psionnooicyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips-. cc/ /tppppulcc//oc//pp weellcc/giidee des CooeeeoeeeSeeeeeeee Sudmuubisss ss ciiooooooiii oolicyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: the experiment settings are presented in section for training data and training method, and evaluation set up are detailed in 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: we report results using an average of k generations with k up to 64 for datasets such as AIME, We additionally report boxplot to show the improvements of the model over the baseline in appendeix Figure 7.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section and Figure 2b provides the compute resources used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /neurips hwtppublic/tthccsuuiio..o
Answer:
[Yes]
Justification:
We have read the code of ethics and conform with it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section A.1 discuss the broader impact of this work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Dataset sourced from open-source dataset and the model is trained for math and code reasoning instead of other capabilities which might involve high risk misuse issue.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Appendix A.3 and A.7 describe the assets used in this paper with license.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw. soerssw thooeee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: will open-source model weights. Dataset open-sourced with public access.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Does not use crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: no crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
This is original research work.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //nuuriipssp cc/ Conferences / 2025/ LLM) for what should should not described.
20

---

A Technical Appendices and Supplementary Material
A.1 Broader Impact and Limitations
This work primarily focuses on enhancing the reasoning capabilities of language models in mathe- matics and coding tasks. As such, it poses minimal negative societal impact in terms of privacy risks or bias concerns. Our study targets medium-scale models (7B/14B) due to the cost of RL training and extensive ablations; we additionally report 1.5B and 32B results in the appendix (Table 9), but do not evaluate models below 1B or above 100B parameters. We intentionally initialize from DeepSeek- -a---R--iii disitsille checkpoints to isolate the effect of the SFT-RL stage; applying RL directly to base pre- wre-trained models (e.g., R1-Zero-style We focus on verifiable domains (math and coding) to enable rigorous evaluation; assessing transfer to open-ended tasks is left for future work. Finally, our conclusions are primarily empirical rather than theoretical
is a distinct setting and out of scope in this study.
A.2 Funding Transparency Statement
1. Funding (financial activities supporting the submitted work): Funding in direct support of this work is from company NVIDIA;
2. Competing Interests (financial activities outside the submitted work): No additional revenues related to this work.
A.3 Dataset
Table 5: Evaluation Dataset
Dataset
#
License
Link
MATI1500 (Lightman al., 202.)
500
Huggi BuggingFocas Sttp///ooo////TTSSS0000 /MATH-500
AIME2024
30
co-by-nc-sa-
4..0
g thub. www///oo///it SOA.T/////eee//// 5-Math
AIME2025
30
oo-by-ne-sa-4.00
MathArena/ aa///mmmm_222222
HMMT Feb
30
co-by-ne-sa-4.0
BRUMO2025
30
cc by me 4.0
Mechaarent/brrmm_2222
LiveCadeRench v5 (Jain al., 2024)
279
MIT
i
LiveCodeBench v6 (Jain et al., 2024)
175
MIT
Table 6: Training Dataset
Dataset
License
Link
NuminaMath (Li et al., 2024)
Apache-2.0 MIT
Al 1 AL--8//buminaaath-1...
DeepScaler (Iuo et al., 2025h)
ArCoder Problems (before 20240731)
oc-by-4.0
htttps://atcodrr.....
Aizu Problems
ec by 4.0
LeetCode Prohlems
ec-by-4.0
https:// ptrpss//leette httpp::/leetooodeec
A.4 Additional Experiments on General Reasoning
In addition to the math and code evaluation, we evaluate the models STEM and logical reasoning benchmarks such as GPQA (Rein et al., 2024), Knight&Knave (Xie et al., 2024), and Zebra-Logic (Lin et al., 2025). In Table 7, we show that after RL, Knave with large margin, achieving 96.1% 1% with the 14B model. This shows
model improves on all datasets,
especially on Knight& Knave the RL reasoning capability learned during math and code training generalizes other domains.
A.5 Additional Experiments on 1.5B and 32B models
In Table 9, we run the same AceReason- EeeReason-Nemtron RL recipe on a smaller and larger distilled checkpoint SeopSeee--R--issillle 5/33B.)/32B. We show that our training recipe generalizes to 1.5B model and 32B model, achieving significant improvements on both math and code benchmarks. Due to computational constraints, we only conduct Math-only RI. for the 32B model. However, we found the coding accuracy on LCB v5 significantly improves to 63.4 without any code RL. training, surpassing Skywork-ORI ----3BB (63), which has done code RL.
21

---

Table 7: Benchmark accuracy (%) on GPQA, Knight Knave, and Zebra- Logic.
Model
GPQA
Knight & Knave
Zebra-Logic
DeepSeek- Baieck-RR--iitiii 1--sstill--BB
49.1
35.2
24.5
7
51.1
56.4
39.6
DeepSeck-R 1-Distill-1444
59.1
83.8
60.1
AceReason AccReusoo--emmotron
61.2
96.1
72.7
Table 8: AIME 2025 pass @ (%).
Model
8
32
64
128
256
512
1024
DeepSeek-RR -isttilled Qwen-7B
62.2
66.8
71.3
76,5
81.2
86.2
90.3
92.6
AccReason-Nemmotron-T7
73.4
76.0
78.3
85,4
89.0
93.0
94.8
96.1
A.6 Additional Math-RL, Analysis
In Figure 6, show the accuracy AIME2025 during math RL training. We found the stage (16K) training leads to a large performance improvement within 300 steps. In Figure 7, we plot boxplots of AIME2024 and AIME2025 accuracy for the 7B and 14B AceReason model comparing to DeepSeck-R1--istiil--ee
We can observe the accuracy of AceReason over 64 generations at 25% and 75% percentile and median, significantly improve Figure 8, we inspect the problem-level solve rate for AceReason- 14B on AIME2024 and AIME2025. On AIME24 and AIME25, the AceReason model solves one additional problem. We also observe large gains on problems with higher initial accuracy, showing the benefits of RL training.
the distilled model. Finally, in
AIME 2025
AceRcason 78
AceReason-14B
65
59
Stage (16K): adceleration
53
Despseek-R1 Dittel
47
41
1000
Training Steps
Figure 6: Model accuracy on AIME2025.
A.7 Code-RL Dataset Curation Details
We collect our Code-RL training data from various modern competitive programming platforms, such as AICoder, LeetCode, Aizu, etc., with public strong test cases, while most open-sourced coding datasets (e.g., TACO, APPs) suffer from noisy problem statements, self self-contamination, and weak LLM synthetic test cases that are unreliable for RL training. To ensure data quality, we performed very strict filtering rules, by filtering out 1) Multi-solution or interactive problems that requires special judge or other external tools; 2) Problems where images within the statement obscure a clear understanding; 3) Problems containing incorrect test cases or those lacking golden solutions; 4) Problems with weak test cases that allow incorrect solutions to pass tests. Furthermore, to prevent self-contaminatie within our collected problem set, conduct strict problem statement and source
22

---

Table 9: Additional results of 1.5B and 32B model on AIME24, AIME25, and LiveCodeBench v5.
Model
AIME24
AIME25
LiveCodeBench v5
DeepSeek- Berreck-----iiiee Pristilled-Qwen-te AceReason-Nemotron- 1.5B
28.9
22.7
16.9 27.2 (10.37)
47.7 (18.81)
35.5 (12.87)
DeepSeek- ---isstiledd--eeee
72.6
54.9
57.2 63.4 (6.21)
AceReason -eeootoonmtt2211 (math-only RL)
79.0 (6.47)
70.6 (15.71)
AceReason-78
80
DeepSeek-RI-Distilee
70
Accuracy
60
Accuracy
60
50
40
40
AceReason-144
30
DeepSeek-RI D--isii-----iitl--1-----
AIME 2024
AIME 20:25
AIME 2024
AIME 2025
Dataset
Dataset
(a) 7B
(b) 14B
Figure 7: Boxplot of AceReason vs Deepseck-RI-Dee on AIME24/25 over generations.
URL matching. avoid any potential contamination of our test set, we exclude all problems released after 20240801, and apply n--rrmm matching (with n = 14) to our testing set problem statement.
To prepare for subsequent curriculum training for Code-RL, estimate the relative difficulty of the collected problems. We deploy the local DeepSeek-RI 11-771 model, allow it to generate 8 attempts each problem, and assign corresponding difficulty score ranging from 0 to 8. Problems that the DeepSeek-RI RI-R11111 model failed to solve in all 8 attempts are excluded from the training set. Following this aggressive filtering process, 8,520 problems remain, forming our final Code-RL training set.
A.8 Topic-wise Accuracy Analysis on LiveCodeBench v5
Given the observation that both Math-RL and Code-RL enhance code generation ability on coding benchmarks, we are interested to see in detail how these two RL stages improve accuracy on topic- specific coding problems. Specifically, this ablation aims identify which problem topics benefit the most from Math-RL and the subsequent Code-RL. Motivated by this, we conducted ablation studies on LiveCodeBench v5 dataset, which consists of coding problems from AtCoder and LeetCode platforms. While LeetCode problems come with numan-annotaii DFS), there is no tag on Atcoder problems. To address this, we first extract a of problem tags from LeetCode. Then, for each AtCoder problem, we query the 04-mini-hin model to infer candidate tags given the problem statement and the set of topics. Furthermore, we group LiveCodeBench v5 problems by their assigned topics and evaluate model performance (avg @8 accuracy) for each topic group.
topic tags (e.g.. Greedy, Math,
We compare the performance of our initial SFT models, corresponding CekeasonnNemooroon-777 that incorporate both Math-RL and Code-RL. As shown in Figure 9, we plot the accuracy for each topic before and after Math-RL and Code-RL. The figure shows that applying math-only RL enhances model performance across all coding problem topics, with especially strong gains in algorithmic domains and math-related areas such as Math, Counting, and Combinatorics combinatories-ddmainn that rely heavily on mathematical concepts and general reasoning abilities. Furthermore, for topics like Simulation, String, and Graph, which rely more heavily on coding implementation and data structure skills, Code-RI. Ieads to significant further improvement.
against after applying Math-RL stage only and final models
RI-Diisttll-Qeerte
23

---

AIME2024
AIME2025
100
100
80
80
Accuracy
Accuracy
60
60
20
Improvements
20
After AL Degracations
Problem ID
Problem
Figure 8: Comparison of problem-solving rates after RL training.
Topis-wie acouracy on LiveCodeBendvv5
Toplc ise accuracy on BivecedBencch
Rash Table
Binary Search
Sorting
Depth-First
Theary
Number Theary
Binary Search
Counting
Graph
Simulation
Math
Math
Sorting
Enumeration
Greedy
Greedy
Hash Table
Combinatoris
Heap (Prionity Queae)
String
String
jPrinnty Queuel
Counting
Dynamic Programming
Gemice to Du
Combiratoric
Bepoock Ded
Craph
Mer Calokk herevessne
Dynamic Programming
Accurscy
Accuracy
Figure 9: RL improvement topic-level for coding problems.
A.9 False Positives and Negatives in Code-RL Training
To highlight the importance of climinating false positive reward (incorrect code passing all within time constraints) and false negative reward (incorrect test cases that fail correct code) in RL Training, we conduct two ablation experiments, showing that both types of errors can harmful to RL training, resulting in early convergence on sub-optimal policies, or even complete training collapse.
DiveCodeBench v5
52
mm
Accuracy (avg@8)
38
AceReason-78
AccReason 78 (+ False Pasitives)
36
AceReason-78 Negatives)
195
273 312
Training Steps
Figure 10: The impact of false positive and false negative rewards in Code RL Training
To simulate the impact of false negative rewards, we introduce subset of problems into the training data where either the correctness of test cases could not be verified, or the official "golden" solution failed to pass all provided tests. From Figure we can see that, the RL model tends converge to sub-optimal point, with no further improvement on the evaluation set. We hypothesize that such

---

fälse negative reward causes the model to discard correct algorithms for certain training problems. Consequently, when encountering lesting problems that share similar underlying concepts and require these discarded algorithms, the model remains unable to identify the correct solution, thereby degrading performance.
To simulate RL training with false positive rewards, we blend problems with weak test cases that allow incorrect solutions to pass into our training set. As notable example, we examined DeepCoder's RL training set. Despite their claims of providing strong test case selection criteria, we find that their tests still cannot cover all tricky edge cases there exists incorrect solutions that can pass all their selected test cases but will subsequently fail on official test cases. After incorporating unfiltered Code-RL training data from these sources into our problem set, we observed sudden drop on testing set accuracy midway through the training process. as shown in Figure 10. We suspect that this performance degradation occurs because the model learns solve some training problems without considering all corner cases or by generating brute-force solutions with incorrect time complexity. This wrongly learned policy is then penalized while evaluating against strong test cases in the testing set.
A.10 Instruction for evaluation
Math
Please reason step by step, and put your final within \bboxe::
No Starter Code (Python)
Write Pythen code to solve the problem. Please place the solution code in the following format:
" 'python
Your solution code here
Has Starter Code
Solve the problem starting with the provided function header.
Function header:
<starter_code>
Please place the solution code in the following format: 'python
Your solution code here
A.11 Example AceReason-14B response to simple query
User: hi
Model:
<think>
Alright the user wrote "hi" in the middle of the message.
Probably just saying hello.
should respond in a friendly and wel welcoming way.
Maybe how can assist them today.
Keep it onen-ended to encourage them to share what they need help with. </think>
Hello! How can I assist you today?
25

---

User: many strawberries do need collect rfs?
Model:
<think>
First, need to understand what the user is asking. They want to know how many strawberries are needed to collect 9 rts.
interpret r's as the letter 'r' Each strawberry is associated with a certain number of 'r's.
Looking at the word strawberry, I count the number of 'r's it contains.
There are three 'r's in strawberry.
If one strawberry provides three 'r's, then to get a total of nine fr's, I divide the total needed by the number each strawberry provides.
So, nine divided by three equals three.
Therefore, the user needs to collect three strawberries to have nine fr's
</think>
To determine how many strawberries you need to collect **9 "r' s**, let's break it down step by step.
### Step 1: Understand the Context
It seems you' re referring to the letter **"r*.e within the word ** strawberry.**
### Step Count the let's s look at the word **strawberry.::
strawberry
.*s t w w r r y* ..re,, ANENA, *AIAN (there are **3 r's**)
### Step Calculate the Number Strawberries Needed
Each r*strawberyy* contains **3 r's 8'5.*
тo collect **9 r's*,, you would need:
M[
Atext (Number of strawberries) \frac(]text (Total. r's needed) (Xtext(r' 8 per strawberry)) (f) (3)
\]
### Final Answer
\
\boxed (3)
A]
26


Reviews:
[{"review_id": "FSe7qu2bkF", "reviewer": "Reviewer_8qff", "summary": "This paper demonstrates that large-scale RL training can significantly enhance the math and code reasoning capabilities of mall- and mid-sized models that have undergone SFT. It proposes a staged training approach (math-only first, followed by code-only) to improve model performance. Additionally, a high-quality, challenging, and open-source dataset is constructed to support reproducible research.", "questions": "N/A", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n- The paper introduces a novel two-stage RL training methodology(math first, then code). This not only significantly improves performance within each domain but also reveals an unexpected benefit: math-only RL substantially improves code reasoning capabilities.\n- The experiments cover a variety of math and code reasoning benchmarks and compare them with multiple SOTA models, verifying the effectiveness of the method.\n- This paper constructs a high-quality and challenging dataset through a rigorous pipeline, ensuring reliable verification-based RL training.\n\n\n### Weaknesses:\n- Although the paper indicates that RL shows a higher performance ceiling for models at 14B and above, the improvement for smaller models (e.g., 7B) is relatively limited, and it still underperforms distillation models in some math tasks.\n- The paper provides a limited theoretical or empirical explanation. For example, why math RL can enhance code task performance and why code tasks (e.g., graph theory, string manipulation) benefit more.", "was_revised": true, "final_justification": "The responses mitigate some of my concerns about Weakness 1. My final Rating is higher than 4 but lower than 5.", "rebuttal": "We sincerely appreciate your thoughtful review and recognition of our contributions on the “**novel two-stage RL training method**” and “**high-quality open-source dataset**”. We're pleased to address your concerns and provide additional results.\n\n>“Although the paper indicates that RL shows a higher performance ceiling for models at 14B and above, the improvement for smaller models (e.g., 7B) is relatively limited, and it still underperforms distillation models in some math tasks.“\n\nWe appreciate this observation, which actually highlights one of our key insights: **RL effectiveness scales with model capacity**.\nOur 14B results prove the concept: AceReason-14B surpasses specialized distillation models (AIME25: 67.4% vs 63% OpenMath-14B; LiveCodeBenchv5: 61.1% vs 59.4% OpenCode-14B), demonstrating that RL can exceed state-of-the-art distillation expert models.\nFor 7B models, while AceReason-7B doesn't surpass OpenMath-7B on math tasks, we achieve substantial **double-digit improvements** over the initial model. In addition, the performance on code tasks are competitive to OpenCode-7B (e.g., LCBv5 51.8% vs 51.3% OpenCode-7B).\n- AIME24: 55.5% → 69.0% (**+13.5%**)\n- AIME25: 39.0% → 53.6% (**+14.6%**)\n- LiveCodeBench v5: 37.6% → 51.8% (**+14.2%**)\n\nThe performance gap with OpenMath-7B stems from fundamental training differences:\n- **Data scale**: OpenMath uses **4.9M+** math-specific samples for SFT vs. **800K** general samples in our initial model (DeepSeek-R1-Distilled-Qwen). This is at least **6x difference** in terms of math training data\n- **Specialization**: OpenMath is math-only, OpenCode is code-only; AceReason handles math, code, and general reasoning capabilities\n- **Model size scaling trajectory**: OpenMath **plateaus** from 7B→14B →32B (AIME25: 61.2→63.0→62.5), while AceReason shows strong scaling potential (7B→14B: 53.6→67.4)\n\nThis scaling behavior suggests that while specialized distillation may be optimal at very small scales, RL becomes increasingly advantageous as model capacity grows at medium size.\n\n>“The paper provides a limited theoretical or empirical explanation. For example, why math RL can enhance code task performance and why code tasks (e.g., graph theory, string manipulation) benefit more.”\n\nFirst, we would like to provide more empirical results for the RL transfer reasoning capability phenomenon. Our investigation reveals that RL's benefits extend beyond math and coding and generalize to other tasks such as STEM (GPQA) and logic reasoning (Knight & Knave, Zebra-logic).\n| Model | GPQA | Knight & Knave | Zebra-Logic |\n|-------|------|----------------|-------------|\n| DeepSeek-R1-Distill-7B | 49.1 | 35.2 | 24.5 |\n| AceReason-7B (Ours) | **51.1 (+2.0)** | **56.4 (+21.2)** | **39.6 (+15.1)** |\n| DeepSeek-R1-Distill-14B | 59.1 | 83.8 | 60.1 |\n| AceReason-14B (Ours) | **61.2 (+2.1)** | **96.1 (+12.3)** | **72.7 (+12.6)** |\n\nIn addition, our detailed analysis (Figure 8 and Section A.7) reveals how these enhanced reasoning capabilities specifically benefit code generation:\n- **Improvements in algorithmic tasks**: Code problems requiring mathematical thinking (combinatorics, counting, number theory) show the largest gains, with direct application of math reasoning.\n- **Gains in logic-heavy tasks**: Problems involving complex control flow, constraint handling, and systematic search (graph algorithms, dynamic programming) benefit from enhanced reasoning capabilities.\n- **Improvements in implementation-focused tasks**: Even string manipulation and data structure problems show gains, suggesting that better reasoning helps with problem decomposition and solution planning.\n\nWe emphasize that our contribution is primarily empirical - we document and quantify these transfer effects systematically across multiple domains. Providing definitive theoretical explanations for these emergent behaviors remains beyond the scope of this work and represents an important direction for future research. We will make this clear in the limitation section of this paper. Thank you!"}, {"review_id": "MQzbY7HqsH", "reviewer": "Reviewer_QBrj", "summary": "The paper proposes a post-training method using Reinforcement Learning (RL) on small and medium-sized models. It implements staged training for math and code tasks, which improves the model's performance on these specific tasks. The paper provides a detailed account of the training setups and practical tricks used to stabilize the training process and enhance its effectiveness. The evaluation shows that RL significantly boosts the reasoning abilities of the initial model, achieving performance that is competitive with state-of-the-art reasoning models.", "questions": "1. Following up on W2, do you have additional experimental data with a larger pass@k to prove that RL expands the model's capability limits?\n2. Does the model show any improvement in general-purpose reasoning abilities after RL training?", "limitations": "See the weaknesses above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- S1: This paper is well-written and easy to follow.\n- S2: The paper offers many practical training tricks, supported by experiments, for managing the instability inherent in reinforcement learning.\n- S3: The paper provides a detailed analysis to demonstrate the performance improvements gained from RL.\nWeaknesses:\n- W1: The paper's novelty and contribution appear limited, as it seems to combine existing techniques. Examples include staged length extension and curriculum learning.\n- W2: The experiments analyzing the model's capability limits are questionable. In the cited paper https://arxiv.org/abs/2504.13837 (Yue et al., 2025), the performance inflection point for the base and RL models is around pass@256. However, this paper only presents results up to pass@64, which is insufficient to prove that RL training actually expands the model's capability limits.\n- W3: The paper does not evaluate the model's general reasoning abilities, for instance, on STEM or other puzzle-related reasoning tasks. Theoretically, an improved reasoning model should demonstrate stronger cross-domain reasoning skills.", "was_revised": true, "final_justification": "The authors provided more experiments and clarifications on the questions raised in the previous review. The responses mitigate my concerns. I decided to raise my rating.", "rebuttal": "Thank you for your detailed review. We hope to address your concerns with concrete evidence:\n>“The paper's novelty and contribution appear limited, as it seems to combine existing techniques. Examples include staged length extension and curriculum learning”\n\nWe agree that AceReason is built based on a combination of methods and techniques, and we appreciate your acknowledgement that our work offers “**many practical training tricks, supported by experiments**”. The novelty of our study lies in our ablation study of the necessity of stage-1 (8K) training (Fig 2b) resulted in training efficiency improvements, effectiveness of curriculum learning (Table 3) which uses harder prompts for later stage training, and demonstration of strict on-policy training which stabilizes RL training. Our analysis on problem-level accuracy also provides the first demonstration of the improvement of RL for both math and coding in Figure 4.\n\nIn addition to that, we provide the systematic demonstration that math-only RL significantly improves code reasoning (6-14% on LiveCodeBench) and proposes the two-stage math-only RL and code-only RL pipeline, which stabilizes the code RL training (Figure 1b). Many prior studies focus **exclusively on either math or code**, showing the difficulty of handling heterogenous prompts and inherent complexity of RL. To the best of our knowledge, this cross-domain transfer was not explicitly mentioned in prior studies and concurrent works mentioning this (Skywork-OR1 May 28, Magistral Jun 12) appeared after our submission deadline.\n\nCombining all these techniques, we are the first to show a medium sized SFT model using RL can surpass a specialized distillation model on both math and code and achieve state-of-the-art performance at the time.  At 14B, we outperform OpenMath-14B (AIME25: 63 vs 67.4) and OpenCode-14B (LCB-v5: 59.4 vs 61.1) - the best expert models in the Qwen2.5 family. Importantly, the challenge of combining these methods to achieve significant improvements is substantial and requires non-trivial technical depth.\n\n>“Following up on W2, do you have additional experimental data with a larger pass@k to prove that RL expands the model's capability limits?”\n\nThanks for your question. We initially used the pass@64 evaluation setting following Yue et al. [1] LiveCodeBench evaluation set up in their Figure 3. We agree showing pass@k at larger k can show more concrete results. Below, we computed pass@8 to 1024 results using the 7B model on two benchmarks (AIME2025 and LiveCodeBench v6), which are released after the release of the distilled model (DeepSeek-R1-Distilled). Each datapoint is calculated based on an average of 100 runs to reduce variance. \n\nAs we can see, AceReason-7B shows improved pass@k compared to the initial distilled model across k=8 to 1024. The gap in the LiveCodeBench v6 is more than **10+ points** as the coding problems require the answer to pass all test cases, which are **difficult to guess correctly**. Our results show that RL improves pass@k on coding tasks and show evidence that RL expands capability limits.\n\n- AIME 2025 Results:\n| pass@k | 8 |  16 | 32 | 64 | 128 | 256 | 512 | 1024 |\n|--------|----|----|----|----|----|-----|-----|------|\n| DeepSeek-R1-Distilled-Qwen-7B | 62.2 | 66.8 | 71.3 | 76.5 | 81.2 | 86.2 | 90.3 | 92.6 |\n| AceReason-7B | **73.4** | **76.0** | **78.3** | **85.4** | **89.0** | **93.0** | **94.8** | **96.1** |\n\n\n- LiveCodeBench v6 Results:\n| pass@k | 8 | 16 | 32 | 64 | 128 | 256 | 512 | 1024 |\n|--------|----|-----| ----|----|----|-----|-----|------|\n| DeepSeek-R1-Distilled-Qwen-7B | 47.4 | 50.9 | 53.6 | 56.3 | 58.2 | 60.4 | 62.3 | 63.4 |\n| AceReason-7B | **58.1** | **61.7** | **64.8** | **67.8** | **70.0** | **71.9** | **73.3** | **74.3** |\n\n\n[1] Yue, Y et al., Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?\n\n> “Does the model show any improvement in general-purpose reasoning abilities after RL training?”\n\nThanks for your comment. We agree examining the math and code RL model on other reasoning tasks can show a more broader understanding of the model’s reasoning capabilities. Below, we evaluated AceReason models on three tasks including STEM (GPQA) and logic puzzles such as (Knight & knave, Zebra-logic). Interestingly, we found AceReason shows significant improvements on logic puzzles (e.g., 12.3%+ on Knight & knave and 12.6% on Zebra-Logic for AceReason-14B). On GPQA, which is a dataset focused mainly on college-level knowledge, AceReason-14B shows 2% improvements. These results further support our previous observations that math-RL improves coding as it shows the math and code **RL improvements are generalizable** to other domains.\n\n| Model | GPQA | Knight & Knave | Zebra-Logic |\n|-------|------|----------------|-------------|\n| DeepSeek-R1-Distill-7B | 49.1 | 35.2 | 24.5 |\n| AceReason-7B (Ours) | **51.1 (+2.0)** | **56.4 (+21.2)** | **39.6 (+15.1)** |\n| DeepSeek-R1-Distill-14B | 59.1 | 83.8 | 60.1 |\n| AceReason-14B (Ours) | **61.2 (+2.1)** | **96.1 (+12.3)** | **72.7 (+12.6)** |\n\nWe hope this evidence addresses your concerns and demonstrates the significance of our contributions."}, {"review_id": "hqF0QTq7dz", "reviewer": "Reviewer_V9Hg", "summary": "The authors present a recipe for using large-scale reinforcement learning to enhance distillation-based reasoning models for math and coding tasks. The proposed approach specifically targets the RL phase. By carefully curating training problems and ablating nuances in the RL pipeline, the authors demonstrate that it is possible to elevate existing distilled models to approach the performance of top models on relevant benchmarks, such as AIME and LiveCodebench. The authors also incorporate a curriculum learning scheme that trains the model on math problems followed by coding tasks. This approach results in simultaneous improvements in both domains. They find that on-policy reinforcement learning is key for stable training.", "questions": "1. In the top right plot of Figure 2, the green line (AceReason-7B with math RL) and the red line (AceReason-7B without math RL) show a clear discrepancy in the final steps, as the latter's performance on LiveCodeBench declines. Has the authors investigated why this occurred?\n2. In Section 4.3 line 305, the authors state that the difficulty score is estimated via the 7B model. It seems like this is different from the previous estimation via DeepSeek-R1, introduced in Section 3.2. I wonder why two different estimation setups are used.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed pipeline is effective and produces models that are significantly stronger than their distillation counterparts, approaching the performance of closed-source or proprietary models.\n2. The proposed approach brings insights into some of the earlier unsolved \"mysteries\". These include the claim that distillation is often more effective for small models, the problem of entropy collapse (or explosion), and the importance of data (problem) curation.\n3. The paper provides extensive ablation studies and detailed analysis on multiple aspects of the pipeline, including problem difficulty, length extension, and curriculum learning.\n\nWeaknesses:\n\n1. Many components in the proposed pipeline mainly involve or borrow small modification from existing work. It uses GRPO with on-policy training, length extension, and problem difficulty estimation, which are not new to the field.\n2. While the experiments include ablation studies and analyses of some modifications, the justification for using each of them is still unclear to me, as most results are highly empirical. Readers are unsure whether these modifications would still be effective in a slightly different setting, such as with a different dataset or initial model.", "was_revised": true, "final_justification": "The authors have provided detailed ablation experiments and clarifications on all weaknesses / questions I raised in my review. These mainly include my concerns about the novelty of the propose method (or pipeline) and findings from empirical experiments. I decided to raise my rating due to the above reasons.", "rebuttal": ">“Many components in the proposed pipeline mainly involve or borrow small modification from existing work. It uses GRPO with on-policy training, length extension, and problem difficulty estimation, which are not new to the field.”\n\nThank you for your comment. We appreciate your acknowledgement of our proposed pipeline “**approaching the performance of close-source models**”, “**brings insights into some of the earlier unsolved mysteries**” and “**provide extensive ablation studies on multiple aspects of the pipeline**”. While we agree these techniques are not new, we believe the originality of our work are as follows:\n- **Effectiveness and efficiency improvement of each technique**: our ablation demonstrates the importance of each technique used in the pipeline such as strict on-policy training (stabilize training), stage-1 8K training (improves efficiency and effectiveness compared to starting from 16K or 24K), and problem difficulty (effectiveness in later stages). \n- **Math and Code RL pipeline**: we present the first systematic demonstration of math-only RL enhances code reasoning. This cross-domain transfer is a core aspect of the insight from our study. While concurrent works (Skywork-OR1, May 28; Magistral, June 12) have mentioned this, they appeared after our submission deadline. We thus introduce a two-stage RL pipeline, combining math-only RL and code-only RL, to boost training efficiency.\n- **State-of-the-art results on both math and code**: combining all the techniques above, we are the first to show that a medium-sized SFT (distilled) model, when enhanced with RL, can outperform specialized distillation models and achieve SOTA results on both math and coding tasks, which is a non-trivial contribution and requires a significant technical depth. In contrast, many prior studies **focus exclusively** on either math or code, highlighting the difficulty of handling heterogeneous prompts and inherent complexity of RL training.\n- **Novel insights on code-RL capability improvements**: we present a series of ablation studies to understand the unique challenge of code-RL training (appendix A.8) and show a unique perspective of code-RL indeed expanding the initial model performance limit on LiveCodeBench (Figure 4 - top). In addition, we add pass@1024 results below to further validate our observation that code-RL expand capability of the initial model. We believe these insights are novel to understand the code RL training.\n\n- LiveCodeBench v6 Results:\n| pass@k | 8 | 16 | 32 | 64 | 128 | 256 | 512 | 1024 |\n|--------|----|-----| ----|----|----|-----|-----|------|\n| DeepSeek-R1-Distilled-Qwen-7B | 47.4 | 50.9 | 53.6 | 56.3 | 58.2 | 60.4 | 62.3 | 63.4 |\n| AceReason-7B | **58.1** | **61.7** | **64.8** | **67.8** | **70.0** | **71.9** | **73.3** | **74.3** |\n\n\n\n>“While the experiments include ablation studies and analyses of some modifications, the justification for using each of them is still unclear to me, as most results are highly empirical. Readers are unsure whether these modifications would still be effective in a slightly different setting, such as with a different dataset or initial model.\"\n\nThank you for your comment. We agree that most results are indeed empirical as RL training itself already involves a lot of randomness and doing RL on top of strong reasoning models that generate long chain-of-thought responses is highly computational intensive. We extensively experiment with a series of techniques to combat training stability (on-policy), improve training efficiency (stage-1 8K training), curriculum learning (using hard prompts) and show the effectiveness of each component. We believe these insights are generally applicable to different models and dataset. We also propose a reliable two-stage training recipe to do math-RL and code-RL for training stability and efficiency. Each of these designs are well motivated to **improve training stability and efficiency** as a way to scale RL.\n\nIn addition to 7B and 14B models, we validated the recipe on smaller model 1.5B and larger model 32B below (due to computational constraint, we only conducted math-RL for the 32B model). Our results consistently demonstrate that **our RL recipe enhances performance across all model scales**, and the conclusion that math-RL improves code generalizes effectively to the 32B model (LCB v5: 57.2 → 63.4), this is close to the DeepSeek-R1 model (65.9) even without code-RL training, which surprise us. Furthermore, to ensure the community can reproduce our study, we are committed to **open-sourcing our training dataset and model**.\n\n| Model | AIME24 | AIME25 | LiveCodeBench v5 |\n|-----------|------------|------------|---------------------|\n| DeepSeek-R1-Distilled-Qwen-1.5B | 28.9 | 22.7 | 16.9 |\n| AceReason-1.5B | 47.7(**+18.8**) | 35.5(**+12.8**) | 27.2(**+10.3**) |\n| DeepSeek-R1-Distilled-Qwen-32B | 72.6 | 54.9 | 57.2 |\n| AceReason-32B (math-only RL) | 79.0 (**+6.4**) | 70.6 (**+15.7**) | 63.4 (**+6.2**) |\n\n## Technical questions\n\n>“In the top right plot of Figure 2, the green line (AceReason-7B with math RL) and the red line (AceReason-7B without math RL) show a clear discrepancy in the final steps, as the latter's performance on LiveCodeBench declines. Has the authors investigated why this occurred?”\n\nOur experiments reveal two distinct advantages of initializing code-RL training from a math-RL checkpoint. Firstly, we observe an improvement in code task performance. Secondly, the math-RL checkpoint consistently produces more effective chains of thought, contributing to a more stable RL training process, particularly in later stages of code-RL. The length of model responses demonstrates a gradual increase throughout training. Conversely, direct initialization from a distilled checkpoint results in fluctuating response lengths and an exponential increase in the clip-ratio (% of responses exceeding maximum length), leading to **instability** during RL training and subsequent **training collapse** in later stages. Therefore, we have elected to commence all subsequent experiments from a math-RL checkpoint.\n\n>“In Section 4.3 line 305, the authors state that the difficulty score is estimated via the 7B model. It seems like this is different from the previous estimation via DeepSeek-R1, introduced in Section 3.2. I wonder why two different estimation setups are used.”\n\nThanks for raising this question. We would like to clarify the distinction between Section 3.2 and Section 4.3.\nIn Section 3.2, we utilize DeepSeek-R1 to determine if the collected math problems are solvable by the R1 model. This process primarily serves to clean the dataset, removing question-answer pairs that may be noisy due to OCR-related errors, thus ensuring the questions are solvable at least with the R1 model.\n\nConversely, in Section 4.3, we employ the 7B model to estimate the problem difficulty (pass rate). This approach is more appropriate as it uses the current policy model to assess the pass rate and filter out questions deemed easy by this model. These filtered data are subsequently used for training the current policy model. Intuitively, this provides a more accurate estimation of problem difficulty for the current policy model, and is beneficial for RL training."}, {"review_id": "laJOndpXDJ", "reviewer": "Reviewer_iccS", "summary": "In this paper, the authors examine the question of whether small- and mid-size reasoning models should be built by training directly with RL, or via distillation from larger and more capable reasoners. Via a relatively simple recipe involving GRPO, they train two reasoning models (which they dub AceReason models), of size 7B and 14B, and show that they can outperform comparable reasoners built via distillation. They outline the details of their training setup, their data curation strategy, and report a fine-grained analysis of how RL impacts the models' reasoning capability.", "questions": "My main question for the authors is to produce evidence supporting the general applicability of their training recipe. They should carry out their training pipeline on at least one model of size ~1B, and most importantly, on at least one small- or mid-sized model that has not been distilled directly from DeepSeek-R1. If the gains observed there were comparable to those observed in the main results table, that would make the paper's message much more solid and would prompt me to raise my score.\n\nBelow are more questions and remarks, preceded by the line number when they refer to a specific line in the text:\n- 47: \"the distilled model is first trained on math-only prompts\" What distilled model is being referred to here?\n- The authors make a big deal of training completely on-policy, by doing only one gradient update per rollout, but how does this happen in practice? In line 191, the authors say that their rollout batch size is 128, with group size $G=8$. Does this mean that they use a minibatch size of 128 when computing the gradient? That seems impossible. Could the authors provide details?\n- 273: \"Ace Reason still remains its superiority\" please fix this typo.", "limitations": "The authors do not discuss the limitations of their work. In my opinion, and considering that the main message of the paper is that large-scale RL can enhance the reasoning capabilities of small- and mid-sized models, these limitations are:\n- The models that authors consider are ultimately mid-sized by today's standards. Capable LLMs can have has little as 1B parameters today, but the authors do not attempt their pipeline on anything below 7B in size.\n- Moreover, the authors only attempt their pipeline on two very specific models, both distilled from DeepSeek-R1, a very capable and large reasoner. This calls into question the applicability of their pipeline to generic models in that size class, and undermines the main message of the paper.\n\nThe authors should address these limitations, or at least clearly outline them in an ad-hoc paragraph.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The question addressed by the authors is timely and pertinent.\n- Their training setup is ultimately quite simple and easily reproducible.\n- The gains w.r.t baseline are substantial.\n- In fine-grained analyses in section 4.3 are very useful and insightful.\n\n**Weaknesses:**\n- While it is true that the DeepSeek paper had argued for distillation instead of RL for building mid-size reasoners, I do not think that the idea that a reasoner of 7B or 14B can be successfully trained via RL is as controversial or unexpected as the authors imply.\n- 7B and 14B are relatively small sizes in today's day and age, but there are capable LLMs that are even smaller, e.g. Qwen2.5-1.5B. It remains to be seen whether the authors' RL pipeline would still be successful at those sizes.\n- The authors only use models distilled from DeepSeek as base models in their experiments. This undermines their point on RL vs. distillation, as well as leaving the possibility that their RL pipeline would not be successful in general.", "was_revised": true, "final_justification": "The authors have clarified the main message of their work, which I believe is supported by the experiments they had show in their submitted manuscript. Moreover, they have ran an additional experiment with a smaller (1.5B) model, which further strengthens the paper's message.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback. We appreciate you finding the question addressed is “**timely and pertinent**” and analysis in section 4.3 are “**very useful and insightful**”. Below, we aim to clarify and address your questions.\n\n## Main Contribution Clarification\n\nWe would like to clarify our research question and define what constitutes \"successful\" RL in our context.\nThe DeepSeek-R1 training pipeline consists of: (1) SFT using 800K distilled long-CoT responses, followed by (2) large-scale RL. Our experiments **explicitly follow** this pipeline, starting from models already distilled from R1 (DeepSeek-R1-Distilled-Qwen-7B/14B). We intentionally do **not** experiment with RL from base models, as this represents a **different paradigm** (similar to DeepSeek-R1-Zero) that yields inferior results.\n\nFor smaller models, the field has converged on two competing approaches:\n- **Distillation-only**: Generate massive amounts of distillation data (e.g., 4.9M+ examples for OpenMath-Nemotron, 700K for OpenCode-Nemotron) and conduct large-scale SFT to create specialized expert models\n- **Distillation + RL**: Start from DeepSeek-R1-Distilled checkpoints (800K SFT data) and apply large-scale RL (our approach, also used by Skywork-OR1-Math and Light-R1-14B)\n\nOur central research question is: \"*For smaller models, can RL on top of distilled models achieve better performance than distillation alone?*\" This question matters because recent influential work (DeepSeek-R1 and Llama-Nemotron-235B) concluded that RL yields suboptimal results for smaller models compared to distillation, recommending RL only for large models like DeepSeek-V3-671B or Llama-Nemotron-Ultra-253B.\n\nTherefore, our definition of \"successful\" RL is not merely improving over the initial distilled model – we aim to **match or exceed** the best specialized distillation-only expert models. As stated in Lines 43-45, “we demonstrate that large-scale RL can significantly enhance strong small- and mid-sized SFT models, achieving performance competitive with state-of-the-art distillation at 7B and surpassing it at 14B”.\n\n| Model | AIME24 | AIME25 | LiveCodeBench v5 | LiveCodeBench v6 |\n|-------|--------|--------|------------------|------------------|\n| OpenMath-14B | 76.3 | 63 | - | - |\n| OpenCode-14B | - | - | 59.4 | 51.1 |\n| AceReason-14B | **78.6** | **67.4** | **61.1** | **54.9** |\n\n\nBelow, we hope to address your questions:\n\n>#1: “I do not think that the idea that a reasoner of 7B or 14B can be successfully trained via RL is as controversial or unexpected as the authors imply”\n\nWe would like to clarify what we mean by \"successful\" RL training. Our definition of success is **not** merely achieving improvements over the initial distilled model - rather, it's matching or exceeding the best specialized distillation-only models, which is highly challenging.\nPrior work, including DeepSeek-R1 and Llama-Nemotron-235B, explicitly recommended distillation as the superior approach. The \"Sober Reasoning\" paper (arXiv:2504.07086 April 2025) reinforced this view, claiming that \"reinforcement learning (RL) approaches yield only modest improvements - far below prior claims\"\n\nIn contrast, our results directly challenge this consensus. At 14B scale, AceReason surpasses the best distillation-only expert models (e.g., OpenMath, OpenCode). One of the key insights is that we reveal that model size critically determines RL effectiveness - as 7B shows competitive results and 14B shows outperforming results. Meanwhile, the distillation model OpenMath performance on math benchmarks saturated when scale model size from 14B to 32B (e.g., AIME25 drops from 63 to 62.5).\n\nWhether this finding is \"controversial\" may be subjective and time-dependent. To the best of our knowledge at the submission time, we believe demonstrating that RL can outperform specialized distillation at 14B scale represents novel insights to the community.\n\n>#2“1.5B…It remains to be seen whether the authors' RL pipeline would still be successful at those sizes.”\n\nThanks for your comments. We agree 1.5B model size is also important and facilitate reproducibility of our study. Therefore, we conducted additional experiments on 1.5B models to show the effectiveness of our RL recipe. As we found below, AceReason-1.5B shows significant improvement over the initial model achieving 18.9%+ on AIME24, 12.8%+ on AIME25. We hope these results resolve your concern of the effectiveness of RL pipeline when applied to smaller model sizes.\n\n| Model | AIME24 | AIME25 | LiveCodeBench v5 |\n|-----------|------------|------------|---------------------|\n| DeepSeek-R1-Distilled-Qwen-1.5B | 28.9 | 22.7 | 16.9 |\n| AceReason-1.5B | 47.7(**+18.8**) | 35.5(**+12.8**) | 27.2(**+10.3**) |\n\n\n\n>#3“The authors only use models distilled from DeepSeek as base models in their experiments.”\n\nThank you for your question. The choice of using models distilled from DeepSeek-R1 as the initial model in our experiment was deliberate and essential to our research design. Since our core research question examines whether RL can enhance already-distilled models following the DeepSeek-R1 pipeline (SFT → RL), we necessarily start from DeepSeek-distilled checkpoints. Using models distilled through other methods would introduce confounding variables and prevent us from isolating the effects of our RL approach. In addition, experimenting with RL from base pre-trained models (as in DeepSeek-R1-Zero) represents a fundamentally different research question and is beyond the scope of this work.\n\n>#4“The authors do not discuss the limitations of their work. … these limitations are:\n… Capable LLMs can have as little as 1B parameters today, but the authors do not attempt their pipeline on anything below 7B in size.\nthe authors only attempt their pipeline on two very specific models, both distilled from DeepSeek-R1, …\nThe authors should address these limitations, or at least clearly outline them in an ad-hoc paragraph.“\n\nThanks for your comment. We put the limitation section of this work in Appendix A.1.\nTo your questions of “attempt their pipeline on … 1B model” and “only attempt their pipeline on two very specific models”, we hope our previous response in #2 (add 1.5B results) and #3 (explain our experiment setting is different from DeepSeek-R1-Zero) resolve your questions. \nWe appreciate your feedback on these limitations and will explicitly mention the research question clearly in the introduction, experiment setting that uses DeepSeek distilled model for RL (L113), and add 1.5B experiment results to the paper. We will also mention we are not using a pre-trained base model for RL in the limitation as it is related to a different experiment setting. Thank you!\n\n## Technical Clarifications\n>Q1: Line 47 - \"the distilled model is first trained on math-only prompts - What distilled model is being referred to here?\"\n\nThe \"distilled model\" refers specifically to DeepSeek-R1-Distilled-Qwen-7B/14B models.\n\n>Q2: On-policy training implementation details\n\nExcellent question about our implementation. Let me clarify with a concrete example:\n- With group size G=8 and batch size 128, we generate 128×8 = 1,024 examples total\n- These 1,024 examples are divided into minibatches for gradient computation\n- We accumulate gradients across all minibatches\n- Crucially: We perform exactly ONE gradient update after processing all 1,024 examples\n\nThis is implemented in veRL by setting \n\ndata.train_batch_size == actor_rollout_ref.actor.ppo_mini_batch_size\n. We'll add this clarification to improve clarity.\n\nWe hope these clarifications and additional evidence address your key concerns while demonstrating that our work makes significant contributions to show how RL can enhance already-capable reasoning models. Please feel free to let us know if you have any other questions. Thank you!"}]

Author rebuttal:

Meta-review: The paper investigates whether large-scale RL can enhance the reasoning capabilities of small- to mid-sized models beyond distillation-based approaches. The authors propose a two-stage RL training pipeline (math-first, then code) and demonstrate significant improvements on math and code benchmarks. Key findings include: (1) math-only RL improves both math and code performance, with extended code-only RL further boosting code results, (2) curriculum learning with progressive response lengths stabilizes training, and (3) on-policy RL is critical for performance. The work also releases a high-quality, open-source dataset for verification-based RL.  

Reviewers acknowledge the novel two-stage RL methodology that improves performance in both domains and reveals math RL’s unexpected benefit for code reasoning. The rigorous data curation pipeline and extensive ablations provide actionable insights for reproducible research. The practical training tricks address RL instability and outperform distillation baselines. The gains are substantial, with the 14B model approaching proprietary model performance. A high-quality open dataset enabling reproducible RL training is also highlighted.

On the other hand, reviewers note limitations. The 7B model’s improvement is modest, and it underperforms distillation in some math tasks. Theoretical explanations for cross-domain benefits are lacking. Generalization to smaller models or non-DeepSeek base models remains unverified.  

In general, the paper makes valuable contributions by demonstrating RL’s effectiveness in enhancing reasoning capabilities of small- and mid-sized models, supported by solid experiments, a useful dataset, and insightful analyses. While it has limitations, its strengths such as novel methodology, significant performance gains, open dataset, and reproducibility outweigh the weaknesses.

During the author rebuttal, the authors have clarified the main message of their work, and have run additional experiments to address reviewers' concerns. These efforts are appreciated by most reviewers.
Decision: Accept (poster)