AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy
Authors: Zihan Liu
Zhuolin Yang
Yang Chen
Chankyu Lee
Mohammad Shoeybi
Bryan Catanzaro
Wei Ping
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Reasoning
Reinforcement Learning
Supervised Fine-tuning
Math
Code
Primary area: foundation or frontier models, including LLMs
TL;DR: We study the synergy between SFT and RL in developing strong reasoning models. Our final 7B model attains top-tier performance among Qwen2.5-based 7B models.

Abstract: In this work, we investigate the synergy between supervised fine-tuning (SFT) and reinforcement learning (RL) in developing strong reasoning models. We begin by curating the SFT training data through two scaling strategies: increasing the number of collected prompts and the number of generated responses per prompt. Both approaches yield notable improvements in reasoning performance, with scaling the number of prompts resulting in more substantial gains. We then explore the following questions regarding the synergy between SFT and RL:
(i) Does a stronger SFT model consistently lead to better final performance after large-scale RL training?
(ii) How can we determine an appropriate sampling temperature during RL training to effectively balance exploration and exploitation for a given SFT initialization?  
Our findings suggest that (i) holds true, provided effective RL training is conducted, particularly when the sampling temperature is carefully chosen to maintain the temperature-adjusted entropy around 0.3, a setting that strikes a good balance between exploration and exploitation. Notably, the performance gap between initial SFT models narrows significantly throughout the RL process. Built on a strong SFT foundation and SFT–RL synergy, our AceReason-Nemotron-1.1 7B model significantly outperforms AceReason-Nemotron-1.0 and achieves new state-of-the-art performance among Qwen2.5-7B-based reasoning models on challenging math and code benchmarks, thereby demonstrating the effectiveness of our post-training recipe.

Published as a conference paper at ICLR 2026
ACEREASON-NEMOTRON 1.1: ADVANCING MATH AND CODE REASONING THROUGH SFT AND RL SYNERGY
Zihan Liu , Zhuolin Yang', Yang Chen', Chankyu Lee', Mohammad Shoeybi', Bryan Catanzaro', Wei Ping 113
ABSTRACT
In this work, we investigate the synergy between supervised fine-tuning (SFT) and reinforcement learning (RL) in developing strong reasoning models. We be- gin by curating the SFT training data through two scaling strategies: increasing the number of collected prompts and the number of generated responses per prompt. Both approaches yield notable improvements in reasoning performance, with scal- ing the number of prompts resulting in more substantial gains. We then explore the following questions regarding the synergy between SFT and RL: (i) Does stronger SFT model consistently lead to hetter final performance after large-scale RL training? (ii) How can we determine an appropriate sampling temperature during RL training to effectively balance exploration and exploitation for a given SFT initialization? Our findings suggest that (i) holds true, provided effective RL training is conducted, particularly when the sampling temperature is carefully chosen to maintain the temperature-adjusted entropy around 0.3, a setting that strikes a good balance between exploration and exploitation. Notably, the perfor- mance gap between initial SFT models narrows significantly throughout the RL process. Built strong SFT foundation and SFT-RL synergy, AceReason- Nemotron-Le 1 7B model significantly outperforms EveReeson-Nemmotee 1.0 and achieves new models on challenging math and code benchmarks, thereby demonstrating the ef- fectiveness of our post-training recipe.
performance among Qwen2.5-77B 2-B-2-----B--eedd reasoning
INTRODUCTION
Math and code reasoning with large language models (LLMs) has been active area of research
for years (Cobbe et al. 2021 Chen et al., 202T). Previous work has primarily focused on enhanc- ing short chain-of-though (CoT) reasoning (Wei et al. 2022). which is typically acquired through
pretraining and supervised fine-tuning (SFT) (e.g., Shao et al. 2024: Hui et al. 2024 Yang et al.)
2024b Liu al. 2024c).
Since the introduction of OpenAl ol (OpenAl 2024) and DeepSeek-RI (Guo et al. 2025 Liu et al.
2024a), long chain-of ofttouggtthhhuhht (CoT) reasoning, which is acquired through large-scale rein reinforcement learning (RI.,, has emerged as a key driver of the remarkable progress in the reasoning capabilities of frontier LLMs (Guo et al. 2025 Qwen-Team [025 Yang et al. 2025), with reward signals typically provided by rulc-based verifiers. Subsequently, much of the follow-up work has focused on distilling these large frontier models into smaller mid-sized models through synthetic data generation and
SFT-only approach (Bercovich et al.] 2025 Moshkov et al. 22025] Ahmad et al. 2025 Yang et al.) 2025). Several recent efforts have sought replicate the success large- scale RL on smaller base or SFT models (i.c., 7B and 14B ) (e.g., Luo et al. 2025a He et al. [025a]] Wen et al., 2025 Liu et al. 2025), often leveraging DeepSeek-RI-Diseee --Distilll-Qee222 checkpoints. However, systematic study of the synergy between SFT and RL has been limited within the research community and is notably absent from the technical reports of frontier models.
(Chen et al. (025) as the initialization
AccReason-Nemooroon-l. (Chen et al., 2025) proposed stage-wise RL method math-only and code-only prompts, demonstrating both effectiveness and efficiency. In this work, we take a
NVIDIA', t: Correspondence to: zihanl @ @nvidia.com, wping@nvidia.com. 3: Technical Lead.

---

Published as a conference paper at ICLR 2026
step further by integrating supervised fine-tuning (SFT) and reinforcement learning (RL), probing their training dynamics and the synergy between them to provide a holistic perspective on these post-training techniques for building sate-oo-thheann reasoning models.
Specifically, we make the following contributions:
1. We begin by scaling SFT training through collecting large number of prompts, increas- ing the number of generated responses per prompt, and increasing the number of training epochs. We find that: (i) Scaling both the number of prompts and the number of gener- ated responses per prompt leads to substantial improvements in reasoning performance on math and code benchmarks, although scaling the number of prompts yields more signifi- cant gains. (ii) We observe consistent performance gains from the first to the fifth epoch, with improvements plateauing between the fifth and sixth epochs, regardless of the specific SFT blend used. This suggests that a certain degree of "overfitting" actually enhances test accuracy with long CoT generation, likely due to exposure bias autoregressive models.
2. We initiate RL training from various SFT models and make the following key observations: (i) Stronger SFT models continue to produce consistently better results after large-scale RL, although the performance gap narrows during RL training.
(ii) For a given initial SFT model, selecting an appropriate temperature during RL training is crucial for achieving a good balance between exploration and exploitation. We provide rule of thumb for setting the sampling temperature such that the temperature-adjuste entropy remains around 0.3, which typically leads effecive RL training.
3. We systematically study the best strategy during RL training when the final answer is not produced within a specific response length budget (e.g.. 24K). Whether we should assign negative reward or mask out the entire sample (i.e., overlong filtering). Our findings show that overlong filtering provides clear benefits when the token limit is short (e.g.. 8K or 16K). However, this advantage diminishes at a 24K token budget, and at 32K, overlong filtering can even degrade model performance.
4. We confirm that the stage-wise RL approach on math-only and code-only prompts remains effective when applied to range of much stronger SFT models beyond DeepSeck-RR Distill-Qwen models, demonstrating the broad applicability of this method (Chen et al. 2025 Building our strong SFT model and insights into the synergy between SFT and RL, our AceReason-Nemotrrn
7B model achieves record-high performance among Qwen2.5-7B-basee reasoning models on challenging math and code reasoning bench- marks, demonstrating the superiority of our post-training recipe.
2 RELATED WORK
The capacity for reasoning is essential to Al and manifests in both the text domain (e.g.. Wei et al.,
2022 Cobbe al. [222]]]] Hui et al. 2024) and the multimodal space (e.g.. Dai al.] 2004 Zhu
et al. 2025 Ghosh et al. 2025). DeepSeek-RI (Guo et al. 2025) has demonstrated the effectiveness reinforcement learning (RL). This approach has proven particularly useful in
of verification-
domains with structured outputs and well-defined verification criteria (c.g.. Chen et al. [025 Yang
al. 2024b; Liu al., 2024c Guo et al., 20255 Luo et al. 2025a). Such rule-based verification
eliminates the need for reward modeling and enhances the accuracy of the reward signal.
Several follow-up studies have explored various RL training techniques improve model perfor- mance on math and code reasoming tasks (Luo et al., 2023b He et al. 2025b, Chen et al.) 2025). BicReason-Nemotron.
(Chen et al., 2025) proposed stage-wise RL method that trains sequen- tially on math and code prompts, and advocated for strict on-policy GRPO (Shao et al. 2024) train- ing. DAPO (Yu al. 2025) investigates the use of an overlong penalty- assigns negative rewards to truncated generations within the response length-in the math domain, and finds that removing it is beneficial. This approach is also adopted by DeepCoder (Luo et al. (025a)). In this work, we conduct more systematic analysis and find that the overlong penalty should be applied during the later stages of RL training, rather than at the early stages.
Another line of work focuses supervised fine-tuning (SFT) via distillation from frontier reason- ing models trained with reinforcement learning. Notable examples include DeepSeek-FR -I-ekkii Qwen (Guo al. 2025), Light-R1 (Wen al. 2025). OpenMathReassonnng (Moshkov al. 2025
2

---

Published as a conference paper at ICLR 2026
Math Code SFT
Math-only Stnge-1 (8K)
Base Model
SFT Model
Math RL. Model
to Stuge-3 (24K)
Code-unly Stage-I (24K) & Stage-1l (32K)
Math Code RL Model
Math-only Stage-4 (32K)
BeeReassonNNeotzee 1.1
Figure Training Pipeline of AceReason- Nemotron 1.I. We start by performing math and code SFT on a base pretrained model. Next, we conduct three stages of math-only RL training with progressively growing response length, i.c., Stage-1 (8K), Stage-2 (16K), and Stage-3 (24K), to develop math-specialized RI. model. We then apply code-only RI. training to enhance model's coding capability. Lastly, earry out a final stage of math-only RI produce AccReason-Nematro 1.1.
-eeeeoonnNemootonn
OpenCode (Ahmad al. 2025), ) and Llama-Nemotron (Bercovich al. 2025). These models leverage large-scale math and code reasoning samples generated by DeepSeek-RI (Guo al. 2025) or QwQ (Qwen-Team) 2025). Some works describe the use of pretraining, supervised fine-tuning (SFT), and reinforcement learning (RL) in building strong reasoning models. However, systematic study of the interplay and integration between SFT and RL is often lacking in technical reports (e.g.. Yang et al. 2025
Xia et al. 2025 Seed et al. 2025).
METHOD
In this section, we present the details of the supervised fine-tuning and reinforcement learning pro- cesses used to train AceReason-Nemotron 1.1. The overall training pipeline is illustrated in Figure
3.1 SUPERVISED FINE-TUNING
3.1.1 PROMPT COLLECTION AND FILTERING
We collect math and code reasoning datasets from high-qualige data sources. For math, we collect prompts from AceMath dataset (Liu al. 2024c), NuminaMath (Li et al.) (024), and Open
Reasoning (Moshkov et al. 2025). ). For coding, we collect prompts from TACO (Li et al. 2023),
APPs (Hendrycks et al. 202 OpenCoder-Stagel (Huang al. 2024), and OpenCodeReason-
ing (Ahmad et a 2025). ). We conduct the dataset deduplication to ensure that each prompt is unique. After that, we conduct data decontamination and filter the sample that has 9-gram overlap with any
test sample in math and coding benchmarks (Mucnnigholf et al. 2025). ). We use DeepSeek-RR (Guo
al. 2025) to generate responses for the collected prompt set.
We aim for our SFT dataset to encompass a diverse range of challenging samples. Intuitively, longer model responses often correspond more difficult questions. Based on this observation, we found that a large portion of the collected prompts are relatively simple, with many responses around or below 2,000 tokens in length. To achieve a better balance across difficulty levels, we randomly filtered out subset of these simpler prompts and adjusted the proportions of other difficulty levels through additional random sampling. This resulted final dataset of 247K math prompts and 136K code prompts, totaling 383K prompts. Response token length distributions are in Appendix
3.1.2 SCALING OF SFT DATA
We investigate how scaling the SFT dataset impacts model performance along two axes: (1) increas- ing the number of unique prompts, and (2) increasing the number responses per prompt. Expanding the set of prompts enriches the coverage of problem types and topics, while adding more responses per prompt allows the model observe diverse reasoning paths for the same input.
To explore these scaling strategies, we construct seven SFT datasets (v1 through v7). each main- taining similar distribution of response token lengths shown in Appendix B The dataset size scale increases from 36K samples in v1 to 2.2M samples in v7. Section Figure prrure[speeeents the number of prompts and the corresponding average number of responses per prompt for each dataset.
3

---

Published as a conference paper at ICLR 2026
For all datasets, SFT training is initialized from the base model Qwen2.5-Math-27 (Yang et al. 2024b), which is also the starting point for DeepSeek-Ri-ee RR--Disiil-Qwen--
(Guo et al., 2025). Since
Qwen2.5-Math- only supports context length of 4,096. we modify the rope theta parameter from 10,000 to 1,000,000 enable support for a context length of 128K.
In Section we investigate and address the following questions with respect to SFT:
1. How does scaling SFT data improve model performance math and code benchmarks?
2. Is it more effective scale the number of unique prompts or the number of responses per prompt? What is the best strategy in practice?
3. How does training for more epochs improve performance? What is the stopping criterion?
3.2
REINFORCEMEN LEARNING
3.2.1
OVERVIEW
We apply the stage-wise RL approach on math-only and code-only prompts in sequence, as de- scribed in Chen et al. (0025), to our SFT models. Specifically, we employ the GRPO algorithm (Shao et al. (024) and strictly adhere to on-policy training by generating G = 8 or 16 rollouts (0,) for each question & in global batch of 128 prompts, followed by a single policy gradient update. The motivation for using on-policy training is to stabilize RL and prevent entropy collapse, as demon- strated in Chen et al. (025). We utilize the token-level policy gradient loss, which assigns greater rewards longer samples when the answer is correct and harsher penalties when it is incorrect. The intuition is that learning to generate longer samples plays a more critical role in enhancing reasoning capabilities. We remove the KL divergence term well.
3.2.2 DATA CURATION
We utilize the high-quality math and code RL data from AceReason-Nemotron--l (Chen et al., 2025). In particular, we find that the difficulty level of prompts and the accuracy of answers are the most critical factors for effective RL training. Prompts should be neither too easy nor too difficult, enabling the model to receive balanced mix of positive and negative rewards across a group of rollouts. In all cases, answers must be as accurate as possible, as this is the only way for the model to receive meaningful signals during verification-batt based RI. Taken together, these findings highlight that the quality of RI. data outweighs its quantity.
3.2.3 TRAINING PROCESS
We follow the math-only and code-only RL training curriculum introduced AceReason- Nemotron-1.0 (Chen et al. 2025). We begin with math-only RL using stage-wise response length extension from 8K to 16K, and then to 24K. Next, we apply code-only RL with response lengths of 24K and 32K. Finally, we conduct length budget. For each stage, we use the same RL training dataset as AceReason-Nemottrn- The entire training process and key takeaways summarized below.
additional round of math-only RL with 32K response
1. Math-only Stage-1 (8K): This initial stage with 8K response length budget serves warm-up phase for RL training. We use relatively simple questions sampled from our collected RL dataset for training. Most of these questions elicit responses from DeepSeek- R1 with token lengths predominantly between 2K and 4K.
2. Math-only Stage-2 (16K): At this stage of training, we increase the proportion of more challenging questions compared to Stage 1. As a result, the model's average response length gradually increases, and observe substantial performance improvement.
3. Math-only Stage-3 (24K): We filter out most of the simple questions and keep around 2500 hard ones for the training of this stage. Our model shows a significant performance improvement on math benchmarks in this stage.
4. Code-only Stage-I (24K): This stage marks the beginning of code RL training, which is initiated after math RL training ensure greater stability.
4

---

Published as a conference paper at ICLR 2026
5. Code-only Stage-1I (32K): In this stage, apply the epoch-wise filtering strategy in AccReasoon-emootrn 1.0. Specifically, we remove problems that can be fully solved by the previous epoch's 's checkpoint, i... problems for which every rollout passes all test cases.
6. Math-only Stage-4 (32K): As in the math-only Stage-3 (24K) setup, filter out most the simple questions- those that can be solved by every rollout-and retain only the challenging ones for training in this final stage.
In this work, explore the following questions related to the RL process:
1. How does initializing RL from different SFT models affect final model performance? Does starting from a stronger SFT model lead to better overall performance?
2. How does the sampling temperature of policy LLM affect RL training, particularly in bal- ancing the exploration-exploiaaiie trade-off?
3. Is the Math-only Stage-1 (8K) truly necessary, given that it initially lowers benchmark performance? If the reasoning process compression from this stage is essential, how long should train? Do we need to wait until benchmark accuracies fully recover?
4. During RL training, what is the best strategy when the final answer is not generated within a specified response length budget (e.g., 24K)? Should we assign a negative reward or mask out the entire sample (i.e., overlong filtering)?
5. Does RL still improve pass @ K large K when the SFT model is much stronger than DeepSeek-RI--isiill--eeaa used in AcoReason-Nemotron--100
4 EVALUATION
4.1 BENCHMARK
For math tasks, we evaluate our models on AIME24, AIME25, Math500 (Hendrycks et al. 202 as well as HMMT2025 Feb and BRUMO2025 from MathArena (Balunovic et al.) (005). For cod- ing tasks, we evaluate our models on EvalPlus (Liu et al. 2023 2024b), and LiveCodeBench v5 (2024/095 01/00//01100//111
and (2025 / / 02/00//0 /0 2025 2025/0002222 05/01) (Jain et al. 2024). Un- less otherwise specified, all benchmarks use the default inference settings: temperature of 0.6, top-p of 0.95, and a maximum sequence length 32,768 (32K). Because reasoning models produce highly variable outputs when sampling is used, we report pass @ performance averaged over runs (denoted as avg @)) For all reported numbers, we set n = 64 for AIME24, AIME25, HMMT2025 Feb, and BRUMO2025; EvalPlus. Note that large n is crucial for obtaining reliable metric, as the standard deviation of pass @ decreases with 1/ espppecillyyy for benchmarks with small number of problems. For AIME2024, avg@ @ 16, avg @ 32, and avg@ 64 64 yield stds of 1.8, 1.2, and 0.7, respectively.
8 for LiveCodeBench V5 and V6; and
4 for MATH500 and
4.2 BASELINES
Since our training begins with the base model Qwen2.5 --athh (Yang et al. 2024a), we primarily compare against state-of-the-art reasoning models built on either Qwen2.5 or Llama-3.1 (Grattationi et al.) 2024) of comparable parameter sizes to isolate the effects of pre-training and ensure a fair comparison. Our baselines include SFT models distilled from much larger frontier mod- els (e.g.. DeepSeek-R1). including Light-R1-7B (Wen et 2025), (Moshkov et al., 2025), and Llama-Nemotronn Nano-8B (Bercovich et al.] 2025), as well math-only RL-based models such as AReal-boba- (He et al.] 2025a), and AceReason-Nemotrue 1.0-7B (Chen et al. (025)) We also compare with sfate-of-the the 7B code LLMs, including OlympicCoder-777 (HuggingFace 2025) and OpenCoddeR (Ahmad et al. 2025).
7B (Guo et al., 2025). OpenMathReasoning.-
RL-7B (RL Lab| 2025). Skywork ORtorr-ORI-Mathh-
4.3 MAIN RESULTS
Table I shows the evaluation results on math and code benchmarks. Compared to other SET models through distillation, our SFT model achieves slightly better results than Llama- Jama-Nemotron-- Nano---n--Nao-88-
5

---

Published as a conference paper at ICLR 2026
AIME
MATH 500
HMMT 2025
BRUMO 2025
LiveCodeBench
EvalPlus
Models
2024
2025
avg
avg 64
avg@4
avg@64
avg@64
avg
mvg
avg44
SFT (distillation) based models:
Light R1-7B
59.1
44.3
92.41
27.67
40.67
Llama-Nemen Nemotronn-anoo-33
61.3
47.1
95.4
46.6
46,21
81.21
OpenMard-Nemesteee Nemotron-7B
74.8
61.2
OpenCudleReuorring Rew.eeennnnnenneeeeeeee NNmmotron--
51.3
46.11
83.4
RL. based models:
AReal boba RL-7B
61.9
48.3
93.8'
29.41
58.91
34.31
Skywork OR Math-B 7B
69.8
52.3
94.41
31.41
60.6
43.6
Skywork-OR1-77
70.2
32.0f
OlympicCoder-7B
37.1
79.8
MiMo 7B RL.
68.2
55.4
95.8
35.71
65.11
57.8
49.3
03-mini (low)
60.0
48.3
95.8
28.3
66.7
60.9
Magistral Small (24B)
70.7
62.8
95.9
55.8
47.4
DeepSeck RI Ri-Dislill Qwen- 7B
55.5
39.01
92.8
26.31
51,2
37.6
34.11
80.41
Naetaoonn-emttr------ 1.0-7B
69.0
53.6
94.1
33.9
62.2
51.8
44.1
84.6
Our SFT-7B (starting point of RL)
62.0
48.4
94.1
31.1
59.4
48.8
43.8
83.4
Nemotron
Table Evaluation of reasoning models primarily based on Qwen2.5-Math 7B and Llama-3.1 to disentan- gle the impact of pretraining. We report pass@ averaged over n generations (avg@n) following the DeepSeek- RI evaluation framework (same template, temperature=0.66 top-p=0.95, max response length=32768) By default, we include official numbers from the model developers if they are available. Otherwise, we eval- uate the model using the official template and same evaluation setting ahove. Note that, unlike the base model Qwen2.5 Math, MiMo-7B-RL is developed from hase model pretrained with extensive synthetic reasoning data from advanced reasoning model (Xia et al. 2025
vl, and achieves much better performance compared to Light-R1 and DeepScek-RI Biibcckk-R-Distlll Disttiil---Qen- 7B. It is worth mentioning that, Bi--sccc-R--issill-----a-e our SFT model, suggesting the high quality of our collected SFT samples.
is trained from Sowen2.5-Math--
we observe that the same AceReason RL training recipe (the same training method applied on the same training data) from Chen al. (2025) substantially improves the performance of our strong SFT model, yielding absolute score gains of 10.6% on AIME24, 16.4% AIME25, 8.4% on LiveCodeBench v5, and 8.3% on LiveCodeBench v6. As result, AceReason -aaissoon- 1.:nn-1-.111 1.1.7B, demonstrates superior performance on math and code reason- ing tasks, achieving the highest accuracy among 7B-scale models on AIME25 and LeveCodeBench v6 be-heechmarks that carry lower risk of contamination compared to their earlier versions. As reference, for AceReason-Nemane (eemotron-110-771 the same RL training recipe improves its starting SFT model, DeepSeek-R 1iii--R--Distti -waeiil---ee-BBBB. 13.5% on AIME24, 14.6% on AIME25, 14.2% on Live- CodeBench v5, and 10.0% on LiveCodeBench v6. This demonstrates that well-designed RL recipe can still largely boost the model's reasoning capability, even from much stronger SFT model.
For BecReason-Nemmorron-1
4.4
SCALING OF SET DATA CONSISTENTLY IMPROVES PERFORMANCE
1.202
Prompt
1,039
V7
Sf Avg Responses per Prompt
Figure 2: Log-scaled data statisties for the number math and code prompts and the total number of samples. Circle size and color indicate the average responses each prompt. Each SFT datasct consists of both math and code SFT samples.
Total T Samples (K)
590
v6
Total Samples (K)
582
210
V5
218.
v5
89
V4
76
V2
42
31
18
18:
5369 130 228
6076 127
Number of Math Prompts (K)
Number of Code Prompts (K)
Figure 2 presents the number of prompts, total samples, and average number of responses per prompt for each SFT dataset. Figure demonstrates the benefits of scaling SFT datasets, from 18K math and 18K code samples in version vl to 1.2M math and 1. 1.0M code samples in version v7. We observe

---

Published as a conference paper at ICLR 2026
4
AIME25
52
LiveCodeBeenc V5
UveCodrBeechWWW
Au ovin-11
Aecurnyy 11N
43.2
Accumnyy (90)
40.
42-
10
90
37.4 Gw0r73
15
0
32
3PT DolasetVorsa
SFT batazet Virsinn
SFT Eotasat
SFT Datasot Varsion
Figure 3: Accuracies AIME24, AIME25, and LiveCodeBench V5 and V6 for different SFT datasets. For
each SFT blend, the model trained until accuracy plateaus.
that both scaling strategies, namely increasing the diversity of prompts and adding more responses per prompt, significantly enhance model performance.
From the SFT dataset v1 to v4, we focus expanding the number of unique prompts while limiting each prompt to single response. Even modest number of unique prompt increases, such as adding 15K 20K math code samples, result in noticeable gains. For instance, expanding the math dataset by 16K samples from v3 to v4 leads to a 4% improvement in AIME24 and a 2% increase in AIME25. Starting with v5, we scale both the number of unique prompts and the number of responses per prompt. At this stage, the SFT v5 model matches the math performance of DeepSeek- RI-Distill-Owenn-7 and surpasses it in coding benchmarks. In our final dataset, v7, we maintain a similar number of prompts but further increase the number of responses per prompt. This additional scaling yields another performance boost, with AIME25 improving by 8% (from 41.3 to 49.3).
We examine which SFT data scaling factor more strongly influences performance and find that in- creasing the number of unique prompts has the greater impact. Detailed analyses are in Appendix C
In addition, we observe that SFT performance continues to improve across multiple training epochs, plateauing around the 5th-6th epoch. Detailed analyses are provided in Appendix
4.5
RL ANALYSES
4.5.1
RL STARTING FROM DIFFERENT SET MODELS
AIME24
Figure 4: Math-only RL. training start- ing from different (distillation) mod- cls. The AIME24 accuracy at step-0 re- flects the performance of the initial SFT checkpoints. The subsequent numbers in the figure show the final accuracy achieved the end of cach training stage; Math-Only Stage-1 (8K), Stage-2 (16K), Stage-3 (24K), and Stage-4 (32K).
70
RL from R1 Dist I Qwen 73 RL from Our SFT-78 vs
-114
67.4
50.8
70.0
80.0
65
64.7
60
61.4
65.5
55
55.4
65.4
50
750
2250
3000
Math-Only Training Steps
Figure presents RL experiments initialized from different SFT models, including two trained on We generally observe signif- icant performance gains at stage-2 (16K) and stage-3 (24K). The performance begins plateau at stage-4 (32K) for the model initialized from SFT-7B v7, as further gains become more challenging on top of an already strong model. It is worth mentioning that these performance gains are accom- panied by an increase in the average response token length at stage-2 and stage-3, suggesting that the model engaged in longer reasoning processes to tackle more difficult problems.
our SFT datasets v5 and v7, as well as Das-Seck-R--isstl---eeeer
Interestingly, we observe that while some SFT models show substantial performance gaps (e.g., between SFT-7B v5 and v7), these differences become much smaller after applying RL training over more steps. For instance, the performance gap decreases from an initial 6.6% to 1.6% on ALME24. This underscores the potential of RL to effectively enhance model performance and bridge the gap between different starting points. Additionally, when RL is applied to SFT models with similar starting performance, such as SFT-7B v5 and DeepSeck-B -iock-RI--iie R1-Distill- -wee---eennBB..- the resulting performance on AIME24 reaches similar level. We put the results for AIME25 in Appendix E
7

---

Published as a conference paper at ICLR 2026
4.5.2
How TRAINING TEMPERATURE AFFECTS THE PROGRESS OF RL
0.5
0.6
temperature
Lempperatiur
ATME24 avg0666
AIME2S argibbi
arg
VS
V6
tomperature
SFT Model (Step
2..4
Inference using lemperature 0.6
62,0
48.4
18.8
43.8
Inderence using tempeerttur 0.85
62.0
46.3
48,9
44.1
0.3
Inference using lemperiture 1.0
61.2
45.4
48.5
43.6
Muth- Stage Models
0.2
Trained temperature
52.4
50.1
45.6
Trained temperature 0.85
56.8
52.1
0.1
Trained with temperature 1,0
63.3
56.7
$1.5
45.7
Math Only
Figure 5: Left: Trajectories of temperature-adjuste entropy during RL training with different policy LLM temperature settings. Right: Impact of varying temperatures for inference and RL training. We observe that using a temperature of 0.6 for inference consistently yields better average results, and thus adopt 0.6 as the default inference temperature unless otherwise specified.
Figure S iillustrates how different training temperature settings affect the trajectory of entropy during RL training and model performance after training. We make the following two observations:
For a given model, the temperature in RL training should be carefully tuned-not too low or too high. A low temperature (e.g., 0.6 in Figure (5) leads to over-exploitation and limited exploration, which can ultimately result in sub-optimal performance. In contrast, high temperature (e.g., 1.0 in F causes excessive exploration and low initial rewards, followed by a reduction in entropy and hindered learning progress.
Through multiple trials, we observed useful rule of thumb: setting the training tem- perature such that the temperature-adjusted entropy remains around 0.3 typically leads to effective RL training, as it strikes good balance between exploration and exploitation.
Note that using temperature of 0.6 for inference consistently yields better average results; therefore, we adopt 0.6 as the default inference temperature across experiments unless otherwise specified. However, with training temperature of 0.6, RL begins with low entropy (around 0.15) and remains below 0.2 throughout the training, causing the policy LLM to favor exploitation over exploration. Such conservative learning behavior ultimately leads sub-optimal performance.
With a training temperature of 1.0, entropy starts high (around 0,4), which gradually drops to ap- proximately 0.22. This behavior may be attributed the poor performance of the SFT model when sampling with temperature of 1.0. As shown in Figure 5 (right table), we can see that using tem- perature 1.0 results in the poorest SFT models compared 0.85 and 0.6. At the early RL stage, we also observe that using a temperature of 1.0 results in average rewards that are roughly 3-4% worse than those achieved with a temperature of 0.85. While a higher initial temperature encour- ages greater "exploration". relatively low reward signal dampens this tendency, leading the model to favor "exploitation". as reflected sharper logits and consequent drop in entropy.
In contrast, moderate temperature of 0.85 starts with entropy of approximately 0.26, which gradually increases to around 0.38 over the course of training. We hypothesize that the relatively higher rewards associated with this setting-also reflected in the table on the right (inference with 0.85 performs only marginally worse than 0.6, but significantly better than 1.0) 1.0)-promote continued exploration, thereby driving the steady increase in entropy during training. This improved explo- ration ration-explonn eiolonnntiotttiionn trade-off results in the highest benchmark performance.
4.5.3 AT WHICH STAGE SHOULD WE APPLY OVERLONG FILTERING?
During our curriculum RL training, model responses are sampled within fixed response length budget (c.g., 8K or 16K tokens), and any outputs exceeding this limit are truncated. When response surpasses this predefined length, a key training consideration emerges; should we mask the whole sample without assigning any reward (i.e., "w/ overlong filtering") should such overlong outputs be penalized with a negative reward (i.e., "w/o overlong filtering")?
In previous studies, DAPO et responses could confuse the model, as a well-reasoned answer might be unfairly penalized simply for its excessive length. Instead, applying overlong filtering helps stabilize training and improves
2025 shows that under 16K token limit, penalizing truncated
8

---

Published as a conference paper at ICLR 2026
APP221
Cuttiong Bisiing
710
72.5
59.5
VT owr.ogg Eoor
01.0
V1 cocoerr Flarig
stegs
Figure 6: Ablation Studies on Math-Only RL training to assess the impact of overiong filtering. In both settings, Stage-1 starts with the same SFT model, and each subsequent stage begins with the same RL model from the previous stage trained under the best-performin the final stage (Stage-4), RL training without overlong filtering leads to superior performance. Evaluations performed with maximum sequence length of 32K. Results on AIME25 are in Appendix
setting (i.c., "w/ overlong filtering"). Notably, in
AIME24
AIME25
LCB V5 avg
LCB V6 avg
Table 2: Comparisons of the effects of increas- ing the maximum out- put length 64K, with and without applying the overlong filtering at the last stage of Math-One Oaly
avg@64 64
#v
Math-Only Stuge-4 RL w/ Overlong Filtering Inference using 32K Maximum Length
70.2
62.3
52.0
45.1
Inference using 64K Maximum Length
72.4
64.5
$3.5
45.7
Math-Only Stage-4 RI. w/o Overlong Filtering Inference using 32K Maximum Length Inference using 64K Maximum Length
71.4
63.5
53.5
48.0
73,0
64,8
$4.5
48.7
RL.
performance. contrast, Skywork-ORI (He et al. 2025b) observes no clear performance benefit from overlong filtering, although their findings are limited stage-1 1 (8K) RL training.
Compared to previous studies, we conduct a more systematic study for overlong filtering across all RL stages. Figure 6 illustrates the effects of "overlong filtering". Unlike Skywork-OR1, observe a notable benefit from applying overlong filtering in Stage-1 (8K). This is largely due to the relatively short 8K token limit, where the beginning of the training, approximately 30% of sample outputs exceed this boundary. Without overlong filtering, the negative rewards on these truncated samples will introduce significant noise into training. As the token limit increases, this noise diminishes-resulince equivalent performance in Stage-3 (24K). In Stage-4 (32K). RL training without overlong filtering outperforms the alternative. This is because removing overlong filtering makes the inference more token efficient and allows more concise generation within the 32K token budget.
in smaller gains from overlong filtering in Stage-2 (16K) and nearly
Table 2 further evaluates models under 64K maximum inference length. Interestingly, the model trained without overlong litterin-desspe generating more concise outputs-is still able to outper- form the model trained with overlong filtering, particularly coding benchmarks. These findings contrast with the conclusions DeepCoder (Luo et al., 2025a), which suggest that overlong filtering improves generalization to longer response lengths (e.g., 64K) during inference.
Additional analyses are provided in the Appendix due to space constraints. In Appendix and study the importance of stage- 1 RL and its appropriate training duration. Appendix pnpenddixiinvesttgaae the impact of math-only RI. on code reasoning performance. In Appendix [ Jand]f K we analyze how the SFT model, particularly for large k. Finally, Appendix L
RL improves pass@k accuracies explores how RL models perform on difficult problems compared to SFT models.
5 CONCLUSION
In this work, we study the training dynamics of supervised fine-tuning (SFT) and reinforcement learning (RL). We begin by analyzing SFT, examining the effects of scaling both the number of unique prompts and the number of responses per prompt. Our results show that both scaling strate- gies substantially improve the reasoning abilities of large language models (LLMss). Notably, per- formance consistently improves from the first to the fifth epoch during SFT, with gains plateauing around the fifth or sixth epoch- even when scaling the number of responses per prompt. We then conduct systematic study of applying RL across different SFT models. Despite large initial per- formance differences among these models, RL training significantly reduces the performance gap. We observe that the first stage of RL training may not yield immediate improvements and some- times degrade performance. However, plays a critical role in encouraging models to generate more concise reasoning, which proves beneficial in later stages of RI. training. Interestingly, even

---

Published as a conference paper at ICLR 2026
strong SFT models with robust coding abilities benefit substantially from math-only RL training. This leads to further gains in coding performance. Our final 7B model achieves stateoofttheaar re- sults among Qwen2.5-based 7B models, scoring 63.2% AIME25 and 52.8% on LiveCodeBench V5 Ve-dmoonttrtitng strong performance on challenging math and code benchmarks.
REFERENCES
Wasi Uddin Ahmad, Sean Narenthiran, Somshubra Majumdar, Aleksander Ficck, Siddhartha Jain, Jocelyn Huang, Vahid Noroozi, and Boris Ginsburg. Opencodereasoning: Advancing data distil- lation for competitive coding. arXiv preprint vrXiv:2504.01443 2025.
Mislav Balunovic, Jasper Dekoninck, Ivo Petrov, Nikola Jovanovic, and Martin Vechev. Matharena: Evaluating Ilms on uncontaminated math competitions, february 2025. URL https://mathrreaa ai, 2025.
Akhiad Bereovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran EI- El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najecb Nabwani, et al. Llama-Nemotroon Efficient Reasoning Mod- cls. arXiv preprint arXiv:2505.009: 00949, 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 2021.
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Acereason-neemotron:: Advancing math and code reasoning through rein- forcement learning. arXiv preprint wxXiv22505.16400 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14166 2021.
Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang. Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NVLM: Open frontier-class multimodal LLMs. arXiv preprint arXiv:2409.1 /402, 2024.
Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi. Jachyeon Kim, Wei Ping, Rafael Valle, Di- nesh Manocha, and Bryan Catanzaro. Audio flamingo 2: An audio-language model with long- audio understanding and expert reasoning abilities. arXiv preprint arXiv:2503.03983. 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178: 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, al. Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning, arXiv preprint arXiv:2501. /2948, 2025.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner series, 2025a. Notion Blog.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner technical report. arXiv preprint arXiv:2505.22312. 2025b.
Dan Hendrycks, Steven Basart, Sauray Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NewrlPS, 2021a.
Dan Hendrycks, Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Sort, (4):0-6, b021b.
10

---

Published as a conference paper at ICLR 2026
Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, JH Liu, Chenchen Zhang, Linzheng Chai, et al. Opencoder: The open cookbook for top-tier code large language models. arXiv preprint arX1v:2411.04905, 2024.
HuggingFace. Open rl: fully open reproduction of deepseek-rl, January 2025. URL https: weeithuu..com/uuggeg...
open-rl
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang. Bowen Yu, Keming Lu, et al. Qwen2.5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 2024.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang. Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. pnttps://githut com nwm.iamminnnaannaaa.amm wwwuuuia_aataseeeeeee dataset pdf)] 2024.
co AL/AAI-MO///A Numi Bwwaaaauaaaaaaataaaaaaaaa aimo
Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topies in algorithmic code generation dataset. arXiv preprint arXiv:2312. 14852,
2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-V. V3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://
openreview. wtrtt/forummeee fid= aid==iqxx60ccr
Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. Evalu- ating language models for efficient code generation. In First Conference on Language Modeling, 2024b. URL htt.:httss//:ppeeeeeeee
net forum? Semm:dddIIBBee... hwtp:/ruutteeaa
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.24866 2025.
Zihan Liu, Yang Chen, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. AceMath: Ad- vancing frontier math reasoning with post-training and reward modeling. arXiv preprint M.Xiv:2412.15084. 2024c.
Michael Luo, Sijun Tan, Roy Huang, Xiaoxiang Shi, Rachel Xin, Colin Cai, Ameen Patel, Alpay Ariyak, Qingyang Wu, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source 14b coder at 03-mini level, 2025a. Notion Blog.
Michael Luo, Sijun Tan, Justin Wong. Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL, 2025b. Notion Blog.
Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building ematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504 16891, 2025.
math-
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501. 2025.

---

Published as a conference paper at ICLR 2026
OpenAl. Learning reason with LLMs, 2024.
Qwen-Team. QwQ-32B: Embracing the Power of Reinforcement Learning, 2025. URL https:
//qwenlm. wtbpb/gg////www/ 32b/
Ant Research RL Lab. Arcal:
Ant reasoning rl.
AReaL, 2025.
ByteDance Seed, Yufeng Yuan, Yu Yue, Mingxuan Wang, Xiaochen Zuo, Jiaze Chen, Lin Yan, Wenyuan Xu, Chi Zhang, Xin Liu, et al. beed-thinking-vl.. Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504. 13914, 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. ReepseekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 2iXi:2240220330000 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An. Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Li: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond. arXiv preprint arXiv:2503. 2025.
Bingquan Xia, Bowen Shen, Dawei Zhu, Di Zhang, Gang Wang, Hailin Zhang, Huaqiu Liu, Jiebao Xiao, Jinhao Dong, Liang Zhao, et al. Mimo: Unlocking the reasoning potential of language model-from pretraining to posttraining. arXiv preprint arXiv:2505. 00550007668, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint a5X1w:2412.15115, 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-Math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. 2024b.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Qiying Yu. Zheng Zhang, Ruofci Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint 2xxiv:2503.144777 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models, arXiv preprint arXiv:2504.10479 2025.
12

---

Published as a conference paper at ICLR 2026
A
INSTRUCTION FOR EVALUATION
A.1 MATH
Please place your final answer inside \boxed().
A.2 No STARTER CODE (PYTHON)
Write Python code to solve the problem. Please place the solution code in the following format:
"python
Your solution code here
A.3 HAS STARTER CODE (PYTHON)
Solve the problem starting with the provided funct header.
Punction header:
<starter_code>
Please place the solution codc in the following format:
"python
Your solution code here
B SFT DATASET RESPONSE TOKEN LENGTH DISTRIBUTION
Porcentage of prompts (8S
Percentage of Prompps (5)
Token Longth
Longth
Figure 7: Response token length distributions for the math SFT dataset (left) and the code SFT dataset (right).
Figure 7 presents the response token length distributions for math and code prompts.
C WHICH SFT DATA SCALING FACTOR HAS LARGER IMPACT
We further analyze which SET data scaling factor contributes significantly performance improvements: increasing the number unique prompts or increasing the number of responses per prompt. To this end, we perform a multiple linear regression analysis model the relationship between overall accuracy (z) and the two independent variables: the number of unique prompts (x) and the number of responses per prompt (z), as described by the following equation:
log, x. + b log,
We apply the least squares method fit this model using seven data points (Figure and estimate the regression coefficients a and b, and bias c. Prior to fitting, x and are tranformed to log base-2 scale to reflect the exponential growth in total samples versus the linear trend in accuracy. Then,
13

---

Published as a conference paper at ICLR 2026
and y standardized (zero mean and unit variance) to ensure they the same scale. The de- pendent variable 2 defined as the average accuracy across AIME24, AIME25, and LiveCodeBench V5 and V6.
The resulting estimates are a = 4.831 and = 2.635, with the coefficient of determination H² 0.989, indicating strong fit. The larger value of compared to b suggests that increasing the number of unique prompts may have a greater impact on SFT model performance than increasing the number of responses per prompt. Given that when the number of unique prompts reaches certain amount, increasing it becomes generally harder due to the difficulties of collecting more diverse data sources. In this case, increasing the number of responses for each prompt serves practical alternative to boost the performance of SFT model.
D SFT PERFORMANCE IMPROVES PROGRESSIVELY OVER EPOCHS
AIME24
LiveCodeBench V5
62
50
60
Accuracy 06
48
58
56
44
SFT Dataset v6 Dataset v7
42
SFT Dataset v6 SFT Dataset v7
52
40
2
Epoch
Epoch
Figure 8: Accuracies over different epochs of training for SFT dataset v6 and v7.
Figure illustrates the accuracies AIME24 and LiveCodeBench V5 over different epochs of training for SFT dataset v6 and v7. We observe that the model's performance gradually improves from the 1st the 5th epoch, and begins plateau around the 5th 6th epoch. This pattern consistent across different versions of the SFT dataset. This suggests that a certain degree "overfitting" may actually enhance test accuracy long chain-of- of-thought (CoT) generation. likely due exposure bias in autoregressive models. Additionally, we find that models trained on dataset v7 consistently outperform those trained on dataset v6 throughout training. While both versions share similar set of prompts, dataset v7 contains nearly twice many responses per prompt on average. Despite being exposed to the same prompts more frequently, models trained on v7 also reach performance plateau around the fifth to sixth epoch. This suggests that increasing the number of responses per prompt in the SFT dataset also greatly benefits from multi-epoch training.
E RL TRAINING FROM DIFFERENT SET MODELS ON AIME25
Consistent with the results shown in Figure 4 we observe from Figure 9 that the final performance gaps between RL models initialized from SFT-7B-v5 and SFT-7B-v7 narrows after training. How- ever, substantial gap remains when comparing RL training from SFT-7B-v7 to DeepSeek-RL. Distill-Qwen-7B. This suggests that when the initial SFT models differ significantly quality, RL has limited ability to further close the performance gap.
F AT WHICH STAGE SHOULD WE APPLY OVERLONG FILTERING?
Extra results on AIME25 can be found in Figure 10 and the conclusion aligns with our findings AIME24.
G IMPORTANCE OF STAGE-1 (8K)
We can find in Figure that during Math-Only Stage-1 (8K), the model exhibits an initial drop in performance followed by recovery. Nevertheless, its final performance the end of Stage-1 may
14

---

Published as a conference paper at ICLR 2026
AIME25
RL from R1 Distill Qwen-78 7B
63.5
60
RL from Our SFT-7B v5
57.2
59.2
Accuracy (%)
RL from Our SFT-7B v7
55
56.7
53.6
50
48.4
44.6
53.1
48.8
45
40.8
40
42.6
39.0
39.0
35
750
1500
2250
3000
3750
Math-Only Training Steps
Figure 9: Math-only RL training starting from different SFT (distillation) models. The AIME25 accuracy at step-0 reflects the performance of the initial SFT checkpoints. The subsequent numbers in the figure show the final accuracy achieved the end of each training stage: Math-Only Stage-1 (8K), Stage-2 (16K), Stage-3 (24K), and Stage-4 (32K).
critiog Eisenp
61.
265
E5 EEC
2000 120
226
305
100
335 156
103
250
305
335
100 155
205
256
Figure 10: Ablation Studies on Math-Only training assess impact overlong filtering. In both settings, Stage-1 starts with the same SFT model, and each subsequent stage begins with the same RL. model from the previous stage trained under the hest-performing setting (i.c., "w/ overlong filtering"). Notably, in the final stage (Stage-4), RL training without overlong filtering leads to superior performance. Evaluations performed with maximum sequence length of 32K.
remain below its initial level. Given the lack of clear improvement during Stage- 1, natural question arises: should Stage-1 be omitted?
To investigate this, we compare Stage-2 (16K) training initialized from the SFT checkpoint (omit Stage-1 (8K)) versus from the final step of Stage- as shown on the left side of Figure II We observe that although Stage-1 training causes initial drop in AIME25 performance from 48.4 to 44.6, enables more rapid and consistent improvement during Stage-2. contrast, skipping Stage- 1 results in slower gains and eventual plateau at lower performance level (56.7 vs. 51.8). As shown on the right side of Figure II we observe that the average response length sharply declines from over 5000 tokens around 4000 tokens approximately step 600 during stage-1 training. This reduction, occurring between steps 0 and 600, is attributed to the 8K token length limit and coincides with period where the model performance drops. Although continued training after 600 steps improves performance, the average response length does not go up.
We conjecture that the 8K token constraint compels the reasoning model to develop more concise thinking process in order to complete responses within the limit and successfully solve the tasks. One motivation for compressing the reasoning process is that the reacher-foreed responses used during SFT are generated by PoppSeck-R1-6111 to effectively learn from or reproduce on their own. This reasoning compression becomes especially valuable during subsequent RI. training, where the model maintains conciseness but benefits from longer sequence length, allowing it to tackle more complex problems. In contrast, training an SFT model directly from stage-2 (with 16K limit) bypasses this reasoning compression stage and thus fails to learn such capability.
and may be too lengthy and complex for smaller models
H How LONG SHOULD WE TRAIN STAGE-1
From Appendix G we learn that the stage-1 1 (8K) plays an important role in RL training. Building on this insight and noting that model performance tends to recover and improve with continued
15

---

Published as a conference paper at ICLR 2026
Resporge Token Length (K)
Accuracy AIME25 (96
w/o Stage-1
5.5
Stage-1
54
51.8
52
50
4.5
48
4.0
46
44.6
3.5
44
200
400
600
800
200
400
600
800
1000 1200
Math-Only Stage-2 (16K) Training Steps
Math-Only Stage-1 (8K) Training Steps
Figure 11: Left: Ablation study comparing models trained with and without Math-Only Stage- For "w/o Stage-1", 1", the step-0 accuracy reflects the performance Stage-1", the step-0 accuracy represents the final performance of Stage-1 RL initialized from the same SFT model. Right: Average response token length during Math-Only Stage-1 (8K) RL training.
our SFT model on AIME25. contrast, for "w/
AIME24
AIME25
LCB V5
LCB V6 avg@8
avg 64
avg@64
avg@8
Our SFT Model
62.0
48.4
48.8
43.8
Stage-1 1200 steps Stage-2 (16K)
61.3
44. (4.6))8)))
48.4
43.7
65.3(3.3))
56. (6.37)
51.6
45.7
Stage-1 1600 steps Stage-2 (16K)
61.6:0.4))
46.5(1.9))
49.4
44.5
66.3(43))
56. 7(.33)))
51.8
45.6
Stage-1 2300 steps
63.2(121)
66.8(1664)
50.1
44.9
Stage-2 (16K)
66.1
54.8(6.49)
51.8
45.6
Table 3: Studies examining how varying the number Stage-1 training steps (e.g., 8K) impacts subsequent RL training. The arrow indicates the accuracy comparison between our RI. models and initial SFT model on math benchmarks.
stage-1 training--va wwe investigate whether extending the duration stage-1 training further enhances the subsequent RL stages.
the results of this investigation, showing how varying the number stage-1 training steps affects final performance at stage-2. We observe that continuing stage-1 training beyond 1200 steps yields slight additional improvements, with models gradually matching or even surpassing the original SFT model math and coding benchmarks. However, these gains do not consistently translate to better outcomes at the final performance of stage-2. Specifically, models initialized from the stage-1 checkpoints 1200, 1600, 2300 steps result in comparable stage-2 performance. We also find that all stage-2 training plateau at similar training steps. This suggests that we could stop stage-I I training early and transition 2tage-2, which leads more substantial improvements.
I MATH-ONLY RL SIGNIFICANTLY IMPROVES CODE REASONING
LiveCodeBench
54
52
.1.6
11.2
51.2
50
49.3
Accuracy
48
RL from SFT v5
44
43.3
RL from SFT v7
SFT
Stage-1
Stage-2 Stage-3 Math-Only RL Stages
Stage
Figure 12: LiveCodeBench V5 accuracy over different Math-Only RL stages.
16

---

Published as a conference paper at ICLR 2026
1.0 (Chen et al. 2025), found that math-only RL significantly improves performance on code reasoning benchmarks. We reaffirm this finding using different SFT model as initialization, which stronger than DeepSeek-RI. -i----Riilll--ww 1.0. To explore this further, we examine the impact of each math-only RI training stage on coding performance. Figure T2]reveals that the majority of the improvement stems from Stage-2, and Stage- I can even potentially help improve coding capability for relatively weaker SFT model. Similar to our observations on math benchmarks, Stage-4 yields minor gains in coding performance.
In
used in AccReason-Nemottr
Notably, the performance gap in coding narrows considerably over the course of RL training-from an initial 5.5% the SFT starting point to just 1.6% by Stage-3. This trend mirrors the results observed on math benchmarks, further underscoring the powerfulness of RL training in enhancing model capabilities.
J RL IMPROVES UPON THE SFT MODEL IN TERMS OF PASS @ EVEN WHEN K IS LARGE
AtcRRason-Nemottrnn-111177
AIME24
AIME25
90
86
Pass@kk (9)
@ (9%)
87
83
84
77
81
32
64 K (# of candidates)
128
K (# of candidates)
LiveCodeBench Vs
LiveCone V6
79
75
C 76
0 72
73
66
70
& 63
67
60
64 8
57
32
64 (# of candidates)
128
32
64 (# of candidates)
128
Figure 13: Comparison of pass @K K scores between PenReasonnNemmoton-...i it is trained from. To compute pass @ K, we generate 256 outputs per sample for AIME24 and AIME25, and 128 outputs LiveCodeBench procedure is repeated 100 times, and the final pass@ K score is computed by averaging the results across all repetitions.
and the SFT-7B v7 model
and V6. then randomly select outputs, evaluate pass K. This
The pass@ K results from AceReason-Nemootron-11 (Chen al. 2025) indicate that RL can con- sistently enhance pass @ K with increasing K. As shown in Figure 13, our experiments on math and code benchmarks reveal that RL training consistently improves @
@k accuracy across the range of K = 8 to K = 128. These findings align with those of AceReason-Nemtt model being initialized from considerably stronger SFT baseline (e.g.. SFT-7B v7) compared to the DeepSeck-RR - --istilll-wwn-P11 model used in their study.
1.0, despite our RI.
In addition, we observe that for the math benchmark AIME25, the improvement from RL diminishes as K increases. For instance, the performance gain decreases from 8.3% at K = 8 to just 1.2% at K = 128. This due to AIME's answer space is quite limited, consisting solely of positive integers with maximum value in the hundreds. In contrast, the gains coding benchmarks such as LiveCodeBench V5 and V6 remain substantial, with LiveCodeBench V6 still showing a 5% improvement K 128.
17

---

Published as a conference paper at ICLR 2026
SFT v5
SFT v7
RL from SFT v5
RL from SFT V7
AIME24
AIME25
90
88
Pass (%)
88
Pass@k (90
83
78
73
68
80
63
64
32
128
K (#f of candidates)
K (# of candidates)
LiveCodeBench V5
LiveCodeBench V6
75
Pass@k (%)
75
(99
70
65
70
60
65
55
60
50
8
32
64
128
8
32
64
128
K of candidates)
K of candidates)
Figure 14: Pass@k results AIME24, AIME25, LiveCod<Bench V5, and V6, showcasing two SFT models their subsequent Math-Only RL-trained versions. To compute pass@k, we generate sample for AIME24 and AIME25, and 128 outputs for LiveCodeBench V5 and V6. We then randomly select k outputs, and evaluate pass pass@k. @k This procedure is repeated 100 times, and the final pass @k score is computed by averaging the results across all repetitions.
outputs per
K PASS@K ACCURACY ON MATH-ONLY RL MODELS
In Figure 14 we demonstrate the pass@k results on two SFT models and their subsequent math-
only RL-trained models. We find that the results are consistent with the observations in Appendix [ Similar to the pass@1 results, we observe that RL from weaker SFT model (i.e., SFT v5) leads to greater improvements estingly, we find that even though the RL training is performed solely on math math--seccific prompts, the resulting models exhibit consistent gains coding benchmarks (LiveCodeBennh V5 and V6) increases.
@ass@k, which further highlights the effectiveness of RL training. Inter-
L RL IMPROVES OVER STRONG SFT MODEL BY SOLVING HARD PROBLEMS
CiReasson-Nemotron--i (Chen et al. 2025) find that RL training is able to unlock a long tail of --sSSsei-R--isttll-------ee
hard coding problems that the starting SFT model, within 64 or even 1024 attempts. In particular, RL training notably boosts performance on problems where the SFT model achieves less than 20% accuracy. As shown Figure [5] we observe that this finding still holds true, even though our initial SFT model is substantially stronger than the one used SccReeason-Nemotte CeRkesson-Nemotroon 1.1BBB is able to tackle long tail of hard coding problems that the SFT model fails to solve within 128 attempts, leading to over ten additional problems solved on both LiveCodeBench V5 and V6.
fails solve
(Chen et al., 2025). Notably, on LiveCodeBench, we observe
M COMPUTATIONAL COST OF TRAINING
For SFT training, the total compute cost is 11,000 H100 GPU hours (458.3 GPU days).
For RL training, the total compute cost is 23,944 H100 GPU hours (997.7 GPU days), with the following breakdown:
Math-only Stage-1: 2,773 H100 GPU hours (115.5 GPU days);
Math-only Stage-2: 3,312 H100 GPU hours (138.0 GPU days);
18

---

Published as a conference paper at ICLR 2026
AIME24
AIME25
80
80-
Accuracy
Accuracy
40
SFT V7
40
SFT V7
RL. improvements
20
AL. improvements
Degradations
Degradations
Problem instances
Problem Instances
LiveCooeBench v5
LiveCodeSench V6
1000
100
80
Accuracy
Accuracy
60
40
20
RL Degradations
RL Degradations
instances
Instances
Figure 15: Comparison of mroblem-level solving rates between AceReason DeceRasoo-Nemootron.. and the SFT-7B v7 model is trained from. For each problem, accuracy averaged over 256 outputs for AIME24 and AIME25, over outputs for LiveCodeBench V5 and V6.
Math-only Stage-3: 5,084 H100 GPU hours (211.8 GPU days);
Math-only Stage-4: 9,458 H100 GPU hours (394,1 GPU days);
Code-only Stage-I: 1,875 H100 GPU hours (78.1 GPU days);
Code-only Stage-II: 1,442 H100 GPU hours (60.1 GPU days)
19


Reviews:
[{"review_id": "UtgEsVqwW5", "reviewer": "Reviewer_aouA", "summary": "The paper studies how supervised fine-tuning (SFT) and reinforcement learning (RL) interact to improve long chain-of-thought reasoning for math and code. It scales SFT along two axes with more unique prompts and more responses per promptand analyzes epoch effects. It then applies a stage-wise, strictly on-policy GRPO RL curriculum (math stages at 8K -> 16K-> 24K-> 32K tokens, interleaved with code RL at 24K and 32K) and examines starting-point sensitivity, sampling temperature, and handling of over-length trajectories. The resulting 7B model, AceReason-Nemotron-1.1, built on Qwen2.5-Math-7B, reports improved results on AIME24/25, MATH500, HMMT2025, BRUMO2025, EvalPlus, and LiveCodeBench v5/v6, with analyses of temperature-adjusted entropy and overlong filtering.", "questions": "See Weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- This is a well-executed empirical paper that systematically probes how supervised fine-tuning and reinforcement learning interact for long chain-of-thought reasoning, with clear takeaways (e.g., entropy/temperature rule-of-thumb, stage-wise curriculum, and overlong-filtering policy). The paper offers generalizable training guidance.\n\n- The empirical program is rigorous and targeted: clearly defined questions, extensive ablations, and transparent evaluation settings.\n\n- The model is evaluated on a wide range of mathematical and coding benchmarks, using repeated sampling avg@n for variance reduction and consistent default decoding across all models.", "weaknesses": "- Please report GPU-days per stage to contextualize the recipe’s practicality.\n- Adding one/more non-Qwen-7B model would validate generality.\n- The authors did not release code, which may limit reproducibility.\n- In Figure 4, why is the “final accuracy achieved at the end of each training stage” for different models not shown at the same step? Does this mean that different models require a different number of training steps in a given stage?\n- Regarding the results in Figure 4, have you conducted experiments where RL is applied directly to the base model (skipping SFT)? Would this lead to similar conclusions? Additionally, the reviewer is also curious whether applying RL directly on an existing instruct model would produce more interesting comparison results.\n- During different stages of RL training, is there any overlap between the samples used in each stage? How many samples were used in total for each stage?\n- How is the number of training steps for each RL stage determined? What are the exact numbers?\n- In the SFT experiments on the number of responses per prompt, does the diversity of the responses per prompt have a significant impact? When sampling multiple responses, how do the authors ensure diversity?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "298BusVXy3", "reviewer": "Reviewer_dEtu", "summary": "This paper conducts a systematic analysis of the alternating interplay and integration between SFT (Supervised Fine-Tuning) and RL (Reinforcement Learning) training. It explores how expanding the SFT dataset improves initialization — including the number of prompts, the number of responses per prompt, and the number of training epochs. The authors also investigate the impact of using different SFT models for initializing RL, as well as how varying the temperature during RL training affects entropy and accuracy. In addition, they examine whether filtering overlong samples during RL affects the outcomes. Through detailed experiments, the authors study all these questions and demonstrate that SFT based on Qwen2.5-7B surpasses DeepSeek-R1-Distill-Qwen in performance. Furthermore, they confirm that stage-wise RL can achieve further improvements on stronger SFT models.", "questions": "See in Weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Compared with most technical reports that provide only brief descriptions of SFT and RL details, this paper conducts a systematic study on the interplay and integration between SFT and RL using extensive resources. This work is highly significant for understanding the training dynamics of RL initialized from SFT in the research community.\n\n2. The authors analyze SFT data in detail — including the number of prompts, the number of responses per prompt, and their effects on SFT performance — across both Math and Code domains. They also study how different SFT models influence RL training dynamics, offering valuable insights. For example, scaling the number of prompts yields more significant improvements, and stronger SFT models lead to higher RL starting points, though the performance gap narrows as RL training progresses.\n\n3. The authors analyze how different temperatures in RL affect entropy and performance, experimentally illustrating healthy entropy dynamics and corresponding temperature settings during RL training.\n\n4. They further study the impact of overlong sample filtering, providing detailed analyses under various maximum-length settings.", "weaknesses": "The experiments are mainly conducted on Qwen2.5-based 7B models. It is unclear whether the authors tested larger models, such as 32B, to verify the generalizability of their conclusions. However, given the large-scale data and the computational resources required, the absence of such experiments is understandable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bz7zheKyPQ", "reviewer": "Reviewer_3y3R", "summary": "This work presents a comprehensive study of SFT and RL training on a 7B reasoning model. The authors investigate different aspects, including the impact of SFT model on RL performance, the sampling temperature and overlong penalty during RL training. The whole RL training recipe starts from math training to code training, with a short-to-long generation length. The resulted model achieves competitive results on both math and code benchmarks.", "questions": "1. Why does the authors choose to apply RL training on math and code in seperate stages, instead of apply RL training on math and code data simultaneously?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a comprehensive SFT&RL recipe for a 7B reasoning models.\n2. Ablation studies from different aspects are conducted, making this work as a solid technical work and providing valuable empirical insights.\n3. The resulted model achieves competitive performance on both math and code benchmarks.", "weaknesses": "1. The question \"Is theMath-onlyStage-1(8K) trulynecessary\" is not answered in the submission.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "990L6Q5YLQ", "reviewer": "Reviewer_hZwB", "summary": "This paper studies the synergy between supervised fine-tuning and reinforcement learning in math and coding domain and and introduces AceReason-Nemotron-1.1-7B model that achieves high performance on math and coding problem. More specifically, they study the effects of scaling SFT data by increasing both prompt diversity and the number of responses per prompt. And they then apply a staged RL curriculum with increasing context length, analyzing how SFT strength, sampling temperature, and overlong response handling affect learning.", "questions": "I have raised most of my concerns in the Weaknesses section, and I would like the authors to address them. Below are a few additional questions:\n\n1. In Lines 396–398, the authors suggest keeping the entropy around 0.3. However, according to Figure 5, the entropy later deviates significantly from 0.3 and approaches 0.4. Could the authors clarify why this happens, and whether this suggestion is universal across settings (e.g. models, datasets)?\n\n2. The paper conducts math and code training in separate stages. Have the authors considered joint training that mixes math and code signals instead of a staged approach?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper collects a large amount of SFT and RL data and conducts thorough preprocessing. In addition, it performs extensive SFT and RL experiments along with many ablation studies, and evaluates the model across comprehensive math and coding benchmarks. Overall, the experiments are fairly thorough.\n\n\n2. The final model achieves leading performance on multiple metrics compared to models of similar size, demonstrating the effectiveness of the proposed approach.", "weaknesses": "1. Although this paper features a very large workload and extensive experiments, I believe the generalization of some of its findings and conclusions is questionable. Specifically, all experiments are conducted on Qwen2.5-Math-7B and focus on math and coding generation tasks. From the model perspective, multiple works [1][2] have already pointed out the uniqueness of Qwen2.5-Math-7B, so the findings of this paper (e.g., sampling temperature choices) may not transfer to other models. From the task perspective, the experiments are primarily on math and coding. Would these conclusions still hold for more challenging tasks such as web agents or SWE-style agent tasks?\n\n2. More broadly, the findings feel more like a report on tuning methods and hyperparameters for a particular model and task type, rather than offering deeper empirical and theoretical insights that generalize to different settings in the community. Therefore, I hope the authors can clearly distinguish which conclusions are generalizable and which only apply to the specific setup studied in the paper.\n\n3. In the RL training phase, the authors divide the process into many stages, but no clear justification is provided. In Line 196, the authors cite AceReason-Nemotron-1.0 and follow a similar setup, yet do not explain the rationale behind these design choices in more detail.\n\n[1] Zeng, Weihao, et al. \"Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.\" arXiv preprint arXiv:2503.18892 (2025). \n\n[2] Shao, Rulin, et al. \"Spurious rewards: Rethinking training signals in rlvr.\" arXiv preprint arXiv:2506.10947 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)