Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
Authors: Xumeng Wen
Zihan Liu
Shun Zheng
Shengyu Ye
Zhirong Wu
Yang Wang
Zhijian Xu
Xiao Liang
Junjie Li
Ziming Miao
Jiang Bian
Mao Yang
Venue: ICLR 2026 Poster (2026)
Keywords: large language models
reasoning
reinforcement learning
Primary area: foundation or frontier models, including LLMs
TL;DR: We show theoretically and empirically that reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base LLMs

Abstract: Recent advancements in long chain-of-thought (CoT) reasoning, particularly through the Group Relative Policy Optimization algorithm used by DeepSeek-R1, have led to significant interest in the potential of Reinforcement Learning with Verifiable Rewards (RLVR) for Large Language Models (LLMs). While RLVR promises to improve reasoning by allowing models to learn from free exploration, there remains debate over whether it truly enhances reasoning abilities or simply boosts sampling efficiency. This paper demonstrates the profound impact that RLVR has on the reasoning capabilities of LLMs. We revisit Pass@K experiments and show that RLVR can extend the reasoning boundary for both mathematical and coding tasks. This is supported by our introduction of a novel evaluation metric, CoT-Pass@K, which captures reasoning success by accounting for both the final answer and intermediate reasoning steps. Furthermore, we present a theoretical framework explaining RLVR's incentive mechanism, demonstrating how it can encourage correct reasoning even when rewards are based solely on answer correctness. Our analysis of RLVR's training dynamics reveals that it incentivizes correct reasoning early in the process, with substantial improvements in reasoning quality confirmed through extensive evaluations. These findings provide strong evidence of RLVR's potential to enhance LLM reasoning, offering valuable insights into its mechanisms and performance improvements.

Published as a conference paper at ICLR 2026
REINFORCEMENT LEARNING WITH VERIFIABLE RE- WARDS IMPLICITLY INCENTIVIZES CORRECT REA- SONING IN BASE LLMs
Xumeng Wen"1 1 Zihan Liu Shun Zheng 11 Shengyu Yel Zhirong Wu', Yang Wang', Zhijian Xut3, Xiao Liang Junjie Li Ziming Miao', Jiang Bian', Mao Yang'
Microsoft Research Asia Peking University
The Chinese University of Hong Kong *University California, Los Angeles
ABSTRACT
Recent advancements in long chain-of-thought (CoT) reasoning, particularly through the Group Relative Policy Optimization algorithm used by DeepSeek-R1 1,, have led significant interest in the potential of Reinforcement Learning with Verifiable Rewards (RLVR) for Large Language Models (LLMs). While RLVR promises to improve reasoning by allowing models to learn from free exploration, there remains debate ply boosts sampling efficiency. This paper demonstrates the profound impact that RLVR has the reasoning capabilities of LLMs. We revisit Pass K experiments and show that RLVR can extend the reasoning boundary for both mathematical and coding tasks. This supported by our introduction of novel evaluation met- ric, CoT-Pass @ K, which captures reasoning success by accounting for both the final answer and intermediate reasoning steps. Furthermore, we present theo- retical framework explaining RLVR's incentive mechanism, demonstrating how it can encourage correct reasoning even when rewards are based solely on answer correctness. Our analysis of RLVR's 's training dynamics reveals that it incentivizes correct reasoning early in the process, with substantial improvements in reasoning quality confirmed through extensive evaluations. These findings provide strong evidence of RLVR's s potential to enhance LLM reasoning, offering valuable in- sights into its mechanisms and performance improvements.
whether it truly enhances reasoning abilities sim-
INTRODUCTION
The successful replication of long chain-of-thouse (CoT) reasoning, similar to that in OpenAl's ol (OpenAl 2024), by DeepSeek-R1 (Guo et al.) 2025) using the Group Relative Policy Optimiza- tion (GRPO) algorithm (Shao et al. 2024). has sparked surge of interest within the open research community. This interest is focused understanding, reproducing. and extending DeepSeek's ap- proach, as evidenced by multitude of recent studies (Liu et al. 2025b Hu et al.) 2025 Zeng et al., 2025 Yu al. 2025 He et al. 2025 Wen al.) 2025 Chen et 2025b). Fundamentally, this emerging paradigm is a form of Reinforcement Learning with Veritiable Rewards (RLVR) (Lambert
al. 2024 Guo al. 2025 Yue et 2025), where a Large Language Model (LLM) acts as pol- icy, generating a CoTl as sequence of actions and receiving feedback on answer correctness from deterministic verifiers. This paradigm holds the promise endowing LLMs with the ability to learn from experience through free exploration, potentially leading to unlimited intelligence (OpenAl
2024
Guo et al. 2025 Silver
Sutton 2025).
However, emerging concerns question the true effectiveness of RLVR. These concerns are motivated by the observation that some post-RLVR models improve the Pass@ metric but fail to enhance the model. This phenomenon was first noted by Shao et al. (2024) during the development of GRPO. Subsequently, a systematic study by Yue et al. (2025)
Pass K metric compared to the base
'These authors contributed equally: Xumeng Wen, Zihan Liu. Shun Zheng.
1 Work done during the internship at Microsoft Research Asia.
Cooresponneece shun.zhengde mierosoft.com.

---

Published as a conference paper at ICLR 2026
Reinforcement Learning with Verifiable Rewards (RLVR)
Hypothests Explalning Pass81 Pass8K Ali reasoning paths are present in the base model. in yit improves samplng aflicweny. AL roduces reasoning capacity.
AIME 2024
3.0
AIME 2025
Base LLM
After ALSH
Pascuggh
0.5
0.4
0.2
0.2
Question
Sampling Number [K)
Sampling Number (K)
Chain of Thought (Car)
1.0
AIME 2024
3.0
AIME 2025
0.8
0.5-
Answer
0.6
Perspective:
RLYR Emplictll Incentivizes Correct Reasoning promotes ALVI minggates spuriour guesses.
Base LLM
0.2
paths.
After R15R
0.0
25 2 27 2- 2" 2"
0.0
27
2
Sampling Number [K]
Sampling Number (8)
Figure I: An illustration of our perspective: RLVR implicitly incentivizes correct reasoning in base LLMs. We visualize how different explanation frameworks lead to varying reasoning paths being activated, with our perspective shown in the lower left and recent popular hypoth- esis explaining Pass@K K observations (ue et al., 2025) summarized in the upper left. In this diagram, the line width represents the sampling probability of a reasoning path, while the color distinguishes correct paths (green) from incorrect ones (red). If all reasoning paths after apply- ing RLVR are already present in the base model, the reasoning model merely adjusts the sam- pling probabilities of these existing paths (visualized in dashed lines). This hypothesis effec- tively accounts for the key observation shown in the upper-right part, where, for a moderately large K, a base LLM can catch up to the reasoning model after RLVR using the Pass K met- ric. In this study, unveil the extended reasoning capability boundary in math tasks using refined metric, CoT-Pass@ K, which emphasizes both the correctness of answers and the validity of reasoning CoT's. Verification results supporting this figure are publicly available https: //tuggingfacc.cc/////oo/o/
/ 25 25_CoT______ Veri mmerificaiion
on more open-weight RLVR models discovered that the Pass@ K metric of the base model increases much faster rate than its RLVR-tumed counterpart. Consequently, for moderately large K, the base model eventually matches and surpasses the reasoning model. This led to their adventurous hypothesis: all correct reasoning paths are already present in the base model, and RLVR merely improves sampling efficiency at the cost of reducing overall reasoning capacity.
While this hypothesis has gained significant support eZhuet 2025 [hhng et al., 2025 Wang et al., 2025a Chen et al. 2025a), conflicting observations have also been reported. For instance, Liu et al. (2025a) detected the emergence of new reasoning patterns after RLVR, while they also acknowl- edged loss in reasoning capacity measured by Pass@K K. Chen al. (2025b) reported persistent improvements in Pass @K K for competitive coding tasks but did not show improved Pass@K K for math tasks. Shojaee et al. (025) observed similar Pass K observations different patterns on puzzles with high complexity. To the best of our knowledge, no systematic explanation exists to reconcile these contradictory findings, leaving a critical question unanswered: "should we accept the hypothesis as a fundamental limitation of RLVR or should we trust new em- pirical findings that challenge the hypothesis?"
math datasets but found
In this work, we address this debate systematically and demonstrate that RLVR can fundamentally enhance the reasoning abilities of LLMs. First, we revisit Pass@ K experiments and unveil the existence of extended reasoning capability boundaries after RLVR for both math and code tasks. In addition to reproducing the extended reasoning boundary in competitive coding, as reported byChe et al.] (2025b), we find that the Pass K performance of base LLMs on math reasoning can be unreliable, as base LLMs are capable of producing incorrect CoTs yet coincidentally arriving at the ground truth, especially for hard mathematical questions where answers are simple and can be easily guessed after multiple attempts. To address this, we introduce a new metric, CoT-Pass @ K, which evaluates success only when both the final answer and the intermediate reasoning CoT are correct. In practice, we verify the correctness of mathematical CoTs by instructing DeepSeek- -iik--R--05 R1-0528-0000 -wee-- 8B (DeepSeek) 2025) and confirm their reliability. Using this new metric, we successfully identify
2

---

Published as a conference paper at ICLR 2026
the extended reasoning boundary of post-RLVR model for math tasks. Figure summarizes key perspectives.
Moreover, develop theoretical framework to explain why RLVR works, even when base LLMs may guess the ground truth and only answer correctness provided as a reward, and how RLVR incentivizes correct reasoning. Our central insight is that once LLMs have been pre-trained tablish strong knowledge and logic priors that distinguish correct from incorrect CoTs, the GRPO gradient will increase the probability of generating more correct CoTs.
Additionally, we investigate the training dynamics of RLVR to understand when this improved rea- soning emerges. By reproducing GRPO-style training using the open-source DAPO recipe (Yu et al. 2025) and performing extensive verifications, we find that RLVR begins to incentivize cor- rect reasoning from the early stages of training, and this capability generalizes well to unseen test questions. The results of our training analysis align with our theorem, which highlights the implicit incentivization of correct reasoning CoT's
Finally, we evaluate the quality of generated CoTs from a learning perspective: if supervised learn- ing on some CoT data results in better generalization performance on test sets, we regard them as high quality. This allows us to evaluate the quality of CoT's generated by model checkpoints at different RLVR stages. Our results show that after RLVR, the quality of reasoning CoT's has been fundamentally improved.
In summary, our contributions include:
A systematic evaluation revealing the extended reasoning capability boundary after RLVR for both code and math tasks.
A theoretical understanding of why RLVR works with only answer correctness as a reward and how RLVR incentivizes correct reasoning.
An analysis of RLVR's training dynamics, delving deeper into optimization effects, gener- alization behaviors, and current limitations.
Confirmation of the quality improvements in reasoning CoTs from a learning perspective, replicating the generalization abilities of post-RLVR models trained with enormous costs simply via supervised fine-tuning.
RELATED WORK
RLVR Since the release of DeepSeck-RP R1 (uu et al.
2025).
there has been a surge of research
interest in the RLVR paradigm (Luo et al.
2025b
Liu al. 2025b
Hu et al. 2025 Cui et al.
2025 Xie et al.
2025 Zeng et al. 2025 Yu et al.
2025 Luo et al.
2025a Chen et al. 2025a
He al. 2025
Wen
2025
Cao al. 2025
Liu et al. 2025a
Chen al.) 2025b). Due
to the high computational cost of RLVR, most studies have focused on small- to medium-sized models (up 32B parameters). These studies span a wide range of aspects, including training data curation, objective design, hyperparameter tuning, base model selection, and various insightful observations. However, only a few studies have addressed the theoretical foundations of RLVR. In this work, we argue that RLVR for LLMs should be understood from different perspective-one that emphasizes the correctness reasoning paths. We hope our empirical findings and theoretical perspective could inspire the community to develop more efficient and effective RLVR approaches, unlocking its broader potential across diverse applications.
Debates on Whether RLVR Really Incentivizes Since Yue et al. (0255) raised the insightful question of whether RLVR truly incentivizes improvements beyond the base L.LMs, and conducted extensive empirical experiments to demonstrate the wide applicability of their key hypothesis that RLVR does not improve Pass@K for the base LLM because all reasoning paths are already present in the base model-there have been varying perspectives on this hypothesis. Some studies agree
with this viewpoint (Wang et al., 2025b Zhu et al.] 2025| Zhang et al. 2025 Wang et al.
2025a
Chen et al.] 2025a). while others report contradictory findings (Liu et al.
2025a Chen et al. 2025b
Shojace et al. 2025), as discussed in the introduction. There is currently no fundamental under-
standing resolve these debates. Liu al. (2025a) speculated that previous RLVR experiments
3

---

Published as a conference paper at ICLR 2026
2.0
Buan Ludt
1.0
1.0
5.4
0.-
0.0
2.2
0.2
Mua Rum
Bast LU
0.7
Alle 10/
0.4
Ala Runh
Sum LU
21
2
Sampling Number (K)
Sampling Number
Sampling Number
Namber [K]
Sampling Number
3.0
1.0
2.0
0.5
0.6
Roc.U M1
Ryn
5.4
San 1M
21 222222222222 222222 21
2222 2 2"222
27 21
2 2
22222 2"22
2' 2"
Sampling Number (K)
Samplnn Number (K)
Sampling Number (S)
Sampligg Number (K]
Sampling Number (K)
Figure 2: Comparisons of @ass@K K (the top row) and CoT-Paas@@ K (the bottom row) on five math benchmarks (different columns) to show how RLVR could improve base LLMs. Here the base LLM is Qwen2.5-320 and the post-RLVR model is DAPO-Qwen-3211 For CoT-Pass @ K. we perform multiple verifications for each CoT using DeepSeek- RI ---0-10-000----------- and display the results determined by any-correct, all-correct, and majority-coneet strategies, which constitute the shaded area in lower subplots.
may have been conducted within a single domain (e.g., math) and were optimized for limited gradi- ent steps before true exploration could occur. Shojaee et al. (2025) suggested that the complexity of puzzles might be the key factor. Chen et al. (2025b) presented statistically significant empirical results to justify that their model indeed improves Pass@K, particularly highlighting a persistent gap the BiveCodeBench v6 (Jain et al., (2025). In this work, extend the Pass K experiments of [Cen et al. (2025b) to more LiveCodeBennh versions and include another similar open-source study (He et 2025) starting RLVR from distilled LLMs as a comparison. All these results clearly disclose the extended reasoning boundary of distilled LLMs on competitive coding after RLVR.
The Importance of Correct CoTs Recent studies have also highlighted the importance of veri- fying the correctness of CoTs (Areuschin et al.) 2025 McGinness & Baumgartner 2022) SShoaaee al. 2025). However, their approaches focus defining synthetic reasoning tasks where the cor- rectness of reasoning CoTs can be verified easily. While this an interesting and effective approach for fully examining reasoning correctness, it is difficult apply unstructured reasoning scenarios, such as in math and code. In this work, we argue that the LLM-as-a-Cofe --o-a--oT-Judge paradigm could play crucial role in more general reasoning tasks, and emphasize the pressing need for the design of evaluation benchmarks to assess the reliability of emerging LLM verifiers.
3 EXTENDED REASONING CAPABILITY BOUNDARY AFTER RLVR
In this section, present concrete benchmark evaluations that demonstrate how RLVR can fun- damentally enhance the reasoning abilities LLMs. This enhancement goes beyond mere im- provements in sampling efficiency; it also expands the reasoning capability boundary. However, to effectively observe this enhancement, it is crucial to adopt an appropriate RLVR training recipe, select challenging benchmarks that are free from data contamination. and utilize reliable evaluation metrics. Without these measures, one might only observe improvements in sampling efficiency, with actual change in reasoning capacity. Below, we discuss representative cases from both the math and code domains, showcasing genuinely extended reasoning boundaries.
3.1 MATH REASONING
We begin by revisiting the P experiments conducted on the open-source model, DAPO- Qwen-32B (Yu et al. 2025), which successfully reproduced R1-Zero (Guo et al.) 2025) using the base LLM, Qwen2.5-333 3 (Qwen 2024), and a curated set 17k mathematical problems. A key contribution of our work is the introduction of novel evaluation metric, CoT--Pase @ K. which em- phasizes the importance of evaluating the correctness of detailed reasoning steps for mathematical questions, rather than relying solely answer correctness.
4

---

Published as a conference paper at ICLR 2026
53
0.5
2222222 22222222
Numbeer (K)
Samp 1ng Number
Samping Number
Nummerr
and the Number 90
Samping Number 3C
Figure 3: Comparisons of Pass@K K across six LiveCodeBench versions show how much RLVR could enhance distilled LLMs. Here the distilled LLM is post-RLVR model is AceReason-Nemooton--77
However, precisely measuring the correctness of CoTs scale is inherently challenging due the unstructured, lengthy, and complex nature of math CoT's Fortunately, we can leverage special- ized open-source LLMs, such yet lightweight verifier, employing an LLM--s-a-CoT-Jude paradigm. In this study, we use this verifier multiple times for each reasoning CoT generated by employing three distinct strategies to assess CoT correctness: any-correct (at least one verification returns correct), all-correct (all verifications must return correct), and majority-corest (the majority vote determines the outcome). To further ensure the reliability of these verifications, we manually inspect cases where the Pass K metric yields small positive value, but the CoT-Pass@ K metric returns zero. The details of our LLM-as- -as-a-CoT-Judeee approach can be found in Appendix A.2
(DeepSeek) (025), as a powerful
and Qwen2.5-32B,
Figure presents a comprchensive comparison using both Pass @ K and CoT-Pass metrics across prominent math-reasoning benchmarks. As shown in the top row, the Pass @ K results align with the observations in Yue et al. 2025): the performance of the base LLM quickly catches up with and even surpasses the post-RLVR model as K increases. However, in stark contrast, the CoT-Pass K results on AIME 2024 and AIME 2025 reveal a consistent and significant performance gap between the models across all values of K (up to 1024). This gap is particularly pronounced 2025, possibly due to its complete absence of unintentional data contamination, as it was released after the base model's training cutoff. We perform manual inspections to ensure the distinct gaps observed using the CoT-Pass @ K metric are reliable (see examples in Appendix A.7.I and A.7.2). Our LLM verifier effectively identifies critical errors, which we agree should be rejected. These results demonstrate the extended reasoning boundary of DAPO-Qwen-32B over Qwen2.5-32B.
AIME
Additionally, we observe that on other benchmarks such as MATH-500 and AMC23, the effects of RLVR seem less pronounced, as the base LLM is already capable of solving these problems correctly with sufficient attempts. This may due to 1) the problems being simple enough for the base LLM to solve using its existing knowledge, 2) the problems being part of its pre- training data, allowing the base LLM to recall the correct solution after multiple trials. Distinguishing between these possi- bilities is challenging without knowing the exact training data used for Qwen2.5-32B Furthermore, on the Minerva benchmark, the PLst-RLVR model shows no improvement, likely due to a train-test domain mismatch. Minerva contains numerous physics problems and free-form answers, while the DAPO training data was restricted to math problems with integer answers. These results do not undermine the effectiveness of RLVR; rather, they highlight the importance of selecting appropriate benchmarks for evaluating RLVR progress.
3.2 CODE REASONING
Unlike RLVR for mathematical problems, where the correctness of extracted answer tokens used as a proxy for passing, code reasoning relies on the actual execution of generated code snippets to verify correctness, significantly reducing the likelihood of guessing. Therefore, Pass @ serves as a reliable evaluation metric for code reasoning tasks.
In this section, we reproduce the @ experiments across different versions of Live- CodeBench (Jain et al. 2025) to compare the performance of AceReason-Nemotrron-7 (Chen et al.) 2025b) with its ppe-RLVR counterpart, DeepSeek-RI- Ri-isttlll- -wllll--ee-BBBBB (Guo et al. 2025). As shown in Figure 3 observe that AceReason-Nemotron-77 exhibits clear Pass@K K improvements on most benchmark versions, even though the latter a distillation
DeepSeek- Ber-Seek--I-iitttee model already demonstrating remarkable reasoning capabilities. These results suggest that even
5

---

Published as a conference paper at ICLR 2026
for distillation models, high-quality RLVR training recipe significantly extend the reasoning capability boundary, particularly for competitive coding tasks.
To further confirm the success of RLVR in extending reasoning boundaries for coding tasks, evaluate another post-RLVR model, Skywork-ORI (He et al. 2025), which has a fully reproducible training recipe publicly available. Detailed results on LiveCodeBench--66 can be found in Ap- pendix A.3] where we observe a consistent Pass @ K gap between Skywork-OR1 and DeepSeek- RI RI-Distill-Qwen-77 Specifically, we find that only medium and hard problems LiveCodeBench- v6 contribute to the differentiation between these two models for large values, underscoring the importance of selecting challenging benchmarks.
4 A THEORETICAL UNDERSTANDING OF RLVR FOR LLMS
In addition empirical evidences, provide theoretical understanding of how RLVR, as imple- mented in the GRPO algorithm (Shao et al. 2024). J fundamentally incentivizes correct reasoning for pre-trained language models. We note key distinction between RLVR for LLMs and traditional RL for randomly initialized models. Pre-trained LLMs, owing to their powerful likelihood estima- tion capabilities obtained during pre-training, can generate various CoTs and then produce possible answers. Some of them could coincidentally arrive at the ground truth, especially when the ground truth in a simple format and can be easily guessed. In contrast, traditional RL simply optimizes for action trajectories that yield high rewards, without necessarily verifying the intrinsic correctness of each action along the path. For instance, in the Go game (Silver al., 2017). every action is valid once the simulation environment is setup correctly. In the following, we start our theoretical analysis from formal problem setup distinguishing CoT and answer tokens in LLM responses.
Given question prompt q we sample G responses Y {y1, 12...2..... ....... yG) from policy na, where my is a LLM model parameterized by A. Let be the CoT in response yes and a; the final answer. We define the following correctness indicators:
Problem Setup
Icor(C:) {1
if c is correct otherwise
JAns(a)) - {
if ag is correct
otherwise
In this study, define the CoT correctness [cor(G)) (c; as the intermediate tokens of response (c²) expressing necessary and accurate logics that lead to the ground truth. We use p Ps. (IcoT 1) to denote the probability of generating correct COT. In practice, verify the CoT correctness because it is inherently unstructured, knowledge-intent intensive, and full of details. In contrast, the answer correctness (a;) is assumed be verified programmatically. So we have verifiable reward R(y;) that is binary and determined solely by answer correctness: R(y:)) (a;). We calculate the standard GRPO advantage A(y,) as:
rather challenging to
Á(w:) R(y:) NY
NY
G
G
R(y)),
ov
1 G Ø=1
Σ
(R(y)) - mx)
(2)
OY
f=1
Without loss of generality, we consider a policy gradient (Sutton et al.
1999)
update:
G A(y.) log (V6 q)-
VoJ(O)
(3)
Assumptions Given the problem setup decoupling CoT and answer correctness, we introduce a critical Logic Prior assumption: compared with incorrect CoT's, correct CoT's have higher probabil- ities to induce correct answers. Thus we have
>>>>> ((Isns(as) = Ico((c)) 1) a P(IAma(as) = Icor(c)) 0) B.
(4)
This assumption is based on the belief that pre-trained LLMs have established strong knowledge and logic priors. Besides, we also assume learnable group (av > 0) and a sufficiently large sampling number G to ensure stable gradient updates. Then, establish the following theorem.

---

Published as a conference paper at ICLR 2026
Theorem (GRPO Implicitly Incentivizes Correct Reasoning) For any prompt q satisfying our assumptions, the expected GRPO advantage [[]():)] satisfies:
E A(ys)) Icor(Ct) > 0.
[A(ye) Icor(c)) = =0] <0
(5)
where A(y.) defined in equation The GRPO policy gradient, as defined in equation increase the probability of generating correct CoTs (p') in the next round, so p increases monotonically.
Below we briefly illustrate our key perspectives why GRPO works and when it may fail. More- over, we include a complete proof for Theorem ] and more discussions in Appendix A.4
Discussions on the effectiveness of GRPO Theorem [ indicates that even though a base LLM may guess the ground truth with imperfect CoTs at the beginning (low initial p²). GRPO could still work as long as the knowledge and logic priors have been established. The driving factor is the gap a 3 O, which amplifies the advantage difference between correct and incorrect CoTs. As training progresses and a increases (due to more sound reasoning across various question prompts) while 33 decreases (reducing spurious correlations, model biases, incorrect knowledge calculation, etc.), causing the gap to widen and further accelerating coherent reasoning. As Pe - 1, (a - () may approach in faster pace because generating a few answer tokens typically much easier than producing long correct CoTs, then E E[A(y,) correct CoT] - 0, ensuring convergence.
Discussions on failure modes in GRPO We note that the Logic Prior assumption may not al- ways hold, potentially leading to the reinforcement of incorrect CoTs, since base LLMs may retain inherent biases and possibly fatal knowledge errors from pre-training. These harmful information might exist in some CoTs that finally yield the correct answer. In such cases, improper model bi- ases could be unintentionally reinforced. We suspect that these unexpectedly reinforeed CoTs are the root cause of the challenges faced by the R1-Zero approach (Guo et al. 2025), including poor readability and multi-lingual behaviors.
5 TRAINING DYNAMICS OF RLVR
To further demystify RLVR, we reproduce and analyze the training recipe of DAPO et al, 2025) which has been demonstrated present extended reasoning capability boundaries Section Our experiments show that its training dynamics align pretty well with Theorem
Key Indicators We first introduce key indicators that we have recorded during the reproduction. For each prompt q sampled with G' responses, we define the number of answer passes and the number both CoT and answer passes as C _ (a,) and = Dg Icar(c:) IAna respectively. We follow Chen et al. (2021) approach to calculate the Pass @K K metric. Accordingly,
( 9) and CoT-Pass@ @ K = ( Besides,
have per-prompt metrics: Pass (o)
we estimate the probability of producing correct answers for prompt P(CA)(1) & and the probability of producing correct CoTs when generating correct answers P(CCICA)(() L. For a dataset of multiple prompts, we take an average of per-prompt metrics as the dataset-level score.
Optimization Effects In our reproduced DAPO training, we observe that most training questions have been fully optimized. As shown in Figure 4 the probability of generating correct answers for these questions almost reach 1. In the meanwhile, we also observe the improvement in producing more correct reasoning CoTs, as indicated by the improvements in P(CC)CA))()) These observa- tions validate the key perspective in Theorem RLVR not only optimizes the final verifiable reward but also implicitly incentivizes correct reasoning.
Generalization Behaviors In the meanwhile, Figure 5 discloses that the optimization of RLVR leads to the generalization improvements of both Pass @ K and CoT-Pass@K K from the very begin- ning. And using the CoT-Pass K metric, we can clearly tell that the reasoning capability boundary has also been enhanced since the beginning. Another interpretation for this empirical observation
7

---

Published as a conference paper at ICLR 2026
P(CA)
PCC|CAY)
1.0
0.8
Probablitty
0.6
0.4
0.2
0.0
0-20
80-100
180-200
280-300
380-400
Training Steps
The evolution of P(CA)(1) (the fraction of correct answers for prompt and P(CC)CA)()) (the fraction of correct CoTs within the correct answers for prompt for fully optimized training questions over the course of DAPO training.
Figure 4:
AIME
AIME 2025
1.0
1.0
0.4
Alter
0.4
RaVi Stsp-210
0.2
RaVk Stop
Stap-30
0.2
Basa
0.0
20
2
22
2a
23
2
2
2
2
2
20
2
2
Sampling Number (K)
Sa hampping Number (K)
AIME 2024
AIME 2025
1.0
1.0
RIVE Siep-t0
Step-30
0.6
0.6
LLM
0.4
0.4
Stap-210
0.2
0.2
2
2
21
Sampling Number (K)
Sampling Number (K)
Figure 5: The evolution of Pass@ K (the top row) and CoT-Pass@# K (the bottom row) performance
on AIME 2024 and 2025 for different model checkpoints during the DAPO training.
that the model has learned to produce more and more reasoning CoTs that DeepSeek-R1-0528. Qwen-8B cannot identify any error. This ability is learned implicitly since we have no formal supervision for the CoT correctness.
Limitations of DAPO Figure also unveils the limitations of DAPO, the R1-zero approach ap- plied to a 32B base LLM. As ((CA)) approaches 1.0 for most fully optimized training questions after 400 steps, which means these questions no longer learnable because cannot calculate valid GRPO advantage for all-correct group. we can still observe non-negligible portion of imperfect Co''s (the median of P(CCICA)(0) is around 0.7). These signals indicate that there are certain unexpected reasoning behaviors learned in DAPO and we may not have chance mitigate them purely based on answer correctness as the reward.
THE QUALITY OF REASONING CoTS ENHANCED BY RLVR
In addition to the LLM-as-a-CCL --aa----CoT-Juueee approach for strictly identifying critical errors reason- ing CoTs, we further leverage supervised fine-tuning (SFT) to assess the quality of reasoning CoTs enhanced by RLVR. Given the training questions DAPO, we conduct multiple SFT procedures, starting from the same base LLM and learning from CoT' generated by different models. If the CoT
8

---

Published as a conference paper at ICLR 2026
AIME 2024
AIME 2024
AIME 2024
DAPO- Oween-326
0.5
Pacuga (Avg (Avg.033) 9321
0.8
0.8
0.4
CoT- ToT-PassBk
SFT on correct CaTs
Pass@kk
0.6
0.6
0.3
SFT on incomect CoTl
Colb
0.4
Q.er2.5
0.4
0.2
SFTBBase Cob)
0.2
SFTIDAPO Cols]
Qwen2.5-378BB 5-328-Base
DAPO Qwen 328
Base
Stepslo-20, 80-200) DAPO
2222''''22' 2-22 2igi0 Sampling Number (K)
2'222222 2222222
SFT Training Data
Sampling Number (K)
AIME 2025
AIME 2025
AIME 2025
DAPO Owin 328
0.8
0.8
Porega (Avg (332)
0.3
SFT on correct
Pezsigk
0.6
CoT.PassaK
0.
SFT on incemee CoT:
SP on all Cola
0.4
Guw 2- 5-30--ess
0.1
SETIDAPO
0.2
Qwen2.5--3 5-320-Base
DAPD-Guee-128
Base Cols
Steps/o-201 StepsilB0- 200) DAPO Cols
0.0-27
""
SFT Training Data
Sampling Number (K)
Sampling Number (K)
(a) The CoT quality at different RLVR stages, using (b) The CoT quality before and after RLVR, using Pass on test sets as the proxy metric.
Passe K on test sets as the proxy metric.
Figure 6: We show the generalization performance of post-SFT LLMs optimized on different CoT data. All these SFT processes start from the same base LLM, Qwen2.5-32D with the only variable being the different CoT data on DAPO training questions. We use the performance on test sets (AIME 2024, 2025) proxy for the quality of the corresponding CoT data.
data is of high quality, we expect the post-SFT model to exhibit improved generalization perfor- mance. Figure 6 presents an overall quality evaluation of various CoT data.
Specifically, Figure 6((a illustrates the evolution of CoT quality during RLVR. As training pro- gresses, the generalization performance of post-SFT LLMs, measured in Pass @ improves steadily. Ultimately, SFT on DAPO CoT data matches the Pass@ performance of DAPO-Qwen-32B. This result indicates that, given sufficient training questions and CoT data from a post- Rost-RLVR model, we can replicate a new model with nearly the same Pass @ I performance simply through SFT. Moreover, an interesting observation that, regardless of whether the CoT data contains identifiable errors, as RLVR progresses, the CoT quality, measured in Pass@1, 1, generally improves. This suggests that although some erroneous steps may be present, the overall quality of these "incorrect" CoTs in the later stages of RLVR improves significantly.
Figure/6(b)come compares the CoT quality before and after RLVR, using both Pass@ K and CoT-Pass@ K as proxy metrics. Comparing DAPO-Qwen-32B with the post-SFT model trained on its CoT data, we observe that this simple SFT approach nearly replicates the performance of a post-RLVR model, which would otherwise require significant computational cost. When comparing Qwen2.5-32B with the post-SFT model trained on its CoT data, we find that the post-SFT model begins to mitigate guessing. This aligns with our expectations, Qwen2.5-32B into the SFT procedure. We can therefore regard this process as round of off-policy RLVR optimization. These observations indicate that the incentivized CoT data through RLVR is crucial, as such CoTs cannot be directly sampled from base LLMs. RLVR optimizes the model's reasoning abilities, ensuring that the generated CoT's are more accurate, coherent, and reliable, which is essential for handling complex tasks.
we only feed CoT data with correct answers from
Limitations A key limitation of our study lies in the use of LLM as the verifier for the correctness of reasoning CoTs, due to the prohibitive cost of manually checking a large volume of generated rea- soning paths. Moreover, our theorem only explains the optimization process of RLVR but provides no guarantee for its generalization. We merely observe the generalization empirically. Due to space limitations, further discussions the implications of our findings are deferred to Appendix A.6

---

Published as a conference paper at ICLR 2026
7 CONCLUSION
In this work, we address the fundamental problem of whether RLVR genuinely incentivizes novel reasoning in base LLMs. Through empirical evaluations and theoretical analysis, we justify new perspective: RLVR implicitly incentivizes correct reasoning. Moreover, our analyses on training dynamics and CoT quality further confirm that the reasoning CoTs after RLVR are fundamentally different and can even help to replicate similar capabilities simply via supervised learning.
We hope these findings can not only resolve conflicting conclusions prior work but also illuminate the untapped potential of RLVR in aligning LLMs with human reasoning systems. We envision promising future where RLVR serves as cornerstone for developing LLMs that learn through interaction, self-correction, and verifiable reasoning.
REFERENCES
Team Anthropic. Introducing Claude 4. httpps//ww..attrooo...c////////s 2025. [Released 23-05-2025).
Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. Chain-of-thought reasoning in the wild is not always faithful. arXiv preprint arXiv:2503.08679, 2025.
Shiyi Cao, Sumanth Hegde. Dacheng Li, Tyler Griggs, Shu Liu, Eric Tang, Jiayi Pan, Xingyao Wang, Akshay Malik, Graham Neubig, Kourosh Hakhamaneshi, Richard Liaw, Philipp Moritz, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. SkyRL-vO: Train real-world long-horizon agents via reinforcement learning. hnttps://novase ai notion. site/skyr1- v0 2025.
Huayu Chen, Kaiwen Zheng, Qinsheng Zhang, Ganqu Cui, Yin Cui, Haotian Ye, Tsung-Yi Lin, Ming- Yu Liu, Jun Zhu, and Haoxiang Wang. Bridging supervised learning and reinforcement learning in math reasoning. arXiv preprint arXiv:2505.18116 2025a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint 1..iv:2207.033744 2021.
Yang Chen, Zhuolin Yang. Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. CeeRRasoo-Nemotree Advancing math and code reasoning through rein- forcement learning. arXiv preprint arXiv:2505. 16400, 2025b.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456. 2025.
Team DeepSeek.
DeepSeek-RL R1-0528 news250528 2025. [Released 28-05-2025).
Team
Gemini. tic
Introducing Gemini 2.0: our new Al model for the agen-
https://blog http://llooo.oooeee/eecocoooooo/ wttpe//hhhooooooogggoeeeeeeeeeee http://oooeddeemmmdddda.
google-gemini-an haamail-a-uuddrr---eeeee------ 2024/ 2024. [Released 11-12-2024).
Team
Gemini.
Gemini 2.5:
Our most intelligent
models are getting even
better.
https://blog. wtw.:/oogg.eeeecoooooooooooo//// wtteechholoog/googieee.... gowgleooooegoooooooooooooo deepmind/ 2025. 20-05-0005..
google-gemin ni htt.://emiini-pppt--------------
Team Grok. Grok 3 Beta The age of reasoning agents.
2025. [Released 19-02-2025].
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arxiv:2501./2944, 2025.
10

---

Published as a conference paper at ICLR 2026
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner technical report. arXiv preprint arXiv:2505.22312, 2025.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-Reasoner-Zerr: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290 2025.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In ICLR, 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.1512 15124, 2024.
Hunter Lightman, Vincet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In ICLR, 2024,
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, al. DeepSeek-V3 technical report. arXiv preprini arXiv:2412.19437, 2024.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. ProRL: Prolonged reinforcement learning expands reasoning boundaries large language mod- els. arXiv preprint arXiv:2505.244 24864, 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding RI-Zero-like training: A critical perspective. arXiv preprint 2:Xi:25503.0078: 2025b.
Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaox- iang Shi, Rachel Xin. Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source 14b coder 03-mini level. http://pretty-rrdio-77....mm w1..5.notion.oo Awtp://iieeeeeeeeoeeeeeeeeeaee mini-Level- 902c 46800bbbee5hh3344551222 2025a. Notion Blog.
Michael Luo, Sijun Tan, Justin Wong. Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa. and lon Stoica. Deepscaler: Surpassing ol o1-preveew with a 1.1...... 1.5b model by scaling rl. worriewwwith-a----e 1-----oee----------------------- tion Blog.
3a42, 2025b. No-
Lachlan McGinness and Peter Baumgartner. Large language models' reasoning stalls: An investi- gation into the capabilities of frontier models, arXiv preprint 96676225067 2025.
Team Mistral.AI. Stands to reason. Magistral. httpps//mmistrallaaaaa/////sggtt 2025. [Released 10-06-2025).
Team OpenAL. Learning reason with LLMs. http:///opeaii.com//deey 2024. [Released 12-09-2024].
Team OpenAL. Introducing OpenAl o3 and 04-mini. htsp:::/openai.coommmme
introducing--on o3 and mini/ 2025. [Released 16-04-2025].
Team Qwen. Qwen2.5 technical report. arXiv preprint arXiv:2412./5115,
Team Qwen. Qwen3: Think deeper, act faster.
htttps:/ htt.:/wwwwn....huu.eee wtt..i....hih.h....oooo.oo.
gwen3/] 2025. [Released 29-04-2025).

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. In NeurlPS, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. In EuroSys, 2025.
Parshin Shojace, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv preprint arXiv:2506. 00940, 2025.
David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 2025.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of Go without human knowledge. Nature, 2017.
Xuerui Su, Shufang Xie, Guoqing Liu, Yingce Xia, Renqian Luo, Peiran Jin, Zhiming Ma, Yue Wang, Zun Wang, and Yuting Liu. Trust region preference approximation: simple and stable reinforcement learning algorithm for LLM reasoning, arXiv preprint arXiv:2504.000 04554444444 2025.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. NewrIPS, 1999.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275. 2022.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math- aath-Shepherd: Verify and reinforce LLMs step-by-stey without human annotations. In ACL, 2024.
Yibin Wang, Zhimin Li, Yuhang Zang, Chunyu Wang, Qinglin Lu, Cheng Jin, and Jiaqi Wang. Unified multimodal chain-of thought reward model through reinforcement fine-tuning. arXiv preprint arXiv:2505.033 033388 2025a.
Yiping Wang, Qing Yang, Zhiyuan Zeng. Liliang Ren, Lucas Liu, Baolin Peng. Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20571, 2025b.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-R1: Curriculum SFT, DPO and RL for long CoT from scratch and beyond. arXiv preprint 2rXiv:2533.10660 2025.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Sid- dartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. LiveBench: A challenging, contamination-llmitc. LLM benchmark. In ICLR, 2025.
Tian Xie, Zitian Gao. Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-RI: Unleashing LLM reasoning with based reinforce- ment learning. arXiv preprint arXiv:2502. 2025.
Shunyu Yao. The second half. http:://eesyyyy..gtthb....... 2025.
12

---

Published as a conference paper at ICLR 2026
Qiying Yu. Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? arXiv preprint arXiv:2504.1333 10022544..337,,, 2025.
Weihao Zeng. Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. SimpleRL- Zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503, 18892, 2025.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised LLM reasoning incentivization. arXiv preprint arXiv:2504.05812, 2025.
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising effectiveness of negative reinforcement in LLM reasoning. arXiv preprint arXiv:2506.0/ 347, 2025.
A APPENDIX
A.1
DATA SOURCES
For math benchmarks studied in this paper, we leverage the following data sources: AIME 2025 AIME 2 Math-500 AMC23 Minerva
For training and evaluation of DAPO (Yu et al. 2025)
their training data (https: /
bugging ace. co/ wtt.d/aatttttseeeeeeeeeee BByeeee http://atttttttttttttteeeeeeeeee
and processed co http://o//aaaaeaeeeeeeeeee dttasaaatttt///aoeeeeeeee Byteessss///tttttsss www.eeddssinnhhhaeee nighanghhiii STA/
version of AIME 2024 (https: /ht.tps://hugtii http:/hhgggggggggge LIME-2022 Please note that they have duplicated questions multiple times and explained in the dataset page that the purpose be compatible with an old version of VERL (Sheng al. 2025 We reuse the prompt template of DAPO to evaluate their Pass@ K and CoT-Passük performance on other benchmarks.
Besides, we follow the official LiveCodeBench repository to perform Pass @ evaluations on com- petitive coding.
A.2 LLM-AS-A -AN-ASS--COTT-JCDER FOR MATH REASONING
We use much more specialized LLM mathematical reasoning (DeepSeck-R1
8B) as the verifier to examine the reasoning steps of base LLM,, Qwen2.5-32B. Meanwhile, we also acknowledge the existence of verification errors and manually checked many of them fimm the reliability of this verification. To further mitigate potential verification errors, we design multi-verification approach, as shown in Figure 7 For each reasoning CoT, we conduct multiple verifications independently and calculate three aggregation metrics:
All-correct: Chains of Thought that pass all verification attempts
Majority-corret: Chains of Thought that pass most verification attempts
Any-correct: Chains of Thought that pass at least one verification attempt (capturing po- tential emror recovery cases)
ht http://wwggnnlooooo/aaec/////llr
13

---

Published as a conference paper at ICLR 2026
DeepSeek-R1-0528--wee
conduct multiple verifications on CoT
wrong CoT
verify1
X
verify2
x
verify3 x
wrong CoT verify1
X
verify2
verify3 x
Qwen2.5-32B answer
Question
right CoT
verifyt
verify2
verify3 x
right CoT
verify
verify2
verify3 x
Figure 7: An intuitive diagram to illustrate the benefits of our mulli-verification system: simultane- ously considering any-correct, all-correct, and majority-correc helps us mitigate false positives and false negatives within individual verifications.
This multi-verification approach because the all-correct strategy mitigates false positives and the any-correct option reduces false negatives. Let Pip and pm represent the per-attempt false positive and false negative rates, respec- tively. For independent verification attempts, we observe:
have comprehensive view of CoT correctness
All-correct: The false positive rate decays exponentially as Pip
Any-correct: The false negative rate decays exponentially as p/
In our study, we employ n = 3 verification attempts for each COT.
Moreover, provide the prompt template used for DeepSeet-R1--52--Qeen---- follows.
Our Prompt Template for Verifier ReepSeek-R11-522--Qeee
You are an expert in mathematics and logical reasoning. Your task is to evaluate the cor- rectness of solution given math problem, with a **strong emphasis the reasoning process** not just the final answer. Below is the ** and the **Solution (Provided by another Al model)**:
**Prbbem=*3; *3;
[question]]
**Solution (Provided by another Al model)**; {(solution]}
Please perform the following tasks:
1. ** Analyze the solution step-by-step** paying close attention to: -Computational accu- racy- Logical consistency Conceptual understanding Whether the reasoning is valid and complete
2. **Identify any issues errors in the reasoning**, even if the final answer is correct. Clas- sify them into the following categories (if applicable): **Calculation Error** Mistakes in arithmetic, algebraic manipulation, numerical computation. ** Logical Error** Invalid reasoning, flawed logic, or incorrect inference. **Conceptaa Error** Misunderstanding or misuse of mathematical concepts definitions. *** **Omission Incompleteness**; Miss- ing steps, incomplete justification, or not addressing all parts of the question. **Other**, Any other type of error that does fit into the above categories.
3. **Provide final judgment** on whether the solution is logically sound and free of errors in reasoning.
Please format your response as follows:
**Issues Identified:*
- [[ssue 1]: [Classification]] [Brief explanation] [[ssue 2]: [Classification]] [Brief expla- nation] -...
Let's think step by step and output your final judgment within \boxed (boxed[yes) (boxed(no)
14

---

Published as a conference paper at ICLR 2026
A.3 REVISITING PASS K EXPERIMENTS FOR SCYWORK-ORI
Skywork-ORI (He et al., 2025) has generously shared their complete training recipes, claiming to enhance distilled LLMs with more powerful reasoning capabilities through RLVR. Therefore, conduct the Pass @ K experiments on their models to understand how RLVR could improve distilled LLMs.
Figure 8 shows the Pass K. comparisons between Skywork-ORl-7B and CrepSeek-Rl-Diseee Qwen- 7B on LiveCodeBench-v6. We can also observe significant improvement of both sampling efficiency (Pass @ 1) and reasoning boundary (Pass@K, K, K up to 1024). This conclusion is consistent with the observations for Figure 3 in the main paper.
LiveCodeBench-v6
Livecod-Bench---eeey
JwcCode@enchw66mmdllm
LiveCodeBench-w--hend
1.00
0.8
Pass
0.4
0.2
DeepSest 41 -
0.4
2 21 2 2 2" 2" 2
2" 27 27 2" 2
2' 2"
2
2' 2'
2' 2'2
Sampling Number
Sampling Number
Sampling Number
Sampling Number
Figure 8: Comparisons of Pass@ K on LiveCoddeBench-vó and its different difficulty-level subsets. Here the distilled LLM is CeepSeck-RI-Disilll-Nee ORI-7B.
and the post-RLVR model is Skywork-
However, in math domains, applying RLVR to distilled LLMs seems to merely deliver sampling efficiency improvements. As shown in Figure we observe that even using the CoT-Pass @ K metric, Skywork-OR1-Math- and DeepSeek-R -1-eek--RR-Dsilll R--iistill---een--- do not have distinct Pass @ K gaps for large K values. We suspect the reason is that the distilled LLM may already master major reasoning capabilities that can be learned with RLVR using answer correctness as the reward. So in math domains, their main improvements lie in Pass In contrast, for code domains, applying RLVR to distilled LLMs can still stimulate them to fit for real-world execution feedback, thereby incentivizing extended reasoning boundary.
AIME 2025
AIME 2024
1.0
1.0
0.9
@k
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
Skywork
Skyworkk 0R1
20 2
2
2
27 2 2
2 2
2
2
2
2 2
Sampling Number (K)
Sampling Number (K)
1.0
1.0
0.9
0.9
CoT-Pass@K
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
DeepSeck RL Detil-Qwen-ee
0.4
DeepSeck- Didil-Qwar 78
OR1 78
2' 2' 2'
2' 2' 2' 2' n0
2" Sampling Number (K)
2"
2
2 2' 2 2" 2" 2
Sampling Number (K)
Figure 9: Comparisons of @ (the top row) and CoT-Pass@K (the bottom row) on AIME 2024, 2025 to show how RLVR could improve distilled LLMs. Here the distilled LLM is DeepSeek- RI-Distill-QWen--7, and the RLst-RLVR model is Swwwork-ORI-Mathhhe For CoT-Pass @ K, we perform multiple verifications for each CoT using DeepSeek-R1-0528 -0ee8---ne-BBB and display the results determined by any-correct, all-correct, and majority-correct strategies, which constitute the shaded area lower subplots.
15

---

Published as a conference paper at ICLR 2026
A.4 PROOF AND ADDITIONAL THEORETICAL ANALYSIS
Below we include the detailed proof for Theorem
Proof Let Pe. = P(Icor(ci) = 1) be the current probability of generating a correct CoT. The expected reward for response Y4 is:
a [c)r((() = {B (c)r(()) =
B[R(yA)]
(6)
The group-level expected reward p a Eluy] is:
μ Pelx + (1-Pe)3.
For large G, the group mean by and variance oz concentrate around their expectations:
Ny G 1
(8)
o
G o² O.
(9)
The expected advantage conditional on CoT correctness is:
E[A(y) Icor(c:) = G-o00 a 1
(10)
8
E[A(y:) Icor(c:) = 0] G- 3 p
(11)
Substituting equation
into equation 70 and equation
B[A(y)) correct CoT] (1 - Pa) a 3)
(12)
0
E ElA(y:) incorrect Coff -Pola B)
(13)
Since 3 (by equation 4 uuner the Logic Prior assumption) and 0, we have:
(1 - Po)(a -8/a > 0,
-Pola - -3/00 O,
proving inequalities equation]
5
VoJ(O) EE A(y)) log To q), average increases the likelihood of responses with A(ys) > (correct CoTs) and decreases it for A(y:) < 0 (incorrect CoTs). Thus, Pe increases monotonically.
The GRPO policy gradient update in equation
Discussions on (u, σ²) in Theorem 1 From equation 7 we know that the group reward mean is given by μ p.a + (1 ([1-20 3. Furthermore, we can derive the exact formula for the variance o² in equation 9 and analyze their impacts together with Pe, a, and / on policy iterations.
The sample variance oy converges to the true variance 2
1 Σ (R(s,) 2x G+000 Var Var(R(y;) o²,
o% G
where Var(R(y,) can computed using the law of total variance:
Var(R(y;)) Var(E(R(y,,) Icar (cpr(c ([)))((())) (cor(c))))
Variance of conditional expoctation
Expoctation of conditional variance
First term:
E(R(y,) [c)r(c;)] a if In
3 [cor(c))) =
16

---

Published as a conference paper at ICLR 2026
The random variable B(R(y;) (cyo(c))) has variance:
Var(IB(((()) [cor(c;)]) =(a (ca-B))po(o -po).
Second term:
Var(R(y;) [cor(e;)) [a(1- a)) if Icor(())) =
(3(1-3) ( ifooo((c)) =
so its expectation is:
B[Var( [coT(c;))] poa(1 a) (1 Pc) 3).
Thus:
-a - () Pa) + p - + pc) -3).
(14)
Substituting y and into equation 12 and equation 13
have
(1 pc)(a (3)
E[A(s:) correct CoT]
(aa- -B)pp(1 - -Pe) + Ppal1 - ()) + (1- (1-pa)a(1- B)
-Pe(a - -3)
E[A(y:) incorrect CoT)
V(a- 3)ppp1 pe) + Pca - aa) + -Pe)B(1 3)
An ideal pre-training high-capacity model could help ensure that 1 and B - 0 at the beginning RLVR. In this condition, we have the following advantage estimates:
[(((v)) correct CoT
Pe
E(A())) incorrect CoT] +
Pe
Pc
In this ideal scenario, the role of human would be to prepare a comprehensive and diverse set of questions and answers, leveraging RLVR to automatically incentivize the model's reasoning capa- bilities. However, in practice-the the "unideal case"-it is often necessary to first fine-tune the base LLM align its output with proper reasoning distribution before applying RLVR.
Discussions on Key Observations in RLVR Grounded in our theoretical analysis, we can now provide our unique explanations for several previously elusive yet important observations reported in DeepSeek-RI (Guo et al. 2025).
Our Explanation of the Observation "DeepSeek-R -l-zekk--R-Zeerr -Zero achieved remarkable Puss @ K performance on AIME 2024 but encountered challenges such poor readability and language mixing.". " Even DeepSeek- V3 (Liu et al., 2024) cannot guarantee ideal conditions where a + 1, - 0. As result, cold-start data required to rectify prior logic biases, motivating the RI approach.
Our Explanation the Observation "The RI Re-Zero approach did not work well for the 32B dense model, yet distillation can be very effective." ": Key factors such as (Pe, a. (3) for the 32B base model are in an even worse state, causing pure RLVR to converge to suboptimal local solutions. Based on analysis, the key to effective reasoning lies in learning correct CoTs. Therefore, the distillation approach can efficiently teach an L.M how to reason properly.
A.5 ADDITIONAL DETAILS IN REPRODUCING DAPO TRAINING
Our reproduction was conducted on 32 AMD MI300X GPUs using the VERL framework (Sheng et al. 2025). and ran for over two weeks. Our run did not fully reproduce the Pass@1 accuracy above 50% as reported by Yu al. (2025). while we reached a comparable performance of around 44% Pass@ 1. in line with third- party reproduction (Chen et al.) (025a). We use the same verifier introduced in Section 3 to assess the correctness both training and evaluation rollouts.
In addition to the performance evolution on fully optimized training questions highlighted in Fig- ure of f the main paper, we include performance evolution on hard questions and more continuous validation performance in Figure 10 to provide more comprehensive information. The additional observations are consistent with what we have introduced in the main paper. It is natural to ob- serve that RLVR begins to incentivize correct reasoning from the very beginning, as evidenced by
17

---

Published as a conference paper at ICLR 2026
P(CA)
PCC|CAY)
1.0
Probablitt
0.8
0.6
0.4
0.2
0.0
0-20
80-100
180-200
280-300
380-400
Training Steps
(a) Distributions of P(CA)( and P(CCICA)) for easy training questions in DAPO.
P(CA)
RCCjCCAN
1.0
0.8
Probability
0.6
0.4
0.2
0.0
0-20
80-100
180-200
280-300
380-400
Training Steps
(b) Distributions of P(CA)() and P(CCICA) for hard training questions in DAPO.
AIME 2024
Pass@l CoT-Passe81
0.4
0.3
0.0
50
100
150
200
250
300
350
400
Training Steps
(c) Generalization performance AIME 2024 across different training steps.
Figure 10: We show gradually optimized performance training prompts throughout RLVR and corresponding generalization behaviors on testing prompts. The top two subfigures use violin plots to visualize the evolution of P(CAA)() (the fraction of correct answers for prompt q) and P(CCICA)(*) (the fraction of correct CoTs within the correct answers for prompt g) over the course of DAPO training. Subfigure (a) shows results for "easy" training questions, while (b) presents the corresponding distributions for "hard" questions (P(CA)(1) i for DAPO). We divide training questions into easy and hard by sampling Qwen2.5-32B, and labeling those with at least one correct answer easy. Subfigure (c) presents the generalization performance on AIME 2024 across different training steps.
rollouts for each of the 17k training questions using
increased P(CCICA)()) values in the early training steps shown in Figures 10(a) and 10(b) These incentivized reasoning capabilities translate into improved generalization on unseen questions, as demonstrated by notable gains in CoT-Pass@ @ on AIME 2024 within the first 20 training steps in Figure [O(c) Note that each training step here corresponds to one round of PPO-style optimiza- tion (Schulman al., 2017), which includes 16 gradient updates, according to the DAPO training script. Thus, we see that correct reasoning abilities begin to generalize after only few gradient updates.
Furthermore, the incentivization of correct reasoning on training questions appears to be contin- uous process, reflected by the steady increase in the mean of P(CCIC throughout train- ing, for both easy and hard questions. Meanwhile, we again observe that P(CAA)()) (equivalent to Pass@1 (9)) is an unreliable metric, particularly for easy training questions. As shown in Fig- ure 10(a). the distribution of P(CA))() becomes highly skewed toward 1.0 after 180 steps, mislead-
18

---

Published as a conference paper at ICLR 2026
ingly suggesting that most questions are perfectly solved. However, examining the distribution of P(CC)CA))) reveals that substantial fraction of responses still contain flawed reasoning. We suspect this is one of the reasons behind the difficulty of achieving strong results with Qwen2.5-32B using the RI-zero approach.
For both easy and hard training questions, improving P(CCICA))) seems to be a slow and chal- lenging process. Since our analysis shows that enhancing correct CoTs is key improving reason- ing capabilities, we believe that future research should explore novel mechanisms to accelerate the improvement of P(CC)CA))) thereby enhancing both the efficiency and effectiveness of RLVR.
A.6 DISCUSSIONS
Call for Live, Challenging Benchmarks Static benchmarks developed prior to the release of modern base models are increasingly susceptible to contamination risks, potentially undermining the reliability of observed improvements. In response, we emphasize the need for live benchmarks that evolve time, suggested in recent studies (Jain al. 2025 White al. 2025). Additionally, we agree with the viewpoint of Yao (2025) that future research advancements may rely more on designing new evaluations, benchmarks, and environments.
serves as a useful CoT verifier, it is not infallible. Conflicting verification results across multiple queries reveal the challenges of false-positive and felse-negative verifications. To tackle this, we combine multiple verification strategies, including different voting rules, to improve robustness. Looking forward, there is pressing need for light yet reliable CoT verifiers that can serve as standardized evaluators beyond the coarse-graiie grained Pass @ K metric. This direction also relates to previous studies on process reward modeling (Lightman et al.
Call for Lightweight yet Powerful CoT Verifiers While
2024 U Uesato et al. 2022 Wang et al. 2024).
Scaling RLVR or Scaling Pre- Training While the scaling of pre-training has led to transforma- tive progress in LLMs (Kaplan al., 2020: Liu al. 2024), enabling the transition the era of artificial general intelligence, we argue that scaling RLVR could be equally pivotal, given the em- pirical evidences and theoretical foundation that all demonstrate its real incentivization beyond base LLMs. As modern LLMs approach the limits of language token exposure, learning from experi- ence (Silver& & Sutton 2025) may represent the next leap. Recent efforts by leading research teams suggest a growing emphasis on this direction (Guo et al.] [2025] DeepSeek] [202]] Gemini 2024
Grok, 2025: OpenAl 2025 Qwen 2025) Gemini] [025; Anthropic| 20255] Mistral.A|| 2025). ). For
the broad open research community, understanding the foundations and limitations of current RLVR
algorithms is crucial to push this direction further.
New RLVR Algorithms and Beyond With insight that RLVR implicitly incentivizes correct reasoning in base LLMs, we anticipate the development of new algorithmic paradigms. These may include optimization formulations or objective functions, such policy-gradient approaches (Sutton
et al. 1999. Schulman et al., 2017), new likelihood-basse optimization objectives (Chen et al. Zhu al. 2025). and preference optimization frameworks (Rafailov al.) [023] [Su et al.,
2025a
2025).
The key principle is that the new algorithms should be designed to more directly incentivize
correct reasoning paths, alleviating inherent logical biases in base LLMs.
19

---

Published as a conference paper at ICLR 2026
A.7 CASE STUDIES
We perform manual verification on difficult problems from multiple benchmarks, focusing on cases where the base model occasionally produces correct final answer despite flawed reasoning in its COT. For each case study, we present the original question, the full response generated by Qwen2.5- 32B, and detailed breakdown of key errors identified by SrepSeck-R1-05222 (0058-Qwen3-8B explicitly linking each error to the corresponding flawed reasoning segments in the CoT. For layout and color schemes, we follow the visualization strategies used Yue al. (2025).
A.7.1 CASE STUDIES ON AIME 2024
Question
AIME24 Question No.9 (dataset 7
Let A, B, C, and D be points on the hyperbola 20 27 whose diagonals intersect at the origin. Find the greatest real number that is less than B for all such rhombi.
n²
1 such that ABCD is a rhombus
Qwen2.5-32B Answer
The diagonals of rhombus are perpendicular and bisect each other, ABCD is square. Let a BD AC.
CoepSeee-RR-0528- R21-058--QQ0 20a-28--wee3-8B verify: **Conceptual Error*:; The solution incor- rectly assumes that ABCD is square, whereas the problem only specifies that is rhombus. A rhombus does not necessarily have equal diagonals, so this assumption invalid.
DeepSeek-Re
Then a 2(AB²). The equation of the hyperbola can be rewritten as y² 24
20
2
24. Let P the point where diagonals intersect. Then OP 3. so
DeepSeek-RI R1-eck---1-5288 R1-0528-Qwent Qwen3-8B verify: **Logical Error**: The solution defines P the intersection point and sets to (a/2, a/2), but the problem states that the diagonals intersect at the origin. This inconsistency leads to incorrect reasoning, as should be (0,0) based on the problem statement.
Since P lies the hyperbola, we have a² 6 (
24,
a² 20
24, a² 480.
5
Therefore, the greatest real number that is less than BD2 for all such rhombi is 480
Question
AIME24 II Question No.1s(dataset id 81
Find the number of rectangles that can be formed inside a fixed regular dodecagon (12-gon) where each side of the rectangle lies either side diagonal of the dodecagon. The diagram below shows three of those rectangles.
httte////rssppoooameoaaa/e/////d AIVE Problens/
Problem
htteesr/aatttppoooolmmeeeeiiiien phpp AIME, II Problens/
Problem_11 15
20

---

Published as a conference paper at ICLR 2026
Qwen2.5-32B Answer
To solve this problem, we need to count the number of rectangles that can be formed inside a fixed regular dodecagon (12-gon) where each side of the rectangle lies on either side or diagonal of the dodecagon.
First, let's consider the sides of the rectangle. Since each side of the rectangle must lie on either a side or diagonal of the dodecagon, we can think of the problem choosing 2 pairs of parallel lines from the set of all lines (sides and diagonals) in the dodecagon.
There are 12 sides in dodecagon. Additionally, there are 12 diagonals that connect every other vertex, and 12 diagonals that connect every third vertex. So, there are a total 36 lines in the dodecagon.
Now, let's count the number ways to choose pairs of parallel lines. We can do this by considering the different types of lines and the number of ways to choose 2 lines from each type:
1. Choosing 2 lines from the 12 sides: There 12 choose ways do this, which is equal to 66.
DeepSeek-FE R Qwen3-8B verify: the method of counting two pairs of parallel lines assumes that any such choice forms a rectangle, which is not necessarily true as the lines may not intersect properly to form rectangle.
2. Choosing 2 lines from the 12 diagonals that connect every other vertex: There are 12 choose 2 ways to do this, which is equal to 66.
Choosing 2 lines from the 12 diagonals that connect every third vertex: There are 12 choose 2 ways to do this, which is equal to 66.
Choosing 1 line from the sides and 1 line from the diagonals that connect every other vertex: There are 12 12 ways to do this, which is equal to 144.
Choosing 1 line from the sides and line from the diagonals that connect every third vertex: There are 12 12 ways to do this, which is equal to 144.
Choosing line from the diagonals that connect every other vertex and line from the diagonals that connect every third vertex: There are 12 12 ways to do this, which is equal to 144.
DeepSeek-RI Biaieekk-R1055 R1-0528- Seee--e-e33BBB verify: **Logical Error**; The solution only con- siders three types of lines (sides, diagonals connecting every other vertex, and diag- onals connecting every third vertex), but there are additional diagonals (steps 4, 5, and 6). By ignoring these, the classification incomplete.
Adding up all these possibilities, we get a total of 66 + 66 + 66 + 144 + 144 144 = 630. However, have counted each rectangle twice (once for each pair of parallel lines), need to divide the total by to get the final answer: 630 / 315.
Therefore, the number of rectangles that be formed inside fixed regular dodecagon (12-gon) where each side of the rectangle lies on either side or diagonal of the dodecagon is 315
Question
AIME24 Question No.11(datasel id 62
Each vertex of a regular octagon is independently colored either red or blue with equal probability. The probability that the octagon can then be rotated that all of the blue vertices end up at positions where there had been red vertices is m where m and are relatively prime positive integers. Find m n.
php/2024 AIKE I. Problems/
Probless
21

---

Published as a conference paper at ICLR 2026
Qwen2.5-32B Answer
To solve this problem, we need to consider the possible colorings of the vertices of the octagon and how they be rotated.
First, let's consider the number of ways to color the vertices of the octagon. Since each vertex can be colored either red or blue, there 2 256 possible colorings. Next, we need to consider the rotations of the octagon. Since the octagon has 8 vertices, there are 8 possible rotations.
Now, let's consider the condition that all blue vertices end up at positions where there were originally red vertices. This means that the number blue vertices must less than or equal to the number of red vertices. Since there are 8 vertices in total, the number of blue vertices can be 0, 1, 2, 3, or 4. For each of these cases, and divide by the total number of colorings to find the probability. Case 1: O blue vertices There is only way color the vertices in this (all red), and it satisfies the condition. The probability is 256
need to count the number of colorings that satisfy the condition
Case 2: 1 blue vertex There are (i) 8 ways to choose the position of the blue vertex, and all of them satisfy the condition. The probability is 256
Case 3: blue vertices There are (*) = 28 ways to choose the positions the blue vertices. However, not all of them satisfy the condition. We need to subtract the cases where the blue vertices are adjacent. There are 8 such cases (one for each rotation of the octagon). So, there are 28- 8 20 colorings that satisfy the condition. The probability is
DeepSeek- R1 R1-0528-- 1198--Qwen3-8B verify: **ooiiaal Error** The solution incor- rectly assumes that subtracting cases based adjacency specific shapes (c.g.. "V" shape) ensures the condition does not hold. However, the condition depends on the existence of a rotation that maps all blue vertices to red positions, not on the ad- jacency or shape of the blue vertices. For example, with two blue vertices, adjacent configurations can still satisfy the condition for some rotations, subtracting them is invalid.
Case 4: 3 blue vertices There are (3) 56 ways to choose the positions of the blue vertices. However, not all of them satisfy the condition. We need to subtract the cases where the blue vertices are adjacent or form "V" shape. There are 16 such cases (one for each rotation of the octagon). So, there are 56 16 = 40 colorings that satisfy the condition. The probability is 40
5
32
Case 5: blue vertices There are (5) 70 ways choose the positions of the blue vertices. However, not all of them satisfy the condition. We need to subtract the cases where the blue vertices are adjacent or form "V" shape. There are 24 such cases (one for each rotation of the octagon). So, there are 70 24 46 colorings that satisfy the condition. The probability
256
128
Now, need up the probabilities for all cases and simplify the fraction:
1
1
5
5
23
+ 8 + 20 + 40 + 46
115
256
32
64
32
128
256
256
The fraction
115
already its simplest form, so m = 115 and n = 256. Therefore,
256
m+ n 115 + 256
371
22

---

Published as a conference paper at ICLR 2026
Question
AIME24 I Question No.tssd id 65 10
Let ABCD be tetrahedron such that AB CD = AC BD = v80, and BC AD V89. There exists point inside the tetrahedron such that the distances from I to each of the faces of the tetrahedron are all equal. This distance can be written in the form myn where m, n, and p are positive integers, m and p relatively prime, and divisible by the square of any prime. Find + n +
=
Qwen2.5-32B Answer
To find the distance from the point / to each of the faces of the tetrahedron, we can use formula for the volume of tetrahedron and the fact that the distances from I to each of the faces are equal.
First, let's find the volume of the tetrahedron using Heron's formula for the of triangle and the formula for the volume tetrahedron:
Ō base area height
We can choose any face as the base, so let's use the face ABC:
AB = 4II, AC = /80, and BC = /89
Using Heron's 's formula,
find the of the triangle ABC: /S1+v0++V89
AB+ AC++C
8
AABC ss(s AB) (C) BC)
Now, let's find the height of the tetrahedron from the point to the face ABC. We know that the distances from to each of the faces equal, let's call this distance h.
AABC
DeepSeek-R1 R1-0552 11528-QQeen-8BB verify: **Conceptual Error** The volume formula AABC is incorrect for the incenter of the tetrahedron
We find the volume of the tetrahedron using the Cayley-Menger determinant:
1
1
0
41
80
89
12
41
89
41
288
80
89
0
80
89
41
80
DeepSeek-RP R1-kk-----2------- -0228-Q verify: **Calculation Error** The Cayley- Menger determinant matrix contains multiple incorrect values. For example, the clement in row 2, column 5 should be 80 (BD2) but is 41, and the element in row 3, column should be 41 (CD2) but is 80. This affects the volume calculation.
Solving for V, we get: -AABC
V
Now, we can solve for h:
h
6V
7ABC
After calculating the values, we find that h can be written in the form my where m, n, and p are positive integers, m and are relatively prime, and n not divisible by the square of any prime.
The final answer is m p 104
ottpes////rrooeellloooo////eeeed AAMME I Problens/ Problem_14
23

---

68, and its height is the same as the Published as a conference paper at ICLR 2026
A.7.2 CASE STUDIES ON AIME 2025
Question
AIME25 Question No.2 11
On AABC points A, D, E, and B lie that order side AB with AD = DE 16, and EB = 8. Points A, F. G. and lie in that order on side AC' with AF = 13, = 52, and
GC 26. Let M be the reflection of D through F, and let N be the reflection of G through
E. Quadrilateral D FGF has area 288. Find the area heptagon AFNBCEM.
M
Qwen2.5-32B Answer
To solve the problem, we need to find the area of the heptagon AFNBCEM. We know that DEGF has an area of 288, and we need to find the areas the triangles AFM, NCE, and MNE. E.
First, let's find the area of triangle AFM, Since M the reflection of D through F, triangle
AFM M is congruent triangle AFD. The base of triangle AFD is AF 13,, and its height
is the same as the height of triangle ADE, which is
288
288
288
72 So, the
DE
EG
16++52
68
17
area of triangle AFM is 1 AF height
13
17
168
DeepSeck-R1-Dne R-0--20---------- RI-0528-Owen3-8B verify: ** *onccptual Error** The solution in- correctly assumes that the height of triangle ADE is derived from the area of quadri- lateral DEGF using DE + EG, but EG is not defined in the problem.
Next, let's find the area of triangle NCE. Since N the reflection of through E, triangle
NC'E is congruent to triangle GCE. The base of triangle GCE is GC 26, and its height
is the same as the height of triangle GEB, which GE EB area of triangle NCE is GC height
288
288
288
24 So, the
52+88
60
26
312
DeepSeek- RI R1-0528-00 1--552--wee3-8R verify: Similarly, for triangle NCE, the height is cal- culated using GE + EB, but GE is not given; the solution uses 52 (FG) incorrectly.
Finally, let's find the area of triangle MNE. Since M is the reflection of D through F and N is the reflection of G through E, triangle MNE is congruent to triangle DGE. The
base of triangle DGE is DE EG 16
52
height of triangle DEG, which is DE+EG
288
288
72
So, the area triangle MNE
68
17-
DE + EG height 68
144.
DeepSeek- Baaeek--RI-e R1-0528-000
R1-0522--wwe3--BB verify: The area calculation for triangle MNE uses DE EG base, which is not valid base unless DE and EG are collinear.
Now, we can find the area of the heptagon AFNBCEM by adding the areas of the triangles AFM.NCH MNE, and the quadrilateral DEGF:
468
17
312
5
144 + 288
588
www.///wwwwiiiiiiiii..i.e.. php/2025_ AIME Problems/
Problem 2
24

---

Published as a conference paper at ICLR 2026
Question
AIME25 Question No.1
A piecewise linear periodic function is defined by f(x) x i E 1-1,1)
and f(r + f(x) for all real numbers z. The graph of f(x) has the sawtooth pat- tern. The parabola x 34y² intersects the graph of f(x) at finitely many points. The sum of the y-coordinates of these intersection points can be expressed in the form atbysE where a, b, c, and d are positive integers, a, b, and d have greatest common divi- sor equal to 1, and e is not divisible by the square of any prime. Find + b b + d.
22 2- if e (1.3)
Qwen2.5-32B Answer
To find the intersection points of the sawtooth function and the parabola, we need to solve the system of equations f(x) 34y² and f(x + 34y². We solve these equa- tions separately for each interval of f(z), and then sum the y-coordinates of the intersection points. Here's a Python code do that:
"python
import sympy sp
Define the variables X, = sp. sp.bollll (' x_y")
Define the piecewise linear periodic function
sp. ( >= 3)1)
>- -1) & (x < 1)), (2 x, (x >- 1) (x
Define the parabola parabola 34*y**2
$ Find the intersection points in each interval Lntersection_points = []
for i in range(-i, 3):
interval in. (i, itl, left_ope right_open--Tree) eq (f,, parabola)
solutions sp.solve y)
for sol in solutions: if sol,is_real: Intersection_points...po a (soled
$ Sum the y-coordi nates of the intersection points y_sum sum (interrsettoorpoontt))
# Simplify the sum y_sum_ simplified = sp.simplify (ym)um)
htteer///rrooppplleoooooo...ccad AIKE Problens/ Problem
25

---

Published as a conference paper at ICLR 2026
Extract the coefficients a, b, c, and d a, b, e, d wiw.didd(aa)) sp.Wild('b'), sp. sp. sg.wild('d') match sp.collect (y_ sy_ssn_simmllfied, sp.tqrt (sp. bxsp.sqrt
Wi c()))) (
a, b, e, d match(a), match(b), match[c), match [d]
Calculate the final answer final answer + print (tinal_answer
The output of this code is 259, Therefore, the answer is 259
verify: **Logical Error** The code solves the equation f(x) = 34y² for which finds the y-values where horizontal lines inter- sect the graph f(z). but this does not correspond to finding the intersection points with the parabola x 34y². The correct approach requires solving the system y = fa) and x = 34y² simultaneously, such as by solving f(34y²) = y.
DeepSeek-R1-052--Qeee
**Omission / Incompleteness** -The code only considers one period of the func- tion (æ [-1,3)) but due to the parabola x 34y² with |y| S 1, æ ranges from 0 to 34, covering multiple periods (since the function has period 4). Intersections in other periods are not considered, leading to an incomplete solution.
A.7.3 CASE STUDIES ON AMC23
Question
AMC 12 2023 A Problem 23(dataset id 15
How many ordered pairs of positive real numbers (a, b) satisfy the equation
(1 + 22) 22 b) 32ab?
Qwen2.5-32B Answer
First, let's simplify the given equation:
(1 + 2a)(2 22) b) 32ab
Expanding the left side, we get:
(2 + 4a + 4b Sab) +b) 32ab
Sww---11-022----n----- verify: Calculation Error The initial expan- sion of (1+22)(2+2b) is incorrect, resulting in 2+4a+4b+8ab instead of the correct 2+4a+2b+4ab. This error propagates through subsequent steps.
Expanding further:
4a + 8a² + 8ab + 16a + 2b + 4ab + 8b2 + 16ab -3abb
httpfp///crttgooooomooooooaaaaas php/2023 AMC 12A Problems/
Problem 23
26

---

Published as a conference paper at ICLR 2026
Combining like terms:
Sa + 12ab + Sa b + 26
862 16ab² 28ab
Rearranging terms:
Sa + 8a²b + 8b2 16ab² 16ab 2b
Dividing both sides by 2:
4a + 4a + 4b² + Sab" Sab
Now, let's try to factor the equation:
4a² + b) + 462 Sab2 8ab
Factoring out common terms:
41 + b) 4b² 2a) 8ab b
Now, let's try to solve for and b:
4a (1 l + b) = Sab - b 46 (1 + 2a)
Dividing both sides by
b
a1 b) 2ab
b² 2a) 4
Now, let's try to find solution for and b:
b
a² + = 2ab
12 (1 + 2a)
4
One possible solution is when = b:
a² (1 = 2a
a 4
a²(1 + 20)
-0128-Q verify: Logical Error - The solution assumes a=b is solution without verification, and this assumption is based on an incorrect equation. The derivation under a=b leads to non-zero result, contradicting the claim of solution.
DeepSeek-RI1 R1-0528-
Simplifying:
a² + a²
2a
a
a
2a 2
4
Combining like terms:
2a a
Multiplying both sides by 4:
S = a
Dividing both sides by a:
8a² 1
Taking the square root of both sides:
/8
Since b, we have:
1
V8
Thus, there is only one ordered pair (a, b) that satisfies the equation:
27

---

Published as a conference paper at ICLR 2026
Question
AMC 12 2023 A Problem 19(dataset id 10 What is the product of all solutions to the equation
log7= 2023 log289 2023 log2023. 2023
Qwen2.5-32B Answer
First, let's rewrite the equation using the change of base formula for logarithms, We have
log 2023
log7. 2023
log(7x)
log 2023
logasur 2023
12g(289x:)
log 2023
log2023< 2023
log(2023:4
Substituting these expressions into the original equation, we get
(log 2023)²
log 2023
log( (x) 10g(289x)
log (2023x)
Cross-multiplinng, we get
(log 2023)² (2023r) (log 7x)) (log 289x) log 2023
Canceling the common factor of log 2023, we have
log(2023r) (log Tx) (log 289x)
DeepSeek-R 1--518-----0 11928-Qwen3-88 verify: **Calculation Error**; The solution incor- rectly canceled the log 2023 factor during simplification, omitting the log 2023 term.
Using the product-to-sum formula for logarithms, we have
log(2023x) log(7æ) log(289x)
DeepSeek-R1-0522 - verify: * *Conceptual Error**; the solution incor- rectly treated the product of logarithms (log 7 (log 289r) as if it were sum, which is not valid.
l2g(2023r) log (2g(2233..2)
2023.r = 2023x
Dividing both sides by 2023r, we get
So the only solution is r = 1. The product of all solutions is
1
"tteer//artcpoool.mmoooicccccacc phppsl23pppp33333// AMC 12A Problens/ Problese @ooblem_19
28

---

Published as a conference paper at ICLR 2026
A.7.4
CASE STUDIES ON MATH-500
Question
MATH500 unique id test/intermediiat.llggone algebrr//822
The equation
10
(13r 1)
has 10 complex roots F1, F1. 12: F2, T3, F4, Fa, Ts. Fs, where the bar denotes complex conjugation. Find the value of
1
1
1
F1
7272
F333
T474
T5
Qwen2.5-32B Answer
To solve the given problem, let's first analyze the equation x 10 + (13x 1)" = 0. This can be rewritten as:
== -13r 1)a0
Taking the 10th root on both sides, we get:
+i(13r - 1).
This gives two cases consider: case separately.
i(13x 1) -i(13x 1) Let's solve each
DeepSeck- Baicck-R1-5522 R1-0--8-0000----- 11552--Qeen-8B verify: **Calculation Error**; The solution incor- rectly assumes that the equation has only two roots, but the equation is of degree 10, it should have 10 roots. The method of taking the 10th root only considers two cases, missing the other roots.
For the first case:
I i(13r - 1) = 13ir -i = x(1-133) - == 13i
For the second case:
i
-((11r - x 13ir = - xx1 + 131) =
1 13i
So, the roots of the equation are of the form conjugate pairs, can pair them up r' T-131 and so
1-137 and
-i
I-13. Notice that these roots come in
P1
1+13: 12 1+13/ 12
Now, we need to find the value of r1 +
1
+
+
1 + roTa
Since ri
r272
roPa
r4t4
1-138
I+13i
1+169
170 we have:
170.
Fi
Similarly, F272
170,
= 170,
170, and
1 170. Therefore, the sum is:
170 + 170 + 170 + 170
170 850.
Thus, the final answer is:
850
29

---

Published as a conference paper at ICLR 2026
Question
MATH500 unique id testintermediatt..ggbbrr22 Find the maximum value of
I + 1²
over all real numbers z and y.
Qwen2.5-32B Answer
To find the maximum value of the expression 46 we can start by analyzing the de- nominator. Since x² and y² are always non-negative, the denominator is always at least 6. This means that the expression is always less than equal to ==y Now, let's consider the numerator y. If we set
and = -1, then the numerator is (-1) ) 2. Substituting these values into the expression, we get 14(-11746
4 Therefore, the
maximum value of the expression is
DeepSeek-R Srrsseck-R--5288 10528-QQeen388B verify: **Omission / Incompleteness*** The lution does not provide complete justification for why the point (1, -1) gives the the maximum
maximum value. only evaluates one specific point and claims without comparing it to other points demonstrating that the expression cannot be larger elsewhere. For example, does not consider points where xa + y is min- imized while y is positive and non-zero, points where the numerator and denominator balance differently.
A.8 RESULTS FROM MORE VERIFIERS
A.8.1 COT-PASS @ K COMPARISONS ACROSS DIFFERENT VERIFIERS
We further evaluate two powerful verifier models--gttooss- and gpt-oss- 120s- 10oos----bbuuinnng using identical evaluation settings (using high reasoning efforts and a maximum token length of 64k) and perform- ing n -3 verification attempts for each COT. Both gpt-oss verifiers use the same verification prompt described in Section A.2 As shown in Figure II the verification results AIME24 and AIME25 exhibit consistent trends across DeepSeek- -0528- 1-28-Qwen3-81 (DS-8B), gpt-oss- and gpt-oss- 120b. While the larger verifiers produce slightly lower absolute pass rates, the relative improvements from RLVR and the overall pass @ p-pass @K k curves remain highly aligned across all verifier mod- els. These results consistent with the core conclusion obtained using DS-8B RLVR enables the model to generate substantially more correct CoT's compared to the base model.
A.8.2 VERIFICATION CORRALATION BETWEEN DS-8B AND GPT-OSS VERIFIERS
We further analyze the consistency between DS-8B and the two gpt-oss verifiers by measuring correlation of their correctness judgments. Each verifier model is run three times every chain-
of-thought, and correctness is summarized using the three criteria introduced Section A.2 All- correct, correi,, and Any-correct (denoted as All. Maj, and Any). Figure 12 presents set of heatmaps across two model Co''s generated by the RLVR-trained model (DAPO) and CoT's from the base model using only samples that yield correct answers AIME25. The first column shows the intra-model correlations among the three DS-8B verification passes, capturing its self- consistency. The remaining columns report the inter-model correlations between DS-8B and gpt- oss-20b gpt-oss- -20b. Across both model groups, most of the correlation structures are stable and indicate a relatively high level of agreement between the verifiers.
Two observations emerge from these results. First, CoTs from the base model exhibit substantially higher intra-verifier and inter-verifier correlations, such the DS-8B intra-correlation between Maj and All/Any reaching around 0.88. This is expected, as base CoT's contain more salient errors,
30

---

Published as a conference paper at ICLR 2026
verffication
D5-88 Verification
gpt-oss-206 Verification
9pt-ss--20b Verification
1.0
Aller RLVIt
1.0
Aller Ravil
Aller RLVil
RiMt
AUKLL 2025
Pa6188
0.8
Base LiK!
0.8
Ease LLM
COT-PaSSS
0.8
Base LLM
CoT-Pas:88K
0.8
Siaee LLM
0.4
0.4
0.4
0.4
0.0
27 21 2 2'22222''22
0.0
21 2'
2'222'22 2222
0.0
2: 27
2' 2 2210
0.0
2:
Sam pling Number
Sampling Number
Sampling Number
Sampling Num
1.0
Afte RUVR
1.0
After RIVA
Afer RLVR
Afteer RIVR
A/ME: 2024
Base
CST-Pacsgg Sgk
CoT-PassiBkk
Pacsille
0.6
0.6
0.6
0.6
0.2
0.2
0.2
0.2
2'22"
2'
2'2"2'222""
2'
2"
2222 2' 2'
Sam pling Number
Sampling Number
Sampling Number
Sampling Num ber
Figure 11: CoT-Pass@ comparisons using three different verifiers: DS-8B, gpt-oss-20b, and gpt- oss-120 on AIME 24-25.
making them easier and more consistent for all verifiers to detect. contrast, DAPO CoTs contain fewer and subtler errors, leading lower agreement: different verifiers may occasionally miss small mistakes and mark these CoTTs as correct. Second, stronger verifier models such as gpt-oss-20b and gpt-oss-120h tend to apply stricter correctness criteria and identify more subtle errors. As result, the correlations between DS-8B and the gpt-oss verifiers typically lie around 0.7 across the correctness metrics.
DS-8B
DS-8B & gpt-oss-20b
DS-8B gpt-oss-120b
DAPO- Qwen- 32B CoTs Any
All
1.00
0.70
0.54
All
0.55
0.63
0.59
All
0.54
0.61
0.63
DS-8B
Maj
0.70
1.00
0.78
Maj
0.59
0.73
0.71
DS-8B
Maj
0.56
0.66
0.72
0.54
0.78
1.00
Any
0.48
0.61
0.62
Any
0.45
0.54
0.61
Maj
Any
All
Maj
Any
Maj
Any
gpt-oss-20b
gpt-oss-1205
DS-8B
DS-8B & gpt-oss-20b
DS-8B gpt-oss-120b
Qwen2 5-32B CoTs Any
All
1.00
0.88
0.78
All
0.68
0.78
0.79
0.69
0.76
0.75
DS-8B
DS-8B
Maj
0.88
1.00
0.88
Maj
0.67
0.78
0.85
Maj
0.67
0.76
0.80
0.78
0.88
1.00
Any
0.61
0.74
0.85
Any
0.61
0.71
0.78
Maj
Any
All
Maj
Any
Maj
Any
gpt-oss-20b
opt-oss-120b
Figure 12: Correlation heatmaps of correctness assessments across verifier models. Each verifier performs three passes per chain-of-thie of-thugghtt summarized using the All, Maj, and Any metrics. Re- sults shown AIME25 for CoTs yielding correct answers, separately for DAPO-Qwen-32B and the base Qwen- 2.5-32B model.
31

---

Published as a conference paper at ICLR 2026
A.9 WHY Do SFT MODELS GENERALIZE EVEN WHEN TRAINED ON INCORRECT COTs?
A notable observation in our experiments is that SFT on RLV --enereneaaeed Co'Ts leads to substan- tial performance gains errors). In contrast, SFT on the base model's s own CoT'swhether correct or incorrect-yields only minor improvements. This prompts an important question: why can "incorrect" RLVR CoTs still support strong generalization?
when CoT is labeled as incorrect (meaning it contains verified
Our notion of incorrectness follows the verification prompt described Section A.2 any CoT con- taining calculation mistakes, invalid logical deductions, conceptual misunderstandings. or missing essential reasoning steps is labeled as incorrect. Thus, incorrect CoTs span broad range of quality. The key insight from our analyses is that Among incorrect Co'Is, those generated by RLVR differ fundamentally from those of the base model.. They are longer, more structured, and contain fewer severe errors, allowing SFT to extract useful reasoning patterns despite the presence of mistakes.
A.9.1 ANALYSIS OF ERROR FREQUENCIES
We examine whether RLVR reduces the severity of reasoning errors. Using gpt-oss- 120b verifica- tion, we categorize four common error types, counting each category independently.
Figure 13 shows that RLVR CoTss have substantially lower error rates across all categories, indicating that RLVR leads more stable and higher-quality reasoning paths.
0.4
Qwin2.35 5-328
Qwen-32e
0.3
Frequency
Calculation Error
Conceptual Error Error Types
Logical Error
Or ssion Incompleteness
Figure 13: Error frequencies in CoTs with correct answers on AIME 2025, verified by gpt-oss- 120b.
RLVR CoTs exhibit lower eπrоr rates across all categories, reflecting higher- higher-quality reasoning paths.
A.9.2 TOKEN LENGTH ANALYSIS CORRECT AND INCORRECT CoTs
compare the token-length distributions of CoTs classified as correct or incorrect for the base model (Qwen2.5-32B) and the of token lengths on the DAPO-17K training set.
model DOAPO-Qwen-32B)) Table I reports quantiles
A key observation is that for DAPO- (HDD--wen-3255 incorrect CoTs are substantially longer than CoTs from the base model. This suggests that even when marked as incorrect, RLVR CoT's cover extended reasoning chains that include high-quality logical steps. Consequently, SFT these incorrect CoTs can still extract useful reasoning patterns and generalize effectively, despite the presence of some errors.
Table 1: Token-length quantiles of correct and incorrect CoT' for Qwen2.5-32B (base) and DAPO- Qwen-32B (RLVR) on DAPO-17K train set.
0.05
0.1
0.25
0.5
0.75
0.9
0.95
DAPO-Qwen-32B incorrect CoTs
1220
1494
2239
3995
7183
10929
13295
DAPO-Qwen-32B correct Co''s
1026
1185
1557
2266
3579
5730
7755
Qwen2.5-32B incorrect CoTs
610
666
783
950
1183
1504
1850
Qwen2.5-32B correct CoTs
623
717
1066
1343
1586
32

---

Published as a conference paper at ICLR 2026
A.10 DETAILED VERIFICATION RESULTS ON AIME BENCHMARKS
Tables 2 and report per-problem verification results on AIME2024 and AIME2025 using gpt- oss- 12s-.20b. For each problem, aggregate statistics over N 1024 sampled CoTs from both the RLVR-trained model and the base model. Each CoT is verified three times, and correctness is determined by the Majority-corres criterion described Section A.2
We include the following metries:
CoT-Pass@k k (A: 1024): whether at least one CoT is correct.
#CC: number of correct CoT'.
@ass@k (k- 1024): whether least answer is correct.
#CA: number of correct answers.
All evaluations use the OpenAl Responses API with reasoning.effortssiigg and max output tokens=64k.
Table 2: gpt-oss-120b verification results AIME 2024. Metries per problem aggregated over
N 1024 CoTs using the Majority-corred criterion.
RLVR
BASE
CoT-Pass
#CC
2024 AIME
Problem [url
1022
1023
14
2024 AIME
Problem
1024
366
2024 AIME
Problem 1r
1015
1024
1.99
183
2024 AIME 11
Problem
899
1023
165
249
2024 AIME
Problem
848
904
357
2024 ATME
Problem 6 [Ur]
523
879
6
2024 AIME
Problem 10 [url
522
914
4
2024 ATME
Problem [ur]
484
928
2024 AIME
Problem ur]
398
550
2024 AIME
Problem [uz]
155
306
12
2024 AIME
Problem 7
[z)
127
1023
53
150
ATME
175
266
2024
AIME
Problem 13 inl
22
666
23
194
2024
AIME
Problem 13
url
544
870
0
81
Problem
280
2024
AIME
Problem 1x url
86
226
0
36
2024
AIME
Problem 15
UE1
48
1010
0
0
2024 AIME
Problem
645
2024 ATME
Problem 5
(ril
483
2024 AIME
Problem
15
2024 AIME:
Problem 1D url
2024 AIME
Problem [ri]
229
0
0
2024 ATME
Problem urr
2024 AIME
Problem
27
0
2024 AIME
Problem 11
un1
1
2024 AIME
Problem
0
0
0
33

---

Published as a conference paper at ICLR 2026
Table 3: 1ot-oss--20b verification results on AIME 2025. Metrics problem are aggregated over N 1024 CoTs using the Hajority-correst criterion.
RLVR
BASE
Problem ID)
CrT-Pass@k
ACC
Pass@lk
#CA
CoT-Pasckk
#CC
Pass@k
#CA
2025 AIME
Problem
uri
1022
1024
17
2025 AIME
Problem
1020
1024
2025 AIME
Problem 2
1000
1024
177
237
2025 AIME
Problem 3
992
69
2025 AIME
Problem 6
940
1024
127
176
2025 AIME
Problem 4
Um
707
833
18
48
2025 AIME
Problem
1024
214
2025 AIME 1-
Problem 8 um
296
452
7
Problem
2025 AIME
Problem 2 ur
978
1016
2025 AIME
Problem 5
uc
623
2025
AIME 11 Problem
209
457
2
2025
AIME
Problem 14 LP]
151
170
2025
AIME
Problem 11 Ln
103
141
2025
AIME
Problem 6
IEL
454
26
2025
AIME
Problem 1
[url
0
2025 ATME
Problem 7
[r)
2025 AIME
Problem [ur]
0
2025 AIME
Problem Grll
108
2023 AIME 1- Problem 13 (url
2025 AIME
Problem
2025 AIME
Problem turl
6
2025 AIME
Problem uri
2025 AIME
Problem
url
2025 AIME
Problem 13
0
2025 AIME
Problem (i
34


Reviews:
[{"review_id": "RgfzrAXUMd", "reviewer": "Reviewer_RnmQ", "summary": "This paper investigates whether Reinforcement Learning with Verifiable Rewards (RLVR) genuinely improves the reasoning capability of large language models, rather than merely increasing sampling efficiency. It provides both theoretical analysis and empirical evaluation to show that RLVR implicitly incentivizes correct reasoning chains. The authors introduce a new evaluation metric, CoT-Pass@K, which verifies both intermediate reasoning correctness and final answers, and demonstrate that GRPO-based RLVR improves reasoning quality across multiple math and code benchmarks. Results suggest that RLVR not only reinforces correctness but also enhances reasoning consistency and generalization.", "questions": "- Could RLVR’s implicit reasoning incentive be combined with explicit step-wise rewards to further boost reasoning reliability?\n- It'd be great to see more details on training cost/efficiency and scaling behavior", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a well-structured theoretical analysis explaining how GRPO gradients implicitly favor logically consistent reasoning traces, offering insights on why RLVR often improves reasoning without explicit step-wise rewards.\n\n2. Experiments on multiple reasoning domains (math and code) and models (Qwen, Nemotron) show consistent gains under both Pass@K and the proposed CoT-Pass@K metrics. The analyses of training dynamics and reasoning correctness lend strong empirical support to the theoretical claims.\n\n3. The new CoT-Pass@K metric could be a useful tool for the community, bridging the gap between evaluating correctness of answers and correctness of reasoning.", "weaknesses": "1. The reliability of CoT-Pass@K hinges on judgments from a single verifier (DeepSeek-R1-Qwen3-8B). While multi-verification is discussed, evaluating under different verifiers would improve confidence in these results. How robust are CoT-Pass@K results when using different verifiers? -- we can fix the verifier used in training and just use different verifiers for evaluation.\n\n2. Potentially questionable assumptions. The theoretical analysis assumes that reasoning traces with correct answers have systematically higher probabilities, which may not always hold in domains like commonsense or multi-hop reasoning. The practical validity of this assumption could be discussed more thoroughly. Do we have any empirical examples?\n\n3. While the paper shows improved correctness, it does not analyze whether RLVR reduces reasoning diversity or leads to overconfident, deterministic reasoning traces. Add some quantitative analysis of reasoning diversity would further strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "opu4mP5Mvn", "reviewer": "Reviewer_BkYC", "summary": "* **Starting Point**: The paper aims to argue that **RLVR truly enhances the model's capabilities**\n* **New Metric**: The paper proposes the **CoT-Pass@K** metric, which is better suited than the commonly used Pass@K to demonstrate the **model's capability improvement after RLVR**\n* **Experimental Validation**: The experiments verify that for multiple math tasks, **CoT-Pass@K better highlights the performance gap** in model capabilities compared to Pass@K", "questions": "1. Why opt for a smaller model to judge the reasoning paths? Doesn't the accuracy of the judgment become constrained by the capacity of the referee model? Should a more powerful referee model be employed instead?\n\n2. For tasks where a direct reward signal is unavailable in standard RL training, the common approach is often to use a rubric combined with a powerful evaluation model to assign rewards. Given this, could your CoT-Pass@K metric be integrated into the RLVR training process to yield beneficial results?\n\n3. Based on the experimental results, we observe that the CoT-Pass@K score for 'no-thinking' models is generally suppressed compared to their Pass@K score. However, this method does not appear to provide a significantly better evaluation metric for RLVR models (as the difference between Pass@K and CoT-Pass@K for RLVR seems minor). Therefore, what are the practical applications or use cases for your proposed method?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper reads smoothly and presents its ideas in a clear, easy-to-follow style.\n2. The experimental design is comprehensive and well thought-out.\n3. RLVR provides only marginal gains at Pass@1024 or Pass@2048, but demonstrably boosts Pass@1—a widely recognized pattern. This supports the prevailing view that RLVR achieves its effect by learning correct reasoning, a claim for which your experiments offer some evidence.", "weaknesses": "Please refer to the \"Questions\" section for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jHUcx0Qdq6", "reviewer": "Reviewer_jCat", "summary": "This manuscript investigates a fundamental open question in reinforcement learning with verifiable rewards (RLVR): does RLVR merely improve sampling efficiency, or does it endow the base model with fundamentally new reasoning capabilities? The authors first revisit the Pass@K metric, which the RL community has often considered a signal of exploration, and identify its potential pitfalls. Specifically, base LLMs are capable of producing incorrect Chains of Thought (CoTs) yet coincidentally arriving at the ground truth. To address this, they introduce a new metric, CoT-Pass@K, which simultaneously considers the correctness of both the reasoning trace and the final answer. By re-examining the model's reasoning performance with CoT-Pass@K, the authors observe that RLVR successfully extends the reasoning boundary of the base model in both code and math tasks. The manuscript also presents primary theorems on GRPO's advantage estimation to interpret these empirical observations.", "questions": "1. How would the authors define the correctness of a CoT? That is, if a model makes mistakes initially but then corrects them in subsequent tokens through self-reflection, should such a CoT be considered incorrect?\n\n2. It would be good to see the influence of SFT on both Pass@K and CoT-Pass@K. For example, DAPO-Qwen-32B and DeepSeek-R1-Distilled-Qwen-32B have similar avg@32 on AIME24. It would be interesting to see which model performs better when using CoT-Pass@K as a metric.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The exact role of RLVR in enhancing the reasoning capabilities of base models is an open and fundamental problem for the RL community. Thus, this manuscript begins with a very strong motivation and is, in my view, highly timely. \n\n- Although the proposed Theorem 1 is relatively simple and straightforward, this preliminary explanation is helpful for the community to understand the underlying reasons why RLVR works. \n\n- The authors base their conclusions on extensive experiments. I particularly appreciate that they not only experimented with the Qwen+Math setup but also conducted additional experiments on coding tasks.", "weaknesses": "- The correctness of a CoT is determined by a R1-distilled-Qwen3-8B model, which may not be a sufficiently strong verifier. Since reasoning models may frequently exhibit rethinking or backtracking behavior, I have concerns about whether an 8B model is reliable enough to verify complex reasoning traces that include self-correction, reflection, and backtracking. For example, what if a model initially thinks incorrectly but then corrects its answer on its own? Is the distilled 8B model strong enough to verify such a CoT? The authors acknowledge this limitation in line 468, but I still feel it is noteworthy. Stronger frontier close-source LLMs maybe a better choice, e.g., Gemini-2.5-pro, if the cost of API calls is affordable.\n\n- The empirical observations from the post-SFT experiments (lines 429-431) are difficult for me to follow. It is counterintuitive to me that: 1) tuning on erroneous CoTs can also improve Pass@1 accuracy, ultimately leading to nearly the same performance, and 2) SFT on all CoTs results in worse accuracy than SFT on only incorrect CoTs. Does this observation indicate that the \"incorrect\" CoTs identified by Distilled-Qwen3-8B only contain erroneous steps at the beginning (which the model self-corrects later) rather than the entire reasoning trace being wrong? And thus SFT on these partially incorrect reasoning traces can also benefits pass@1 accuracy. If this is true, I have further concerns about the reliability of the CoT verifier and the definition of a \"correct\" CoT.\n\nIn summary, I am positive about this manuscript, and I find its findings inspiring and important. However, due to the two concerns mentioned above, I do not have enough confidence to give it a higher rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)