Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs

## Abstract

Abstract Recent advancements in long chain-of-thought (CoT) reasoning, particularly through the Group Relative Policy Optimization algorithm used by DeepSeek-R1, have led to significant interest in the potential of Reinforcement Learning with Verifiable Rewards (RLVR) for Large Language Models (LLMs). While RLVR promises to improve reasoning by allowing models to learn from free exploration, there remains debate over whether it truly enhances reasoning abilities or simply boosts sampling efficiency. This paper systematically investigates the impact of RLVR on LLM reasoning. We revisit Pass@K experiments and demonstrate that RLVR can extend the reasoning boundary for both mathematical and coding tasks. This is supported by our introduction of a novel evaluation metric, CoT-Pass@K, which captures reasoning success by accounting for both the final answer and intermediate reasoning steps. Furthermore, we present a theoretical framework explaining RLVR’s incentive mechanism, demonstrating how it can encourage correct reasoning even when rewards are based solely on answer correctness. Our analysis of RLVR’s training dynamics reveals that it incentivizes correct reasoning early in the process, with substantial improvements in reasoning quality confirmed through extensive evaluations. These findings provide strong evidence of RLVR’s potential to enhance LLM reasoning, offering valuable insights into its mechanisms and performance improvements.

## 1 Introduction

The successful replication of long chain-of-thought (CoT) reasoning, similar to that in OpenAI’s o1 (OpenAI, 2024), by DeepSeek-R1 (Guo et al., 2025) using the Group Relative Policy Optimization (GRPO) algorithm (Shao et al., 2024), has sparked a surge of interest within the open research community. This interest is focused on understanding, reproducing, and extending DeepSeek’s approach, as evidenced by a multitude of recent studies (Liu et al., 2025b; Hu et al., 2025; Zeng et al., 2025; Yu et al., 2025; He et al., 2025; Wen et al., 2025; Chen et al., 2025b). Fundamentally, this emerging paradigm is a form of Reinforcement Learning with Verifiable Rewards (RLVR) (Lambert et al., 2024; Guo et al., 2025; Yue et al., 2025), where a Large Language Model (LLM) acts as a policy, generating a CoT as a sequence of actions and receiving feedback on answer correctness from deterministic verifiers. This paradigm holds the promise of endowing LLMs with the ability to learn from experience through free exploration, potentially leading to unlimited intelligence (OpenAI, 2024; Guo et al., 2025; Silver & Sutton, 2025).

However, emerging concerns question the true effectiveness of RLVR. These concerns are motivated by the observation that some post-RLVR models improve the Pass@1 metric but fail to enhance the Pass@K metric compared to the base (pre-RLVR) model. This phenomenon was first noted by Shao et al. (2024) during the development of GRPO. Subsequently, a systematic study by Yue et al. (2025) on more open-weight RLVR models discovered that the Pass@K metric of the base model increases at a much faster rate than its RLVR-tuned counterpart. Consequently, for a moderately large K, the base model eventually matches and surpasses the reasoning model. This led to their adventurous hypothesis: all correct reasoning paths are already present in the base model, and RLVR merely improves sampling efficiency at the cost of reducing overall reasoning capacity.

While this hypothesis has gained significant support (Zhu et al., 2025; Zhang et al., 2025; Wang et al., 2025a; Chen et al., 2025a), conflicting observations have also been reported. For instance, Liu et al. (2025a) detected the emergence of new reasoning patterns after RLVR, while they also acknowledged a loss in reasoning capacity as measured by Pass@K. Chen et al. (2025b) reported persistent improvements in Pass@K for competitive coding tasks but did not show improved Pass@K for math tasks. Shojaee et al. (2025) observed similar Pass@K observations on math datasets but found different patterns on puzzles with high complexity.
To the best of our knowledge, no systematic explanation exists to reconcile these contradictory findings, leaving a critical question unanswered: “*should we accept the hypothesis as a fundamental limitation of RLVR or should we trust new empirical findings that challenge the hypothesis?*”

In this work, we address this debate systematically and demonstrate that RLVR can fundamentally enhance the reasoning abilities of LLMs.
First, we revisit Pass@K experiments and unveil the existence of extended reasoning capability boundaries after RLVR for both math and code tasks.
In addition to reproducing the extended reasoning boundary in competitive coding, as reported by Chen et al. (2025b), we find that the Pass@K performance of base LLMs on math reasoning can be unreliable, as base LLMs are capable of producing incorrect CoTs yet coincidentally arriving at the ground truth, especially for hard mathematical questions where answers are simple and can be easily guessed after multiple attempts.
To address this, we introduce a new metric, CoT-Pass@K, which evaluates success only when both the final answer and the intermediate reasoning CoT are correct.
In practice, we verify the correctness of mathematical CoTs by instructing DeepSeek-R1-0528-Qwen3-8B (DeepSeek, 2025) and confirm their reliability.
Using this new metric, we successfully identify the extended reasoning boundary of a post-RLVR model for math tasks.
Figure [1](https://arxiv.org/html/2506.14245v2#S0.F1) summarizes our key perspectives.

Moreover, we develop a theoretical framework to explain why RLVR works, even when base LLMs may guess the ground truth and only answer correctness is provided as a reward, and how RLVR incentivizes correct reasoning. Our central insight is that once LLMs have been pre-trained to establish strong knowledge and logic priors that distinguish correct from incorrect CoTs, the GRPO gradient will increase the probability of generating more correct CoTs.

Additionally, we investigate the training dynamics of RLVR to understand when this improved reasoning emerges. By reproducing GRPO-style training using the open-source DAPO recipe (Yu et al., 2025) and performing extensive verifications, we find that RLVR begins to incentivize correct reasoning from the early stages of training, and this capability generalizes well to unseen test questions. The results of our training analysis align with our theorem, which highlights the implicit incentivization of correct reasoning CoTs.

Finally, we evaluate the quality of generated CoTs from a learning perspective: if supervised learning on some CoT data results in better generalization performance on test sets, we regard them as high quality. This allows us to evaluate the quality of CoTs generated by model checkpoints at different RLVR stages. Our results show that after RLVR, the quality of reasoning CoTs has been fundamentally improved.

In summary, our contributions include:

- •
A systematic evaluation revealing the extended reasoning capability boundary after RLVR for both code and math tasks.
- •
A theoretical understanding of why RLVR works with only answer correctness as a reward and how RLVR incentivizes correct reasoning.
- •
An analysis of RLVR’s training dynamics, delving deeper into optimization effects, generalization behaviors, and current limitations.
- •
Confirmation of the quality improvements in reasoning CoTs from a learning perspective, replicating the generalization abilities of post-RLVR models trained with enormous costs simply via supervised fine-tuning.

## 2 Related Work

##### RLVR

Since the release of DeepSeek-R1 (Guo et al., 2025), there has been a surge of research interest in the RLVR paradigm (Luo et al., 2025b; Liu et al., 2025b; Hu et al., 2025; Cui et al., 2025; Xie et al., 2025; Zeng et al., 2025; Yu et al., 2025; Luo et al., 2025a; Chen et al., 2025a; He et al., 2025; Wen et al., 2025; Cao et al., 2025; Liu et al., 2025a; Chen et al., 2025b).
Due to the high computational cost of RLVR, most studies have focused on small- to medium-sized models (up to 32B parameters).
These studies span a wide range of aspects, including training data curation, objective design, hyperparameter tuning, base model selection, and various insightful observations.
However, only a few studies have addressed the theoretical foundations of RLVR.
In this work, we argue that RLVR for LLMs should be understood from a different perspective—one that emphasizes the correctness of reasoning paths.
We hope our empirical findings and theoretical perspective could inspire the community to develop more efficient and effective RLVR approaches, unlocking its broader potential across diverse applications.

##### Debates on Whether RLVR Really Incentivizes

Since Yue et al. (2025) raised the insightful question of whether RLVR truly incentivizes improvements beyond the base LLMs, and conducted extensive empirical experiments to demonstrate the wide applicability of their key hypothesis—that RLVR does not improve $Pass@K$ for the base LLM because all reasoning paths are already present in the base model—there have been varying perspectives on this hypothesis.
Some studies agree with this viewpoint (Wang et al., 2025b; Zhu et al., 2025; Zhang et al., 2025; Wang et al., 2025a; Chen et al., 2025a), while others report contradictory findings (Liu et al., 2025a; Chen et al., 2025b; Shojaee et al., 2025), as discussed in the introduction.
There is currently no fundamental understanding to resolve these debates.
Liu et al. (2025a) speculated that previous RLVR experiments may have been conducted within a single domain (e.g., math) and were optimized for limited gradient steps before true exploration could occur.
Shojaee et al. (2025) suggested that the complexity of puzzles might be the key factor.
Chen et al. (2025b) presented statistically significant empirical results to justify that their model indeed improves $Pass@K$, particularly highlighting a persistent gap on the LiveCodeBench v6 (Jain et al., 2025).
In this work, we extend the Pass@K experiments of Chen et al. (2025b) to more LiveCodeBench versions and include another similar open-source study (He et al., 2025) starting RLVR from distilled LLMs as a comparison.
All these results clearly disclose the extended reasoning boundary of distilled LLMs on competitive coding after RLVR.

##### The Importance of Correct CoTs

Recent studies have also highlighted the importance of verifying the correctness of CoTs (Arcuschin et al., 2025; McGinness & Baumgartner, 2025; Shojaee et al., 2025).
However, their approaches focus on defining synthetic reasoning tasks where the correctness of reasoning CoTs can be verified easily.
While this is an interesting and effective approach for fully examining reasoning correctness, it is difficult to apply to unstructured reasoning scenarios, such as in math and code.
In this work, we argue that the LLM-as-a-CoT-Judge paradigm could play a crucial role in more general reasoning tasks, and emphasize the pressing need for the design of evaluation benchmarks to assess the reliability of emerging LLM verifiers.

## 3 Extended Reasoning Capability Boundary After RLVR

In this section, we present concrete benchmark evaluations that demonstrate how RLVR can fundamentally enhance the reasoning abilities of LLMs. This enhancement goes beyond mere improvements in sampling efficiency; it also expands the reasoning capability boundary. However, to effectively observe this enhancement, it is crucial to adopt an appropriate RLVR training recipe, select challenging benchmarks that are free from data contamination, and utilize reliable evaluation metrics. Without these measures, one might only observe improvements in sampling efficiency, with no actual change in reasoning capacity. Below, we discuss two representative cases from both the math and code domains, showcasing genuinely extended reasoning boundaries.

### 3.1 Math Reasoning

Figure: Figure 2: Comparisons of Pass@K (the top row) and CoT-Pass@K (the bottom row) on five math benchmarks (different columns) to show how RLVR could improve base LLMs. Here the base LLM is Qwen2.5-32B, and the post-RLVR model is DAPO-Qwen-32B. For CoT-Pass@K, we perform multiple verifications for each CoT using DeepSeek-R1-0528-Qwen3-8B, and display the results determined by *any-correct*, *all-correct*, and *majority-correct* strategies, which constitute the shaded area in lower subplots.
Refer to caption: x2.png

We begin by revisiting the Pass@K experiments conducted on the open-source model, DAPO-Qwen-32B (Yu et al., 2025), which successfully reproduced R1-Zero (Guo et al., 2025) using the base LLM, Qwen2.5-32B (Qwen, 2024), and a curated set of $17k$ mathematical problems. A key contribution of our work is the introduction of a novel evaluation metric, CoT-Pass@K, which emphasizes the importance of evaluating the correctness of detailed reasoning steps for mathematical questions, rather than relying solely on answer correctness.

However, precisely measuring the correctness of CoTs at scale is inherently challenging due to the unstructured, lengthy, and complex nature of math CoTs. Fortunately, we can leverage specialized open-source LLMs, such as DeepSeek-R1-0528-Qwen3-8B (DeepSeek, 2025), as a powerful yet lightweight verifier, employing an LLM-as-a-CoT-Judge paradigm. In this study, we use this verifier multiple times for each reasoning CoT generated by DAPO-Qwen-32B and Qwen2.5-32B, employing three distinct strategies to assess CoT correctness: *any-correct* (at least one verification returns correct), *all-correct* (all verifications must return correct), and *majority-correct* (the majority vote determines the outcome). To further ensure the reliability of these verifications, we manually inspect cases where the Pass@K metric yields a small positive value, but the CoT-Pass@K metric returns zero.
The details of our LLM-as-a-CoT-Judge approach can be found in Appendix [A.3](https://arxiv.org/html/2506.14245v2#A1.SS3).

Figure [2](https://arxiv.org/html/2506.14245v2#S3.F2) presents a comprehensive comparison using both Pass@K and CoT-Pass@K metrics across prominent math-reasoning benchmarks. As shown in the top row, the Pass@K results align with the observations in (Yue et al., 2025): the performance of the base LLM quickly catches up with and even surpasses the post-RLVR model as $K$ increases. However, in stark contrast, the CoT-Pass@K results on AIME 2024 and AIME 2025 reveal a consistent and significant performance gap between the models across all values of $K$ (up to 1024). This gap is particularly pronounced on AIME 2025, possibly due to its complete absence of unintentional data contamination, as it was released after the base model’s training cutoff. We perform manual inspections to ensure the distinct gaps observed using the CoT-Pass@K metric are reliable (see examples in Appendix [A.8.1](https://arxiv.org/html/2506.14245v2#A1.SS8.SSS1) and [A.8.2](https://arxiv.org/html/2506.14245v2#A1.SS8.SSS2)). Our LLM verifier effectively identifies critical errors, which we agree should be rejected. These results demonstrate the extended reasoning boundary of DAPO-Qwen-32B over Qwen2.5-32B.

Additionally, we observe that on other benchmarks such as MATH-500 and AMC23, the effects of RLVR seem less pronounced, as the base LLM is already capable of solving these problems correctly with sufficient attempts. This may be due to 1) the problems being simple enough for the base LLM to solve using its existing knowledge, or 2) the problems being part of its pre-training data, allowing the base LLM to recall the correct solution after multiple trials. Distinguishing between these possibilities is challenging without knowing the exact training data used for Qwen2.5-32B. Furthermore, on the Minerva benchmark, the post-RLVR model shows no improvement, likely due to a train-test domain mismatch. Minerva contains numerous physics problems and free-form answers, while the DAPO training data was restricted to math problems with integer answers. These results do not undermine the effectiveness of RLVR; rather, they highlight the importance of selecting appropriate benchmarks for evaluating RLVR progress.

### 3.2 Code Reasoning

Figure: Figure 3: Comparisons of Pass@K across six LiveCodeBench versions to show how much RLVR could enhance distilled LLMs. Here the distilled LLM is DeepSeek-R1-Distill-Qwen-7B, and the post-RLVR model is AceReason-Nemotron-7B.
Refer to caption: x3.png

Unlike RLVR for mathematical problems, where the correctness of extracted answer tokens is used as a proxy for passing, code reasoning relies on the actual execution of generated code snippets to verify correctness, significantly reducing the likelihood of guessing. Therefore, Pass@K serves as a reliable evaluation metric for code reasoning tasks.

In this section, we reproduce the Pass@K experiments across different versions of LiveCodeBench (Jain et al., 2025) to compare the performance of AceReason-Nemotron-7B (Chen et al., 2025b) with its pre-RLVR counterpart, DeepSeek-R1-Distill-Qwen-7B (Guo et al., 2025). As shown in Figure [3](https://arxiv.org/html/2506.14245v2#S3.F3), we observe that AceReason-Nemotron-7B exhibits clear Pass@K improvements over DeepSeek-R1-Distill-Qwen-7B on most benchmark versions, even though the latter is a distillation model already demonstrating remarkable reasoning capabilities. These results suggest that even for distillation models, a high-quality RLVR training recipe can significantly extend the reasoning capability boundary, particularly for competitive coding tasks.

To further confirm the success of RLVR in extending reasoning boundaries for coding tasks, we evaluate another post-RLVR model, Skywork-OR1 (He et al., 2025), which has a fully reproducible training recipe publicly available. Detailed results on LiveCodeBench-v6 can be found in Appendix [A.4](https://arxiv.org/html/2506.14245v2#A1.SS4), where we observe a consistent Pass@K gap between Skywork-OR1 and DeepSeek-R1-Distill-Qwen-7B. Specifically, we find that only medium and hard problems in LiveCodeBench-v6 contribute to the differentiation between these two models for large $K$ values, underscoring the importance of selecting challenging benchmarks.

## 4 A Theoretical Understanding of RLVR for LLMs

In addition to empirical evidences, we provide a theoretical understanding of how RLVR, as implemented in the GRPO algorithm (Shao et al., 2024), fundamentally incentivizes correct reasoning for pre-trained language models. We note a key distinction between RLVR for LLMs and traditional RL for randomly initialized models. Pre-trained LLMs, owing to their powerful likelihood estimation capabilities obtained during pre-training, can generate various CoTs and then produce possible answers. Some of them could coincidentally arrive at the ground truth, especially when the ground truth is in a simple format and can be easily guessed. In contrast, traditional RL simply optimizes for action trajectories that yield high rewards, without necessarily verifying the intrinsic correctness of each action along the path. For instance, in the Go game (Silver et al., 2017), every action is valid once the simulation environment is setup correctly. In the following, we start our theoretical analysis from a formal problem setup distinguishing CoT and answer tokens in LLM responses.

##### Problem Setup

Given a question prompt $q$ , we sample $G$ responses $\mathbf{Y}=\{y_{1},y_{2},\dots,y_{G}\}$ from policy $\pi_{\theta}$, where $\pi_{\theta}$ is a LLM model parameterized by $\theta$. Let $c_{i}$ be the CoT in response $y_{i}$, and $a_{i}$ the final answer. We define the following correctness indicators:

$$ $\mathcal{I}_{\text{CoT}}(c_{i})=\begin{cases}1&\text{if }c_{i}\text{ is correct}\\ 0&\text{otherwise}\end{cases},\quad\mathcal{I}_{\text{Ans}}(a_{i})=\begin{cases}1&\text{if }a_{i}\text{ is correct}\\ 0&\text{otherwise}\end{cases}.$ (1) $$

In this study, we define the CoT correctness $\mathcal{I}_{\text{CoT}}(c_{i})$ as the intermediate tokens of a response ($c_{i}$) expressing necessary and accurate logics that lead to the ground truth. We use $p_{c}^{\theta}=P_{\pi_{\theta}}(\mathcal{I}_{\text{CoT}}(c)=1)$ to denote the probability of generating a correct CoT.
In practice, it is rather challenging to verify the CoT correctness because it is inherently unstructured, knowledge-intensive, and full of details.
In contrast, the answer correctness $\mathcal{I}_{\text{Ans}}(a_{i})$ is assumed to be verified programmatically.
So we have a verifiable reward $R(y_{i})$ that is binary and determined solely by answer correctness: $R(y_{i})=\mathcal{I}_{\text{Ans}}(a_{i})$.
We calculate the standard GRPO advantage $\hat{A}(y_{i})$ as:

$$ $\hat{A}(y_{i})=\frac{R(y_{i})-\mu_{\mathbf{Y}}}{\sigma_{\mathbf{Y}}},\quad\mu_{\mathbf{Y}}=\frac{1}{G}\sum_{j=1}^{G}R(y_{j}),\quad\sigma_{\mathbf{Y}}=\sqrt{\frac{1}{G}\sum_{j=1}^{G}(R(y_{j})-\mu_{\mathbf{Y}})^{2}}.$ (2) $$

Without loss of generality, we consider a policy gradient (Sutton et al., 1999) update:

$$ $\nabla_{\theta}J(\theta)\approx\frac{1}{G}\sum_{i=1}^{G}\hat{A}(y_{i})\nabla_{\theta}\log\pi_{\theta}(y_{i}\mid q).$ (3) $$

##### Assumptions

Given the problem setup decoupling CoT and answer correctness, we introduce a critical *Logic Prior* assumption: compared with incorrect CoTs, correct CoTs have higher probabilities to induce correct answers. Thus we have

$$ $P(\mathcal{I}_{\text{Ans}}(a_{i})=1\mid\mathcal{I}_{\text{CoT}}(c_{i})=1)=\alpha>P(\mathcal{I}_{\text{Ans}}(a_{i})=1\mid\mathcal{I}_{\text{CoT}}(c_{i})=0)=\beta.$ (4) $$

This assumption is based on the belief that pre-trained LLMs have established strong knowledge and logic priors.
Besides, we also assume a learnable group ($\sigma_{\mathbf{Y}}>0$) and a sufficiently large sampling number $G$ to ensure stable gradient updates.
Then, we establish the following theorem.

###### Theorem 1 (GRPO Implicitly Incentivizes Correct Reasoning)

For any prompt $q$ satisfying our assumptions, the expected GRPO advantage $\mathbb{E}[\hat{A}(y_{i})]$ satisfies:

$$ $\displaystyle\mathbb{E}\left[\hat{A}(y_{i})\mid\mathcal{I}_{\text{CoT}}(c_{i})=1\right]>0,\quad\mathbb{E}\left[\hat{A}(y_{i})\mid\mathcal{I}_{\text{CoT}}(c_{i})=0\right]<0,$ (5) $$

where $\hat{A}(y_{i})$ is defined in equation [2](https://arxiv.org/html/2506.14245v2#S4.E2).
The GRPO policy gradient, as defined in equation [3](https://arxiv.org/html/2506.14245v2#S4.E3), increase the probability of generating correct CoTs ($p^{\theta}_{c}$) in the next round, so $p^{\theta}_{c}$ increases monotonically.

Below we briefly illustrate our key perspectives on why GRPO works and when it may fail. Moreover, we include a complete proof for Theorem [1](https://arxiv.org/html/2506.14245v2#Thmtheorem1) and more discussions in Appendix [A.5](https://arxiv.org/html/2506.14245v2#A1.SS5).

###### Theorem 1 (GRPO Implicitly Incentivizes Correct Reasoning)

##### Discussions on the effectiveness of GRPO

Theorem [1](https://arxiv.org/html/2506.14245v2#Thmtheorem1) indicates that even though a base LLM may guess the ground truth with imperfect CoTs at the beginning (low initial $p^{\theta}_{c}$), GRPO could still work as long as the knowledge and logic priors have been established.
The driving factor is the gap $\alpha-\beta>0$, which amplifies the advantage difference between correct and incorrect CoTs.
As training progresses and $\alpha$ increases (due to more sound reasoning across various question prompts) while $\beta$ decreases (reducing spurious correlations, model biases, incorrect knowledge or calculation, etc.), causing the gap to widen and further accelerating coherent reasoning.
As $p_{c}\to 1$, $(\alpha-\beta)$ may approach $1$ in a faster pace because generating a few answer tokens is typically much easier than producing long correct CoTs, then $\mathbb{E}[\hat{A}(y_{i})\mid\text{correct CoT}]\to 0$, ensuring convergence.

##### Discussions on failure modes in GRPO

We note that the *Logic Prior* assumption may not always hold, potentially leading to the reinforcement of incorrect CoTs, since base LLMs may retain inherent biases and possibly fatal knowledge errors from pre-training. These harmful information might exist in some CoTs that finally yield the correct answer. In such cases, improper model biases could be unintentionally reinforced. We suspect that these unexpectedly reinforced CoTs are the root cause of the challenges faced by the R1-Zero approach (Guo et al., 2025), including poor readability and multi-lingual behaviors.

## 5 Training Dynamics of RLVR

To further demystify RLVR, we reproduce and analyze the training recipe of DAPO (Yu et al., 2025), which has been demonstrated to present extended reasoning capability boundaries in Section [3](https://arxiv.org/html/2506.14245v2#S3). Our experiments show that its training dynamics align pretty well with Theorem [1](https://arxiv.org/html/2506.14245v2#Thmtheorem1).

##### Key Indicators

We first introduce key indicators that we have recorded during the reproduction.
For each prompt $q$ sampled with $G$ responses, we define the number of answer passes and the number of both CoT and answer passes as $C=\sum_{i=1}^{G}\mathcal{I}_{\text{Ans}}(a_{i})$ and $D=\sum_{i=1}^{G}\mathcal{I}_{\text{CoT}}(c_{i})\cdot\mathcal{I}_{\text{Ans}}(a_{i})$, respectively.
We follow Chen et al. (2021)’s approach to calculate the Pass@K metric.
Accordingly, we have per-prompt metrics: $\text{Pass@K}^{(q)}=1-\frac{\binom{G-C}{K}}{\binom{G}{K}}$ and $\text{CoT-Pass@K}^{(q)}=1-\frac{\binom{G-D}{K}}{\binom{G}{K}}$.
Besides, we estimate the probability of producing correct answers for prompt $q$ as $P(CA)^{(q)}=\frac{C}{G}$ and the probability of producing correct CoTs when generating correct answers as $P(CC|CA)^{(q)}=\frac{D}{C}$.
For a dataset of multiple prompts, we take an average of per-prompt metrics as the dataset-level score.

Figure: Figure 4: The evolution of $P(CA)^{(q)}$ (the fraction of correct answers for prompt $q$) and $P(CC|CA)^{(q)}$ (the fraction of correct CoTs within the correct answers for prompt $q$) for fully optimized training questions over the course of DAPO training.
Refer to caption: x4.png

Figure: Figure 5: The evolution of Pass@K (the top row) and CoT-Pass@K (the bottom row) performance on AIME 2024 and 2025 for different model checkpoints during the DAPO training.
Refer to caption: x5.png

##### Optimization Effects

In our reproduced DAPO training, we observe that most training questions have been fully optimized.
As shown in Figure [4](https://arxiv.org/html/2506.14245v2#S5.F4), the probability of generating correct answers for these questions almost reach $1$.
In the meanwhile, we also observe the improvement in producing more correct reasoning CoTs, as indicated by the improvements in $P(CC|CA)^{(q)}$.
These observations validate the key perspective in Theorem [1](https://arxiv.org/html/2506.14245v2#Thmtheorem1): RLVR not only optimizes the final verifiable reward but also implicitly incentivizes correct reasoning.

##### Generalization Behaviors

In the meanwhile, Figure [5](https://arxiv.org/html/2506.14245v2#S5.F5) discloses that the optimization of RLVR leads to the generalization improvements of both Pass@K and CoT-Pass@K from the very beginning.
And using the CoT-Pass@K metric, we can clearly tell that the reasoning capability boundary has also been enhanced since the beginning.
Another interpretation for this empirical observation is that the model has learned to produce more and more reasoning CoTs that DeepSeek-R1-0528-Qwen-8B cannot identify any error.
This ability is learned implicitly since we have no formal supervision for the CoT correctness.

##### Limitations of DAPO

Figure [4](https://arxiv.org/html/2506.14245v2#S5.F4) also unveils the limitations of DAPO, the R1-zero approach applied to a 32B base LLM.
As $P(CA)^{(q)}$ approaches $1.0$ for most fully optimized training questions after 400 steps, which means these questions are no longer learnable because we cannot calculate a valid GRPO advantage for a all-correct group, we can still observe a non-negligible portion of imperfect CoTs (the median of $P(CC|CA)^{(q)}$ is around $0.7$).
These signals indicate that there are certain unexpected reasoning behaviors learned in DAPO and we may not have a chance to mitigate them purely based on answer correctness as the reward.

## 6 The Quality of Reasoning CoTs Enhanced by RLVR

In addition to the LLM-as-a-CoT-Judge approach for strictly identifying critical errors in reasoning CoTs, we further leverage supervised fine-tuning (SFT) to assess the quality of reasoning CoTs enhanced by RLVR.
Given the training questions in DAPO, we conduct multiple SFT procedures, starting from the same base LLM and learning from CoTs generated by different models. If the CoT data is of high quality, we expect the post-SFT model to exhibit improved generalization performance. Figure [7(a)](https://arxiv.org/html/2506.14245v2#S6.F7.sf1) presents an overall quality evaluation of various CoT data.

Figure: (a) We show the generalization performance of post-SFT LLMs optimized on different CoT data. All these SFT processes start from the same base LLM, Qwen2.5-32B, with the only variable being the different CoT data on DAPO training questions. We use the performance on test sets (AIME 2024, 2025) as a proxy for the quality of the corresponding CoT data.
Refer to caption: x6.png

##### Limitations

A key limitation of our study lies in the use of a LLM as the verifier for the correctness of reasoning CoTs, due to the prohibitive cost of manually checking a large volume of generated reasoning paths. Moreover, our theorem only explains the optimization process of RLVR but provides no guarantee for its generalization. We merely observe the generalization empirically.
Due to space limitations, further discussions on the implications of our findings are deferred to Appendix [A.7](https://arxiv.org/html/2506.14245v2#A1.SS7).

## 7 Conclusion

In this work, we address the fundamental problem of whether RLVR genuinely incentivizes novel reasoning in base LLMs. Through empirical evaluations and theoretical analysis, we justify a new perspective: RLVR implicitly incentivizes correct reasoning. Moreover, our analyses on training dynamics and CoT quality further confirm that the reasoning CoTs after RLVR are fundamentally different and can even help to replicate similar capabilities simply via supervised learning.

We hope these findings can not only resolve conflicting conclusions in prior work but also illuminate the untapped potential of RLVR in aligning LLMs with human reasoning systems. We envision a promising future where RLVR serves as a cornerstone for developing LLMs that learn through interaction, self-correction, and verifiable reasoning.

## References

- Anthropic (2025)
Team Anthropic.
Introducing Claude 4.
[https://www.anthropic.com/news/claude-4](https://www.anthropic.com/news/claude-4), 2025.
[Released 23-05-2025].
- Arcuschin et al. (2025)
Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy.
Chain-of-thought reasoning in the wild is not always faithful.
*arXiv preprint arXiv:2503.08679*, 2025.
- Cao et al. (2025)
Shiyi Cao, Sumanth Hegde, Dacheng Li, Tyler Griggs, Shu Liu, Eric Tang, Jiayi Pan, Xingyao Wang, Akshay Malik, Graham Neubig, Kourosh Hakhamaneshi, Richard Liaw, Philipp Moritz, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica.
SkyRL-v0: Train real-world long-horizon agents via reinforcement learning.
[https://novasky-ai.notion.site/skyrl-v0](https://novasky-ai.notion.site/skyrl-v0), 2025.
- Chen et al. (2025a)
Huayu Chen, Kaiwen Zheng, Qinsheng Zhang, Ganqu Cui, Yin Cui, Haotian Ye, Tsung-Yi Lin, Ming-Yu Liu, Jun Zhu, and Haoxiang Wang.
Bridging supervised learning and reinforcement learning in math reasoning.
*arXiv preprint arXiv:2505.18116*, 2025a.
- Chen et al. (2021)
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.
Evaluating large language models trained on code.
*arXiv preprint arXiv:2107.03374*, 2021.
- Chen et al. (2025b)
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping.
AceReason-Nemotron: Advancing math and code reasoning through reinforcement learning.
*arXiv preprint arXiv:2505.16400*, 2025b.
- Cui et al. (2025)
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al.
Process reinforcement through implicit rewards.
*arXiv preprint arXiv:2502.01456*, 2025.
- DeepSeek (2025)
Team DeepSeek.
DeepSeek-R1-0528 release.
[https://api-docs.deepseek.com/news/news250528](https://api-docs.deepseek.com/news/news250528), 2025.
[Released 28-05-2025].
- Gemini (2024)
Team Gemini.
Introducing Gemini 2.0: our new AI model for the agentic era.
[https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/](https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/), 2024.
[Released 11-12-2024].
- Gemini (2025)
Team Gemini.
Gemini 2.5: Our most intelligent models are getting even better.
[https://blog.google/technology/google-deepmind/google-gemini-updates-io-2025/](https://blog.google/technology/google-deepmind/google-gemini-updates-io-2025/), 2025.
[Released 20-05-2025].
- Grok (2025)
Team Grok.
Grok 3 Beta — The age of reasoning agents.
[https://x.ai/news/grok-3](https://x.ai/news/grok-3), 2025.
[Released 19-02-2025].
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- He et al. (2025)
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al.
Skywork open reasoner 1 technical report.
*arXiv preprint arXiv:2505.22312*, 2025.
- Hu et al. (2025)
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum.
Open-Reasoner-Zero: An open source approach to scaling up reinforcement learning on the base model.
*arXiv preprint arXiv:2503.24290*, 2025.
- Jain et al. (2025)
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica.
LiveCodeBench: Holistic and contamination free evaluation of large language models for code.
In *ICLR*, 2025.
- Kaplan et al. (2020)
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
Scaling laws for neural language models.
*arXiv preprint arXiv:2001.08361*, 2020.
- Lambert et al. (2024)
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al.
Tulu 3: Pushing frontiers in open language model post-training.
*arXiv preprint arXiv:2411.15124*, 2024.
- Lightman et al. (2024)
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
Let’s verify step by step.
In *ICLR*, 2024.
- Liu et al. (2024)
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.
DeepSeek-V3 technical report.
*arXiv preprint arXiv:2412.19437*, 2024.
- Liu et al. (2025a)
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong.
ProRL: Prolonged reinforcement learning expands reasoning boundaries in large language models.
*arXiv preprint arXiv:2505.24864*, 2025a.
- Liu et al. (2025b)
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin.
Understanding R1-Zero-like training: A critical perspective.
*arXiv preprint arXiv:2503.20783*, 2025b.
- Luo et al. (2025a)
Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica.
Deepcoder: A fully open-source 14b coder at o3-mini level.
https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349a512a51, 2025a.
Notion Blog.
- Luo et al. (2025b)
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica.
Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl.
https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2, 2025b.
Notion Blog.
- McGinness & Baumgartner (2025)
Lachlan McGinness and Peter Baumgartner.
Large language models’ reasoning stalls: An investigation into the capabilities of frontier models.
*arXiv preprint arXiv:2505.19676*, 2025.
- Mistral.AI (2025)
Team Mistral.AI.
Stands to reason. Magistral.
[https://mistral.ai/news/magistral](https://mistral.ai/news/magistral), 2025.
[Released 10-06-2025].
- OpenAI (2024)
Team OpenAI.
Learning to reason with LLMs.
[https://openai.com/index/learning-to-reason-with-llms/](https://openai.com/index/learning-to-reason-with-llms/), 2024.
[Released 12-09-2024].
- OpenAI (2025)
Team OpenAI.
Introducing OpenAI o3 and o4-mini.
[https://openai.com/index/introducing-o3-and-o4-mini/](https://openai.com/index/introducing-o3-and-o4-mini/), 2025.
[Released 16-04-2025].
- Qwen (2024)
Team Qwen.
Qwen2.5 technical report.
*arXiv preprint arXiv:2412.15115*, 2024.
- Qwen (2025)
Team Qwen.
Qwen3: Think deeper, act faster.
[https://qwenlm.github.io/blog/qwen3/](https://qwenlm.github.io/blog/qwen3/), 2025.
[Released 29-04-2025].
- Rafailov et al. (2023)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
In *NeurIPS*, 2023.
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*arXiv preprint arXiv:1707.06347*, 2017.
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al.
DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Sheng et al. (2025)
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu.
HybridFlow: A flexible and efficient RLHF framework.
In *EuroSys*, 2025.
- Shojaee et al. (2025)
Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar.
The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity.
*arXiv preprint arXiv:2506.06941*, 2025.
- Silver & Sutton (2025)
David Silver and Richard S Sutton.
Welcome to the era of experience.
*Google AI*, 2025.
- Silver et al. (2017)
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.
Mastering the game of Go without human knowledge.
*Nature*, 2017.
- Su et al. (2025)
Xuerui Su, Shufang Xie, Guoqing Liu, Yingce Xia, Renqian Luo, Peiran Jin, Zhiming Ma, Yue Wang, Zun Wang, and Yuting Liu.
Trust region preference approximation: A simple and stable reinforcement learning algorithm for LLM reasoning.
*arXiv preprint arXiv:2504.04524*, 2025.
- Sutton et al. (1999)
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour.
Policy gradient methods for reinforcement learning with function approximation.
In *NeurIPS*, 1999.
- Uesato et al. (2022)
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins.
Solving math word problems with process-and outcome-based feedback.
*arXiv preprint arXiv:2211.14275*, 2022.
- Wang et al. (2024)
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui.
Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations.
In *ACL*, 2024.
- Wang et al. (2025a)
Yibin Wang, Zhimin Li, Yuhang Zang, Chunyu Wang, Qinglin Lu, Cheng Jin, and Jiaqi Wang.
Unified multimodal chain-of-thought reward model through reinforcement fine-tuning.
*arXiv preprint arXiv:2505.03318*, 2025a.
- Wang et al. (2025b)
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al.
Reinforcement learning for reasoning in large language models with one training example.
*arXiv preprint arXiv:2504.20571*, 2025b.
- Wen et al. (2025)
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al.
Light-R1: Curriculum SFT, DPO and RL for long CoT from scratch and beyond.
*arXiv preprint arXiv:2503.10460*, 2025.
- White et al. (2025)
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum.
LiveBench: A challenging, contamination-limited LLM benchmark.
In *ICLR*, 2025.
- Xie et al. (2025)
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo.
Logic-RL: Unleashing LLM reasoning with rule-based reinforcement learning.
*arXiv preprint arXiv:2502.14768*, 2025.
- Yao (2025)
Shunyu Yao.
The second half.
[https://ysymyth.github.io/The-Second-Half/](https://ysymyth.github.io/The-Second-Half/), 2025.
- Ye (2025)
Shengyu Ye.
EvalHub: All-in-one benchmarking platform for evaluating large language models (LLMs) with comprehensive metrics and standardized testing frameworks.
[https://github.com/ysy-phoenix/evalhub](https://github.com/ysy-phoenix/evalhub), 2025.
[Accessed 16-06-2025].
- Yu et al. (2025)
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al.
DAPO: An open-source LLM reinforcement learning system at scale.
*arXiv preprint arXiv:2503.14476*, 2025.
- Yue et al. (2025)
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang.
Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?
*arXiv preprint arXiv:2504.13837*, 2025.
- Zeng et al. (2025)
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He.
SimpleRL-Zoo: Investigating and taming zero reinforcement learning for open base models in the wild.
*arXiv preprint arXiv:2503.18892*, 2025.
- Zhang et al. (2025)
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian.
Right question is already half the answer: Fully unsupervised LLM reasoning incentivization.
*arXiv preprint arXiv:2504.05812*, 2025.
- Zhu et al. (2025)
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng.
The surprising effectiveness of negative reinforcement in LLM reasoning.
*arXiv preprint arXiv:2506.01347*, 2025.

## Appendix A Appendix

### A.1 Contributions

We summarize individual contributions as follows:

- •
Xumeng Wen conducted all rollout and verification experiments and led the $Pass@K$ analysis on mathematical benchmarks.
- •
Zihan Liu performed meticulous examination of numerous mathematical reasoning cases and verified the reliability of the LLM-as-a-CoT-Judge paradigm.
- •
Shun Zheng designed this research, conducted RLVR experiments, identified CoT correctness as the key factor, built the theoretical framework, and wrote the manuscript.
- •
Shengyu Ye developed the EvalHub tool (Ye, 2025), supporting most rollout and verification experiments, and performed Pass@K experiments for AceReason-Nemotron-7B.
- •
Zhirong Wu conducted independent rollout studies and provided insightful thoughts during group discussions.
- •
Yang Wang provided crucial infrastructure support, enabling all experiments through AMD MI300 GPUs.
- •
Zhijian Xu contributed to training analysis and proposed the visualization of $P(\text{CC}|\text{CA})$.
- •
Xiao Liang shared valuable insights regarding mathematical benchmarks, training question selection, and effective mathematical verifiers.
- •
Junjie Li participated actively in group discussions and shared RL experiences.
- •
Ziming Miao provided additional infrastructure support.
- •
Jiang Bian and Mao Yang supervised the project and provided comprehensive support to the entire team.

### A.2 Data Sources

For math benchmarks studied in this paper, we leverage the following data sources:
AIME 2025(^1^11[https://huggingface.co/datasets/opencompass/AIME2025](https://huggingface.co/datasets/opencompass/AIME2025)),
AIME 2024(^2^22[https://huggingface.co/datasets/HuggingFaceH4/aime_2024](https://huggingface.co/datasets/HuggingFaceH4/aime_2024)),
Math-500(^3^33[https://huggingface.co/datasets/HuggingFaceH4/MATH-500](https://huggingface.co/datasets/HuggingFaceH4/MATH-500)),
AMC23(^4^44[https://huggingface.co/datasets/math-ai/amc23](https://huggingface.co/datasets/math-ai/amc23)),
Minerva(^5^55[https://huggingface.co/datasets/math-ai/minervamath](https://huggingface.co/datasets/math-ai/minervamath)).

For training and evaluation of DAPO (Yu et al., 2025), we reuse their training data ([https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k](https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k)) and processed version of AIME 2024 ([https://huggingface.co/datasets/BytedTsinghua-SIA/AIME-2024](https://huggingface.co/datasets/BytedTsinghua-SIA/AIME-2024)).
Please note that they have duplicated questions multiple times and explained in the dataset page that the purpose is to be compatible with an old version of VERL (Sheng et al., 2025).
We reuse the prompt template of DAPO to evaluate their $Pass@K$ and $CoT$-$Pass@K$ performance on other benchmarks.

Besides, we follow the official LiveCodeBench repository(^6^66[https://github.com/LiveCodeBench/LiveCodeBench](https://github.com/LiveCodeBench/LiveCodeBench)) to perform Pass@K evaluations on competitive coding.

### A.3 LLM-as-a-CoT-Judge for Math Reasoning

Figure: Figure 7: An intuitive diagram to illustrate the benefits of our multi-verification system: simultaneously considering *any-correct*, *all-correct*, and *majority-correct* helps us to mitigate false positives and false negatives within individual verifications.
Refer to caption: x6.png

We use a much more specialized LLM on mathematical reasoning (DeepSeek-R1-0528-Qwen3-8B) as the verifier to examine the reasoning steps of base LLM, Qwen2.5-32B.
Meanwhile, we also acknowledge the existence of verification errors and manually checked many of them to confirm the reliability of this verification.
To further mitigate potential verification errors, we design a multi-verification approach, as shown in Figure [7](https://arxiv.org/html/2506.14245v2#S6.F7)). For each reasoning CoT, we conduct multiple verifications independently and calculate three aggregation metrics:

- •
*All-correct*: Chains of Thought that pass all verification attempts
- •
*Majority-correct*: Chains of Thought that pass most verification attempts
- •
*Any-correct*: Chains of Thought that pass at least one verification attempt (capturing potential error recovery cases)

This multi-verification approach can ensure us to have a comprehensive view of CoT correctness because the *all-correct* strategy mitigates false positives and the *any-correct* option reduces false negatives.
Let $p_{\text{fp}}$ and $p_{\text{fn}}$ represent the per-attempt false positive and false negative rates, respectively. For $n$ independent verification attempts, we observe:

- •
*All-correct*: The false positive rate decays exponentially as $p_{\text{fp}}^{n}$
- •
*Any-correct*: The false negative rate decays exponentially as $p_{\text{fn}}^{n}$

In our study, we employ $n=3$ verification attempts for each CoT.

Moreover, we provide the prompt template used for DeepSeek-R1-0528-Qwen3-8B as follows.

### A.4 Revisiting Pass@K Experiments for Skywork-OR1

Skywork-OR1 (He et al., 2025) has generously shared their complete training recipes, claiming to enhance distilled LLMs with more powerful reasoning capabilities through RLVR. Therefore, we conduct the Pass@K experiments on their models to understand how RLVR could improve distilled LLMs.

Figure [8](https://arxiv.org/html/2506.14245v2#A1.F8) shows the Pass@K comparisons between Skywork-OR1-7B and DeepSeek-R1-Distill-Qwen-7B on LiveCodeBench-v6. We can also observe a significant improvement of both sampling efficiency (Pass@1) and reasoning boundary (Pass@K, K up to 1024). This conclusion is consistent with the observations for Figure [3](https://arxiv.org/html/2506.14245v2#S3.F3) in the main paper.

Figure: Figure 8: Comparisons of Pass@K on LiveCodeBench-v6 and its different difficulty-level subsets. Here the distilled LLM is DeepSeek-R1-Distill-Qwen-7B, and the post-RLVR model is Skywork-OR1-7B.
Refer to caption: x7.png

However, in math domains, applying RLVR to distilled LLMs seems to merely deliver sampling efficiency improvements.
As shown in Figure [9](https://arxiv.org/html/2506.14245v2#A1.F9), we observe that even using the CoT-Pass@K metric, Skywork-OR1-Math-7B and DeepSeek-R1-Distill-Qwen-7B do not have distinct Pass@K gaps for large K values.
We suspect the reason is that the distilled LLM may already master major reasoning capabilities that can be learned with RLVR using answer correctness as the reward.
So in math domains, their main improvements lie in Pass@1.
In contrast, for code domains, applying RLVR to distilled LLMs can still stimulate them to fit for real-world execution feedback, thereby incentivizing extended reasoning boundary.

Figure: Figure 9: Comparisons of Pass@K (the top row) and CoT-Pass@K (the bottom row) on AIME 2024, 2025 to show how RLVR could improve distilled LLMs. Here the distilled LLM is DeepSeek-R1-Distill-Qwen-7B, and the post-RLVR model is Skywork-OR1-Math-7B. For CoT-Pass@K, we perform multiple verifications for each CoT using DeepSeek-R1-0528-Qwen3-8B, and display the results determined by *any-correct*, *all-correct*, and *majority-correct* strategies, which constitute the shaded area in lower subplots.
Refer to caption: x8.png

### A.5 Proof and Additional Theoretical Analysis

Below we include the detailed proof for Theorem [1](https://arxiv.org/html/2506.14245v2#Thmtheorem1).

###### Proof 1

Let $p_{c}=P(\mathcal{I}_{\text{CoT}}(c_{i})=1)$ be the current probability of generating a correct CoT. The expected reward for a response $y_{i}$ is:

$$ $\mathbb{E}[R(y_{i})]=\begin{cases}\alpha&\text{if }\mathcal{I}_{\text{CoT}}(c_{i})=1\\ \beta&\text{if }\mathcal{I}_{\text{CoT}}(c_{i})=0\end{cases}$ (6) $$

The group-level expected reward $\mu\triangleq\mathbb{E}[\mu_{\mathbf{Y}}]$ is:

$$ $\mu=p_{c}\alpha+(1-p_{c})\beta.$ (7) $$

For large $G$, the group mean $\mu_{\mathbf{Y}}$ and variance $\sigma_{\mathbf{Y}}^{2}$ concentrate around their expectations:

$$ $\displaystyle\mu_{\mathbf{Y}}$ $\displaystyle\xrightarrow{G\to\infty}\mu$ (8) $\displaystyle\sigma_{\mathbf{Y}}^{2}$ $\displaystyle\xrightarrow{G\to\infty}\sigma^{2}>0.$ (9) $$

The expected advantage conditional on CoT correctness is:

$$ $\displaystyle\mathbb{E}[\hat{A}(y_{i})\mid\mathcal{I}_{\text{CoT}}(c_{i})=1]$ $\displaystyle\xrightarrow{G\to\infty}\frac{\alpha-\mu}{\sigma}$ (10) $\displaystyle\mathbb{E}[\hat{A}(y_{i})\mid\mathcal{I}_{\text{CoT}}(c_{i})=0]$ $\displaystyle\xrightarrow{G\to\infty}\frac{\beta-\mu}{\sigma}.$ (11) $$

Substituting equation [7](https://arxiv.org/html/2506.14245v2#A1.E7) into equation [10](https://arxiv.org/html/2506.14245v2#A1.E10) and equation [11](https://arxiv.org/html/2506.14245v2#A1.E11):

$$ $\displaystyle\mathbb{E}[\hat{A}(y_{i})\mid\text{correct CoT}]$ $\displaystyle\rightarrow\frac{(1-p_{c})(\alpha-\beta)}{\sigma}$ (12) $\displaystyle\mathbb{E}[\hat{A}(y_{i})\mid\text{incorrect CoT}]$ $\displaystyle\rightarrow\frac{-p_{c}(\alpha-\beta)}{\sigma}.$ (13) $$

Since $\alpha>\beta$ (by equation [4](https://arxiv.org/html/2506.14245v2#S4.E4) under the *Logic Prior* assumption) and $\sigma>0$, we have:

$$ $\displaystyle(1-p_{c})(\alpha-\beta)/\sigma$ $\displaystyle>0,$ $\displaystyle-p_{c}(\alpha-\beta)/\sigma$ $\displaystyle<0,$ $$

proving inequalities equation [5](https://arxiv.org/html/2506.14245v2#S4.E5).

The GRPO policy gradient update in equation [3](https://arxiv.org/html/2506.14245v2#S4.E3),
$\nabla_{\theta}J(\theta)\approx\frac{1}{G}\sum_{i=1}^{G}\hat{A}(y_{i})\nabla_{\theta}\log\pi_{\theta}(y_{i}\mid q)$,
on average increases the likelihood of responses with $\hat{A}(y_{i})>0$ (correct CoTs) and decreases it for $\hat{A}(y_{i})<0$ (incorrect CoTs).
Thus, $p_{c}$ increases monotonically.

###### Proof 1

##### Discussions on ( μ , σ 2 \mu,\sigma^{2} ) in Theorem 1

From equation [7](https://arxiv.org/html/2506.14245v2#A1.E7), we know that the group reward mean is given by $\mu=p_{c}\alpha+(1-p_{c})\beta$. Furthermore, we can derive the exact formula for the variance $\sigma^{2}$ in equation [9](https://arxiv.org/html/2506.14245v2#A1.E9) and analyze their impacts together with $p_{c}$, $\alpha$, and $\beta$ on policy iterations.

The sample variance $\sigma_{\mathbf{Y}}^{2}$ converges to the true variance $\sigma^{2}$:

$$ $\sigma_{\mathbf{Y}}^{2}=\frac{1}{G}\sum_{j=1}^{G}(R(y_{j})-\mu_{\mathbf{Y}})^{2}\xrightarrow{G\to\infty}\text{Var}(R(y_{j}))\equiv\sigma^{2},$ $$

where $\text{Var}(R(y_{j}))$ can be computed using the law of total variance:

$$ $\text{Var}(R(y_{j}))=\underbrace{\text{Var}(\mathbb{E}[R(y_{j})\mid\mathcal{I}_{\text{CoT}}(c_{j})])}_{\text{Variance of conditional expectation}}+\underbrace{\mathbb{E}[\text{Var}(R(y_{j})\mid\mathcal{I}_{\text{CoT}}(c_{j}))]}_{\text{Expectation of conditional variance}}.$ $$

First term:

$$ $\mathbb{E}[R(y_{j})\mid\mathcal{I}_{\text{CoT}}(c_{j})]=\begin{cases}\alpha&\text{if }\mathcal{I}_{\text{CoT}}(c_{j})=1\\ \beta&\text{if }\mathcal{I}_{\text{CoT}}(c_{j})=0\end{cases}.$ $$

The random variable $\mathbb{E}[R(y_{j})\mid\mathcal{I}_{\text{CoT}}(c_{j})]$ has variance:

$$ $\text{Var}(\mathbb{E}[R(y_{j})\mid\mathcal{I}_{\text{CoT}}(c_{j})])=(\alpha-\beta)^{2}p_{c}(1-p_{c}).$ $$

Second term:

$$ $\text{Var}(R(y_{j})\mid\mathcal{I}_{\text{CoT}}(c_{j}))=\begin{cases}\alpha(1-\alpha)&\text{if }\mathcal{I}_{\text{CoT}}(c_{j})=1\\ \beta(1-\beta)&\text{if }\mathcal{I}_{\text{CoT}}(c_{j})=0\end{cases},$ $$

so its expectation is:

$$ $\mathbb{E}[\text{Var}(R(y_{j})\mid\mathcal{I}_{\text{CoT}}(c_{j}))]=p_{c}\alpha(1-\alpha)+(1-p_{c})\beta(1-\beta).$ $$

Thus:

$$ $\displaystyle\sigma^{2}=(\alpha-\beta)^{2}p_{c}(1-p_{c})+p_{c}\alpha(1-\alpha)+(1-p_{c})\beta(1-\beta).$ (14) $$

Substituting $\mu$ and $\sigma$ into equation [12](https://arxiv.org/html/2506.14245v2#A1.E12) and equation [13](https://arxiv.org/html/2506.14245v2#A1.E13), we have

$$ $\mathbb{E}[\hat{A}(y_{i})\mid\text{correct CoT}]\rightarrow\frac{(1-p_{c})(\alpha-\beta)}{\sqrt{(\alpha-\beta)^{2}p_{c}(1-p_{c})+p_{c}\alpha(1-\alpha)+(1-p_{c})\beta(1-\beta)}},$ $$

$$ $\mathbb{E}[\hat{A}(y_{i})\mid\text{incorrect CoT}]\rightarrow\frac{-p_{c}(\alpha-\beta)}{\sqrt{(\alpha-\beta)^{2}p_{c}(1-p_{c})+p_{c}\alpha(1-\alpha)+(1-p_{c})\beta(1-\beta)}}.$ $$

An ideal pre-training on a high-capacity model could help to ensure that $\alpha\rightarrow 1$ and $\beta\rightarrow 0$ at the beginning of RLVR. In this condition, we have the following advantage estimates:

$$ $\mathbb{E}[\hat{A}(y_{i})\mid\text{correct CoT}]\rightarrow\sqrt{\frac{1-p_{c}}{p_{c}}},\quad\mathbb{E}[\hat{A}(y_{i})\mid\text{incorrect CoT}]\rightarrow-\sqrt{\frac{p_{c}}{1-p_{c}}}.$ $$

In this ideal scenario, the role of human would be to prepare a comprehensive and diverse set of questions and answers, leveraging RLVR to automatically incentivize the model’s reasoning capabilities. However, in practice—the “unideal case”—it is often necessary to first fine-tune the base LLM to align its output with a proper reasoning distribution before applying RLVR.

##### Discussions on Key Observations in RLVR

Grounded in our theoretical analysis, we can now provide our unique explanations for several previously elusive yet important observations reported in DeepSeek-R1 (Guo et al., 2025).

Our Explanation of the Observation “*DeepSeek-R1-Zero achieved remarkable Pass@K performance on AIME 2024 but encountered challenges such as poor readability and language mixing.*”:
Even DeepSeek-V3 (Liu et al., 2024) cannot guarantee ideal conditions where $\alpha\rightarrow 1,\beta\rightarrow 0$. As a result, cold-start data is required to rectify prior logic biases, motivating the R1 approach.

Our Explanation of the Observation “*The R1-Zero approach did not work well for the 32B dense model, yet distillation can be very effective.*”:
Key factors such as $(p_{c},\alpha,\beta)$ for the 32B base model are in an even worse state, causing pure RLVR to converge to suboptimal local solutions. Based on our analysis, the key to effective reasoning lies in learning correct CoTs. Therefore, the distillation approach can efficiently teach an LLM how to reason properly.

### A.6 Additional Details in Reproducing DAPO Training

Our reproduction was conducted on 32 AMD MI300X GPUs using the VERL framework (Sheng et al., 2025), and ran for over two weeks.
Our run did not fully reproduce the $Pass@1$ accuracy above 50% as reported by Yu et al. (2025), while we reached a comparable performance of around 44% $Pass@1$, in line with a third-party reproduction (Chen et al., 2025a).
We use the same verifier introduced in Section [3](https://arxiv.org/html/2506.14245v2#S3) to assess the correctness of both training and evaluation rollouts.

Figure: (a) We show gradually optimized performance on training prompts throughout RLVR and corresponding generalization behaviors on testing prompts. The top two subfigures use violin plots to visualize the evolution of $P(CA)^{(q)}$ (the fraction of correct answers for prompt $q$) and $P(CC|CA)^{(q)}$ (the fraction of correct CoTs within the correct answers for prompt $q$) over the course of DAPO training. Subfigure (a) shows results for “easy” training questions, while (b) presents the corresponding distributions for “hard” questions ($P(CA)^{(q)}$ ¡ 1 for DAPO). We divide training questions into easy and hard by sampling 64 rollouts for each of the $17k$ training questions using Qwen2.5-32B, and labeling those with at least one correct answer as easy. Subfigure (c) presents the generalization performance on AIME 2024 across different training steps.
Refer to caption: figures/aime24-15.png

### A.7 Discussions

##### Call for Live, Challenging Benchmarks

Static benchmarks developed prior to the release of modern base models are increasingly susceptible to contamination risks, potentially undermining the reliability of observed improvements. In response, we emphasize the need for *live benchmarks* that evolve over time, as suggested in recent studies (Jain et al., 2025; White et al., 2025). Additionally, we agree with the viewpoint of Yao (2025) that future research advancements may rely more on designing new evaluations, benchmarks, and environments.

##### Call for Lightweight yet Powerful CoT Verifiers

While DeepSeek-R1-0528-Qwen3-8B serves as a useful CoT verifier, it is not infallible. Conflicting verification results across multiple queries reveal the challenges of false-positive and false-negative verifications. To tackle this, we combine multiple verification strategies, including different voting rules, to improve robustness. Looking forward, there is a pressing need for light yet reliable CoT verifiers that can serve as standardized evaluators beyond the coarse-grained Pass@K metric. This direction also relates to previous studies on process reward modeling (Lightman et al., 2024; Uesato et al., 2022; Wang et al., 2024).

##### Scaling RLVR or Scaling Pre-Training

While the scaling of pre-training has led to transformative progress in LLMs (Kaplan et al., 2020; Liu et al., 2024), enabling the transition to the era of artificial general intelligence, we argue that scaling RLVR could be equally pivotal, given the empirical evidences and theoretical foundation that all demonstrate its real incentivization beyond base LLMs. As modern LLMs approach the limits of language token exposure, learning from experience (Silver & Sutton, 2025) may represent the next leap. Recent efforts by leading research teams suggest a growing emphasis on this direction (Guo et al., 2025; DeepSeek, 2025; Gemini, 2024; Grok, 2025; OpenAI, 2025; Qwen, 2025; Gemini, 2025; Anthropic, 2025; Mistral.AI, 2025). For the broad open research community, understanding the foundations and limitations of current RLVR algorithms is crucial to push this direction further.

##### New RLVR Algorithms and Beyond

With our insight that RLVR implicitly incentivizes correct reasoning in base LLMs, we anticipate the development of new algorithmic paradigms. These may include optimization formulations or objective functions, such as policy-gradient approaches (Sutton et al., 1999; Schulman et al., 2017), new likelihood-based optimization objectives (Chen et al., 2025a; Zhu et al., 2025), and preference optimization frameworks (Rafailov et al., 2023; Su et al., 2025). The key principle is that the new algorithms should be designed to more directly incentivize correct reasoning paths, alleviating inherent logical biases in base LLMs.

### A.8 Case Studies

We perform manual verification on difficult problems from multiple benchmarks, focusing on cases where the base model occasionally produces a correct final answer despite flawed reasoning in its CoT.
For each case study, we present the original question, the full response generated by Qwen2.5-32B, and a detailed breakdown of key errors identified by DeepSeek-R1-0528-Qwen3-8B, explicitly linking each error to the corresponding flawed reasoning segments in the CoT.
For layout and color schemes, we follow the visualization strategies used in Yue et al. (2025).

#### A.8.1 Case Studies on AIME 2024

#### A.8.2 Case Studies on AIME 2025

#### A.8.3 Case Studies on AMC23

#### A.8.4 Case Studies on MATH-500

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Could RLVR’s implicit reasoning incentive be combined with explicit step-wise rewards to further boost reasoning reliability?
- It'd be great to see more details on training cost/efficiency and scaling behavior
soundness: 3
strengths: 1. The paper provides a well-structured theoretical analysis explaining how GRPO gradients implicitly favor logically consistent reasoning traces, offering insights on why RLVR often improves reasoning without explicit step-wise rewards.

2. Experiments on multiple reasoning domains (math and code) and models (Qwen, Nemotron) show consistent gains under both Pass@K and the proposed CoT-Pass@K metrics. The analyses of training dynamics and reasoning correctness lend strong empirical support to the theoretical claims.

3. The new CoT-Pass@K metric could be a useful tool for the community, bridging the gap between evaluating correctness of answers and correctness of reasoning.
summary: This paper investigates whether Reinforcement Learning with Verifiable Rewards (RLVR) genuinely improves the reasoning capability of large language models, rather than merely increasing sampling efficiency. It provides both theoretical analysis and empirical evaluation to show that RLVR implicitly incentivizes correct reasoning chains. The authors introduce a new evaluation metric, CoT-Pass@K, which verifies both intermediate reasoning correctness and final answers, and demonstrate that GRPO-based RLVR improves reasoning quality across multiple math and code benchmarks. Results suggest that RLVR not only reinforces correctness but also enhances reasoning consistency and generalization.
weaknesses: 1. The reliability of CoT-Pass@K hinges on judgments from a single verifier (DeepSeek-R1-Qwen3-8B). While multi-verification is discussed, evaluating under different verifiers would improve confidence in these results. How robust are CoT-Pass@K results when using different verifiers? -- we can fix the verifier used in training and just use different verifiers for evaluation.

2. Potentially questionable assumptions. The theoretical analysis assumes that reasoning traces with correct answers have systematically higher probabilities, which may not always hold in domains like commonsense or multi-hop reasoning. The practical validity of this assumption could be discussed more thoroughly. Do we have any empirical examples?

3. While the paper shows improved correctness, it does not analyze whether RLVR reduces reasoning diversity or leads to overconfident, deterministic reasoning traces. Add some quantitative analysis of reasoning diversity would further strengthen the paper.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Why opt for a smaller model to judge the reasoning paths? Doesn't the accuracy of the judgment become constrained by the capacity of the referee model? Should a more powerful referee model be employed instead?

2. For tasks where a direct reward signal is unavailable in standard RL training, the common approach is often to use a rubric combined with a powerful evaluation model to assign rewards. Given this, could your CoT-Pass@K metric be integrated into the RLVR training process to yield beneficial results?

3. Based on the experimental results, we observe that the CoT-Pass@K score for 'no-thinking' models is generally suppressed compared to their Pass@K score. However, this method does not appear to provide a significantly better evaluation metric for RLVR models (as the difference between Pass@K and CoT-Pass@K for RLVR seems minor). Therefore, what are the practical applications or use cases for your proposed method?
soundness: 2
strengths: 1. The paper reads smoothly and presents its ideas in a clear, easy-to-follow style.
2. The experimental design is comprehensive and well thought-out.
3. RLVR provides only marginal gains at Pass@1024 or Pass@2048, but demonstrably boosts Pass@1—a widely recognized pattern. This supports the prevailing view that RLVR achieves its effect by learning correct reasoning, a claim for which your experiments offer some evidence.
summary: * **Starting Point**: The paper aims to argue that **RLVR truly enhances the model's capabilities**
* **New Metric**: The paper proposes the **CoT-Pass@K** metric, which is better suited than the commonly used Pass@K to demonstrate the **model's capability improvement after RLVR**
* **Experimental Validation**: The experiments verify that for multiple math tasks, **CoT-Pass@K better highlights the performance gap** in model capabilities compared to Pass@K
weaknesses: Please refer to the "Questions" section for details.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How would the authors define the correctness of a CoT? That is, if a model makes mistakes initially but then corrects them in subsequent tokens through self-reflection, should such a CoT be considered incorrect?

2. It would be good to see the influence of SFT on both Pass@K and CoT-Pass@K. For example, DAPO-Qwen-32B and DeepSeek-R1-Distilled-Qwen-32B have similar avg@32 on AIME24. It would be interesting to see which model performs better when using CoT-Pass@K as a metric.
soundness: 2
strengths: - The exact role of RLVR in enhancing the reasoning capabilities of base models is an open and fundamental problem for the RL community. Thus, this manuscript begins with a very strong motivation and is, in my view, highly timely. 

- Although the proposed Theorem 1 is relatively simple and straightforward, this preliminary explanation is helpful for the community to understand the underlying reasons why RLVR works. 

- The authors base their conclusions on extensive experiments. I particularly appreciate that they not only experimented with the Qwen+Math setup but also conducted additional experiments on coding tasks.
summary: This manuscript investigates a fundamental open question in reinforcement learning with verifiable rewards (RLVR): does RLVR merely improve sampling efficiency, or does it endow the base model with fundamentally new reasoning capabilities? The authors first revisit the Pass@K metric, which the RL community has often considered a signal of exploration, and identify its potential pitfalls. Specifically, base LLMs are capable of producing incorrect Chains of Thought (CoTs) yet coincidentally arriving at the ground truth. To address this, they introduce a new metric, CoT-Pass@K, which simultaneously considers the correctness of both the reasoning trace and the final answer. By re-examining the model's reasoning performance with CoT-Pass@K, the authors observe that RLVR successfully extends the reasoning boundary of the base model in both code and math tasks. The manuscript also presents primary theorems on GRPO's advantage estimation to interpret these empirical observations.
weaknesses: - The correctness of a CoT is determined by a R1-distilled-Qwen3-8B model, which may not be a sufficiently strong verifier. Since reasoning models may frequently exhibit rethinking or backtracking behavior, I have concerns about whether an 8B model is reliable enough to verify complex reasoning traces that include self-correction, reflection, and backtracking. For example, what if a model initially thinks incorrectly but then corrects its answer on its own? Is the distilled 8B model strong enough to verify such a CoT? The authors acknowledge this limitation in line 468, but I still feel it is noteworthy. Stronger frontier close-source LLMs maybe a better choice, e.g., Gemini-2.5-pro, if the cost of API calls is affordable.

- The empirical observations from the post-SFT experiments (lines 429-431) are difficult for me to follow. It is counterintuitive to me that: 1) tuning on erroneous CoTs can also improve Pass@1 accuracy, ultimately leading to nearly the same performance, and 2) SFT on all CoTs results in worse accuracy than SFT on only incorrect CoTs. Does this observation indicate that the "incorrect" CoTs identified by Distilled-Qwen3-8B only contain erroneous steps at the beginning (which the model self-corrects later) rather than the entire reasoning trace being wrong? And thus SFT on these partially incorrect reasoning traces can also benefits pass@1 accuracy. If this is true, I have further concerns about the reliability of the CoT verifier and the definition of a "correct" CoT.

In summary, I am positive about this manuscript, and I find its findings inspiring and important. However, due to the two concerns mentioned above, I do not have enough confidence to give it a higher rating.


Decision: Accept (Poster)