Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
Authors: Yang Yue
Zhiqi Chen
Rui Lu
Andrew Zhao
Zhaokai Wang
Yang Yue
Shiji Song
Gao Huang
Venue: NeurIPS 2025 oral ()
Keywords: reinforcement learning with verifiable reward
LLM reasoning
Primary area: deep_learning
TL;DR: We systematically examine the current state of RLVR and surprisingly find that it does not elicit fundamentally new reasoning patterns—revealing a gap between the potential of RL and the actual impact of current RLVR methods.

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has recently demonstrated notable success in enhancing the reasoning performance of large language models (LLMs), particularly in mathematics and programming tasks. 
It is widely believed that, similar to how traditional RL helps agents to explore and learn new strategies, RLVR enables LLMs to continuously self-improve, thus acquiring novel reasoning abilities that exceed the capacity of the corresponding base models. 
In this study, we take a critical look at \textit{the current state of RLVR} by systematically probing the reasoning capability boundaries of RLVR-trained LLMs across diverse model families, RL algorithms, and math/coding/visual reasoning benchmarks, using pass@\textit{k} at large \textit{k} values as the evaluation metric.
While RLVR improves sampling efficiency towards the correct path, we surprisingly find that current training does \emph{not} elicit fundamentally new reasoning patterns.
We observe that while RLVR-trained models outperform their base models at smaller values of $k$ (\eg, $k$=1), base models achieve higher pass@$k$ score when $k$ is large.
Moreover, we observe that the reasoning capability boundary of LLMs often narrows as RLVR training progresses.
Further coverage and perplexity analysis shows that the reasoning paths generated by RLVR models are already included in the base models' sampling distribution, suggesting that their reasoning abilities originate from and are \textit{bounded} by the base model. 
From this perspective, treating the base model as an upper bound, our quantitative analysis shows that six popular RLVR algorithms perform similarly and remain far from optimal in fully leveraging the potential of the base model.
In contrast, we find that distillation can introduce new reasoning patterns from the teacher and genuinely expand the model’s reasoning capabilities.
Taken together, our findings suggest that current RLVR methods have not fully realized the potential of RL to elicit genuinely novel reasoning abilities in LLMs. This underscores the need for improved RL paradigms—such as continual scaling and multi-turn agent-environment interaction—to unlock this potential.

Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
Yang Yue l+t
Zhiqi Chen
Rui Lu
Andrew Zhao' Zhaokai Wang? 2 Gao Huang
Yang Yue
Shiji Song'
LeapLaab, Tsinghua University
2 Shanghai Jiao Tong University
[le-y22, zq-chen23) maii://iiii..ssinadddu.........
(gaohuang/ @ Esinghua.edu.cn
http:://imitt-of-rvv..gi.g
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has recently demon- strated notable success in enhancing the reasoning performance of large language models (LLMs), particularly in mathematics and programming tasks. is widely believed that, similar to how traditional RL helps agents explore and learn new strategies, RLVR enables LLMs to continuously self-improve, thus acquiring novel reasoning abilities that exceed the capacity of the corresponding base models. In this study, we take a critical look at the current state of RLVR by systematically probing the reasoning capability boundaries of RLVR-trained LLMs across various model families, RL algorithms, and math/coding/ / reasoning benchmarks, using pass @K large values the evaluation metric. While RLVR improves sampling efficiency towards correct paths, we surprisingly find that current train- ing does not elicit fundamentally new reasoning patterns. We observe that while RLVR-traine models outperform their base models at smaller values of l (e.g., k=1), base models achieve higher pass@ k score when l: is large. Moreover, we observe that the reasoning capability boundary of LLMs often narrows as RLVR training progresses. Further coverage and perplexity analysis shows that the rea- soning paths generated by RLVR models are already included in the base models' sampling distribution, suggesting that their reasoning abilities originate from and are bounded by the base model. From this perspective, treating the base model an upper bound, our quantitative analysis shows that six popular RLVR algorithms perform similarly and remain far from optimal in fully leveraging the potential of the base model. In contrast, we find that distillation can introduce new reasoning patterns from the teacher and genuinely expand the model's reasoning capabilities. Taken together, our findings suggest that current RLVR methods have not fully realized the potential of RI. to elicit genuinely novel reasoning abilities in L.MMs. This underscores the need for improved RL paradigms-Soa as continual scaling and multi-turn agent-environment interaction-to unlock this potential.
Introduction
The development of reasoning-centric large language models (LLMs), such as OpenAI-ol [1]. DeepSeek-R1 [2], and Kimi-1.5 [3], has significantly advanced the frontier of LLM capabilities, particularly in solving complex logical tasks involving mathematics and programming. In contrast to traditional instruction-tuned approaches that rely on human-curated annotations [4, 5], the key driver
Equal Contribution. Project Lead. Corresponding Author. The first author Yang Yuc (frF)) and the sixth author Yang Yue ( H) share the same English name different Chinese names.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Base Model
RLVR Model
Omni-MATHH-Tran
More Efficient Sumpling
0.7
Problem A
Coverage (pass(ic))
0.6
0.5
0.4
0.3
Qwen2.5 70
Roduced Scope of Reasoning Capacity
PRR
Problem
0.2
GRPO stsp-150
0.1
2
8 16 32 64 128256
Number of Samples k
Figure 1: (Left) The effect of current RLVR on LLM's reasoning ability. Search trees are generated by repeated sampling from the hase and RIVR-traiied models for given problem. Grey indicates paths that are unlikely he sampled by the model, while black indicates paths that are likely he sampled. Green indicates correct paths, which has positive rewards. Our key finding is that all reasoning paths in the RLVR model are already present in the base model. For certain problems like Problem A, RLVR training biases the distribution toward rewarded paths, improving sampling efficiency. However, this comes at the cost of reduced scope of reasoning capacity: For other problems like Problem B, the base model contains the correct path, whereas that of the RLVR model does not. (Right) RLVR training progresses, the average performance (i.e., pass@ 1) improves, but the coverage of solvable problems (i.e., pass@ (26) decreases, indicating a reduction in LLM's reasoning boundary.
behind this leap forward is large-scale Reinforcement Learning with Verifiable Rewards (RLVR) [6. 2]. RLVR starts with a pretrained base model or one fine-tuned on long chains of thought (CoT) data, optimizing it via reinforcement learning based on simple, automatically computable rewards. These rewards are determined by whether the model's 's output matches ground-truth solution in mathematics or passes unit tests in code, thus enabling scalability without human labeling. This framework has gained significant attention due to its simplicity and practical effectiveness. In traditional RL settings such as game playing (e.g.. Atari, Go), agents often autonomously discover new strategies and surpass even human-level performance through self-improvement [7, 8). Inspired by this success, it is widely believed that RLVR similarly enables LLMS to autonomously develop novel reasoning patterns, including enumeration, self-reflection, and iterative refinement, surpassing the capabilities of their base models [2]. Consequently, RLVR has been considered a promising path toward continuously self-evolving LLMs, potentially bringing us closer to more powerful intelligence [2].
However, despite its empirical success, the underlying effectiveness of current RLVR remains underexamined. This raises a fundamental question: Does current RLVR genuinely enable LLMs to acquire novel reasoning abilities-simila to how traditional RL discovers exploration-or or does it simply utilize reasoning patterns already in the base model?
strategies through
To rigorously answer this question, we must first assess the reasoning capability boundaries of both base and R models. Traditional evaluation metrics rely on average score from greedy decoding nucleus sampling [9]. which reflects average-case behavior. However, these metrics risk underestimating the true potential of a model, especially it fails on difficult problems after limited attempts, despite being capable of solving them with more sampling. To overcome this limitation, we adopt the pass @+ k metric [10], where a problem is considered solved if any of the k sampled outputs is correct. By allowing multiple attempts, pass @ reveals whether model has the potential to solve a problem. The average pass @k across a dataset thus reflects the proportion of problems a model can potentially solve within & trials, offering more robust view of its reasoning boundary. This provides a rigorous test on whether the RLVR training yields fundamentally transcending capacity, enabling the model solve problems that the base model cannot.
Using the pass @K metric, we conduct extensive experiments across various benchmarks, covering multiple LLM families, model sizes, and RLVR algorithms to compare base models with their counterparts. We uncover several surprising findings that offer a more comprchensive assessment of the effectivenees of current RLVR training and reveal the gap between existing RLVR methods and the ideal goals of RL-discovering genuinely new reasoning strategies:
Current RLVR models exhibit narrower reasoning coverage than their base models. In pass@k & curves, although RLVR models outperform their base models at small k. it is surprising that base models consistently surpass RLVR models across all benchmarks and LLM families as k increases. This suggests that current RLVR training does not expand, and even reduce the scope of reasoning over solvable problems. Manual inspection of model responses shows that, for most

---

problems, the base model can produce at least one correct COT, implying that it can already generate correct reasoning paths for problems that were previously considered only solvable for RLVR models.
Reasoning paths generated by current RIVR model already exist in its base model. To further investigate this phenomenon, we analyze the accuracy distribution. The results show that although RLVR improves average performance (i.e., pass@) 1) by sampling more efficiently on problems already solvable by the base model, does not enable the model to solve new problems. Further perplexity analysis reveals that the reasoning paths produced by RLVR models already exist within the output distribution of the base model. These findings indicate that RLVR does not introduce fundamentally new reasoning capabilities and that the reasoning capacity of current RLVR models remains bounded by that of its base model. This effect of RLVR is illustrated in Figure (left).
Current RLVR algorithms perform similarly and remain far from optimal. Treating the base model as upper bound, we define the sampling efficiency gap (ASE). shown in Figure 8 (top), as the difference between an RL model's pass @ and the base model's pass @ (with & = 256 as a proxy for upper-bound performance). This metric quantifies how closely an RL algorithm approaches the optimal bound. Across all algorithms (e.g., PPO, GRPO, Reinforce++), ASE shows only minor variation yet remains consistently large, suggesting that current RLVR methods, while improving sampling efficiency, are still far from optimal.
RLVR and distillation are fundamentally different. While RLVR improves reasoning scores by more efficiently sampling high-reward outputs, does not elicit new reasoning capabilities and remains constrained within the base model's capacity. In contrast, distillation can transfer new reasoning patterns from a stronger teacher to the student. As a result, distilled models often demonstrate expanded reasoning scope beyond that of the base model.
In conclusion, our findings show that current RLVR methods, while improving sampling efficiency, do not elicit novel reasoning beyond the base model's capabilities. This highlights gap between existing RLVR methods and the goals reinforcement learning, underscoring the need for improved RL paradigms such continual scaling, better exploration, and multi-tur turn agent interaction.
2
Preliminaries
In this section, we first outline the fundamentals of RLVR, then introduce the pass@k @k metric to evaluate reasoning boundaries, and explain why it is preferred over alternatives like best-of-N.
2.1 Reinforcement Learning with Verifiable Rewards
Verifiable Rewards. Let IT be an LLM with parameters 0 that generates a token sequence y (yi...., )r) conditioned on a natural-language prompt x. A deterministic verifier V returns a binary reward: r V(xx,) E {0, 1}, where
if and only if the model's final answer is exactly correct. A format reward may also be added to encourage the model to explicitly separate the reasoning process from the final answer. The goal of RL is to learn a policy maximize the expected reward:
[0) = Eg-D [Ey)ma(+[e][]]]] where D is the distribution of prompts.
RLVR Algorithms. Proximal Policy Optimization (PPO) [11) proposed using the following clipped
surrogate to maximize the objective:
LCUP = E (O),,,, clip(r, - c),)).
(1) and A, is the advantage estimated by a value net-
where re(0)= = Te ( y<e) Teue work Vg. KI. divergence term is optionally applied, to constrain the model from deviating too far from the original policy. More algorithms are introduced in Section D.5.
Policy Gradient. PPO and its variants belong the policy gradient class of RL [12, 13]. These methods learn exclusively from on-policy samples, i.e., samples generated by the current LLM. In the context of verifiable rewards, the training objective generally maximizes the log-likelihood of samples with correct answers and minimizes the likelihood of those with incorrect answers.
Zero RL Training applies RL directly to the base model without any supervised fine-tuning (SFT) [2]. To clearly study the effect of RLVR, we follow this zero-RL setting for all math tasks using pretrained models as start model. However, for coding and visual reasoning tasks, open-source work typically uses instruction-tunde effectiveness of using a pure zero-RL setting. Following this convention, we compare the finetuned model with its RLVR-trainee counterpart to focus solely on the effect of RLVR.
models as starting points, primarily due to the training instability and limited
3

---

Qwen-2.5-7B
Qwen-2.5-14B
Qwen-2.5-32B
LLaMA-3.1-8B
Coverage (passii))
0.8
Basc
0.8
0.8
0.8
AIME24
8.6
0.6
0.6
0.5
04
82
0.2
0.2
0.2
0
4
16
61
256
1024
16
61
236
1024
0
1
4
16
64
256
1024
16
64
256
1024
Number of Samples
MATH500
0.6
0.6
0.6
0.5
0.4
0.4
0.4
0.4
0
128
0.8
0.8
0.8
0.8
Minerva
0.6
0.2
0.2
0.2
0.2
16
2
32
64
126
2
16
32
628
0.8
Olympiad
0.6
0.6
0.6
0.5
0.4
0.4
0.4
0
16 32 64 128
16 52
64
128
0
15
32 54
16
64
256
1024
Figure 2: Pass @ & curves of base models and their ELVR-trraine' counterparts across multiple mathematical benchmarks. When & small, RL-trained models outperform their base versions. However, increases to the tens or hundreds, base models consistently catch up and surpass RL-trained models. More results on GSM8K and AMC23 can be found at Figure 10.
2.2 Metrics for LLM Reasoning Capacity Boundary
Pass@k Metrics. Accurately measuring reasoning ability boundary of base and RL models is challenging, as methods like greedy decoding the average of nucleus samplings [9] only reflect average-case performance. To accurately measure the reasoning ability boundary, we extend the commonly used pass@k metric from code generation [14] all tasks with verifiable rewards. Given a problem, we sample outputs from the model. The pass & value for this question is at least one of the A samples passes verification; otherwise, it is O. The average pass @ k value over the dataset reflects the proportion of problems the dataset that the model can solve within trials, providing a rigorous evaluation of the reasoning capacity coverage of LLMs. We adopt unbiased, low-variance estimator for computing to calculate pass @ k. as detailed Section A.2.
Comparison with Best-of-N and Majority Voting. Best-of-N [15] and majority voting are practical methods for selecting correct answers, but they may overlook model's full reasoning potential. In contrast, we use pass @ & not to assess practical ufility but to investigate the boundaries of reasoning capacity. a model produces correct solution in any of the samples, we treat the problem as within its potential scope. Thus, if RL enhances reasoning, the RL-trained model should succeed in more such problems than the base model. Methods like Best-of-N or majority voting may miss these successes if the correct answer is not selected by the verifier or voting.
Random Guessing Issue. For coding tasks, where compiler and predefined unit test cases are used as verifiers, the pass@1 & value can accurately reflect whether the model can solve the problem. In mathematics, the issue "guessing" can become pronounced as increases, where a model may generate an incorrect CoT but still accidentally arrive at the correct answer. To address this, we manually check the correctness of CoT for subset of model outputs as detailed in Section 3.1. By combining reuslts math with manually checking and coding, rigorously evaluate the scope

---

of LLM's reasoning capacity. Another caveat is that, with an astronomically large k. even uniform sampling over the token dictionary would stumble upon the correct reasoning path-though this is infeasible within today's s time and compute budgets. Crucially, we find that the base model already produces correct outputs at realistic values (k- 128 or 1024), well within practical resource limits.
Table I: Experimental setup for assessing RLVR's effect on the reasoning boundaries of LLMs.
Task
Start Model
RL Frumework
Algorithm(ss)
Benchmark(s)
(LaMA-3.1188
SimpleRLZoo
GSMBK, MATH500
Mathematies
Qwen2.5-7B/14B/22BB
Oat-Zero
GRPO
Minerva, Olympiad
5-Mtth
DAPO
AIME24, AMC23
Qwen2.5-7BB --n--tuutttttt
Code-RI
LiveCodeBench
Code Generation
DeepSeck 1 Dis--istil-QQO Distill--wen-1-4 Qwin---wenn
DeepCoder
GRPO
HumanEival+
Visual Reasoning
Qwen2.5-V1.2 V-LBBB
EasyR1
GRPO
MathVista
MathVision
Qwen2.5-7B Base
PPO, GRPO
Ommi-Math-Rule
Deep Analysis
Qwen2.5-72 10-2.-57--nnntruu
VeRL
Reinforce++
MATH500
DeepSeek-HR R---R-iiDDiisiI- Distill-Qwen-7H Qwen-Te
RLOO, ReMax, DAPO
3 RLVR's Effect on Reasoning Capacity Boundary
With the evaluation metrics for reasoning boundaries established, we now conduct comprehensive evaluation of the base and RLVR models through extensive experiments. Our analysis is organized by task category, covering three representative domains: mathematics, code generation, and visual reasoning. The overall experimental setup is summarized in Table 1.
For sampling procedures for both base and RLVR models, we a temperature of 0.6 and a top-p value of 0.95, allowing a maximum generation of 16,384 tokens. We also show the effect of different temperature settings in Figure 17. For evaluation of the base model, a common practice is to include few-shot examples in the prompt to guide the output [5, 16, 17]. However, to ensure a fair and unbiased comparison, we deliberately avoid using few-shot prompts for base models, eliminating any potential confounding effects on reasoning that might be introduced by in-context examples. For evaluating both the base and RLVR models, we use the same zero-shot prompt as in RLVR training, or the default prompt provided by the benchmark, ensuring a consistent setup across both models. Interestingly, although base models often produce unformatted or non- sensical responses without few-shot guidance, we observe that with sufficient sampling, they are still capable of generating correctly formatted outputs and successfully solving complex problems. Prompt templates for training and evaluation are provided in Section E.
Evaluation Protocol.
3.1 RLVR for Mathematical Reasoning
Models and Benchmarks. In math problems, models are required to generate a reasoning process (i.e.. CoT) along with the final answer. To ensure the robustness of conclusions, we experiment with multiple LLM families, primarily Qwen2.5 (7B/14B/32B base variants) [16] and additionally LLaMA-3.1 1-BB [5]. We adopt RLVR models released by SimpleRL.Zoo [18], which train zero-RL models using GRPO on GSM8K and the MATH training set, with correctness reward only, excluding any format-based reward. We compare the pass @ ls curves of base and zero-RL models on benchmarks
of varying difficulty: GSM8K [15]. MATH500 [19], Minerva [20]. Olympiad [21], AIME24, and AMC23. Additionally, we include the RLVR model Oat-Zero-7h and DAPO-32B [22, 23]. These two models are characterized by strong performance on the challenging AIME24 benchmark.
The Effeet of RLVR: Increased Likelihood of Correct Samples, Decreased Coverage of Solvable Problems. As shown in Figure 2, we consistently observe a contrasting trend between small and large k values. When small (e.g., models outperform their base counterparts. This aligns with the common observation that RL improves performance, suggesting that RLVR makes models significantly more likely to sample correct responses. However, as k increases, with steeper curves, base models consistently catch up to and eventually surpass RL-trained models across all benchmarks, indicating their broader coverage of solvable problems. For example, on the Minerva benchmark with a 32B-sized model, the base model outperforms the RI--trained model by approximately 9% at k = 128, implying that it can solve around 9% more problems in the validation set.
1, equivalent to average-case accuracy). RL-trained
5

---

We further examine RL models trained with Oat and DAPO. As shown in Figure 11, although the RL model initially demonstrates a strong performance, nearly 30% higher than the base model, is eventually surpassed by the base model. Based on these results, we conclude that RLVR increases the likelihood of sampling correct responses at low k, but narrows the model's s overall coverage. We further analyze the root cause of this phenomenon in Section 4.1.
CoT Case Analysis. We present the correct CoTs sampled from the base model in Figure 20 and Figure 21, manually selected from 2048 samplings for the hardest questions in AIME24. The responses from the base model tend to be long CoTs and exhibit reflective behavior, highlighting the strong reasoning ability inherent in the base model.
Validity of Chain-of-Thouun -hoouugh.. For mathematical problems, the common evaluation is based solely on the correctness of the final answer, with the risk of "hacking". To accurately reflect the reasoning ability boundary using pass @ k it is important to assess how many solved problems result from sampling genuinely correct CoT's, rather than from lucky guesses. Following [10], we manually inspect all CoTs that led correct answers to the most challenging solvable problems in the GSM8K dataset- - those with an average accuracy below 5% but above 0%. The base model answered 25 such questions, with 24 containing at least one correct COT. Similarly, the RL-trained model answered 25 questions, 23 of which included at least one correct COT. We also manually check the CoT' for problems in the challenging AIME24 benchmark with an average accuracy below 5%. Details can be found in Section D.2. The base model answered 7 such questions, with 5 out of 6 containing at least one correct CoT (excluding one ambiguous case of correctness due to skipped reasoning steps). Similarly, the RL-trained model answered 6 questions, 4 of which included at least one correct CoT. These results suggest that the base model can sample valid reasoning paths to solve the problems.
3.2 RLVR for Code Generation
Models and Benchmarks. We adopt the open-sourced RLVR- trained model, CodeR -aaeeRR-ZZr--- [24], which trains zero- RL models on 12K LeetCode and TACO samples over 832 steps, based on Qwen2.5-7B Dwwe2.5-7BBLnnsruutt-l [25]. For evaluation, mod- els are assessed on LiveCodeBench v5, comprising 279 problems that span from August 2024 to January 2025 [26], as well as Hu-
LiveCodeBeech (2024.8-2055,
0.8
Coverage (pass
0.75
0.7
0.65
manEval+ and MBPP+ [27]. We also evaluate the most powerful open-source RLV coding LLM, DeepCoder- 14B [28], built Here both models take 32k re-
0.6
0.55
on DeepSeek-R1 RI-Distill---enn-
0.5
DoppeabbsiS-ttheeee. (RL.)
sponse length. Due to their high computational cost, we evaluate them only on LiveCodeBench as a representative benchmark.
16
Number of Samples (k)
Figure 3: RLVR for Coding.
The Effect of RLVR. Since passing all unit tests is nearly impossible to achieve by guesswork, pass @+ k provides reliable measure of a model's reasoning boundary. As shown in Figure 3, Figure 12, and Figure 4 (left), the effects of RLVR on three code generation benchmarks exhibit trends that are highly consistent with those observed in mathematical benchmarks.
3.3 RLVR for Visual Reasoning
Models and Benchmarks. In visual reasoning, models must jointly interpret visual and textual inputs solve complex reasoning problems. This has gained significant attention in the multimodal community since the rise of L.L. reasoning [. For our experiments, we select math within visual contexts as a representative task. We use the EasyRI framework [31] train Qwen2.5-VL. 7B [32] on Geometry3K [33], and evaluate its visual reasoning capabilities on filtered Math Vista- TestMini [34] and Math MesiMini [35], where multiple-choice questions are removed.
The Effect of RLVR. As shown in Figure 4 (right), the effects of RLVR on visual reasoning are highly consistent with those observed in math and coding benchmarks. This suggests that the original model has broader coverage of solvable questions even in multimodal tasks.
Validity of Chain-of-TT Tfooghhht........ Similarly, we manually inspect subset of the most challenging problems, teet with an average accuracy below 5%. We find that for both the original and RI. models, 7 out of 8 problems have at least one correct COT. These results support the validity of CoTs.

---

Eval+
MBPP+
MittVissa
Math Vishon
0.9
0.9
Coverage (pars
Coverage (pass:8k))
0.8
01.9
0.8
19
0.7
0.6
0.8
0.6
0.7
0.75
0.5
0.2
2
16 32 64 128
8
16
52
64
128
X
16
52
64
128
2
16
32
64
128
Number Samples
Number Samples
Samples
k
Number Samples
Figure 4: Pass@k curves of base and RLVR models. (Left) Code Generation. (Right) Visual Reasoning.
Deep Analysis
In this section, we conduct a deeper analysis the effects of current RLVR training. We also highlight the distinct characteristics of distillation in comparison to RLVR. In addition, we design controlled experiments to examine the impact different RL algorithms and design choices.
4.1 Reasoning Paths Already Present in Base Models
Accuracy Distribution Analysis. Experiments in Section 3 reveal surprising trend: the base model covers wider range of solvable problems than the RLVR-trained model. To bet- ter understand this, we analyze how the accuracy distribution changes before and after RLVR training. As shown in Figure 5, RLVR increases the frequency of high accuracies near 1.0 and reduces the frequency of low accuracies (e.g.. 0. 0.2). How- a deviation from this trend is the increased frequency at accuracy 0 - indicating that RLVR leads to more unsolvable problems. This also explains the improvement of RLVR in average scores, driven not by solving new problems but by improving sampling efficiency on problems already solvable by the base model. Additional accuracy histograms are provided in Figure 14.
Base
100
RL
Percentage (%)
Frequency
80
28
40
20
10
66
(t:84)
(048.1)
(a:de)
(0783
Accunacy Interva/
Figure 5: Qwen2.5-7B Accuracy His-
togram Minerva.
Problee Coverage Analysis. To further investigate, we compare the set of solvable questions for both the base model and its corresponding RL- trained version on AIME24 and MATH500. We find that there many cases where the base model solves a problem but the RLVR model fails, and very few where RLVR succeeds while the base model does not, as shown in Table 2. Details can be found at Sec- tion D.7. As shown in Table 5, the set of problems solved by the RL-trained model is nearly subset of those solvable by the base model. similar trend is observed in coding tasks as shown in Table 6.
Table We evaluate on AIME24 (k = 1024) and MATH500 (k = 128). The table reports the solvable/unsolvable fraction of problems falling into four categories.
Base
SimpleRLZoo
AIME24
MATH500
/
63.3%
92.4%
13.3%
3.6%
X
/
0.0%
1.0%
23.3%
3.0%
This raises the natural question: Do all reasoning paths generated by RL-trained models already exist within the output distribution of their base models?
Perplexity Analysis. To answer this question, we utilize the metric perplexity. Given model m, a problem æ, and a response Y = (y1...., .yr) (can be generated by the same model, another model. humans), the perplexity is defined as the exponentiated average negative log-likelihood of a sequence:
x) = exp (- (-1 T
PPL.. Y
Σ log P(u. I. 22........................ ............))
which reflects the model's ability to predict the given response conditioned on the prompt x. Lower perplexity indicates that the model has higher likelihood of generating this response.
We randomly sample two problems from AIME24 and employ Diven2.577B-B Base and SimpleRL- Qwen2.5-7B-Bass to generate 16 responses for each problem, denoted as Yise and YRI. respectively. We also let OpenAl-ol [1] generate responses, denoted as YGT. As shown in Figure 6, the distri-
bution of PPL Y RL. (z) closely matches the lower portion of the PPL Y Base (z) distribution,
7

---

corresponding to responses that the base model tends to generate. This suggests that the responses from RL-trained models are highly likely to be generated by the base model. In Section D.4, we show that PPl- YRL//x) gradually decreases as RI. training progresses, indicating that RLVR mainly sharpens the distribution within the base model's prior rather than expanding beyond
Summary. Combining the above analyses, we arrive at three key observations. First, problems solved by the RLVR model are also solvable by the base model; the observed improvement average scores stems from more efficient sampling on these already solvable problems, rather than learning to solve new problems. Second, after RLVR training, the model often exhibits narrower reasoning cov- erage compared to its base model. Third, all the reasoning paths exploited by the RLVR model are already present in the sampling distribution of the base model. These findings indicate that RLVR does not introduce fundamentally new reasoning capabilities and that the reasoning capacity of the trained model remains bounded by that of its base model.
2.2
1.25
2.0
Perplexity
1.20
1.6
1.10
1.4
1.05
1.2
1.00
Lo
PPi
MMare Wasel Prim PrilawBiee PPlnitial p Primellar)
Figure 6: Perplexity distribution of responses. The conditioning problem is omitted in the figure.
4.2 Distillation Expands the Reasoning Boundary
In addition to direct RL training, another effective approach to improving the reasoning ability of small base models is distillation from powerful reasoning model [2]. This process is analogous to instruction-following fine-tuning in post-training. However, instead of using short instruction- response pairs, the training data consist of long CoT reasoning traces generated by the teacher model. Given the limitations of current RLVR in expanding reasoning capabilities, it is natural to ask whether distillation exhibits similar behavior. We focus on representative model, DeepSeek- RL-DDssill-Qweet 7B, which distills DeepSeek-RR RI into -renh.5-Math-7B. We compare it with the base model Qwen2.5-Mann 5-tthh- and its RL-trained counterpart
Minerva
Coverage (pars(k)
Baso
RL
and include Qwen2.5-Math-77 Instruct as an additional baseline. As shown in Figure the pass@1 curve of the distilled model is consistently and significantly above that of the base model. This indicates that, unlike RL that is fundamentally bounded by the reasoning capacity of the base model, distillation introduces new reasoning patterns Iearned from a stronger teacher model. As a result, the distilled model is capable of surpassing the reasoning boundary of the base model.
Disiil
128
Number of Samples
Figure 7: pass of base, Instruct, RLVR, and dis- tilled models.
4.3 Effects Different RL Algorithms
As discussed previously, the primary effect of RL is to enhance sampling efficiency rather than to expand a model's reasoning capacity. To quantify this, we propose the Sampling Efficiency Gap (ASE), defined as the difference between the RL-trained model's 's pass@ @ and the base model's 's pass@ k (we use k = 256 in our evaluation). Lower ASE is better. Here we conduct clean experiments to study the effect of different RL algorithms in enhancing sampling efficiency.
Experiment Setup. We re-implement popular RL algorithms using the VeRL framework [36] for fair comparison, including PPO [11], GRPO [37], Reinforce++ [38], RLOO [39], ReMax [40], and DAPO [23]. Following DAPO [23] and Oat-Zero [22), we remove the KL term to avoid constraining model Icarning. During training, we use the AdamW optimizer [41] with constant learning rate of 10-6, For rollout, we employ a prompt batch size of 256 and generate responses per prompt. The maximum rollout length is set to 8,192 tokens, and the sampling temperature is set as 1.0. We use PPO mini-batch size of 256.
To assess in-domain and out-of-domain generalization under RLVR, we split Omni-MATH-RRin subset of Omni-MATH [42] containing verifiable problems, into training set (2,000 samples) and an in-domain test set (821 samples), and use MATH500 the out-of-domain benchmark.

---

Omni-MATH Train
Omni-MATHH Test (In Domain)
MATH500 (Out Domain)
0.7
ASE 0.359
ASE 0.410
Qwan2.5-7
ASE 0.206
GRPO
0.4
ReMax
0.3
RL0O
0.2
0.2
Reinforcet+
0.1
0.1
2 4 8 16 52 64 128256
4
8
16
32 64 128256
4
8
16 32 64 128256
Number of Samples k
Number of Samples k
Number of Samples K
Omni-MATH-TTee Train
Omni-MATH- Test (In Domain)
MATH500 (Out of Domain)
0.7
0.7
1.0
0.6
ASE = 0.247
0.6
AsE
0.5
0.5
0.8
ASE 0.1.99
GRPD-4pp150
Qwen2.5-77
0.4
0.4
GRPO--t.p:000
0.6
GRPO-step150
0.3
0.3
0.4
0.1
Number of Samples k
Number of Samples K
Number of Samples
Figure 8: (Top) Different RL algorithms. (Bottom) Different RL training steps. The detailed values for each point at pass @ and pass@ 256 are provided in Table and Table
Results. As shown in Figure 8 (top), although diffferent RL algorithms exhibit slight variations in both pass@1 and pass @ 256, these differences are not fundamental. Different RL algorithms yield slightly different ASE values (i.e., ranging from GRPO's 43,9 to RLOO's best 42.6 the in-domain test set). Furthermore, we observe that ASE remains consistently above points across different algorithms, highlighting that existing RL methods are still far from achieving optimal sampling efficiency. This suggests that novel RL algorithms or entirely new paradigms may be necessary to approach the upper bound. Additional observations can be found Section D.5.
4.4 Effects of RL Training
Asymptotic Effects. Based on the setup in Section 4.3, we investigate the effect of the training steps on the asymptotic performance of the model. As shown in Figure (right), as RL training progresses, pass@) I on the training set consistently improves from 26.1 to 42.5. However, as RLVR training progresses, @ 256 progressively decreases, indicating a reduced reasoning boundary.
Effect of Number of Rollouts n. The training hyperparameter n. the number of responses per prompt, affect p & by enabling broader exploration during training. We increase from 8 to 32. As shown in Figure 16, pass@A k improves slightly over n = 8, but the RL-trained model still eventually outperformed by the base model. We leave the question of whether scaling RLVR training can eventually surpass the base model to future investigation.
Effeet of KL Loss. To control model deviation, some prior work adds KL penalty. We ablate this by applying a KL term with coefficient 0.001. As shown Figure 16, the KL-regularized model achieves similar pass @ to GRPO without KL, but with much lower pass @ 128.
4.5 Effects Entropy
As RL training progresses, the model's 's output entropy typically decreases [23], which may contribute reduced reasoning boundary due to less diverse output. To investigate this factor, increase the generation temperature of the RLVR-trained model to match the output entropy of the base model T - 0.6. As shown in Figure 18, although the RLVR model performs slightly better pass@ l: higher temperatures compared to its own performance at T = 0.6, it still underperforms the base model across pass @ k. This suggests that while reduced entropy contributes to the narrowing of the reasoning boundary, it alone does not fully account for the reduction.

---

4.6 Effects of Model Size Scaling
Scaling plays a central role in the capabiliities of contemporary LLMs. It remains an important question whether the conclusions drawn continue to hold as model size increases. For many large models, isolating the effect of RLVR is not feasible. For example, in the case of GPT-o1, the base model is not publicly accessible. Qwen3-235B [43] is trained through multiple stages, including RLVR and long-context CoT supervised fine-tuning, which makes it impossible to disentangle the impact of RLVR alone. For Deepseek-R1 R-Zeeo, the absence of publicly hosted API forced us to self- host the model, but throughput was limited to around 50 tokens per second length of 32k, rendering pass@ h evaluation currently impractical. As a more tractable alternative, we selected the Magiisral-Meciuum2555 2506 API to conduct a preliminary set of experiments. This model is trained using pure RL, with Histral-Mediume --25005---555--- as the starting model [44]. Although the model size is not disclosed, Magistral-Mediin performs comparably to Deepseek-Re RI and is positioned near the frontier in terms of reasoning capability.
maximum sequence
We queried the models using a maximum context length of 40k as the original pa- per does. Once again, we observed that RLVR provides significant gains low k, but little no improvement at higher k. Specifically, at & = 1, the RLVR- enhanced model solves approximately 7 more problems on AIME24 and 8 more on AIME25 compared to its base version. However, as & increases, the performance gap steadily narrows. These observations suggest that our conclusion continues to hold even for current, highly capable, near- frontier reasoning models. Whether this trend persists as more compute, such as pre-training scale budgets, dedicated to RL training remains a critical question for the future of LLM reasoning.
AIME24
AIME25
0.8
Coverage (peosggt))
0.6
0.6
0.4
0.4
Mistal-Mediam-- (Baac]
MismilMMdiin--
Magittad Madiaa (RLVR)
Magictal-Mrddmm (RUNV))
15 64
1024
16
64
256
1024
Number of Samples
Number of Samples
Figure 9: pass @ k curves of Magistral-Medium.
Related Work
We summarize key related works on the analysis of RLVR here and provide a more comprehen- sive discussion in Appendix B. While recent RLVR methods have achieved impressive empirical results [2. 6], their fundamental impact on reasoning remains underexplored. Several studies [45-47] suggest that reflective behaviors in RLVR models originate from the base models rather than being learned through reinforcement learning. Dang et al. [48] observed a decline in pass @k k performance Post-RLVR training, but their analysis was limited in scope. More importantly, they did not explore the relationship between the base model and the RL model. Deepseek- trends, but their study was limited to single instruction-tun tuned model and two math benchmarks. In contrast, our work systematically investigates a wide range of models, tasks, and RL algorithms to accurately assess the effects of current RLVR methods and models. We further provide in-depth anal- yses, including accuracy distributions, reasoning coverage, perplexity trends, and comparison against distilled models, offering comprehensive understanding of RLVR's capabilities and limitations.
[37] also observed similar
6 Conclusion and Discussion
RLVR is widely regarded promising approach to enable LLMs to continuously self-improve and acquire novel reasoning capabilities. In this paper, we systematically investigate the effect of current RLVR methods the reasoning capacity boundaries of LLMs. Surprisingly, our findings reveal that current RLVR does not elicit fundamentally new reasoning patterns; instead, the reasoning capabilities of RLVR- R-traineed models remain bounded by those of their base models. These results indicate that current RLVR methods have not fully realized the potential of reinforcement learning to elicit novel reasoning abilities in L.MSs through exploration and exploitation. This limitation may stem from the lack of effective exploration strategies in the vast language space or the absence of multi-turn agent-environment interactions needed to generate novel experience. We provide more detailed discussion of the possible causes of this gap and promising future directions in Section C.
10

---

Acknowledgements
This work is supported in part by the National Key R&D Program of China under Grant 20222D01 14903, the National Natural Science Foundation of China under Grants 42327901 and U24B20173, and the Scientific Research Innovation Capability Support Project for Young Faculty under Grant ZYGXQNSXYCKNIZCMMD.,
Author Contributions
All authors made valuable contributions to the experimental design, analysis, and iteration, well as to the writing, editing, and overall management of the project.
Yang Yue (F led the project, first discovered the phenomenon where RL pass @ is surpassed by the base model, and proposed the idea; designed the experiments and partially conducted experiments; took primary responsibility for writing the manuscript.
Zhiqi Chen conducted substantial experiments, including pass@ @k evaluation across models and benchmarks, and the perplexity analysis; contributed discussions, figure creation, and manuscript review.
Rui Lu contributed to inspiration the idea and conceptualization of the project, story writing and manual check of AI reasoning trajectory.
Andrew Zhao contributed to discussions on experimental design, proposed the perplexity-base
analysis, and contributed to the early implementation of the RL training code.
Zhaokai Wang contributed to discussions of RLVR's effect on reasoning boundary, writing, proofreading, and comprehensive manuscript review.
Yang Yue (AF) contributed to the training of visual reasoning model, discussions, proofreading and figure refinement.
Gao Huang & Shiji Song supervised the research, and assisted in writing the paper.
References
[1] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey, al. Openai ol system card. arXiv preprint arXiv:2412.167200 2024. 7, 16
[2] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.
2025. 1, 2, 3, 8, 10, 16
[3] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao. et al. Kimi k1. 5: Scaling reinforcement learning with 1lms. arXiv preprint arXiv:2501. 125999 2025.
[4] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0m 08774, 2023.
[5] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 1,
[6] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint 2 2024. 2, 10,
16
[7] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-535 2015.
2, 17
11

---

[8] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 550(7676):354333) 2017. 2, 17
[9] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. ICLR, 2020. 2,4
[10] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:240..2178/ 2024. 2,6
[11] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (7Xiv:7777.06377, 2017. 3, 8
[12] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8:229-256 1992. 3
[13] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[14] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021. 4, 16
[15] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14166 2021. 4,5
[16] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2.5 technical report. arXiv preprint arXiv:2412. 15115, 2024. 5
[17] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 19X4472412.19437,
2024.
[18] Weihao Zeng. Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503: 1889, 2025. 5, 16
[19] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021. 5
[20] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag. Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. NeurIPS, 2022.
[21] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-lewe bilingual multimodal scientific problems. ACL, 2024. 5
[22] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.2073
2025. 5, 8, 16
[23] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system scale. arXiv preprint 2xXx::5033.44//
2025. 5, 8,8
[24] Jiawei Liu and Lingming Zhang. Code-rl: Reproducing for code with reliable rewards. https: // /g github. com/
wwt.cooolo/////////ce wtwggaannecooeeeee r1, 2025. GitHub repository. 6, 16
12

---

[25] An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang. Jianhong Tu, Jianwei Zhang, Jingren Zhou, al. Qwen2.5- technical report. arXiv preprint ar0iv:2501. 15383, 2025. 6
[26] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. ICLR, 2025. 6
[27] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Neur/PS, 2023. 6
[28) Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang. Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepcoder: fully open-source 14b coder 03-mini level, 2025, Notion Blog.
[29] Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforeing super generaliza- tion ability in vision-language models with less than $3. https: : /ttpt//gttttttt/ee www. V1 2025, Accessed: 2025-02-02. 16
[30] Haozhan Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. Vlm-rl: A stable and generalizable rl-style large vision-language model. https: //github.
cww/om///////o om- ai 1ab/-aaabVVMMRR R1, 2025. Accessed: 2025-02-15.
[31] Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyrl: An efficient, scalable, multi-modality rl training framework. https://gittee thub.. com hamm/hyyouga/EasyL.. 2025. 6, 16
[32] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13922; 2025. 6
[33] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chup Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In ACL, 2021. 6
[34] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024. 6
[35] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In NeurlPS Datasets and Benchmarks Track, 2024. 6
(36] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu, Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024. 8
[37] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03.2 2024. 8, 10, 16
[38] Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprini 250:::20000006666 2025. 8
[39] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstun, and Sara Hooker. Back basics: Revisiting reinforce style optimization for learning from human feedback in llms. ACL, 2024. 8, 16
[40] Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. ICML, 2024. 8
[41] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In ICLR, 2017. 8
13

---

[42] Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang. Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-math: A universal olympiad level mathematic benchmark for large language models, 2025. 8
[43] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arX.v:2505.09388 2025. 10
[44] Abhinav Rastogi, Albert Q Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, et al. Magistral. arXiv preprini arXiv:2506. 10910, 2025. 10
[45] Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang, Chao Du, and Min Lin. There may not be aha moment in ri-zero-like training a pilot study, https: //opsll/ notion. site/ oat-zero, 2025. Notion Blog. 10, 16
[46] Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RI post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912, 2025.
[47] Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, et al. Rethinking reflection in pre-training. arXiv preprint arXiv:2504.0400 2025. 10, 16
[48] Xingyu Dang, Christina Back, J Zico Kolter, and Aditi Raghunathan. Assessing diversity collapse reasoning. In Scaling Delf-Impooiin Foundation Models without Human Supervision, 2025. 10, 16
[49] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NeurIPS, 2022. 16
[50] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning open resources. NeurlPS, 2023. 16
[51] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. NeurlPS, 2022. 16
[52] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et Reinforeed self-training (rest) for language modeling. arXiv preprini arXiv:2308..888000
2023. 16
[53] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. NeurlPS, 2023. 16
[54] Yang Yue, Bingyi Kang, Zhongwen Xu, Gao Huang, and Shuicheng Yan. Value-consistent representation learning for data-efficient reinforcement learning. In AAAI, 2023. 17
[55] Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannanch Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In ICLR, 2023. 17
[56] David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 2025. 17
14

---

Appendix
Appendix Contents
A Implementation Details
16
A.1 RLVR Algorithms
16
A.2 Low-Variance pass@k Estimation
16
B More Related Works
16
Discussion
17
D Detailed Experimental Results
18
D.I. More Results Mathematics and Coding
18
D.2 Validity of Chain-of-Thoon -hhnuuhh on AIME24
19
D.3 Accuracy Distribution Visulization
20
D.4 Perplexity Analysis
21
D.5 Different RLVR Algorithms
D.6 Effects of KL and Rollout Number
22
D.7 Solvable Problem Coverage Analysis
22
D.8 Temperature and Entropy Analysis
23
D.9 Training Dynamics
24
D.10 CoT Case Analysis
25
Prompt Templates
27
Broader Impacts
29
15

---

A Implementation Details
A.1 RLVR Algorithms
To reduce memory and computational overhead, several critic-free variants have been proposed. GRPO [37] estimates the advantage with a normalized reward within group of responses to the same question: A; [ri mean(r)]/std(m where r= fri,,...,Ta denotes the set of rewards for group of G sampled responses. RLOO [39] instead adopts a leave-one-out baseline within each Ez
batch B. Its advantage is defined as A, B
A.2 Low-Variance pass@k Estimation
Directly computing pass @ using only l sampled outputs per problem can lead to high variance. To mitigate this, we follow the unbiased estimation method proposed by Chen er al. [14]. Specifically, for each problem 2; from the evaluation dataset D, we generate n samples (n 2 k) and count the number of correct samples as The unbiased estimator of pass@k k over the dataset is given by:
pass @ k: EE
With this formulation, we can easily estimate pass l with low variance across all k S n.
In our experiments, we set n to the largest (i.e., rightmost) l value in the pass@ k: curves, typically 128, 256, 1024. For example, in Figure 2, we and n 1024 for AMC23 and AIME24. For the Olympiad benchmark, we set n 128 for the Qwen models and 1024 for LLaMA-3. 1-1--B, due its relatively lower base model capacity.
128 for MATH500, Minerva, and GSM8K,
B More Related Works
Reinforcement Learning for LLM Reasoning. Since the emergence of LLMs, the post-training
phase has proven crucial to enhance problem solving and reasoning abilities [49]. This stage typically falls into three main categories: supervised fine-tuning using numan-curated or distilled data [50], self-improvement iteration [51, 52], and reinforcement learning [49]. Previously, a reward model or preferences between responses were employed for reward modeling 53]. Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has gained significant traction as a method to improve the reasoning capabilities of LLMs domains such as mathematics and programming [6. 37]. An encouraging landmark work is OpenAl's ol model [1]. which was among the first large-scale applications of RL for reasoning, achieving state-of-the-ar art results at the time of its release. Following this, Deepscek-RI [2] became the first open-weight model to match or surpass the performance of ol. significant innovation introduced with RI is the "zero" setting, where reinforcement learning is applied directly to the base LLM, bypassing any intermediate supervised tuning. This approach inspired a wave of open-source efforts replicate or extend R1's 's methodology and improve RL algorithms [18, 22-24J. In parallel, reinforcement learning has also gained attention in the multimodal domain, driving advancements in multimodal reasoning [29-31].
Analysis of RIVR. Although there are many excellent open-source works and algorithmic designs in the field of RLVR, there remains lack of deep understanding regarding the root effects of RLVR on LLM reasoning abilities and its limitations when starting from the base model. Several studies [45- 47] highlight that the reflective behaviors observed in RI-like models actually emerge from the base models, rather than being introduced by RLVR training. Dang et al. [48] observed a phenomenon similar to our findings: Pass @k k deteriorates rapidly and fails to recover with reinforcement learning. but this was seen only in limited experimental setup with Qwen-2.5-0.55 on GSM8K. More importantly, they did not explore the relationship between the base model and the RL model. In contrast, our paper conducts systematic and rigorous experiments to show that not only reflective behaviors but all reasoning paths are already embedded the base model. We further demonstrate that RLVR does not elicit any new reasoning abilities beyond the base model.
16

---

C Discussion
In Section 3 and Section 4, we identified key limitations of RLVR in enhancing LLM reasoning capabilities. In this section, we explore possible underlying factors that may explain why RLVR remains bounded by the reasoning capacity of the base model.
Discussion I: Key Differences Between Traditional RL and RLVR for LLMs are Vast Action Space and Pretrained Priors. Traditional RL such as AlphaGo Zero and the DQN series [8. 7, 54] can continuously improve the performance of a policy in environments like Go and Atari games without an explicit upper bound. There for LLMs. First, the action space in language models is exponentially larger than that of Go or Atari games [55]. RL algorithms were not originally designed handle such a vast action space, which makes it nearly impossible to explore the reward signal effectively if training starts from scratch. Therefore, the second distinction is that RLVR for LLMs starts with a pretrained base model with useful prior, whereas traditional RL in Atari and GO games often begins from scratch. This pretrained prior guides the LLM in generating reasonable responses, making the exploration process significantly easier, and the policy can receive positive reward feedback.
key differences between traditional RI. and RLVR
Double-Edged Sword in This Vast Action Space. Since the sampling of responses guided by the pretrained prior, the policy may struggle to explore new reasoning patterns beyond what the prior already provides. Specifically, in such complex and highly combinatorial space, most responses generated during training are constrained by the base model's prior. Any sample deviating from the prior highly likely to produce invalid or non-sensical outputs, leading to negative reward. As discussed in Section 2.1, policy gradient algorithms aim to maximize the log- likelihood of responses within the prior that receive positive rewards, while minimizing the likelihood of responses outside the prior that receive negative rewards. As a result, the trained policy tends to produce responses already present in the prior, constraining its reasoning ability within the boundaries of the base model. From this perspective, training RL models from a distilled model may temporarily provide beneficial solution, as distillation helps inject a better prior.
Discussion 2: Priors
Possible Future Work. Developing more efficient exploration strategies may be essential for navigating the vast action space, facilitating the discovery of out-of-prior reasoning patterns and the development of more capable reasoning models. Furthermore, current RLVR frameworks are limited to single-turn interactions with the verifier, lacking the ability to iteratively revise or improve based on environmental feedback. A multi-turn RL agent paradigm wwith richer, ongoing interaction with a grounded environment could enable models to generate novel experiences and learn from them. This agent paradigm has been described as the beginning of an "era of experience" [56].
17

---

D Detailed Experimental Results
D.1 More Results on Mathematics and Coding
Qwen-2.5-7B
Qwen-2.5-14B
Qwen-2.5-32B
LLaMA-3.1-8B
Basc
Coverage (gessl
0.8
0.3
0.8
0.8
AMC23
6.66
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.2
16
256
01
236
1024
0
64
256
1024
16
1024
Number of Samples k
0.95
0.95
0.95
GSM8K
0.6
0.9
0.4
0.85
0.85
0.85
0.2
16
32
64
16
128
16
32
64
128
Figure 10: More results of SimpleRLZooo on GSM8K and AMC23.
AIME24
AIME24
1.0
Qweas25-Mui-18
Coverage (pass(@)))
0.8
0.8
Coverage (pact(())
0.6
0.4
0.4
0.2
Qwen DAPO-32B
4
16
64
256
1024
0
16
64
256
1024
Number of Samples
Number of Samples
Figure 11: Oat-Zero-7B and DAPO-32B are evaluated respective base models.
AIME24 and compared against their
LiveCodeBench (2024.8-2025.)
LiveCodeBench (2023.5-2025.)
0.5
Qwen--2.5-22 2aa-sn..---B--tttt-lll Inst-nut----M
Coverage (pass(0k @ik)
Codce Zecro Q.57-2.5-7B
Coverage (pass(@k) (kk)
0.3
0.4
0.3
0.2
0.2
Qwen-2.
0.1
0.1
1odde-RR--zxo-uueee222
4
8
16
32
64 128
2
8
16
32
64 128
Number of Samples k
Number of Samples
Figure 12: C1 on LiveCodeBench.
18

---

D.2 Validity of Chain-of- -foo-uhhthhhthtttth on AIME24
We manually check the CoTs for the most challenging AIME24 benchmark. To begin, we introduce filtering mechanism designed to eliminate easily guessable problems. Specifically, we prompt Qwen2.5-7B-Bin
answer questions directly, without using chain-of-thought reasoming, and sample answers multiple times. If problem can he answered correctly with low but non-7ero probability (e.g., <5%, we consider to be guessable and remove Problems that can be directly answered correctly with high probability are retained, as they are likely easier and solvable using valid CoT's
AIME24
0.8
Coverage (pars
0.6
0.4
022
Base
RL
16
256
1024
Number of Samples k
Figure 13: P curves of the base and SimpleRLZoo-71 models in the filtered AIME24.
The base and RL model pass@k k curves on this filtered AIME24 can be found in Figure 13, showing similar trending to previous results. Although this filtering method is heuristic, it proves to be effective. Applying it to AIME24 (30 questions) results in a subset of 18 questions. We then prompt the models answer these filtered questions using CoT reasoning. Then we perform a manual inspection of all CoTs that led to correct answers on the hardest problems those with an average accuracy below 5%. The base model answered such questions, with 5 out of 6 containing at least one correct CoT (excluding one ambiguous case of correctness due to skipped reasoning steps). Similarly, the RL-trained model answered questions, 4 of which included least one correct CoT. These results suggest that even for the hardest questions in the challenging AIME24, base model can sample valid reasoning paths to solve the problems.
19

---

D.3 Accuracy Distribution Visulization
Qwen-2.5-7B
Qwen-2.5-14B
Qwen-2.5-32B
LLaMA-3.1-8B
80
AIME24
10
110
8.8.5 S.5. 2882888 288
AMC23
180
X6
MATH500
Minerva
1000
1010
40
180
100
GSM8K
681
380
Olympiad
100
16
Figure 14: Accuracy histogram before and after RLVR with SimpleRI.Zoo models.
20

---

D.4 Perplexity Analysis
To analyze how perplexity evolves over the course RLVR training. we evaluated three RLVR checkpoints-eane middle, and final in Section 4.3 RL training, For each checkpoint, we sampled 32 responses per problem, computed the median among 32 perplexity values, and reported the average over the first 10 problems in the table. As expected, we observed that Paee (RR.) (z) gradually decreases as RL training progresses, indicating that RLVR mainly sharpens the distribution within the base model's prior rather than expanding beyond it.
Perplexity Score Evolution during Training
1.28
PPland
1.24
1,219
1.22
1.20
1.176
1.16
Training Progress
Figure 15: Perplexity Evolution during RL Training.
D.5 Different RLVR Algorithms
We report several additional observations on different RLVR algorithms in Figure 8. First, DAPO achieves slightly higher pass@1 scores across all three datasets; however, its dynamic sampling strategy requires approximately 3 ~ 6x more samples per batch during training compared to other algorithms. Moreover, its performance drops significantly at k 256. Second, RLOO and Reinforce++ perform consistently well across the entire k range (from to 256), while maintaining efficient training costs, achieving a good balance between effectiveness and efficiency. Third, ReMax shows lower performance at both pass@1 and pass @ 256. We hypothesize that this is due to its use of the greedy response reward as the advantage baseline, which in the RLVR setting is binary (0 or 1) and highly variable. This likely results in unstable gradient updates during training.
Table 3: Detailed values for each point at pass @ and pass @ 256 across different RL algorithms in Figure 8.
Model
Omni-MATH-Trrin
Omni-MATH-TE
MATH500
pass@1 1
pass @ 256
pass@1
pass 256
pass@1
pass @ 256
Qwen2.5-7B
9.9
67.2
10.2
34.5
96.2
GRPO
26.1
66.3
25.1
97.2
PPO
27.2
65.8
26.8
69.2
75.2
97.2
ReMax
24.4
65.5
23.8
67.5
73.5
96.6
RLOO
28.6
66.4
28.1
69.2
75.0
97.4
Reinforce++
28.2
67.7
28.0
69.7
75.4
96.8
DAPO
31.4
66.1
26.5
67.0
75.6
96.4
Table 4: Detailed values at pass and pass@255 @256 across different RL training steps in Figure (right).
Model
Omni-MATH-TTraie
Omai-MATH--Tes pass 256
MATH500 pass
pass@1
pass 256
pass
pass @566
Qwen2.5-7B
9.9
67.2
10.2
69.1
34.5
96.2
GRPO-step150
26.1
66.3
25.1
68.3
74.4
97.2
GRPO-step300
33.6
65.3
27.1
66.6
75.4
96.0
GRPO-step450
42.5
64.3
28.3
63.9
76.3
95.4
21

---

D.6
Effects of KL and Rollout Number
Omni-MATH-TTran
Omni-MATH-Test (In Domain)
MATH500 (Out of Domain)
0.7
0.7
0.9
0.6
0.5
0.5
0.8
0.4
0.4
0.7
4
8
16
32 64 128256
8 16 32 64 128256
2
4
8
16
32 64 128256
Number of Samples K
Number of Samples k
Number of Samples k
GRPO (n-8, steps)
n-32, 220 steps
kl-00001, 450 steps
Qwen2.5-7B
Ablation Study on KL Loss and Rollout Number n. For increasing from 8 to 32, we keep the prompt batch size unchanged, which results in increased computation per training step. Due resource constraints, we train for only 220 steps under this setting, leading to lower pass@ I as the model has not yet converged. Nevertheless, the model with 32 achieves a higher pass@ I 128, highlighting the positive effect of larger rollout numbers in improving pass i: higher values of k.
Figure 16:
D.7 Solvable Problem Coverage Analysis
Table 2 reports the fraction of problems categorized as four conditions: (1) both models solve the problem least once, (2) only the base model solves it, (3) only the RLVR model solves it, and (4) neither model solves it in any of the l samples. It highlights that there are many cases where the base model solves problem but RLVR fails (type 2), and very few where RLVR succeeds while the base does not (type 3). Even in the rare type 3 cases (e.g.. 1% or about 5 problems in MATH500), the base model able to solve all of them when sampling 1024 times. These results support our conclusion that RLVR rarely solves problems the base model cannot and generally results in reduced coverage.
Table Indices of solvable problems AIME24 (starting from 0). An approximate subset relation- ship can be observed: most problems solved by the RL model are also solvable by the base model.
Models
Problem Indices
Qwen2.5-7B
0, 1, 4, 6, 9.
19,
26, 28,
0, 6. 6. 7, 8, 9. 12, 14, 15, 16, 18, 22, 23, 24, 25, 26, 27, 28, 29
Table 6: Indices of solvable problems in LiveCodeBench (ranging from 400 to 450, starting from 0).
Model
Solvable Problem Indices
400, 402, 403, 407, 409, 412, 413, 417, 418, 419, 423. 427, 432, 433, 436, 438, 439, 440, 444, 445, 448, 449
0wen2.5-7B-Inssrccttte
400, 402, 403, 407, 412, 413, 417, 419, 422, 423,
Coder-RI
427, 430, 433, 438, 439, 440, 444, 445, 449
22

---

D.8
Temperature and Entropy Analysis
T = 0.6
T 1.0
T= 1.2
0.8
0.8
0.8
Basc
AIME24
Coverage (passt
0.6
RL
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
16
64
256
1024
1
256
1024
0-
16
64
256
1024
Number of Samples
MATH500
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
2
4
8
16
32
64
128
0.2
8
16
32
64
128
0.2
2
4
16
32
64 128
Figure found base model's performance drops when the lemperature exceeds 1.0, as tends to generate more random and less coherent tokens. In contrast, the RL modellss performance remains relatively stable across different temperature settings. Therefore, we use T = 0,6 in the main experiments, as it allows both models demonstrate their best reasoning performance.
AIME24 (Ebase = 0.47)
AMC23 (Ebase 0.51)
GSM8K (Ebase = 0.75)
0.8
1.0
0.6
0.8
0.4
0.6
0.9
0.85
0.2
Bese (T 00.)
0.4
(T-90
Base (T 0.9)
RL. (T-05)
(L(T-06)
RL (T-0.6)
0
4
16
64
256 1024
0.2
4
16
64
256
1024
16 32 64 128
MATH500 (Eoase 0.72)
Minerva (Enase 0.63)
Olympiad (Epase
Coverage (pass(@k)
0.9
0.8
0.8
0.8
0.6
0.6
0.7
(T-00)
0.4
(T-(0)
0.4
(T-06)
0.6)
RL. (T...5
RL (T-088)
RL (T-10)
0.6
0.2
0.2
64 128
64
128
16 32 64 128
Number of Samples
Figure 18: Comparison of Base and RLVR Models with Matched Output Entropy. We evaluate the base model (Qwen2.5-7BB each dataset using temperature T 0.6 and report its output entropy E in the title each figure. enable a comparison, we increase until its output entropy approximately matches Erse. For example, on AMC23, we set 1' = 0.9 to achieve ERI. = 0.47. We also include RLVR results at T = 0.6 as an additional baseline, which has lower entropy e.g.. 0.22 AMC23 and 0.33 MATH500.
temperature of the RLVR model (SimpleRI. .o00)
23

---

D.9 Training Dynamics
Actor Reward
0.35
Reward
0.25
0.20
PPO
Reinforce
GRPO (a-32)
0.10
GRPO (KL=0.001)
180
360
420
Actor Response Length
1200
GRPO
1100
PPO
Reinforce
GRPO (n-32)
Response Lengtt
GRPO (KL-0.001)
900
800
100
500
400
60
120
240
420
Actor Generation Entropy
GRPO
PPO
Reinforce-+
GRPO (e=32)
0.8
GRPO (KL-0.001)
Entropy
0.6
0.1
0.2
120
180
240
300
360
420
480
Training Steps
Figure 19: The curves of training reward, response length, and generation entropy during training, corresponding to experiments in Section 4.
24

---

D.10
CoT Case Analysis
Question
(AIME24 Question No. 16)
Among the 100 residents moville, there are 195 who own diamond digg.67 367 who own or golf cluts, and 562 who own garden spadl, addrtiion each the 900 residents bog candy hearts. Them 437 resdents who esartl these things, and residents own cractly three these things. the sumber resients Aimeville who own
trings
Qwen- 2.5- 7B nswer
Response L Length: 1058
priccple of inclusion exclusion. We will denote number of reiidnts who diamond rings, dlubs, gardon spodes, and candy hoarts S, C, mapectivoly. We given followtng information:
solre this problem,
-D =196
-G =367
-562
every resident beg candy bearts)
We are also given that 437 miiidnts aeactly two these things and residents own ecactly these things.
these things Fiust, we need Eid the number residents who own at leest one these things. This be calculated the priaciple inclusion-
dencte
residents
fins.
Total D G.G 1 C (number of residents who oun exsctly two things) (sumberr residents who
of residents
things)
Since there are 900 moidents in total, we wiitee the equation as:
900 = 195 367 562 900
437 2 234 3
Now we need to solve X:
195 367
-905-33
900 3 X
012 x
812 -900
X
-00
88
Since the number of residents cannot be negalivee there is an error in our calculations I Let's re-evaluate the problem. Win know that them are 437 residents who own pvaclly two things and nsidmnt who own extrtly thme things. can subtract thrse from the botal number of rezdents who own least one thnng find the mumber of residents who own exartly one thing:
Number of residents who own exactly one thing (D G + C) (sumber of residets who own esactly two things)
-2 (number of residents who own exaetly three things) 3 (number residents who all four things)
Let's denote the number of residents who own exacly one thing as Y:
367
9000
= 1924 905 - x
Y= 1019 x
Since every resident least thing,
equation:
900 Y
New need to solve for
900 = [1019-
3
+-37 +-77+4001 + 3
900
1019 + 437 460
900
1924
This equation is not true, which meons there is still an error in our calculations. Let's try different approach.
We can use the fact that the total number of residents who own lesst one thing is 900 We can write the oquation: residents who exactly things)
900 (number residents who own esactly one thing)
+(mumber of residents who own exarlly three things) + tzumber resiõents who own all four things)
dencte
resdents own ecactly thing
900 V 437 2 234
900 - V 805 +
Now need to find We use the foct that the total number residents who losut one who own each minus the residents who
the sum of the resdents
Itmms
900 (195 + 367 + 562 900) (mumber of residennss who exactly things)
(mumber residents
exactyy things) (number residents things)
900 1924
437
2 234
x
900
1924-
605
x
1119
Now we can solvv for X
3x 1119 900
3x - 219 x
So, the namber of residents of A imeville who all four of these things 73
Figure 20: Qwen2.5- 5-Base-BB-- 7B Correct Response Case 1.
25

---

Question
(AIME24 Question No.24)
Find the number of ways to place digit in each cell of 2x3 gıd so that the sum of the two numbers formed by reading left right 999, and 99. The gnd below is an example of such an arrangement because B + 991 999 and 9 9 9 B1 99.
8
the three numbers formed reading to
Qwen- 2.5- 7B nswer
Response Length: 967
deternine the number ways place digits a 2x3 grid that sum the numbers formed reading left right 999, and the the thrse numbers formed reading top to bottom 99, Let's break this down step by step.
solve this problem,
Define the grid: Let the grid be represented as:
b
The numbers formed by reading left right are 100a 10b + and 100d + 10e The numbers formed by reading top to hottom are 10a + d, 10b + e, and 10c +
2. Set up the equations: The sum the two numbers formed by reading left to right 999:
100a + 10b 100d 10e + f
The the three numbers formed reading top bottom 99:
10a d 106 10c
99
Simplify equations:
From the first equation:
100a + 106 + 100d 10c 999
This implies:
- 9,c
From the second equation:
10a + 10b + 10c 99
This is consistent with the previous simplification since:
10(a + b c) + (d + + f) = 99
Given 9, b b = 9, and and c follows
10(a+b+c)+++ (9-
Simplifying, get:
10(a + 1
9(a +(=+b+c)+27=0
9( ((+
a atb+c=8
Since
itollows that d= -
Determine possible values:
b, must be digits (0-9) such that + b 8. For each valid combination of b. f. are uniquely determinod as a. b,
Count the combinations
We need to count the number of non-negative integer solutions to a + b 4 = 8 with s b, s 9.
This classic stars and bars" problem with an additional constraint. number solutions is given by the binomial coefficient:
8 3-
10
3-1
Therefore, the number of ways to place the digits in the grid is 45
Figure 21: Qwen2.5-Base-7 Correct Response Case 2.
26

---

E Prompt Templates
We provide the prompt templates used for training and evaluation in our experiments. The prompt for SimpleRL training and evaluation is shown in Figure 22, while the prompt for Oat-Zero is shown adopted. For Code-RI evaluation, we follow the original codebase and adopt the default templates from the benchmarks, including LiveCodeBench prompt (Figure 25), HvmanEval+, and MBPP+ prompt (Figure 26). The prompt used for EasyR1 I training and evaluation is shown in Figure 27. For VeRL-trained RL models, as discussed in Section and Section 4.4, the training and evaluation prompts are provided in Figure 28. For evaluating Mistral and Magistral models AIME24/25, prompts provided in Figure 29. To ensure fair comparison, the base models use the same prompts as their corresponding RL-trained counterparts during evaluation.
in Figure 23. For Code-R1 training, prompt Figure
SimpleRL Prompt
<im startjsssstem
You helpful assistant <fim_ end|>
<iim start]>user (question)
Please reason step by step, and put your final answer withint(looxed(()))) end|>
<im, start]aassistant
Figure 22: Prompt for SimpleRL. Training and Evaluation. The base model uses the prompt as the RL model during evaluation.
Oat Prompt
<fim_ start|ssystem
Please reason step by step, and put your final answer within ([boxed <lim end|> <iim start]>user
{question <iim, end|> <im start|aassistant
Figure 23: Prompt for Oat-Zero training and evaluation.
Code-RI R1 Prompt
<Qim, startjsssstem
You are helpful programming assistant. The user ask you question and you as the assistant solve it. The assistant first thinks how solve the task through reasoning and then provides the user with the final answer. The reasoning process and answer are enclosed within <think>... <thinko..</thnkke </think> and <answer>.. <answer>..<sanswer> <fanswer> tags,
respectively. <iim end|>
<iim start]-user
(question) <iim_ end|> <im start/assssstnnt
Figure 24: Prompt for Code-R1 training.
27

---

LiveCodeBench (Code Generation) Prompt
start]]sssstem
You are a helpful assistant. <lim end|>
<fim start]uuser
You will be given question (problem specification) and will generate correct
Python program that matches the specification and passes all tests. You will NOT
return anything except for the program.
Question: gquestion.questinn content)
if question.starte starter code
#** Format:
(PromptC onstants.
wwwttnnts..ORMMTTTIII FORMATTING MESSAGE, WITH_STA STARTERCCODE)
py thon
(question.starte code) 6..
<fim -ineeeddd> else }
*** Format:
(ProompConstaansoooRAAAAA MESSAGE WITHOUT_STARTER,
python
OUR CODE
<lim end|>
<lim start]aassistant
'python
Figure 25: Since Code-RI does not specify an evaluation prompt, we adopt the original Live- CodeBench evaluation prompt. To encourage both the base and RL-trained models generate code, we append " ""ython to the end of the prompt. Using this setup, we reproduce a pass @ score of 28.6, which close to the reported 29.7.
HumanEval+ MBPP+ Prompt
<im_ start]
You are helpful assistant <qim_ end|>
<im start|>user
Please provide self- contained Python script that solves the following problem in markdown code block:
(python task cocm_prompt
<im, end]>
<im, start)passistant
Helow Python script with self-contained function solves the problem and passes corresponding tests:
"python
Figure 26: Prompt for Code-R1 Evaluation on HumanEval+ and MBPP+
28

---

EasyR1 Prompt
<iim startjssystem
You are Qwen, created by Alibaba Cloud. You are a helpful assistant. You FIRST think about the reasoning process as an internal monologue and then provide the final answer: The reasoning process MUST BE enclosed within <think> </think> tags. The final answer MUST BE put in (ooxed() .4im end|> <fim start]suser
<(vision (imaae token) <tvision __dd|>
{question) --fimoonn--mmmmm end|>
<jim start|aassistnnt
Figure 27: Prompt for EasyR1 training and evaluation.
VeRL Training and Evaluation Prompt
A conversation between User A ssistant. The user asks a question, and ssistant solves it. The assistant first thinks about reasoning process in the mind and then provides the user with the answer: The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, Le., <think> reasoning process here </think> <answer> answer here <'answer>, Usen O A ssistant:
Figure 28: Prompt for VeRL training Omni-math-train and evaluation on Omni-math-eval and MATH500.
Mistral & Magistral Prompt
A conversation between User and Assistunt. The asks question, and the Assistant
The assislant first thinks about reasoning process the mind and then provides user with answer. The reasoning process answer are enclosed within <hhink> </think> and <answer> </answer> tags, respectively, i.e. <think> reasoning process here </think> <answer> answer here </unswer>. User: (question). Assistant:
Figure 29: Prompt for Mistral and Magistral model evaluation.
Broader Impacts
The potential negative social impacts of our method align with those typically associated with general I.LM reasoning technologies. We emphasize the importance of adhering to the principles of fair and safe deployment L.M systems.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss the limitations of the work in the section of conclusion and limitation.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
30

---

Answer:
[NA]
Justification:
This paper does not involves theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed information about experiments in the appendix and provide the source code that can reproduce reported results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
31

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will release our code, models and data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//niss httppsccccpbccccccceececcccccccc htt.g/edede/eeeemeiiiaeeeeee /Cooeeeooiissseeeeeee wwssssnollicyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We specify all the training and test details in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We repeated experiments with several runnings and found error bar is relatively
small.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
32

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provided sufficient information on the computer resources the main text and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: https://meee /tte://///eupp///eeee cc wwtp///bl.ccccccccccc.e.eee.l.le inees?
Answer: [Yes]
Justification: The research conducted in the paper conformed, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential societal impacts in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
33

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification:
We describe safeguards for responsible release of models in the social impacts
section.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited the creators original owners of assets (e.g., code, data, models), used in the paper and conformed the license and terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
34

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We communicated the details of the dataset/ccoeemmodl part of their submission.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve study participants.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
35

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy btttp::/neeuips... /ttpr//ps..c/ooofeeeeee/////255/ for what should or should not be described.
36


Reviews:
[{"review_id": "5FpsCzihZw", "reviewer": "Reviewer_79qN", "summary": "The authors are interested in the following question: can LLMs fine-tuned with current RLVR approaches develop new reasoning capabilities, beyond those already present in (even if hard to elicit from) the base models? The key methodology used to examine this question is comparing the pass@k performance of base and fine-tuned models for increasingly high k values (up to 256). \n\nConsistently, and across different models, RL algorithms and benchmarks, the authors find that base models match or outperform fine-tuned models at high k values. All of the results are on 7-32B base models (Qwen2.5, Llama-3.1), trained with standard RL algorithms (GRPO, PPO, RLOO) on a variety of reasoning benchmarks (GSM8K, MATH, Geometry3K). \n\nThe intuition provided for these results is that RLVR training narrows down the model distribution towards more-likely-to-be-rewarded reasoning paths, in the process approaching the base model's high k pass@k performance at much lower k values; but without expanding from the base model's distribution (despite the base models having the capacity to learn so, as illustrated by distillation results).", "questions": "1. Did you run any experiments on the models closer to the frontier, and if so, did you observe the same pattern? For example, DeepSeek-V3(-Base) vs DeepSeek-R1-Zero.\n\n2. Similarly, do you have any experiments with SFT pre-training, does it modify the conclusions? \n\n3. The term \"reasoning capacity/boundary\" seems like a bit of a misnomer; shouldn't it be solution-space/reasoning coverage or scope? Curious about the reasoning behind this term.", "limitations": "The discussion on Limitations should be improved, see the first point in the Weaknesses section. \n\nNo concerns about negative societal impact.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nThe paper's research questions is very topical and important, with significant implications for developing LLMs capable of discovery. While the main claim isn’t too surprising and has been observed already in prior literature as the authors acknowledged, this paper makes a significant contribution by providing a thorough analysis and stronger evidential basis, as well as simple & intuitive explanations. This includes demonstrating the same phenomena across a large number of training domains, models and RL algorithms; alongside other supporting experiments that strengthen the story. The results are remarkably consistent across all these variations.\n\nI also appreciate the variety of supporting/control experiments provided, from which I'd highlight the following: \n- comparing surprisal of the RL responses under base and RL policy (--> RL responses are within the base model distribution)\n- comparison of models trained at different checkpoints (--> shows not improved by training longer)\n- distillation from reasoning traces of a much bigger model (DeepSeek-R1) (--> shows base model’s intrinsic capacity isn’t the obstacle to learning better reasoning trajectories)\n- comparison over different training GRPO hyperparameters and KL divergence (--> does not change the results)\n- controlling for output entropy (--> does not change the results)\n\n## Weaknesses\n\nThe biggest limitation of the paper is that the results are all on relatively small models (7-32B parameters) and low training data diversity (for example, mathematical reasoning models are trained only on GSM8K + MATH, coding models on LeetCode and TACO): quite far from the RLVR-trained frontier systems of today. As we've seen repeatedly in the LLM literature, both factors can significantly & suddenly change properties of models (Wei et al, 22). Given the overall thoroughness of analysis, the results are still very convincing for the small-to-mid scale RLVR pipelines (which are still important and worth publishing on!), but this limitation should be acknowledged better in the paper. As is, there is no mention of this in the Discussion section.    \n\nA couple of other points, which if addressed, would make the paper stronger:\n- Random guessing explanation: low % of problems with at least one correct CoT in answers on a sample of 8-25 problems is a good sanity check, but a pretty low bar to clear. Example of something that would be more assuring: report % of trajectories containing flawed CoTs, do this for larger sample sizes using LLM-as-judge, use these measurements to estimate the bound on performance attributable to random guessing \n- Over-fitting to test set: it's another alternative explanation I don't see addressed in the paper (distillation and train vs test results should indicate overfitting can't fully account for it, but this connection is not made explicit)  \n\n## Minor\n\n- In Table 1: could you also somehow note training, not just test sets? \n- Fig 6: add a sentence on the relevance of comparing to perplexity of o1 responses\n- Fig 8: hard to read, perhaps show only k@1 and k@256 or $\\Delta_{SE}$?\n- Table 4 & 5: this would be a more effective demonstration if you just reported % overlap for a larger number of benchmarks\n- Fig 16, AIME24: it looks like higher temperature actually benefits the RL model at high pass@k (in contrast to the provided explanation that T=0.6 was chosen as the best regime for both models)", "was_revised": true, "final_justification": "Overall, a very thorough paper and a solid contribution to understanding the effects of RLVR training. \n\nThe rating was updated from Accept to Strong Accept in rebuttal, after adding results from models closer to the frontier and fixing a number of smaller issues.", "rebuttal": "# Response\n\nThanks for supporting the acceptance of our work. Please find our responses to your questions below.\n\n### `W1: The biggest limitation of the paper is that results are all on relatively small models (7-32B parameters) and low training data diversity; Q1: Did you run any experiments on the models closer to the frontier?`\n\nThank you for the insightful comment. We also care deeply about whether scaling up model size affects our conclusions.\n\nAfter careful investigation, we found that **only three models over 70B parameters have both base and RLVR-trained versions publicly available: DeepSeek-R1-Zero, Tulu-3-70B [1], and Magistral [2].** We summarize them in the table below.\n\nFor many other large models, isolating the effect of RLVR is not feasible. For example:\n- **GPT-o1**: The base model is not accessible, and training details are unclear; the final model may not rely solely on RLVR.\n- **Qwen3-235B**: The reasoning model undergoes multiple stages, including RLVR and long-CoT SFT, impossible to isolate the effect of RLVR.\n- **LLaMA-3-70B**: no corresponding RLVR version exists.\n| Model Pair | Size | API Access | Weights Access |\n| - | - | - | - |\n| DeepSeek-V3-Base / R1-Zero | 671B | ✗ | ✓ |\n| Tulu-3-SFT / Tulu-3-RLVR | 70B | ✗ | ✓ |\n| Mistral-Medium-3 / Magistral-Medium | undisclosed* | ✓ | ✗ |\n\n*: **undisclosed size but reasoning performance close to DeepSeek-R1-671B**\n\n**For Deepseek-R1-Zero**, we have done our best to make it run with limited resources (two 8xH100 GPU nodes). However, without expert engineering optimization, the throughput remains very low, around 50 tokens/s at a maximum sequence length of 32k. Runnings on AIME24 with 1024 samplings would take approximately 4 months, which makes it currently infeasible. We are actively seeking additional resources and hope to resume the process later.\n\nThus, we selected Tulu-3-70B and Magistral.\n\n**For Tulu-3-70B**, which we hosted locally with 8 H100 GPUs, we conducted evaluations on the first 100 subsampled problems from the MATH500 benchmark due to high computational cost. The results below are consistent with our main conclusion.\n\n### **Tulu3 on Math500**\n\n|Model|k=1|k=4|k=16|k=32|\n|-|-|-|-|-|\n|Tulu-3-SFT|58.6%|77.2%|88.1%|91.0%|\n|Tulu-3-RLVR|64.4%|80.9%|87.7%|89.0%|\n|Δ(RLVR-SFT)|+5.8%|+3.7%|-0.4%|-2.0%|\n\n**For Magistral, one of the most advanced reasoning models**, we queried the API using a maximum context length of 40k. Again, we observed the RLVR model has great improvements at low k, but little to no gain at larger k. Specifically, at k = 1, the RLVR model solves approximately 7 more problems than the base on AIME24 and 8 more on AIME25. However, as k grows, the performance gap steadily narrows. **The trend observed in this preliminary experiment further supports that our conclusion holds even for a frontier-level reasoning model.**   \n### **Magistral on AIME24**\n\n|*k*|1|4|64|256|1024|\n|-|-|-|-|-|-|\n|Mistral-Medium-3(Base)|53.5%|69.7%|80.0%|87.6%|90.0%|\n|Magistral-Medium(RLVR)|76.6%|88.8%|90.0%|93.3%|93.3%|\n|Δ(RLVR-Base)|+23.1%|+19.1%|+10.0%|+5.7%|+3.3%|\n\n\n### **Magistral on AIME25**\n\n|*k*|1|4|64|256|1024|\n|-|-|-|-|-|-|\n|Mistral-Medium-3(Base)|33.3%|51.5%|73.2%|81.3%|93.3%|\n|Magistral-Medium(RLVR)|62.0%|75.8%|88.7%|92.1%|93.3%|\n|Δ(RLVR-Base)|+26.7%|+24.3%|+15.5%|+10.8%|0.0%|\n\n\nWe will include these results  in the revision and explicitly discuss the scalability limitation. A more comprehensive evaluation on very large models is currently constrained by resources, but this is an important direction for future work.\n\n[1] Allen Institute for AI. \"Tulu 3: Pushing frontiers in open language model post-training.\"\n\n[2] Mistral AI. \"Magistral.\" arXiv:2506.10910.\n\n### `W2: Random guessing explanation: Example of something that would be more assuring: report % of trajectories containing flawed CoTs, use these measurements to estimate the bound on performance attributable to random guessing`\n\nThanks for your suggestion. If we understand correctly, by \"% of trajectories containing flawed CoTs,\" you’re referring to the proportion of incorrect reasoning paths across multiple samples per problem. While this is a useful metric, it serves a different purpose from our analysis.\n\nAs stated in Section 2.2, our goal is to assess the **coverage of model capacity—specifically, whether the model can generate at least one correct CoT for a given problem**. This is why we manually check whether at least one correct CoT appears per problem. In contrast, measuring the percentage of flawed CoTs aligns more with **average-case evaluation**, which, while valuable, does not directly capture capacity coverage. \n\nTo further address concerns about random guessing, we refer to the **coding task** in Section 3.2, where success requires passing unit tests—making random guessing highly unlikely. **The same trends hold in that setting, which supports our conclusions and helps rule out random guessing as an explanation.**\n\n### `W3: Over-fitting to test set`\n\nThank you for raising this point. If we understand correctly, by \"overfitting to the test set,\" you are referring to potential test data leakage during RLVR training. We will add an explicit discussion to clarify in revision, as follows:\n\nFor the RLVR checkpoints in the main experiments in Section 3, the RL training data is publicly documented and does not overlap with the evaluation benchmarks. Further, in our deep analysis (e.g., RLVR train with OmniMath data), we further ensure a clean train/test split, so there is no data leakage. \n\nIf we have misunderstood your concern, please let us know—we would be happy to clarify further.\n\n### `Q2: do you have any experiments with SFT pre-training, does it modify the conclusions?`\n\nIf we understand correctly, you're asking whether our conclusions hold when RLVR is applied to models already fine-tuned via SFT or distillation. If it's yes, we have included such experiments in our paper\n\nIn coding task in Section 3.2, we evaluated DeepCoder-14B, which is trained using RLVR on top of DeepSeek-R1-Distill-Qwen-14B, an SFT model based on the pretrained Qwen-2.5-14B. As shown in Figure 3 in the paper and summarized below, as k increases, base models consistently catch up to and eventually surpass RL-trained models:\n\n|pass@k|1|4|16|64|\n|-|-|-|-|-|\n|DeepSeek-R1-Distill-Qwen-14B|52.7%|63.1%|70.0%|75.6%|\n|DeepCoder-14B (RLVR)|57.2%|66.6%|71.9%|75.2%|\n\n To further support our conclusion on SFT models, we add results on math task and will add it later in revision. Here, the RLVR model is trained on DeepSeek-R1-Distill-Qwen-1.5B (an SFT model from Qwen-2.5-1.5B). Evaluated on AIME24. Similar trends are observed.\n\n|pass@k|1|4|16|64|\n|-|-|-|-|-|\n|DeepSeek-R1-Distill-Qwen-1.5B|29.9%|52.0%|71.2%|80.4%|\n|RLVR|47.3%|66.1%|75.0%|77.0%|\n\nIn summary, these results reinforce our main findings, even when RLVR is applied starting from an SFT model. \n\n### `Q3: The term \"reasoning capacity/boundary\" seems like a bit of a misnomer; shouldn't it be solution-space/reasoning coverage or scope? `\n\nThank you for pointing this out. You're right that our use of terms like reasoning capacity/boundary may have been somewhat inconsistent and potentially confusing.  **\"Reasoning coverage\" or \"reasoning scope\"** more accurately reflect the concept we intended. We will revise the terminology in the revision.\n\n### `Minors`\n\nThank you for your careful review and valuable suggestions. We address each point below and will incorporate the corresponding changes in the revision.\n\n- **Table 1**: We currently omit training set details due to space limitations. In the revision, we will adjust the layout to make the training data clearly identifiable for each experiment.\n- **Figure 6**: The comparison with the perplexity of o1 responses serves as a baseline, illustrating how \"surprising\" responses from other reasoning models (like o1) appear to the base.  We will add a sentence to clarify this in the revision.\n- **Figure 8**: The original purpose of using a folded y-axis was to highlight the ordering differences between algorithms at both k=1 and k=256.  However, we acknowledge the readability concerns of fig.8, and we will replace it with a more reader-friendly version in the revision.\n- **Figure 16 (AIME24)**:  Our choice of T=0.6 follows the evaluation protocol of DeepSeek-R1 and provides stable performance across benchmarks. We found empirically that the optimal temperature for RLVR model varies by task (e.g., higher T benefits AIME24, while lower T helps AMC23), and T=0.6 offers a reasonable overall trade-off. As shown in Figure 17, we also experimented with increasing T for RLVR. While this improves performance on AIME24, on most other benchmarks it performs comparably to RLVR at T=0.6. Even on AIME24, RLVR with higher T still lags behind the base model at large k, reinforcing our main conclusion.\n- **Tables 4 & 5**: We appreciate your suggestion and will revise the tables accordingly. Specifically, we will revise the tables to show the fraction of problems in four categories: (1) both models solve, (2) only base solves, (3) only RLVR solves, and (4) neither solves. Results are as follows:\n\n|Base|SimpleRLZoo|AIME24(k=1024)|MATH500(k=128)|\n|-|-|-|-|\n|✓|✓|63.3%|92.4%|\n|✓|✗|13.3%|3.6%|\n|✗|✓|0.0%|1.0%|\n|✗|✗|23.3%|3.0%|\n\nThe results highlight key observations:\n\nThe overlap in correct solutions  (type 1) is dominant. There are many cases where the base model solves a problem but RLVR fails (type 2), and very few cases where RLVR succeeds while the base does not (type 3). Even in rare type 3 cases (e.g., 1% or 5 problems in MATH500), when sampling 1024 times, the base model can solve all of them.\n\nThis supports our findings that RLVR rarely solve problems the base cannot, and generally achieve lower pass@k at large k, indicating reduced coverage.\n\n### `conclusion`\n\nWe hope our responses have addressed your concerns and strengthened confidence in our paper. Should there be any further questions, feel free to reach out."}, {"review_id": "jQklQGVvNT", "reviewer": "Reviewer_tKuY", "summary": "This paper systematically investigates whether RLVR genuinely enhances LLM reasoning beyond base model capabilities. Using pass@k metrics across math, coding, and visual reasoning tasks, the authors find that RLVR improves sampling efficiency (pass@1) but narrows reasoning boundaries (pass@k for large k). Key finding: RLVR models only exploit reasoning paths already present in base models rather than discovering new strategies.", "questions": "- How do the findings extend to larger base models (e.g., 70B+ parameters)? Would the reasoning boundary limitations persist?\n- Could different reward structures (e.g., process rewards vs. outcome rewards) change the conclusions about RLVR's effectiveness?", "limitations": "- Evaluation scope: Limited to current RLVR methods and may not apply to future RL paradigms\n- Base model selection: Results may be influenced by the specific pretrained models used", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths*\n- Comprehensive evaluation: Multiple model families, tasks, and RLVR algorithms tested systematically\n- Novel methodology: Pass@k curves reveal hidden limitations invisible to traditional metrics\n- Rigorous analysis: Perplexity analysis and manual CoT validation strongly support main claims\n- Important negative result: Challenges widespread belief about RLVR's capability expansion\n- Clear presentation: Well-structured with effective visualizations\n\n*Weaknesses*\n- Limited theoretical insight: Lacks explanation for why RLVR fails to expand reasoning boundaries\n- Narrow scope: Findings may not generalize to future RL paradigms or different reward structures\n- Missing solutions: Identifies problems clearly but provides limited improvement suggestions\n- Base model dependency: Conclusions may be influenced by specific base models tested", "was_revised": true, "final_justification": "Rating 6.\nMost of my concerns are sovled", "rebuttal": "# Response\n\nThanks for supporting the acceptance of our work. Please find our responses to your questions below.\n\n### `W1: Limited theoretical insight: Lacks explanation for why RLVR fails to expand reasoning boundaries`\n\nThank you for the thoughtful comment. While our paper focuses on empirical findings, we provide an intuitive explanation in Appendix C, summarized here:\n\nCurrent RLVR methods (e.g., PPO, GRPO) are **on-policy**, meaning they rely on the model to generate its own training samples. Given **the vast language space** (roughly $V^L \\approx 150000^{4096}$, where $V$ is the vocab size and $L$ the avg response length), training from scratch is infeasible because the model would rarely encounter any positive reward under outcome-based reward settings.\n\nAs a result, RLVR relies on the **pre-trained prior** to produce responses likely to earn positive reward. This dependence restricts exploration: most on-policy samples stay within the prior’s region, and every rewarded trajectory further reinforces this behavior through gradient updates. Soft-max sampling can occasionally generate out-of-prior trajectories, but they are exceedingly rare; when they do appear, they are usually invalid and receive negative outcome reward. This feedback loop amplifies the exploration bottleneck.\n\nInterestingly, we find that the findings, and intuitive explanations presented in our paper have been formalized into a theoretical framework in a recent work [1]. It provides theoretical support for the limitation we identify: under both KL-constrained and KL-free RLVR settings, the probability mass assigned by RLVR-trained models outside the base model’s support is theoretically bounded, and this upper bound is typically very small in practice. (Theorem 2.5, Corollary 2.7). This theory work further supports our findings, and the proof approach closely reflects the intuition we provide in Appendix C.\n\nWe will clarify this explanation for our findings further in the revision.\n\n[1] The Invisible Leash: Why RLVR May Not Escape Its Origin. 2025 07.\n\n### `W2&Q2: Narrow scope: Findings may not generalize to future RL paradigms or different reward structures;  Could different reward structures (e.g., process rewards vs. outcome rewards) change the conclusions about RLVR's effectiveness?`\n\nThank you for the insightful question. As stated in the abstract and introduction, our study primarily aims to clarify the limitations of **current** RLVR. \n\n**By identifying what today’s RLVR can and crucially cannot achieve, we hope to motivate future RL research toward unlocking genuinely novel reasoning abilities.**\n\nProcess-level rewards are indeed a promising direction for addressing current RLVR's limitation: in principle, assigning credit to intermediate reasoning steps could help alleviate the exploration inefficiency. However, current process rewards tend to be noisy and vulnerable to hacking, which is why most practical implementations still rely on outcome-based rewards—where the limitations we expose remain significant.\n\nHistory has shown that RL can elicit new capabilities (e.g., AlphaZero in Go). However, RLVR for LLM faces a different set of challenges, most notably, the exponentially larger action space and the strong influence of the pretrained prior, as discussed in detail in Appendix C. We believe that with further advances such as improved process rewards, RL may eventually overcome these limitations (we elaborate on potential directions in the next question). **However, before that, clearly identifying the current gap, as our paper does, is a critical first step toward that goal.**\n\n### `W3: Missing solutions: Identifies problems clearly but provides limited improvement suggestions;`\n\nThank you for the comment. Pinpointing the problem is only the first step; finding viable remedies is equally important for advancing LLM reasoning models. We see several promising research directions:\n1. **Finer-grained reward structures**: step-wise rewards guide intermediate reasoning and reduce exploration bottlenecks.\n2. **Improved exploration**: Instead of naive softmax sampling, introduce structured or hierarchical search to enhance exploration efficiency.\n3. **Better long-horizon credit assignment**: Use techniques to propagate reward more effectively over long CoT chains and enabling the model to assign credit to crucial intermediate steps instead the whole response\n4. **Scaling up RL training**: Match RLVR compute and data scale to that of pre-training\n5. **Multi-turn tool use & external knowledge**: Allow the agent to interact with tools or retrieve external facts, broadening the reasoning space beyond single-pass generation\n\nWhile Appendix C already hints at several possible solutions, we will make these directions clearer and more concrete in the revision. Ultimately, addressing this limitation will require collective effort, and we hope our paper and the directions outlined above can help catalyze progress in this important area.\n\n### `W4: Base model dependency: Conclusions may be influenced by specific base models tested`\n\nThank you for the comment.  In our paper, we include  Qwen, LLaMA-3.1 and  DeepSeek-R1-Distill-Qwen-14B as a starting model for RLVR.\n\nTo further demonstrate the generality of our findings, we additionally conducted experiments on other base models and their RLVR-trained versions, including: Gemma-2-2B [1], Mimo-7B [2], Tulu-3-70B [3].\n\nThe results are as follows:\n### **Gemma-2-2B on Math500**\n\n|  | pass@1 | pass@4 | pass@16 | pass@64 | pass@256 |\n| --- | --- | --- | --- | --- | --- |\n| Base | 25.8% | 39.3% | 52.4% | 64.6% | 75.0% |\n| RLVR | 28.0% | 42.1% | 54.4% | 65.2% | 73.8% |\n| Δ (Base - RLVR) | -2.2% | -2.8% | -2.0% | -0.6% | +1.2% |\n\n### **Mimo-7B on Math500**\n\n|  | pass@1 | pass@4 | pass@16 | pass@64 | pass@256 |\n| --- | --- | --- | --- | --- | --- |\n| Base | 39.1% | 69.4% | 89.4% | 96.6% | 98.6% |\n| RL-Zero | 65.3% | 90.6% | 96.5% | 97.8% | 98.4% |\n| Δ (Base - RLVR) | -26.2% | -21.2% | -7.1% | -1.2% | +0.2% |\n\n### **Tulu-70B on Math500**\nFor Tulu-3-70B, we evaluate on the first 100 problems of MATH500 due to the high computational cost.\n\n| Model          | k=1  | k=4  | k=16 | k=32 |\n| -------------- | ---- | ---- | ---- | ---- |\n| Tulu-3-SFT*     | 58.6% | 77.2% | 88.1% | 91.0% |\n| Tulu-3-RLVR    | 64.4% | 80.9% | 87.7% | 89.0% |\n| Δ (SFT - RLVR) | -5.8% | -3.7% | +0.4% | +2.0% |\n\n*: Tulu-3-RLVR is RL-trained based on Tulu-3-SFT.\n\n**These results further support the robustness and broader applicability of our conclusions across model families.**\n\n[1] Gemma 2: Improving Open Language Models at a Practical Size.\n\n[2] MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining\n\n[3] Tulu 3: Pushing Frontiers in Open Language Model Post-Training\n\n### `Q1: How do the findings extend to larger base models (e.g., 70B+ parameters)?`\n\nThank you for the insightful comment. We also care deeply about whether scaling up model size affects our conclusions. During the rebuttal, we conducted extensive investigations and experiments on this topic, detailed in our response to **Reviewer 79qN’s Weakness 1 (W1)**.\n\n**Our results show that the findings hold even for 70B+ models and state-of-the-art frontier models**. Due to space constraints and the absence of a global response section, we kindly refer you to that response for the full discussion.\n\n### `conclusion`\n\nWe hope our responses have addressed your concerns and strengthened confidence in our paper. If you have any further questions, please feel free to reach out."}, {"review_id": "UEhUiXE5Wh", "reviewer": "Reviewer_6ryS", "summary": "This paper investigates whether RLVF actually improves pass@k for reasoning LLMs beyond the base model. \nThrough extensive experiments, the authors find that across different datasets and RLVF algorithm settings, a consistent observation is that the fine-tuned model does not outperform the base model in pass@k when k is sufficiently large. \nCase studies further show that the solved questions are largely a subset of those solvable by the base model.", "questions": "- What are the specific policy optimization parameters used during RLVF? For example, what values did you set for ppo_mini_batch_size and ppo_epochs in VeRL?\n- Why does the \"Perplexity Analysis\" paragraph only analyze two samples from AIME?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n- This work serves as a relatively systematic experimental survey on an important question—whether RLVF improves pass@k. Although it does not fully resolve the question nor provide theoretical explanations, the clear and detailed experimental results are likely to contribute positively to ongoing discussions in the reasoning LLM community.\n- The writing is easy to follow, and the results are clearly organized, which is also aided by the accessibility of the core topic.\n\n### Weaknesses:\n- There is no theoretical explanation—only extensive experimental results. The paper includes some intuitive descriptions, such as the repeated reference to base models’ boundaries, but they remain informal.\n- The experiments mainly focus on the Qwen model series.\n- The evaluation is too centered on pass@k curves. I would appreciate more analyses like the one in the “Perplexity Analysis” paragraph.", "was_revised": true, "final_justification": "This is a solid empirical contribution to the RL+LLM, reasoning, and post-training domains. While the claims are not entirely novel, prior work has only offered partial insights in limited settings. Therefore, I believe this work has the potential to make a meaningful impact on the community.", "rebuttal": "# Response\n\nThanks for supporting the acceptance of our work. Please find our responses to your questions below.\n\n### `W1: no theoretical explanation. The paper includes some intuitive descriptions, such as the repeated reference to base models’ boundaries, but they remain informal.`\n\n\nThank you for the thoughtful comment. While our paper focuses on empirical findings, we provide an explanation for findings in Appendix C, summarized here:\n\nCurrent RLVR methods (e.g., PPO, GRPO) are **on-policy**, meaning they rely on the model to generate its own training samples. Given **the vast language space** (roughly $V^L \\approx 15000^{4096}$, where $V$ is the vocab size and $L$ the avg response length), training from scratch is infeasible because the model would rarely encounter any positive reward under outcome-based reward settings.\n\nAs a result, RLVR relies on the **pre-trained prior** to produce responses likely to earn positive reward. This dependence restricts exploration: most on-policy samples stay within the prior’s region, and every rewarded trajectory further reinforces this behavior through gradient updates. Soft-max sampling can occasionally generate out-of-prior trajectories, but they are exceedingly rare; when they do appear, they are usually invalid and receive negative outcome reward. This feedback loop amplifies the exploration bottleneck.\n\nInterestingly, we find that the findings, and intuitive explanations presented in our paper have been formalized into a theoretical framework in a recent work [1]. It provides theoretical support for the limitation we identify: under both KL-constrained and KL-free RLVR settings, the probability mass assigned by RLVR-trained models outside the base model’s support is theoretically bounded, and this upper bound is typically very small in practice. (Theorem 2.5, Corollary 2.7). This theory work further supports our findings, and the proof approach closely reflects the intuition we provide in Appendix C.\n\nWe will clarify this explanation for our findings further in the revision.\n\n[1] The Invisible Leash: Why RLVR May Not Escape Its Origin. 2025 07.\n\n### `W2: mainly focus on the Qwen model series.`\n\nThank you for the comment. In addition to Qwen models, we also include  LLaMA-3.1 and  DeepSeek-R1-Distill-Qwen-14B as a starting model for RLVR in our paper.\n\nTo further demonstrate the generality of our findings, we additionally conducted experiments on other base models and their RLVR-trained versions, including: Gemma-2-2B [1], Mimo-7B [2], Tulu-3-70B [3].\n\nThe results are as follows:\n### **Gemma-2-2B on Math500**\n\n|  | pass@1 | pass@4 | pass@16 | pass@64 | pass@256 |\n| --- | --- | --- | --- | --- | --- |\n| Base | 25.8% | 39.3% | 52.4% | 64.6% | 75.0% |\n| RLVR | 28.0% | 42.1% | 54.4% | 65.2% | 73.8% |\n| Δ (Base - RLVR) | -2.2% | -2.8% | -2.0% | -0.6% | +1.2% |\n\n### **Mimo-7B on Math500**\n\n|  | pass@1 | pass@4 | pass@16 | pass@64 | pass@256 |\n| --- | --- | --- | --- | --- | --- |\n| Base | 39.1% | 69.4% | 89.4% | 96.6% | 98.6% |\n| RL-Zero | 65.3% | 90.6% | 96.5% | 97.8% | 98.4% |\n| Δ (Base - RLVR) | -26.2% | -21.2% | -7.1% | -1.2% | +0.2% |\n\n### **Tulu-70B on Math500**\nFor Tulu-3-70B, we evaluate on the first 100 problems of MATH500 due to the high computational cost.\n\n| Model          | k=1  | k=4  | k=16 | k=32 |\n| -------------- | ---- | ---- | ---- | ---- |\n| Tulu-3-SFT*     | 58.6% | 77.2% | 88.1% | 91.0% |\n| Tulu-3-RLVR    | 64.4% | 80.9% | 87.7% | 89.0% |\n| Δ (SFT - RLVR) | -5.8% | -3.7% | +0.4% | +2.0% |\n\n*: Tulu-3-RLVR is RL-trained based on Tulu-3-SFT.\n\n**These results further support the robustness and broader applicability of our conclusions across model families.**\n\n[1] Gemma 2: Improving Open Language Models at a Practical Size.\n\n[2] MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining\n\n[3] Tulu 3: Pushing Frontiers in Open Language Model Post-Training\n\n### `W3: The evaluation is too centered on pass@k curves. I would appreciate more analyses like the one in the “Perplexity Analysis” paragraph.`\n\nWe appreciate your recognition of the perplexity analyses beyond pass@k. In our paper, we also include **accuracy distributions, reasoning coverage, and comparisons with distilled models**, intended to provide a more comprehensive view of RLVR’s capabilities and limitations.\n\nDuring the rebuttal, we further expanded the perplexity analysis, **adding the dynamics of perplexity throughout training**, which will be included in the revision  (see details in our response to Reviewer KdU6’s Q4 for details if you are interested).\n\nAdditionally, we plan to incorporate more plausible explanations and potential solutions for RLVR's limitations to deepen the discussion. We believe these additions will strengthen the paper beyond pass@k curves and help enrich understanding of the findings. Thank you again for your constructive suggestion.\n\n### `Q1: What are the specific policy optimization parameters used during RLVF? For example, what values did you set for ppo_mini_batch_size and ppo_epochs in VeRL?`\n\nThank you for the question. For broad experiments, we use publicly released checkpoints from existing RLVR works such as SimpleRLZoo, DAPO, OAT, and DeepCoder. The RL training hyperparameters for these models can be found in their respective papers.\n\nFor our in-depth analysis, we trained RLVR models ourselves using the following settings:\n\n| Parameter             | Value |\n| --------------------- | ----- |\n| Rollout temperature   | 1.0   |\n| Prompt batch size     | 256   |\n| Rollout number        | 8     |\n| ppo_mini_batch_size | 256   |\n| ppo_epochs          | 1     |\n\n\nThis results in 8 gradient updates per rollout step, calculated as:\n(Prompt batch size × Rollout number × PPO epochs) / PPO mini-batch size.\n\n### `Q2: Why does the \"Perplexity Analysis\" paragraph only analyze two samples from AIME?`\n\nDue to space limitations, we included only two examples in the paper as representative cases—the remaining problems exhibit similar patterns. In fact, we conducted perplexity analysis on all 30 AIME problems, and in most cases, the base model assigns lower perplexity to RLVR responses than to its own, and other cases comparable.\n\n|problem id|$PPL_{base}(Y_{base})$|$PPL_{base}(Y_{RL})$|\n|----------|----------------------|---------------------|\n|0|1.133|**1.065**|\n|1|1.166|**1.155**|\n|2|1.254|**1.209**|\n|3|**1.128**|1.140|\n|4|1.117|**1.076**|\n|5|**1.092**|1.103|\n|6|1.182|**1.138**|\n|7|1.122|**1.070**|\n|8|1.176|**1.076**|\n|9|1.121|**1.107**|\n|10|1.212|**1.115**|\n|11|1.307|**1.253**|\n|12|1.122|**1.066**|\n|13|1.186|**1.173**|\n|14|**1.130**|1.145|\n|15|1.110|**1.052**|\n|16|1.201|**1.106**|\n|17|1.112|**1.090**|\n|18|1.132|**1.115**|\n|19|**1.145**|1.156|\n|20|1.163|**1.044**|\n|21|**1.256**|1.283|\n|22|1.225|**1.160**|\n|23|1.154|**1.092**|\n|24|1.085|**1.055**|\n|25|**1.225**|1.247|\n|26|1.151|**1.098**|\n|27|1.130|**1.075**|\n|28|1.171|**1.125**|\n|29|1.302|**1.278**|\n|Avg.|1.167|**1.129**|\n\n\nWe will include the full set of figures in the appendix of the revision.\n\n`conclusion`\n\nWe hope our responses have addressed your concerns and strengthened confidence in our paper. If you have any further questions, please feel free to reach out."}, {"review_id": "07LTZxNTZB", "reviewer": "Reviewer_KdU6", "summary": "This paper thoroughly investigates the effect of RLVR on the reasoning ability of LLMs, measured by a pass@k metric under different sampling budget k. \nThey find that RLVR actually does not improve the reasoning performance upper bound of LLMs, but instead make the LLM sample promising CoTs with higher probability,at the cost of a more restricted sampling distribution and correspondingly lower performance than the baseline LLMs that sample from a wider distribution when given a larger sampling budget. \nThese findings provide a critical reflection on people's common belief that RLVR can improve the reasoning ability of LLMs, which are very straightforward to understand and insightful.", "questions": "1. The models evaluated in this paper mainly have ~7B parameters, with a maximal evaluated model size of 32B. I think an important question here is if the conclusion drawn from the paper still holds when we scale up the model size? Some evaluation results on larger models like o1 and DeepSeek-R1-Zero will be quite helpful here. I understand that this may introduce significantly higher computational cost, so some priliminary results would be great enough, otherwise this should be clearly stated as a limitation of this paper due to computational constraints and call for future work. \nMoreover, this paper gives a great discussion on the limitations of existing RLVR methods and the reasons, but a discussion on the limitations of this paper itself seems to be missed. \n2. Another dimension of variability that I'm interested about is how the gap in the pass@k curves changes w.r.t. the difficulty of the evaluation tasks. Intuitively, I think more challenging task will reduce the gap between base model and RLVR model when k is large, as it's harder to sample the correct CoT from a prior distribution. I'm an RL instead of LLM researcher, so I'm not sure if there are some clear order of difficulty in the evaluation tasks used in the paper. If yes, I think some analysis on how task difficulty influences the results here would make this paper even more valuable to the community. \n3. If I understand correctly, Most pass@k curves reported in the paper are the results on some hold-out test sets that are not used for RLVR training right? If so, how would the pass@k curves look like on the training tasks?\n4. The Perplexity analysis in Figure 6 is very interesting. But it only shows the perplexity score at the end of RLVR training. I would suggest to further add a figure to show how the perplexity score evolves during the RLVR training process to better support your analysis. E.g., based on your analysis, I would expect that $PPL_{RL}(Y_{base})$ to start from a uniform distribution and gradually converge to two clusters after training, while $PPL_{base}(Y_{RL})$ gradually have lower scores during training. Furthermore,  the very high score of $PPL_{base}(Y_{GT})$ indicates that a larger LLM like o1 may behave very differently from smaller ones, which again motivate my first point in this section. \n\nOverall, I'm happy to further raise my rating if the authors can help answer or address these questions, especially the first point on scaling with model size.", "limitations": "I don't find a clear discussion of the limitations of this paper in the text, and would suggest the authors to add it, e.g., the first point I mention in the question section.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. Clear and insightful motivation, which leads to conclusions that could be very beneficial for the RLVR community; \n2. The experiments are designed in a very logical way that start from some surprising benchmark results questioning some common belief before, then give detailed analysis on the potential reasons of the issue with further experiments that well support these hypothesis. \n3. The paper is well written and structured, with clear definitions, thorough experimental evaluation and details. \n\nWeakness: \nSee my questions in the next section.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns during rebuttal with supportive experiments and analysis. So I'm happy to further raise my score.", "rebuttal": "# Response \n\nThanks for supporting the acceptance of our work. Please find our responses to your questions below.\n\n### `Q1: Does the conclusion drawn from the paper still hold when we scale up the model size? some priliminary results would be great enough`\n\nThank you for the insightful question. We also care deeply about whether scaling up model size affects our conclusions.\n**After careful investigation, we found that only three models over 70B parameters have both base and RLVR-trained versions publicly available**: DeepSeek-R1-Zero, Tulu-3-70B [1], and Magistral [2]. We summarize them in the table below.\nFor many other large models, isolating the effect of RLVR is not feasible. For example:\n\n- **GPT-o1**: The base model is not accessible, and training details are unclear; the final model may not rely solely on RLVR.\n- **Qwen3-235B**: The reasoning model undergoes multiple stages, including RLVR and long-CoT SFT, making it impossible to isolate the effect of RLVR alone.\n- **LLaMA-3-70B**: no corresponding RLVR version exists.\n\n| Model Pair   | Size    | API Access | Weights Access |\n| - | ---| -- | -- |\n| DeepSeek-V3-Base / R1-Zero | 671B    | ✗          | ✓                 |\n| Tulu-3-SFT / Tulu-3-RLVR  | 70B     | ✗          | ✓                 |\n| Mistral-Medium-3 / Magistral-Medium       |    undisclosed*    | ✓          | ✗                 |\n\n*: **undisclosed size but reasoning performance close to DeepSeek-R1-671B**\n\n**For Deepseek-R1-Zero**, we have done our best to make it run with limited resources (two 8xH100 GPU nodes). However, without expert engineering optimization, the throughput remains very low, around 50 tokens/s at a maximum sequence length of 32k. Runnings on AIME24 with 1024 samplings would take approximately 4 months, which makes it currently infeasible. We are actively seeking additional resources and hope to resume the process later.\n\nThus, we turned to Tulu-3-70B and Magistral.\n\n**For Tulu-3-70B**, which we hosted locally with 8 H100 GPUs, we conducted evaluations on the first 100 subsampled problems from the MATH500 benchmark due to high computational cost. The results below are consistent with our main conclusion.\n\n### **Tulu3 on Math500**\n\n| Model          | k=1  | k=4  | k=16 | k=32 |\n| -------------- | ---- | ---- | ---- | ---- |\n| Tulu-3-SFT     | 58.6% | 77.2% | 88.1% | 91.0% |\n| Tulu-3-RLVR    | 64.4% | 80.9% | 87.7% | 89.0% |\n| Δ (RLVR - SFT) | +5.8% | +3.7% | -0.4% | -2.0% |\n\n**For Magistral, one of the most advanced reasoning models whose. reasoning performance is colse to Deepseek-R1**, we queried the API using a maximum context length of 40k. Again, we observed RLVR's great improvements at low k, but little to no gain at larger k. Specifically, at k = 1, the RLVR model solves approximately 7 more problems than the base on AIME24 and 8 more on AIME25. However, as k grows, the performance gap steadily narrows. **The trend observed in this preliminary experiment further supports that our conclusion holds even for a frontier-level reasoning model.**\n\n### **Magistral on AIME24**\n| *k*  | 1      | 4      | 64     | 256   | 1024  |\n| - | ------ | - | - | - | - |\n| Mistral-Medium-3 (Base) | 53.5%  | 69.7%  | 80.0%  | 87.6% | 90.0% |\n| Magistral-Medium (RLVR)| 76.6%  | 88.8%  | 90.0%  | 93.3% | 93.3% |\n| Δ (RLVR - Base) | +23.1% | +19.1% | +10.0% | +5.7% | +3.3% |\n\n\n### **Magistral on AIME25**\n| *k*  | 1      | 4      | 64     | 256    | 1024  |\n| - | - | - | - | - | -|\n| Mistral-Medium-3 (Base) | 33.3%  | 51.5%  | 73.2%  | 81.3%  | 93.3% |\n| Magistral-Medium (RLVR) | 62.0%  | 75.8%  | 88.7%  | 92.1%  | 93.3% |\n| Δ (RLVR - Base)         | +26.7% | +24.3% | +15.5% | +10.8% | 0.0% |\n\nWe will include these results  in the revision and explicitly discuss the scalability limitation. We also appreciate your suggestions, which help us strengthen the paper. A more comprehensive evaluation on very large models is currently constrained by compute resources, but this is an important direction for future work. \n\n[1] Allen Institute for AI. \"Tulu 3: Pushing frontiers in open language model post-training.\"\n\n[2] Mistral AI. \"Magistral.\" arXiv:2506.10910.\n\n###  `Q2:  how the gap in the pass@k curves changes w.r.t. the difficulty of the evaluation tasks.`\n\nThank you for the thoughtful question. To explore this, we analyzed the MATH500 benchmark, which includes difficulty annotations (Levels 1–5) based on AoPS (Art of Problem Solving) criteria. These levels are based on  knowledge domain, solution technique, and complexity. Math500 dataset contains 43, 90, 105, 128, and 134 problems at Levels 1 to 5, respectively. Results are as follows:\n\n### **level 1**\n\n| pass@k|1 | 4 | 16 | 64 | 128 |\n| --- | --- | --- | --- | --- | --- |\n|Base| 86.4% | 96.2% | 97.9% | 99.4% | 100.0% |\n|RLVR| 93.4% | 94.7% | 95.3% | 95.3% | 95.3% |\n|Δ(Base-RLVR)| -7.0% | 1.5% | 2.6% | 4.1% | 4.7% |\n\n### **level 2**\n\n| pass@k|1 | 4 | 16 | 64 | 128 |\n| --- | --- | --- | --- | --- | --- |\n|Base| 79.4% | 93.3% | 97.2% | 98.3% | 98.9% |\n|RLVR| 91.1% | 94.7% | 96.8% | 97.8% | 97.8% |\n|Δ(Base-RLVR)| -11.7% | -1.4% | 0.4% | 0.5% | 1.1% |\n\n### **level 3**\n\n| pass@k|1 | 4 | 16 | 64 | 128 |\n| - | - | - | - | - | - |\n|Base| 74.9% | 93.2% | 98.1% | 99.9% | 100.0% |\n|RLVR| 89.4% | 96.0% | 98.6% | 99.0% | 99.0% |\n|Δ(Base-RLVR)| -14.5% | -2.8% | -0.5% | 0.9% | 1.0% |\n\n### **level 4**\n\n| pass@k|1 | 4 | 16 | 64 | 128 |\n| - | - | - | - | - | - |\n|Base| 57.5% | 80.2% | 91.0% | 95.8% | 97.7% |\n|RLVR| 76.5% | 85.7% | 91.0% | 93.2% | 94.5% |\n|Δ(Base-RLVR)| -19.0% | -5.5% | 0.0% | 2.6% | 3.2% |\n\n### **level 5**\n\n| pass@k|1 | 4 | 16 | 64 | 128 |\n| - | - | - | - | - | - |\n|Base| 34.9% | 60.8% | 77.7% | 86.0% | 88.1% |\n|RLVR| 56.6% | 68.8% | 77.4% | 82.5% | 84.3% |\n|Δ(Base-RLVR)| -21.7% | -8.0% | 0.3% | 3.5% | 3.8% |\n\nPreliminary observations:\n\n- **At k=1, the gap between base and RLVR increases with difficulty level**. Easy problems are often solved by the base on the first try. For harder problems, it's more difficult for the base to sample the correct CoT initially, while RLVR better optimizes sample efficiency. As a result, the crossover point—where the base overtakes RLVR—shifts with difficulty: around k=2 for Level 1, and k=16 for Levels 4–5.  These aligns well with your intuition that more difficult tasks require more samples to retrieve correct CoTs from the prior for base model.\n- However, at very large k, the gap becomes less correlated with difficulty level. The gap depends on how many problems the base model can eventually solve while the RLVR model cannot, due to reduced output diversity during RLVR training. How this relates to task difficulty is still unclear and warrants further investigation.\n\nWe appreciate this suggestion and will include this analysis in the revision. \n\n###  `Q3: Most pass@k curves reported in the paper are the results on some hold-out test sets that are not used for RLVR training right? If so, how would the pass@k curves look like on the training tasks?`\n\nYes, the most pass@k curves are reported on hold-out test sets that were not used during RLVR training. \n\nIn Sections 4.3 and 4.4, we present pass@k curves on both training and test sets as fig.14 shows, and we find the trends to be highly consistent to the test set.\n\nTo further support this, we subsampled 100 examples from the training set for SimpleRLZoo-7B and plotted the pass@k curves. **The results on the training set mirror those on the test set**: RLVR outperforms the base model at small k, but is overtaken by the base model at larger k.\n\n### **Training set**\n\n| k | 1 | 4 | 16 | 64 | 128|\n| - | - | -| -| - | - |\n| Base  | 43.4% | 58.4% | 70.0% | 80.2% | 84.5% |\n| RLVR | 52.7% | 57.6% | 61.4 | 64.9% | 66.7% |\n| Δ (Base - RLVR) | -9.3% | 0.8%  | 8.6%  | 15.3% | 17.8% |\n\n\n### **Test set (Math500)**\n\n| k | 1 | 4 | 16 | 64 | 128|\n| -- | - | -| - | - | - |\n| Base | 61.5%  | 81.5% | 90.6% | 94.8% | 96.0% |\n| RLVR  | 78.0%  | 85.7% | 90.4% | 92.6% | 93.4% |\n| Δ (Base - RLVR) | -16.5% | -4.2% | 0.2%  | 2.2%  | 2.6%  |\n\n\n### `Q4: further add a figure to show how the perplexity score evolves during the RLVR training process`\n\nThank you for the thoughtful suggestion. To analyze how perplexity evolves over the course of RLVR training, we evaluated three RLVR checkpoints—early, middle, and final. For each checkpoint, we sampled 32 responses per problem, computed the median among 32 perplexity values, and reported the average over the first 10 problems in the table (since images are not allowed in rebuttals).\n\n**As expected, we observed that $PPL_{base}(Y_{RL})$ gradually decreases as RL training progresses**, indicating that RLVR mainly sharpens the distribution within the base model’s prior rather than expanding beyond it.\n\n| Metric | Value |\n| - | - |\n| $PPL_{base}(Y_{base})$   | 1.244 |\n| $PPL_{base}(Y_{early})$  | 1.219 |\n| $PPL_{base}(Y_{middle})$ | 1.176 |\n| $PPL_{base}(Y_{final})$  | 1.159 |\n\n**For $PPL_{RL}(Y_{base})$, we observed the emergence of two clusters over time**, consistent with your expectation. One cluster shows a slow decrease in perplexity, corresponding to base samples that remain within RLVR’s sharpened distribution. The other cluster exhibits a rapid increase in perplexity, reflecting base samples that fall outside the RLVR’s sharpened distribution.\n\n| Metric  | Cluster 1 | Cluster 2 |\n| - | --- | - |\n| \\$PPL\\_{RL-early}(Y\\_{base})\\$  | 1.195     | 1.347     |\n| \\$PPL\\_{RL-middle}(Y\\_{base})\\$ | 1.193     | 1.368     |\n| \\$PPL\\_{RL-final}(Y\\_{base})\\$  | 1.190     | 1.506     |\n\nSince this dynamic behavior is well aligned with the prediction from our conclusion, we believe this analysis further supports our claims.\n\n### `conclusion`\n\nWe hope our responses, especially regarding the first point on scaling to larger model sizes, have addressed your concerns and strengthened your confidence in the paper. If you have any further questions or suggestions, please don’t hesitate to let us know."}]

Author rebuttal:

Meta-review: The paper examines an important topic --- whether math-based RL aiming to elicit reasoning abilities improves pass@k (for high enough k) or merely pass@1. The question is intriguing, because while standard RL setups optimize pass@1 in their current formulation, pass@k is equally important in some areas (e.g., when a verifier at test time is available), it ensures sufficient exploration, and can be linked with diversity of reasoning.

The paper answers this important question negatively, i.e., pass@k does not increase, even for close to frontier models. The paper is masterfully executed, and the rebuttal essentially addresses all reviewers' concernes, as highlighted by the paper perfect score, where all reviewers strongly recommend acceptance. I am happy to do the same and strongly recommend the paper for acceptance.
Decision: Accept (oral)