Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards
Authors: Xiaoyuan Liu
Tian Liang
Zhiwei He
Jiahao Xu
Wenxuan Wang
Pinjia He
Zhaopeng Tu
Haitao Mi
Dong Yu
Venue: NeurIPS 2025 poster ()
Keywords: LLM Reasoning
RLVR
Self-verification
Primary area: deep_learning
TL;DR: We introduce a RL framework to train LLM's reasoning and self-verification ability simultaneously.

Abstract: Large Language Models (LLMs) show great promise in complex reasoning, with Reinforcement Learning with Verifiable Rewards (RLVR) being a key enhancement strategy. However, a prevalent issue is ``superficial self-reflection'', where models fail to robustly verify their own outputs. We introduce RISE (Reinforcing Reasoning with Self-Verification), a novel online RL framework designed to tackle this. RISE explicitly and simultaneously trains an LLM to improve both its problem-solving and self-verification abilities within a single, integrated RL process. The core mechanism involves leveraging verifiable rewards from an outcome verifier to provide on-the-fly feedback for both solution generation and self-verification tasks. In each iteration, the model generates solutions, then critiques its own on-policy generated solutions, with both trajectories contributing to the policy update. 
Extensive experiments on diverse mathematical reasoning benchmarks show that RISE consistently improves model's problem-solving accuracy while concurrently fostering strong self-verification skills. Our analyses highlight the advantages of online verification and the benefits of increased verification compute. Additionally, RISE models exhibit more frequent and accurate self-verification behaviors during reasoning. These advantages reinforce RISE as a flexible and effective path towards developing more robust and self-aware reasoners.

Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards
Xiaoyuan Liu Tian Liang" Zhiwei He Jiahao Xu² Wenxuan Wang'
Pinjia Helt Zhaopeng Tu
Haitao Mi Dong Yu?
Schooo of Data Science, The Chinese University of Hong Kong, Shenzhen Tencent Shanghai Jiao Tong University *Renmin University of China
Abstract
Large Language Models (LLMs) show great promise in complex reasoning, with Reinforcement Learning with Verifiable Rewards (RLVR) being key enhancement strategy. However, a prevalent issue is "superficial self-reflection" where models fail robustly verify their own outputs. We introduce RISE (Reinforcing Rea- soning with Self- Seff-Verrficationn) a novel online RL framework designed to tackle this. RISE explicitly and simultaneously trains L.M to improve both its problem- solving and self-verification abilities within a single, integrated RI, process. The core mechanism involves leveraging verifiable rewards from an outcome verifier to provide on-the-fly feedback for both solution generation and self-verification tasks. In each iteration, the model generates solutions, then critiques its own on- policy generated solutions, with both trajectories contributing to the policy update. Extensive experiments on diverse mathematical reasoning benchmarks show that RISE consistently improves model's fostering strong self- celf-verification skills. Our analyses highlight the advantages of online verification and the benefits of increased verification compute. Additionally, RISE models exhibit more frequent and accurate self-verification behaviors during reasoning. These advantages reinforce RISE as flexible and effective path towards developing more robust and self-aware reasoners.
accuracy while concurrently
Introduction
Large Language Models (LLMs) have demonstrated remarkable potential in complex reasoning tasks. A promising avenue for further enhancing these capabilities Reinforcement Learning (RL), particularly methods that utilize verifiable rewards (RLVR) from outcome verifiers [Gao et al.) 2024 Guo et al.) [2055] Lambert et al.) 2024 Yue et al., (025). This paradigm, often applied to domains like mathematics where solution correctness can be programmatically evaluated, enabling models to improve through direct feedback on their generated solutions.
However, even with outcome-based RLVR, models may still learn generate spurious reasoning without truly understanding the underlying logical process or developing robust self-assessment skills. This can lead to "superficial self-reflection" [Liu al. 2025), where models struggle to reliably identify errors in their own reasoning and verify the correctness of their outputs, ultimately resulting in flawed solutions and suboptimal performance. While some approaches explicitly incorporate self-critique [Xi et al. 2024 Xie et al. 2025) to provide additional signals, the process of learning to solve problems and learning verify solutions often decoupled or lack direct, contemporaneous feedback for the verification skill itself within the RL loop.
Work done when Xiaoyuan and Zhiwei interning Tencent. 1 Correspondence to: Pinjia He chepinjia@cuhk, wwu.iiiauuuuku..ccc.. edu.ce. and Zhaopeng Tu tentueeeennntc.m....
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To address this limitation and foster more robust reasoning, we introduce RISE (Reinforcing Rea- soning with Self- Verification) as novel online reinforcement learning framework. RISE is designed to explicitly and simultaneously train an L.M to improve both its mooblem-ollving ability and its capacity to verify its own generated solutions within a single, integrated RI. process. The key idea is to leverage the verifiable reward signal from a rule-based outcome verifier not only to guide the generation of correct solutions but also to align the model's self-verification ability on-the-fly.
In the RISE framework, during each training iteration, the model first generates solutions for a batch of problems. Subsequently, using these on-policy generated solutions and the original problems, verification problems are constructed with predefined template, prompting the model to critique its own solution and provide score. The same outcome verifier used to assess problem solutions also provides ground-truth supervision for the verification task, based on an exact match between the predicted verification score and the ground-truth solution score. Both the problem-solving trajectories and the self- seff-veriiicatio trajectories, along with their respective verifiable rewards, are then combined to update the model's 8 parameters using unified RI. objective. This tight coupling enables the model to learn not only to solve problems, but also to critique and verify its own outputs, fostering more dynamic and grounded self-improvement loop.
In our experiments, we implement and evaluate RISE using the Proximal Policy Optimization (PPO) algorithm, applying to the 1.5B, 3B, and 7B base models from the Qwen2.5 series. Compared to a Zero-RL RL baseline, which incorporates only problem-solving supervision, RISE consistently improves reasoning accuracy and achieves up to 2.8x increase verification accuracy on challenging mathematical benchmarks. Moreover, RISE outperforms instruction-tuned models across both tasks. For instance, RISE-3B improves the reasoning accuracy by 3.7 points and self-verification accuracy by 33.4 points compared with Qwen2.5-3BB Instruut
We also find that this enhanced self-verification ability contributes improved test-time performance. Specifically, RISE-3B and RISE-7B outperform standard majority voting by +0.2% and + +1.9%, respectively, under a k= inference budget. Further analysis reveals that RISE enhances the internal reasoning process by encouraging more frequent and effective verification behaviors. Finally, our ablations demonstrate that online verification is crucial to the success of RISE.
Our main contributions are as follows:
We introduce RISE (Reinforeing Reasoning with Seff-Verificationn) novel online reinforcement learning framework that explicitly and simultancously trains LLMs to improve both problem- solving and celf-verification capabilities within single, integrated RL process, leveraging verifi- able rewards for both tasks on-the-fly.
We demonstrate, through extensive experiments on challenging mathematical reasoning bench- marks using PPO-based implementation, that RISE significantly boosts problem-solving perfor- mance while instilling robust self- velfiveriiication skills the LLM.
We provide comprehensive analyses elucidating the critical role of RISE's online verification mechanism, the benefits of scaling verification training compute, and how the developed self- verification capability contributes to more accurate and reliable solution generation.
2 Related Work
RLVR for LLM Reasoning In the literature, reinforcement learning has been widely used to align
language models with human preferences, typically through reward models or pairwise preference comparisons [Christiano et al. 2017 Ouyang et al. 2022. Rafailov al.) 2023). More Recently, Reinforcement Learning with Verifiable Rewards (RLVR) R)) has emerged as powerful approach for im- proving the reasoning capabilities of LLMs in domains such as mathematics and programming [Jaech] et al., 2024 Guo et al. 2025]. Using only outcome rewards, recent work has demonstrated the scalability of RL algorithms for LLM reasoning [Guo et al., 2025 Team al., 2025| Zeng et al.. 2025 Hu et al. [025]. However, leveraging verifiable rewards not only for reasoning supervision but also as direct training signal for velf-veriiication remains underexplored, which the main focus of
RISE.
Learning to Solve and Verify Solution generation and verification are two foundational capabilities of LLMs HHung al. 2024 Song et al.) 2024), echoing the classic P versus NP dichotomy in

---

computer science [Wikipedia contributors 2025]. In the context of LLM reasoning, previous work has focused on teaching models either to solve problems [uo al. 2025 Zelikman al. 2022], to
verify solutions (Wang et al. 2024 Lightman et al. 2023 Shi and Jin 2025 Zhang et al. 2024 or
to leverage the verification capability to perform self-improvement Yuan et al. 2024] Xiong et al.
2025) and self-calliration [Huang et al. 2025]. More recently, Lin et al. [2055 introduced a self-play framework that trains LLMs to generate code and corresponding test cases through two-stage training, and Ma et al. [2025] proposed training methods that teach LLMs
velf-v and self-correct contrast, we introduce an online RL framework that explicitly leverages verifiable reward signals to jointly align the model's problem-solving and self-verification abilities in a unified training process.
based on deliberately constructed trajectories.
Reinforcement Learning Preliminaries
Policy Gradient Methods The goal RL to learn policy that maximizes the expected cumulative reward (namely return), denoted as the performance measure J, Policy gradient methods learn parameterized policy that can select actions maximize without consulting other value functions. Grounded by the policy gradient theorem [Sutton and Barto 2018), the optimization is performed as gradient ascent based on the gradient of J(0) with respect to the policy parameter 0.
large language model is naturally a parametrized policy Tg. The state at time i, denoted as 8t, is the concatenation the prompt and the response y< generated so far, while the action o, is the next token y/. 7 refers to total timestamps (response length + 1). Thus, the gradient can be expressed as:
T
VoJ(O) = -..WDD yIuz
Σ
vo log To( x,y A
The core part of this method is the advantage function A,, which determines the extent to increase decrease the probability of selecting this action (token) in the given state. In practice, the advantage function is implemented as cumulative discounted rewards subtracting an optional baseline, representing how much better an action is compared to the alternatives:
A 7 to b(st))
where E [0, 1] is the discount factor for the future rewards and r, = R(*E, Gt 8841) is the re- ward from the environment at time Different implementations of the bascline formulate mul- tiple variants of policy gradient methods, including using learned state-value functions (c.g., RE- INFORCE (Willimms 1992]. Actor-Critic [Barto et al., 1983)), group-level reward means (c.g.. GRPO
Shao et al. 2024)), and leave-one-out (e.g. RLOO [Ahmadian et al. 2024
Proximal Policy Optimization Proximal Policy Optimization (PPO) [Schulman et al., 2017] is popular algorithm of Actor-Critic method, which incorporates critic model ø to help estimate advantage for training the actor model 0 (i.e., policy). One major improvement of PPO is penalizing excessive policy updates and thereby maintaining training stability. In practice, the objective of the actor model is defined as follows:
[(0) = E, min (r,()) clip (,00),,1-,,,11
- 3KL (Te (Tel(TTe()
(2)
where (+) To (ve (.))))
uu [. ClipO and KL() are two techniques used for limiting update magnitudes.
With Generalized Advantage Estimation (GAE) [chullan al. 2015), the advantage is estimated
as A-weighted sum of step-emporal Samporl-Diiffrrenc (TD) errors:
A, = 6, (7A)5,+1
(TA)T-1- 87-1:
where 8, = r. + VV Vo(ss))
T denotes response length with token indexes from 0 to T 1. Vo(s)) is the value predicted by the
critic model & at state 8t, rt is the scalar reward from the environment at time 4, and E [0, 1] is
the GAE parameter that trades off between bias and variance. In practice, set X 7 1, thus
3

---

Problem Solution Verilication Generation
AL Optimization
Training Data
Batch Data
Model
Generate CoTs
Compute Rewards
Compute Advantage
Train
B gev
Problem Solations
13-y11
Solation Rewards
Af
(xx,))
Venification Prompt Tamplate 16
Orowng
N
A6
A4
PFO)
Ag.
Solutiin Vertfications
Venfication Rewards
Alaa
AB
Abaxe
varther
(G,,30))
10..37.331
upoite
Figure 1: Illustration of RISE, which consists of two stages: (i) Problem Solution & Verification Generation: problems from the training batch are used to generate maainoff-though solutions from the model. Problems and model solutions are then formatted as verification prompts to generate verifications of the solutions. (ii) RL Optimization: the original generation data and their verification mixed as the new batch, and the model is optimized based on the RL objective.
making the per-token loss averaged over the full response length T. By design, r, for < T and r; for = T 1 (i.e., outcome reward). After we update the actor model, the critic model should also be updated for accurate value estimations. practice, we use Mean Squared Error (MSE) to measure the prediction loss and perform the update:
Et max (Vo((s)--V V/ (clip(Va Vol (8c) - e, Vinaa (8x) +))- -V, Vitars) }){
I (0)
where Viarg = Vouu (8)) + Å
Verifiable Reward Unlike the rewards from conventional reward models which are continuous values denoting the goodness of the response, verifiable rewards are usually discrete signals repre- senting the correctness of the final result (Lambert al.) 2024 Guo al.) [023). Given the prompt x and the complete response from the LLM тT, a classic verifiable reward is defined as binary value produced by a deterministic outcome verifier OV: r OV (x,,, y) E {0, 1), where
if and only if the final answer is exactly correct (e.g., the numeric result is mathematically equivalent to the ground truth answer) and r 0 otherwise. In practice, an auxiliary format reward be included to encourage the model to present its answer in prescribed style.
4 Methodology: Reinforcing Reasoning with Self- Verification (RISE)
To address the challenge of superficial self-refiection, we propose RISE for self-improving reasoners, which is a scalable online RL method with explicit verification objective. The key idea of RISE is the use of the verifiable reward signal from the rule-based outcome verifier align the model's verification ability on-the-fly. This enables us to teach the model to verify its own response at the same time solves the problem, as depicted in Figure and Algorithm
4.1 Online Reasoning and Verification
Problem Solution Generation Given an initial model Ta and a training set D = {(x y: } consisting of problems x,, and their corresponding ground-truth answers y we begin each RL iteration by sampling a data batch. At iteration t, the model first generates solutions for each problem in the batch, each comprising chain-of-thought reasoning followed by final answer.
Next, the reward is computed for each generated response. Following prior RLVR approaches, we define mle-bbsed outcome verifier (OV) that incorporates both answer and format correctness:
1, 0.5, boxed but not matched
boxed and matched
To =
(5)
unboxed
Here "matched" means the final answer in the generated solution y mathematically identical the provided ground truth y", and "boxed" means the final answer in y is wrapped in the \boxed().

---

Algorithm RISE (PPO)
Input Language model : outcome verifier OV; dataset D; rollout number K; generation batch size y verification batch size B,: verification prompt template T; total iteration N.
1: Initialize: actor Th f- Thm old-actor Thuy critic TΦ. reference Tref
2: for iteration = to N do
3: Sample B, samples for generation Ps = [xx, y.) yi) ~
4: Get generation batch:
Generate solutions
+ y Fov (y: (k)
.Y:))
y (-())x;) is k B.
(k)
5: Select B, triples P ((x
r.)
C g for verification
=
6: P. < {T(( y), r) (x, E P')
I/ each element a new prob-ans tuple y") D Verify generations
7:
Get verification batch:
V y( Fo (y 33,
x -x kSK]
8:
Get complete training batch + GUV
9: Estimate advantages Å using Eq.
Joint optimization
10: Update critic Ts by critie loss in Eq.
11: Update actor my by actor loss in Eq.
update oold <- 0
12:
end for
Output Optimized actor model n
This produces the generation batch g yx r), where each element includes the input problem, model-generated solution, and its associated reward.
Online Solution Verification To construct verification data, we apply a predefined prompt template verification prompt X that
(see Figure { to G, formatting the problem-solution pair into explicitly states the verification criteria and asks the model to critique the provided solution and assign score. Since the criteria specified in the prompt are exactly the rules employed by the outcome verifier, the original reward from the generation phase is reused as the ground-truth score for the verification task. Thus, for each triple (.) E G. we construct the verification data as (Xur = T (xx.)) y), Yver = r). In practice, the amount of verification data is controlled by the verification batch size, allowing a flexible balance between groblem-solving and solution verification.
For each verification prompt, the model generates K responses, each containing a natural language critique and final score. These responses evaluated using the same OV criteria Eq. (5). Concretely, the verification reward determined by comparing the score extracted from the model's verification response with the reward assigned by the rule-based outcome verifier for the same judged solution. A reward of +1.0 is given when the model correctly predicts the score in the specified format (e.g., \boxeed((1)) reward of -0.5 assigned when the predicted score is incorrect but the format is valid; and reward of - 1.0 is applied when the response is in an invalid format (missing the (boedd[] wrapper). This process yields the verification batch V {(x,y) y, r)}. maintaining the same structure as the problem-solution batch.
4.2 RL Integration
The preceding Online Reasoning and Verification stage is architecturally agnostic to the choice of the underlying aolicy-gradien algorithm; its only algorithm-specin interface the advantage estimator Á used the policy update. our formulation, advantage values are computed from a concatenated mini-batch B G V. encompassing samples from both the reasoning and verification tasks. Since every sample in annotated with a scalar reward and the action log- prgoobbbbil under the current policy Tg. any estimator that maps sequence of state-action-reward tuples to an advantage can be incorporated without further structural change.
For our main experiments with PPO (see Algorithm []. we apply GAE (Eq. 3) independently to each trajectory. The generation and verification trajectories are jointly processed within the same stochastic gradient descent (SGD) step, enabling the actor to be optimized with respect to both types of data. Meanwhile, the shared critic learns unified value function across tasks. PPO's 's clipping mechanism further ensures that updates remain stable within a consistent trust region.
5

---

Table 1: Detailed results of RISE and other baseline methods various math benchmarks. Zero-RL models are trained under the same setting as RISE, but without the verification objective.
Reasoning
Self- Delf-Verification
Model
MATH
AIME
AMC
Mine.
Olym.
Avg.
MATH
AIME
AMC
Mine.
Olym.
Avg.
GPT-4o
79.0
13.3
55.0
50.0
42.5
48.0
83.4
33.3
67.5
50.4
54.4
57.8
Qwen2.5-1.5B
Base
2.0
0.0
1.9
0.8
0.6
1.1
19.4
21.9
22.7
15.9
21.1
20.2
Instruct
37.5
0.8
19.4
8.3
11.7
15.5
48.8
22.1
36.5
36.9
29.6
34.8
SFT
10.1
0.0
4.1
1.8
2.0
3.6
19.0
5.8
12.3
10.5
10.9
11.7
Zero-RL
55.3
2.1
25.9
17.4
19.5
24.0
54.1
5.0
30.7
21.0
23.0
26.8
RISE
54.6
2.9
27.5
17.2
19.8
24.4
75.9
85.0
70.6
66.0
74.9
74.5
Qwen2.5-3B
Base
32.7
1.3
15.3
10.3
10.7
14.1
39.5
13.6
22.5
29.9
21.2
25.3
Instruct
61.0
3.8
34.1
25.6
24.6
29.8
65.6
21.0
45.5
37.6
35.0
40.9
SFT
14.4
0.4
5.3
2.9
2.8
5.2
21.5
2.1
10.9
17.9
13.2
13.1
Zero-RL
64.2
6.7
37.5
27.4
26.6
32.5
64.9
13.0
39.7
30.3
31.2
35.8
RISE
64.3
7.9
42.5
26.2
26.6
33.5
81.0
86.3
74.4
56.1
73.6
74.3
Qwen2.5-7B
Base
38.3
2.1
21.9
11.9
13.2
17.5
58.4
45.9
51.5
48.4
48.4
50.5
Instruct
73.8
10.0
50.6
35.9
35.8
41.2
77.2
26.3
57.0
40.2
45.2
49.2
SFT
28.7
0.8
13.8
6.2
7.2
11.3
40.5
36.6
47.4
39.2
36.1
40.0
Zero-RL.
74.5
12.1
51.3
34.2
36.7
41.7
75.9
21.7
56.5
37.3
41.6
46.6
RISE
74.8
12.5
55.9
34.6
36.7
42.9
83.8
75.0
72.5
48.6
65.9
69.2
Experiment
5.1 Experiment Setup
Dataset We follow the previous study (Zng et al. 2025 to utilize MATH-Hard (Level 3-5) [Hendrycks al.] [202T] our training set, which total comprising 8,523 problems. This training set is used for all SFT baselines, Zero-RI. baselines, and RISE models.
main experiments three Qwen2.5 models Yang et al. 2024 with different sizes (i.e., 1.5B, 3B, and 7B) for their strong reasoning capabilities. The RL training of our models is based the verl (Sheng et al. 2025 framework with train batch size of 1024 and mini-batch size of 128. We follow [Zng et al. 2025) by setting the sampling temperature to 1.0 and rollout 8 responses for each problem. The RISE models have default verification batch size 128. We set the RL configurations same for RISE models and Lero-RL models, ensuring a fair comparison. Additionally, we include the RL experiments on Qwen3 models in Appendix H..l where we observe larger performance gains, further demonstrating the effectiveness of RISE on newer models.
Models We conduct
Benchmarks We evaluate model performance standard mathematical reasoning benchmarks: MATH500 [Hendrycks et al. 2021 Lightman et al. 2023), Minerva Math [Lewkowycz et al.) 2022]
OlympiadBench [He al. [2024]. and competition-lewe benchmarks AIME 2024 and AMC 2023.
Following Zeng et al. 2025). we generate responses per problem using a sampling temperature of 1.0, and report Pass accuracy [Chen al. (221]) the evaluation metric. Solution correctness
is based on exact match the final answer, and verification correctness depends on the agreement between the predicted verification score and the score from the outcome verifier.
5.2 Experimental Results
Table presents the results of RISE across model sizes and benchmarks.
RISE significantly enhances veff-verification capabilities while improving reasoning perfor- mance. RISE models consistently outperform their Zero-RI. counterparts across both reasoning and self-verification tasks on all model sizes. The improvement in self-verification is particularly dra-

---

matic: RISE-1.5B achieves 74.5% average verification accuracy compared to just 26.8% for Zero-RL, representing a 47.7 percentage point improvement. This demonstrates that our integrated approach successfully develops robust self-verificatios skills while simultaneously enhancing problem-solving capabilities. Notably, the verification improvements are particularly pronounced on the most challenging benchmarks like AIME24 and OlympiadBench, suggesting that RISE enables models to better recognize their limitations and errors on difficult problems.
Scaling model size improves reasoning performance while maintaining strong verification capabilities. Scaling model size from 1.5B to 7B parameters consistently enhances reasoning performance across all benchmarks. Interestingly, the verification performance of RISE models remains consistently high across model sizes, with models achieving over 69% average accuracy. The ability to maintain strong verification capabilities while scaling reasoning performance aligns with our contribution of developing a framework that simultaneously improves both critical capabilities.
RISE models outperform standard SFT and base models by substantial margin. The results clearly demonstrate that RISE models substantially outperform their SFT and base model counterparts. For instance, RISE-7B achieves 42.9% average reasoning accuracy compared just 1 1.3% for SFT- 7B and 17.5% for the base model. This substantial improvement demonstrates the effectiveness of our integrated, online learning approach built upon state-of-the-art RLVR paradigms.
5.3 Test-Time Scaling with Self-verification
K=
K 8
50
50
Zero-RL, RL.
Zero-RL.
RISE
RISE
Ressoning Accurccy
RISE Veriffy
Reasoning Accuracy
RISE - Self--errriyy
40
40
30
30
31 011
Qwen-1,5B
Qwen-33
Qwen-7B
Qwen-1,5B
Qwen-33
Qwen-7BB
(a) k=4
(b) k=8
Figure 2: Test-time scaling performance across different sampling budgets ("k").
To further evaluate the benefits of the enhanced self-verification capabilities developed by RISE, we investigate its impact test-time using self-consistency majority voting ("maj @k") [Wang et al. 2023] and verification- -eeetttee majority voting. In the latter, following [Wang et al.) 2024], the model's 's self-generated verification scores for each candidate solution used to weight its contribu- tion in the majority vote. The results, presented in Figure 2 compare RISE models against Zero-RL -LL models across different sampling budgets ("k=4" and "k=8"). With the lightweight verification cost (Appendix G.2) the experiments maintain fairness in terms of total sampling cost.
RISE consistently improves test-time scaling performance with ceff-verification and majority voting. RISE models outperform their Zero-RL counterparts when employing test-time strategies such as majority voting and criffiattion-eiighee selection. Across model sizes and sampling budgets, RISE achieves higher average accuracy, with the largest relative gains observed when self-verification scores are used re-rank majority votes. For example, RISE-7B achieves an average score 49.8% with k = 8 + self-verify, surpassing Zero-RL's 48.3% 3% under the same conditions. This consistent improvement confirms the effectiveness of integrating self selfiveeriiiatiio both training and inference.
Verification voting delivers further accuracy gains. Incorporating self- seff-verification scores as weights in the voting process leads to additional accuracy improvements for all RISE models. For instance, RISE-3B and RISE-7B models see improvements of +0.2% and +1. +1.9% over standard majority voting at the k = budget, respectively. These results indicate that the self-verification policy learned by RISE provides meaningful confidence signals for answer calibration.
7

---

5.4 Comparison with Off-the-shelff Verifiers
We further compare the verification accuracy between our RISE models self-verifiers and off-the-shelf veri- fiers, including discriminative verifier (Math-Shepherd- 7B [Wang ct al. 2024)) and generative verifier (GPT- Specifically, we use the verification prompt in Figure 8 for both RISE models and GPT-4o and adhere to the original logic for Math-Shepherd to ver- ify the generated solutions. The results of RISE- 3B and 7B are presented in Figure 3 which show that RISE models consistently outperform existing outcome verifiers in judge their solutions' correctness. This serves great advantage for the model further improve its test-time performance, by leveraging the self-verification signal either externally or internally. Detailed results and evaluation implementation can be found in Appendix
Gen. Verifier (GPT-44) Disc. Verifier (Math Shepherd)
RISE Self-Verity
Verification Accuracy
4o (0enAA][2024])
1-4.0 (1.00
RISE-1.5B
RISE-3B
RISE-7B
Figure 3: Comparisons between RISE (self-verify) and off-the-shelf verifiers.
5.5 Analysis
In this section, we provide some insights into how RISE improves performance.
RISE demonstrates robust and si- multancous learning of problem- solving and self-verification, with self-verification skills developing no- tably faster across different model scales. The learning curves, il- lustrated by the reward trends in Figure 4 reveal a consistent and steady improvement in both rea- soning solvvig and self- verification rewards throughout the RI training process for all evaluated models. This uniform positive pro- gression across varying model sizes highlights the robustness of the RISE framework in co-training these two abilities, a core contribution of our work. A key observation is that the self-verification reward generally exhibits more rapid inerease and reaches higher relative level compared the problem-solvine vinng reward within the same training period. This aligns with the "CGnnraatonn-Vrrffiattooo Gap" posited by [Song al. 2024]. suggesting that models might acquire verification capabilities more readily than problem-solving abilities.
RISE-1.5B
RISE-3B
Reasoning
Verification
0.5
Reward
-0.5
-0.5
Training Step
Training Stop
Figure 4: Reasoning and verification reward at train time.
Impact of Verification Compute trained
In the main experiment, our RISE models with a verification
417
Verificationn Accuracy
batch size of 128, which is 12.5% of the generation batch 1024. We fur- ther explore the model performance
Reasoning Accuracy
70
35
103
50
by scaling up the verification data batch, i.c., the train-time compute, up to 100% of the generation batch. In practice, we choose the percentages of {0, 12.5%, 23%, 50%, 100%) and perform experiment on our RISE mod- els. The results are shown in Fig- ure 5 The problem-solving perfor- mance first increases, and then slightly decreases, and finally increases across the benchmark, maintaining at high level. Furthermore, the verification performance keeps scaling with more training compute, indicating the robustness of scalability of our RISE method.
25
23.
30
RISE-10
ROSE-38
RISE 38
15
ROSE-1.58
158
10
RIST-1.9
0
25
50
75
100
25
50
75
100
Verification Data Portion (%)
Verification Data Partion
Figure 5: Impact of verification data ratio.

---

Online and Offline Verification We validate the effectiveness of online verification by com- paring it to offline variant, where the verifica- tion data are collected from a distant policy and directly added to the training set. In practice, we select the policy at step 96 (final step) of the Zero-RL model and use its generated solutions to construct offline verification set. In the exper- iment, keep the portion of verification data and the training batch size same to eliminate other influence factors, making the only chang- ing variable the source of the verification data. Figure shows the results. While the problem- solving performance of offline verification models are on par with the online ones, they have a significant drop in terms of self-ve seffvveriiiationn accuracy, which indicates the importance of online verification designed in our RISE method.
50
80
Offine
Online
Ofline
Online
Reasoning Accuracy
Verification Accuracy
20.7
30
40
1.5B
3B
7B
1.5B
3B
7B
Figure 6: Comparisons between online and offline verification.
Enhanced Verification for Reasoning Be-
sides leveraging the self-verification ability ex- ternally during the test-time as in $ [.3] such
10
40
Zonoiti
RISE
Zuno-Ri
RISE
ability is also internalized by the model to en- hance its reasoning generation process. To anal- ysis this effect from the quantitative perspec- tive, measure the Verification Frequency and Self-Veriffee Verified Reasoning Accuracy in model's problem-solving process. Inspired by YYoo ct al., 2025], we use a set verification key- words select the responses containing self- verification behaviors, namely ("verifyy" "ver- ifying", "recheck", "validate", "re-evaluate").
Vorification Frequency
8
Reasoning Accuracy
30
26.7
20
2
0
10
1.5B
Figure 7: Verification frequency (left panel) and its effectiveness in reasoning (right panel).
Figure 7 (left) presents the first result, where the proportion of responses containing explicit verification phrases consistently rises after RISE training. The increase is modest for the 1.5 B model (+0.05 absolute) but becomes more pronounced as model scale grows (+1.09 for B and +1.05 for B). Since the decoding syper-parameters higher verification frequency arises from policy updates that jointly optimize for both solution and verification rewards, effect enabled by our first contribution, which integrates verification into unified RL training loop rather than treating as a post-hoc heuristic. This mechanism allows models to check their intermediate solutions before reaching the final conclusion in more
fixed, the
Table 2: Examples of verification behaviors of different methods on Qwen2.5-7B: RISE can incur more effectiveness self-verification the reasoning.
Problem
What the least positive integer multiple of 30 that can be written with only the digits 0 and 2?
Zero-RI.
To find the least positive integer multiple of 30 that can be written with only the digits 0 and 2, need to consider the properties of multiples of 30... Since 222 divisible by both 2 and 3, a multiple of 30. Let's verify: 222 is even, so divisible by The sum of the digits of 222 2 2 6, which is divisible by 3. Thus, 222 is indeed multiple of 30. Therefore, the least positive integer multiple of 30 that can be written with only the digits 0 and is 222
RISE
To find the least positive integer multiple of 30 that can be written with only the digits and 2, we need ... Therefore, the smallest number N is 2220. To verify: 2220 ends in O, so is multiple of 10. The sum of the digits of 2220 is 2 2 2 + = 6, which is a multiple of 3. Since 2220 satisfies both conditions,
the least positive integer multiple of 30 that can be written with only the
digits and 2. Thus, the final answer is 2220

---

Simply asking the model to "verify" is not sufficient; the verification needs to be correct. In Figure 7 (right), we measure the accuracy of problems on which the model verifieri RISE yields clear improvements over RI. for self-verified reasoning accuracy at every scale: +3.3% (1.5 B), +1.7% (3 B), and a striking +6.5% (7 B). These gains show that RISE's online verifier reward shapes the policy toward not only producing more verifications, but also ones that align with ground truth.
The case Table iillustraees this distinction. Zero-RL "verifies" 222 by merely restating divisibility rules, overlooking the necessity of a trailing zero for multiples of 30. RISE, in contrast, recomputes both the units-digit constraint and the digit-sum divisibility test, ultimately validating the answer of 2220. Such structured. multi-step verification reflects genuinely internalized skill and explains the quantitative trend that higher verification frequency correlates with higher reasoning accuracy.
6 Conclusion
In this work, introduced RISE, novel online reinforcement learning framework that integrates problem-solving with explicit self-verification training for LLMs. By leveraging verifiable rewards for both generation and verification tasks within a unified RL objective, RISE aims overcome superficial self-reflection and foster more robust reasoning capabilities. Our experiments, primarily using PPO with Qwen2.5 models on diverse mathematical reasoning benchmarks, demonstrate that RISE significantly improves problem-sol verification skills. Notably, RISE models learn to verify their own solutions more effectively than off-the-shelf verifiers and further benefit from leveraging this capability at test time. Together, RISE provides a promising direction for building more reliable and self-aware LLM reasoners, adaptable to various policy-
accuracy while concurrently developing strong self-
policy-gradient gradient algorithms and extendable to other domains with verifiable rewards.
Acknowledgment
This paper was supported by the Guangdong Basic and Applied Basic Research Foundation (No. 2024A1515010145 and the Shenzhen Science and Technology Program (Shenzhen Key Laboratory Grant No. DDSYS200236226011000000
References
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace. Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker, Back to basies: Revisiting reinforce-style optimization for learning from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 12248-12267, 2024.
Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics. SMC-13(5):834 846, 1983, doi: 11899TTSM..188333311111
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet. Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021. URL http::tttt////tt wwwpexivvooor/aeo absp///a///////
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017.
10

---

Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang. and Yi Wu. On designing effective rl reward at training time for Ilm reasoning. arXiv preprint OrXiv:2410.55111 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprins arXiv:2501. 2025.
Chaoqun He, Renjie Luo. Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with dlympiad-evee bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2228-8850, 2024.
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k. A large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprins arXiv:2504. /4456, 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0333 2021.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Shum. Open-reasoneet-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.
2025.
Audrey Huang, Adam Block, Dylan Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jor- dan T Ash, and Akshay Krishnamurthy, Self-improvement in language models: The sharpening mechanism. arXiv preprint arXiv:2412.01955 2024.
Chengsong Huang, Langlin Huang. Jixuan Leng. Jiacheng Liu, and Jiaxin Huang. Efficient test-time scaling via self-calibration. arXiv preprint arXiv:2503.00m0 00031, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameyy, et al. Openai ol system card. arXiv preprint arXiv:2412.167200 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposhum on operating systems principles, pages 61 1-626, 2023.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint 24Xiv:241..15124 2024.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Zi Lin, Sheng Shen, Jingbo Shang, Jason Weston, and Yixin Nie. Learning to solve and verify: A self-play framework for code and test generation. arXiv preprint arXiv:2502. 14948. 2025.
Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang, Chao Du, and Min Lin. There may not be aha moment rl rl-zero-like like training - a pilot study. https://oatImm.no notiinooomnnoon taatee/at-zen 2025. Notion Blog.

---

Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S²r: Teaching Ilms to self-verify and self-correct via reinforcement learning. arXiv preprint arXiv:2502. 12853, 2025.
OpenAl. Gpt-4o, 2024. URL attp:s//opeaal.cco//nneeeeellllp 09-18.
Accessed: 2025-
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 11553728-53744- 2023.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprini arXiv: 1XXiv:5506.02335 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279-1297,202
Wenlei Shi and Xing Jin. Heimdall: test-time scaling on the generative verification. arXiv preprint arXiv:2504. 10337, 2025.
Yuda Song, Hanlin Zhang, Carson Eisenach, Sham Kakade, Dean Foster, and Udaya Ghai. Mind the gap: Examining the iell-improveeent capabilities of large language models, arXiv preprint arXiv:2412.02674 2024.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with 1lms. arXiv preprint arXiv:2501. 2025.
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Opennathinstruct-22 Accelerating ai for math with massive open-source instruction data. arXiv preprint 1000022400.01560, 2024.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lam- bert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformers Reinforcement Learning, 2020.
com/nuggingface/tzz
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9426-9439, 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreviex wwt.:/oooordddd-P.......
Wikipedia contributors. P versus np problem - Wikipedia, the free encyclopedia, 2025. URL. wtt.ps://eeee.t wikipeiia org/w/index. index.idxxxx///e wwwptpteeettteeeteeccceee wwwsus_evu... _P_ NP-_problemoolddd 1287287306 [Online; accessed 2025).
12

---

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine learning, 8(3):229-255) 1992.
Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do, Wenyu Zhan, et al. Enhancing 11m reasoning via critique models with test-time and training-times supervision. arXiv preprint arXiv:2411.16577, 2024.
Zhihui Xie, Jie Chen, Liyu Chen, Weichao Mao, Jingjing Xu, and Lingpeng Kong. Teaching language models to critique via reinforcement learning. arXiv preprint 2xXv::5522.0392,,, 2025.
Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self-rewarding correction for mathematical reasoning, arXiv preprint arXiv:2502. 19613, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 15, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 2025.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in llms. arXiv preprint arXiv:2502.03373. 2025.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning, 2024.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model?, 2025. httphttss://arttt
org/ abs/ /504. 13837
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 15:15476-15488, 2022.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 700: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.188/) 2025.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.1515 15240, 2024.
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contributions of this paper, including the proposed method, performance results, ablation studies, and further analysis, are clearly outlined in the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification: Due to space limit, we discuss the limitations of this work and future work in detail in Appendix [
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
This is an empirical paper which does not include thcorctical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Experiment details are provided in Section 5.1 Appendix C and Appendix D Prompt templates are provided in Appendix B
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code and documentations are provided in the supplementary materials to facilitate the reproduction of experiment results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.cc/pullcc/guut hww.b//iccguiddess/ddeeuuubbib
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experiment setting and details provided in Section 5.1 Appendix C and Prompt templates are provided in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Following common practice in the RL post-training literature, we do not report error bars due the high computational cost of running multiple trials. Additionally, follow established conventions by reporting the pass ratio, which inherently averages over multiple generations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
17

---

Justification:
We provide a detailed description of the experimental environment in Ap-
pendix F
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://p///aurppo..ccooooo//hhaa
Answer: [Yes]
Justification: We have ensured that the research conducted in our paper complies with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The paper does not identify any immediate direct societal risks or benefits, as the proposed method primarily introduces a training framework for reasoning tasks such mathematics. We emphasize that future works building upon method should be developed and applied responsibly.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
18

---

11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The work in this paper does not involve such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: License details are provided Appendix E We have fully complied with the licensing terms and usage policies for all data, code, and models used in our experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: For the new assets introduced in this paper, detailed documentation provided alongside the assets in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
19

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The LLM is used only for writing and does not impact the core methodology, scientific rigorousness, or originality of the research in this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy for what should or should not be described.
20

---

A Limitations
On experiments: First, after training the LLM to self-verify its outputs, it may act as pseudo-rule- based verifier further guide RL training without relying on ground-truth labels. This opens the possibilityy of self-im celf-fimrovement the main scope of this study. Second, the training data is exclusively drawn from math reasoning tasks. Given the observed generalization performance, we expect the effectiveness of RISE to persist when trained on other domains with verifiable rewards. Third, the metrics adopted for evaluating verification effectiveness have certain limitations, as they do not capture the necessity of the emerged verification behaviors. In this work, we use final reasoning accuracy as an indicator of goodness, while future work could develop more fine-grained criteria that better quantify the necessity such as partial trajectory correctness or model uncertainty.
unlabeled data, which we leave for future work as it lies beyond
On algorithm: RISE trains the LLM as generative verifier that produces natural language critiques, which has shown benefits for both reasoning and test-time scaling. An alternative design co-train a discriminative verifier with a separate classification head. While it remains unclear how RISE would perform in that setting, we believe this does not affect our main contributions which demonstrate the effectiveness of generative verification in improving groblem-solving capabilities.
B
Prompt Templates
Prompt Template
Below you presented with question and a tentative response. Your task is evaluate and assign rating to the response hased on the following clear criteria:
Rating Criteria:
Missing final answer enclosed in Whoxed() the end: assign Whoxed(-1). Correct response with the final answer enclosed in Wboxed(] at the end: assign Mboxed 1). 3. Incorrect response with the final answer enclosed in Wboxed at the end: assign Wboxed(- 0.5).
### Question Begin ### [Question] ### Question End ###
### Response Begin ###
[Response]
### Response End ###
Briefly summarize your analysis, then clearly state your final rating value enclosed in Wbboxed( J at the end.
Figure 8: Verification prompt used in the experiment.
Prompt Template
blim_starl>sssteer
Please reason step by step, and put your final answer within Wboxed ><<lim_endl> endl> start/>utr
Input]<lin endl>
tin_staatbaasssttan
Figure 9: Prompt template used in the training and evaluation.
21

---

Prompt Template
hii__strtbsssstmm
You are a helpful assistant.<ling <lims-liidddeeee endl>> stn__strtb>user
Input] Please reason step by step, and put your final answer within \boxed <_lim_endd>>>>
<lim_startie stasstratbassit
Figure 10: Prompt template used for Qwen base model evaluation.
C Training Details
During RL training, we set the actor's clipping ratio to 0.2 and disable the KL penalty loss. The critic uses a clipping range of 0.5. The learning rates are fixed at 10-7 for the actor and 9 x 10-6 for the critic. The KL divergence coefficient is set to 1 x 10-2, We limit the maximum response length to 3000 tokens, which already results negligible clip ratio. The full dataset is trained for 12 epochs. This configuration is shared across both the Zero-RL and RISE models.
For the SFT baseline models, we use batch size of 32 and apply a cosine learning rate scheduler with learning rate of 2 x 10 and warm-up ratio of 1 x 10 The dataset is trained for 3 epochs.
D Evaluation Details
D.1 Verification Evaluation with Other Verifiers
To evaluate the verification accuracy of RISE and GPT-4o (prompted verifier), extract the final verification score from each response and normalize it to either +1 (predicted correct) or (predicted incorrect). The normalization is defined as:
1, 8
porrmalizdd
0,
otherwise
which aligns with the criteria used by the rulc-based outcome verifier. For the Math-Shepherd model, which outputs a continuous score in the range [0, (with 0 indicating the solution/step predicted to be incorrect and 1 indicating correct), we apply a threshold of 0.5 for normalization:
1,
0.5
Snormalized
otherwise
After normalization, we compute verification accuracy by directly comparing the predicted scores against those returned by the outcome verifier.
D.2 Weighted Majority Voting with Self- Velf-Verifiiation
In 5.3 we explore the combination of self-consistency and self-verification in test time following Wang et al. 2024]. In practice, we initially classify solutions into distinct groups according to their final answers. Following that, extract and normalize the self-verification scores and normalize them as +1 (correct) and 0 (incorrect) as in D.I Since our score are binary and could lead to an unexpected zero sum, we integrate Laplace smoothing for computing the mean score for the answer. Formally. the final selected answer based on N candidate solutions is:
a Σ S(p,8:)
Gmajöd veriff-verify argmax,
Σ I(a,=al = a)
(6)
N ad
Frequency
stmoothed mean score
where S(p, 5,) the score the i-th solution assigned by the self- self-verification. In practice, we set and d 2 empirically, suggesting prior belief of 0.5 average score.
22

---

E Licenses
Datasets and Benchmarks. The training dataset is derived from MATH (MIT License). We evaluate on five benchmarks: MATH 500 (MIT License), AIME 2024 (CCO: Public Domain), AMC 2023 (Apache License 2.0), Minerva Math (license not found), and Olympiad Bench (MIT License).
Framework. RL training is based on verl v0.2 (Apache-2.0) license), and SFT training is based on v0. (Apache-2.0) license). Evaluation is performed using vllm frame- 2023 v0.7.2 (Apache-2.0 License) and the script is based on OpenMathlnstruct-
trl von Werra al. 2020]
work Kwon et al.
2 Toshniwal et al. 2024
Models. We train our models based on the Qwen2.5 series. Specifically, Owen2.5-1.5B and Qwen2.5-7H are released under the Apache License 2.0, while 20wen2.5-3E released under custom Qwen Research license. We also compare against watth-Sepherdd Seepherr@moeell model (license not found), and GPT-4o (accessed via OpenAl API, governed by OpenAl Terms of Use
F Computing Resources
All experiments conducted machines with AMD EPYC 9K84 96-core CPUs and NVIDIA H20 GPUs. Our code is primarily based on Python 3.12.2 and PyTorch 2.5.1. RL training took approximately day for Qwen-1.5B and Qwen-3B, and days for Qwen-7B. The SFT baseline training required about 2 hours per model, and evaluation took roughly hour per model. In total, the experiments consumed approximately 700 GPU hours. Including preliminary and failed runs, the overall project required more compute.
G Detailed Experiment Results
G.1 Detailed Comparison with off-the-shelf verifiers
we report the average verification accuracy across the five benchmarks, Here, we present the detailed verification accuracy comparison between RISE models, Wath-Shepherd, and GPT-4o each evaluation benchmark.
5.4
Gen. Verifier (GPT-4o)
Disc. Verifier (Math-Stee Shephhh))
Self Verify (RISF-1.5B)
Generator: RISE-1.5B
Verification Accuracy
80
75.7
75.8
71.7
71.3
57.1
64.7
60
40
20
MATH500
AIME24
AMC23
Minerva
Olympiad
Figure 11: Detailed comparisons verification accuracy between RISE-1.5B and other verifiers.
23

---

Gen. Verifier (GPT-40)
Disc. Verifier -athh-Shepherd)
Self (RISE-3B)
100
Generator: RISE-3B
86.3
78,0
81.0
70.1
Verification Accuracy
BP.5
60
57.5
66.1
40
20
MATH500
AIME24
AMC23
Minerva
Olympiad
Figure 12: Detailed comparisons of verification accuracy between RISE-3B and other verifiers.
Gen. Verifier (GPT-40)
Disc. Verifier (Math-Shepherd)
Self Verify (RISE-7B)
100
Generator: RISE-7B
62.5
83.8
80
79.1
Verification Accuracy
67.02
65
542
000.0
07:22
48.6
20
MATH500
AIME24
AMC23
Minerva
Olympiad
Figure 13: Detailed comparisons of verification accuracy between RISE-7B and other verifiers.
G.2 Verification Cost
To assess the verification cost, compute the average verification token usage across the five evaluated benchmarks,
Table Average reasoning and verification token usage across RISE models.
summarized in Ta- 3 In general, the cost of solution verification lightweight compared the problem-sole solving process, with ratios ranging from 0.02 to 14. The verification responses optionally highlight the critical issues and provide the final score suc- einctly. This property allows to incorporate
Model
Reason.
Veri.
Ratio (V/R)
RISE-1.5B
676
13
0.02
RISE-3B
686
94
0.14
RISE-7B
693
52
0.08
the verification outputs from RISE models at test time (e.g., through weighted majority voting) without introducing significant computational overhead, thereby maintaining fairness comparison to baseline budgets and preserving the robustness of RISE under realistic evaluation settings. We also provide the results with adjusted sampling budgets in Table for reference, where RISE continues to improve upon vanilla majority voting and outperforms the Zero-RL basclines in most cases.

---

Table 4:
Test-time scaling (maj@ (k performance under adjusted sampling budgets. Zero-RL and RISE results are measured under baseline budget; SV denotes Self-verify.
Model
RISE + SV Budget
Baseline Budget
Zero-RL
RISE
RISE + SV
=
Qwen2.5-1.5B
4.08 sols
sols
26.5
28.3
28.5
Qwen2.5-3B
4.52 sols
sols
37.3
38.6
38.5
Qwen2.5-7B
4.32 sols
sols
45.6
46.7
48.6
Qwen2.5-1.5B
8.16 sols
sols
29.1
31.4
31.7
Qwen2.5-3B
9.12 sols
sols
39.1
39.2
40.7
Qwen2.5-7B
8.64 sols
sols
49.1
48.2
49.8
G.3 Detailed Analysis for Enhanced Verification
In Figure 7 report the average verification frequency and accuracy of self-verified solutions the five benchmarks. Here, we present the fine-grained results between RISE models and Zero-RL baseline each evaluation benchmark.
Table 5: Performance comparison between RISE models and Zero-RI. models on verification frequency and effectiveness for the generation.
Method
Verification Frequency
MATH
AIME
AMC
Minerva
Olympiad
Avg.
Qwen2.5. Zero-RL
6,45
6.67
2.25
15.59
7.75
RISE-1.5B
7.10
5.31
2.53
15.31
7.80
Owen2.553B-22. Zero-er
4.90
8.33
14.29
2.99
8.72
7.85
RISE-3B
4.63
9.17
18.18
3.08
9.67
8,94
Qwen2.557B-22ro-RL
5.30
7.19
1.56
8.19
5.45
RISE-7B
6.08
7.92
8.13
1.79
8.57
6,50
Self Verified Reasoning Accuracy
Qween25511.5B-22ro-RRl
37.21
0.00
24.00
24.49
15.59
20.26
RISE-1.5B
38.73
4.76
35.29
23.64
15.31
23.55
Owen2.5-3B-Zero-
45.92
0.00
14.29
20.00
19.96
20.03
RISE-3B
43.78
4.55
18.18
22.39
19.54
21.69
Qwwen..5-7B-7cno-Rl
62.74
0.00
8.70
35.29
26.70
26.68
RISE-7B
65.43
5.26
38.46
28.21
28.73
33.22
Table 6: Reflection Keywords Rate between RISE models and Zero-RI. models.
Method
Verification Frequency in Generation
MATH
AIME
AMC
Minerva
Olympiad
Avg.
Owwen2.5-1.5B--Zooo-RR.
0.16
0.40
0.26
0.16
0.29
0.25
RISE-1.5B
0.19
0.45
0.29
0.16
0.32
0.28
Qwen2.5-3B-Zerr--P
0.14
0.40
0.24
0.11
0.27
0.23
RISE-3B
0.16
0.45
0.20
0.13
0.29
0.25
Dwwen2.5-1B-22no-RL..
0.13
0.38
0.23
0.08
0.23
0.21
RISE-7B
0.14
0.50
0.29
0.10
0.27
0.26
G.4 Reflection Keywords Analysis
Following Yeo et al. 2025 we track the self-reflection keywords ("wait", "however", "alterna- tively", "retry", "recheck" ) to quantitatively measure the general reflection behaviors beyond the
25

---

self-verification among the model problem-solving responses. In practice, we sum the total word counts for the keyword set and normalize by the dataset size.
The results Table show that our RISE model constantly have higher level of reflection-related behaviors than the Zero-RI. models, indicating the positive effect of self-verification training.
H Generalization Results
H.1
Generalization on Qwen3 Models
Table 7: Performance comparison of RISE and baseline methods for Qwen3 models.
Reasoning
Self- Verification
Model
MATH
AIME
AMC
Mine.
Olym. Avg.
MATH
AIME
AMC
Mine.
Olym. Avg.
Qwen3-4B-Base
Base 39.4
6.3
24.1
12.6
17.8
20.0
60.9
72.6
61.9
59.4
63.8
63.7
Zero-RL
73.7
13.3
45.9
29.5
37.2
39.9
73.7
39.9
52.9
37.9
47.8
50.4
RISE
77.8
12.9
52.8
43.4
40.6
45.5
87.4
79.6
70.9
50.8
68.0
71.3
Owen3-8B-Ban
Base
42.5
8.3
28.4
15.4
18.4
22.6
67.0
65.5
64.4
62.9
62.0
64.4
Zero-RI. 77.6
13.8
58.1
37.7
41.6
45.7
79.7
54.1
68.8
46.9
56.9
61.3
RISE 83.0
21.3
59.4
48.4
44.4
51.3
91.8
85.4
87.4
53.4
72.2
78.1
To further demonstrate the effectiveness of RISE, we conduct a new set of experiments on the latest Qwen3 models (Apache 2.0 License). As the mathematical reasoning capability of Qwen3 has been shown to be substantially higher than that of Qwen2.5 (Yang et al.) [225), we construct a more challenging training set better match their ability. Specifically, we downsample the DeepMath- 103K [[e et al., 2025) dataset (MIT License) by difficulty level to obtain a 10K subset. Using this data, train both Qwen3-4B-Bar baseline and our RISE, under identical configurations except for the inclusion of the verification objective. For RISE models, we set the verification batch size to 128 by default, consistent with the Qwen2.5 experiments. After training, we follow the same evaluation protocols as in the main experiment assess model performance on both problem-solving and solution verification tasks. The results, summarized in Table show that RISE achieves substantial improvements in reasoning accuracy over the Zero-RL baseline, with average gains of +5.6% and +5.6% for the 4B and models, respectively. Verification accuracy also improves markedly (+20.9% for the 4B model and +16.8% for the 8B model). Together with our main experiments on Qwen2.5, these results demonstrate the generalizability and effectiveness of RISE across different model scales and families.
and Qwen3-8B --B-Bas models with the Zero-RL (vanilla PPO)
H.2
Generalization on Diverse Tasks
Table Performance of RISE and baseline models on diverse tasks.
Model
MMLU-Pro
GPQA
HumanEval
Veri. Acc. (Avg)
Qwen2.5.1.5B.2err-PP.
19.7
20.0
39,9
21.4
RISE-1.5B
21.3
23.0
44.7
67.7
Owen2.5-7B-Zerro-
46.3
27.8
63.7
49.5
RISE-7B
47.4
28.3
64.1
62.1
To evaluate the cross-domain generalization of RISE beyond mathematical reasoning, we further conduct zero-shot transfer study on RISE models. Concretely, we directly test the math-tuned models on diverse tasks, including general knowledge (MMLU-Pro), science (GPQA-Dianond), and code generation (HumanEval). The results in Table & outperform their Zero-RI. baselines both reasoning and verification accuracy across these out-of- distribution domains. These findings indicate that the self-verification ability Ieamned by RISE is
show that RISE-1.55 and RISE-7B consistently
26

---

robust and transferable, providing benefits even in tasks unseen during training. Future work could try applying RISE to broader domains where verifiable reward signals are available.
H.3 Comparison to Separate Training of Reasoning and Verification
While our main focus is to improve the rea- soning capability of single LLM rather than building multi-agent system with specialized components, we provide pilot results on sepa- rately fine-tuned velver-verifier models using the medium-sizee Qwen2.5-3B. Specifically, we use the same MATH-Hard dataset and finetune an additional verifier model on the Zero-RL. model's s (i.e., solver) solutions, formatted as ver- ification problems. As shown in Table 9 the results averaged over five benchmarks indicate that RISE achieves higher overall reasoning perfor- than the separately trained verifier. Unlike this separate training scheme, RISE improves model performance holistically and scales more efficiently, requiring significantly less training and inference cost.
Table 9: Reasoning and verification accuracy for Qwen2.5-3B Finetuned Models.
Model
Reason.
Veri.
Base
14.1
25.3
Zero-RL (Solver)
32.5
78.3
Zero-RL (Verifier)
RISE
33.5
74.3
mance, despite being slightly less verification-accureee
H.4 Verification on External Solution Set
Although we mainly use the verification ability of RISE models self-improvement purposes, we construct a test set with external solutions to address potential concerns about verification generalizability. Specifically, we use GPT-4o's solutions on the five evaluated benchmarks and ask the models to verify their correctness. The results are shown in Table TO which indicate that RISE models still maintain high level of verification accuracy on the external solutions.
Table 10: Verification performance of RISE mod- els and other verifiers on the external solution set.
Verifier Model
Verification Accuracy
RISE-1.5B
67.9
RISE-3B
74.4
RISE-7B
70.7
GPT-4o
57.8
Math- Shath-Shepherd
58.9
This suggests that the verifier learned by RISE generalizes well beyond its own generation style and has acquired a robust notion of correctness.
I
Cases Study
Case Study
### Analysis:
The response provided contains several errors and incorrect calculations. Here's step-by- step breakdown of the issues:
1. **Trinngle Type Identification The response incorrectly identifies the triangle as a right triangle based on the Pythagorean theorem. but it does correctly apply the theorem to confirm the right angle.
** Angle Bisector Theorem Application The Angle Bisector Theorem applied incorrectly, leading to an incorrect ratio for BD and DC.
**Length Calculation of AD**: The length of the angle bisector AD is calculated incorrectly using the formula for the length of the angle bisector in right triangle.
**Power of a Point Theorem*** **; The application of the Power of Point theorem incorrect, leading to an incorrect value for AP.
### Final Rating: llboxed(-055)
Figure 14: Example verification returned by RISE-7B AIME 24.
27


Reviews:
[{"review_id": "J3f3jIMNAS", "reviewer": "Reviewer_rqQc", "summary": "The work introduces RISE, an online reinforcement-learning framework that jointly optimises an LLM’s problem-solving and self-verification skills. In each PPO iteration the policy first generates chain-of-thought solutions, then critiques its own on-policy answers through a templated verification prompt; both trajectories receive binary verifiable rewards from an outcome checker and are mixed in a single RL objective. Experiments on five mathematics benchmarks with Qwen-2.5 models how that RISE raises reasoning accuracy over a Zero-RL baseline and boosts verification accuracy, the learned verifier even outperforms off-the-shelf discriminative and generative judges.", "questions": "Discussed in Weakness.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The work pinpoints “superficial self-reflection” in RLVR and proposes an elegant, integrated solution. This is simpler than multi-stage or reward-model pipelines.\n2. Table 1 shows that RISE improves both reasoning and verification on different sized models and multiple datasets, demonstrating robustness to model size and task difficulty.\n3. The learned verifier surpasses GPT-4o and Math-Shepherd in judging correctness, suggesting that in-distribution, joint training can yield a  accurate critic and useful for low-latency deployment. \n4. The author also discusses the frequency and efficacy of self-verification,  and qualitative cases reveal that RISE encourages multi-step consistency checks rather than re-statements.\n\n# Weaknesses\n\n1. All experiments are in maths with exact-match verifiers. It is unclear whether the method extends to domains where verification is noisy or differen domain (e.g., code-related QA, general reasoning, factual QA). A transfer study or cross-domain demonstration would strengthen generality claims.\n2. While verification accuracy soars, reasoning improvements over Zero - RL are relatively small (≤ 2 pp on most benchmarks). The paper could discuss why better self - checking does not translate into larger solving gains. Also, is there any quantified correlation between the verification accuracy and the final RL performance?\n3. Because the verifier learns from on-policy data and shares parameters with the generator, it may overfit to its own solution style. Evaluating on external solutions or holding out a verifier dev set would alleviate this concern.\n4. The paper does not address cases where the verifier is confidently wrong, or how such failures might compound during training.", "was_revised": true, "final_justification": "I have checked reviewers-authors discussion, most of my concerns has been addressed. I will keep my score.", "rebuttal": "We thank Reviewer rqQc for the positive and constructive feedback, highlighting our elegant solution and robust results.\n\n> **W1:** All experiments are in maths with exact-match verifiers. It is unclear whether the method extends to domains where verification is noisy or differen domain (e.g., code-related QA, general reasoning, factual QA). A transfer study or cross-domain demonstration would strengthen generality claims.\n\nThis is an excellent point. While our math-trained models were not exposed to other domains, we conducted a zero-shot transfer evaluation to test their generalizability. We evaluated the models on general knowledge (MMLU-Pro), science (GPQA), and code (HumanEval).\n\n**The results show that RISE, trained only on math, consistently improves both reasoning and verification accuracy across these diverse, out-of-distribution domains.** This demonstrates that the underlying self-verification skill learned by RISE is robust and transferable.\n\n|  | MMLU-Pro (General) | GPQA (Science) | HumanEval (Code) | Averaged Verification Accuracy |\n| --- | --- | --- | --- | --- |\n| Qwen2.5-1.5B-PPO (Zero-RL) | 19.7 | 20.0 | 39.9 | 21.4 |\n| RISE-1.5B | **21.3 (+1.6)** | **23.0 (+3.0)** | **44.7 (+4.8)** | **67.7 (+46.3)** |\n| Qwen2.5-7B-PPO (Zero-RL) | 46.3 | 27.8 | 63.7 | 49.5 |\n| RISE-7B | **47.4 (+1.1)** | **28.3 (+0.5)** | **64.1 (+0.4)** | **62.1 (+12.6)** |\n\nDue to resource constraints, we have not yet conducted full RL training on non-math domains, but we believe these results strongly support our generality claims and we will add them to the paper.\n\n---\n\n> **W2:** While verification accuracy soars, reasoning improvements over Zero - RL are relatively small (≤ 2 pp on most benchmarks). The paper could discuss why better self - checking does not translate into larger solving gains. Also, is there any quantified correlation between the verification accuracy and the final RL performance?\n\nThank you for this thoughtful question. The difference in gains stems from the fact that verification is an easier task than multi-step reasoning. As noted in prior work, a \"Generation-Verification Gap\" exists where models learn to verify more easily than they learn to generate correct solutions. Our results are consistent with this phenomenon.\n\nWhile verification gains don't translate 1-to-1 to reasoning gains, **stronger verification directly contributes to better reasoning**. As we show in **Section 5.5 (Figure 7 right)**, the accuracy of solutions that invoke self-verification is significantly higher in RISE models than in the baseline. This shows that the learned skill is being used effectively.\n\nRegarding a quantified correlation, we show a positive trend between the two reward types during training in **Figure 4**. However, the exact correlation is complex and depends on the base model and dataset. The key takeaway is that improved self-verification, enabled by RISE, provides a more reliable mechanism for the model to identify and correct error reasoning paths, leading to overall performance improvements.\n\n---\n\n> **W3:** Because the verifier learns from on-policy data and shares parameters with the generator, it may overfit to its own solution style. Evaluating on external solutions or holding out a verifier dev set would alleviate this concern.\n\nThis is a valid concern. To test for overfitting, we evaluated our RISE-trained verifiers on their ability to judge the correctness of **external solutions generated by GPT-4o**.\n\n**Our results show that RISE models outperform strong, dedicated verifiers like GPT-4o and Math-Shepherd on these external solutions.** This indicates that the verifier learned by RISE generalizes well beyond its own generation style and has acquired a robust, non-overfit notion of correctness.\n\n| Verifier Model | Verification Acc. on GPT-4o Solutions (Avg.) |\n| --- | --- |\n| **RISE-1.5B** | 67.9 |\n| **RISE-3B** | **74.4** |\n| **RISE-7B** | 70.7 |\n| GPT-4o (Verifier) | 57.8 |\n| Math-Shepherd | 58.9 |\n\n---\n\n> **W4:** The paper does not address cases where the verifier is confidently wrong, or how such failures might compound during training.\n\nThis is handled directly by our reward mechanism. As described in Lines 153-163, the verifier receives a reward signal based on whether its predicted score (e.g., correct/incorrect) matches the ground-truth score from the outcome verifier.\n\n**Critically, if the verifier is confidently wrong (e.g., it labels a wrong solution as correct), it receives a negative reward and is penalized.** This explicit feedback loop directly discourages overconfident errors and prevents them from compounding during training, forcing the verifier’s prediction to align with the ground-truth outcome."}, {"review_id": "J8RT3iYulE", "reviewer": "Reviewer_RZbs", "summary": "This work presents RISE, an RL framework for LLMs that enhances RLVR by jointly learning self-verification as an additional verifiable task. This framing allows for a powerful auxiliary task that augments RLVR techniques with an additional task. They then show the efficacy of RISE through extensive experimentation and ablations to show the model improving both self-verification and task performance. They show that learning promotes both task capabilities and self-verification performance which also leads to improved self-consistency based test-time compute strategies. Finally they ablate the online nature of the learning demonstrating that learning both capabillities online is critical to the augmented performance.", "questions": "1. No big questions other than weaknesses section. More out of curiosity, have the authors tried testing RISE on other verifiable domains such as code?\n2. How would the authors consider applying a similar strategy to learned reward models? \n3. Related to question 1, what would the authors think would be the effects of doing multi-task RLVR with this auxiliary loss? \n4. Finally, how sensitive is the performance of this method to the prompt chosen for the verification task?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Thorough experimentation. Particularly want to highlight the ablation of adding in offline self-verification and showed the importance of online verification\n- Convincing baselines and experiment design to answer the question of \"does self-verification help\"\n- Showed further utility of learning self-verification through test-time compute experiments.\n\nWeaknesses:\n- Small clarity comments\n1. perhaps the baseline shouldn't be called zero rl but instead PPO. \n2. For section 4.1, could the authors more explicitly detail the verficiation reward?", "was_revised": true, "final_justification": "This paper presents a method in a growing body of work augmenting RLVR with other verifiable auxiliary tasks. Their implementation is simple and would scale well across verifiable domains. They further showed the robustness of this methodology to prompts during the rebuttal.", "rebuttal": "Thank you for the positive evaluation and for recognizing the thoroughness of our experiments and the utility of our method.\n\n> **W1&W2:** Clarity on baseline name and verification reward.\n\nThank you for these excellent suggestions for improving clarity.\n\n1. **Baseline Name:** We agree. \"PPO\" is a more precise and standard name for the baseline. We will change \"Zero-RL\" to \"PPO\" in the revised paper.\n2. **Verification Reward:** Absolutely, let us further clarify it. The verification reward is computed based on the exact match between the score extracted from the model’s verification response and the reward provided by the rule-based outcome verifier for that solution. Specifically:\n  - **+1.0 reward:** for correctly predicting the score in the specified format (e.g., `\\\\boxed{1}`).\n  - **-0.5 reward:** for predicting an incorrect score in the correct format.\n  - **-1.0 reward:** for generating an invalid format (e.g., missing the `\\\\boxed{}` wrapper).\n\nWe will add these explicit details to Section 4.1.\n\n---\n\n> **Q1:** More out of curiosity, have the authors tried testing RISE on other verifiable domains such as code?\n\nThis is an excellent point. While our math-trained models were not exposed to other domains, we conducted a zero-shot transfer evaluation to test their generalizability. We evaluated the models on general knowledge (MMLU-Pro), science (GPQA), and code (HumanEval).\n\n**The results show that RISE, trained only on math, consistently improves both reasoning and verification accuracy across these diverse, out-of-distribution domains.** This demonstrates that the underlying self-verification skill learned by RISE is robust and transferable.\n\n|  | MMLU-Pro (General) | GPQA (Science) | HumanEval (Code) | Averaged Verification Accuray |\n| --- | --- | --- | --- | --- |\n| Qwen2.5-1.5B-PPO (Zero-RL) | 19.7 | 20.0 | 39.9 | 21.4 |\n| RISE-1.5B | **21.3 (+1.6)** | **23.0 (+3.0)** | **44.7 (+4.8)** | **67.7 (+46.3)** |\n| Qwen2.5-7B-PPO (Zero-RL) | 46.3 | 27.8 | 63.7 | 49.5 |\n| RISE-7B | **47.4 (+1.1)** | **28.3 (+0.5)** | **64.1 (+0.4)** | **62.1 (+12.6)** |\n\nDue to resource constraints, we have not yet conducted full RL training on non-math domains, but we believe these results strongly support our generality claims and we will add them to the paper.\n\n---\n\n> **Q2:** How would the authors consider applying a similar strategy to learned reward models?\n\nThis is an insightful direction. In this work, we pair a more challenging task (reasoning) with a relatively easier auxiliary task (verification), and find that co-training improves both. Reversing this setup, i.e., using generative reasoning to facilitate training a learned reward model, may be less straightforward, as the auxiliary task is harder than the main on. However, it remains a promising direction worth exploring, particularly for the learned generative reward models.\n\n---\n\n> **Q3:** Related to question 1, what would the authors think would be the effects of doing multi-task RLVR with this auxiliary loss?\n\nWe believe RISE is extendable to multi-task RLVR settings. The key would be to ensure that the tasks (and their verification counterparts) are balanced in difficulty and that their reward scales are properly aligned. With a well-curated multi-task dataset, we expect RISE to be highly effective.\n\n---\n\n> **Q4:** Finally, how sensitive is the performance of this method to the prompt chosen for the verification task?\n\nThis is an important practical question. We ran an additional experiment on Qwen2.5-7B using a paraphrased verification prompt with minor structural changes.\n\n**The results show that RISE is robust to the paraphrasing of the verification prompt.** While verification accuracy saw a minor drop, the primary reasoning performance remained stable and still significantly outperformed the PPO baseline.\n\n| Qwen2.5-7B | Reasoning | Self-Verification |\n| --- | --- | --- |\n| RISE (Original Prompt) | 42.9 | 69.2 |\n| **RISE (Modified Prompt)** | **42.5** | **65.4** |\n| PPO Baseline | 41.7 | 46.6 |"}, {"review_id": "o4WWQ0mXws", "reviewer": "Reviewer_PrjU", "summary": "The paper proposes RISE, a novel RL framework to fine tune an LLM to both reason and verify its own reasoning steps.", "questions": "Why would it be desirable to constrain the generator and the verifier to be the same model?\nAs far as I know, and as the authors state in line 31, these are two independent tasks which are often performed by separate models.\nThese models could even be of different sizes, to better adapt to the different complexities of the two tasks.\n\nLine 26-27: I don't understand, why would that be a problem? Again, why is it desirable / important for the generator model to have robust self-assessment skills?\nLine 33: it seems it is assumed this is a limitation, but it's never argued why is it so.\n\n\nI am quite unfamiliar with the related literature and I have little to no expertise on this topic, but I suspect the claim in Sec 2 line 80-82 might require some further clarification.\nThe authors seem to claim no previous work fine-tuned the verifier model using verifiable rewards, is that correct?\n\n\nSec 5.1: do you consider any baseline where generator and verifier are RL-tuned independently? That seems a much more relevant baseline than Zero-RL, which only finetunes the generator/problem-solver. Also, how is SFT trained? \n\nSec 5.2: lines 208-211: this paragraph seems to suggest the authors expect the verification performance of a model to decrease with model size and that RISE limits this problem.\nHowever, I would expect the verification performance to increase with model size; instead, in Table 1, it seems that the verification performance of RISE decreases. \nStill, other models in Table 1 generally show improved verification performance when increasing parameters.\nHence, I have some doubts about the strength of the authors' argument, can they maybe comment on this?\n\n\nLines 320-325: the author claims \"substantial increase in verification frequency\", but that seems a rather modest increase to me (between 0 and 1%?).\nAlso, I don't see why an increased verification frequency is necessarily a desirable property. I imagine one would like the model to ask for verification only when needed (for example, when it suspects the reasoning trajectory is mistaken and can, therefore, be pruned). A model requiring verification at each single step would achieve maximum verification frequency but seems undesirable to me.\n\nIs it maybe possible to formulate a different and more insightful metric to do this comparison? E.g. can we measure how often both models require verification versus how often verification is actually needed? (e.g. an intersection-over-union metric).\nOn this note, the following comparison between Zero-RL and RISE in terms of overall reasoning accuracy rather than verification frequency seems much more useful to me.", "limitations": "LIMITATIONS\n\nYes, the authors discuss the limitations of the proposed method.\n\n\nSOCIETAL IMPACT\n\nThe authors claim no societal impacts as their work focuses on improving reasoning capabilities on tasks such as mathematics.\nHowever, claiming that improving reasoning capabilities just has no societal impacts seems a bit an oversimplification to me, although, again, this topic is outside my field of expertise and, therefore, I do not have any strong opinion here.\n\nStill, I know at least that some research on this topic does exist (e.g. potential societal impact when these methods are applied outside mathematics data) so I suspect the conclusion might not be as straightforward as presented here.\n\nWhat do the authors and the other reviewers think of this?", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper includes an extensive experiment section, with different ablation studies.\nThe proposed idea seems interesting and probably quite novel as far as I know, although I have very limited expertise and knowledge of the related literature, so I might be missing some relevant previous work.\n\nOn the other hand, I feel like the explanation of the motivation behind the main idea is a bit weak (why should the generator and the verifier be the same model?) and that the manuscript contains a few hand-wavy claims which I am not sure are properly supported.\nSee Questions below.\n\nFinally, I feel like the authors should consider some additional baselines for proper comparison; see questions below.", "was_revised": true, "final_justification": "The authors clarified all my doubts, included some initial promising analysis and promised to try to implement the baseline suggested.", "rebuttal": "We thank the reviewer PrjU for finding our idea “interesting and probably quite novel”.\n\n> **Q1&Q2:** Motivation for a single, self-verifying model.\n\nThese questions get to the heart of our motivation. Our goal is **not just to build a separate verifier, but to improve the generator's own reasoning process.**\n\nThe key issue we address is \"superficial self-reflection\", where the emergent self-reflections do not lead to correct final answers. By training a **single, unified model** to both solve and verify, RISE encourages the model to develop an **internally consistent** **and correct** reasoning process. A model that can reliably check its own work is more likely to produce robust and trustworthy reasoning. This aligns with trends in state-of-the-art models (e.g., OpenAI's o1, DeepSeek-R1) which heavily feature self-reflection as a core capability.\n\n---\n\n> **Q3:** I am quite unfamiliar with the related literature and I have little to no expertise on this topic, but I suspect the claim in Sec 2 line 80-82 might require some further clarification. The authors seem to claim no previous work fine-tuned the verifier model using verifiable rewards, is that correct?\n\nTo clarify our claim in Lines 80-82: our novelty lies in **explicitly and simultaneously training both the generator and the verifier within a single, online RL loop, using on-policy data and a unified objective.** While others have trained verifiers separately or used them to generate rewards, to our knowledge, this tightly-coupled, joint-training framework for both skills is novel.\n\n---\n\n> **Q4:** Sec 5.1: do you consider any baseline where generator and verifier are RL-tuned independently? That seems a much more relevant baseline than Zero-RL, which only finetunes the generator/problem-solver. Also, how is SFT trained?\n\nTuning a separate verifier is an interesting direction, but it addresses a different research question (building the best separate components). Our primary goal is to ablate the effect of our proposed joint-training objective. Therefore, the most direct and scientifically sound baseline is **Zero-RL**, which uses the identical RL setup but without our proposed verification objective. This allows for a clean comparison demonstrating the benefit of RISE's core contribution.\n\nRegarding SFT: SFT models are instruction-tuned on the golden solutions from the MATH-Hard (Level 3-5) dataset, and detailed implementations can be found in **Appendix C.** \n\n---\n\n> **Q5:** Sec 5.2: lines 208-211: this paragraph seems to suggest the authors expect the verification performance of a model to decrease with model size and that RISE limits this problem. However, I would expect the verification performance to increase with model size; instead, in Table 1, it seems that the verification performance of RISE decreases. Still, other models in Table 1 generally show improved verification performance when increasing parameters. Hence, I have some doubts about the strength of the authors' argument, can they maybe comment on this?\n\nThank you for the opportunity to clarify. We do **not** claim that verification performance should decrease with model size. We believe the self-verification results in Table 1 show variance across scales, not a decreasing trend. This is likely because self-verification on its own outputs is a relatively easy task that even smaller models can learn well, leading to comparable performances.\n\nTo further validate our claim, we evaluated the models on a fixed external solutions (GPT-4o responses on the 5 math benchmarks). Here, we see no decreasing trend; rather, all RISE models maintain strong verification accuracy across scales.\n\n| Verifier Model | Verification Acc. on GPT-4o Solutions (Avg.) |\n| --- | --- |\n| RISE-1.5B | 67.9 |\n| RISE-3B | 74.4 |\n| RISE-7B | 70.7 |\n| GPT-4o | 57.8 |\n| Math-Shepherd | 58.9 |\n\n---\n\n> **Q6 & Q7**: Verification frequency and metrics.\n\nWe completely agree that higher frequency is not the goal; **more *effective* verification is**. As we argue in the paper, the increased frequency is meaningful precisely because it leads to better outcomes.\n\n- First, we kindly note that the paper does not claim a \"substantial increase\" for all models; we explicitly state the increase for the 1.5B model is modest and the increase \"becomes substantial as scale grows (+1.09% for 3B and +1.05% for 7B)\" (Line 322).\" We will rephrase this in the revised version to ensure clarity.\n- More importantly, as shown in **Figure 7 (right)**, the **accuracy of the self-verified subset of responses is significantly higher for RISE models**. This shows the verifications are not just more frequent, but more *impactful*, helping the model correct or confirm its reasoning path.\n- Regarding new metrics (Q7), we agree that measuring when verification is \"needed\" would be insightful. However, this is non-trivial to define objectively, as we cannot know the model's internal state of uncertainty. We believe our current metric—showing improved accuracy when self-verification is invoked—is also a direct indicator of its effectiveness.\n\n---\n\n> **Societal Impact:** The authors claim no societal impacts as their work focuses on improving reasoning capabilities on tasks such as mathematics. However, claiming that improving reasoning capabilities just has no societal impacts seems a bit an oversimplification to me.\n\nThank you for this important point. You are right that \"no societal impact\" is an oversimplification. We will revise this section. Our intended meaning was that our work, as a foundational training framework evaluated on public math datasets, does not introduce *immediate, direct* societal risks. However, we acknowledge that any advance in AI reasoning capabilities has potential downstream impacts, both positive and negative. We will add a sentence stating that the method we propose should be developed and applied responsibly."}, {"review_id": "K5ooPQaS8o", "reviewer": "Reviewer_KxXf", "summary": "The paper introduces a new framework for improving mathematical reasoning of language models. The core idea is to train a language model jointly on generation and verification tasks using reinforcement learning with verifiable rewards. The proposed method achieves modest gains on the generation task and substantial gains on the verification task across a range of mathematical reasoning benchmarks.", "questions": "Please refer to the second and last bullet points of the weaknesses above.", "limitations": "The authors clearly states the limitations of the proposed method in Appendix A.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- (Originality) Training a single unified model for generation and verification is novel.\n- (Quality) The proposed method is evaluated on various mathematical reasoning benchmarks.\n\nWeaknesses\n- (Significance) The improvement in generation accuracy is not statistically significant, except on the AMC dataset.\n- (Significance) Verification does not seem to offer much benefit at test time. Because the method uses a generative verifier, each verification step consumes roughly the same number of tokens as producing a solution. For a fair comparison, \"k=4 + verification\" should be compared against \"k=8 without verification.\" Based on Figure 2, the latter achieves higher accuracy.\n- (Clarity) The online and offline verification experiment (Figure 6) shows that high verification accuracy does not necessarily translate into high generation accuracy, thereby unintentionally weakening the paper’s claim.", "was_revised": true, "final_justification": "I initially considered their evaluation unfair and insufficiently robust, but their rebuttal addressed my concerns, so I have raised my score from 2 to 3. I still feel that the results on Qwen2.5 are not strong enough to convincingly demonstrate that the proposed method outperforms the baselines. The Qwen-3 experiments are highly promising, but I excluded them from the evaluation to maintain fairness (GPU-poor vs. GPU-rich).", "rebuttal": "We thank Reviewer KxXf for their critical feedback, which has pushed us to further strengthen our paper with new experiments and analysis.\n\n> **W1:** The improvement in generation accuracy is not statistically significant, except on the AMC dataset.\n\nWhile the gains with the Qwen2.5 series are consistent, we acknowledge that stronger improvement is helpful. To further demonstrate the effectiveness and significance of RISE, we ran new experiments on the more recent **Qwen3 base models, Qwen3-4B-Base and Qwen3-8B-Base**, using 10K training data from DeepMath.\n\n**On these newer models, RISE achieves substantial and significant improvements in reasoning accuracy over the Zero-RL (PPO) baseline**, with average gains of **+5.6%** and **+5.6%** for the 4B and 8B models, respectively. Verification accuracy also sees a massive boost (+20.9% for the 4B model and +16.8% for the 8B model).\n\n| Reasoning Acc. (**4B**)| MATH | AIME | AMC | Minera | Oly. | **Avg.** |\n| --- | --- | --- | --- | --- | --- | --- |\n| PPO | 73.7 | 13.3 | 45.9 | 29.5 | 37.2 | 39.9 |\n| **RISE** | **77.8** | 12.9 | **52.8** | **43.4** | **40.6** | **45.5 (+5.6)** |\n\n| Reasoning Acc. (**8B**)| MATH | AIME | AMC | Minera | Oly. | **Avg.** |\n| --- | --- | --- | --- | --- | --- | --- |\n| PPO | 77.6 | 13.8 | 58.1 | 37.7 | 41.6 | 45.7 |\n| **RISE** | **83.0** | **21.3** | **59.4** | **48.4** | **44.4** | **51.3 (+5.6)** |\n\nThese results clearly demonstrate the significant impact of our method. We will add them to the paper.\n\n---\n\n> **W2:** Test-time benefit and fair comparison of compute.\n\nThis concern is based on a **critical factual misunderstanding about the compute cost of verification.** A verification response can be very short (e.g., \"The final rating for the response is \\\\boxed{-0.5}.\"), whereas a solution response is a long chain-of-thought. Thus, the premise that their token costs are \"roughly the same\" is incorrect.\n\nWe calculated the actual token consumption. On average, a **verification response uses only 2% to 14% of the tokens of a full solution generation.**\n\n| Model | Avg. Reasoning Tokens | Avg. Verification Tokens | **Ratio (Verif/Reason)** |\n| --- | --- | --- | --- |\n| RISE-1.5B | 676 | 13 | **0.02** |\n| RISE-3B | 686 | 94 | **0.14** |\n| RISE-7B | 693 | 52 | **0.08** |\n\nTherefore, comparing `k=4 + verif` to `k=8` is not a fair comparison. A much fairer comparison, which generously **overestimates** the verification cost, would be against `k=4 + 4*0.14 ≈ k=5` for the RISE models.\n\nHere is the adjusted comparison. **RISE with self-verification at k=4 consistently outperforms the stronger PPO baseline at k=5.** This demonstrates a clear and compute-efficient benefit at test time.\n\n| maj@k | RISE + self-verify (k=4) | PPO (original, k=4) | PPO (adjusted, k=5) |\n| --- | --- | --- | --- |\n| RISE-1.5B | **28.5** | 26.5 | 28.1 |\n| RISE-3B | **38.5** | 36.2 | 37.3 |\n| RISE-7B | **48.6** | 45.6 | 46.9 |\n\n---\n\n> **W3:** Offline verification experiment weakens the claim.\n\nWe believe there is a misunderstanding of the purpose of this ablation. The result in Figure 6 does not weaken our claim; on the contrary, **it strengthens our central thesis that *online* verification is crucial.** The entire point of the offline experiment was to show that simply mixing in offline verification data is *not* effective, as the verification skill is not learned jointly with on-policy generation. This highlights the limitation of decoupled approaches and motivates the need for our integrated, online framework.\n\nAdditionally, as discussed in Section 5.5, verification is generally an easier task than reasoning, so the translation from verification accuracy to reasoning accuracy is not always linear. In Figure 7 (right), we show that **the accuracy of self-verified responses improves substantially with RISE**, providing a more direct evidence that strengthened self-verification contributes to better reasoning performance."}]

Author rebuttal:

Meta-review: This paper proposes RISE, a new online RL framework for LLMs that learns self-verification as an additional verifiable task, together with the original problem-solving. Initially, the reviewers raised some concerns such as the demonstration on diverse tasks (rqQc, RZbs, KxXf) or the fairness due to the cost from verification step (KxXf). During the rebuttal, the authors successfully resolved these concerns as denoted by the reviewer’s replies. Therefore, we would like to suggest accept this work.
Decision: Accept (poster)