Title: Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Reasoning

RLVR

Self-verification
Abstract: Large Language Models (LLMs) show great promise in complex reasoning, with Reinforcement Learning with Verifiable Rewards (RLVR) being a key enhancement strategy. However, a prevalent issue is ``superficial self-reflection'', where models fail to robustly verify their own outputs. We introduce RISE (Reinforcing Reasoning with Self-Verification), a novel online RL framework designed to tackle this. RISE explicitly and simultaneously trains an LLM to improve both its problem-solving and self-verification abilities within a single, integrated RL process. The core mechanism involves leveraging verifiable rewards from an outcome verifier to provide on-the-fly feedback for both solution generation and self-verification tasks. In each iteration, the model generates solutions, then critiques its own on-policy generated solutions, with both trajectories contributing to the policy update. 
Extensive experiments on diverse mathematical reasoning benchmarks show that RISE consistently improves model's problem-solving accuracy while concurrently fostering strong self-verification skills. Our analyses highlight the advantages of online verification and the benefits of increased verification compute. Additionally, RISE models exhibit more frequent and accurate self-verification behaviors during reasoning. These advantages reinforce RISE as a flexible and effective path towards developing more robust and self-aware reasoners.


Peer reviews:
comments: SUMMARY:
The paper introduces a new framework for improving mathematical reasoning of language models. The core idea is to train a language model jointly on generation and verification tasks using reinforcement learning with verifiable rewards. The proposed method achieves modest gains on the generation task and substantial gains on the verification task across a range of mathematical reasoning benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths
- (Originality) Training a single unified model for generation and verification is novel.
- (Quality) The proposed method is evaluated on various mathematical reasoning benchmarks.

Weaknesses
- (Significance) The improvement in generation accuracy is not statistically significant, except on the AMC dataset.
- (Significance) Verification does not seem to offer much benefit at test time. Because the method uses a generative verifier, each verification step consumes roughly the same number of tokens as producing a solution. For a fair comparison, "k=4 + verification" should be compared against "k=8 without verification." Based on Figure 2, the latter achieves higher accuracy.
- (Clarity) The online and offline verification experiment (Figure 6) shows that high verification accuracy does not necessarily translate into high generation accuracy, thereby unintentionally weakening the paper’s claim.

QUESTIONS:
Please refer to the second and last bullet points of the weaknesses above.

LIMITATIONS:
The authors clearly states the limitations of the proposed method in Appendix A.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes RISE, a novel RL framework to fine tune an LLM to both reason and verify its own reasoning steps.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper includes an extensive experiment section, with different ablation studies.\nThe proposed idea seems interesting and probably quite novel as far as I know, although I have very limited expertise and knowledge of the related literature, so I might be missing some relevant previous work.\n\nOn the other hand, I feel like the explanation of the motivation behind the main idea is a bit weak (why should the generator and the verifier be the same model?) and that the manuscript contains a few hand-wavy claims which I am not sure are properly supported.\nSee Questions below.\n\nFinally, I feel like the authors should consider some additional baselines for proper comparison; see questions below.\n\nQUESTIONS:\nWhy would it be desirable to constrain the generator and the verifier to be the same model?\nAs far as I know, and as the authors state in line 31, these are two independent tasks which are often performed by separate models.\nThese models could even be of different sizes, to better adapt to the different complexities of the two tasks.\n\nLine 26-27: I don't understand, why would that be a problem? Again, why is it desirable / important for the generator model to have robust self-assessment skills?\nLine 33: it seems it is assumed this is a limitation, but it's never argued why is it so.\n\n\nI am quite unfamiliar with the related literature and I have little to no expertise on this topic, but I suspect the claim in Sec 2 line 80-82 might require some further clarification.\nThe authors seem to claim no previous work fine-tuned the verifier model using verifiable rewards, is that correct?\n\n\nSec 5.1: do you consider any baseline where generator and verifier are RL-tuned independently? That seems a much more relevant baseline than Zero-RL, which only finetunes the generator/problem-solver. Also, how is SFT trained? \n\nSec 5.2: lines 208-211: this paragraph seems to suggest the authors expect the verification performance of a model to decrease with model size and that RISE limits this problem.\nHowever, I would expect the verification performance to increase with model size; instead, in Table 1, it seems that the verification performance of RISE decreases. \nStill, other models in Table 1 generally show improved verification performance when increasing parameters.\nHence, I have some doubts about the strength of the authors' argument, can they maybe comment on this?\n\n\nLines 320-325: the author claims \"substantial increase in verification frequency\", but that seems a rather modest increase to me (between 0 and 1%?).\nAlso, I don't see why an increased verification frequency is necessarily a desirable property. I imagine one would like the model to ask for verification only when needed (for example, when it suspects the reasoning trajectory is mistaken and can, therefore, be pruned). A model requiring verification at each single step would achieve maximum verification frequency but seems undesirable to me.\n\nIs it maybe possible to formulate a different and more insightful metric to do this comparison? E.g. can we measure how often both models require verification versus how often verification is actually needed? (e.g. an intersection-over-union metric).\nOn this note, the following comparison between Zero-RL and RISE in terms of overall reasoning accuracy rather than verification frequency seems much more useful to me.\n\nLIMITATIONS:\nLIMITATIONS\n\nYes, the authors discuss the limitations of the proposed method.\n\n\nSOCIETAL IMPACT\n\nThe authors claim no societal impacts as their work focuses on improving reasoning capabilities on tasks such as mathematics.\nHowever, claiming that improving reasoning capabilities just has no societal impacts seems a bit an oversimplification to me, although, again, this topic is outside my field of expertise and, therefore, I do not have any strong opinion here.\n\nStill, I know at least that some research on this topic does exist (e.g. potential societal impact when these methods are applied outside mathematics data) so I suspect the conclusion might not be as straightforward as presented here.\n\nWhat do the authors and the other reviewers think of this?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents RISE, an RL framework for LLMs that enhances RLVR by jointly learning self-verification as an additional verifiable task. This framing allows for a powerful auxiliary task that augments RLVR techniques with an additional task. They then show the efficacy of RISE through extensive experimentation and ablations to show the model improving both self-verification and task performance. They show that learning promotes both task capabilities and self-verification performance which also leads to improved self-consistency based test-time compute strategies. Finally they ablate the online nature of the learning demonstrating that learning both capabillities online is critical to the augmented performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Thorough experimentation. Particularly want to highlight the ablation of adding in offline self-verification and showed the importance of online verification\n- Convincing baselines and experiment design to answer the question of \"does self-verification help\"\n- Showed further utility of learning self-verification through test-time compute experiments.\n\nWeaknesses:\n- Small clarity comments\n1. perhaps the baseline shouldn't be called zero rl but instead PPO. \n2. For section 4.1, could the authors more explicitly detail the verficiation reward?\n\nQUESTIONS:\n1. No big questions other than weaknesses section. More out of curiosity, have the authors tried testing RISE on other verifiable domains such as code?\n2. How would the authors consider applying a similar strategy to learned reward models? \n3. Related to question 1, what would the authors think would be the effects of doing multi-task RLVR with this auxiliary loss? \n4. Finally, how sensitive is the performance of this method to the prompt chosen for the verification task?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work introduces RISE, an online reinforcement-learning framework that jointly optimises an LLM’s problem-solving and self-verification skills. In each PPO iteration the policy first generates chain-of-thought solutions, then critiques its own on-policy answers through a templated verification prompt; both trajectories receive binary verifiable rewards from an outcome checker and are mixed in a single RL objective. Experiments on five mathematics benchmarks with Qwen-2.5 models how that RISE raises reasoning accuracy over a Zero-RL baseline and boosts verification accuracy, the learned verifier even outperforms off-the-shelf discriminative and generative judges.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The work pinpoints “superficial self-reflection” in RLVR and proposes an elegant, integrated solution. This is simpler than multi-stage or reward-model pipelines.\n2. Table 1 shows that RISE improves both reasoning and verification on different sized models and multiple datasets, demonstrating robustness to model size and task difficulty.\n3. The learned verifier surpasses GPT-4o and Math-Shepherd in judging correctness, suggesting that in-distribution, joint training can yield a  accurate critic and useful for low-latency deployment. \n4. The author also discusses the frequency and efficacy of self-verification,  and qualitative cases reveal that RISE encourages multi-step consistency checks rather than re-statements.\n\n# Weaknesses\n\n1. All experiments are in maths with exact-match verifiers. It is unclear whether the method extends to domains where verification is noisy or differen domain (e.g., code-related QA, general reasoning, factual QA). A transfer study or cross-domain demonstration would strengthen generality claims.\n2. While verification accuracy soars, reasoning improvements over Zero - RL are relatively small (≤ 2 pp on most benchmarks). The paper could discuss why better self - checking does not translate into larger solving gains. Also, is there any quantified correlation between the verification accuracy and the final RL performance?\n3. Because the verifier learns from on-policy data and shares parameters with the generator, it may overfit to its own solution style. Evaluating on external solutions or holding out a verifier dev set would alleviate this concern.\n4. The paper does not address cases where the verifier is confidently wrong, or how such failures might compound during training.\n\nQUESTIONS:\nDiscussed in Weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}