Title: Incentivizing LLMs to Self-Verify Their Answers
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language model

math reasoning

reinforcement learning
Abstract: Large Language Models (LLMs) have demonstrated remarkable progress in complex reasoning tasks through both post-training and test-time scaling laws. While prevalent test-time scaling approaches are often realized by using external reward models to guide the model generation process, we find that only marginal gains can be acquired when scaling a model post-trained on specific reasoning tasks. We identify that the limited improvement stems from distribution discrepancies between the specific post-trained generator and the general reward model. To address this, we propose a framework that incentivizes LLMs to self-verify their own answers. By unifying answer generation and verification within a single reinforcement learning (RL) process, we train models that can effectively assess the correctness of their own solutions. The trained model can further scale its performance at inference time by verifying its generations, without the need for external verifiers. We train our self-verification models based on Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-1.5B, demonstrating their capabilities across varying reasoning context lengths. Experiments on multiple mathematical reasoning benchmarks show that our models can not only improve post-training performance but also enable effective test-time scaling. Our code is available at https://github.com/mansicer/self-verification.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a unified RL framework that trains LLM by synergizing answer generation and verification via Group Relative Policy Optimization (GRPO), eliminating the need for external reward models at inference time. The authors integrate a generative-verifier reward into GRPO, using a policy‐aligned buffer and a dynamic verification reward to focus the model on challenging cases. This method is then evaluated on mathematical reasoning benchmarks (MATH500, AIME24/25, AMC23, OlympiadBench), showing that “Self-Verification” improves post-training performance over standard GRPO and enables more effective test-time scaling compared to self-consistency, best-of-N, beam search, etc.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The single‐model approach bridges post-training and test-time scaling, avoiding distributional mismatch with external verifiers.\n\n2. Evaluation: On multiple benchmarks, Self-Verification models outperform base models and prior RL baselines (e.g., SimpleRL, DeepScaleR) in both problem-solving and verification tasks. In terms of efficiency, verification consumes significantly fewer tokens (24% in problem-solving) and lower inference overhead compared to external-verifier methods. Ablation studies \n\nWeaknesses:\n\n1. (Data Contamination): The paper does not discuss whether evaluation sets (e.g.\\ AIME24, MATH500) might overlap with the training data or pretraining corpora, especially for the self-verification verifier (Tables 2–3). Without contamination analysis, reported verification accuracies may be overly optimistic.\n\n2. The authors focus exclusively on LLM-based generative verification (which may contain hallucinations and black-box mechanisms) and do not compare to symbolic or formal engines (e.g.\\ SAT/SMT solvers) that could provide deterministic verification paths for many math sub‐tasks.\n\nQUESTIONS:\n1. Verification without Ground Truth: During test-time scaling (Section 3.2), how does the model reliably assess correctness in the absence of explicit ground-truth labels? What confidence calibration mechanisms prevent propagation of false positives/negatives?\n\nLIMITATIONS:\nYes, the authors have explained the limitations, including the scope and potential constraints of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel \"self-verification\" framework to address the distributional shift between specialized language models and external verifiers, a key problem that limits performance scaling at test-time. The authors propose unifying answer generation and verification within a single RL process, training the model to simultaneously solve complex reasoning tasks and assess the correctness of its own solutions. By leveraging a policy-aligned buffer to train on its own recent outputs and a dynamic reward function that incentivizes learning on difficult cases, the model effectively becomes its own highly capable and efficient verifier. Experiments on mathematical reasoning benchmarks demonstrate that this method not only improves the model's intrinsic problem-solving ability but also enables superior test-time scaling by self-evaluating multiple candidate responses, outperforming traditional methods like self-consistency and those relying on external reward models.\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed unified framework for problem-solving and self-verification is quite intriguing, as it appears, at least on the surface, to lead to more strong reasoning capabilities. Furthermore, the authors' point regarding verifier distribution shift is compelling, and I concur with this perspective.\n\nHowever, a conceptual challenge arises from this design. While using a single model for both problem-solving and self-verification elegantly addresses the issue of distribution shift, it seems that, theoretically, the responses generated by the model itself do not introduce new, external information. This raises the crucial question of why this unified training approach should enhance the model's fundamental reasoning abilities. The paper seems to lack a more thorough or intuitive explanation for this synergistic effect.\n\nQUESTIONS:\n1、Regarding Table 1, the comparison seems potentially imbalanced. If I understand correctly, the \"Self-Verification (Ours)\" method involves generating multiple responses which are then verified, whereas the baseline methods only generate a single response. Could you clarify if this is an apples-to-apples comparison, or am I misinterpreting the experimental setup for this table?\n\n2、The performance improvement shown in Table 1 appears to be somewhat marginal. Specifically, when comparing the \"Self-Verification\" method to the \"GRPO (Problem-solving Only)\" baseline, the gain seems modest. Could you elaborate on the significance of this improvement?\n\n\n3、Regarding the training process, my understanding is that for each problem-solving sample, G responses are generated, which then create G verification samples. If G=8, this seems to imply a significant increase in computational overhead for training the verification task. Could you comment on the cost-benefit analysis here, as the performance improvement from this additional training seems modest?\n\n\n4、I acknowledge that the dynamic reward function for self-verification is quite sophisticated. However, looking at Figure 6 in the appendix, the performance of the simple 0-1 reward function doesn't seem substantially worse, especially for the smaller R1-1.5B model?\n\n5、I'm interested in the claim that problem-solving and self-verification are mutually beneficial. From one perspective, the verification task could be seen as a simple re-framing of the original task's output, potentially not adding substantial new information. Could you provide a more detailed or intuitive explanation for how training a model to verify its own solutions leads to a genuine improvement in its core problem-solving abilities?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInspired along the line of the generative verifier works[14-16], the paper applies GRPO algorithm to align the model to answer and verify the question at the same time. During the inference time scaling, the same model is used to answer and question and verify the generated answer. A weighted aggregation is used to determine the final answer.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The clarity of the paper's writing and presentation is good.\n\nWeakness:\n\nMethod-wise:\n\n1. The novelty of the proposed method, both technically and method-wise, is quite limited. The method builds on the success of previous generative-verifier works [14–16]. The major difference lies in: (1) instead of using SFT, the model uses an RL method; and (2) following naturally from point 1, the method relies on RL–online sampled data instead of the pre-constructed dataset used in previous generative-verifier works.\n2. One proposed benefit of the method is that no external verifier is needed for test-time scaling. However, the same amount—or even more—compute is required, since the model must be prompted to verify the generated answer. Thus, compute-wise, the proposed method offers no advantage beyond saving a copy of the model weights in storage. Moreover, when scaling to larger models, this approach is debatably limited by the need for both generation and verification to rely on the same large model, thereby losing the flexibility of using a large model for generation and a smaller model for more efficient verification.\n\nExperiment-wise:\n\n3. The performance improvement seems to be marginal compared to the GRPO baselines shown in Table 1.\n4. Despite being closely related to previous Gen-Verifier works, there seems to be no comparison against previous work [14], It's expected to see a comparison on verifier's accuracy, as well as performance difference of using different verifiers for test-time-scaling\n5. Ablation studies against some of the important method design are missing, e.g. Policy-Aligned Buffer and $\\alpha$ from Test-time Scaling\n\nQUESTIONS:\n* In table 1, is the verification used in avg@10, if yes, how?\n* In table 1, GRPO-Qwen-7B seems duplicate, what's the GRPO-R1-1.5B performance?\n* What's the verifier model for best-of-N baselines? How good is the trained model's ability to serve as verifier other than the answer generated by itself? (e.g we can verify this by using the trained model as another model's verifier)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a unified reinforcement learning framework that trains large language models to both generate and verify their own solutions within a single policy. They introduce a dynamic verification reward and a policy-aligned buffer to address distribution shifts and data imbalance during joint optimization. The approach is tested on mathematical reasoning benchmarks using Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-1.5B, their self-verification models improve post-training performance and enable effective test-time scaling without external verifiers, achieving good results.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n1. Strong performance advantages.\n2. Clear motivation. \n\n## Weakness\n1. The paper’s primary impetus—that a “distribution discrepancy” between a post-trained generator and a general reward model constrains test-time scaling—is asserted but never substantiated with distributional distance metrics or error‐mode analyses\n2.  Although joint RL training is said to eliminate external verifiers, there is no measurement of inference latency, sampling overhead, or energy consumption to demonstrate a net practical benefit.\n3. Complexity - deploying a unified RL framework requires managing complex components—policy and value networks, replay buffers, scheduler tuning, distributed rollout servers—versus the far simpler pipelines of prompt-based self-consistency or supervised fine-tuning.\n\nQUESTIONS:\n- How well does self-verification generalize to non-mathematical domains such as code generation or open-ended commonsense reasoning?\n\n- How sensitive are results to hyperparameters of the dynamic verification reward and buffer size, and have the authors performed ablations on these components?\n\n- Recent discussions have revealed that even entropy-minimization alone can be a strong baseline. Can the authors provide comparison against those?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}