Title: Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Self-Rewarding

Reinforcement Learning

Large Language Models
Abstract: Recent advances of Reinforcement Learning (RL) have highlighted its potential in complex reasoning tasks, yet effective training often relies on external supervision, which limits the broader applicability. In this work, we propose a novel self-rewarding reinforcement learning framework to enhance Large Language Model (LLM) reasoning by leveraging the consistency of intermediate reasoning states across different reasoning trajectories. Our key insight is that correct responses often exhibit consistent trajectory patterns in terms of model likelihood: their intermediate reasoning states tend to converge toward their own final answers (*high consistency*) with minimal deviation toward other candidates (*low volatility*). Inspired by this observation, we introduce CoVo, an intrinsic reward mechanism that integrates ***Co***nsistency and ***Vo***latility via a robust vector-space aggregation strategy, complemented by a curiosity bonus to promote diverse exploration. CoVo enables LLMs to perform RL in a self-rewarding manner, offering a scalable pathway for learning to reason without external supervision. Extensive experiments on diverse reasoning benchmarks show that CoVo achieves performance comparable to or even surpassing supervised RL. Our code is available at https://github.com/sastpg/CoVo.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work authors propose an unsupervised reinforcement learning (RL) procedure for LLM finetuning using reasoning stability as a reward function. For each step of a reasoning trajectory $\\tau_i$, authors define a distance to final answer as the average negative log likelihood (log perplexity) of that answer. Then for an entire trajectory, they define the consistency metric $Con(\\tau_i)$ as the average number of steps which are `closer' to its own answer than alternatives, and a volatility metric $Vol(\\tau_i)$ that measures how late in the reasoning process the model strays toward other answers. Using a combination of these metrics as well as a curiosity penalty, the authors arrive at their unsupervised reward function (CoVo). This reward function is demonstrated on average to outperform other unsupervised methods (EMPO, TTRL) and even some supervised methods (GRPO, Reinforce++) as measured in terms of accuracy on popular reasoning tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Significance: Analyzing the internal dynamics of CoT reasoning is a very relevant line of work\n* Consistent Experimental Results: The difference in correct and incorrect trajectories are well motivated in Fig. 2, and while the improvement in accuracy on individual tasks is marginal it seems consistent across all tasks.\n* Some motivating theory: Proposition 5.1 provides good explanation as to why majority voting rewards can fail\n\nWeaknesses: \n* Accuracy Saturation: The fact that this method improves maybe ~0.5% on average over unsupervised baselines (Table 2), as well as the overall saturation across all methods seems to suggest a possible limit to the power of this method to discover new reasoning patterns.\n* I would be interested in seeing an in depth example of how individual reasoning trajectories qualitatively change before and after training. The authors mention in the appendix that this reward encourages \"more coherent and logically grounded\" reasoning but I would like more evidence here.\n* I appreciate the up front discussion of complexity $\\mathcal{O}(NTK)$ for this method, I would like more discussion comparing this to other unsupervised methods as this seems very expensive.\n* Proposition 5.2 seems like the standard ELBO derivation for variational inference, is the new insight here just that reasoning path $s$ is the latent variable?\n\nQUESTIONS:\n* How would authors explain the apparent `saturation' of accuracy across all finetuning methods in table 2? Is this a limitation of model size/ training resources?\n* Would this objective have an implicit bias toward longer reasoning chains, where the model has `made up its mind' but continues to generate reasoning steps?\n* Does this finetuning affect OOD and non-reasoning tasks such as text summarization?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes CoVo, a self-rewarding RL framework for LLM reasoning that leverages internal response trajectories rather than relying on external rewards or final answer matching. By modeling reasoning paths through consistency and volatility metrics, and encoding them in a vector space, the approach enables trajectory-level reward signals. A curiosity bonus is added to encourage exploration. Empirical results demonstrate that CoVo outperforms prior self-rewarding methods and sometimes even supervised RL.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n\n- The paper addresses a key limitation of RL for LLMs: reliance on expensive human labels or biased reward models.\n- It introduces a novel and intuitive intrinsic reward scheme based on consistency and volatility over intermediate reasoning steps.\n- The proposed vector-space representation enables stable learning and avoids reward hacking.\n- Results are strong across multiple reasoning benchmarks, including math and general QA.\n\n### Weaknesses:\n\n- Limited exploration: Although a curiosity bonus is introduced, the method still relies solely on self-generated samples without external perturbation or guided sampling, which may restrict exploration and cause convergence to locally optimal but incorrect reasoning patterns.\n- Dependence on initial policy: The effectiveness of the reward signals (consistency and volatility) hinges on the quality of the initial model. Poor base models may produce unreliable reasoning paths, reducing the robustness of self-rewarding learning.\n- Risk of reinforcing faulty reasoning: The framework may unintentionally reward trajectories that are confidently wrong but internally consistent, especially when no ground-truth supervision or reward model is used to validate answer correctness.\n- Simplistic distance metric: The reliance on token-level negative log-likelihood as the sole measure of distance may overlook deeper semantic or structural alignment between reasoning states and final answers, potentially limiting reward fidelity in complex tasks.\n\nQUESTIONS:\n1. Can the authors explain why likelihood distance was chosen over other possible measures (e.g., mutual information) for consistency/volatility scoring?\n\n2. How does CoVo handle failure cases where wrong answers have consistent but misleading trajectories?\n\nLIMITATIONS:\nWhile the authors claim no significant limitations, we believe the method may still suffer from issues related to exploration, dependency on initial policy quality, and distance metric simplicity, as discussed above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CoVo (Consistency and Volatility), a self-rewarding reinforcement learning framework for enhancing LLM reasoning capabilities without external supervision. The key insight is that correct reasoning trajectories exhibit distinct patterns: high consistency (intermediate states lead toward their final answer) and low volatility (minimal deviation toward alternative answers). The authors propose an intrinsic reward mechanism that combines these trajectory-level features through robust vector-space aggregation, supplemented by a curiosity bonus for exploration. Experiments on mathematical and commonsense reasoning benchmarks show that CoVo achieves performance comparable to supervised RL methods, improving Qwen2.5-3B-Instruct from 63.6% to 68.2% on MATH dataset.\n\nSTRENGTHS AND WEAKNESSES:\n- The biggest concern: Despite including tests on other domains, the training setup is restricted to Qwen on math. Therefore, I am not able to assess if the improvements come from the algorithm, or simply from the base model, as pointed out by recent work [1,2] rethinking the effectiveness of unsupervised/weakly-supervised RL methods. The gain may simply come from eliciting certain meta-capabilities [3] and then using these general reasoning skills to solve problems, both in-domain and out-of-domain, or may just learn the format.\n- The improvements are very marginal compared to baselines. Specifically, compared to TTRL, the difference is usually (or always) smaller than 1.0%, and it even underperforms in many cases. RL is known to be sensitive, and so is the evaluation. In practice, even an update on training infra or python library can easily lead to a 2% accuracy change on MATH. This is the second aspect why I am not able to assess the effectiveness of the algorithm proposed in this paper.\n- For the analysis in section 3.2.1, the paper says “for correct answers, the reasoning paths rapidly approach to the final answer within a few steps. In contrast, incorrect responses typically exhibit delayed convergence, with intermediate states remaining relatively distant from the final answer for an extended period.” However, the analysis does not ablate difficulty. That is, easier problems are more likely to get correct, and solving them requires fewer steps. A more rigorous analysis is required to set up such a causal correlation.\n\n[1] Spurious Rewards: Rethinking Training Signals in RLVR. Shao et al. 2025.\n\n[2] The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning. Agarwal et al. 2025.\n\n[3] Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs. Gandhi et al. 2025.\n\nQUESTIONS:\n- If this method underperforms the simpler methods like TTRL, why do we need a more complex method that’s harder to implement?\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes CoVo, a novel self-rewarding reinforcement learning framework for improving the reasoning abilities of LLMs without relying on external supervision. The method utilizes the consistency and volatility of intermediate reasoning steps across sampled trajectories to construct intrinsic rewards, and introduces a curiosity bonus to encourage exploration. The authors conduct extensive empirical studies and theoretical analysis, showing that correct answers tend to follow consistent and low-volatility reasoning paths.

STRENGTHS AND WEAKNESSES:
Strength:

1. The paper introduces a principled and original approach for constructing self-reward signals using internal trajectory structure, addressing a key limitation in prior self-rewarding methods.
2. The authors provide a solid theoretical justification by framing the learning objective as a variational inference problem over reasoning paths, adding conceptual clarity to the method.
3. The paper is well written and easy to follow.

Weakness:
1. The computation of distance matrices and multiple trajectory sampling per prompt may lead to significant training costs.
2. Experiments are restricted to relatively small models, generalization to larger models (e.g., 13B+) or production settings is unclear.

QUESTIONS:
1. How sensitive is CoVo to the number of sampled reasoning paths per query? Could performance degrade with fewer samples?
2. Would combining CoVo with external reward models further enhance performance, or cause interference?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05