Title: Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: PRM

LLM

RL

Reasoning
Abstract: The development of reasoning capabilities represents a critical frontier in large language models (LLMs) research, where reinforcement learning (RL) and process reward models (PRMs) have emerged as predominant methodological frameworks. Contrary to conventional wisdom, empirical evidence from DeepSeek-R1 demonstrates that pure RL training focused on mathematical problem-solving can progressively enhance reasoning abilities without PRM integration, challenging the perceived necessity of process supervision.
In this study, we conduct a systematic investigation of the relationship between RL training and PRM capabilities. Our findings demonstrate that problem-solving proficiency and process supervision capabilities represent complementary dimensions of reasoning that co-evolve synergistically during pure RL training. In particular, current PRMs underperform simple baselines like majority voting when applied to state-of-the-art models such as DeepSeek-R1 and QwQ-32B. To address this limitation, we propose Self-PRM, an introspective framework in which models autonomously evaluate and rerank their generated solutions through self-reward mechanisms. Although Self-PRM consistently improves the accuracy of the benchmark (particularly with larger sample sizes), analysis exposes persistent challenges: The approach exhibits low precision (<10\%) on difficult problems, frequently misclassifying flawed solutions as valid. These analyses underscore the need for combined training with process supervision and continued RL scaling to enhance reward alignment and introspective accuracy. We hope these findings provide actionable insights for building more reliable and self-aware complex reasoning models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper challenges the conventional necessity of Process Reward Models (PRMs) in enhancing the reasoning capabilities of large language models (LLMs). Through empirical analysis, the authors demonstrate that reinforcement learning (RL) training alone, without explicit PRM supervision, can implicitly develop robust PRM capabilities in models like DeepSeek-R1 and QwQ-32B. These RL-trained models not only excel in problem-solving but also exhibit strong process-level judgment, outperforming models explicitly fine-tuned with PRM-labeled data. The study introduces \"Self-PRM,\" a framework where models autonomously evaluate and rerank their solutions using internal rewards. However, Self-PRM struggles with low precision on challenging problems, often misclassifying incorrect solutions. The findings suggest that PRMs may not be essential for complex reasoning, as RL training inherently fosters both problem-solving and process-supervision abilities.\n\nSTRENGTHS AND WEAKNESSES:\n#### Strengths\n\n- The idea is novel and interesting.\n- This paper provides insightful analysis on the relationship between RL, problem-solving abilities, and process supervision capabitlities. Recent works on LLM reasoning and RL rarely focus on the point of PRMs, but this paper provides valuable and interesting insights.\n- The paper is easy to follow.\n\n#### Weaknesses\n\n- The improvement of Self-PRM is not significant compared with majority voting and BoN w/ PRM.\n- This paper mainly focuses on mathematical reasoning tasks, which may limit the generality of the findings. It would be beneficial to explore the implications of the findings in other domains, such as coding.\n- This paper does not discuss limitations.\n\nMinor:\n- The Justification should be filled in the checklist.\n- L232: SSelf-PRM -> Self-PRM.\n\nI am willing to adjust the rating if the authors can address the weaknesses and the questions below.\n\nQUESTIONS:\nBesides the weaknesses above, further questions are as follows:\n\n- Self-REF is only evaluated using one non-reasoning model (Qwen2.5-32B-Instruct). If we use a weaker model, such as Qwen2.5-7B-Instruct or Qwen2.5-1.5B-Instruct, will the performance of Self-REF be worse? I am wondering if the performance of Self-REF relies on the reasoning abilties of the model since the improvement of Qwen2.5-32B-Instruct w/Self-REF is more significant on easy tasks (GSM8K) than difficult tasks (Omni-MATH).\n- How does the ProcessBench performance evaluated in Figure 1? Do the authors prompt the trained models to answer the first errorneous step in the reasoning process?\n- How does Self-PRM rerank the output? Is it the same way as the evaluation method of ProcessBench?\n- How do the authors divide the reasoning step in BoN w/ PRM experiments?\n- In Table 5, how do the authors choose these problem indices and why some places are filled with \"-\"?\n- Will the code be open-sourced? It would be helpful for the community to reproduce the results and further explore the findings.\n\nLIMITATIONS:\nThis paper does not discuss limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-05"}

comments: SUMMARY:
This paper investigates whether explicit process reward models (PRMs) are necessary for strong reasoning in language models. Through thorough experiments, the authors demonstrate that reinforcement learning (RL) can implicitly induce PRM-like abilities in large language models. They further propose “self-PRM,” a method that uses the model itself to judge and rerank its outputs, thereby acting as its own PRM. Self-PRM leads to improved overall performance, though it can exhibit low precision on particularly difficult problems.

STRENGTHS AND WEAKNESSES:
### Strengths
1. The experimental setup is comprehensive, covering a wide range of benchmarks, models, and evaluation methods.
2. I appreciate that the paper is framed more as a careful analysis rather than simply a new method introduction. The authors are transparent about the limitations of their proposed method (self-PRM).

### Weaknesses
I believe there is some misalignment between the paper’s framing and the experimental findings.
1. The results primarily show that external PRMs can be suboptimal for strong RL-trained models, but that self-PRM (using the model itself) is still beneficial. This may reflect the limitations of current external PRMs rather than the broader claim that PRMs are unnecessary. In other words, the experiments demonstrate that current PRMs may be insufficient, but not that process-level guidance itself lacks value.
2. The results indicate that gains in process judgment often precede gains in answer accuracy during RL training, which suggests that guidance via process-level supervision could still be helpful.

QUESTIONS:
Does models with self-PRM suffer from low precision on difficult problems more than no-PRM? It is not clear from the paper. Maybe I missed it.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates whether explicit Process Reward Models (PRMs) are necessary for developing strong reasoning capabilities in large language models, or if pure reinforcement learning (RL) training can implicitly induce these capabilities. The authors conduct a systematic empirical study demonstrating that RL-trained models like DeepSeek-R1 and QwQ-32B develop robust PRM capabilities without explicit process-level supervision, often outperforming models explicitly trained with PRM data. The study reveals a statistically significant correlation between problem-solving proficiency and process supervision capabilities through chi-square analysis. The authors introduce Self-PRM, where models use their internal reward signals to rerank their own outputs, showing consistent improvements over external PRMs and majority voting, particularly at higher sample sizes. However, the analysis reveals limitations in Self-PRM's precision on challenging problems, highlighting the need for better reward alignment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- This work provides some interesting insights that shed light into the future of PRM\n- Comprehensive empirical evaluation: The study covers multiple datasets (GSM8K, MATH, OlympiadBench, OmniMath) and various model types, providing robust evidence for the claims\n- I especially like the Chi-square tests provide statistical validation of the correlation between problem-solving and process supervision capabilities across models\n- Experiments are well-designed: The RL training experiments with learning curves effectively demonstrate the co-evolution of accuracy and PRM capabilities during training\n- Self-PRM offers an alternative to external PRMs with demonstrated usefulness\n\nWeakness:\n\n- The scope of this paper is limited to the math reasoning domain due to the selection of MATH data as the evaluation dataset. It is unclear if the conclusion of this paper will transfer to other domains (e.g., coding)\n- Not really a weakness but it'll be interesting to see if self-PRM can be used to further improve the policy its self by acting as a PRM during RL\n\nQUESTIONS:\n1. Generalization across domains (as mentioned in the weakness section): How do these findings extend to non-mathematical reasoning tasks such as coding\n2. RL training specifics: The paper shows results for one RL training setup (DAPO with Qwen2.5-7B-Base). How sensitive are the findings to different RL algorithms (PPO, DPO)? Is it possible that different algorithm might exhibit different pattern?\n3. Efficiency-wise: What is the computational overhead of Self-PRM compared to external PRMs and majority voting (e.g., wall clock time would be nice to have)? How does this scale with the number of candidate solutions and model size?\n4. Threshold: How sensitive is Self-PRM performance to the selection of confidence thresholds or scoring mechanisms? Are there principled ways to set these parameters?\n\nLIMITATIONS:\nno.\nJustification is \"[TODO]\" in the paper check list", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work analyzes RL algorithms that focus solely on outcome correctness, such as GRPO, which can enhance LLMs' capability of process supervision. By comparing different versions of open-source LLMs and conducting RL training experiments, the author points out that verifying the final result can still endow models with process supervision abilities. This work also highlights a potential correspondence between problem solving ability and process supervision capability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The analysis in L141 \"Problem-Solving V.S. Judgment\" is insightful.\n2. This work is well-written and easy to understand.\n\nWeaknesses:\n1. L207–209 state: \"indicating that Self-REF can inject meaningful structure into process modeling where it is otherwise absent. In contrast, RL-trained models such as QwQ-32B and DeepSeek-R1 exhibit marginal or negative changes in F1 when using Self-REF\". However, Qwen2.5-32B-Instruct has undergone an RL training process. While R1-Distill-Qwen-32B has not, the performance gain is already minimal. Therefore, the experimental results do not sufficiently support the stated conclusion.\n\n2. AIME benchmark only have a few questions, and the performance differences reported in Sec 4.3 are not particularly significant. The conclusion that \"External PRMs provide little to no benefit for RL-based reasoning models\" may not be broadly generalizable.\n\nQUESTIONS:\nTable 3, for the claim \"while RL-trained models (QwQ-32B, DeepSeek-R1) show minimal or negative gains, suggesting limited benefit when process reasoning is already reinforced.\", another reason could be the QwQ and R1's performance is higher than Qwen2.5-32B-Instruct and R1-Distill-Qwen-32B. Similarly, the advantage of Qwen2.5-32B-Instruct is 18.8, but the advantage of R1-Distill-Qwen-32B is only 2.6. How about testing a smaller RL-trained model?\n\nIt's better to provide more statistical results of Sec 4.4. I think drawing a curve of the pass ratio and precision could be very helpful to support the claim \"Self-PRM also suffers from low precision on hard problems, misclassifying many incorrect solutions as correct\"\n\nLIMITATIONS:\nThe statement made in this work is generalizable, but experiments only involve the mathematical domain.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nChallenging the necessity of external Process Reward Models (PRMs), this paper's systematic investigation reveals that pure reinforcement learning not only improves an LLM's problem-solving skills but also synergistically co-develops its internal process supervision capabilities, suggesting that continued RL scaling is a more direct path to advancing complex reasoning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- This paper presents a comprehensive empirical study with a very broad scope. It investigates a range of topics, from the limitations of state-of-the-art Process Reward Models (PRMs) in LLM reasoning to the advantages and disadvantages of a Self-PRM approach. Following the precedent set by Deepseek R1, this work further demonstrates that an external PRM may not be essential for successfully scaling reasoning abilities via reinforcement learning.\n- The experimental finding presented in Figure 1 is particularly interesting. It reveals a strong correlation where the improvement in a model's problem-solving ability is accompanied by an enhancement of its process supervision capabilities. This insight could potentially unlock novel applications, such as using the reasoning model itself as a PRM.\n\nWeaknesses\n- My main criticism is that the paper attempts to tackle too many broad questions at once. This ambitious scope seems to have come at the cost of depth, leading to some topics being underexplored and a general lack of profound insights.\nThe paper does not adequately explain why RL scaling enhances the ability of a reasoning model to act as a PRM. Does this process make the internally generated reward signals more robust against being 'hacked' by the policy, or are there other underlying mechanisms at play? I was expecting a dedicated section (e.g., a Section 3.4) with targeted experiments to delve into this crucial question. Given that the main body of the paper is currently under the page limit, there appears to be ample space to include such a vital analysis.\n- The study overlooks a critical scenario that merits investigation. A key potential advantage of an external PRM is its ability to provide denser supervision signals during RL training compared to a simple, binary rule-based verifier. However, the paper includes no experiments to test this. If an external PRM can guide the RL process more effectively than a rule-based verifier, it would demonstrate that PRMs are not entirely obsolete and still hold significant value—a conclusion that this paper currently seems to dismiss.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}