Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMs
Authors: Zhangyin Feng
Qianglong Chen
Ning Lu
Yongqian Li
Siqi Cheng
Shuangmu Peng
Duyu Tang
Shengcai Liu
Zhirui Zhang
Venue: NeurIPS 2025 poster ()
Keywords: PRM
LLM
RL
Reasoning
Primary area: reinforcement_learning

Abstract: The development of reasoning capabilities represents a critical frontier in large language models (LLMs) research, where reinforcement learning (RL) and process reward models (PRMs) have emerged as predominant methodological frameworks. Contrary to conventional wisdom, empirical evidence from DeepSeek-R1 demonstrates that pure RL training focused on mathematical problem-solving can progressively enhance reasoning abilities without PRM integration, challenging the perceived necessity of process supervision.
In this study, we conduct a systematic investigation of the relationship between RL training and PRM capabilities. Our findings demonstrate that problem-solving proficiency and process supervision capabilities represent complementary dimensions of reasoning that co-evolve synergistically during pure RL training. In particular, current PRMs underperform simple baselines like majority voting when applied to state-of-the-art models such as DeepSeek-R1 and QwQ-32B. To address this limitation, we propose Self-PRM, an introspective framework in which models autonomously evaluate and rerank their generated solutions through self-reward mechanisms. Although Self-PRM consistently improves the accuracy of the benchmark (particularly with larger sample sizes), analysis exposes persistent challenges: The approach exhibits low precision (<10\%) on difficult problems, frequently misclassifying flawed solutions as valid. These analyses underscore the need for combined training with process supervision and continued RL scaling to enhance reward alignment and introspective accuracy. We hope these findings provide actionable insights for building more reliable and self-aware complex reasoning models.

Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMs
Zhangyin Feng" Huawei Technologies Ltd. fengzhangyin@huavee com
Qianglong Chen" Huawei Technologies Ltd. chenqianglong a ai@gmail.... com
Ning Lu HKUST
nluab0cse. ust hk
Yongqian Li Huawei Technologies Ltd. Liyonggian20huarei. com
Siqi Cheng
Shuangmu Peng
Huawei Technologies Ltd.
Huawei Technologies Ltd.
chengaiqi4@huarer com
pengshuangmuuhuavei.. com
Duyu Tang
Shengcai Liu
Huawei Technologies Ltd. tangduyu@huavei.. com
Guangdong Provincial Key Laboratory of Brain-Inspired Intelligent Computation, Department CSE, SUSTech
liusc3@sustech. edu.
Zhirui Zhang Huawei Technologies Ltd. crustc110gmaile com
Abstract
The development of reasoning capabilities represents a critical frontier in large language models (L.LMs) research, where reinforcement learning (RL. and process reward models (PRMs) have emerged as predominant methodological frameworks. Contrary conventional wisdom, empirical evidence from DeepSeek-RI demon- strates that pure RL training focused on mathematical problem-solvingg can pro- gressively enhance reasoning abilities without PRM integration, challenging the perceived necessity of process supervision. In this study, we conduct systematic investigation of the relationship between RL training and PRM capabilities. Our findings demonstrate that problem-solving proficiency and process supervision capabilities represent complementary dimensions reasoning that co-evolve syn- ergistically during pure RL training. In particular, current PRMs underperform simple baselines like majority voting when applied to state-of-the-arr models such DeepSeek-R1 and QwQ-32B. To address this limitation, we propose Self-PRM. an introspective framework in which models autonomously evaluate and rerank their generated solutions through self-reward mechanisms. Although Self-PRM consistently improves the accuracy of the benchmark (particularly with larger sample sizes), analysis exposes persistent challenges: The approach exhibits low precision (<10%) on difficult problems, frequently misclassifying flawed solutions as valid. These analyses underscore the need for continued RL scaling to improve reward alignment and introspective accuracy. Overall, our findings suggest that PRM may not essential for enhancing complex reasoning, pure RL not only improves soblem--olling skills but also inherently fosters robust PRM capabilities. We hope these findings provide actionable insights for building more reliable and self-aware complex reasoning models.
"qqual contribution.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Recent advances in reinforcement learning (RL) scaling have substantially improved the ability of large language models (LLMs) to perform complex reasoning tasks. This progress is exemplified by the emergence of state-of-the-an (SOTA) models such as Kimi 1.5 [25]. OpenReasonerrZero [TO],
DeepSeek-RI [8] QwQ-32B B2) 26], and Claude-3.7-Somee []. all of which demonstrate remark-
able problem-solving capabilities achieved through pure RL training.
have emerged as an alternative paradigm for reasoning supervision, with the aim of evaluating and optimizing the quality of intermediate reasoning steps. Despite their conceptual appeal, PRMs encounter three fundamental limitations: (1) the inherent ambiguity in defining granular reasoning steps, (2) prohibitive annotation costs, and (3) systemic vulnerability to reward hacking mechanisms. In particular, the DeepSeek-RI technical report highlights these limitations 8,, demonstrating that PRMs did not yield significant gains in the performance of complex reasoning. However, recent work continues to explore the potential of PRMs. For example, PRIME introduces implicit reward mechanisms to bypass the need for labeled steps, GenPRM 37 uses code verification and relative progress estimation to improve PRM performance, and Qwen PROCESSBENCH (38) offers a benchmark to assess the quality of the reasoning trace. These efforts underscore the ongoing interest in PRMs as path to better reasoning supervision. However, a crucial question remains largely unexplored: Is explicit PRM training truly necessary for developing models capable of process-level reasoning? ? In other words, can RL training alone- - in the absence of any process supervision lead to models that are both strong problem solvers and capable evaluators of reasoning quality?
Alongside this RL-driven progress, process reward models (PRMs)
16
To answer this, in this work, we present the first systematic empirical study exploring the intrinsic connection between RL training and PRM capabilities. Our key insight is that problem-solving and process supervision are not orthogonal skills, but rather complementary competencies that co-evolve during pure RL training. Through a series of experiments, we examine how reasoning models trained solely with rule-based rewards develop strong process-level judgement capabilities, without access to fine-grained supervision. Through a series of controlled experiments we demonstrate that RL-trained models like DeepSeek-RI and QwQ-32B exhibit strong process judgement abilities, exceeding those of models explicitly trained with PRMs.
math reasoning tasks,
In addition, we find that existing PRMs provide little to no benefit when applied to these models for reranking, often underperforming compared to simple heuristics such as majority voting. To address this, we introduce Pelf-PRM, an introspective approach where a model leverages its own internal reward signal to re-rank sampled outputs. Self-PRM consistently improves performance particularly at higher sample sizes - by aligning candidate selection with the model's own reasoning preferences. Despite these improvements, detailed analysis reveals that Self-PRM suffers from low precision on challenging problems, often misclassifying incorrect solutions as correct. Stronger models, such as DeepSeek-RI and QwQ-320 exhibit more reliable self-evaluation, but the issue persists, highlighting the limitations of introspective scoring when reward alignment is imperfect. These findings challenge conventional assumptions about the necessity of process supervision and suggest that lightweight, introspective training signals whether through self-evaluation or continued RL scaling may provide viable path forward.
In summary, our main contributions are as follows:
We provide the first systematic study demonstrating that RL training alone without any process-level supervision - can endow models with strong PRM capabilities.
We empirically show that existing PRMs fail to improve performance when applied strong RL-trained models, and often underperform simple baselines such as majority voting.
We introduce the Self-PRM framework, in which models rerank their own outputs using internal reward signals, achieving consistent performance gains while also revealing limitations in precision that highlight the need for improved reward alignment.
2 Preliminaries
Reinforcement Learning. RL provides a framework in which an agent learns to make sequential decisions by interacting with environment to maximize cumulative rewards. Formally, RL is

---

Table 1: Evaluation results of different LLMs on PROCESSBENCH, We report the error, correct and F1 score of the respective accuracies on erroneous and correct samples.
GSM8K
MATH
OlympladBench Correct
OmmiMATH
Model
Average
Ervor
Proprictory LLMs (Crite)
GPT-10-0806
70.0
91.2
79.2
54.1
76.6
63.5
45.8
58.4
514
45.2
65.6
53.5
61.9
ol-mnini
88.9
97.9
95.2
83.5
95.1
88,9
80.3
95.6
87.2
74.8
91.7
82.4
87.9
DeepSeek Disnilee prompted Generative
BL.DiisillQQenn-7B
73.5
70.1
Distill--ween333B
74.4
98.4
84.7
71.5
89.4
79.5
64.3
85.5
73.4
59.0
83.8
69.3
76.7
DeepSeck-BB R1
84.1
95.3
89.3
82.3
91.1
86.5
78.2
86.4
82.1
71.7
80.9
76.0
83.5
Models, prompted Generative PRM
Qwen2.5 Instruct
49.8
52.1
97.33
19.0
96.3
Owen2.5 1rren2--7B-lnstrue
40.6
33.2
36.5
30.8
45.1
36.5
26.5
33.9
29.7
26.2
28.6
27.4
32.6
0wen225- Instruct
54.6
94.8
69.3
38.4
87.4
53.3
31.5
78.8
4.5.0
28.3
76.3
41.3
52.2
Qwen22553320 5--2tt-nnttu
19.3
97.9
65.6
36.7
95.8
53.1
25.3
95.8
40.0
24.1
92.5
38.3
49.3
Qwen2.5-7288 I-sstnntt
62.8
76.2
46.3
61.8
92.6
54.6
36.6
90.9
52.2
QwQQ-38B
87.3
Other Models with Training PRM duta, az Diseriminative PRM
Skywark-PRM-LL5 PRM-1.5B 1.5B
50.2
11.5
59.0
37.9
65.2
48.0
15.4
26.0
19.3
13.6
32.8
19.2
36.4
Math Shepherd PRM-7B
32.4
91.7
47.9
18.0
82,0
29.5
15.0
71.1
24.8
14.2
73.0
23.8
31.5
RI.HFko 7RMM-istral-888
33.8
$0.4
72.2
33.4
43.1
9.6
45.2
RLHFFos-PRM--D PRM-Deepeeca-8B
38.8
51.0
Skywark-PRM-77 PRM-7B
61.8
82.9
70.8
43.8
62.2
53.6
17.9
31.9
22.9
14.0
41.9
21.0
42.1
Qwen2.5 Math- 7B- Maath--hhhhhhhttt
16.4
95.9
62.5
18.9
96.6
31.6
7.4
93.8
13.7
1.0
95.0
7.7
28.9
Qwen2.5-Mamt Mamh-- 1B-PR000KK
53.1
95.3
68.2
48.0
90.1
62.6
35.7
87.3
50.7
29.8
86.1
44.3
56.3
Retrievall2M-7B
64.7
88.1
74.6
75.6
71.1
56.0
65.2
60.2
52.8
62.7
57.3
Qwen2.5 Math
82.4
55.2
Qwen2.5 Owen2.5-Mauh- 3-Maub-rRM--223
78.7
97.9
87.3
74.2
88.2
80.6
67.9
82.0
74.3
64.8
78.8
71.1
78.3
typically modeled as a Markov Decision Process (MDP), defined by the tuple where S is the state space, A the action space, P the state transition dynamics, R the reward function, and y the discount factor. In the context of LLMs, RL widely used to align model behavior with human preferences. One prominent approach is Reinforcement Learning from Human Feedback (RLHF) (20), where a reward model (RM) is trained to approximate human judgements generated outputs. The base language model is then optimized using RL to maximize the reward provided by this RM. Several methods have been proposed to implement this optimization process effectively. including PPO (22). DPO (21). GRPO 23). DAPO (35) and VAPO [36 etc. These methods differ in how explicitly they use reward signals, but all share the goal of training LLMs to produce outputs that are more helpful, truthful, and aligned. Notably, recent RL-based models
including DeepSeek- R1 8, Kimil.5 [25] QwQ-32B 226 OpenReasonerrZero I0 achieve state- of-the-art performance
complex reasoning tasks without process-level supervision. Meanwhile,
some works 77 12 24 14 5 are exploring RI. with tools for efficient reasoning. While Retool [5] use DAPO-MATH-TT 17k and DAPO for RI. training to guide L.MSs towards optimal strategies for leveraging external computational tools during reasoning, ToRL. [14] leverage 7B-base to improve the complex reasoning ability.
are designed to evaluate the reasoning process of LLMs, just the final answer. They provide rewards scores for each intermediate step in a reasoning path, helping models learn to think in a more structured, logical, and interpretable way. While PRM800K [13] proposed train the reward model, AlphaMath 2 leverage Monte Carlo Tree Search (MCTS) to unleash the potential of a well-pretrained LLMM to autonomously enhance its mathematical reasoning. PRM is especially useful for tasks that require multi-step problem solving, such as math or logical reasoning. Compared to traditional outcome- supervised reward models [ 3 that focus only on whether the final answer correct, PRMs can give more detailed feedback, making them valuable for improving model alignment and transparency. They are typically trained on labeled reasoning examples, learning distinguish good reasoning steps from incorrect or irrelevant ones. Despite their potential, PRMs face challenges such as limited high- quality reasoning data, difficulties in defining stepwise correctness, and risks of models exploiting the reward function. To improve PRMs, researchers are exploring better data collection methods, combining them with reinforcement learning algorithms (e.g., PPO 22),, ) and even letting models evaluate their own reasoning enhance learning 2 or extend them to visual reasoning [29].
Process Reward Models. PRMs 1
16
I
29
30
39
27
13

---

Table 2: Chi-square test results for model performance on PROCESSBENCH, including GSM8K, MATH, OlympiadBench, and OmniMath datasets. We classify the math problems into True and False categories, depending on whether the LLMs can provide correct solutions. Similarly, we categorize the judgment results into Correct and Error categories, based on whether the LLMs deliver the correct judgments.
Sobution
GSM&K
OLYMPIADBENCH
OMNIMATH
Coneec
Envor
P
Correci
Erioe
P
Correct
Eroe
P
Conectt
Erie
P
Correct
Ener
Qwen2.5-328
Tree
327
2.9c-5
399
201
296
131
243
1467
545
False
11
63
137
351
134
428
960
Rl-Dieill-Qwem--tRR
True
515
47
0.01
735
159
3.08e-
439
146
3.20e-
444
171
0
1963
359
QuQ-32R
Thee
838
0011
826
147
0.125
907
110
0.003
545
149
3.9--
2214
468
False
20
7
292
91
206
530
208
DeepSeck-R1
Thue
330
0025
831
141
0,039
1.15e-
0.012
2138
474
Fabe
19
20
8
284
118
224
108
547
241
3 Intrinsic Connection between RL Training and PRM
3.1 Experimental Setup
We evaluate the PRM capabilities of various models using PROCESSBENCH publicly available benchmark designed to assess the reasoning abilities of LLMs across diverse domains of mathe- matical problem-solving. PROCESSBENCH comprises four distinct datasets: GSM8K [3), MATH, OLYMPIABBENCH [9], and OMNIMATH [6]. For each dataset, we report three metries: Error Rate, Correct Rate, and Fl Score. We categorize the evaluated models into three main groups:
Proprietary LLMSs (Critic Models): This group includes closed-source models, such as GPT-40-0806 11 and ol-mini [9). which serve as reference points.
Qwen-series and DeepSeeksseries Models (Prompted Generative PRM): These models are not directly fine-tuned on RM or PRM datasets; instead, they are prompted as PRMs. Representative models include (wen2.5-Mathh7B-Instte
[33 and DeepSeek-RI (8)
Models Fine-Tunne with PRM Data (Discriminative PRM): These models are explicitly trained on datasets constructed with PRM-style supervision. Models include Math-Shepherd- PRM (28). Skywork-PRMM-7 18] RetrievalPRM-77 40). and RLHFlow-PRM-Misthe 8B 31
Additionally, to further validate the emergence of PRM capabilities during RL training, we conduct experiments using Qwen2.5-7B-Br. with DAPO- (35), employing DAPO as the policy optimization algorithm. The hyperparameters are set follows: learning rate of le-6, batch size of 256, prompt length of 2048, output length of 10240, group size of 16, clipping ratio (high) of 0.28, overlong buffer length of 4096, and overlong penalty factor of 1.0.
3.2 Empirical Analysis on PROCESSBENCH
The evaluation results of different LLMs on PROCESS- BENCH are presented in Ta which summarizes the PRM performance of various models across the GSM8K, MATH, OLYMPIADBENCE models trained purely with reinforcement learning (RL), such as DeepSeek-RI and QwQ-32B, exhibit strong inherent PRM capabilities, despite not being explicitly supervised with PRM-labeled data. These RL-trained models consistently outperform discriminative PRM models that were fine-tumed directly on PRM-annotated datasets. For instance, DeepSeek-RI achieves an average score of 83.5, while QwQ-32B achieves 83.7, both surpassing all models trained with PRM labels in discrininative setting, including Skywork-PRM-N 1.t2255-MMt--RMMM
Evaluation Results on PROCESSBENCH.
and OMNIMATH datasets. Our findings indicate that
(42.1), 2ree2.5-MMtt-PRMM. (73.5), and (78.3). This suggests that PRM capabilities can emerge implicitly through RL, without the need for explicit PRM supervision.
Problem-Solving V.S. Judgment: A Statistical Perspective. We further conduct chi-square tests to evaluate the correlation between problem-solving proficiency and process supervision capabilities in the same model. Specifically, we categorize the mathematical problems in ProcessBench into two groups for each model: (1) True: Problems for which the L.M generates correct solutions; (2) False: Problems where the LLM fails to produce correct solutions. Similarly, we classify the model's

---

GSMBK
Sccuracy
72
Acuuscy
-5
70
76
20
25
Procosa Boatcn F2
70
06
03
30
100
150
200
250
50
100
150
200
250
Steps
Steps
OlympiadBench
OmniMath
4ccuIaCy
Proces
Accumtc
Process
23.0
35
Process Bonch
Accuncy
24
Accuncyy
22.0
21.0
Figure 1: Problem Solving Accuracy and Fl Results of the RL-trained model on ProcessBench. We leverage Qwen2.5-77-base for RL training.
judgment outcomes into: (1) Correct:
Cases where the LLM accurately evaluates the correctness of the paired problem-solutie instances in ProcessBench; (2) Error: Cases where the LLM's judgment is incorrect. Based on these categorizations, of the four groups and calculate the p-value to test the hypothesis that problem-solving proficiency independent of each other. As summarized in Table all
count the total number of instances in each
and process supervision capabilities models exhibit statistically significant correlations (p < 0.05) in the aggregate analysis, strongly rejecting the null hypothesis. This indicates strong correlation between problem-solving proficiency and process supervision capabilities. Specifically, for the GSM8K dataset, three models achieve significant results: Qwen2.5-32B (p 2.9 10-9), QwQ-32B (p 0.011), 1), and DeepSeek-RI did not reach statistical significance (p = 0.071). For the MATH dataset, all models except QwQ-32B exhibited significance: Qwen2.5-32B (p 0), R B1-Distill-Qwen-B3 32B (p = 3.08 x 10-3), and DeepSeek-RI (p = 0.039). QwQ-32B's performance non-significant (p 0.125). For the OLYMPIADBENCI and OMNIMATII datasets, all models achieve statistically significant results. The chi-square tests conclusively validate that problem-solving proficiency and process supervision capabilities are intrinsically linked in LLMs, and this correlation persists across diverse mathematical reasoning tasks.
(p 0.025), whereas LiDDiiilll-Qee--3222
Takeaway
Empirical results demonstrate that RL-trained models (e.g.. DeepSeek-R1 and QwQ-32B) develop robust PRM capabilities without explicit PRM supervision, consistently surpassing discrimina- tive PRM baselines. Chi-square analysis (p 0.05) confirms statistically significant positive correlation between process-level judgment accuracy and problem-solving proficiency.
3.3 Can RL Training Improve PRM Capability?
Since the RL-trained models shows strong PRM capability, further conduct the experiment to observe the change of PRM capability during RL. training. As illustrated in Figure T we analyze the learning curves of the RL-trained model through four mathematical reasoning benchmarks (GSM8K.
5

---

MATH, OLYMPIADBENCH, and OMNIMATH), evaluating both final-answer accuracy (solid blue lines) and ProcessBench F1 scores (red dashed lines). The results demonstrate consistent and parallel enhancement in both final-answer accuracy and process judgement ability as training steps increase. salient trend is that gains in process judgement often precede or surpass those in accuracy, particularly in the early training phases. For instance, on the MATH and GSM8K benchmarks, the Fl score on ProcessBench increases sharply within the first 80 steps, while accuracy improves more gradually. This suggests that RL enables the model to internalize and identify valid reasoning structures before fully mastering the corresponding solution strategies. such OLYMPIADBENCH, the F1 score on ProcessBench improves consistently, whereas accuracy exhibits greater volatility. This divergence highlights the value of process-level learning as a stable and transferable signal, even when final-answer correctness is harder to optimize. Similarly, in OMNIMATH, where domain generalization is required, the model achieves modest but consistent improvements in both metrics, indicating the scalability of process supervision under domain shift. These findings indicate that problem-solving and process supervision are not orthogonal skills, but rather complementary competencies that co-evolve during pure RL training.
more challenging problems,
Takeaway 2:
RL improves both the accuracy of problem solving and the PRM capability in a coordinated manner. Crucially, improvements in process judgement often emerge earlier and more steadily than accuracy, highlighting RL's capacity to implicitly foster interpretable reasoning even in complex or out-of-domain tasks.
4 Explore the Performance Limit of Reasoning Models
4.1 Experimental Setup
In order to further explore whether a model's own problem-solving ability can enhance its capability PRM, we propose the self-reference (Self-REF) paradigm, where the model's generated solutions are used as reference signals to supervise PRM alignment. Using Qwen2.5-32B-Insst Qwen-32B, QwQ-32B, and DeepSeek-RI
R1-Distill- the base model, we evaluate two variants on the PROCESSBENCH benchmark (:4.2): (1) we prompt these models as generative PRMs, and (2) a Self-REFF-enhanced PRM incorporating the model's own solutions as supervisory signals.
Meanwhile, we evaluate whether PRMs can enhance the performance of strong reasoning models using two experimental settings ( (4 (1) BoN w/ PRM. where use -wen2.5-MATH-PRM.. to select the best output from A = 8, 16, 32, 64. (2) BoN w/ Self-PRM, where each model (QwQ- 32B, DeepSeek-RI 1) reranks its own outputs using internally derived reward scores. Evaluations are conducted on AIME24, AIME25, and CNMO24, with performance compared across direct sampling (Pass @ k), majority voting, BoN with external PRM, and BoN with Self-PRM. To analyze the limitations of Self-PRM, we further compute the number of solutions labeled correct by Self-PRM (SPRM) and the subset that truly correct (STp), allowing to quantify precision across individual problem indices.
4.2 Improving PRM Capability with Self-REF
To understand when self-generated reasoning traces serve as effective supervisory signals for PRM alignment, we access the impact of Self-REF across models with distinct training strategies as shown in Table We observe a clear trend: Self-REF substantially improves PRM performance only for models that have not undergone RL. Specifically, the Fl score of Dwen2.5-32B-Instree
purely instruction-tuned tuned model, improves sigmificantly from 45.1 - 63.9 9 (+16 (+18.8) with Self-REF. This gain most pronounced on MATH (+15.6 F1) and OlympiadBench (+13.9 FI), indicating that Self-REF can inject meaningful structure into process modeling where it is otherwise absent. In contrast, RL-trained models such as QwQ-32B and DeepSeek-RI exhibit marginal or negative changes in F1 when using Relf-REF (e.g., QwQ-32B: 83.7 - 83.0; DeepSeek-R1: 83.5 + 81.1). This suggests that these models have already internalized process reasoning through reinforcement objectives, and additional weakly supervised signals from self-generated traces may introduce noise rather than improve fidelity. Interestingly, the distilled variant R Distill-Qwen-322B, which inherits process judgement behaviors through distillation rather than direct RL training, does benefit modestly from

---

Table 3: Evaluation results of several models on PROCESSBENCH with solutions as references, R1-Distill-Qwen QwQ-32B, DeepSeek-RI.
including Qwen2.5-32B Instruct,
OlympiudBench
OmniMATH
Model
Average
Error
Correct
F1
Error
Correct
F1
Ervor
Correct
F1
Ervor
Correct
F1
Qwen2.5-1. 1-.5..--.-Inntuu
23.2
18.9
12.4
24.5
16.5
12.3
1644
18.8
W/ REEF
174
10.4
13.0
14.0
19.7
33.6
12.1
9.0
27.8
13.6
13.8
Dwwen:5-7B-Inuruct
44.9
43.0
43.9
30.1
48.0
37.0
26.5
33.5
29.5
27.1
28.2
27.7
34.5
QwenZ 3-3 Instruct
43.0
97.9
59.8
35.3
95.66
29.0
22.4
9000
35.9
22.4
87.6
35.7
45.1
Self-RFF
72.9
50.7
64.6
35.4
24.5
79.3
37.4
63.9
R1-Disaill Qwen
74.4
98.4
84.7
71.3
89.4
79.5
64.3
85.3
73.4
59.0
83.8
69.3
76.7
Self-REF
77.3
96.4
85.8
76.1
92.4
83.4
68.8
90.9
78.3
58.0
87.1
79.3
QwQ-328
84.1
97.4
90.2
90.4
75.9
71.1
77.0
83.7
w/ Self REF
82.6
93.8
$7.8
81.6
88.9
85.1
77.2
85.3
81.0
71.8
83.8
77.3
83.0
DeepSock-RI
83.5
w/ RelF
81.2
93.8
87.0
83.3
87.7
85.5
74.0
79.9
76.8
70.2
79.7
74.6
81.1
ReF (F1: 76.7 79.3), further supporting that Self-REF effective in settings where explicit reward-driven reasoning supervision is absent or weak. For the small language models, we can observe that qwen2.5-7B-B Swwen.5-7B-inntrucc with Self-REF shows improvements on GSM8K and MATH, while its Fl scores decrease OlympiadBench and OmniMATH. In contrast, qwen2.5-1 1..e2.55--.B-instii .5B-instruct with Self-REF exhibits a decline in performance across all test sets. These results are consistent with our expectations: the effectiveness of Self-REF does depend on the base model's reasoning ability. A weaker instruction-tume -unonnd model would generate lower-quality reasoning traces, which would act as noisy incorrect supervisory signals.
Takeaway 3:
Self-Reference (Self-REF) improves PRM performance primarily for instruction-tuned models lacking RL supervision (e.g., +18.8 for Owen2.5-32B-IInttrcc]) while RL-trained models (QwQ-32B, DeepSeek-RI RI) show minimal negative gains, suggesting limited benefit when process reasoning is already reinforced.
4.3 Self-PRM: Can Current PRM Models further Enhance Reasoning Models?
Table 4: Comparison of sampling@k, (k 8, 16, 32, 64) across OwQ-32B and DeepSeck-R1 -1-on three benchmarks under three settings: direct sampling (Pass @), majority voting, BoN with External PRM (BoN w/ PRM), and BoN with Self-PRM.
QwQ-32B AIME25
DeepSeek RI A1ME25
Method
@k
AIME24
CNM024
AIME24
CNM024
1
73.3
66.7
77.8
80.0
63.3
77.8
8
90.0
73.3
88.9
93.3
86.7
88.9
Pass
16
90.0
80.0
94.4
93.3
90.0
88.9
93.3
86.7
94.4
93.3
90.0
88.9
64
93.3
93.3
94.4
93.3
93.3
88.9
8
80.0
76.7
83.3
80.0
76.7
77.8
16
83.3
76.7
83.3
83.3
76.7
83.3
Majority Voting
32
86.7
76.7
83.3
86.7
76.7
83.3
64
86.7
76.7
83.3
86.7
76.7
83.3
8
83.3
73.3
76.7
83.3
76.7
77.8
16
83.3
73.3
83.3
83.3
76.7
77.8
BoN w/ PRM
86.7
83.3
86.7
76.7
77.8
64
86.7
76.7
83.3
86.7
76.7
83.3
8
83.3
76.7
83.3
83.3
76.7
77.8
16
86.7
80.0
88,9
86.7
76.7
83.3
BoN w/ Self-PRM
32
90.0
80.0
88.9
86.7
83.3
83.3
64
90.0
80.0
83.3
86.7
83.3
83.3

---

To evaluate whether current state-of-the seef-thh-an art PRMs can enhance the performance of reasoning models in complex mathematical reasoning tasks, we conduct experiments on AIME24, AIME25, and CNM024, using
as the PRM for reranking. Specifically, we apply the Best-of-N with PRM (BoN PRM) strategy, which selects the highest-scoring solution based on the PRM's minimum-step reward among the candidates. As shown in Table f models that already exhibit strong reasoning abilities through RL, such as QwQ-32B and DeepSeek-RL, do not benefit from external PRMM-asedd reranking compared to majority voting. Across all datasets and sampling sizes = 8, 16, 32, 64, BoN w/ PRM achieves performance that is comparable to or slightly worse than majority voting. For example, on AIME25 with QwQ-32B, the accuracy plateaus at 76.7 for both BoN w/ PRM and majority voting across all values of k. Similarly, DeepSeek- R1 gains no measurable improvement from PRM reranking any benchmark. These findings highlight key limitation of current PRMs: while they are effective guiding weaker models, they do not effectively enhance or complement diready-alignee reasoning models, and may even misalign with their internal reward signals.
Since reasoning models demonstrate strong PRM performance, introduce Self-PRM method, where each model (e.g.. QwQ-32B or DeepSeek-RII RI) serves as its own PRM to rerank its sampled outputs. As shown in Table the BoN with Self-PRM strategy consistently outperforms both Pass @ and Majority Voting, particularly at larger sample sizes(e.g.. k = 32, 64). For instance, QwQ-32B on AIME24 improves from 86.7 (majority voting) to 90.0 (BoN w/ Self-PRMM A 32 and maintains this gain at = 64. Similar improvements are observed across other datasets and for DeepSeek-PR RRI. These improvements suggest that the model's ' internal reward signal is better aligned with its reasoning behavior than those of externally trained PRMs, enabling a more effective utilization of its latent reasoning capabilities. Thus, Self-PRM offers more introspective and model-consistent evaluation strategy for complex reasoning tasks.
Takeaway 4:
External PRMs provide little to no benefit for RI.-based reasoning models like QwQ-32B and DeepSeek-RI and may even perform worse than majority voting. In contrast, Self-PRM, where models use their own internal scoring as process reward to rerank outputs, consistently improves accuracy, particularly higher sampling sizes. This approach demonstrates better alignment with the model's 's problem-solving abilities in complex reasoning tasks and greater effectiveness in leveraging its latent PRM capabilities.
4.4
The Limitations of Self-PRM
Table 5: Model performance comparison across different datasets and problem indices.
Dataset
AIME24
AIME25 1
AIME25 II
CNM024
Model
index
62
81
89
6
9
10
13
14
12
17
Difficulty
0/64
0/64
3/64
3/64
4/64
2/64
0/64
1/64 0/64
3/64
2/64
0/64
QwQ-32B
SprM
10
14
18
25
6
2
8
5
55
13
16
STP
0
2
0
Difficulty 1/64 0/64 0/64
10/64
22/64
14/64
0/64
1/64
0/64
34/64
6/64
19/64
0/64 0/64
DeepSeek-R1
SPRM
16
32
44
11
7
1
1
2
39
7
34
STP
3
18
0
10
0
Note: AIME24 problem indices are sourced from the aatthaaiaainne dataset on Hugging Face.
The indices in Table 5 were specifically chosen from problems that the respective models (QwQ-32B and DeepSeek-R11 R1) initially failed to solve correctly (i.e., failed at pass @ 1). This selection allows us to analyze the behavior of Self-PRM on what are considered 'hard problems' for each model. For these hard problems, we sampled 64 solutions to see how many the Self-PRM would judge correct (SpRAs), ), and out of those, how many were truly correct (SPRM). The "_" symbol indicates that the problem was not a hard problem for that particular model (i.e., the model solved it correctly on its first try). For instance, in AIME24 index 62, QwQ-32B succeeded initially, so was excluded from this failure analysis and marked with "_".
8

---

To better understand the limitations of Self-PRM, we conduct a fine tine-griined analysis at the problem level using N 64 sampled solutions per problem instance. For each case, we define Sprm as the number of solutions labeled as correct by the Self-PRM, and STP C SPRM as the subset that are truly correct. Thus, the ratio STP/ /SRM reflects the precision of Self-PRM's judgment. As shown in Ta while Self-PRM often selects a substantial number of candidate solutions as correct (e.g., 55 on CNM024-index-1 for QwQ-32B), the precision often low. For example, on CNM024-index-L only 2 of the 55 selected solutions are truly correct, yielding precision of 3.6%. This trend is especially pronounced for high-difficulty problems, such as CNMO24 and certain AIME25 instances, where Self-PRM, misclassifies a large number of incorrect reasoning traces valid. In contrast, DeepSeek-FR RI, while selecting fewer candidates overall, tends to show higher precision in Self-PRM filtering. On the same CNMO24-index- problem, DeepSeek-PR R1 selects 39 solutions correct, 10 of which are true positives, resulting in a much higher precision of 25.6%. This suggests that stronger solvers exhibit more reliable self-PRM behaviors, possibly due more coherent internal alignment between their reasoning and scoring mechanisms. These observations highlight a key challenge: although Self-PRM improves performance at the aggregate level, its fine-grained precision can vary substantially, especially on difficult tasks, where the model tends to overestimate the correctness of its own outputs. Future work may focus strengthening a model's Self-PRM capabilities through joint training with fine-ge graingr process supervision or continued RL scaling, enabling tighter alignment between the model's problem-sol solving ability and PRM capabilty.
Takeaway
Self-PRM also suffers from low precision on hard problems, misclassifying many incorrect solutions as correct. Stronger models like DeepSeek-RI exhibit more reliable self-evaluation.
Improving Self-PRM may require joint training with PRM-related tasks or continued RL scaling to enhance internal alignment.
5
Conclusion
This work challenges the prevailing assumption that explicit process-level supervision is necessary for enabling effective reasoning in large language models. Through systematic empirical analysis, we demonstrate that RL alone - without access to fine- line-grained reasoning annotations - can endow models with strong PRM capabilities. RL-trained models such DeepSeek-RI and QwQ-32B not only solve complex problems with high accuracy but also implicitly learn to distinguish between valid and flawed reasoning steps. Furthermore, we find that existing state-of-the-arr PRMs fail to enhance the performance of slow now-thinking reasoning models, and in some cases, underperform compared to simple baselines like majority voting. In contrast, leveraging internal signals, i.e., Self-PRM consistently improves performance, particularly at larger sampling sizes. These findings reveal a tight coupling between problem-solving and process-level judgement in RL-trained models. Overall, our findings suggest that PRM may not be essential for enhancing complex reasoning, as pure RL. not only improves problem-solving skills but also inherently fosters robust PRM capabilities. We hope these findings provide actionable insights for building more reliable and self-aware complex reasoning models.
Limitations
In this work, since mathematical problems offer clear, objective correctness criteria and structured, step-by-step solutions, we chose the math reasoning domain for studying the co-evolution of problem- solving and process supervision capabilities. Our primary goal was to first systematically establish and validate the existence of the strong positive correlation between model's problem-solving accuracy and its process judgment Fl score, phenomenon we demonstrate persists across multiple datasets as RL training progresses. However, a deeper investigation of the underlying mechanism is valuable direction.

---

Acknowledgments
This work was supported by National Key Research and Development Program of China under Grant 2022YFA1004102, and in part by National Natural Science Foundation of China under Grant 62502192, and National Natural Science Foundation of China under Grant 62272210.
References
[1] Anthropic. Introducing claude.
https: // www. anthropéc. com/ indew/
introducing-claude 2023.
[2] Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: Process supervision without process. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 27689-27777 227244. Curran Associates, Inc., 2024.
[3] Karl Cobbe, Vineet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, abs/21 10. 10.1168,,, 2021.
[4] Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.0144 4556,, 2025.
[5] Jiazhan Feng. Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in Ilms, 2025.
[6] Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong. Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang. Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-math: A universal olympiad level mathematic benchmark for large language models.
ArXiv, abs/2410.07985. 2024.
[7] Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang. Minlie Huang, Nan Duan, and Weizhu Chen. ToRA: tool-integrated reasoning agent for mathematical problem solving. In The Twelfth International Conference Learning Representations, 2024.
[8] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
[9] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Annual Meeting of the Association for Computational Linguistics, 2024.
[10] Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung- Yeung Shum Xiangyu Zhang. Open-reasoner--eero: An open source approach scaling reinforcement learning on the base model.
com http://hww.....oo..eeee/aaaaaee/ Retpe//Opppn-Raeaeee---eeeee
2025.
[11) Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint arXiv:2410.2127 2024.
[12] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-rl: Training 1lms to reason and leverage search engines with reinforcement learning. arxiv preprint arXiv:2503.09512 2025.
[13] Wendi Li and Yixuan Li. Process reward model with q-value rankings. In The Thirteenth International Conference Learning Representations, 2025.
10

---

[14] Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrall inllinttegrted rl, 2025.
[15]
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. ArXiv, 006/2305.200550 2023.
[16] Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou, Can lb Ilm surpass 405b Ilm? rethinking compute-optimal test-time scaling, arXiv preprint arXiv:2502.066733, 2025.
[17] Qianli Ma, Haotian Zhou, Tingkai Liu, Jianho Yuan, Pengfei Liu, Yang You, and Hongxia Yang. Let's reward step by step: Step-level reward model as the navigators for reasoning. arXiv preprint arXiv:2310. /0000000000000000 2023.
[18] Skywork o1 Team. Skywork-ol open series. http:://hugginnfacccc///// ber 2024. [19] OpenAl. Learning to reason with llms, 2024.
Novem-
[20]
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.022555 2022.
[21] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. ArXív, abs/2305.1 1es/2305.11290, 2023.
[22] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, fes/1707.063477 2017.
[23] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, abs/2402.033000, 2024.
[24] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint WxXiv:2503305522 2025.
[25] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501. 12599, 2025.
[26]
Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024.
[27] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint 1.. 2022.
(28) Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLMS step-by-step without human annotations. In Lun- Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 9426-9439, Bangkok, Thailand, August 2024, Association for Computational Linguistics.
[29] Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao. Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503. 10291, 2025.
[30] Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of coppute-optimal inference for problem-solving with language models. arXiv preprint arXivv2408.000 2024.

---

[31] Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang, An implementation of generative prm. https://gi thub. com OLHFFlo/RLHH.
neward-Modell1e 2024.
[32]
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang. Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint Ar0iv:2407. 10671, 2024.
[33] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu. Jianhong Tu, Jingren Zhou, Junyang Lin, et Qwen2. 5-math technical report: Toward mathematical expert model via self- self-improvement. arXiv preprint arXiv:2409. 2024.
[34] Fei Yu, Anningzhe Gao, and Benyou Wang. OVM, outcome-supervised value models for planning mathematical reasoning. Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACI. 2024, pages 858-875, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
[35] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang. Honglin Yu, Weinan Dai, Yuxuan Song, Xiang Wei, Haodong Zhou, Jingjing Liu, Wei Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yong-Xu Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale. 2025.
[36] Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504. 05/ 18, 2025.
[37] Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, et al. Genprm: Scaling test-time compute of process reward models via generative reasoning. arXiv preprint (iXXiv:5540009)) 2025.
[38] Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu. Dayiheng Liu, Jingren Zhou, and Junyang Lin. Processhench: Identifying process errors in mathematical reasoning. arXiv preprini arXiv:2412.06559, 2024.
[39] Jianyuan Zhong, Zeju Li, Zhijian Xu, Xiangyu Wen, and Qiang Xu. Dyve: Thinking fast and slow for dynamic process verification. ArXiv, abs/2502. 11157, 2025.
[40] Jiachen Zhu, Congmin Zheng, Jianghao Lin, Kounianhua Du Ying Wen, Yong Yu, Jun Wang. and Weinan Zhang. detrieval-augmente process reward model for generalizable mathematical reasoning. arXiv preprint arXiv:2502.14361 2025.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
this paper, we conduct a systematic investigation of the relationship be- tween RI. training and PRM capabilities. Our findings demonstrate that problem-solving proficiency and process supervision capabilities represent complementary dimensions of reasoning that co-evolve synergistically during pure RL training.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See in Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
13

---

Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See in Section 3.1 and Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https
/ 'nipsscc/puullc//uuu hotp:///ggiddeeeeCouuuuumbbbbi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
15

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See in Section 3.1 and Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research presented in this paper conforms to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All external assets, including datasets, code libraries, and pre- trained models, are properly credited in the paper, primarily in the experiments section and the references.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We utilized pre-trained Large Language Model and the model version explicitly detailed in Section 3.1 and 4.1.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
19


Reviews:
[{"review_id": "BucMXFfeuX", "reviewer": "Reviewer_cPkA", "summary": "Challenging the necessity of external Process Reward Models (PRMs), this paper's systematic investigation reveals that pure reinforcement learning not only improves an LLM's problem-solving skills but also synergistically co-develops its internal process supervision capabilities, suggesting that continued RL scaling is a more direct path to advancing complex reasoning.", "questions": "See Weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- This paper presents a comprehensive empirical study with a very broad scope. It investigates a range of topics, from the limitations of state-of-the-art Process Reward Models (PRMs) in LLM reasoning to the advantages and disadvantages of a Self-PRM approach. Following the precedent set by Deepseek R1, this work further demonstrates that an external PRM may not be essential for successfully scaling reasoning abilities via reinforcement learning.\n- The experimental finding presented in Figure 1 is particularly interesting. It reveals a strong correlation where the improvement in a model's problem-solving ability is accompanied by an enhancement of its process supervision capabilities. This insight could potentially unlock novel applications, such as using the reasoning model itself as a PRM.\n\nWeaknesses\n- My main criticism is that the paper attempts to tackle too many broad questions at once. This ambitious scope seems to have come at the cost of depth, leading to some topics being underexplored and a general lack of profound insights.\nThe paper does not adequately explain why RL scaling enhances the ability of a reasoning model to act as a PRM. Does this process make the internally generated reward signals more robust against being 'hacked' by the policy, or are there other underlying mechanisms at play? I was expecting a dedicated section (e.g., a Section 3.4) with targeted experiments to delve into this crucial question. Given that the main body of the paper is currently under the page limit, there appears to be ample space to include such a vital analysis.\n- The study overlooks a critical scenario that merits investigation. A key potential advantage of an external PRM is its ability to provide denser supervision signals during RL training compared to a simple, binary rule-based verifier. However, the paper includes no experiments to test this. If an external PRM can guide the RL process more effectively than a rule-based verifier, it would demonstrate that PRMs are not entirely obsolete and still hold significant value—a conclusion that this paper currently seems to dismiss.", "was_revised": true, "final_justification": "I believe the second part of my concern has been addressed, but the first part remains unclear. Therefore, I'm inclined to keep my score.", "rebuttal": "We sincerely thank the reviewer for their valuable time and constructive feedback. We would like to address the identified weaknesses and questions to further clarify and strengthen our work.\n\nQ1: Regarding the lack of depth on *why* RL scaling enhances PRM capability\n- Our primary goal was to first systematically establish and validate the *existence* of the strong positive correlation between a model's problem-solving accuracy and its process judgment F1 score, a phenomenon we demonstrate persists across multiple datasets as RL training progresses. However, we agree that a deeper investigation into the underlying mechanism is a valuable addition.\n- In our revision, we will add a new section with targeted analyses to explore this *why* ; these experiments will be conducted after the rebuttal period due to time constraints. Our hypothesis is that as RL training improves the policy, the model's internal value function becomes a more reliable estimator of the correctness of reasoning steps. To test this, we will analyze the evolution of the model's internal reward signals and their correlation with ground-truth step-wise correctness at different checkpoints during training. This will provide a more direct explanation for how RL implicitly fosters robust PRM capabilities without explicit supervision.\n\nQ2: Regarding the overlooked scenario of using an external PRM for RL training\n\n- We thank the reviewer for highlighting this important scenario. We acknowledge that our study did not test the efficacy of an external PRM as a source of dense reward signals during the RL training phase. Our experimental scope was focused on a different, but equally relevant, question: whether existing state-of-the-art PRMs can further enhance already strong reasoning models through post-hoc reranking. Our results show that for models like QwQ-32B and DeepSeek-R1, external PRMs offer no significant benefit over simpler methods like majority voting, whereas an internally derived Self-PRM does.\n\n-  While using a PRM for dense supervision is a valid approach, our work challenges the prevailing assumption that is considered necessary. The paper highlights that conventional PRMs face significant hurdles, including prohibitive annotation costs and vulnerability to reward hacking. Our core finding is that pure RL scaling presents a more direct and efficient path, as it not only boosts problem-solving but also inherently develops the desired process supervision abilities. We believe this finding, which shows that a complex capability can be an emergent property of a simpler training objective, is a significant contribution. We will clarify this distinction in the paper and position the dense supervision angle as a valuable direction for future research. In our subsequent research, we are attempting to use a self-PRM to guide the RL training process, where both the policy model and the self-PRM are improved simultaneously. More related results will be disclosed in future studies."}, {"review_id": "r4XaJSJzep", "reviewer": "Reviewer_SEpk", "summary": "This work analyzes RL algorithms that focus solely on outcome correctness, such as GRPO, which can enhance LLMs' capability of process supervision. By comparing different versions of open-source LLMs and conducting RL training experiments, the author points out that verifying the final result can still endow models with process supervision abilities. This work also highlights a potential correspondence between problem solving ability and process supervision capability.", "questions": "Table 3, for the claim \"while RL-trained models (QwQ-32B, DeepSeek-R1) show minimal or negative gains, suggesting limited benefit when process reasoning is already reinforced.\", another reason could be the QwQ and R1's performance is higher than Qwen2.5-32B-Instruct and R1-Distill-Qwen-32B. Similarly, the advantage of Qwen2.5-32B-Instruct is 18.8, but the advantage of R1-Distill-Qwen-32B is only 2.6. How about testing a smaller RL-trained model?\n\nIt's better to provide more statistical results of Sec 4.4. I think drawing a curve of the pass ratio and precision could be very helpful to support the claim \"Self-PRM also suffers from low precision on hard problems, misclassifying many incorrect solutions as correct\"", "limitations": "The statement made in this work is generalizable, but experiments only involve the mathematical domain.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The analysis in L141 \"Problem-Solving V.S. Judgment\" is insightful.\n2. This work is well-written and easy to understand.\n\nWeaknesses:\n1. L207–209 state: \"indicating that Self-REF can inject meaningful structure into process modeling where it is otherwise absent. In contrast, RL-trained models such as QwQ-32B and DeepSeek-R1 exhibit marginal or negative changes in F1 when using Self-REF\". However, Qwen2.5-32B-Instruct has undergone an RL training process. While R1-Distill-Qwen-32B has not, the performance gain is already minimal. Therefore, the experimental results do not sufficiently support the stated conclusion.\n\n2. AIME benchmark only have a few questions, and the performance differences reported in Sec 4.3 are not particularly significant. The conclusion that \"External PRMs provide little to no benefit for RL-based reasoning models\" may not be broadly generalizable.", "was_revised": true, "final_justification": "Thanks for the author's detailed response and additional results. However, the claim made in this work has a large scope, and the current results are still not fully convincing me, such as the RLHF and RLVR series models have different behaviours. So I keep my score.", "rebuttal": "We sincerely thank the reviewer for their valuable time and constructive feedback. We are pleased that you found the paper well-written, easy to follow, and the analysis in the \"Problem-Solving V.S. Judgmen'' section insightful. We would like to address the identified weaknesses and questions to further clarify and strengthen our work.\n\nQ1: On the conclusion regarding Self-REF (Table 3)\n\nWe appreciate the reviewer's close reading of this section. There seems to be a slight misunderstanding we wish to clarify. The reviewer states that \"Qwen2.5-32B-Instruct has undergone an RL training process'', but according to the source technical reports, this model is a purely instruction-tuned model without reinforcement learning. Qwen2.5-32B-Instruct is a non-thinking model. Our paper explicitly categorizes it this way to test the effect of Self-REF in a setting where process reasoning has not been reinforced. The substantial F1 score improvement of +18.8 for this model provides strong evidence for our claim.\n\nConversely, R1-Distill-Qwen-32B, while not directly trained with RL, was distilled from DeepSeek-R1 and thus inherits its process judgment behaviors. In fact, R1-Distill-Qwen-32B is a thinking model. Therefore, it represents an intermediate case. The fact that its performance gain from Self-REF is minimal (+2.6) further supports our conclusion that as a model's internal reasoning becomes more aligned through RL (or distillation from an RL model), the benefit of weakly supervised signals from its own generated traces diminishes.\n\nQ2: On the generalizability of the AIME benchmark results (Sec 4.3)\n\nThis is a fair point. We acknowledge that the AIME benchmarks contain a limited number of problems. Our intention was to use these highly challenging competition-level problems to test the limits of current PRMs in a difficult, out-of-distribution setting.\n\nWhile the AIME results alone might not be broadly generalizable, they are presented as complementary evidence to our larger-scale findings on Processbench. Across Processbench, our core finding holds: strong RL-trained models like DeepSeek-R1 and QwQ-32B consistently outperform models explicitly trained with PRM labels. The fact that an external PRM also fails to provide a significant boost over majority voting on the difficult AIME problems further reinforces our main thesis. In our revision, we will soften the language in this section to more accurately reflect the limited sample size of the AIME benchmark while maintaining the core insight.\n\nWe have supplemented the analysis on the GPQA-diamond dataset, which contains a total of 198 questions specifically designed for the most challenging, PhD-level science questions. The specific results are as follows:\n\n| QwQ-32B           | k=8   | k=16  | k=32  | k=64  |\n|-------------------|-------|-------|-------|-------|\n| Pass@k            | 81.3  | 85.9 | 88.9  | 89.9\n| Majority Voting@k | 64.1 | 63.6  | 65.2  | 64.1 |\n| BoN w/ PRM@k      | 64.6 | 63.6 | 65.2 | 64.6 |\n| Self-PRM@k        | 62.6 | **64.3**| **65.7**| **66.2**|\n\n\n| DeepSeek-R1        | k=8   | k=16  | k=32  | k=64  |\n|--------------------|-------|-------|-------|-------|\n| pass@k         | 93.4  | 93.9  | 94.9  | 95.6  |\n| Majority Voting@k | 82.8  | 82.8  | 82.8  | 82.3  |\n| BoN w/ PRM@k     | 81.8  | 82.8  | 83.3  | 82.3  |\n| Self-PRM@k    | 80.8  | 81.3  | 82.6  | **83.8**  |\n\nBased on the analysis of results on GPQA-diamond dataset, we find that the conclusion remains consistent with our previous findings: External PRMs provide little to no benefit for RL-based reasoning models.\n\nQ3: Testing a smaller RL-trained model (Table 3)\n\nFollowing your suggestion, we directly used an open-source, smaller RL-trained model: Qwen-2.5-7B-SimpleRL-Zoo.\nThe specific results are as follows:\n\n\n| **Model**               | **GSM8K**               |              |              | **MATH**                |              |              | **OlympiadBench**       |              |              | **OmniMATH**            |              |              | **Average** |\n|-------------------------|-------------------------|--------------|--------------|-------------------------|--------------|--------------|-------------------------|--------------|--------------|-------------------------|--------------|--------------|-------------|\n|                         | **Error**               | **Correct**  | **F1**       | **Error**               | **Correct**  | **F1**       | **Error**               | **Correct**  | **F1**       | **Error**               | **Correct**  | **F1**       |             |\n| Qwen-2.5-7B-SimpleRL-Zoo | 42.5                    | 95.3         | 58.8         | 27.4                    | 83.7         | 41.3         | 17.1                    | 76.7         | 28.0         | 12.4                    | 71.0         | 21.1         | 37.3        |\n| w/ Self-REF             | 37.7                    | 93.3         | 53.7         | 26.9                    | 87.7         | 41.2         | 21.3                    | 82.3         | 33.9         | 19.0                    | 78.4         | 30.6         | 39.9        |\n\nBased on the results, we can observe that the conclusions drawn from the 7B model are consistent with our previous findings: RL-trained models show minimal or negative gains, suggesting limited benefit when process reasoning is already reinforced.\n\nQ4: More statistical results for Self-PRM (Sec 4.4)\n\nWe agree completely. A visualization would be much more powerful. We thank the reviewer for this constructive idea. In the revised version, we will add a plot showing the curve of the solution pass ratio versus Self-PRM precision. This will provide a clearer and more intuitive illustration to support our claim that Self-PRM's precision is low on difficult problems.\n\nQ5: On the Limitation of the Mathematical Domain\n\nWe acknowledge this limitation. We chose the mathematical domain because it offers objective and verifiable ground truth for both final answers and intermediate reasoning steps, making it an ideal environment to systematically study the relationship between problem-solving and process supervision. \n\nCurrently, there are no process reward evaluation benchmarks similar to ProcessBench in other domains, and constructing such benchmarks is challenging. Therefore, we have not extended the analyses in Table 2 and Table 3 to other domains. Regarding the results in Table 4, we have extended our investigation to the GPQA-diamond dataset (The results are shown in Q2.) and found that the conclusions remain consistent.\n\nWe will explicitly state in the conclusion that our findings are currently confined to mathematical reasoning and that future work should investigate whether these principles generalize to other complex reasoning domains.\n\n\nThank you again for your insightful feedback. We believe that incorporating these changes will significantly improve the quality and impact of our paper."}, {"review_id": "uKARTMTZ3w", "reviewer": "Reviewer_shMK", "summary": "This paper investigates whether explicit Process Reward Models (PRMs) are necessary for developing strong reasoning capabilities in large language models, or if pure reinforcement learning (RL) training can implicitly induce these capabilities. The authors conduct a systematic empirical study demonstrating that RL-trained models like DeepSeek-R1 and QwQ-32B develop robust PRM capabilities without explicit process-level supervision, often outperforming models explicitly trained with PRM data. The study reveals a statistically significant correlation between problem-solving proficiency and process supervision capabilities through chi-square analysis. The authors introduce Self-PRM, where models use their internal reward signals to rerank their own outputs, showing consistent improvements over external PRMs and majority voting, particularly at higher sample sizes. However, the analysis reveals limitations in Self-PRM's precision on challenging problems, highlighting the need for better reward alignment.", "questions": "1. Generalization across domains (as mentioned in the weakness section): How do these findings extend to non-mathematical reasoning tasks such as coding\n2. RL training specifics: The paper shows results for one RL training setup (DAPO with Qwen2.5-7B-Base). How sensitive are the findings to different RL algorithms (PPO, DPO)? Is it possible that different algorithm might exhibit different pattern?\n3. Efficiency-wise: What is the computational overhead of Self-PRM compared to external PRMs and majority voting (e.g., wall clock time would be nice to have)? How does this scale with the number of candidate solutions and model size?\n4. Threshold: How sensitive is Self-PRM performance to the selection of confidence thresholds or scoring mechanisms? Are there principled ways to set these parameters?", "limitations": "no.\nJustification is \"[TODO]\" in the paper check list", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- This work provides some interesting insights that shed light into the future of PRM\n- Comprehensive empirical evaluation: The study covers multiple datasets (GSM8K, MATH, OlympiadBench, OmniMath) and various model types, providing robust evidence for the claims\n- I especially like the Chi-square tests provide statistical validation of the correlation between problem-solving and process supervision capabilities across models\n- Experiments are well-designed: The RL training experiments with learning curves effectively demonstrate the co-evolution of accuracy and PRM capabilities during training\n- Self-PRM offers an alternative to external PRMs with demonstrated usefulness\n\nWeakness:\n\n- The scope of this paper is limited to the math reasoning domain due to the selection of MATH data as the evaluation dataset. It is unclear if the conclusion of this paper will transfer to other domains (e.g., coding)\n- Not really a weakness but it'll be interesting to see if self-PRM can be used to further improve the policy its self by acting as a PRM during RL", "was_revised": true, "final_justification": "I appreciate that the authors provide results on the additional experiments and show the efficacy of self-PRM outside the MATH domain, as well as answering a couple of questions. All of my concerns have been addressed, and I will modify my score accordingly and vote for acceptance.", "rebuttal": "We are extremely grateful to the reviewer for their thoughtful and encouraging feedback, as well as the high scores across the board. We are delighted that you found our work to offer interesting insights, appreciated the comprehensive empirical evaluation and the statistical validation via Chi-square tests, and found the experiments to be well-designed. Your positive assessment is a strong validation of our research direction. We will address the weakness you've pointed out and answer your excellent questions.\n\nQ1: On the Limitation to the Math Reasoning Domain\n\nThis is a very fair point, and we agree that the current work's empirical validation is focused on the mathematical domain. We chose this domain because mathematical problems offer clear, objective correctness criteria and structured, step-by-step solutions, making it an ideal \"laboratory\" for studying the co-evolution of problem-solving and process supervision capabilities.\n\nCurrently, there are no process reward evaluation benchmarks similar to ProcessBench in other domains, and constructing such benchmarks is challenging. Therefore, we have not extended the analyses in Table 2 and Table 3 to other domains. Regarding the results in Table 4, we have extended our investigation to the GPQA-diamond dataset (The specific results are presented below.) and found that the conclusions remain consistent.\n\n| QwQ-32B           | k=8   | k=16  | k=32  | k=64  |\n|-------------------|-------|-------|-------|-------|\n| Pass@k            | 81.3  | 85.9 | 88.9  | 89.9\n| Majority Voting@k | 64.1 | 63.6  | 65.2  | 64.1 |\n| BoN w/ PRM@k      | 64.6 | 63.6 | 65.2 | 64.6 |\n| Self-PRM@k        | 62.6 | **64.3**| **65.7**| **66.2**|\n\n\n| DeepSeek-R1        | k=8   | k=16  | k=32  | k=64  |\n|--------------------|-------|-------|-------|-------|\n| pass@k         | 93.4  | 93.9  | 94.9  | 95.6  |\n| Majority Voting@k | 82.8  | 82.8  | 82.8  | 82.3  |\n| BoN w/ PRM@k     | 81.8  | 82.8  | 83.3  | 82.3  |\n| Self-PRM@k    | 80.8  | 81.3  | 82.6  | **83.8**  |\n\nRegarding generalization, we hypothesize that the core finding will indeed transfer to other complex reasoning domains like coding, planning, or legal analysis. The underlying principle—that an agent learning to solve a complex task (the policy) simultaneously develops an internal model for evaluating the quality of its actions (the value function)—is a fundamental concept in reinforcement learning. As long as a domain requires multi-step, logical reasoning, we expect a similar synergistic development. We will explicitly add this discussion to our limitations section and highlight cross-domain validation as a critical direction for future work.\n\nQ2: On Using Self-PRM to Improve the Policy during RL\n\nThis is not a weakness but a fantastic suggestion for future work, and we thank you for it. You have accurately identified a natural and exciting extension of our findings. The current work uses Self-PRM in a post-hoc reranking setting. Using the model’s own internal reward signal to provide dense, step-level feedback during RL training could create a powerful self-improvement loop, potentially accelerating learning and pushing performance even further. This is an excellent idea that directly builds on our work, and we will add it to our future work section as a promising research avenue.\n\nQ3: Sensitivity to RL algorithms\n\nDue to time constraints, we directly tested the performance of the open-source model Open-Reasoner-Zero-7B, which is based on the qwen2.5-7B-base model and trained using the PPO algorithm. On the Math, GSM8K, OlympiadBench, and OmniMath datasets, the problem-solving accuracy (%) of Open-Reasoner-Zero-7B are 77.9, 89.8, 42.0, and 37.4 respectively. On ProcessBench, its F1 scores (%) are 41.3, 58.8, 28.0, and 21.1 respectively. Based on these results, we can draw the same conclusion: after RL training, both the problem-solving ability and PRM capability of the model have improved.\nIn the revised version, we will retrain a model based on the PPO algorithm, plot the corresponding curves in Figure 1, and provide a more detailed analysis.\n\nQ4: Computational overhead of Self-PRM\n\nMajority voting outperforms both Self-PRM and external PRMs in terms of efficiency. Majority voting merely derives the final answer based on the consistency of results, whereas both Self-PRM and external PRMs require invoking the model to judge the process scores before selecting the answer. In this paper, Self-PRM and external PRMs are utilized in the same manner; the only distinction lies in that external PRMs are derived from independently trained models, while self-PRM does not require additional training. Therefore, the efficiency of Self-PRM and external PRMs depends solely on the model parameters. The external PRMs employed in this paper is a 72B model, while QwQ-Self-PRM, being a 32B model, exhibits superior efficiency. In contrast, R1-Self-PRM, with a size of 671B, demonstrates significantly slower inference speed.\n\nQ5: Self-PRM sensitivity to thresholds\n\nWe adopt the LLM-as-judge approach, where the model is directly prompted to judge the correctness of each step. Since this is a generative model, there is no involvement of a confidence threshold in this process."}, {"review_id": "ip9sfnnj2h", "reviewer": "Reviewer_GVjb", "summary": "This paper investigates whether explicit process reward models (PRMs) are necessary for strong reasoning in language models. Through thorough experiments, the authors demonstrate that reinforcement learning (RL) can implicitly induce PRM-like abilities in large language models. They further propose “self-PRM,” a method that uses the model itself to judge and rerank its outputs, thereby acting as its own PRM. Self-PRM leads to improved overall performance, though it can exhibit low precision on particularly difficult problems.", "questions": "Does models with self-PRM suffer from low precision on difficult problems more than no-PRM? It is not clear from the paper. Maybe I missed it.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n1. The experimental setup is comprehensive, covering a wide range of benchmarks, models, and evaluation methods.\n2. I appreciate that the paper is framed more as a careful analysis rather than simply a new method introduction. The authors are transparent about the limitations of their proposed method (self-PRM).\n\n### Weaknesses\nI believe there is some misalignment between the paper’s framing and the experimental findings.\n1. The results primarily show that external PRMs can be suboptimal for strong RL-trained models, but that self-PRM (using the model itself) is still beneficial. This may reflect the limitations of current external PRMs rather than the broader claim that PRMs are unnecessary. In other words, the experiments demonstrate that current PRMs may be insufficient, but not that process-level guidance itself lacks value.\n2. The results indicate that gains in process judgment often precede gains in answer accuracy during RL training, which suggests that guidance via process-level supervision could still be helpful.", "was_revised": true, "final_justification": "Regarding AC's questions:\n\n- Yes, the revised framing does address the overclaiming issue. However, fully shifting the narrative may be nontrivial. For example, Table 3 suggests that Self-PRM is primarily beneficial for models that haven’t been RL-trained, and Section 4.4 acknowledges limitations of self-reflection.\n- If the external PRM is not a reasonable baseline (e.g., due to OOD mismatch), then this does weaken the strength of the comparison. That said, I’m not an expert on this specific setup and can’t fully judge its appropriateness.\n- I don’t consider math-only scope a major concern.\n- I view the paper primarily as an analysis piece, and in that light, it provides valuable insights. For instance, that models can naturally acquire process judgment capabilities during RL. If it were framed purely as a new method, I’d find the evidence less compelling.", "rebuttal": "We sincerely thank the reviewer for their careful reading and insightful feedback. We are very grateful that you appreciated the comprehensive experimental setup and our transparent approach to analyzing the results and limitations of our proposed method. Your main criticism regarding the potential misalignment between our framing and findings is very well-taken, and we would like to clarify our position.\n\nQ1: Regarding the Framing:  \"PRMs are Unnecessary'' vs. \"Current PRMs are Insufficient''\n\nThis is an excellent and sharp observation. We agree that our results do not dismiss the value of process-level guidance itself. In fact, our findings strongly support its value. Our central argument, perhaps not framed as clearly as it could be, is aimed at the conventional approach of building separate, external, and explicitly annotated PRMs. Our title, \"Is PRM Necessary?\", is meant to question this specific paradigm, which faces major challenges like prohibitive annotation costs and vulnerability to reward hacking.\n\nOur work demonstrates that:\n\n1. Pure RL can implicitly develop this process-level judgment capability without explicit supervision.\n\n2. The resulting internal capability can be harnessed via Self-PRM to achieve performance gains, even outperforming existing external PRMs on strong models.\n\nTherefore, we are not arguing that process supervision is without value. Rather, we are proposing that it is possible to achieve effective process supervision *without* a separate, externally trained PRM. Self-PRM is presented as a more direct, scalable, and model-consistent way to leverage this valuable guidance. We will revise the paper's introduction and conclusion to refine our framing, shifting the emphasis from \"PRMs are unnecessary\" to \"A separate, externally-annotated PRM may not be necessary,\" making this more nuanced argument clearer.\n\nQ2: Regarding Process Judgment Gains Preceding Accuracy Gains\n\nThank you for highlighting this key result from Figure 1. You are absolutely correct in your interpretation: the fact that process judgment improves early in training *does* suggest that process-level supervision could be a very effective and dense signal to guide RL. We see this not as a contradiction, but as a crucial piece of the puzzle that our work helps illuminate. Our contribution is showing that this powerful early signal does not necessarily need to come from an external PRM. Instead, it is a capability that emerges organically. This opens up exciting future possibilities, such as using Self-PRM *during* RL training to create a self-improvement loop, a point we will emphasize more in our future work section.\n\nQ3: Does Self-PRM suffer from low precision on difficult problems more than no-PRM?\n\nIn our experiments, \"no-PRM\" refers to baselines like majority voting. Majority voting does not produce a confidence score or a precision metric in the same way; it simply selects the most frequent answer. The key finding is that while Self-PRM's precision can be low on difficult problems (i.e., it is overconfident and labels many flawed solutions as valid ), its ranking is still highly effective. As shown in Table 4, using Self-PRM to select the single best candidate (BoN w/ Self-PRM) consistently leads to higher final accuracy than both majority voting and direct sampling. This indicates that although Self-PRM's absolute judgment is imperfect, its relative judgment is superior to the baselines. We will add a paragraph to Section 4.4 to make this important distinction clear.\n\nThank you once again for your constructive criticism. We believe your feedback will help us significantly sharpen the paper's message and better contextualize our contributions."}, {"review_id": "9smxTeceeh", "reviewer": "Reviewer_kCye", "summary": "This paper challenges the conventional necessity of Process Reward Models (PRMs) in enhancing the reasoning capabilities of large language models (LLMs). Through empirical analysis, the authors demonstrate that reinforcement learning (RL) training alone, without explicit PRM supervision, can implicitly develop robust PRM capabilities in models like DeepSeek-R1 and QwQ-32B. These RL-trained models not only excel in problem-solving but also exhibit strong process-level judgment, outperforming models explicitly fine-tuned with PRM-labeled data. The study introduces \"Self-PRM,\" a framework where models autonomously evaluate and rerank their solutions using internal rewards. However, Self-PRM struggles with low precision on challenging problems, often misclassifying incorrect solutions. The findings suggest that PRMs may not be essential for complex reasoning, as RL training inherently fosters both problem-solving and process-supervision abilities.", "questions": "Besides the weaknesses above, further questions are as follows:\n\n- Self-REF is only evaluated using one non-reasoning model (Qwen2.5-32B-Instruct). If we use a weaker model, such as Qwen2.5-7B-Instruct or Qwen2.5-1.5B-Instruct, will the performance of Self-REF be worse? I am wondering if the performance of Self-REF relies on the reasoning abilties of the model since the improvement of Qwen2.5-32B-Instruct w/Self-REF is more significant on easy tasks (GSM8K) than difficult tasks (Omni-MATH).\n- How does the ProcessBench performance evaluated in Figure 1? Do the authors prompt the trained models to answer the first errorneous step in the reasoning process?\n- How does Self-PRM rerank the output? Is it the same way as the evaluation method of ProcessBench?\n- How do the authors divide the reasoning step in BoN w/ PRM experiments?\n- In Table 5, how do the authors choose these problem indices and why some places are filled with \"-\"?\n- Will the code be open-sourced? It would be helpful for the community to reproduce the results and further explore the findings.", "limitations": "This paper does not discuss limitations.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "#### Strengths\n\n- The idea is novel and interesting.\n- This paper provides insightful analysis on the relationship between RL, problem-solving abilities, and process supervision capabitlities. Recent works on LLM reasoning and RL rarely focus on the point of PRMs, but this paper provides valuable and interesting insights.\n- The paper is easy to follow.\n\n#### Weaknesses\n\n- The improvement of Self-PRM is not significant compared with majority voting and BoN w/ PRM.\n- This paper mainly focuses on mathematical reasoning tasks, which may limit the generality of the findings. It would be beneficial to explore the implications of the findings in other domains, such as coding.\n- This paper does not discuss limitations.\n\nMinor:\n- The Justification should be filled in the checklist.\n- L232: SSelf-PRM -> Self-PRM.\n\nI am willing to adjust the rating if the authors can address the weaknesses and the questions below.", "was_revised": true, "final_justification": "I keep my score since my concerns are not fully addressed. The explainations are detailed here (https://openreview.net/forum?id=ooiHIklvN5&noteId=yfKd8Nlz4O).", "rebuttal": "We sincerely thank the reviewer for their detailed feedback and for recognizing the novelty of our idea and the value of our analysis. We are grateful for your constructive criticism and the opportunity to clarify our work. We will address all weaknesses and questions, and we have a clear plan to revise our paper accordingly.\n\nQ1: On the significance of Self-PRM's improvement\n\nWe acknowledge that the improvements in Table 4 might seem modest at first glance. However, we would like to highlight three key points: \n\n1. **Consistency:** Self-PRM consistently matches or outperforms both majority voting and the external PRM across nearly all benchmarks and sampling sizes. In contrast, the external BoN w/ PRM is sometimes worse than simple majority voting, indicating that its guidance can be misaligned. \n\n2. **Meaningful Gains on Hard Problems:** On challenging benchmarks like AIME, even a small percentage increase represents solving additional difficult problems. For example, with QwQ-32B on AIME24 (@k=32), Self-PRM achieves 90.0\\% accuracy, a notable improvement over both baselines at 86.7\\%. \n\n3. **Computational Efficiency:** Crucially, Self-PRM achieves these gains with **virtually zero additional computational overhead**, as it uses internal reward signals calculated during the initial generation pass . An external PRM, however, can nearly double the inference cost. When factoring in efficiency, the benefit of Self-PRM becomes highly significant. We will add a discussion on this cost-benefit analysis to the paper.\n\nQ2: On the generality of findings (mathematical reasoning)\n\nThis is a valid point. We chose this domain because mathematical problems offer clear, objective correctness criteria and structured, step-by-step solutions, making it an ideal \"laboratory\" for studying the co-evolution of problem-solving and process supervision capabilities.\n\nCurrently, there are no process reward evaluation benchmarks similar to ProcessBench in other domains, and constructing such benchmarks is challenging. Therefore, we have not extended the analyses in Table 2 and Table 3 to other domains. Regarding the results in Table 4, we have extended our investigation to the GPQA-diamond dataset, which contains a total of 198 questions specifically designed for the most challenging, PhD-level science questions. The specific results are presented below and we  found that the conclusions remain consistent.\n\n| QwQ-32B           | k=8   | k=16  | k=32  | k=64  |\n|-------------------|-------|-------|-------|-------|\n| Pass@k            | 81.3  | 85.9 | 88.9  | 89.9\n| Majority Voting@k | 64.1 | 63.6  | 65.2  | 64.1 |\n| BoN w/ PRM@k      | 64.6 | 63.6 | 65.2 | 64.6 |\n| Self-PRM@k        | 62.6 | **64.3**| **65.7**| **66.2**|\n\n\n| DeepSeek-R1        | k=8   | k=16  | k=32  | k=64  |\n|--------------------|-------|-------|-------|-------|\n| pass@k         | 93.4  | 93.9  | 94.9  | 95.6  |\n| Majority Voting@k | 82.8  | 82.8  | 82.8  | 82.3  |\n| BoN w/ PRM@k     | 81.8  | 82.8  | 83.3  | 82.3  |\n| Self-PRM@k    | 80.8  | 81.3  | 82.6  | **83.8**  |\n\nRegarding generalization, we hypothesize that the core finding will indeed transfer to other complex reasoning domains like coding, planning, or legal analysis. The underlying principle—that an agent learning to solve a complex task (the policy) simultaneously develops an internal model for evaluating the quality of its actions (the value function)—is a fundamental concept in reinforcement learning. As long as a domain requires multi-step, logical reasoning, we expect a similar synergistic development. We will explicitly add this discussion to our limitations section and highlight cross-domain validation as a critical direction for future work.\n\n\nQ3: On the lack of a limitations section\n\nWe sincerely apologize for this oversight. We will add a dedicated \"Limitations\" section in the revised manuscript. This section will address the domain generality issue mentioned above and other points to provide a more balanced perspective on our work.\n\nQ4: Self-REF on weaker models\n\nIn the Self-REF section, we have supplemented the results of qwen2.5-7B-instruct and qwen2.5-1.5B-instruct.\nBased on the results, we can observe that qwen2.5-7B-instruct with Self-REF shows improvements on GSM8K and MATH, while its F1 scores decrease on OlympiadBench and OmniMATH. In contrast, qwen2.5-1.5B-instruct with Self-REF exhibits a decline in performance across all test sets. These results are consistent with our expectations: the effectiveness of Self-REF *does* depend on the base model's reasoning ability. A weaker instruction-tuned model would generate lower-quality reasoning traces, which would act as noisy or incorrect supervisory signals. We will add this analysis to the final paper.\n\n| **Model**               | **GSM8K**               |              |              | **MATH**                |              |              | **OlympiadBench**       |              |              | **OmniMATH**            |              |              | **Average** |\n|-------------------------|-------------------------|--------------|--------------|-------------------------|--------------|--------------|-------------------------|--------------|--------------|-------------------------|--------------|--------------|-------------|\n|                         | **Error**               | **Correct**  | **F1**       | **Error**               | **Correct**  | **F1**       | **Error**               | **Correct**  | **F1**       | **Error**               | **Correct**  | **F1**       |             |\n| qwen2.5-7B-instruct  | 44.9                    | 43.0         | 43.9         | 30.1                    | 48.0         | 37.0         | 26.5                    | 33.3         | 29.5         | 27.1                    | 28.2         | 27.7         | 34.5        |\n| w/ Self-REF             | 45.4                    | 82.9         | 58.7         | 26.1                    | 80.0         | 39.4         | 15.6                    | 72.9         | 25.7         | 15.8                    | 70.5         | 25.8         | 37.4        |\n|                         |                         |              |              |                         |              |              |                         |              |              |                         |              |              |             |\n| qwen2.5-1.5B-instruct  | 23.2                    | 17.6         | 20.0         | 18.9                    | 27.6         | 22.4         | 12.4                    | 24.5         | 16.5         | 12.3                    | 24.9         | 16.4         | 18.8        |\n| w/ Self-REF             | 17.4                    | 10.4         | 13.0         | 14.0                    | 19.7         | 16.4         | 7.4                     | 33.6         | 12.1         | 9.0                     | 27.8         | 13.6         | 13.8        |\n\n\nQ5: ProcessBench evaluation in Figure 1\n\nFor the evaluation shown in Figure 1, we prompt the model to act as a \"Generative PRM\". For each instance in ProcessBench (which contains a problem and a candidate solution), the model is asked to analyze the solution and output a judgment on its correctness, identifying the first erroneous step if one exists. We then compare the model's textual judgment to the ground-truth label to calculate the ProcessBench F1 score.\n\nQ6: Self-PRM reranking method\n\nWe adopt the same evaluation method as used in ProcessBench to judge the correctness of each step. The self-PRM evaluates the correctness of all steps in each response, and then among the responses where all steps are correct, we select the final answer based on the consistency of the answers.\n\nQ7: Reasoning step division in BoN w/ PRM\n\nFollowing the protocol outlined in the ProcessBench paper, we utilize Qwen2.5-72B-Instruct for step segmentation. We will provide the specific prompt in the revised version.\n\nQ8: Problem indices in Table 5: Problem indices in Table 5\n\nThe indices in Table 5 were specifically chosen from problems that the respective models (QwQ-32B and DeepSeek-R1) **initially failed to solve correctly** (i.e., failed at pass@1). This selection allows us to analyze the behavior of Self-PRM on what are considered 'hard problems' for each model. For these hard problems, we sampled 64 solutions to see how many the Self-PRM would judge as correct ($S_{PRM}$ ), and out of those, how many were truly correct ($S_{PRM}$ ). The \"-\" symbol indicates that the problem was **not a hard problem** for that particular model (i.e., the model solved it correctly on its first try). For instance, in AIME24 index 62, QwQ-32B succeeded initially, so it was excluded from this failure analysis and marked with \"-\". In contrast, DeepSeek-R1 failed the same problem, so its Self-PRM performance is shown. We will clarify this selection methodology in the paper to make the purpose of this fine-grained analysis clear.\nThank you again for your thorough and helpful review. We are confident that by addressing these points, we can significantly improve the paper and hope you will consider re-evaluating its rating."}]

Author rebuttal:

Meta-review: This paper investigates whether explicit PRMs are necessary for reasoning in LLMs, showing that RL alone can induce process supervision capabilities and introducing Self-PRM as a reranking mechanism. The study is timely, well-executed, and provides valuable empirical insights, though the framing initially overstated claims, Self-PRM’s gains are often modest, and evaluation is limited mostly to math reasoning with potential OOD issues in PRM baselines. Reviewers are split between borderline accept and reject, but agree the revised framing is more accurate. Overall, the work advances understanding of RL vs. PRM in reasoning and is worth sharing.
Decision: Accept (poster)