SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learning
Authors: Yuyang Ding
Xinyu Shi
Juntao Li
xiaobo liang
Zhaopeng Tu
Min Zhang
Venue: NeurIPS 2025 poster ()
Keywords: LLM Reasoning
Process Supervision
Robust Learning
Primary area: deep_learning

Abstract: Process reward models (PRMs) offer fine-grained, step-level evaluations that facilitate deeper reasoning processes in large language models (LLMs), proving effective in complex tasks like mathematical reasoning. 
However, developing PRMs is challenging due to the high cost and limited scalability of human-annotated data.
Synthetic data from Monte Carlo (MC) estimation is a promising alternative but suffers from a high noise ratio, which can cause overfitting and hinder large-scale training.
In this work, we conduct a preliminary study on the noise distribution in synthetic data from MC estimation, identifying that annotation models tend to both underestimate and overestimate step correctness due to limitations in their annotation capabilities. 
Building on these insights, we propose {\bf S}elf-Denoising Monte {\bf C}arlo {\bf An}notation (\textsc{Scan}), an efficient data synthesis and noise-tolerant learning framework.
Our key findings indicate that:
(1) Even lightweight models (e.g., 1.5B parameters) can produce high-quality annotations through self-denoising strategy, enabling PRMs to achieve superior performance with only 6\% the inference cost required by vanilla MC estimation.
(2) With our robust learning strategy, PRMs can effectively learn from this weak supervision, achieving a 39.2 F1 score improvement (from 19.9 to 59.1) in ProcessBench.
Despite using only a compact synthetic dataset, our models surpass strong baselines, including those trained on large-scale human-annotated datasets such as PRM800K.
Furthermore, performance continues to improve as we scale up the synthetic data, highlighting the potential of \textsc{Scan} for scalable, cost-efficient, and robust PRM training.

SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learning
Yuyang Ding', Xinyu Shi',
Juntao Li Xiaobo Liang', Zhaopeng Tu', Min Zhang'
Soochow University
Tencent
fyyding23, , 02) suda. edu.
f1jt, mingmmminnn @suda. edu. cn zptu@tencent.com
Project Page: https://scaat github.io
Abstract
Process reward models (PRMs) offer fine-grained, step-level evaluations that facili- tate deeper reasoning processes in large language models (LLMs), proving effective in complex tasks like mathematical reasoning. However, developing PRMs is chal- lenging due the high cost and limited scalability Synthetic data from Monte Carlo (MC) estimation is promising alternative but suffers from high noise ratio, which can cause overfitting and hinder large-scale training. In this work, we conduct preliminary study on the noise distribution in synthetic data from MC estimation, identifying that annotation models tend to both underestimate and overestimate step correctness due to limitations in their anno- tation capabilities. Building on these insights, we propose Self-Denoising Monte Carlo Annotation (SCAN), an efficient data synthesis and noise-tolerant learning framework. Our key findings indicate that: (1) Even lightweight models (e.g.. 1.5B parameters) can produce high-quality annotations through self-denoising strategy, enabling PRMs to achieve superior performance with only 6% the inference cost required by vanilla MC estimation. (2) With our robust learning strategy, PRMs effectively learn from this weak supervision, achieving 39.2 F1 score improve- ment (from 19.9 to 59.1) 1) in ProcessBench. Despite using only compact synthetic dataset, our models surpass strong baselines. including those trained on large-scale human-annotated datasets such as PRMS00K. Furthermore, performance continues to improve as we scale up the synthetic data, highlighting the potential of SCAN for scalable, cost-efficient, and robust PRM training.
auman-annotated data.
Introduction
The recent advent of large language models (LLMs) such OpenAl ol [ 13 and DeepSeek RI [9] has sparked significant interest in scaling test-time compute to encourage slower and deeper reasoning processes. In this context, process reward models [1 35 have emerged promising approach, offering grainer,,,,, step-level evaluations that facilitate iterative self-refinement [29 and exploration of solution spaces [18]. This proves be particularly effective in tackling complex problems like mathematical reasoning tasks.
However, critical challenge for developing process reward models (PRMs) lies in data annotation. While numan-annotale methods [17] can produce high-quality data that effectively guides PRM training. they come at a high cost. To address this, many works (35] g 26 have explored synthetic data generation via Monte Carlo estimation. However, synthetic data still falls short of matching the quality of human-annotater data, explored by Zheng et al. [44). The primary challenge stems
Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

from the high noise ratio inherent in Monte Carlo-generater generated data, models tend to quickly overfit noisy samples, hindering effective training at larger data scales. Recent studies 30
28 43 have demonstrated that introducing stronger supervision from large-scale critic models (e.g., Qwen-72B) an effective strategy for mitigating noise by retaining only consensus samples agreed upon by both the critic model and Monte Carlo estimation [5] [4]. However, how noise is distributed and how to train PRMs robustly in the presence of such noise are still underexplored. In this paper, we investigate the full potential of denoising in MC estimation itself and robust learning in PRMs, without relying on any external stronger supervision. To the best of our knowledge, we are the first systematically explore Process Reward Learning from the perspective of noise distribution and robust learning.
We begin with preliminary study to investigate the noise distribution present in synthetic data generated through Monte Carlo estimation. Our findings indicate that this noise arises mainly from two factors: the annotation model tends to under-estimate and over-estimate step correctness, primarily due to inherent limitations in its annotation capabilities. To address this issue, we introduce self-confidence metric designed to assess the reliability of model-generated annotations. Guided by these insights, we propose strategies to mitigate two distinct types of noise: noisy positive samples and inaccurate negative samples. We develop selective sampling approach to reduce overall sample noise and design a model-wise self-denoising loss for robust learning. By leveraging the self-confidence metric, systematically reduce the annotation model's bias and enhance the overall data quality. Furthermore, to improve data synthesis efficiency, we selectively apply Monte Carlo annotation to the most informative samples, optimizing both accuracy and computational resources.
Using weak supervision from lightweight annotation model, Qwen2.5 5-555---hhh----- 1.5h-1.5B-BBa 1.5t-Inttruct construct synthetic dataset consisting of 101K samples. We evaluate the trained PRM from two per- spectives: test-time scaling (best-of-8 evaluation) and step-wise error detection (ProcessBench [44]). The results demonstrate that our model consistently outperforms strong existing baselines, achieving performance comparable to that the human-annotated dataset PRM800K. Further scaling with an additional 97K synthetic samples generated by Llama3.2 2...--BB-Instruue and -.ren...-atth--B--. leads to our PRM surpassing the performance of the ouman-annotate PRM800K on both evaluation benchmarks. As the dataset size continues to grow, observe further scaling potential.
39 we
2
Preliminary: Unveiling the Noise Distribution in Monte Carlo Annotaion
2.1 Problem Definition
Monte Carlo Estimation In the context of process annotation, Monte Carlo estimation 335 is proposed as an automated approach for evaluating the correctness of each step. Formally, given a question q and corresponding response x containing steps, i..., x [Z1, [ Fx). the correct- ness score of the tth step, denoted ct, can be estimated by completer model through multiple rollouts or simulations. Specifically, the completer model is prompted to sample k completions based on the question y and prefix steps X [x....., Fe]
Ø2,...,
until a terminate state is reached.
& & - - I(()) aa), C = Po (T
(1)
i=1
where [r, a²) is an indicator function that equals if the sampled response matches the ground truth answer a", and 0 otherwise.
PRM Training After obtaining the correctness score c, for each step, train the PRM using binary classification loss:
(BC - B(pg [ye (1)
(1 v.)) log(1 Po xcx))],
(2)
where y/ represents the correctness label of for the th step. We use hard labels to annotate y/, defined as y/ = [[c, > OJ, assigning y = 1 for any positive correctness score and yr = otherwise. We consider a step correct if at least one rollout leads to the correct final answer.
Noise issue in MC estimation A significant effectiveness gap exists between synthetic data and human- annotated data 44 for training PRMs. This discrepancy can be attributed to the noise label

---

completer: Liama31-88--nntrruc
completer: 1amma..--8-Instreee
completer: 1amma.1--8--nnttere
002
Torer (Noise)
Gyer inf, Dose in! Spee fyu
125
Taow lise
Torg Tree
Taver Tive
Count
600
(Clean)
Count
200
faver Ieee
75
403
50-
200
25-
completar:
wwwnn2.-Mathh. 1a-tth--B-nsstucct
Qwen M-5tttt------ M.aa5
600
Tese Trce (Nois))
Gave lose
200
laver Tive
(Noise)
fyee
farw lase
Count
400
Terer in Tree (Cleon)
Count
Jase lase
200
202
01
215
216
419
5.
7/8
56
Low Conf
Via Cond
High Cori
Model's Confiione
Self Contiidnne Level
Figure I: Noise distribution analysis of Llama3. --I--nnsttutt and Qwen2.5-Math, Baaahh-7BInstruce Left: Overall distribution of noise samples across varying self-confidence levels. Middle: Noise distribu- tion in predicted positive samples where tpred = inf. Right: Distance distribution between pred and ttrue for inaccurate negative samples. Additional results of more models can be found in Figure
where the correctness of each reasoning step is often misestimated, primarily due the limitations the completer model. More specifically, the correctness of the current step is annotated by the completer model based on whether the completions lead the correct final answer. As result, the PRM trained on this data only estimates the potential of the "completer model" reach the correct final answer from the current step. This differs significantly from correctness of the current step, as the completer model is not perfect and can make mistakes. This is why the noise phenomenon exists in Monte Carlo estimation.
issue inherent in Monte Carlo estimation 18 42
2.2 Noise Distribution in MC estimation
Given a question g and corresponding response x [ T2....; F.,) we focus only on the first error step during the training 17 and evaluation 44 of the PRM. We denote the first error location predicted by the completer through Monte Carlo estimation as lpred, while the ground truth error location is bewwe- For cases where the response is entirely correct, use the label "inf" (i.e., +oo) to indicate the absence any errors.
Then we can categorize the mistakes made by the completer model into the following two types (fully correct samples are also taken into account):
Under-Fstinge Estimation (ppred < lrue):: This likely occurs because the completer struggles with complex or nuanced reasoning. Even when provided with correct prefix steps, may fail to generate correct rollouts, leading to early error detection.
(fpred > (true): This happens when the completer initially corrects an error, producing correct rollout. However, subsequent errors accumulate, the model cannot fully address them, causing delayed detection of the true error location.
To quantify this relationship between the noise distribution and the completer's problem-solving capability, we propose a metric called self-confidence, which measures the confidence level of the completer model the annotated question:
SCo(g) Pe-
where J(r, a²) an indicator function that evaluates whether the response r matches the ground truth answer a". In practice, SCo (q) is estimated by computing the empirical mean over multiple randomly sampled responses given question 4.
Experimental Settings We select four representative open-source models, i.e., Llama3.1- 1- 8n-trutt Llama3.2-3B--Inste (6]. Qwen2.5-Math- 1.BBahh-t.BInsttruct and 0wen2.5-Math-77 Instruct [39], the completer models. For the dataset, ProcessBench 44], which contains

---

3,400 annootaed process data points spanning multiple difficulty levels. To compute self- confidence, we sample 16 completions for each question. For evaluating step correctness, we perform 8 rollouts per step to determine its correctness.
Key Observations The left of figure illustrates the overall noise distribution of different models across different self-confidence values We mainly have the following observations:
Observation 1:
Noisy cases where tpred < ttrue are predominantly concentrated in low self- confidence regions, supporting the under-estimation hypothesis discussed before.
Observation 2: The noise distribution of lpred > Itrue varies across models. For the more capable Qwen model, its stronger error-correction ability results in noise being concentrated in high self-confidence regions. In contrast, noise more evenly distributed for the Llama model. Observation 3: Clean samples (ppred= lerue) are primarily located in high self-confidence regions. Overall, high-confidence regions exhibit lower proportion of noise.
2.3 Detailed Analysis of Noise Distribution
At a finer granularity, we classify noise into two distinct categories: Noisy Positive Samples and Inaccurate Negative Samples.
Noisy Positive Samples Samples observed as fully correct positives with (pred inf, but actually contain errors, i.e., truee * inf. These cases indicate that the model fails to detect existing errors, leading to noisy positives its predictions. For clarity, we categorize self-confidence into three levels: low confidence as [0, 0,25], medium confidence as (0.25, 0.75), and high confidence as [0.75, 1]. These thresholds are empirically determined through the overall noise distribution. The middle of figure [ illustrates the distribution of noise positive samples, from which we can conclude that:
Observation 4: For predicted positive samples (fyred = inf), the noise positive ratio is significantly lower in high self-confidence samples, making them more suitable for training.
Inaccurate Negative Samples Samples observed as containing errors (tpred * inf), but the predicted error location is incorrect, i.c., tpred # tyrue- This reflects the model's inability to precisely identify the true error location, even when it detects the presence of errors. From Observations 3, 4, we can conclude that the completer model makes fewer mistakes in high self-confidence samples. Therefore, we focus on the high-confidencee subset investigate how noise is distributed inaccurate negative samples. We visualize the relationship between ppedd and ttrue in the right of figure T from which we can conclude:
Observation 5: In most cases, the model can roughly predict the error location but often lacks precision. Furthermore, the model tends overestimate the error location, i.e., fpred >> Itraes and the number of noisy samples decreases as the deviation increases. This also supports the over-estimation hypothesis discussed before.
3 Methodology
Building on the observations of noise distribution discussed in section 2 we propose targeted approaches to address key challenges in Monte Carlo annotation. Specifically, our method contains two modules: (1) an efficient data synthesis framework to reduce substantial inference costs, and (2) robust training methods to mitigate the high noise ratio synthetic data and enable robust learning with noisy labels. Figure 2 illustrates the overall workflow of our proposed method.
3.1 Efficient Data Synthesis Framework
Generate Responses
The synthesis process begins with dataset of questions and their corre- sponding golden answers, denoted as D {(g, M- For each question y,, a generator model
We include fully correct samples, assigning them error location label of +oo for convenience.

---

Module 1: Efficient Data Synthesis Framework
Module 2: Robust Training
Problem &
Positive Samples
Negative Samples
Response
Generator
Protlem Response
(94%)
Problem:
Portive sarpha
for sact ouartion
Requate
bi Sapp
Negative Samples (6%)
Solf-Confidence 66.7%
Mocificttoo
Cord based Fibering
Problem & Resporse
Probiem & Response
Prablees
Completer Model
Pretlers
Collect
2J
Golden oraver
Soldes
Napaha extple saToot
Conf-wise Reweighing
Requeto::
the comeciress each
Shep
Step
100% sample ultilization
only 6% inference time
Figure 2: Overview of our data synthesis and robust training framework.
parameterized by Ti produces responses, denoted as of the generator in % is then computed as:
The confidence score
SC (4:)
N Σ ,a:,, where r; Po(. (
where [rr a') evaluates the correctness of the generated response. Next, we collect the negative samples with ipred * inf from these responses for further step-wise annotation. this process, we do not select the annotated correct (positive) samples for subsequent annotation. Although these positive samples may contain false positives that could potentially be filtered out through detailed step-wise correctness checks, we find that the annotation cost for this process is prohibitively high. For instance, performing rollouts for each step in 10-step response requires 8 Moreover, the positive samples, particularly those in high self-confidence regions, contain minimal noise (from Observation 4). Therefore, we directly use these high-confidence samples as positive examples for training without further annotation. By applying Monte Carlo estimation exclusively to negative samples, we ensure I 100% sample dillizationevve sample annotated through Monte Carlo estimation is included in the final training dataset.
80 rollouts per sample.
Step-wise correciness annotation Negative samples with high self-confidence scores, SCx, (4:) > selected for step- wise correctness annotation using completer model parameterized by 722 We employ vLLM [15] and implement distributed inference via Ray [2 to accelerate the annotation process. To support the subsequent robust learning process, also need collect the self-confidence scores SC,,,o (9:) during annotation. For convenience, we use the same model as the generator and completer models, i.e., æ π2 n, allowing us reuse the self-confidence scores and further enhance data generation efficiency.
3.2 Robust Learning with Noisy Labels
The final annotated dataset, denoted as Diinat, consists of tuples (q, SC+ where q is the .... cx] contains the
question, x = [I1, T22 2n. represents the n-step responses, and c [Ca, ....... corresponding correctness scores annotated via Monte Carlo (MC) estimation. The term SCx (q) denotes the self-confidence score of the completer model æ for question q. We then train PRMs with the reweighted step label:
(oCAN(0) E(ego.v))DDal [yr log( (P xx+)) + (1 (1- w:) Poluelg,
(c.(Sc/S((()) 0 = I[a > 0),
if if t Otherwise
where G: = =P = correct x x
(5)
where ffeed denotes the first emoor location with c, 0. Compared to the traditional BCE loss. modifications focus two main aspects: noise-tolerant labeling and confidence-wirg reweighting.
5

---

Noise-tolerant Labeling The completer model tends to overestimate the correctness of the current step due its strong self-correction capability. As enrors continue accumulate. the model eventually makes mistakes, leading to 'predd > ltrues with high probability of similar errors occurring at nearby positions (Observation 5). To enable more robust learning with these noisy labels, we propose noise-tolerant labeling strategy that applies soft labels to steps preceding the error, within tolerance distance d. We discuss the choice in section 4.4 Through experiments, we demonstrate that the strategy allows the PRM to learn more effectively from noisy labels without overfitting.
Confidence- wise Reweighting After applying the denoising process, a critical issue remains: the annotated labels are still heavily influenced by the capability of the completer model, i.e.,
= Ply: correct Tgold, 9, Xgt), while C; = P(y = correct no, g. xgt).
(6)
Here, of represents the true correctness probability assigned by an assumed golden annotator Tgold- while o is the estimated probability derived from the completer model x Since we can only estimate e, through rollouts performed by the imperfect completer тæ, the true label cf remains unobservable. To mitigate the model-dependent bias, introduce correction factor ó; = of /c,. We leverage the self-confidence score SC,(0) to approximate and reduce this model-induced bias. The rationale behind this approach is follows: consider two models annotating the same sample- strong model with Stnnnng (q) and a weak model with We (q). Naturally, the stronger model will yield a higher correctness score C₁. However, regardless of their inherent strengths. Thus, we adjust the estimated correctness score using the self-confidence score as follows:
aim for the final corrected scores to be consistent across models,
= min((a/SC..(())
This adjustment helps to normalize the influence of model capability on the annotated labels, leading more reliable and unbiased training data. This reweighting procedure is particularly effective when integrating annotations from multiple completer models, as demonstrated by the experimental results in section 4.3
4 Experiment
4.1 Training Dataset Construction
SCAN-Base (101K) We use 7,500 questions with golden answers from the MATH training set. For each question, we generate k responses and perform eight rollouts per step to annotate process correctness via MC estimation. We set tolerance distance SCAN-Base is generated by Qwen2.5-Math-- 1.5--truttt [39 as both the generator and completer. We experiment with varying numbers of responses from & E (64, 128} and found that larger number of responses provides more accurate estimation of the model's self-confidence, which is crucial for denoising and robust learning processes, as elaborated in Appendix [B.4] Additionally, we explore other data sources, including GSM8K (2) and Numina-Math [16], and discover that the MATH dataset offers a more suitable level of difficulty and higher data quality for training (Appendix B.4).
2, with discussion in section 4.4
SCAN-Pro (197K) We further incorporate Qwen2.5-Math- -nnstruct 39 and Llama3,2-33 3- Instruct (6) for Monte Carlo estimation. With these two models, we generate an additional 97K data points. Integrated with SCAN-Base. we construct mixed dataset containing 197K samples. More dataset details are shown in Appendix(A.1 A.T We observe that (1) combining these two datasets increases the diversity of the data, thereby boosting model performance, and (2) the misel-wise reweighted loss design mitigates any data inconsistency arising from the capability gap between the annotation models, with experiment results in section 4.4
4.2 Experimental Setup
We evaluate the effectiveness of the Process Reward Model (PRM) from two key perspectives:
Best-of-N (BoN) Evaluation In this evaluation, the PRM functions verifier to select the best response from multiple candidate answers generated by policy model. Specifically, the PRM assigns scores to each step within response and then aggregates these step- wise scores into overall

---

Table 1: Best-of-8 evaluation results of the policy model -eaenn..--tt-------ee Results of
policy model 1-8--3..1---Istttttt can be found in Appendix B.3] 3 re-evaluated all these models based the same of responses.
To reduce potential errors, we
Training
Annotation Method
College Math
Olympiad Bench
Model
GSMSK
MATH
Avg.
Samples
Greedy
95.6
83.6
46.9
40.6
66.7
Majority Vote@8
96.9
87.3
47.4
43.0
68.7
Pass@8 8 (Upper Bound)
98.0
92.0
52.3
60.4
75.7
UniversalPRM-77
690K
MC,
96.8
86.9
47.6
48.0
69.8
Qwen2.5-Math-PRRe
1500K
MC.
96.8
88.1
47.7
47.6
RLHFFowPPMM- BiM--Mstral-8BB
273K
MC-Only
97.1
87.3
47.3
43.0
68.7
253K
MC-Only
96.8
87.3
47.9
43,9
69.0
FurusPRM-Stagel
463K
Implicit
96.9
86.0
47.4
42.8
68.3
BurusPRM-Stagc2
693K
Implicit
97.0
86.7
47.9
45.3
69.2
Math-PSA-7B
1395K
MC-Only
96.4
86.0
47.7
45.9
69.0
Qwen2.5-Math-72
Baath--B--Mtt--hee
445K
MC-Only
96.9
86.8
47.6
42.3
68.3
Skywork-PRM Nowwok-PRM-QQee... FRRM--we2255777
97.0
87.9
47.8
44.6
69.3
Qwen2.5-Math-79
PB-M------- PRRMO0OK
264K
Human
87.6
47.7
45.0
Swen2.5-Math-7B-SS5 7B-SCAN-Bass
101K
MC-Only
97.1
86.9
47.8
44.4
Qwen2.5 -att2-5--Mth--B--555 Bauth-BBSSAA--PP SCAN--ro
197K
MC-Only
47.7
Table 2: Evaluation Results on ProcessBench. MC denotes Monte Carlo estimation, while KD
represents knowledge distillation from more capable critic models (with 32B or larger parameters).
GSMSK
MATH
Olympiad Bench
Omni Math
Avg.
Model
F1
error
F1
F1
F1
Cririe Models ILLM-as-a--uudel)
GPT-40-0080
63.6
51.4
Qwen2.5 -aau2.5- BB-nstruet
15.5
100.0
26.8
14.8
96.8
25.7
7.7
91.7
14.2
6.9
88.0
12.7
19.9
Llama-3. 3.3--33..00------ Instuutt
72.5
96.9
82.9
43.3
83.2
59.4
31.0
94.1
46.7
28.2
90.5
43.0
Qwen2.5-72B-Inste
62.8
76.2
46.3
61.8
38.7
54.6
36.6
52.2
QuQ-33B-Pheview
81.6
95.3
88.0
78.1
79.3
78.7
61.4
54.6
57.8
55.7
68.0
61.3
71.5
Process Reward Models (MC KD)
UniversalPRM-1
85.8
77.7
67.6
66.4
74.3
Owenn.5-MathhPRRE.
72.0
96.4
82.4
68.0
90.4
77.6
55.7
85.5
67.5
55.2
83.0
66.3
73.5
7-8B Process Reward Models (w/o KD)
RLHFlow-PRRM
33.8
99.0
50.4
21.7
72.2
33.4
8.2
43.1
13.8
9.6
45.2
15.8
28.4
24.2
98.4
38.8
21.4
33.8
10.1
16.9
10.9
16.9
FunuPPM--Saan Stagel
46.9
42.0
44.3
33.3
38.2
35.6
23.9
19.8
21.7
21.9
24.5
23.1
31.2
EurusPMM-Stagee Stage2
51.2
44.0
47.3
36.4
35.0
35.7
25.7
18.0
21.2
23.1
19.1
20.9
31.3
Bwen2.5-Math-7B-Maaa
46.4
95.9
62.5
18.9
31.6
7.4
13.7
95.0
7.7
61.8
82.9
70.8
43,8
62.2
53.6
17.9
31.9
22.9
14.0
41.9
21.0
42.1
Swen2,5-Math-7B-PPP
53.1
95.3
68.2
48.0
90.1
62.6
35.7
87.3
50.7
29.8
86.3
44.3
56.5
Qwen2.5-Math Base
67.1
73.8
61.7
45.2
Qwen2.5 Math-7B 7B 17---SCA--PP Pro
72.9
90.7
80.9
58.6
73.6
65.3
44.2
47.8
45.9
37.8
53.1
44.2
59.1
reward score for the entire response. The response with the highest reward score is selected the final answer. For the aggregation process, we take the lowest score among all steps as the overall
reward score. Further experiments and discussions regarding different aggregation methods
provided in Appendix [ The evaluation datasets cover various difficulty levels, including GSM8K 2
(elementary), MATH [ (competition), College Math (31 (college), and Olympiad Bench (10 (Olympiad). We employ Sween..5-Math-7B.. IB-ssrructt and Llama3. 1-..3..188---1 1-8B-Instruct as the policy model and set N
reasonably practical setting in real-world applications). We report the majority voting 36) as the baseline and pass @8 as the upper bound.
Step-wise Error Detection We further investigate the PRM's ability to detect error locations within responses accurately. We use ProcessBench [44] as the evaluation benchmark, which measures the PRM's capability to identify the first error location the model's performance in recognizing fully correct samples and accurately identifying the error in incorrect responses. Formally, the evaluation metrics can be defined as: The final F1 score the harmonic mean of the accuracies correct and erroneous samples.
given response. This evaluation focuses on

---

of
Bust-ol Accuracy
60.5
Procods dendh Ovain
52-
50
Overfiti
50-
534
48
Overfit!
46
#ads8ne
46-
44
Training OK)
Training
Fitanng
Lipding
(SCA-79))
Figure 3: Ablation results in BoN evaluation and ProcessBench. For ProcessBench, we directly calculate the overall Fl score of full samples. Left: Scaling curve of the PRM training of different datasets. Middle: Scaling curve of selection of tolerance distance. Right: Effectiveness of each component. "Baseline" here represents the vanilla MC estimation method.
Compared Baselines Our primary comparisons against 7B-scale process reward models, including Math-Shephen [35], RLHFlow-PPe [38. Skywork-PRM [23,, Math-PSA (4,, and EurusPRM 40 3 We also include models trained with strong supervision, such Qwen2.5- Math-PRM-78B and UniversalPRM[30 as points of reference. However, we do not include direct comparisons with these strongly supervised models, they rely large-scale external critic models for guidance. As shown in Table 2 supervision, typically via knowledge distillation, plays key role in the final PRM performance. However, we address different issue from these approaches, i.e., denoising MC estimation itseif. For the Best-of-N evaluation, we directly test the publicly available checkpoints of the compared models. For ProcessBench, the results are directly sourced from Zheng et al. (4).
these critic models are already highly capable, and their
4.3 Main Results
Owen2.5-Math-7B: Table presents the Best-of-8 evaluation results based on the policy model With only 101K synthetic samples from the SCAN-Base dataset, gener- ated by the 1.5B model, the Swen2.5-Math-7B-SCAA. larger synthetic datasets and approaches the performance PRMs trained on human-annotated data (PRM800K), Notably, when trained with additional synthetic data generated by the 7B model, the resulting -aaa2.5-MMt--7B-SCAA
Best-of-8 Evaluation
model outperforms PRMs trained on
model surpasses PRM800K. This demonstrates that even with 1 1.5B model, it is possible to synthesize data of comparable quality to human annotations.
ProcessBench Table presents the evaluation results on ProcessBench. Both Qwen2.5-Man 1.2.5---tt-7---- SCAN-Bare and Owen2.5-Maate those trained on PRMS0OK. Remarkably, their eтror detection capabilities even surpass those of the 70B-scale critic model Llama-3.3-700B-nnntt
outperform all other process reward models, including
Through the SCAN approach, Qwen2.5-Math-7BB able to generate process data for self-training. leading to a substantial improvement in its own error detection capability (from 19.9 59.1), demonstrating strong celf-improvement potential.
4.4 Ablation Study
We conducted an ablation study to further demonstrate the effectiveness of each component. Addi- tional results can be seen in Appendix
Scaling Curve of SCAN-Base and SCAN-Pro The left of figure 3 illustrates the performance variation of the model during training. We observe that without any denoising strategy, the model quickly overfits to noisy samples (see Baseline results). Our denoising strategy enables the model to grow steadily without overfitting to noisy samples and further improves the model's performance. By comparing the results of SCAN-Base and SCAN-Pro, we conclude that incorporating additional data sources enhances the diversity of the data, which in turn optimizes the model's upper bound.
Choices of Tolerance Distance Choosing appropriate tolerance distance d is critical, both very small and very large values can introduce noise. When d 0, it results in hard labeling, leading to severe noise, as shown in the scaling curve. Conversely, when becomes soft labeling,

---

which also adds significant noise and hinders scaling 42 The middle of figure shows the results for different values of d. We find that 2 offers good balance, reducing overfitting during training.
Effectiveness of each component The right of TigureB the effectiveness of each component. We begin with a baseline where denoising strategy is applied to the SCAN-Pro dataset. As we progressively incorporate denoising techniques, we observe consistent improvements in model performance across both best-of-n and ProcessBench evaluations. Both tolerance distance labeling and confidence reweighting contribute to performance gains. Tolerance distance labeling enhances the model's robustness when handling noisy samples, while confidence reweighting helps de-bias the probability estimation from different models on annotated samples.
Related Work
Reward Models in Reasoning Tasks Reward models play crucial role in enhancing the capabili- ties of large language models (LLMs), particularly in complex reasoning tasks such as mathematical
problem-solving 6 39] and competitive programming [2] In this context, reward models act as
verifiers to assess the correctness of generated responses 4 or directly enhance LLM capabilities
through preference alignment [9 27.. Unlike outcome reward models (ORMs), which evaluate only
the correctness of the final answer, process reward models [32
20 (PRMs) provide more fine-grained
evaluations by assessing each step the reasoning process. Recent advancements 35 18
have
demonstrated the significant potential of PRMs in scaling test-time compute [26 42 and preference
learning 35 further highlighting their importance the development of more capable and reliable LLMs.
Process Reward Learning Human-annotated data 20 is the primary solution, but due to the complexity of the task, manual annotation is highly costly. Recent works show the potential of Monte Carlo estimation as a promising alternative; however, also introduces considerable noise [3 18
This noise be mitigated by incorporating critic models [7141737 37), where LLMs prompted to assess the correctness of each step directly. However, recent research [4] indicates that only large-scale models (e.g., Swven2.5-Math-7 (aaah-72BBInstruel possess strong critic capabilities. Thus, this approach essentially distills the error-detectio ability of large models into smaller process reward models through data distillation. Numerous works extend the strong supervision more complex methods, like code verification 43), reverse verification [30], and data synthesis for preference learning (28). Another line of work (40 3 focuses on a different training paradigm by simply training an ORM on response-level labels, as optimized language models inherently function as reward models well (24]. Our work takes a novel moisy-leanning perspective and aims address two key challenges in Monte Carlo annotation: high computational overhead and high noise ratio
problem.
Conclusion
In this paper, introduce an effective approach to improving process reward models from the perspective of data annotation. We first conduct a preliminary study to reveal the noise distribution in the Monte Carlo annotation process by introducing confidence metric. We then propose efficient data synthesis and robust learning framework to address the key challenges in Monte Carlo estimation. Through extensive experiments, demonstrate the effectiveness of our proposed approach and the potential of model self self-improvemmnt from a robust learning perspective.
Acknowledgement
We want to thank all the anonymous reviewers for their valuable comments. This work was supported by the National Science Foundation of China (NSFC No. 62206194), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488), and the Young Elite Scientists Sponsorship Program by CAST (2023QNRC001), We also acknowledge MetaStone Tech. Co. for providing us with the software, optimisation on high performance computing and computational resources required by this work.

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08772 2023.
[2] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
[3] Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint urXiv:2502.014555 2025.
[4] Yuyang Ding, Xinyu Shi, Xiaobo Liang, Juntao Li, Qiaoming Zhu, and Min Zhang. Unleashing reasoning capability of Ilms via scalable question synthesis from scratch. arXiv preprint arXiv:2410. 18693, 2024.
[5] Keyu Duan. Zichen Liu, Xin Mao, Tianyu Pang, Changyu Chen, Qiguang Chen, Michael Qizhe Shieh, and Longxu Dou. Efficient process reward model training via active learning. arXiv preprint arXiv:2504. 10559, 2025.
[6] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Daalle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint ar arXiv:2407.21783 2024.
[7] Bofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Keming Lu, Junyang Lin, Chang Zhou, Wen Xiao, et al. Llm crities help catch bugs in mathematics: Towards better mathematical verifier with natural language feedback. CoRR, 2024.
[8] Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang, rstar-math: Small Ilms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04511 2025.
[9] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint urXiv:2501.12944 2025.
[10] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. /4008, 2024.
[11] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0387. 2021.
[12] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu. Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024.
[13] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameyy, al. Openai ol system card. arXiv preprint arXiv:2412.16721 2024.
[14] Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jackyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, and Lu Wang. Process reward models that think. arXiv preprint ar0iv:2504. 16828, 2025.
[15] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph F. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
10

---

[16] Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang. Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
[17] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305. 2023.
[18] Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint
2tt::2406006992, 2024.
[19] Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. arXiv preprint NXiv:2401.00966 2024.
[20] Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. Let's reward step by step: Step-level reward model as the navigators for reasoning. arXiv preprint arXiv:23 10080, 2023.
[21] Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Llm critics help catch Ilm bugs. arXiv preprint arXiv:2407.002/5 2024.
[22] Philipp Moritz, Robert Nishihara, Stephanie Wang. Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, al. Ray: A distributed framework for emerging {AI] applications. In 13th USENIX symposium on operating systems design and implementation (OSDI 18), pages 561-577, 2018.
[23] Skywork o1 Team. Skywork-ol open series. https://bugiingface.c ber 2024.
Novem-
[24] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[25] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
[26] Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. arXiv preprint arXiv:24 2024.
[27] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint X2x:22402.0030000 2024.
[28] Shuaijie She, Junxiao Liu, Yifeng Liu, Jiajun Chen, Xin Huang, and Shujian Huang. R-prm: Reasoning-driven process reward modeling. arXiv preprint 2rXiv:2503.2/295 2025.
[29] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 11m test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 2024.
[30] Xiaoyu Tan, Tianchu Yao, Chao Qu, Bin Li, Minghao Yang. Dakuan Lu, Haozhe Wang, Xihe Qiu, Wei Chu, Yinghui Xu, et al. Aurora: Automated training framework of universal process re- ward models via ensemble prompting and reverse verification. arXiv preprint arXiv:2502.1 1520, 2025.
[31] Zhengyang Tang, Xingxing Zhang, Benyou Wan, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning, arXiv preprint 2tt.:22433002884. 2024.
11

---

[32] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprini urXiv:2211.14277 2022.
[33] Chaojie Wang. Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and Bo An. Q*: Improving multi-step reasoning for llms with deliberative planning. arXiv preprint 0rXiv:2406.14283 2024.
[34] Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, et al. Openr: An open source framework for advanced reasoning with large language models. arXiv preprint arXiv:2410.09671, 2024.
[35] Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: A label-free step-by-step verifier for llms in mathematical reasoning. arXiv preprint arXiv:2312.008935 2023.
[36] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint urXiv:2203.11171 2022.
[37] Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu, Evaluating mathematical reasoning beyond accuracy. arXiv preprint DeXii:240400562. 2024.
[38] Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang. An implementation of generative prm. https:// http:::///tttt..g.. com Btt.///H//O////RRRRReEAEAA Rewardd 2024.
[39] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu. Jiamhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via seff-improvement, arXiv preprinl arXiv:2409. /2/22, 2024.
[40] Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint arXiv:2412.019811 2024.
[41] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.1 15240, 2024.
[42] Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07300) 2025.
[43] Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, et al. Genprm: Scaling test-time compute of process reward models via generative reasoning. arXiv preprint arXiv:2504.0055: 2025.
[44] Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical reasoning. arXiv preprint arXiv22412.06552 2024.
[45] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging Ilm-as- aun-dge- with mt- bench and chatbot arena, 2023.
12

---

A Data Synthesis and Training Details
A.1
Data Synthesis Details
Table 3: Data component of SCAN-Base and SCAN-Pro datasets.
Generate Response
Perforn Rollouts
Dataset
Samples
Generator & Completer
GPU hours
k
temp.
top_p
lemp.
top_p
15K
Qwen2.5-Math 1.55-.nnttrcc
64
0.7
0.8
0.7
0.8
16K
1.0
1.0
0.7
0.8
SCAN-Base
174.4
31K
Swen2.5-Math-1.5BB11
128
0.7
0.8
0.8
40K
128
1.0
0.7
0.8
13K
0wet2.5-Math-77
BB-athhh--ssttuu Instrrcct
64
0.7
0.8
0.7
0.8
SCAN-Pro (Increment)
13K
Qwen2.5-Math- -B-ath--B-Instruct
1.0
0.8
200.1
71K
1aamaa.2-3B-nntmuue
128
1.0
0.8
SCAN-Pro (Full)
374.5
(Total)
Data Components and Inference Cost Table presents the composition of the SCAN-Base and SCAN-Pro datasets, along with specific hyperparameter settings, including the total inference cost. Through our efficient framework design, we generated 197K samples using 374.5 GPU hours. All experiments were conducted single machine equipped with 8 GPUs, requiring only 47 hours in real time.
Monte Carlo Tree Search Regarding the choice inference strategy, we found that the commonly used Monte Carlo Tree Search (MCTS) method is inefficient for large-scale data synthesis. Annotating a single sample with MCTS involves multiple sequential steps, where each action depends on the state of the tree, making parallelization impractical. Additionally, maintaining extensive information for each sample results in substantial memory overhead. As an alternative, we adopted the vanilla Monte Carlo method for annotation. This approach not only delivers strong annotation performance but also achieves results comparable to those of MCTS [88 while offering significantly higher efficiency.
node
Implementation Details For inference deployment, we explored two approaches: (1) launching multiple vL..M servers and making API calls, using distributed routing strategies powered by FastChat [45), and (2) batching multiple queries and utilizing Ray to schedule resources for distributed inference. We found the second approach be significantly more efficient, achieving over twice the speed of the first method. When estimating step-wise correctness via Monte Carlo (MC) estimation, we begin from the first step of the response and proceed sequentially until encountering the first position where = 0, at which point the evaluation stops. While this introduces some dependency between steps. the depth of dependency limited to the number of steps, which is considerably shallower and less complex compared to the recursive depth of Monte Carlo Tree Search (MCTS).
A.2 PRM Training
We train our process reward models based on Brren..5-Mtth-BB-B1 rate of x 10-6 and batch size of 128. The model is trained for one epoch, as we observed that multiple epochs lead to rapid overfitting, particularly on synthetic data. Although overfitting occurs more slowly on annottaed data, it remains a concern with extended training.
with constant learning
B Additional results
B.1
Noise Distribution Across More Models
We present the noise distribution of additional models in Figure including 1-8-a3.--8--Instuuue Qwen2.5-Math- 1.5h-11.B-Insttrrr and Diwen2.--Math-7B-mme Overall, the observations discussed in section 2 remain consistent across these models well.
13

---

completer: Liama32238B-nsstuu
completer: L.mma.2-38-
completer: 1lama3.-28--nntteee
Torer (Noise)
Cyer inf, Dose in!
Tuow lase
Torg
Spee Yyu
100
Tuoe Tixe
Count
600
(Clean)
Count
faner Ieee
400
200
200
25
completer: Libma.118B-inntrre
completer: Llama3.1-88 Inst-nstuuctt
completeri Libma3.108--nsttuuct
lere si frce (Noise)
Spee fosa
Sarer lixe
Tere?h (Noise)
Tar Taxe
Count
Teom in biee (Clean)
Count
100
Sace laxe
75
56
0.1
1.00
200
120
4m
5:1
6/8
7m
bn
Low Cont
Mid Cond
High Gord
completer: Qwen2. Swen2.5-Muth-1.38B 5-W.ath--------isttrrtt
completer: Qwen2.5-Maan 58-IIsstru
completer: Qwen225-Math-1.-05 3-Math-1-5B-mmtrruc
(Noise)
600
Core
Saru Tupe
Terer True (Notse)
Gree - Oppe = In!
Jaco Taue
400
Count
200
200
Low
High
completer: Qwon2.5-Math-77
completer: Qwen2.5.Mathh
completer: Qwen S---ath------nteeeee
800
Tese Tece (Noise)
fpee winf, fosa in
Taver hae
602
fuoe inf
Toxer (Clean)
Count
Count
200
Jasu luse
403
203
4/8
Low
Model's Self-Conffdence
SelfCanffidecc Level
Figure 4: Noise distribution of additional models. Similar observations can be concluded across these models, further validating the consistency of our findings.
Table 4: Best-of-8 evaluation results of ----aat--------N- and Qwen2.5-Math-78
SCAN-Pro using different aggregation methods.
Aggregation Method
College Math
Olympiad Bench
Models
GSMSK
MATH
Average
Min-Max
97.1
86.9
47.8
44.4
69.1
Min- Vote
96.9
87.5
47.7
44.9
69.3
Qwen2.5-Matt -a..55-MMt-------- AB-CANB- Base
Last-Max
96.8
86.8
48.1
45.3
69.3
Last-Vote
96.8
87.4
47.6
45.3
69.3
Min-Max
97.1
87.3
48.1
47.7
70.1
Min-Vore Vote
97.2
87.8
47.9
46.7
70.2
-een2.5-Math7BB-SCA.
Last-Max
96.9
86.4
44.2
68.9
Last- Vote
96.7
87.3
47.7
45.5
69.3
B.2 Aggregation Methods
We experiment with the following four aggregation methods:
Min-Max:
This is the method used in our main experiments. The final selected response is given
by r* argmax, min(p), where p = [pI.p2..... ..p] represents the predicted step scores of trained process reward model.
Last-Max: The final selected response argmax, where pt the reward score of the last step.
14

---

Min-Vote: Inspired by the majority vote method, we designed a weighted voting strategy. The final
R Il(;; - a) min(p), where I(a; a) is indicator function that equals if the i-th response matches answer and 0 otherwise.
answer computed at argmax
R
Vost-Vote: The final answer is computed a* argmax, Σ I(a = pr.
Table 4 presents the results. We observe that the voting-based final answer yields better performance compared to the Min-based strategy. Therefore, if the primary objective is to obtain a single accurate answer, the Min-Vote strategy may be the more effective choice.
B.3
BoN results on 1lamaa31-8B-Instrue
Table Best-of-8 evaluation results of the policy model Llama3. 1-8B-Instruct.
Training Samples
Annotation Method
College Math
Olympiad Bench
Model
GSM8K
MATH
Avg.
Greedy
86.1
51.5
34.0
16.9
47.1
Majority Vote@8
90.5
24.7
Pass@8 (Upper Bound)
95.7
75.6
48.3
40.2
Owen2.5-Math-PPee
1500K
MC. KD
92.5
63.3
40.2
25.6
55.4
UniversalPRM-78
690K
MC, KD
93.3
65.6
40.4
26.5
56.5
RLHFFlww
SHAiFlow-PRM-issiie PRM-Mistral-8B
273K
MC-Only
37.7
RLHHllo-PRM--DeeppSe Deeee
253K
MC-Only
60.6
37.1
24.7
53.2
EurusPRM-Stagel
463K
Implicit
93.0
64.7
40.7
28.3
56.7
BunssPRMSStage
693K
Implicit
93.4
66.4
41.3
28.6
57.4
Math-PSA-7B
1395K
MC-Only
92.5
63.8
39.8
27.7
56.0
93.3
67.1
41.3
28.4
57.5
Qwen2.5-Math-7
--ath---- 7B-PRM00KK
264K
Human
92.0
64.0
40.9
28.2
56.3
Owen2.5-Math-7B- SCAN--asc-
101K
MC-Only
93.1
64.8
40.9
27.6
56.6
Qwen2.5 1reen25-MMth-22- ------------------ SCAN-Pro
197K
MC-Only
93.0
65.8
41.5
28.4
57.2
In addition to the Qwen model, we also evaluated the performance of our trained PRMs Llama3. 1-
8B- Instrucct.. The responses were generated with a temperature setting of 0.5. Table prreentss the results, showing that with only 197K data samples, our PRM achieves performance comparable to other PRMs and outperforms the human- annaantdd PRM800K dataset.
B.4 Ablation in Data Components
Best-ofNN
ProcessBench
50
Best-of-8 Accuracy
70-
69.4
09.1
$9.8
69.7
70.2
56
$4.0
53.9
53.9
-54
Bench
66-
50.7
51.7
52
50
48.2
48.3
40
46
resiquestion
Only (SCAN-Buse, LOLK)
S0K
J0K
(L10K)
(129K)
(15UK)
(L55K)
(222K)
Figure 5: Left: Ablation results of different responses per question estimating self confidenee value. Right: Ablation results on external data sources.
Accurate Estimation of Self-Confidence We estimate the model's self-confidence for given question by sampling multiple responses. Our findings indicate that more accurate estimation of self-confidence scores significantly improves data quality, primarily in: (1) more precise selection
15

---

of high-confidence samples, as high-confidence samples tend to have lower noise, and (2) more accurate reweighting of step-wise correctness scores, reducing bias in the learning process. We experiment with different numbers of responses per question, setting k E {64, 128). Notably, a larger leads to more precise self-confidence estimation. The left of 5 fair comparison, we maintain fixed training dataset size of 30K samples. Our results demonstrate that a more accurate self-confidence estimation brings significant improvements in both Best-of-8 and ProcessBench evaluations.
the results. To ensure
Incorporation More Data Sources We explore the potential incorporating additional data sources, including GSM8K [2] and Numina-Math [16]. Numina-Mati consists of various data sources, such as Olympiad problems and synthetic math datasets. We selected specific subsets from for our experiments. improvement; however, we find that generating responses for GSM8K incurs a higher computational cost compared to MATH. This is because the model makes fewer emrors when generating responses for GSM8K, given its relatively simpler problem set. Moreover, we observe that incorporating subsets of synthetic math data significantly degrades performance. Upon further investigation, we suspect this is due high proportion of unsolvable problems or incorrect reference answers the dataset. Overall, we find that problem difficulty and the quality of question-answer pairs are two key factors that significantly impact performance.
the results. Incorporating GSM8K provides a certain degree of
B.5 Reduction in Noise Ratio
We further investigate whether our method effectively reduces the noisy data. To quantify this, we measure the noise ratio levels. While direct noise measurement is infeasible for our generated data due to the absence of ground truth, we employ ProcessBench as a proxy benchmark, given its similar data sources (including MATH) and response generation process (using Qwen and Llama series models). As shown 6 our denoising process achieves a significant reduction in noisy data content.
Table 6: Noise ratio of synthetic data with and without SCAN denoising.
SCAN
Completer
Vanilla MC
Denoising
Llama-3.1 1.mma-3.1-8B-In 1-8-Ins Qwen2.5-Math-75 Ins
56.2% 51.8%
19.1% % (37. 161)
29.4% (22.4%))
B.6 SCAN Inference Speed
Table/]showw the comparison of performance and inference speed of SCAN Models and other large critic models. The inference speed is tested on 4 A A100-40G GPUs. Experimental results demonstrate that discriminative models exhibit significantly superior inference speed compared to critic models. (long CoT) critic models such as DeepSeek-R --Distill-Qwen-777 show even more pronounced efficiency bottlenecks.
Particularly,
Table Performance and inference speed comparison between SCAN models and critic models.
Best-of-N (Avg, Acc.)
ProcessBench (Avg. Fl)
Model
Infer Speed
Owen2.5-Math-77B8
17.3
1.5 samples / s
Qwen2.55-78-Insrruct
36.8
10.8 samples /
DeepScek-RR I-DDistill-wwen--
53.4
0.5 samples s
M-thahBB----AN
69.1
56.8
44 samples
Maen2..--ath-7BBSCCC
70.1
59.1
44 samples /
16

---

B.7 Incorporate SCAN with Knowledge Distillation Method
Zheng al. demonstrated the effectiveness of consensus filtering, where Monte Carlo estimation can be combined with knowledge distillation to further improve data quality. Since any improvement in MC estimation directly benefits downstream pipelines, we integrated our method (SCAN) into consensus filtering framework to evaluate its effectiveness.
Table 8: Results of SCAN models with knowledge distillation (KD) from QwQ-32B.
Best-of-N (Avg. Acc)
ProcessBench
Model
Method
# Samples
(Avg. F1)
Qwen2.5-Math-PPe PRMahh-PRM-BB
MC KD
1500K
70.1
73.5
Baseline-7B (wlo denoising) +
MC KD MC & KD
100K
52.5
SCAN-Base-7B (w/ denoising) + KD
100K
70.3 (+1.3)
60.8 (+8.3)
using SCAN's denoised MC data with KD (SCAN KD) significantly outper- forms using standard MC data with KD (Baseline + KD), especially the fine ain-ggrined ProcessBench. Therefore, the improved version of MC can be readily used as a plug-in in or substitute in any framework that involves MC estimation, offering further performance gains.
As shown in Table 8
B.8 SCAN Effectiveness in Broader Domains
We further examine the generalization ability of SCAN beyond mathematical reasoning. Specifically, we extend our Math PRM to a general-domain task, GPQA-Diamond [25]. with results reported in Table Our PRM consistently outperforms the majority-vote baseline, indicating that the reasoning capability it acquires is transferable beyond mathematics. This suggests that developing robust, domain-specialized PRMs represents an important direction for future research.
Table Best-of-N results of SCAN-Pro in GPQA-Diamone
Method
=
N =
N
4
N 8
Maj @ N 33.8
33.8
38.9
37.3
PRM @ N
33.8
36.4
40.4
39.4
B.9 Analysis of Process Error Types
To further understand the noise in MC-annotated data, we conducted qualitative analysis by categorizing errors 60 samples from ProcessBench, with results illustrated Table 10
Table 10: Error Type Analysis Results and PRM Accuracy.
Error Type
Description
PRM Accuracy
Calculation Error
Mistakes in arithmetic or computation.
15 20
Logical Error
Inconsistencies unjustified steps.
9/ 20
Conception Error
Misunderstanding of concepts or formulas.
10/ 20
SCAN-PRM highly effective at detecting calculation errors but less sensitive abstract logical or conceptual mistakes, indicating that SCAN primarily captures procedural correctness. Achieving deeper semantie accuracy, however, remains challenging and will require further investigation in future work.
C Discussion
C.1 Performance discrepancy of models in test-time scaling and ProcessBench
We observe notable discrepancy between model performance in Best-of-N evaluation and Process- Bench. For example, models such Owen2.5-MMathPP perform comparably to SCAN models
17

---

under Best-of-N,, yet show larger performance gains on ProcessBench. This contrast reflects the fundamental difference between Monte Carlo (MC) and knowledge distillation (KD) annotation.
MC annotation is coarse-grained: it judges correctness solely based on the final outcome of solution. This provides a strong global signal of solution quality, which explains why MC-trained models excel in Best-of-N evaluation, where the objective is to identify the best overall response. KD annotation is fine-grained: a powerful critic model can analyze reasoning step by step, pinpointing the exact location of errors. Such supervision is crucial for ProcessBench, which explicitly evaluates step-level error detection.
These differences in supervision quality directly influence model performance. In Best-of-N evalua- tion (Table D). MC-trained models perform well, as their coarse error modeling still captures useful global signals. However, in ProcessBench (Table which requires precise step-level correctness, models trained only with MC supervision underperTorm due to their weaker error localization ability, Therefore, while our cost-efficient SCAN method achieves strong results in selecting the best overall solution, the more expensive fine-graing umneggrinedd supervision used by Qwen-PRM naturally yields better step-level error detection. Notably, our experiments Table show that integrating SCAN with KD significantly reduces this performance gap.
C.2 Effectiveness of SCAN Annotator Model
The effectiveness of SCAN is closely tied to the capability of the base annotator, and central insight of our work that this capability must be well-matched to the difficulty of the problem. To quantify this alignment, we propose self-confidence metric.
In general, the quality of process annotation depends on two factors: (I) the capability of the base model and (2) the difficulty of the problem. Effective supervision arises when these two are properly aligned For instance, if highly capable model is applied to an extremely simple problem, it may always recover the correct final answer, even when given erroneous prefix steps, as it can effortlessly detect and correct mistakes. Conversely, the model is too weak for challenging problem (e.g.. Olympiad- it is unlikely to reach the correct answer under any prefix, producing noisy or uninformative annotations.
The key insight, therefore, is that annotation quality improves when the model's capability matches the task difficulty, and this match can be estimated using SCAN'S self-confidence metric.
D Limitations and Future Work
Despite the promising results, our work still faces several limitations.
Linitations of Monte Carlo Estimation During our experiments, we observed type of noise that cannot be effectively handled by Monte Carlo (MC) estimation false positives, where a response is predicted to be entirely correct but actually contains errors. This issue arises from a strong assumption underlying MC estimation: if the final answer is correct, all intermediate steps are assumed to be correct well. Consequently, step-level errors embedded in seemingly correct responses remain undetected. This limitation makes purely MC-based annotation strategies insufficient for ensuring process-level fidelity.
Limitations of Process Reward Models (PRMs) in Al alignment Although PRMs show encour- aging improvements in test-time scaling and eттor localization, their effectiveness in AI alignment and safety remains limited. In particular, PRMs are still vulnerable to reward hacking, where model exploits imperfections in the reward signal to achieve high scores without genuine reasoning. evaluations, where models may
This issue becomes especially pronounced in inadvertently optimize for misaligned objectives.
Promising Direction: Generative Process Reward Models (GenPRMs). To address both the annotation and alignment challenges, we view generative process-level reward models (GenPRMs) as a promising path forward. Unlike discriminative PRMs, GenPRMs are tasked with producing explicit rationales that identify both the location and nature of reasoning errors. This grounds the
18

---

reward signal in the actual reasoning process, thereby reducing the likelihood of reward hacking and improving interpretability. In data synthesis, large critic models can further enhance GenPRMs by pinpointing step-level mistakes, which can be integrated with MC sampling to yield higher- quality supervision. We believe GenPRMs represent key direction for future research, offering stronger process-consistent reward signals and a safer, more interpretable framework for supervising high-capability reasoning models.
Human Verification
We further analyze the capability of our trained PRMs through human evaluation. Specifically, manually inspect whether the PRM accurately identifies correct and incorrect samples, as well as its ability locate errors within responses precisely. Our evaluation focuses two challenging settings in the best-of-n scenario:
High-confidence samples, where almost every response is correct-can the PRM identify the few erroneous responses hidden among predominantly correct ones?
Low-confidence samples, where only few responses are correct- can the PRM successfully identify those correct responses?
We list some representative cases below, with human verification and analysis.
19

---

Best-Of-N Evaluation Example
Task:
Best-of-8 Evaluation
Purpose: Whether PRM can select incorrect responses high-conf
questions.
Dataset Name: GSM8X
Response Generator: Qwen2. Me--Math-BB-Inteee
Process Reward Model: Qwen2. Beath-7B-ScAN Problem:
file. Normally can download GB/minute, but 40% of the way through the download, Windows forces restart to install updates, which takes 20 minutes. Then Carla has to restart the download from the beginning. How load does it take to download the file?
Carla is dowllodding
Response (incorrect, others correct):
To determine the total time it takes Carla to dowmload the 200 GB file, we need to break down the process into segments and calculate the time for each segment.
Ground Truth: Correct, Predicted Score: 96.1,
1. Calculate the time to download 40% of the file:** The file size is 200 GB.
40% of 200
0.40 200 80 GB.
Carla downloads at a rate of 2 GB per minute.
Time to download GB
40 minutes.
Ground Truth: Correct, Predicted Score: 91.8,
2. **Add the time taken for the restart:**
The restart takes minutes.
Total time far is 40 20 = 60 minutes.
Ground Truth: Correct, Predicted Score: 94.1,
3. **Calculate the time to download the remaining 160 GB:**
The remaining file size is 200 80 120 GB.
Carla downloads at a rate of 2 GB per minute.
Time to download 120 GB is 120 60 minutes.
2
Ground Truth: Incorrect, Predicted Score: 3.4, Explanation: Since the reboot requires downloading from scratch, she needs to re-download the entire 200 GB instead of the remaining 120 GB.
**Add the time taken for the second restart:**
The restart takes 20 minutes.
Total time far is 60 + 60 + 20 140 minutes.
Therefore, the total time takes Carla to download the file 140 minutes.
Analysis: PRM select incorrect responses with accurate error location from numerous correct ones, demonstrating strong discriminative capability.
20

---

Best-Off -f Evaluation Example 2 (part 1)
Task:
Best-of-8 Evaluation
Purpose:
Whether PRM can select correct responses in lov-conf
questions.
Dataset Name: Olympiad Bench
Response Generator: Bren22---att-77-Innteee
Process Reward Model:
Problem:
Find all triples b, of positive integers such that a + b3 + c² (abe)?.
Response (correct, 7 others incorrect):
To find all triples ba, of positive integers such that a + + (abc)?, we start by analyzing the equation and considering possible values for a, b, and c.
Ground Truth: Correct, Predicted Score: 95.3,
First, let's check if there are any small solutions. We start with b=b=c:
a + a + a - = 3a a -a =
-a
3) 0.
Since is positive integer, a² #0 have:
a =3
3/3
Since V3 not an integer, a= = does not provide a solution. Ground Truth: Correct, Predicted Score: 85.2,
Next, let's try some small distinct values for a, b, and Suppose 1=1: + d' = (1- bllbb.c) b.2 = 1 63 + =
Rearranging terms, we get:
b 0 (bc)2 1
We need find positive integers and e that satisfy this equation.
Let's try b= 2:
23 -
= 8+ = 4o - δ 4ca +9 0.
We test possible integer values for c:
If c 1, then 1a ----1----1-- 9 6
2, then --44-2+--- 9 9
If c= 3, then -4.33 -4-43349-27 So, e
-9-- 27-2771 39=
is a solution. Therefore, (,, By symmetry, (1,3,2), (2, 1,3), (2,3,1), (3,1,2), and 13,21 are also solutions.
(1,2 is a solution.
Ground Truth: Correct, Predicted Score: 67.2, /
(continue the next page)
21

---

Best-Of-N Evaluation Example 2 (part 2)
Now, let's check if there are any other solutions. Suppose 2:
2 + b + te = (2bc)² = 22
Rearranging terms, we get:
8 + 462 8.
We need find positive integers and that satisfy this equation. Let's try b=1:
1 - = 1+c 4e
e 4e +9=0
We already know that c 4c² + has no positive integer solutions. Let's try b= 2:
2 e 2 2² 8 = 8 e² 16c - 8 = d² 166 + 16 = 0.
We test possible integer values for C:
If -= 1, then 1 - 16 1a + 16 1- 16 16 = 1
c = 2, then 2 16 2 = 64
-40 +
If c= 4, then 43 - 16 16 16 64 10--54 256 + 16 1166 0.
Since c² 16c + 16 0 has no positive integer solutions for c 2 2, there are other solutions when 2.
Ground Truth: Correct, Predicted Score: 61.7,
By symmetry, there are no other solutions when Therefore, the only solutions are:
3 or higher.
(1,2,3), (1, 1,3), (2, 1), (3, 23
Ground Truth: Correct, Predicted Score: 81.3,
Analysis: PRM can select incorrect responses from numerous incorrect ones, but it is less confident in some of the correct steps, as demonstrated by the relatively low step scores of intermediate steps.
22

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix D
23

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We list the key observation (or assumptions) in section 2 to support our design for our main method (section 3)
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide code implementation of preliminary study and main exper- iments in https:: /aap:/://///omouuu./mm 44p.: sciencees science/r/SCAN PRMM Detailed configations of data synthesis, training, and evaluation can be seen in Appendix|A] and section
Guidelines:
24

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
See supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
public//uiide//ooeSuubaua oow.eedSuuumsssiooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https c/ipps.cc/publi///uue
Codeeuumiiii wettsbbmmssiiioi ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
25

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: To reduce the error bars, we use the same responses results of all the Process Reward Models, as we detailed in Table II
the Best-of-N
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information of computer resources in Appendix A.I
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
26

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper fully conforms to the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The data and models used in our study are sourced from public repositories, and our work does not introduce additional risks or modifications that might require new safeguards.
27

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the original paper and detailed information of assets in section
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
See supplementary materials the review system.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
28

---

Answer:
[NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not involve LLMs as any important, original, or non-standard compo-
nents.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanndaddrrd components.
Please refer to our LLM policy (https://nen) / for what should or should not be described.
29


Reviews:
[{"review_id": "teaqrDdciD", "reviewer": "Reviewer_zmmN", "summary": "The research paper introduces a novel framework, SCAN, to address the challenges of training high-quality Process Reward Models (PRMs) from synthetically generated data. The authors identify that standard Monte Carlo (MC) estimation methods produce noisy labels, which causes models to overfit and limits their performance. The paper first presents a preliminary study that analyzes and categorizes the noise in MC-generated data into two types: under-estimation (incorrectly flagging correct steps as errors) and over-estimation (failing to detect the true first error). To quantify this, they propose a \"self-confidence\" metric, which measures the annotation model's own success rate on a given problem.", "questions": "1. As mentioned in weaknesses, how well does the SCAN framework transfer to tasks outside of mathematical reasoning (e.g., coding, multi-modal reasoning)?\n2. How does the choice of base annotation model (size/capability) affect the ultimate PRM performance? Is there a “minimum” effective annotator?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. It proposes an efficient and practical solution (SCAN) that improves both data quality and training robustness via self-denoising and confidence-based approaches. It tackles a real, practical challenge—reducing the cost and scaling limitations of human-annotated PRM data.\n2. This paper presents strong empirical results and made code and data available, aiding reproducibility. It achieves comparable results with only ~6% of the inference cost compared to traditional MC methods. It outperforms or matches models trained on much larger or human-annotated datasets, on challenging benchmarks like ProcessBench and Best-of-N evaluation. In addition, it shows continued improvement with increased data, without apparent overfitting.\n\n## Weaknesses\n1. The self-denoising approach’s effectiveness is inherently limited by the capability of the base model used for annotation. Additional empirical exploration of this limit could strengthen the claims.\n2. While SCAN-trained PRMs surpass baselines, a deeper discussion or ablation on the qualitative differences (e.g., error types, robustness to distribution shift) between human- and synthetically-annotated data could be valuable.\n3. The method is demonstrated mainly for mathematical reasoning datasets. It is unclear how well SCAN generalizes to other domains or types of reasoning tasks.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive and constructive review, highlighting SCAN's practicality, efficiency, and strong empirical results.\n\n**Concern 1:**\nThe self-denoising approach's effectiveness is inherently limited by the capability of the base model used for annotation. Additional empirical exploration of this limit could strengthen the claims.\n\n**Responses to Concern 1:**\nThis is an excellent point. The effectiveness of SCAN is indeed tied to the base annotator's capability, and a key insight of our work is that this capability should be well-matched to the problem's difficulty. Our proposed self-confidence metric is designed precisely to measure this match.\n\nIn general, the quality of process annotation depends on two key factors: (1) the capability of the base model, and (2) the difficulty of the problem. Effective annotation tends to occur when these two factors are well-matched — a core idea behind our SCAN framework. For example, if a very strong model is used on an extremely simple problem, it may always recover the correct final answer, even when provided with erroneous prefix steps, because it can easily detect and correct the errors. On the other hand, if the model is not powerful enough to solve a challenging problem (e.g., one at Olympiad level), it is unlikely to produce the correct answer regardless of the input, resulting in noisy or uninformative supervision.\n\nSo the key insight is that the model's capability should ideally match the difficulty of the problem, and this alignment can be estimated via self-confidence metirc proposed in SCAN. Given a specific problem, we can try several base models of varying capabilities and select the one whose self-confidence score falls within an appropriate range (e.g., [0.75, 0.9]).\n\nIn SCAN, we retain samples where self-confidence is 1 to maximize data utilization. To mitigate the risk of over-estimation noise (where the model self-corrects and misses the true error), we introduce a tolerance distance for soft labels. This provides a robust learning signal even from imperfect annotations.\n\nIn summary, SCAN provides a principled framework to manage the annotator's capability by: **(1) using self-confidence to select an appropriately-sized annotator for the task**, and **(2) using a tolerance distance to robustly learn from its outputs**.\n\n---\n\n**Concern 2:**\nWhile SCAN-trained PRMs surpass baselines, a deeper discussion or ablation on the qualitative differences (e.g., error types, robustness to distribution shift) between human- and synthetically-annotated data could be valuable.\n\n**Response to Concern 2:**\nThank you for this valuable suggestion. To better understand our PRM's capabilities, we conducted a qualitative analysis by categorizing errors in 30 samples from ProcessBench.\n\n| Error Type | Description | PRM Accuracy |\n| ----------- | ----------- | ----------- |\n| Calculation Error | Mistakes in arithmetic or computation. | 9 / 10 |\n| Logical Error | Inconsistencies or unjustified steps. | 5 / 10 |\n| Conception Error | Misunderstanding of concepts or formulas. | 6 / 10 |\n\nOur PRM is highly effective at detecting calculation errors but is less sensitive to more abstract logical and conceptual mistakes. This suggests that while SCAN effectively captures procedural correctness, future work could focus on improving the detection of deeper semantic errors. We will expand this analysis and add case studies to the final paper.\n\n---\n\n**Concern 3:** The method is demonstrated mainly for mathematical reasoning datasets. It is unclear how well SCAN generalizes to other domains or types of reasoning tasks.\n\n**Question 1:** As mentioned in weaknesses, how well does the SCAN framework transfer to tasks outside of mathematical reasoning (e.g., coding, multi-modal reasoning)?\n\n**Response to Concern 3 & Question 1:**\nWe appreciate the reviewer’s interest in evaluating the generalization capability of SCAN beyond the mathematical reasoning domain. To this end, we extend our Math PRM to a general-domain task, GPQA-Diamond, and report the following results:\n\n|  | N=1 | N=2 | N=4 | N=8 | \n| ----------- | ----------- | ----------- | ----------- | ----------- |\n| Majority Vote @ N | 33.8 | 33.8 | 38.9 | 37.3\n| PRM @ N | 33.8 | **36.4** | **40.4** | **39.4** |\n\nOur PRM consistently outperforms the majority vote baseline, demonstrating that the reasoning capabilities it learned are transferable beyond mathematics.\n\nThat said, building high-quality, domain-specific PRMs for tasks like coding or multi-modal reasoning remains challenging, primarily due to:\n\n- **Data Scarcity:** High-quality process-level data is rare in these domains. As noted in a concurrent work [1], math-trained PRMs can outperform PRMs trained on synthetic code data, likely due to the quality gap (e.g., PRM800K vs. MC-generated code data).\n- **Evaluation Reliability:** Annotating and evaluating processes in other domains can be unreliable. For example, multi-modal models often fail to follow specific output formatting instructions, complicating automated evaluation.\n\nWhile we believe SCAN's data synthesis strategy can be extended to these domains, we see the development of robust, domain-specific PRMs as a significant future research direction. We will add this discussion to the paper.\n\n[1] From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling.\n\n---\n\n**Question 2:**\nHow does the choice of base annotation model (size/capability) affect the ultimate PRM performance? Is there a \"minimum\" effective annotator?\n\n**Response to Question 2:**\nThank you for the thoughtful question. As discussed in our Response to Concern 1, we explain how to select an appropriate annotator to ensure the effectiveness of the self-denoising approach.\nThe effectiveness of a base annotation model should be considered in conjunction with the difficulty of the target questions. For example, in relatively easier benchmarks such as GSM8K or Math, a 1.5B-sized model may already serve as a sufficient annotator, as it can achieve reasonably high accuracy. In contrast, for more challenging problems such as AIME24 or AIME25, stronger annotators (e.g., QwQ-32B) may be required to produce reliable supervision."}, {"review_id": "C37iUMFLbZ", "reviewer": "Reviewer_5Gg8", "summary": "This paper presents SCAN, a framework for robust process reward learning that analyzes and mitigates noise in Monte Carlo (MC) estimation. The core contributions are an efficient data synthesis strategy and a noise-tolerant training objective.", "questions": "The baseline categorization in Table 2 is somewhat unconventional, as it appears to group models with different training paradigms (e.g., some baselines are not strictly MC-based), which may complicate direct comparisons.\n\nCould the authors explain the performance discrepancy of in Table1 and Table2? For example, Qwen-PRM performs on par with SCAN models in the Best-of-N evaluation (Table 1) yet wins by a large margin on ProcessBench (Table 2). \n\n\nSuggestions\nThe discussion on related work could be enhanced by including recent studies on alternative PRM evaluation paradigms, such as:\n[1] R-PRM: Reasoning-Driven Process Reward Modeling\n[2] Inference-Time Scaling for Generalist Reward Modeling", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe paper's primary strength lies in its insightful analysis of the noise distribution in MC-generated data, offering a novel and interesting perspective on the problem.\n\n** Weaknesses**\n\nA notable performance gap remains between this MC-only approach and methods that leverage Knowledge Distillation (KD) or human annotations.", "was_revised": true, "final_justification": "The author address my concerns", "rebuttal": "Thank you for your feedback and the opportunity to clarify our contributions and positioning.\n\n**Concern 1:**\nA notable performance gap remains between this MC-only approach and methods that leverage Knowledge Distillation (KD) or human annotations.\n\n**Response to Concern 1:**\nThis is a crucial point. We offer three clarifications:\n\n1. **SCAN is Complementary to KD, Not Competing**: Monte Carlo (MC) estimation and Knowledge Distillation (KD) are not mutually exclusive. In fact, many state-of-the-art methods like Qwen-PRM [1] and GenPRM [2] use MC as part of a consensus filter alongside KD. The goal of our paper is to improve the MC component itself. **A better MC method can directly benefit any pipeline that uses it**. To demonstrate this, we ran a new experiment integrating SCAN into a KD pipeline.\nTo further verify the effectiveness of our MC-based approach, we integrated our method into a consensus filtering pipeline. The results are shown below:\n\n| Model | Method | Samples | BoN (Avg. Acc) | ProcessBench (Avg. F1) | \n| ----------- | ----------- | ----------- | ----------- | ----------- |\n| Qwen-PRM | MC & KD | 1500K | 70.1 | 73.5 |\n| Baseline (w/o denoising) + KD | MC & KD | ~100K | 69.0 | 52.5 |\n| SCAN (base, w/ denoising) + KD | MC & KD | ~100K | **70.3 (+1.3)** | **60.8 (+8.3)** |\n\nAs shown, using SCAN's denoised MC data with KD (**SCAN + KD**) significantly outperforms using standard MC data with KD (**Baseline + KD**), especially on the fine-grained ProcessBench. This confirms that **our improvements to MC are additive and boost KD-based methods**.\n\n2. **Vast Difference in Cost:** It is important to compare performance in light of computational cost. Qwen-PRM uses 1.5M samples annotated by a 72B critic model. Our largest model uses ~200K samples annotated by a 7B model. We estimate the data generation cost for Qwen-PRM is **40-50x higher** than ours. SCAN achieves competitive BoN performance and strong ProcessBench scores at a fraction of the cost.\n3. **Research Focus:** Our primary goal is to investigate the fundamentals of MC annotation—a widely used but under-explored technique. We show that by understanding and mitigating noise *within* the MC process, we can substantially improve its effectiveness without relying on expensive external supervisors.\n\nThe results of SCAN demonstrate that our denoising strategy can significantly improve the upper bound of MC-based methods. In other words, **the improved version of MC can be readily used as a plug-in or substitute in any framework that involves MC estimation, offering further performance gains.**\n\nIn conclusion, SCAN **(1) significantly pushes the performance ceiling of low-cost, MC-only methods** and **(2) provides a better, drop-in replacement for the MC component in more complex KD-based pipelines**.\n\n[1] The Lessons of Developing Process Reward Models in Mathematical Reasoning\n\n[2] GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning\n\n---\n\n**Question 1:**\nThe baseline categorization in Table 2 is somewhat unconventional, as it appears to group models with different training paradigms (e.g., some baselines are not strictly MC-based), which may complicate direct comparisons.\n\n**Response to Question 1:**\nThank you for pointing this out. We sincerely apologize for the oversight — some baselines that are not strictly trained using MC estimation are mistakenly placed in the \"MC-only\" category, which may have caused confusion. \nWe will correct this to ensure a clearer and more accurate comparison across different training paradigms in the revised version.\n\n---\n\n**Question 2:**\nCould the authors explain the performance discrepancy of in Table1 and Table2? For example, Qwen-PRM performs on par with SCAN models in the Best-of-N evaluation (Table 1) yet wins by a large margin on ProcessBench (Table 2).\n\n**Response to Question 2:**\nThis discrepancy highlights the core difference between MC and KD annotation.\n\n- **MC annotation is coarse-grained.** It determines correctness by checking the final outcome of a completed solution. This provides a strong signal for the *overall quality* of a solution trace, which is why MC-trained models perform well on Best-of-N evaluation, where the goal is to pick the best overall response.\n- **KD annotation is fine-grained.** A powerful critic model can analyze the logic of each step, allowing it to pinpoint the *exact location* of an error. This supervision is crucial for ProcessBench, which evaluates step-level error detection accuracy.\n\nSuch differences in supervision quality ultimately affect the performance of the trained models.\n- In the Best-of-N evaluation (Table 1), where models are evaluated based on the overall reward score aggregated from multiple responses, MC-trained models perform reasonably well because their coarse error modeling still captures useful global signals. \n- However, in ProcessBench (Table 2), which requires fine-grained, step-level correctness evaluation, models trained with MC supervision alone fall short due to their weaker error localization capability — hence the observed performance gap.\n\nTherefore, while our cost-efficient SCAN method produces models that are excellent at identifying the best overall solution (BoN), the expensive, fine-grained supervision used by Qwen-PRM naturally leads to better performance on precise error localization (ProcessBench). Our new experiments in W1 show that when SCAN is combined with KD, this gap closes significantly.\n\n---\n\n**Question 3:**\nThe discussion on related work could be enhanced by including recent studies on alternative PRM evaluation paradigms, such as: [1] R-PRM: Reasoning-Driven Process Reward Modeling [2] Inference-Time Scaling for Generalist Reward Modeling\n\n**Response to Question 3:**\nThank you for the excellent suggestion. These are important related works. We will add a detailed discussion and the following results to the appendix. A key takeaway is that while generative PRMs are a promising direction, our discriminative PRM is far more inference-efficient.\n\nGenerative PRM (4 x A100-40G):\n\n| Model      | ProcessBench (Avg. F1) | Infer Speed |\n| ----------- | ----------- | ----------- |\n| Qwen2.5-Math-7B-Instruct | 17.3 | 1.5 samples / s |\n| Qwen2.5-7B-Instruct | 36.8 | 10.8 samples / s |\n| DeepSeek-R1-Distill-Qwen-7B | 53.4 | 0.5 samples / s |\n| GenPRM-7B | 75.2 | 1.2 samples / s |\n| R-PRM-7B | 70.4 | 0.8 samples / s |\n| **Qwen2.5-Math-7B-SCAN-Base** | **56.8** | **44 samples / s** |\n| **Qwen2.5-Math-7B-SCAN-Pro** | **59.1** | **44 samples / s** |\n\nThis highlights a critical trade-off: SCAN offers highly practical and scalable inference, while current generative PRMs trade speed for higher accuracy.\n\n[1] R-PRM: Reasoning-Driven Process Reward Modeling\n\n[2] GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning"}, {"review_id": "9w7GOPOWj6", "reviewer": "Reviewer_MDHU", "summary": "This paper introduces a method for training process reward models (PRMs) without requiring step-by-step human annotation data that builds upon prior self-supervised methods for training PRMs known as \"Monte Carlo annotation / estimation\", where rollouts from a completion model are used to estimate the correctness of a particular step, based on how often the generated completions result in the correct final answer.\n\nIn particular, the paper contributes a study as to when and why Monte Carlo annotations can become inaccurate or noisy, due to weaknesses in the completion model: When the completion model is weak at a certain question, it is likely to fail to reach the correct answer even when providing with a small number of correct steps, resulting in under-estimation of the first step index where reasoning fails. In contrast, when the completion model is able to self-correct after a true reasoning error, it ends up over-estimating the first step index where reasoning fails. Furthermore, by using another set of Monte-Carlo rollouts to estimate the completion model's confidence in the correct answer, the authors find that under-estimation usually occurs when the model has low self-confidence on a question, whereas correct estimation (and some over-estimation) generally occurs when the model has high self-confidence in a questions.\n\nUsing these findings and others as a guide, the authors develop a \"self-denoising\" scheme for PRM training, which (i) filters out low-confidence questions from the training data for the PRM; (ii) generates step-by-step annotations via Monte Carlo rollouts only for questions where the completion model gets the wrong answer, improving efficiency; (iii) corrects for over-estimation of the first failure index by adding soft labels; (iv) corrects for models that are too strong at self-correcting via confidence-reweighting. In experiments, they show that this method, SCAN, outcompetes most other PRMs in the 7-8B range on math reasoning and ProcessBench, including those trained on human-annotated data. They also find that SCAN on a 7B model is competitive with some (but not all) 70B models.", "questions": "- Could you address my concern about the justification of confidence-wise reweighting?\n- Could you address the notational issues that affect the clarity of the paper?\n- How is 100% sample utilization achieved?\n- Could you more carefully explain which negative samples are collected for Monte Carlo step-wise annotation?\n\nThere are also a number of typos, e.g. \"Noise-torlerant\" should be \"noise-tolerant\", and \"noisy positive\" seems like a non-standard term compared to \"false positive\".", "limitations": "I think the authors should include a Broader Impacts statement about the implications of this PRM training strategy on AI alignment and safety. PRMs were initially introduced partly to overcome the limitations of outcome-based RMs, which can lead to safety issues due to over-optimization and reward-hacking, as we've already seen in recent studies of reasoning models (Denison et al, 2024; Bondarenko et al 2025).\nBecause PRMs are supposed to reward only valid chains of thought, the hope is that they are more resistant to this reward hacking behavior. However, Monte Carlo annotation loses this a lot of this benefit of PRMs, since the process-level labels are effectively generated entirely from outcome-based self-supervision, and also creates noise/errors in the reward specification that a policy can learn to exploit. This could also incentivize unsafe specification-gaming / reward hacking in reasoning models (e.g. rewriting test cases, so that the test is eventually passed). I think some discussion of these safety implications is warranted, since this paper uses the Monte Carlo annotation approach.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I don't follow the literature on LLM reward modeling closely, so I'm not well positioned to evaluate originality and significance. Nonetheless, I found the preliminary study quite insightful, and helpful for understanding the failure modes of outcome-based self-supervision for generating process-level labels. Given that process-based supervision has sometimes been pitched as a \"safer\" alternative to outcome-based supervision (which can incentivize reward hacking, etc.), it's valuable to understand out how process-based supervision can also fail or lead to a bad / misaligned training signal, especially when using synthetically generated annotations.\n\nAs for the SCAN method itself, empirically it appears to perform as well as training on human process-level annotations, and I found the experiments and ablations to be thorough.\n\nI do, however, think the justification of the confidence-wise reweighting requires some work. The authors say that, \" $c_i*$ represents the true correctness probability, while $c_i$ is the estimated probability derived from the completer model $π$ \", implying that $c_i*$ is some kind of model-independent property. However, it seems to me that the \"probability of correctness\" is never model-independent --- it's always relative to some generative process of how the remainder of the chain of thought is completed, whether that generative process is parameterized by a neural network, or by how a population of human reasoners would tend to complete a chain-of-thought. So what exactly is this \"true correctness probability\"? I don't think this is a coherent concept unless one defines some kind of \"ground-truth\" or \"reference\" generative process $\\pi^*$. Furthermore, if correctness is defined in logical terms (e.g. whether a reasoning step is mathematically valid), then a particular step is either correct on incorrect --- so the concept of \"probability of correctness\" doesn't really make sense in that context either. Instead, in the context of Monte Carlo annotation, $c_i$ is actually *the probability that some model (or human) eventually reaches a correct answer* --- which is at best a *proxy* of whether that step is actually correct.\n\nOne way to fix the issue above is to make clear that there is some expert / reference policy that we care about, $\\pi*$, (e.g. the distribution of how an expert human would generate reasoning traces). If that expert policy were used to generate Monte Carlo annotations of the dataset, we could estimate $c_i*$ as the probability that the expert would eventually generate a correct answer. We would like to use $c_i*$ as a soft label for training that is independent of the completion model $\\pi$ (even though it's not independent of $\\pi*$), but we only have $c_i$ for some other completion model $\\pi$ -- which might overestimate $c_i*$ when the completion model is stronger than the expert, or underestimate $c_i*$ when the completion model is weaker than the expert. So to estimate $c_i*$, you divide by the completion model's self-confidence. IMO that would be a stronger justification for the confidence-wise reweighting.\n\nThere are a number of other clarity issues that could be fixed. For example, it was confusing that the symbol $y_i$ is used to refer the (hard) label for whether step $i$ is correct, but then later $y_{pred}$ and $y_{true}$ are used to denote the index/step of the first error. I would suggest change it to $t_e^{pred}$ and $t_e^{true}$, to denote that this an index-valued variable. and to make it consistent with the $t_e$ notation used in Equation 5.\n\nI also didn't understand the 100% sample utilization point -- is it the case that all samples that are either negative or not high in self-confidence undergo Monte Carlo annotation? If so, that should be made clear. The line \"we collect the negative samples with $y_{pred} = inf$ from these responses for further step-wise annotation\" was also confusing to me. Are all negative samples collected, or only those with  $y_{pred} = inf$? And are these so-called \"noisy positive samples\", since  $y_{pred} = inf$ but $y_{true} < inf$? If so, why are they suddenly called \"negative samples\"? \n\nOverall though, I don't see reasons to reject this paper -- I just think the clarity and explanation / justification could be improved at various points.", "was_revised": true, "final_justification": "The authors have addressed all of my concerns and questions, and I continue to recommend acceptance.", "rebuttal": "Thank you so much for your exceptionally thorough review and insightful, constructive feedback. Your suggestions have been invaluable for improving the paper's clarity and rigor.\n\n**Concern 1 & Question 1:**\nabout the justification of confidence-wise reweighting\n\n**Response to Concern 1:**\nWe sincerely apologize for the mistakes in the definition of $c*$ and the lack of clarity in our original explanation.\nIn our formulation, $c*$ denotes the predicted step correctness score provided by an ideal or golden annotator. Based on our previous findings, we observed that the noise in step-level annotations primarily stems from the mismatch in model annotation capabilities — leading to systematic under-estimation or over-estimation errors.\n\nWe assume the existence of a golden MC annotator $\\pi_{\\text{gold}}$ that can perfectly model the correctness of each intermediate step via MC estimation, without introducing any systematic bias (i.e., under-estimation or over-estimation). The correct formulation of $c^*$ should be:\n\n$$c_t^* = P(y_{i} = \\text{correct}| \\pi_{\\text{gold}}, q, x_{\\leq t})$$\n\nHere we retain soft labels within a predefined tolerance distance, assuming these steps can be partially correct.\n\nHowever, we can only estimate this probability using our imperfect completer model $\\pi_{\\theta}$, which yields an estimated probability $\\hat{c_t} = P(y_{i} = \\text{correct} | \\pi_{\\theta}, q, x_{\\leq t})$. This estimation is biased by $\\pi_{\\theta}$'s own capabilities (i.e., under-estimation or over-estimation). To address this, we introduce confidence-wise reweighting as a proxy to (1) reduce the bias introduced by imperfect base models, and to (2) normalize the annotation quality across different policy models.\n\nThis mechanism allows us to mitigate annotation noise while preserving the learning signal from partially correct responses, ultimately leading to more robust training of process reward models.\n\n---\n\n**Concern 2 & Question 2:**\nnotational issues that affect the clarity of the paper?\n\n**Response to Concern 2:**\nWe will change $y_{pred}, y_{true}$ to $t_e^{pred}, t_e^{true}$ in the relevant equation and figure.\nThank you for pointing this out — we believe this change will make the presentation clearer and more precise.\n\n---\n\n**Concern 3 & Question 3:**\nHow is 100% sample utilization achieved?\n\n**Response to Concern 3:**\nYes, we only collect negative samples (i.e., those where the predicted answer does not match the ground truth), indicating that process-level errors are definitely present.\n\nWe achieve this by being more efficient. Unlike prior work (e.g., Math-Shepherd) that performs expensive MC annotation on all generated samples (even correct ones), we only perform MC annotation on samples that result in a wrong final answer. We assume that if a model gets an answer right, its reasoning is likely sound by its own standards, so re-annotating with the same model is redundant. Thus, every sample we choose to annotate via MC is used for training, achieving 100% utilization of the annotated data and saving significant computation.\n\n[1] Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations\n\n\n---\n\n**Concern 4 & Question 4:**\nExplain which negative samples are collected for Monte Carlo step-wise annotation?\n\n**Response to Concern 4:**\nWe apologize for the confusion. The statement \"the negative samples with $y_{pred} = \\text{inf}$\" is incorrect — it should be $y_{pred} \\neq \\text{inf}$.\n\nThe negative samples refer to all responses where $y_{pred} \\neq \\text{inf}$, indicating that the annotater model detect some error in the response. Our intention is to clearly convey the definition of negative samples (i.e., $y_{pred} \\neq \\text{inf}$), but we mistakenly used the wrong symbol.\nWe sincerely apologize for the misstatement and any confusion it have caused.\n\n---\n\nThank you for your suggestion, which made us aware of some issues in our phrasing and the lack of sufficient clarification.\nWe have carefully reviewed the entire paper and revised several potentially misleading expressions to ensure clearer and accurate understanding of our method.\n\n---\n\n**Response to Concern about Limitations**\n\nThis is an excellent and important point. We agree completely and will add a dedicated section discussing these implications. You rightly note that because MC annotation derives process labels from outcome supervision, it is still vulnerable to reward hacking, where a model finds flawed reasoning paths that happen to produce correct final answers. This could incentivize specification-gaming and undermines the safety promises of process-based supervision.\n\n**Limitations of PRMs in AI alignment:**\n\nWhile PRMs have shown promising improvements in test-time scaling and error localization, their effectiveness in AI alignment and safety remains limited, primary due to the reward hacking [1].\n\nReward hacking occurs when a model exploits imperfections or ambiguities in the reward signal to achieve high scores without performing the intended reasoning or task. This issue is not unique to ORMs; it also affects PRMs, especially when the model generalizes well but inadvertently optimizes for a misaligned objective. This problem becomes particularly pronounced when evaluated on out-of-distribution data.\n\n**Limitations of MC Method in Process Annotation:**\n\nAs the reviewer rightly pointed out, the current process-level annotations are ultimately derived from outcome correctness. Consequently, it is difficult to identify step-level errors in responses that yield correct final answers but contain faulty intermediate reasoning. This limitation makes purely Monte Carlo-based annotation strategies insufficient for ensuring alignment fidelity.\n\n**A Promising Direction: Generative Process Reward Models (GenPRMs)**\n\nTo address both the alignment and annotation challenges, we believe generative process-level reward models (GenPRMs) offer a promising path forward:\n- For AI Alignment: Unlike discriminative PRMs, GenPRMs are tasked with producing explicit rationales that identify both the location and nature of reasoning errors. This enhances interpretability and grounds the reward signal in the actual reasoning process, making it harder for models to exploit superficial correlations—thereby reducing reward hacking and improving safety.\n- In Process Data Synthesis: GenPRMs provide more accurate error localization. In data generation settings, large critic models can be used to pinpoint precise step-level mistakes, which can then be integrated with Monte Carlo sampling to synthesize higher-quality process supervision. We discuss this further in Appendix D.\n\nIn fact, we consider GenRMs [2] (or GenPRMs) a key direction for future research. We believe they offer strong potential to mitigate reward hacking by enforcing process-consistent reward signals and offering a more interpretable framework for reasoning supervision—both of which are crucial for the safe deployment of high-capability reasoning models.\n\n[1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning\n\n[2] Inference-time scaling for generalist reward modeling"}]

Author rebuttal:

Meta-review: This paper presents SCAN, a framework for robust process reward learning that analyzes and mitigates noise in Monte Carlo (MC) estimation. The core contributions are an efficient data synthesis strategy and a noise-tolerant training objective.

All the reviewers thought of the paper highly and felt that it should be accepted. Please incorporate the discussion into the final version of the paper.
Decision: Accept (poster)