Title: SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Reasoning

Process Supervision

Robust Learning
Abstract: Process reward models (PRMs) offer fine-grained, step-level evaluations that facilitate deeper reasoning processes in large language models (LLMs), proving effective in complex tasks like mathematical reasoning. 
However, developing PRMs is challenging due to the high cost and limited scalability of human-annotated data.
Synthetic data from Monte Carlo (MC) estimation is a promising alternative but suffers from a high noise ratio, which can cause overfitting and hinder large-scale training.
In this work, we conduct a preliminary study on the noise distribution in synthetic data from MC estimation, identifying that annotation models tend to both underestimate and overestimate step correctness due to limitations in their annotation capabilities. 
Building on these insights, we propose {\bf S}elf-Denoising Monte {\bf C}arlo {\bf An}notation (\textsc{Scan}), an efficient data synthesis and noise-tolerant learning framework.
Our key findings indicate that:
(1) Even lightweight models (e.g., 1.5B parameters) can produce high-quality annotations through self-denoising strategy, enabling PRMs to achieve superior performance with only 6\% the inference cost required by vanilla MC estimation.
(2) With our robust learning strategy, PRMs can effectively learn from this weak supervision, achieving a 39.2 F1 score improvement (from 19.9 to 59.1) in ProcessBench.
Despite using only a compact synthetic dataset, our models surpass strong baselines, including those trained on large-scale human-annotated datasets such as PRM800K.
Furthermore, performance continues to improve as we scale up the synthetic data, highlighting the potential of \textsc{Scan} for scalable, cost-efficient, and robust PRM training.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for training process reward models (PRMs) without requiring step-by-step human annotation data that builds upon prior self-supervised methods for training PRMs known as \"Monte Carlo annotation / estimation\", where rollouts from a completion model are used to estimate the correctness of a particular step, based on how often the generated completions result in the correct final answer.\n\nIn particular, the paper contributes a study as to when and why Monte Carlo annotations can become inaccurate or noisy, due to weaknesses in the completion model: When the completion model is weak at a certain question, it is likely to fail to reach the correct answer even when providing with a small number of correct steps, resulting in under-estimation of the first step index where reasoning fails. In contrast, when the completion model is able to self-correct after a true reasoning error, it ends up over-estimating the first step index where reasoning fails. Furthermore, by using another set of Monte-Carlo rollouts to estimate the completion model's confidence in the correct answer, the authors find that under-estimation usually occurs when the model has low self-confidence on a question, whereas correct estimation (and some over-estimation) generally occurs when the model has high self-confidence in a questions.\n\nUsing these findings and others as a guide, the authors develop a \"self-denoising\" scheme for PRM training, which (i) filters out low-confidence questions from the training data for the PRM; (ii) generates step-by-step annotations via Monte Carlo rollouts only for questions where the completion model gets the wrong answer, improving efficiency; (iii) corrects for over-estimation of the first failure index by adding soft labels; (iv) corrects for models that are too strong at self-correcting via confidence-reweighting. In experiments, they show that this method, SCAN, outcompetes most other PRMs in the 7-8B range on math reasoning and ProcessBench, including those trained on human-annotated data. They also find that SCAN on a 7B model is competitive with some (but not all) 70B models.\n\nSTRENGTHS AND WEAKNESSES:\nI don't follow the literature on LLM reward modeling closely, so I'm not well positioned to evaluate originality and significance. Nonetheless, I found the preliminary study quite insightful, and helpful for understanding the failure modes of outcome-based self-supervision for generating process-level labels. Given that process-based supervision has sometimes been pitched as a \"safer\" alternative to outcome-based supervision (which can incentivize reward hacking, etc.), it's valuable to understand out how process-based supervision can also fail or lead to a bad / misaligned training signal, especially when using synthetically generated annotations.\n\nAs for the SCAN method itself, empirically it appears to perform as well as training on human process-level annotations, and I found the experiments and ablations to be thorough.\n\nI do, however, think the justification of the confidence-wise reweighting requires some work. The authors say that, \" $c_i*$ represents the true correctness probability, while $c_i$ is the estimated probability derived from the completer model $π$ \", implying that $c_i*$ is some kind of model-independent property. However, it seems to me that the \"probability of correctness\" is never model-independent --- it's always relative to some generative process of how the remainder of the chain of thought is completed, whether that generative process is parameterized by a neural network, or by how a population of human reasoners would tend to complete a chain-of-thought. So what exactly is this \"true correctness probability\"? I don't think this is a coherent concept unless one defines some kind of \"ground-truth\" or \"reference\" generative process $\\pi^*$. Furthermore, if correctness is defined in logical terms (e.g. whether a reasoning step is mathematically valid), then a particular step is either correct on incorrect --- so the concept of \"probability of correctness\" doesn't really make sense in that context either. Instead, in the context of Monte Carlo annotation, $c_i$ is actually *the probability that some model (or human) eventually reaches a correct answer* --- which is at best a *proxy* of whether that step is actually correct.\n\nOne way to fix the issue above is to make clear that there is some expert / reference policy that we care about, $\\pi*$, (e.g. the distribution of how an expert human would generate reasoning traces). If that expert policy were used to generate Monte Carlo annotations of the dataset, we could estimate $c_i*$ as the probability that the expert would eventually generate a correct answer. We would like to use $c_i*$ as a soft label for training that is independent of the completion model $\\pi$ (even though it's not independent of $\\pi*$), but we only have $c_i$ for some other completion model $\\pi$ -- which might overestimate $c_i*$ when the completion model is stronger than the expert, or underestimate $c_i*$ when the completion model is weaker than the expert. So to estimate $c_i*$, you divide by the completion model's self-confidence. IMO that would be a stronger justification for the confidence-wise reweighting.\n\nThere are a number of other clarity issues that could be fixed. For example, it was confusing that the symbol $y_i$ is used to refer the (hard) label for whether step $i$ is correct, but then later $y_{pred}$ and $y_{true}$ are used to denote the index/step of the first error. I would suggest change it to $t_e^{pred}$ and $t_e^{true}$, to denote that this an index-valued variable. and to make it consistent with the $t_e$ notation used in Equation 5.\n\nI also didn't understand the 100% sample utilization point -- is it the case that all samples that are either negative or not high in self-confidence undergo Monte Carlo annotation? If so, that should be made clear. The line \"we collect the negative samples with $y_{pred} = inf$ from these responses for further step-wise annotation\" was also confusing to me. Are all negative samples collected, or only those with  $y_{pred} = inf$? And are these so-called \"noisy positive samples\", since  $y_{pred} = inf$ but $y_{true} < inf$? If so, why are they suddenly called \"negative samples\"? \n\nOverall though, I don't see reasons to reject this paper -- I just think the clarity and explanation / justification could be improved at various points.\n\nQUESTIONS:\n- Could you address my concern about the justification of confidence-wise reweighting?\n- Could you address the notational issues that affect the clarity of the paper?\n- How is 100% sample utilization achieved?\n- Could you more carefully explain which negative samples are collected for Monte Carlo step-wise annotation?\n\nThere are also a number of typos, e.g. \"Noise-torlerant\" should be \"noise-tolerant\", and \"noisy positive\" seems like a non-standard term compared to \"false positive\".\n\nLIMITATIONS:\nI think the authors should include a Broader Impacts statement about the implications of this PRM training strategy on AI alignment and safety. PRMs were initially introduced partly to overcome the limitations of outcome-based RMs, which can lead to safety issues due to over-optimization and reward-hacking, as we've already seen in recent studies of reasoning models (Denison et al, 2024; Bondarenko et al 2025).\nBecause PRMs are supposed to reward only valid chains of thought, the hope is that they are more resistant to this reward hacking behavior. However, Monte Carlo annotation loses this a lot of this benefit of PRMs, since the process-level labels are effectively generated entirely from outcome-based self-supervision, and also creates noise/errors in the reward specification that a policy can learn to exploit. This could also incentivize unsafe specification-gaming / reward hacking in reasoning models (e.g. rewriting test cases, so that the test is eventually passed). I think some discussion of these safety implications is warranted, since this paper uses the Monte Carlo annotation approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper presents SCAN, a framework for robust process reward learning that analyzes and mitigates noise in Monte Carlo (MC) estimation. The core contributions are an efficient data synthesis strategy and a noise-tolerant training objective.

STRENGTHS AND WEAKNESSES:
**Strengths**

The paper's primary strength lies in its insightful analysis of the noise distribution in MC-generated data, offering a novel and interesting perspective on the problem.

** Weaknesses**

A notable performance gap remains between this MC-only approach and methods that leverage Knowledge Distillation (KD) or human annotations.

QUESTIONS:
The baseline categorization in Table 2 is somewhat unconventional, as it appears to group models with different training paradigms (e.g., some baselines are not strictly MC-based), which may complicate direct comparisons.

Could the authors explain the performance discrepancy of in Table1 and Table2? For example, Qwen-PRM performs on par with SCAN models in the Best-of-N evaluation (Table 1) yet wins by a large margin on ProcessBench (Table 2). 


Suggestions
The discussion on related work could be enhanced by including recent studies on alternative PRM evaluation paradigms, such as:
[1] R-PRM: Reasoning-Driven Process Reward Modeling
[2] Inference-Time Scaling for Generalist Reward Modeling

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe research paper introduces a novel framework, SCAN, to address the challenges of training high-quality Process Reward Models (PRMs) from synthetically generated data. The authors identify that standard Monte Carlo (MC) estimation methods produce noisy labels, which causes models to overfit and limits their performance. The paper first presents a preliminary study that analyzes and categorizes the noise in MC-generated data into two types: under-estimation (incorrectly flagging correct steps as errors) and over-estimation (failing to detect the true first error). To quantify this, they propose a \"self-confidence\" metric, which measures the annotation model's own success rate on a given problem.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. It proposes an efficient and practical solution (SCAN) that improves both data quality and training robustness via self-denoising and confidence-based approaches. It tackles a real, practical challenge—reducing the cost and scaling limitations of human-annotated PRM data.\n2. This paper presents strong empirical results and made code and data available, aiding reproducibility. It achieves comparable results with only ~6% of the inference cost compared to traditional MC methods. It outperforms or matches models trained on much larger or human-annotated datasets, on challenging benchmarks like ProcessBench and Best-of-N evaluation. In addition, it shows continued improvement with increased data, without apparent overfitting.\n\n## Weaknesses\n1. The self-denoising approach’s effectiveness is inherently limited by the capability of the base model used for annotation. Additional empirical exploration of this limit could strengthen the claims.\n2. While SCAN-trained PRMs surpass baselines, a deeper discussion or ablation on the qualitative differences (e.g., error types, robustness to distribution shift) between human- and synthetically-annotated data could be valuable.\n3. The method is demonstrated mainly for mathematical reasoning datasets. It is unclear how well SCAN generalizes to other domains or types of reasoning tasks.\n\nQUESTIONS:\n1. As mentioned in weaknesses, how well does the SCAN framework transfer to tasks outside of mathematical reasoning (e.g., coding, multi-modal reasoning)?\n2. How does the choice of base annotation model (size/capability) affect the ultimate PRM performance? Is there a “minimum” effective annotator?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}