SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward
  Learning
Authors: name: Yuyang Ding, name: Xinyu Shi, name: Juntao Li, name: Xiaobo Liang, name: Zhaopeng Tu, name: Min Zhang

Process reward models (PRMs) offer fine-grained, step-level evaluations that
facilitate deeper reasoning processes in large language models (LLMs), proving
effective in complex tasks like mathematical reasoning. However, developing
PRMs is challenging due to the high cost and limited scalability of
human-annotated data. Synthetic data from Monte Carlo (MC) estimation is a
promising alternative but suffers from a high noise ratio, which can cause
overfitting and hinder large-scale training. In this work, we conduct a
preliminary study on the noise distribution in synthetic data from MC
estimation, identifying that annotation models tend to both underestimate and
overestimate step correctness due to limitations in their annotation
capabilities. Building on these insights, we propose Self-Denoising Monte Carlo
Annotation (SCAN), an efficient data synthesis and noise-tolerant learning
framework. Our key findings indicate that: (1) Even lightweight models (e.g.,
1.5B parameters) can produce high-quality annotations through a self-denoising
strategy, enabling PRMs to achieve superior performance with only 6% the
inference cost required by vanilla MC estimation. (2) With our robust learning
strategy, PRMs can effectively learn from this weak supervision, achieving a
39.2 F1 score improvement (from 19.9 to 59.1) in ProcessBench. Despite using
only a compact synthetic dataset, our models surpass strong baselines,
including those trained on large-scale human-annotated datasets such as
PRM800K. Furthermore, performance continues to improve as we scale up the
synthetic data, highlighting the potential of SCAN for scalable,
cost-efficient, and robust PRM training.