The Lessons of Developing Process Reward Models in Mathematical
  Reasoning
Authors: name: Zhenru Zhang, name: Chujie Zheng, name: Yangzhen Wu, name: Beichen Zhang, name: Runji Lin, name: Bowen Yu, name: Dayiheng Liu, name: Jingren Zhou, name: Junyang Lin

Process Reward Models (PRMs) emerge as a promising approach for process
supervision in mathematical reasoning of Large Language Models (LLMs), which
aim to identify and mitigate intermediate errors in the reasoning processes.
However, the development of effective PRMs faces significant challenges,
particularly in data annotation and evaluation methodologies. In this paper,
through extensive experiments, we demonstrate that commonly used Monte Carlo
(MC) estimation-based data synthesis for PRMs typically yields inferior
performance and generalization compared to LLM-as-a-judge and human annotation
methods. MC estimation relies on completion models to evaluate current-step
correctness, leading to inaccurate step verification. Furthermore, we identify
potential biases in conventional Best-of-N (BoN) evaluation strategies for
PRMs: (1) The unreliable policy models generate responses with correct answers
but flawed processes, leading to a misalignment between the evaluation criteria
of BoN and the PRM objectives of process verification. (2) The tolerance of
PRMs of such responses leads to inflated BoN scores. (3) Existing PRMs have a
significant proportion of minimum scores concentrated on the final answer
steps, revealing the shift from process to outcome-based assessment in BoN
Optimized PRMs. To address these challenges, we develop a consensus filtering
mechanism that effectively integrates MC estimation with LLM-as-a-judge and
advocates a more comprehensive evaluation framework that combines
response-level and step-level metrics. Based on the mechanisms, we
significantly improve both model performance and data efficiency in the BoN
evaluation and the step-wise error identification task. Finally, we release a
new state-of-the-art PRM that outperforms existing open-source alternatives and
provides practical guidelines for future research in building process
supervision models.