GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for
  Step-Level Reasoning
Authors: name: Yao Zhang; name: Yu Wu; name: Haowei Zhang; name: Weiguo Li; name: Haokun Chen; name: Jingpei Wu; name: Guohao Li; name: Zhen Han; name: Volker Tresp
Published: 2025-10-16 17:54:07

Abstract: Process Reward Models (PRMs) aim to improve multi-step reasoning in Large
Language Models (LLMs) by supervising intermediate steps and identifying
errors. However, building effective PRMs remains challenging due to the lack of
scalable, high-quality annotations. Existing approaches rely on costly human
labeling, LLM-based self-evaluation that is prone to hallucination, or Monte
Carlo (MC) estimation, which infers step quality solely from rollout outcomes
and often introduces noisy, misaligned supervision due to credit
misattribution. These issues result in three core limitations: noisy rewards,
low factual fidelity, and misalignment with step-level reasoning objectives. To
address these challenges, we introduce GroundedPRM, a tree-guided and
fidelity-aware framework for automatic process supervision. To reduce reward
noise and enable fine-grained credit assignment, we construct structured
reasoning paths via Monte Carlo Tree Search (MCTS). To eliminate hallucinated
supervision, we validate each intermediate step using an external tool,
providing execution-grounded correctness signals. To combine both step-level
validation and global outcome assessment, we design a hybrid reward aggregation
mechanism that fuses tool-based verification with MCTS-derived feedback.
Finally, we format the reward signal into a rationale-enhanced, generative
structure to promote interpretability and compatibility with instruction-tuned
LLMs. GroundedPRM is trained on only 40K automatically labeled samples,
amounting to just 10% of the data used by the best-performing PRM trained with
auto-labeled supervision. Nevertheless, it achieves up to a 26% relative
improvement in average performance on ProcessBench. When used for reward-guided
greedy search, GroundedPRM outperforms even PRMs trained with human-labeled
supervision, offering a scalable and verifiable path toward high-quality
process-level reasoning.