MASPRM: Multi-Agent System Process Reward Model
Authors: name: Milad Yazdani; name: Mahdi Mostajabdaveh; name: Zirui Zhou; name: Ying Xiong
Published: 2025-10-28 00:48:20

Abstract: Practical deployment of Multi-Agent Systems (MAS) demands strong test-time
performance, motivating methods that guide inference-time search and
selectively spend compute to improve quality. We present the Multi-Agent System
Process Reward Model (MASPRM). It assigns per-action, per-agent values to
partial inter-agent transcripts and acts as an inference-time controller.
MASPRM is trained from multi-agent Monte Carlo Tree Search (MCTS) rollouts
without requiring step-level human annotations, by propagating returns to local
targets. At inference, MASPRM guides step-level beam search and MCTS, focusing
computation on promising branches and pruning early. On GSM8K and MATH,
MASPRM-guided decoding with an outcome reward model (ORM) applied to the final
answer, improves exact match (EM) over a single straight-through MAS pass by
+30.7 and +22.9 points, respectively. A MASPRM trained on GSM8K transfers
zero-shot to MATH without retraining, adding 8.4 EM points at the same
budget. MASPRM is a plug-in value model that estimates per-agent progress and
complements verifier-style decoders, enabling more reliable, compute-aware
multi-agent reasoning. Code: https://github.com/milad1378yz/MASPRM