Rethinking Reward Models for Multi-Domain Test-Time Scaling
Authors: name: Dong Bok Lee; name: Seanie Lee; name: Sangwoo Park; name: Minki Kang; name: Jinheon Baek; name: Dongki Kim; name: Dominik Wagner; name: Jiongdao Jin; name: Heejun Lee; name: Tobias Bocklet; name: Jinyu Wang; name: Jingjing Fu; name: Sung Ju Hwang; name: Jiang Bia; name: Lei Song
Published: 2025-10-01 04:21:14

Abstract: The reliability of large language models (LLMs) during test-time scaling is
often assessed with external verifiers or reward models that
distinguish correct reasoning from flawed logic. Prior work generally assumes
that process reward models (PRMs), which score every intermediate reasoning
step, outperform outcome reward models (ORMs) that assess only the final
answer. This view is based mainly on evidence from narrow, math-adjacent
domains. We present the first unified evaluation of four reward model variants,
discriminative ORM and PRM (\DisORM, \DisPRM) and generative ORM and PRM
(\GenORM, \GenPRM), across 14 diverse domains. Contrary to conventional wisdom,
we find that (i) \DisORM performs on par with \DisPRM, (ii) \GenPRM is not
competitive, and (iii) overall, \GenORM is the most robust, yielding
significant and consistent gains across every tested domain. We attribute this
to PRM-style stepwise scoring, which inherits label noise from LLM
auto-labeling and has difficulty evaluating long reasoning trajectories,
including those involving self-correcting reasoning. Our theoretical analysis
shows that step-wise aggregation compounds errors as reasoning length grows,
and our empirical observations confirm this effect. These findings challenge
the prevailing assumption that fine-grained supervision is always better and
support generative outcome verification for multi-domain deployment. We
publicly release our code, datasets, and checkpoints at
https://github.com/db-Lee/Multi-RM{\small\texttt{https://github.com/db-Lee/Multi-RM}}
to facilitate future research in multi-domain settings.