Process Reinforcement through Implicit Rewards
Authors: name: Ganqu Cui; name: Lifan Yuan; name: Zefan Wang; name: Hanbin Wang; name: Wendi Li; name: Bingxiang He; name: Yuchen Fan; name: Tianyu Yu; name: Qixin Xu; name: Weize Chen; name: Jiarui Yuan; name: Huayu Chen; name: Kaiyan Zhang; name: Xingtai Lv; name: Shuo Wang; name: Yuan Yao; name: Xu Han; name: Hao Peng; name: Yu Cheng; name: Zhiyuan Liu; name: Maosong Sun; name: Bowen Zhou; name: Ning Ding
Published: 2025-02-03 15:43:48

Abstract: Dense process rewards have proven a more effective alternative to the sparse
outcome-level rewards in the inference-time scaling of large language models
(LLMs), particularly in tasks requiring complex multi-step reasoning. While
dense rewards also offer an appealing choice for the reinforcement learning
(RL) of LLMs since their fine-grained rewards have the potential to address
some inherent issues of outcome rewards, such as training efficiency and credit
assignment, this potential remains largely unrealized. This can be primarily
attributed to the challenges of training process reward models (PRMs) online,
where collecting high-quality process labels is prohibitively expensive, making
them particularly vulnerable to reward hacking. To address these challenges, we
propose PRIME (Process Reinforcement through IMplicit rEwards), which enables
online PRM updates using only policy rollouts and outcome labels through
implict process rewards. PRIME combines well with various advantage functions
and forgoes the dedicated reward model training phrase that existing approaches
require, substantially reducing the development overhead. We demonstrate
PRIME's effectiveness on competitional math and coding. Starting from
Qwen2.5-Math-7B-Base, PRIME achieves a 15.1% average improvement across several
key reasoning benchmarks over the SFT model. Notably, our resulting model,
Eurus-2-7B-PRIME, surpasses Qwen2.5-Math-7B-Instruct on seven reasoning
benchmarks with 10% of its training data.