Reward Reasoning Model
Authors: name: Jiaxin Guo; name: Zewen Chi; name: Li Dong; name: Qingxiu Dong; name: Xun Wu; name: Shaohan Huang; name: Furu Wei
Published: 2025-05-20 17:58:03

Abstract: Reward models play a critical role in guiding large language models toward
outputs that align with human expectations. However, an open challenge remains
in effectively utilizing test-time compute to enhance reward model performance.
In this work, we introduce Reward Reasoning Models (RRMs), which are
specifically designed to execute a deliberate reasoning process before
generating final rewards. Through chain-of-thought reasoning, RRMs leverage
additional test-time compute for complex queries where appropriate rewards are
not immediately apparent. To develop RRMs, we implement a reinforcement
learning framework that fosters self-evolved reward reasoning capabilities
without requiring explicit reasoning traces as training data. Experimental
results demonstrate that RRMs achieve superior performance on reward modeling
benchmarks across diverse domains. Notably, we show that RRMs can adaptively
exploit test-time compute to further improve reward accuracy. The pretrained
reward reasoning models are available at
https://huggingface.co/Reward-Reasoning.