Pre-Trained Policy Discriminators are General Reward Models
Authors: name: Shihan Dou, name: Shichun Liu, name: Yuming Yang, name: Yicheng Zou, name: Yunhua Zhou, name: Shuhao Xing, name: Chenhao Huang, name: Qiming Ge, name: Demin Song, name: Haijun Lv, name: Songyang Gao, name: Chengqi Lv, name: Enyu Zhou, name: Honglin Guo, name: Zhiheng Xi, name: Wenwei Zhang, name: Qipeng Guo, name: Qi Zhang, name: Xipeng Qiu, name: Xuanjing Huang, name: Tao Gui, name: Kai Chen

We offer a novel perspective on reward modeling by formulating it as a policy
discriminator, which quantifies the difference between two policies to generate
a reward signal, guiding the training policy towards a target policy with
desired behaviors. Based on this conceptual insight, we propose a scalable
pre-training method named Policy Discriminative Learning (POLAR), which trains
a reward model (RM) to discern identical policies and discriminate different
ones. Unlike traditional reward modeling methods relying on absolute
preferences, POLAR captures the relative difference between one policy and an
arbitrary target policy, which is a scalable, high-level optimization objective
suitable for modeling generic ranking relationships. Leveraging the POLAR
pre-training paradigm, we present a series of RMs with parameter scales from
1.8B to 7B. Empirical results show that POLAR substantially outperforms
traditional non-pre-trained methods, significantly enhancing RM performance.
For instance, POLAR-7B could improve preference accuracy from 54.8% to 81.0% on
STEM tasks and from 57.9% to 85.5% on creative writing tasks compared to SOTA
baselines. POLAR also shows robust generalization capabilities in RLHF using
Reinforcement Fine-tuning (RFT), providing reliable reward signals and markedly
enhancing policy performance--improving LLaMa3.1-8B from an average of 47.36%
to 56.33% and Qwen2.5-32B from 64.49% to 70.47% on 20 benchmarks. Moreover,
scaling experiments reveal a clear power-law relationship between computation
and performance, supported by linear correlation coefficients approaching 0.99.
The impressive performance, strong generalization, and scaling properties
suggest that POLAR is a promising direction for developing general and strong
reward models.