ICML 2025

GRAM: A Generative Foundation Reward Model
for Reward Generalization
Chenglong Wang 1 Yang Gan 1 Yifu Huo 1 Yongyu Mu 1 Qiaozhi He 1 Murun Yang 1 Bei Li 2 Tong Xiao 1 3
Chunliang Zhang 1 Tongran Liu 4 Jingbo Zhu 1 3
Code
Datasets&Models
Abstract
In aligning large language models (LLMs), re-
ward models have played an important role, but
are standardly trained as discriminative models
and rely only on labeled human preference data.
In this paper, we explore methods that train re-
ward models using both unlabeled and labeled
data. Building on the generative models in LLMs,
we develop a generative reward model that is first
trained via large-scale unsupervised learning and
then fine-tuned via supervised learning. We also
show that by using label smoothing, we are in fact
optimizing a regularized pairwise ranking loss.
This result, in turn, provides a new view of train-
ing reward models, which links generative models
and discriminative models under the same class
of training objectives. The outcome of these tech-
niques is a foundation reward model, which can
be applied to a wide range of tasks with little or no
further fine-tuning effort. Extensive experiments
show that this model generalizes well across sev-
eral tasks, including response ranking, reinforce-
ment learning from human feedback, and task
adaptation with fine-tuning, achieving significant
performance improvements over several strong
baseline models.
1. Introduction
Reward models are a fundamental concept in reinforcement
learning and define what an agent optimizes for. For large
language models (LLMs), fine-tuning with reward models
is a common post-training step to align the model outputs
with desired behaviors and objectives. A widely adopted
1School of Computer Science and Engineering, Northeastern
University, Shenyang, China 2Meituan Inc. 3NiuTrans Research,
Shenyang, China 4CAS Key Laboratory of Behavioral Science,
Institute of Psychology, CAS, Beijing, China. Correspondence
to: Chenglong Wang <clwa...