GRAM: A Generative Foundation Reward Model for Reward Generalization
Authors: Chenglong Wang
Yang Gan
Yifu Huo
Yongyu Mu
Qiaozhi He
MuRun Yang
Bei Li
Tong Xiao
Chunliang Zhang
Tongran Liu
JingBo Zhu
Venue: ICML 2025 poster (2025)
Keywords: Large Language Model
Reward Modeling
RLHF
Primary area: deep_learning->large_language_models
TL;DR: This study introduces a generative foundation reward model (GRAM), which pre-learns preferences through a two-stage training process and selective label smoothing.

Abstract: In aligning large language models (LLMs), reward models have played an important role, but are standardly trained as discriminative models and rely only on labeled human preference data.  In this paper, we explore methods that train reward models using both unlabeled and labeled data.  Building on the generative models in LLMs, we develop a generative reward model that is first trained via large-scale unsupervised learning and then fine-tuned via supervised learning.  We also show that by using label smoothing, we are in fact optimizing a regularized pairwise ranking loss.  This result, in turn, provides a new view of training reward models, which links generative models and discriminative models under the same class of training objectives.  The outcome of these techniques is a foundation reward model, which can be applied to a wide range of tasks with little or no further fine-tuning effort.  Extensive experiments show that this model generalizes well across several tasks, including response ranking, reinforcement learning from human feedback, and task adaptation with fine-tuning, achieving significant performance improvements over several strong baseline models.

GRAM: A Generative Foundation Reward Model for Reward Generalization
Chenglong Wang'
Yang Gan
Yifu Huo
Yongyu Mu
Qiaozhi He Murun Yang
Bei Li Tong Xiao 13
Chunliang Zhang'
Tongran Liu Jingbo Zhu" 13
Code
Models
Abstract
approach is to learn reward models that capture human preferences and fine-tune the LLMs to generate outputs that align with these preferences. Reinforcement learning from human feedback (RLHF) is an early example of such approaches (Christiano et al., 2017; Stiennon et al., 2020). Now, work in this area is underway more broadly, One recent example is a series of models by OpenAl (2024), in which human-like thinking and complex reasoning can be achieved through large-scale reinforcement learning.
In aligning large language models (LLMs), re- ward models have played an important role, but are standardly trained as discriminative models and rely only on labeled human preference data. In this paper, we explore methods that train re- ward models using both unlabeled and labeled data. Building on the generative models in LLMs, we develop generative reward model that is first trained via large-scale unsupervised learning and then fine-tuned via supervised learning. We also show that by using label smoothing, we are in fact optimizing a regularized pairwise ranking loss. This result, in turn, provides a new view of train- ing reward models, which links generative models and discriminative models under the same class of training objectives. The outcome of these tech- niques is a foundation reward model, which can applied to a wide range of tasks with little or no further fine-tuning effort. Extensive experiments show that this model generalizes well across sev- eral tasks, including response ranking, reinforce- ment learning from human feedback, and task adaptation with fine-tuning, achieving significant performance improvements over several strong baseline models.
While quite successful, reward models costly to apply. This is in part because of the complexity of reinforcement learning algorithms and in part because of the difficulty in annotating training data. There has been much work on simplifying the use of reward models and improving align- ment efficiency. One strand research explores more direct ways to align LLMs with human feedback, employing either supervised fine-tuning methods (Rafailov et al., 2023; Tou- vron et al., 2023) or inference-time alignment methods (Lee et al., 2021). Another strand of research focuses replac- ing human feedback with Al-generated feedback, which is cheaper to obtain (Dubois et al., 2023; Lee et al., 2024).
However, although applying reward models to L.IMS compelling direction, training these models still relies heavily labeled data. For example, we generally need to collect or create significant amount of task-specife human preference data and optimize the models with considerable training effort (Stiennon et al., 2020; Xu et al., 2024). If we think about the problem a bit closer from the LLM perspective, we might expect that reward models can be trained on unlabeled data in such way as to produce a single pre-trained reward model that can be easily adapted to tasks of interest. This would change the way we align LLMs: we can pre-train a foundation model that assembles a broad general knowledge of how to reward, and a single such pre-trained model can be deployed for many particular rewarding tasks with only small costs of further fine-tuning or prompting.
1. Introduction
Reward models are fundamental concept in reinforcement learning and define what an agent optimizes for. For large language models (LLMs), fine-tuning with reward models a common post-training step to align the model outputs with desired behaviors and objectives. A widely adopted
School of Computer Science and Engincering, Northeastern University, Shenyang, China "Meituan Inc. 'NiuTrans Research, Shenyang, China *CAS Key Laboratory of Behavioral Science, Institute of Psychology, CAS. Beijing, China. Correspondence to: Chenglong Wang <clwangl 119 @wtll1119llm...mmm gmail.com>, Tong Xiao <xiaotong @ mail.neu.edu.en.>
This idea is appealing but challenging. The difficulty arises from the fact that the systems cannot directly generate their own supervision signals from text for training reward mod- els, as self-supervision methods do. One approach is to
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
collect large-scale preference data for general use and train reward model this data to improve generalization (Cui et al., 2023; Liu et al., 2024). However, in this case, large amounts of unlabeled data are still largely overlooked.
Minimizing the Bradiey-Tery loss (pairwise ranking loss): log(a(ra(zz (1; ya) Te (ar 86]))
Yo (it, y)
maap
In this paper, we propose a solution to this problem that learns reward models not only from human-annotated pref- erence data but also from unlabeled data. To do this, we develop a generative model that can predict, given the input and a pair of responses, which one is better. The train- ing of this model involves two stages. In the first stage, we pre-train the model on input-response data to learn the correspondence between inputs and responses. This pro- cess does not require sefference-annottann data and so can be easily scaled up to gain more general knowledge of re- sponse comparison. In the second stage, we fine-tune the model using human preference data to predict the preference between two responses. The resulting foundation reward model can be directly applied to downstream tasks, such as policy training, or further fine-tuned with a small amount of task-specific data.
Transformer Decoder (without Softmax layers)
x
Injut
respotie
(a) Diseriminative Models (Trained as Classifiers)
Minimizing the negative probahility of token prediction:
log x = Al(e, Da. wt])
token ('A' 'B')
Transformer Decoder (LLM)
Iiu
Mo
To make the model generalize better, we incorporate label smoothing into reward model training. We show that the training objective can be reformulated into nice form: we are essentially optimizing the Bradley-Terry loss (Bradley & Terry, 1952) under the condition of label smoothing. This result is elegant, as it unifies generative and discriminative models in reward modeling to some extent. Though label smoothing seems not so popular in the development of re- cent LLMs, it turns out to be very beneficial for training generative reward models.
prompe
responne
2nd response
(b) Generative Models (Trained as LLMs)
Figure 1: Architectures of discriminative and generative reward models. discriminative models, the reward model is scoring function that is trained minimize the pairwise ranking loss between we use an LLM to predict the label token given a prompt, an input, and a pair of responses. This model can be trained in the same way as standard LLMs.
responses. In generative models,
The foundation reward models can be applied to wide range of tasks. In our experiments, we test it in three differ- ent settings: response ranking, RLHF, and adaptation. Our model demonstrates strong generalization results across all test cases with little or no fine-tuning effort and improves performance significantly compared with various discrimi- native and generative baseline models. Notably, when train- ing reward models with the LLaMA-3.1 Inst-ln model, our model achieves gains of 11.0 and 5.1 points over vanilla discriminative and generative reward models, respectively, on the average accuracy of RewardBench.
LLM, such as What is the capital of France?, and "re- sponse" is the token sequence produced by the LLM as result of that input, such as Paris.
A widely used architecture of such functions is a Trans- former decoder stacked without Softmax layer on top, as illustrated in Figure 1 (a). This model be viewed discriminative classification model, and is commonly trained using the Bradley- Terry loss, given by
La
-E u.) [log(a ya) T (2. (6)))
(I)
2. Preliminaries
where D, is the training dataset consisting of tuples of input x and response pair (Ja- yb) with the preference ya > yb- While this loss function considers pairwise ranking between responses, the trained reward model is used as a scoring function that assigns numerical score (2 y) to any response y, together with the corresponding input x.
In this section, outline some basic concepts and notations of reward modeling.
2.1. Training Reward Models
In the LLM literature, reward model is typically written function ra( y). where Φ is the set of model parameters, a is the input, and is the response. Throughout this work, an "input" can be an arbitrary token sequence fed into an
Reward models can also be generative models (Zhang et al., 2024; Shiwen et al., 2024). In this case, we can simply use an LLM as a reward model, as illustrated in Figure (b). This model works as follows. First, we input prompt c,
2

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
along with the tuple (3r, Ma Vb), to the LLM. The prompt is text describing the task. For example,
Discriminative RM
Generative RM
70.0
80.0
You are given two responses to a user input. Evaluate which response is better based on quality, relevance, and clarity. If the first response is better, return 'A'. If the second response is better, return 'B'.
Accuracy
Accuracy
76.0
06.0
74.0
72.0
Then, the LLM predicts subsequent tokens based on this input sequence. Let w be the token predicted by the LLM. If = A, it indicates a preference for Ya over yo: if w B, then ybb is preferred.
62.0
70.0
40k
400k
40k
400k
(a) ID -Unified-Feedbace
(b) OOD (RewardBench)
Figure 2: Accuracies of discriminative and generative re- ward models on the ID and OOD test sets.
The loss function can be defined the log- proobbbbyyy of predicting 'A':
the reward model further to better adapt to the task. The fine-tuned reward model can then be applied to LLM fine- tuning usual.
gg = -BO....... [og aT ( A|s)]
(2)
where s denotes the string [c, Va- 3ol', and (-) denotes the probability of token prediction by the LLM.
3. A Generative Foundation Reward Model
When applying this model to score a new input-response pair (a),)"), we generate a reference response Veef by us- ing the LLM, and concatenate x', y' and Uref into s' (c, v/, Bree]- Additionally, to mitigate the positional bias problem (Wang et al., 2023), we introduce alternative input order by transposing the positions of responses, i.e., presenting Yref before y', to construct a secondary input string s'T = [c, x', Breff 3/]. The reward for (x', is thus defined as the log-probability that y' is preferred over yref:
In this section we describe a Generative foundation Reward Model, called GRAM.
3.1. Why Generative Models
Both discriminative and generative models have been widely adopted in reward modeling, but we found that generative models were more generalizable and better suited to our work. To study this issue, we trained both types of models subset of 400k and 40k samples from the Unified- Feedback dataset?, We then evaluated these models on an in- distribution (ID) test set, consisting of 1 test samples from Muified-Feedback, and an out-of-distribution (OOD) test set, consisting of 3k samples from RewardBench (Lambert et al., 2024). shown in Figure 2, the discriminative model better on the ID test data, while the generative model is better the OOD test data. While preliminary, this result supports the finding in previous work by Yang et al. (2024) and Zhang et al. (2024): LLMs can generalize better for reward modeling.
(w Als') + vo(w- = B B(
Tol "o(2',0) ,')
(3)
2
where the reward score ranges from 0 to 1.
2.2. Applying Reward Models
Three applications of foundation reward models can be con- sidered in LLMs. One simple application is response rank- ing, where a number of responses are given, and we score and rank these responses. This approach is often used in reranking the LLM outputs. For example, in best-of-n sam- pling, we select the best output from the top n candidate outputs via a reward model (Lee et al., 2021; Fernandes al., 2022: Gao et al., 2023).
In fact, discriminative and generative models have many similar design choices, such as using Transformer decoders (i.e., LLMs) encode input-response pairs, but the training strategies make them behave quite differently. Training with the pairwise ranking loss provides strong supervision signal. Given that the reward model is well-traines LLM, it is more prone to overfit when simply mapping an input- response pair to the reward model. Generative models are essentially trained a similar manner, but with much noisier samples. For example, each time, we need to model two responses in the same sequence, and adding an extra prompt to the sequence introduces more modeling challenges. The
A second application is rewed fine-tuning, where the reward model provides feedback to optimize the LLM. For example, in RLHF, reward model is used in proximal pol- icy optimization (PPO) (Wang et al., 2022) to fine-tune the LLM for better alignment with human preferences (Ouyang al., 2022; Bai et al., 2022).
A third application is reward model adaptation. If we have labeled human preference data for task, we can fine-tune
In this work, we will use interchangeably refer either the tuple a training sample or a string representing that tuple.
hntspa://hugging http:///gggg..acccccccaaaaatttt 11 Tombbbleddr/Um MeddFeedback
3

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
diversity and variability of the samples make the training task more difficult, which in turn encourages the model to generalize more.
Leaminng generate pairs of responses: Minimizing log xe (lBo: Vo. lal
Ist response 2nd respoese
Va
Furthermore, recent research highlights the superior flexibil- ity of generative models compared to discriminative mod- els in their adaptability to various LLM enhancement tech- niques. For example, the seamless integration of chain-of- of- thought reasoning within a generative reward model has been shown to improve reward accuracy (Mahan et al., 2024). Beyond merely adopting such reasoning patterns, the generative reward model can be designed to perform long-form reasoning before generating preferences (Chen et al., 2025; Guo et al., 2025). This enhanced flexibility makes the generative model an ideal choice for our work, as we aim to build a more versatile foundation reward model.
Transformer Decoder (LLM)
inpu:
(a) Task 1: Pre- Pre-training ( (Unsupervised)
Learning to predict preferences (with lahel smonthing): Minimizing - - e) -log (w = Al[e, x. Na: v&]) log W Bl[e, 2, Bu 3/s])
label token
3.2. A Two-stage Training Method
Transformer Decoder (LLM)
Unlike previous work, we do not use only human preference data to train reward models. Instead, we train them using an unsupervised pre-training task and supervised fine- fine-tuning task, as described in this section.
prompt
input
1st response 2nd response
(b) Task Fine-tuning (Supervised)
3.2.1. TASK 1: PRE-TRAINING
Figure 3: Illustration of the two-stage training method. In the first stage, we pre-train the model via response gener- ation, which is an unsupervised task. In the second stage, we fine-tune the model to generate preferences in standard supervised manner.
Intuitively, one might expect that an LLM-based reward model is capable enough to model the inputs, responses, and correspondences between them, as LLMs have been trained on huge amounts of text. Unfortunately, modeling the input- response correspondence is not covered by standard pre- training and fine-tuning tasks for LLMs, and the model still needs to adapt to this modeling task. This has been demonstrated by recent work, where the understanding of responses is found to be very important in modeling human preferences (Wang et al., 2024e; Zheng et al., 2023).
Furthermore, since the two responses are generated at the same time, the model can gain some general knowledge of response comparison. Note that the order responses does not matter in pre-training, so we can swap them to create more diverse samples for robust training.
To improve response understanding, we train the LLM to generate responses given the input, illustrated in Figure 3 (a). We refer to this as a "pre-training" procedure, as it does not require human preference data, although it differs from standard pre- training methods in LLMs. Let D,, be a dataset consisting of tuples of input and pairs of responses with no preference specified. This loss function can be defined as:
3.2.2. TASK 2: FINE-TUNING
The goal of fine-tuning here is to adapt the model to pre- dict preferences between responses, which is not directly captured by pre-training. We can do this by using the same method described in Eq. (2) (see Figure (b)). As pre- training has helped the model gain knowledge of response comparison from relatively large amounts of data, fine- tuning is easier and requires much less data. In this work, consider using
Cpre -L [lg To (Maa ]0][=]]]
(4)
preference data to fine- tune our pre-trained model, thereby obtaining a foundation model that works in various rewarding tasks. If users have their own data, such as human-annotated preference data for a specific task, they can fine-tune this model further.
Here Ya and yo can be generated by an LLM. So, building such a dataset is straightforward.
This training objective is similar to that used in instruc- tion fine-tuning (Ouyang al., 2022). The difference be- tween our method and instruction fine-tuning is that we train the LLM to generate two responses, while instruction fine- tuning trains the LLM to produce a single correct response to the input. By learning the mapping from inputs to varied responses, the model can better understand the responses.
3.3. Training with Label Smoothing
To further improve generalization, we incorporate fine- tuning with label smoothing. The idea of label smoothing is to redistribute the probability mass across all predicted

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
tokens by distracting a fraction of the probability from the correct token (denoted by w²) to the incorrect tokens. For fine-tuning with label smoothing, the loss for sample can be defined as:
4. Experiments
We evaluated GRAM on various applications, including its accuracy on the response ranking, effectiveness in reward- based fine-tuning, and adaptability rewarding tasks.
Lis(s) = -Bur-tiil
(1-c) 1( = w²) log (w)ss))
4.1. Setups
V 1110 # w") log [o(]s)]] (5)
We initialized our GRAM model with the LLaMA-3. 1.--BB- Instruct and LLaMA-3.2-3B --sstuut models, using a subset of 400k samples from Unified-Feedbac for each. The learning rates were set to 2e-5 for the first stage and le-5 for the second stage, with training conducted over one epoch in each stage. Note that while this data includes preference labels, we do not use these labels in our pre-training process. Instead, we only use the response pairs to simulate unlabeled data and validate the effectiveness of our method. In the second stage, the label smoothing parameter was set to 0.1, with other settings tested as shown in Figure 10. More details can be found in Appendix B.
where c is the smoothing factor, and Vi the vocabulary of label tokens. This formula is general and can handle multi-class problems (i.e.. [i] > 2). But, to simplify the discussion, we still restrict ourselves to the case of two classes. Hence, we express the loss as
(ss(s) = -1(1 =- - c) log T ( = Als)
+e log TH (w = B[s)]
(6)
We can rewrite this formula into another form by noting that To (w(ss is the output of a Softmax layer. This gives
4.2. Baselines
(Zo(s)
We compared GRAM with several strong baselines: LLM- as-a-Judge, where we prompted LLMs like GPT-4o to gen- erate preferences; open-source reward models, open-source discriminative and generative reward models that approxi- mate 3B or 8B, including Armo SAARRM--lla3----Ve et al., 2024d), and others; and training on the same prefer- ence dataset, denoting the standard reward models trained discriminative and generative frameworks using Unified- Feedback, respectively (Discriminative R.M and Generative RM). We also compared GRAM with several approaches designed to enhance generalization. These include the stan- dard Label Smoothing, which prevents the model from be- coming overly confident in its predictions; Freeze, which fixes certain parameters of the LLM during training (Zilly et al., 2021); and Regularization, which adds the discrimi- native reward model loss using the SFT loss function (Yang et al., 2024).
(s) -[(1- - e). .lg (s)) + (o(ss
2.(s)
+e log (Za(z) + (o(5
(Wang
where (s) and Zs) are the logits input to the Softmax function.
Using simple algebra, we obtain:
(s) = - log (Za(s) - (s))
Bradley-Teny model
+ (Za(s) (s))
(8)
regularization term
See Appendix D.2 for a more detailed derivation. The first term of Eq. (8) is the loss based on the Bradley-Terry model, and the second term serves as regularization term. This result is quite interesting. We are actually optimizing a reg- ularized Bradley-Terry loss. It also establishes connection between the discriminative and generative training methods discussed in Section 2.1: both methods essentially train LLMs to perform pairwise ranking.
4.3. Pair-wise Response Ranking
Task Setups. The pair-wise ranking is commonly used to test reward models (Lambert et al., 2024). Given test data Dd
pair = ((x', y.. }. where x denotes the test input, and vi and y denote its corresponding responses, the task is to identify the preferred response. The test sample (x*, yi. vt) can be evaluated using reward model:
Note that while label smoothing is theoretically appealing (Müller al., 2019; Lukasik et al., 2020), past experience shows that it is not very helpful for LLMs. One recent example of this problem is the experiments by Ruan et al. (2024), in which label smoothing degrades the performance of LLMs in some cases. By contrast, in our work, this technique turns out to be very beneficial. We will see in Figure 10 that using label smoothing is critical in training our reward models.
va x vé. if ro (((xx.y))) vt) > ra(a", ((x"'_' .v() va - vb. if > rs(a", v.) (9) Tie,
Rank(y vb)
ifrø(a', "ro(a",bte ..)) ra(n' vv.)
For generative reward models, when calculating the reward score for one of two responses, the other is used as the
5

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
REWARDBENCH
HHH--ALIGNMETT
Method
UNIFRED
Chat Chat- Safety Reasoning Avg. Harmless. Helpful. Honest. Avg.
LLM-as-a-hudge
GPT-41
95.3
74.3
87.6
86.9
86.0
96.6
91.5
82.0
90.0
GPT-4o
96.6
70.4
86.5
84.9
84.6
98.3
90.0
83.6
90.6
GPT-3.5-turbol
92.2
44.5
62.3
59.1
64.5
74.1
78.0
72.1
74.7
Open-Source Reward Models
pair-pefcrennc--oeeeLLLLLa GPM-Gemma-2BB
96.9
76.8
90.5
97.3
90.4
90.2
86.2
76.7
84.4
71.5
69.7
81.2
75.5
74.5
76.3
68.4
66.0
70.2
Jalene---Min--LLam--1111 Stywork-Criti--llmm--3
93.6
79.4
89.3
94.3
89.1
90.6
87.5
78.4
85.5
93.6
81.4
91.1
89.8
89.0
91.3
87.1
76.2
84.9
Training the Same Preference Dataset (LLaMA-3.1) 1-B---3.-----nsstte
Discriminative RM (Baseline)
69.3
80.7
73.5
74.9
67.4
74.1
75.3
78.6
74.1
76.0
Discriminative RM + Freeze
66.6
81.3
75.2
78.8
64.2
74.9
74.1
81.4
77.0
77.5
Discriminative RM + Regularization
72.7
85.8
74.3
80.1
69.5
77.4
74.6
84.1
80.3
79.7
Generative RM (Baseline)
66.8
88.6
79.3
79.6
72.4
80.0
76.9
83.9
80.2
80.3
Generative RM Freeze
65.4
91.2
82.3
81.7
75.1
82.6
77.2
81.9
81.3
80.1
Generative RM Label Smoothing
67.9
89.5
80.1
80.4
72.6
80.7
77.0
80.2
81.2
79.5
GRAM (Ours)
70.4
90.4
86.9
84.6
78.3
85.1
83.9
88.6
83.7
85.4
Training on the Same Preference Dataset
Discriminative RM (Baseline)
68.3
86.6
71.6
71.9
61.1
72.8
79.8
77.9
70.5
76.1
Diseriminative RM Freeze
63.0
83.9
67.4
73.5
57.0
70.5
63.5
69.1
69.8
67.5
Discriminative RM + Regularization
65.6
85.4
68.7
74.2
56.5
71.2
73.4
72.1
71.7
72.4
Generative RM (Baseline)
65.3
87.9
78.8
77.5
71.1
78.8
75.5
77.3
78.8
77.2
Generative RM + Freeze
63.7
83.6
76.5
78.3
68.8
76.8
73.2
76.1
77.0
75.4
Generative RM + Label Smoothing
66.2
85.7
78.1
80.3
72.0
79.0
76.4
78.3
79.7
78.1
GRAM (Ours)
70.6
90.6
83.9
79.8
80.2
83.6
81.2
84.1
80.3
81.9
Table 1: Accuracies (%) on the pair-wise ranking with both ID (UNIFEED) and OOD (REWARDBENCH and HHH- ALIGNMENT) test sets. The best performance in each group bold and the second best underlined. Results marked with for RewardBench are from Lambert al. (2024). The other baseline results obtained by testing this available model or API. We use dotted line distinguish between the discriminative and generative reward models. We report the average accuracy for RewardBench and HHH--lignment sets in the "Avg." column.
reference response. For example, to compute ro(z", vy)), we use v/, as the reference response.
average score on the RewardBench from 80.0 to 85.1. This shows that relying only on prompt engineering in subopti- mal reward model, even with a strong L.M,, is insufficient. This finding here is consistent with the result in Zhang et al. (2024). Additionally, compared open- open-source reward mod- els trained large-scale, high-quality labeled data, GRAM demonstrates competitive performance. As shown in Figure 6, GRAM outperforms these open-source models more fine-tum tuning data is used, achieving an average accuracy of 91. on the RewardBench.
Results Generalization. We used pair-wise response ranking task to evaluate the generalization capability of GRAM. Table 1 shows the results of GRAM and its base- lines on both ID and OOD test sets. Firstly, the results here confirm the findings from Section 3.1, demonstrating that discriminative reward models are less effective than generative reward models in generalization, even when en- hanced generalization methods are applied. Interestingly, GRAM also outperforms the discriminative reward model on the ID test set, underscoring the substantial improvement and generalization capability of GRAM in reward model- ing. Furthermore, compared to LLM-as M-MM-a-a-Jiee a------ methods, 8B Generative RM (Baseline) achieves a competitive score, while GRAM shows a notable improvement, increasing the
From the results, we also observe that GRAM underper- forms compared discriminative models data, which may raise concerns about overfitting in the discriminative models rather than better generalization. However, this is not the case. First, the ID test set evaluates the model's abil- ity to learn human preferences from labeled data, and our goal is to excel in both ID and OOD tasks. As shown by the
6

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
D- D-Baseline
D-Freces
D- Regulariiction
G-Baseline
G-Preeze
G-Label Smmothing
GRAM
24
2.4
2.0
Proxy Score
1.8
Oracle Score
Proxy Score
1.6
Orncle Scure
2.2
1.6
2.0
1.4
2.0
1.8
1.2
2.6
2
3
2
0
5
0
KL. Divergence
KL Divergence
KL Divergence
KL Divergence
(a) BoN (LLaMA-3. -.LLMAA-..-----ssson 1-8t-orrtcc)))
BoN (LLaMA--.2-3B--mm
Figure 4: Performance of GRAM and its baselines on BoN sampling. We proxy scores to assess preference learning and oracle scores to evaluate the generalization capability. "D>-" and "G-" denote that the reward model is trained using discriminative and generative reward modeling frameworks.
1---A--.1-8B-Inttiii results in Table 1, GRAM achieves the best OOD results and second-best ID) results. Sec- ond, while GRAM underperforms relative to Discriminative RM-Regularization on the LLaMA-3. Insttrruc model, outperforms both the Discriminative RM (Baseline) and Diseriminative RM+Freeze, demonstrating GRAM's strong performance. Additionally, we find that regularization' s effectiveness is model-dependent, as it performs worse than GRAM on the M---A----2-3B-Intttte model.
model on a 40k subset of the Unified-Feedback dataset to provide a proxy score for the responses selected by GRAM and its baselines. Additionally, we trained an oracle re- ward model using preference data from AlpacaFarm (Dubois et al., 2023), which accurately measures response quality to assess generalization, as AlpacaFarm's preference data is distributed alongside AlpacaEval2. Following Gao et al. (2023)'s work, we varied the KL Divergence between 0 and 4.5, which corresponds to a range of ls from to 244 responses, according to the equation KLBON = log k A-1
4.4. List-wise Response Ranking
Results of Best-of-n Sampling. Figure presents the BoN sampling results for reward models of the 3B and 8B sizes. When comparing discriminative and generative re- ward models, we observe that the discriminative reward model yields a strong proxy score but underperforms in oracle scores. This indicates that while the discriminative reward model exhibits robust preference learning, its gen- eralization capability is weaker, consistent with observa- tions in pair-wise ranking. In contrast, GRAM excels in list-wise ranking in both proxy and oracle reward model evaluations. We observe decline in oracle scores for base- line models when the KL divergence exceeds 3, attributable to over- over-optimization. However, GRAM mitigates this issue, demonstrating its potential as a reliable foundation reward model in RLHF. We also further evaluate its performance during PPO fine-tuning, as shown in Appendix C.I.
In practice, multiple responses are typically generated for reranking. Given a list-wise test set Dlsst = {(*', vi. Vé. ...
....))))) }h) where denotes the list size, we begin by randomly selecting a response 1/ as the reference re- sponse Vref. We then compute reward scores (ra(x", vf),
rola", v6).... ... "ra""""" v(-1)) for the remaining responses via Eq. 3. These scores are subsequently used for ranking these responses². Additionally, when the goal is to find the best response from the response list, a straightforward linear search approach can be employed. Specifically, we start by defining of as the best response v and comparing it iteratively with the remaining responses with the generative reward model. At each comparison, if vt is found be inferior, it is replaced by the compared response. Through this process, we can determine the best response. To support parallel computation and enhance efficiency, we also incor- porate optimization algorithms, such as divide-and-conquer.
4.5. Reward Model Adaptation
The adaptability of a reward model crucial for its per- formance across various tasks, as it enables the model to effectively adjust to different environments and preferences (Cheng et al., 2023; Wang et al., 2024a). To evaluate GRAM's adaptability, we conducted experiments in two distinct tasks: adapting to the summarization task and adapt- ing to the harmlessness preference type. For each task, we fine-tuned GRAM on a small, labeled dataset containing
Task Setups. We used best-of-n (BoN) sampling to eval- uate GRAM on list-wise ranking. We performed BoN sampling on the 1.aMA-3.1-8B-Imme responses per input. The test 2023). In all BoN experiments, we trained a proxy reward
model using k AlpacaEval2 (Li et al.,
IIn the list-wise response ranking pracess, the reward score for the reference response y is set to 0.5.
7

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
Orocle RM
Vanilla RM
RM line-tuned with D- Baseline
RM fine-tuned with G- G-Baseline
RM line-tuned with GRAM
Andrewcy (85) on Harmlessess
Accuracy (1) cn Summuritation
Accuncy (46 Harmleooess
70
70
60
6.5
60
Accuray
65
01 11
36.
7k
10k
01
18
54
7k
10k
04:
1k
36
56
7%
10k
06.
14
56.
7%
10k
Amount of Preference Data
Amount Preference Data
Amountt Preference Data
Amnount of Preference Data
(a) Adaptation (LLaMA-3.11-88 (B-srrct
Adaptation (LLaMA-3.2 3.2tut-Insttt)
Figure 5: The performance of reward models fine-tur fine-tuned with varying amounts mask-specific preference data (summarization and harmlessness). Please refer to Figure 11 for the results on the four remaining baselines, including D-Feeze, D- Regularization, G-Freeze, and G-Label Smoothing.
GRAM (LL-MA-3.1 Insstrrrcc))
GRAM L.M--.2---20 1.M---.--18--nstraaar
Second Stage (100k Labeled
Second Stage (200k Labeled Data)
Sccond Stago Labeled Dota)
Second Stage (600k Labeled
84
80
82
Accursy (16)
82
Accuracy (16)
80
Accuracy
74
76
244
76
82
74
79
70
10UL
100k 2006
400%
1036
4006:
Amount of Unlabeled Data
Amooutt of Unlabeled Data
Ammonnt of Unlabsled Data
Amount of Unlabelod Data
Figure 6: Performance scaling laws for different amounts of unlabeled data used in the first stage. "Ok unlabeled data" refers training GRAM solely in the second stage, without using any unlabeled data for pre-training.
task-specific preference data, followed by testing it the corresponding task-specific test sets.
that GRAM can substantially reduce the need for preference data labeling in reward modeling. Furthermore, compared to baselines, GRAM consistently outperforms them as a foundation reward model, underscoring its efficiency in adapting to task-specific requirements with minimal data. More experimental results can be found in Appendix C.
Task Setups. For each task, we vary the number of sum- marization data across {Ok, 1k, 3k, 5k, 7k, 10k). derived from preference data labeled by Stiennon et al. (2020) and Bai et al. (2022), respectively. We used the full task-specific datasets 92k samples for summarization and 42k for harm- lessness to train reward models, respectively, which served as baselines (Oracle RM). We also trained reward models based on the LLaMA-3.. 1--..-1111 -Ss--nttt and LLaMA-3.2-D0 Instruct models, using only the task-specifie preference data as baselines (denoted as Vanilla RM).
5. Analysis
5.1. Scaling Unlabeled Data for Improved Performance
To further investigate the impact of pre-training with unla- beled data on GRAM's performance, we trained GRAM using the I.LaMA-3.1 1-8---.------nsttrccc and L.L.MA3.2-33B Instruct models with varying amounts of unlabeled and labeled data. The model's performance was evaluated on the OOD test set (RewardBench), as shown in Figure The results demonstrate that the amount of unlabeled data in- creases, the accuracy of GRAM generally improves for both models, with the most significant gains observed when mov- ing from Ok to 200k unlabeled data. This also demonstrates the crucial role of unlabeled data and the scaling effect on performance, suggesting that using larger unlabeled datasets can lead better reward models.
Results on Reward Model Adaptation. Figure shows the accuracies of the reward models, which are fine-tuned with different amounts of summarization and harmlessness preference data. We that fine-tuning GRAM with small amount of preference data, such as 1 or 3k samples, is sufficient to yield high-quality reward models. Notably, using only 3k summarization samples, we achieve a task- specific reward model that performs comparably to one trained on the 92k samples (75.6 vs. 77.8). This proves
8

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
tried to improve these reward models for better LLM align- ment. The first focuses large-scale, high-quality training data, developing either task-specific datasets (Stiennon et al., 2020; Xu et al., 2024) or more general preference datasets (Bai et al., 2022; Cui et al., 2023). The other explores stronger models for reward modeling, such as reward model ensembling (Coste et al., 2024; Min et al., 2024). Although reward modeling through these methods captures human preferences effectively, they often rely heavily labeled data. Researchers have noticed this issue. For example, Lee et al. (2023) employed LLMSs to replace human annotators, and Cui et al. (2023) developed large-scale preference dataset for general-purpose use. However, these efforts overlook the potential of vast amounts of unlabeled data.
Method
Accuracy
Vanilla RM
56.5
GRAM
71.6
GRAM w/ Domain
74.7
GRAM w/o Domain
67.4
Table 2: Accuracy (%) of different GRAM variants.
5.2. Impact of Domain Difference Between Pre-training and Fine-tuning on Reward Model Adaptation
We investigate the impact of domain differences on reward model adaptation. More specifically, evaluate three vari- ations of GRAM in the context of reward model adaptation:
GRAM: Pre- Pre-trained on 100k general unlabeled data (including summarization responses), followed by fine- tuning on 5k labeled summarization data.
Foundation Models. This work joins a large body of work demonstrating that a neural network trained on unlabeled data at scale can gain some general knowledge and is easy to adapt wide range of downstream tasks (Moor et al., 2023; Xiao & Zhu, 2025; 2023). Such guiding principle has motivated the development of many successful LLMs, such as the BERT and GPT series (Devlin et al., 2019; Brown et al., 2020). Here we extend this idea to training reward models, though the training scale much less than that of LLMs. The result of this work is somewhat unsurpris- ing but encouraging: the broad effectiveness of foundation models can be verified in more research areas, and it shows that models of this kind can be successfully applied in fields that traditionally rely on highly specialized models.
GRAM w/ Domain:
Pre-trained on 100k unlabeled summarization response pairs derived from TL:DR comparison data (Stiennon et al., 2020), followed by fine-tuning on 5k labeled summarization data.
GRAM w/o Domain: Pre-trained on 100k general un- labeled data (excluding summarization-relaa relatte data; specifically, preference data related to summarization is filtered out using 4o)), followed by fine-tuning on 5k labeled summarization data.
The experimental results are listed in Table 2. These results demonstrate that pre-training on data more closely aligned with the target domain leads better performance in that do- main. Specifically, as shown in the table, GRAM pre-trained with domain-specinn data (GRAM w/ Domain) achieves an accuracy of 74.7, significantly outperforming the model without pre-training (RM w/o Pre-training), which achieves only 56.5. This observation aligns with the common practice in L.Mss, where incorporating pre-training typically improves performance on downstream tasks. Furthermore, our results show that the pre-training approach cxhibits strong robustness. Even with significant domain differences, pre-training still contributes positively to performance, with GRAM (71.6) outperforming the non- pre-trained model and the GRAM w/o Domain (67.4).
7. Conclusions
We have explored training methods for reward models utilizing both labeled and unlabeled data. By leverag- ing the generative capabilities of LLMs, we have devel- oped generative foundation reward model, called GRAM. This model undergoes initial training through extensive unsupervised learning, followed by fine-tuning using su- pervised learning methods. Extensive experiments show that GRAM yields large improvements in generalization over various baselines. Our codebase could be found https://githul. www..///iaaan/e........
-speiii data during
Acknowledgments
See more analysis in Appendix D.
This work was supported in part by the National Science Foundation of China (Nos. 62276056 and U24A20334), the Fundamental Research Funds for the Central Univer- sities, the Yunnan Fundamental Research Projects (No. 302401BC070WD
6. Related Work
Reward Modeling. Reward models, trained human preference data, are central to RLHF or other alignment approaches like reject sampling (Lee et al., 2021; Chu et al., 2023). More recently, researchers have extended the use of reward models beyond training and into inference (Wu et al., 2024; Li et al., 2025). Two strands of research have
and the Program of Introducing Tal- ents of Discipline to Universities, Plan 111 (No.B 16009). We would like to thank anonymous reviewers for their valu- able comments. We also thank Hang Zhou for his assistance in open-sourcing the GRAM model series.

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
Impact Statement
Processing Systems 30: Annual Conference on Neural In- formation Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4299-4307, 2017.
This work does not need ethical considerations, as only utilizes open-source foundation models and publicly avail- able datasets. This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Chu, Y., Xu, J., Zhou, X., Yang. Q. Zhang, S., Yan, Z., Zhou, C., and Zhou, J. Qwen-audio: Advancing uni- versal audio understanding via unified large-scale audio- language models. ArXiv preprint, 2023.
Coste, T., Anwar, U., Kirk, R., and Krueger, D. Reward model ensembles help mitigate overoptimization. The Twelfth International Conference on Learning Represen- tations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
References
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assistant as a laboratory for alignment. ArXiv preprint, 2021.
Cui, G., Yuan, L.,, Ding, N., Yao, Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedbckk: Boosting language models with high-quality feedback. ArXiv preprint, 2023.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. ArXiv preprint, 2022.
Devlin, J., Chang, M.- W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, 2019. Association for Computa- tional Linguisties. doi: 10. 1...1165/1//111111 1423.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 1952.
Brauwers, G. and Frasincar, A survey on aspect-based sentiment classification. ACM Computing Surveys, pp. 1-37, 2022.
Dong, G., Yuan, H., Lu, K., Li, C.,, Xue, M., Liu, D., Wang, W., Yuan, Z., Zhou, C., and Zhou, J. How abilities in large language models are affected by supervised fine-tuning data composition. ArXiv preprint, 2023.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,
J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler. D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray. S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Dubois, Y., Li, C. X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpaca- farm: A simulation framework for methods that learn from human feedback. In Oh, A., Naumann, T., Glober- son, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Neural Information Processing
Chen, X., Li, G., Wang. Z., Jin, B., Qian, C., Wang, Y., Wang, H., Zhang, Y., Zhang, D., Zhang, T., et al. Rm- rl: Reward modeling as reasoning. ArXiv preprint, abs/2500.02288,, 2025.
Fernandes, P., Farinhas, A., Rei, R., C. de Souza, G., Ogayo, P., Neubig, G., and Martins, A. Quality-aware decoding for neural machine translation. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V. (eds.), Pro- ceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1396-1412, Seattle, United States, 2022. Association for Computational Lin- guistics. doi: 10. 10.186533w1/2022..nom www.l-mainnl.an.a. 100.
Cheng, P., Xie, J., Bai, K., Dai, Y., and Du, N. Everyone de- serves a reward: Learning customized human preferences. ArXiv preprint, 2023.
Christiano, P. F., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from hu- man preferences. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information
Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.).
10

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 10835-10866. PMLR, 2023.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tun- ing. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural In- formation Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- - 16, 2023, 2023.
Guo, J., Chi, Z., Dong, L.. Dong, Q.. Wu, X., Huang. S., and Wei, F. Reward reasoning model. ArXiv preprint, abs/2505. 14674, 2025.
Lukasik, M., Bhojanapalli, S., Menon, A. K., and Kumar, S. Does label smoothing mitigate label noise? In Pro- ceedings of the 37th International Conference on Ma- chine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 6448 6458. PMLR, 2020.
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y. et al. Rewardbench: Evaluating reward models for language modeling. ArXiv preprint, 2024.
Lee, A., Auli, M., and Ranzato, M. Discriminative rerank- ing for neural machine translation. In Zong, C.,, Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the I 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7250-7264, Online, 2021. Association for Computa- tional Linguistics. doi: 10.
Mahan, D., Van Phung, D., Rafailov, R., Blagden, C., Lile, N., Castricato, L., Fränken, J.-P., Finn, C., and Albalak, A. Generative reward models. ArXiv preprint, 2024.
Min, D. J., Perez-Rosas, V., Resnicow, K., and Mihalcea, R. Dynamic reward adjustment in multi-reward rein- forcement learning for counselor reflection generation. In Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguisties, Language Resources and Evaluation 2024), pp. 5437-5449, Torino, Italia, 2024. ELRA and ICCL.
Lee, H., Phatale, S., Mansoor, H., Lu, K., Mesnard, T., Bishop, C., Carbune, V., and Rastogi, A. Rlaif: Scal- ing reinforcement learning from human feedback with ai feedback. ArXiv preprint, 2023.
Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V., Rastogi, A., and Prakash, S. RLAIF vs. RLHF: scaling reinforcement learning from human feedback with Al feedback. In Forty-first International Conference on Machine Learn- ing, ICML 2024, Vienna, Austria, July 21-27, 2024. Open- Review.net, 2024.
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., and Rajpurkar, P. Founda- tion models for generalist medical artificial intelligence. Nature, 2023.
Müller, R., Kornblith, S., and Hinton, G. E. When does label smoothing help? In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' A'ché-Buue F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. pp. 1696-4700 2019.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of mnstruction-folloiigg models, 2023.
Li, Z., Xu, T., Zhang, Y., Lin, Z., Yu, Y., Sun, R., and Luo. Z. Remax: A simple, effective, and efficient rein- forcement learning method for aligning large language models. In Forty-first International Conference on Ma- chine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenRevie 2024.
Munikar, M., Shakya, S., and Shrestha, A. Fine-grained sentiment classification using bert. In 2019 Artificial In- telligence for Transforming Business and Society (AITB), pp. 1-5, 2019.
OpenAI. Learning to reason with 1mms, 2024.
Li, Z.-Z., Zhang, D., Zhang, M.-L., Zhang. J.. Liu, Z., Yao, Y., Xu, H., Zheng, J., Wang, P.-J., Chen, X., et al. From system 1 to system 2: A survey of reasoning large language models. ArXiv preprint, 2025.
Ouyang. L.. Wu, J.. Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing
Liu, C. Y., Zeng, L., Liu, J., Yan, R., He, J., Wang. C., Yan, S., Liu, Y., and Zhou, Y. Skywork-reward Bag of tricks for reward modeling in lims. ArXiv preprint, 2024.

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Wang, C., Zhou, H., Hu, Y., Huo, Y., Li, B., Liu, T. Xiao, T., and Zhu, J. ESRL: efficient sampling-basee baaeed reinforcement learning for sequence generation. In Wooldridge, M. J., Dy, J. G., and Natarajan, S. (eds.), Thirty-Fighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Appli- cutions of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intel- ligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 19107-19115. AAAI Press, 2024c. doi: 10. (1.0///AA..3111.......
Rafailov, R., Sharma, A., Mitchell, E.,, Manning, C. D.,
Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Inter- pretable preferences via multi-objective reward modeling and mixurre-of-expert. In EMNLP, 2024d.
Ruan, J., Abudula, A., Liu, X., Li, B., Li, Y., Wang, C.,, Fan, Y., Ge, Y., Xiao, T., and Zhu, J. Ndp: Next distribution prediction as a more broad target. ArXiv preprint, 2024.
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., and Sui, Z. Large language models are not fair evaluators. ArXiv preprint. 2023.
Shiwen, T., Liang, Z., Liu, C. Y., Zeng, L., and Liu, Y. Skywork critic model series, 2024.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Process- ing Systems 33: Annual Conference on Neural Informa- tion Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Wang, Y., Yu, 7.,, Yao, W., Zeng, Z.,, Yang, L., Wang, C., Chen, H., Jiang, C., Xie, R., Wang, J., Xie, X., Ye, W., Zhang, S., and Zhang, Y. Pandalm: An automatic evalua- tion benchmark for LLM instruction tuning optimization. The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.nett 2024e.
Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang, Y. An empir- ical analysis of compute-optimal inference for problem- solving with language models. 2024.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, B. Alpaca: A strong, replicable cntructtionfollowiin model. Stanford Center for Research on Foundation Models. https://erfm. stanford. edu/2023/03// 3/alppcc. html, 2023.
Xiao, T. and Zhu, J. Introduction transformers: an nip perspective. ArXiv preprint, abs/2311.176333 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. ArXiv preprint, 2023.
Xiao, T. and Zhu, J. Foundations of large language models. ArXiv preprint, 2025.
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Durme, B. V., Murray, K., and Kim, Y. J. Contrastive preference optimization: Pushing the boundaries of LLM perfor- mance in machine translation. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
Wang, C., Lu, Y., MU, Y., Hu, Y., Xiao, T. and Zhu, J. Improved knowledge distillation for pre-trained lan- guage models via knowledge selection. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Findings of the Association for Computational Linguisties: EMNLP 2022, pp. 6232-6244, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. doi: 10.18653/1120222.mmdmm cerpggs--emlpp.4....
Yang. R., Ding, R., Lin, Y., Zhang, H., and Zhang, T. Regu- larizing hidden states enables learning generalizable re- ward model for 1lms. In Globersons. A., Mackey, L.. Bel- grave, D., Fan, A., Paquet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Wang, C., Gan, Y., Huo, Y., Mu, Y., Yang, M., He, Q., Xiao, T., Zhang, C., Liu, T., Du, Q., et al. Rovrm: A ro- bust visual reward model optimized via auxiliary textual preference data. ArXiv preprint, 2024a.
Wang, C., Zhou, H., Chang, K., Li, B., MU, Y., Xiao, T., Liu, T., and Zhu, J. Hybrid alignment training for large language models. ArXiv preprint, 2024b.
Zhang, L., Hosseini, A., Bansal, H., Kazemi, M., Kumar, A., and Agarwal, R. Generative verifiers: Reward modeling as next-token prediction. ArXiv preprint, 2024.
12

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
Zheng, L.,, Chiang, W., Sheng, Y., Zhuang, S., Wu, Z Zhuang. Y., Lin, Z., Li, Z.,, Li, D., Xing. E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Zilly, G., Achille, A., Censi, A., and Frazzoli, E. On plasticity, invariance, and mutually frozen weights se- quential task learning. In Ranzato, M.. Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021. NeurlPS 2021, December 6-14, 2021, vir- tual, pp. 12386-12399, 2021.
13

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
A. Theoretical Motivations of the Two-Stage Training Method
We divide reward modeling in generative reward models into response understanding and preference generation. To derive a potential formulation for the response understanding optimization objective, we consider the following optimization problem: learning a generative reward model based on features. This loss function can be given by:
(0)
CEp (lp)
(g
(10)
generate preferences based on features
optimize features with preference labels
where gg denotes a small part of the parameters used to generate the final prediction preferences, i.., the final feedforward layer in the LLM. The term of feature optimization (0) is implicitly defined and can be optimized by adjusting the preference generation specified in Eq. 2. The analytical solution of Cgr( is formulated follows:
Ler(
(a.Ma.aaa)) (1, Ya: M6) (xx, ya, yb)l
(11)
where f()) denotes the implicit features and (-) ) is the gold features. While (0) can assist in optimizing Eq. 11, it often requires labeled data, which is challenging to acquire at scale and with high quality. We consider that these features typically aim to provide knowledge that supports preference generation, with interrelationships between responses. Alternatively, we propose to an auxiliary task to optimize these features without relying labeled preferences, using unlabeled data D..- Specifically, we employ conditional probabilities to characterize interrelationships (Dong et al., 2023), simplifying the problem. Additionally, we account for distributional error arising from both Tá and the responses to given input z. This error primarily stems from the possibility that the response may have been sampled from a different model, which is a mismatch between the data distribution and the current LLM. To mitigate this, we introduce an SFT loss as a regularization term. This auxiliary loss function can be written as
(0) = - E Do [log To (yalaz vo) + log To (16lle (z, ya) + log To (z) + log [T(b//z]]
(12)
Since Vo and yb in the unlabeled data do not explicitly distinguish between preferences, simplify the loss function by focusing on the conditional probabilities between the response pair and the input, resulting in the following simplified form:
Cgr(
[log Ne (36122 1. ya) + log To [x)]]
(13)
We provide an equivalent substitution for this original loss. To do so, we consider the joint distribution To (T(a 36 s1]1z), which accounts for the dependency between ya and you Specifically, the joint distribution can be factored as follows:
To (Va. 30]l = TT(va< (1bl2,, Ma)
(14)
We now proceed by applying the log of the joint distribution. Since log To([ya, (/]lar)) is equal to the sum of the individual log-probabilities (by the logarithmic properties of probabilities), we can conclude that:
log no (ya x)) + log (y% (x ya) = log no (Iya, yol(ar)
(15)
Therefore, the original loss function can be optimized by the negative expectation of the log-joint probability:
Let (0) - TE un Dr. (log No (lva: 3o] [x))
(16)
We denote the negative expectation, as defined in Eq. 4, by Lpre and use it for optimization. The objective of Cpre is to maximize the likelihood of generating both responses Yo and yb- Intuitively, in unlabeled data, responses typically exhibit one of three relationships: yo is preferred over yo- Yb is preferred over ye, or ya and yo are roughly equivalent. By learning the mapping from inputs to diverse responses, the model identifies the implicit features that characterize each scenario, thereby gaining general knowledge for response comparison.
In machine learning, it is common practice to use auxiliary tasks for pre- pre-training features that improve task performance. For instance, Devlin et al. (2019) employed next sentence prediction and masked language model tasks to learn features like semantics, which was then used to augment other tasks, such as sentiment classification (Munikar et al., 2019; Brauwers & Frasincar, 2022). Similarly, Liu et al. (2023) utilized an image caption task to train a feature projector, which was then utilized for general visual instruction tasks. To our knowledge, this work is the first to pre-train preference features through an auxiliary task that only leverages unlabeled data.
14

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
B. Details of Experiments
B.1. Settings
SFT Training. During conducting the SFT training, we set the learning rate, batch size, and training epoch to 1e-5, 256, and 2. We did not conduct tuning of these syper-parameters specific the task and the model, as our experiments with other hyper-parameters did not yield significant performance improvement.
Reward Model Training. We trained the discriminative and generative reward model baselines for epoch with learning rate of le-5 and batch size of 256. In the Label Smoothing baseline for the generative reward model, we set the smoothing factor to 0.1 for fair comparison, consistent with the value used in the label smoothing strategy. In the Freeze baseline, we fixed the embedding parameters while training the generative reward model. Additionally, we fixed the base model's features and only fine-tuned the nonlinear components during the training of the discriminative reward model. In the Regularization baseline for discriminative reward model, we incorporated the SFT loss function, as proposed by Yang al. (2024), into Eq. In the testing of GRAM, we train both the proxy reward model and goal reward model using the LLaMA-3.1 1-8B----Bttuuuue model, employing discriminative framework for training. Also, when adapting GRAM through fine-tuning, we set the learning rate to le-5 and the size of epochs two. Furthermore, during the training of the generative reward model, we utilized the input structure depicted in Figure 7.
Please act as impartial judge and evaluate the quality of the responses provided by two Al assistants to user question displayed below. You should choose the assistant that follows the user's instructions and answers the user's questions better. Your evaluation should consider factors such the helpfulness, relevance, accuracy. depth, creativity, and level of detail of their responses. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be objective possible. Please directly output your final verdict by strictly following this format: [[A]] if assistant A is better, [[B]] if assistant B is better.
[User Question] {input)
x
[The Start of Assistant A's Answer] [Response
Ya
[The End of Assistant A's Answer]
[The Start of Assistant B's Answer] RessponseB)
Ub
[The End Assistant B's Answer]
Preferred:
Figure 7: An example of input for the generative reward model as defined in Eq. 2.
Best-of-n Sampling. When conducting the best-of-n sampling the AlpacaEval2 benchmark, we generated candidate responses using the top-p sampling approach, setting to 0.95 and temperature to 0.75, Then, we picked a final generated response with the maximum reward score.
PPO Fine-tuning. We trained the LLM using PPO via the trlx implementation'. For all experiments, the learning rate was set to le-5 and 5e-6 for the policy model and the value model, respectively. We settled on a batch size of 64 for each PPO step, which consisted of epoch of gradient steps and 4 epochs of mini-batch PPO steps. When using GRAM
thttps:/ thub. com wtt.:///paaaa///ee.e.emmm
15

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
D D-Baseline
D-Froere
Regulaaricaionn
G- G-Baseline
G-Frevze
G-Label Smoothing
GRAM
2.0
1.9
Oraale Score
Prony Score
1.7
1.7
1.6
1.5
06 24 4k 69 St 106
12A 144
16k 28
0R
26
4k
66
88
10%
125 14k,
161 18L 201
08
28
4G
6K SK
14L, Training Samples
10L
12%
16k, 181 2016
06
24
42
10L
12L 142 FAK 1SK 20
Training Samples
Training Samples
Training Samples
(LL.MM-3.1 3.1-8BBB.BBBBB- (8-Instruct)
(b) (LLaMA- 3.2-3B (n-ttunst)
Figure 8: Results on Reinforcement Learning
compute reward scores, this optimization objective is then defined as:
LPPO = -Pgg--ePo. l7 x r (1, ú)) - a x DEL (ū)))))) (ú]=)
(17)
where 7 denotes a scaling factor, DpPO denotes the data for PPO fine-tuning, and Th, denote reference LLM. Here, we set to 10. Additionally, to address the over-optimization issue as described in Gao et al. (2023)'s work, we implemented a strategy that saved checkpoints at regular intervals during the training process. Specifically, we evaluated checkpoints at intervals of 200 steps for all tasks against their respective validation sets and selected the optimal checkpoint with the best reward score. Following Wang et al. (2024b), we also employed a cold-start trick for PPO to alleviate the damage caused by the inaceurate estimation of the early value model. Specifically, we only updated the value model and did not update the policy model during the first 30 steps of PPO training. Following Wang et al. (2024c)'s 's work, we also standardized our reward scores using reward queue, which stored the previous 1k reward scores compute the mean and variance. All of our experiments were done on eight A800 GPUs
B.2. Evaluation
To evaluate the generalization of GRAM, conducted tests using OOD test sets, including HHH--lignmeet (Askell et al., 2021), and RewardBench (Lambert et al., 2024). The HHHH-Alggmeet evaluates language models regarding their helpfulness, harmlessness, and honesty. The test set focuses on the preference evaluation of the specific task. Furthermore, RewardBench is a newly introduced benchmark designed to evaluate the performance of reward models across various dimensions, including chat, reasoning, and safety.
C. Additional Experimental Results
C.1. Reinforcement Learning
In reinforcement learning, the reward score is computed for single input-response pair (x, f), where ý is sampled from the model. In this process, we select Vref arg max TT(-|ar) by greedily sampling the response.
Task Setups. To assess the performance of GRAM in reinforcement learning, we conducted PPO fine-tuning experiments using the Alpaca dataset (Taori et al., 2023), which includes 52k training samples. We used the data splits provided by AlpacaFarm (Dubois et al., 2023) in performing SFT and PPO fine-tuning, Note that we selected the LLaMA-3.1 model as the policy model, as the SFT and RLHF processes for the LLaMA-3.1-8BB I-8B-Instruct model had not been publicly disclosed, leading to a data shift that is incompatible with our dataset.
Results on PPO Fine-tuning. We apply GRAM and its baselines as reward models in PPO fine-tuning. As shown in Figure 8, reinforcement learning shows greater tendency to exploit the learned reward models than BoN. The oracle scores of the baseline methods begin to decline early in training, while their proxy scores rise, highlighting a clear overoptimization issue. In contrast, GRAM exhibits better generalization capability, as evidenced by the increase in the oracle score alongside rising proxy scores. This confirms that GRAM effectively mitigates overoptimization in PPO fine-tuning. Additionally, we
16

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
Method
WinRate
LC-WinRatd
SFT
4.56
3.08
PPO Fine-tuning
+ D-Regularizztion
5.32
4.13
+ G- Basellll
6.75
5.80
+ G-Freeze
8.01
7.82
G-Label Smoothing
8.82
7.94
GRAM (Ours)
12.63
11.12
Table 3: Win rate of models after PPO fine-tuning with GRAM and strong baselines trained with the model. "WinRate" and "LC- CC-WinRate" denote win rate and length-controlled win rate, respectively.
observe that generative models generally outperform discriminative models. Beyond their strong generalization ability, we hypothesize that using the reward of "how much better than the reference output" is more effective than focusing on the output quality itself. This observation can be aligned with Li et al. (2024)'s work, which shows that such reward schemes lead to more stable training during the PPO process.
C.2. Performance on PPO Fine-tuning with Different Reward Models
In addition to evaluating proxy and oracle reward scores, we compute the win rate of models after PPO fine-tuning with GRAM, along with its strong baselines: D-RReullariztion,, G-Baseline, G-Freeze, and G-Label Smoothing. Evaluation conducted using alpaca_eval system', with GPT-4 serving proxy for human evaluation of response quality and the baseline system. As shown in Table 3, the results demonstrate that GRAM outperforms the baselines, serving more effective reward model to enhance PPO training.
C.3. Comparing GRAM with Methods for Enhanced Generalization
observe that the freezing method improve generalization, but this improvement is not always consistent. For example, the RewardBench, freezing parameters can enhance performance for the 8B reward model (e.g., obtaiiing +2.6 points), but similar gains are not observed for the 3B model. A similar trend is evident across other experiments. We hypothesize that the specific parameters frozen play a crucial role, and freezing the embedding layer in the 3B model may not be optimal. Additionally, we note a clear drawback of freezing parameters: can degrade the reward model's s performance on the ID) test set. For example, freezing parameters in the 8B model results in a loss of 1.4 points on the unified feedback test set. We also observe that the standard label smoothing method slightly boosts generalization. In contrast, by incorporating a pre-training process unlabeled data, GRAM can gain general knowledge for comparing responses while preserving generalization, demonstrating strong performance with ID and OOD test sets.
In Table 1,
D. Analysis
D.I. Ablation Study on Two-Stage Training
As shown in Table 4, we design four GRAM variants further to elucidate the functionality of the two-stage training method.
First Stage
Variant
Second Stage
Description
log no (v)
log me no(ubl H.)
GRAM-v1
x
X
V
Using only the second stage, without pre-training via the unlabeled data. Using only the first stage, without fine-tuning via the laheled data. Excluding the update yo the stage. Excluding the update of ys in the first stage.
GRAM-v2
V
X
G3AM-v3
GRAM-v4
V
Table 4: Deseription GRAM variants.
We conduct experiments on pair-wise response ranking and reward model adaptation to evaluate the performance of these
http:///ggttuu...ootttttttttt/ la wtt.c//m//tattttaaaaaaaaaaeeeeee evai
17

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
90.0
75.0
85.1
84.3
83.2
70.0
80.0
Accuracy (%)
Accuracy (%)
74.1
65.0
RM line-tuned with GRAM
70.2
70.0
RM line-tuned with GRAM-v1
60.0
RM fine-tuned with GRAM-12
RM fine tuned with GRAM-v3
60.0
RM fine-tuned with GRAM-v4
55.0
RM fine tuned with G Baxeline
50.0
50.0
GRAM
GRAM-Ví
GRAM-12
GRAM-13
GRAM-v4
D-Baseline
G-Baseline
0k
Sk
7k
10k
Amount of Preference Data
(a) Performance on Pair-wise Response Ranking
(b) Performance on Reward Model Adaptation
Figure 9: We evaluate different GRAM variants based on pair-wise response ranking (RewardBench) and reward model
adaptation (Summarization).
GRAM variants. Note that since GRAM-v2 has not undergone supervised fine-tuning, it does not always follow instructions effectively in pair-wise response ranking, often generating either preference 'A' or 'B', To address this, we use suffix to force it to do so, i.e., appending #Preferred to the input.
The results are summarized in Figure 9. The results show that our two-stage training significantly improves the performance of GRAM. Notably, removing the first stage (GRAM-v1) leads to a substantial performance decline, c.g., losing 3.9 points on the RewardBench. Interestingly, we find that removing the second stage (GRAM-v2), i.e., fine-tuning solely with unlabeled data, still allows GRAM to achieve comparable performance in pair-wise response ranking and reward model adaptation. Additionally, the performance of GRAM-v3 and GRAM-y4 demonstrates the importance of optimizing with the terms log Re (ya|2) and log me (ys|z,, ya). Notably, compared to GRAM, GRAM-v4 experiences a 1.9 points drop on the RewardBench, with this decline being particularly pronounced in reward model adaptation. Compared variants of GRAM, we can conclude that: (1) pre-training on large-scale unlabeled data can gain general knowledge for response comparison, and (2) both terms, log (yo (z) and log To (,,,,, Va), are crucial for learning the knowledge during pre-training.
D.2. Derivation of Label Smoothing
Our label smoothing strategy optimizes the generative reward model based on regularized Bradley-Terry model. To illustrate this, we use a sample with label 'A' in the following derivation:
- -[1 t) log (.(s) + Zo(s)
(x)
Zole)
(s)(s)
llog (z) + eZa(s)
- (1 c)- [lo 2o -lg(e + E (
- [loge log(( (0())]
c) (s()) log(e + =2((o)) -c [Z((s) log(e2a()) + c20(0)) - (Zo(s) (s)) + log(e Zo(0) + zs(o)) log (c(s)
2o)))
e. (2.(s) - (x)) log 1 -
log σ (s) Zo(s)) c (z) Zs(s))
(18)
Bradley- Terry model
regularization term
The final derived form shows that this label smoothing essentially guides the model to learn preferences by optimizing Bradley-Terry model with additional regularization.
We consider that this regularization mitigates overfitting in the generative reward model, thereby enhancing its generalization capability. To verify this, we conduct experiments with varying values of e on ID and OOD test sets. From Figure 10 (a) (ID Unified-Feeddaace we notice that performance improves the value of increases from 0 to 0, after which
18

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
LLaMA-3.1 --8-3-8-8--nntrect
LLaMA-3.2-33 3.-MA--32--B-Instruuc
72.0
Accuracy (%)
Accuracy (%)
70.0
68.0
66.0
76.0
0.01
0.05
0.1
0.2
0.3
0.4
0
0.01
0.05
0.1
0.2
0.3
0.4
Smoothing Factor (6)
Smoothing Factor (c)
(a) (Unifeed-Feedback)
(b) OOD (RewardBench)
Figure 10: Performance of the GRAM trained by label smoothing with different Here, 0 denotes that do label smoothing during training our reward models.
Method
UNIFEED
REWARDBENCH
GRAM w/o Label Smoothing
68.4
82.1
GRAM w/ Our Label Smoothing
70.6
83.6
GRAM w/ Vanilla Label Smoothing
69.1
82.5
G- fi-Bselline w/ Our Label Smoothing
66.7
80.2
Gi-asselnee w/ Vanilla Label Smoothing
66.2
79.0
Table 5: Accuracies (%) of generative reward models with our label smoothing and the vanilla label smoothing.
starts decline slightly. The model achieves its best performance around 0.01, with accuracy peaking 71.9. On the other hand, in Figure 10 (b) (OOD RewardBench), observe more pronounced effect of the regularization term. As increases, performance improves significantly. At 0.1, the model reaches an accuracy of 85.1, demonstrating that the regularization helps reduce overfitting and enhances the model's ability to generalize to unseen data. For smaller values of e (e.g., 0.01), the performance on the OOD test set drops, highlighting the importance of the regularization term in ensuring robust generalization. However, when e exceeds 0.1, we see a negative impact on performance for both ID and OOD tests, suggesting that too much regularization may hinder the model's ability learn effectively. Based on our results, select 0.1 the optimal value for experiments, balancing both ID and OOD performance.
Additionally, as listed in Table 5, we have conducted further experiments on the LLaMA-3.2-3 this in greater detail. As the experimental results show, our label smoothing leads to better reward modeling compared to vanilla label smoothing. These results also show the role of label smoothing in training GRAM.
model to investigate
19

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
D.Freese
D Regularizzzton
Smoothing
fine-tuned
Accuracy Homnlesnne
Accuncy (%0 co Summesizaion
Accuracy Hamlleane
70
65
65
6.5
60
60
55
60
60
55
00
11.
11
36.
10k.
01.
5L
7k
10k.
04
18
36
5L
76
10k.
06.
14
31
5L.
10k
Amount of Preference
Amount Preference Data
Amount Preference Data
Amount of Preference Data
(a) Adaptation (LLaMA-3.1--8) Insttructttt
Adaptation (LLaMA-332-3B-B)
Figure 11: The performance of reward models fine-tuned with our baselines, including D-Freeze, D-Regularization, G- Freeze, and G-Label Smoothing. The results show that GRAM surpasses these baselines in adapting to specific tasks, highlighting its superior generalization capability.
Input
What are the lyrics to the third verse of Rick Astley's 's Never Gonna Give You Up?
Here are the lyrics to the third verse of Never Gonna Give You Up by Rick Astley: Never gonna make you cry.
SFT
The third verse Rick Astley's "Never Gonna Give You Up" goes as follows: I'm never gonna give, never gonna give Give you up, never gonna give, never gonna give you up I'm never gonna give, never gonna give Give you up, never gonna give, never gonna give you up I'm never gonna give, never gonna give Give you up. never gonna give...
D-Baseline
Sorry, 1 can't provide the lyrics to the third verse of "Never Gonna Give You Up" as it is copyrighted copyrighted copyrighted copyrighted copyrighted...
D Regularrizztion
The third verse of Rick Astley's "Never Gonna Give You Up" is: "Inside we both know what's on your mind".
G-Baseline
G-Freeze
The actual third verse Rick Astley's song "Never Gonna Give You Up" is is is
Here is the third verse of Rick Astley's "Never Gonna Give You Up": And if you'l re ever feeling down Just remember what I told ya Good things get better all the time So keep on trying, keep on shining Keep on trying, keep on shining Keep trying, keep on shining
G-Label Smoothing
The third verse of Rick Astley's s "Never Gonna Give You Up" goes like this: "We've 've known each other for so long Your heart's been aching, but you't re too shy to say Inside we both know what's been going on We know the game and we'r gonna play it just wanna tell you how I'm feeling Gotta get you alone"
GRAM
Table 6: A case in the PPO fine-tuning experiments. GRAM optimizes better LLM aligned with human intention, while other baseline reward models produce inaccurate or incomplete lyrics that can be more easily exploited by PPO.
20

---

GRAM: A Generative Foundation Reward Model for Reward Generalization
Input
Is it true that the native Australians were really prisoners of England?
Yes, it is true that the native Australians were really prisoners of England. The British government transported thousands of Aboriginal and Torres Strait Islander people to Australia as prisoners between the 1780s and the 1860s.
SFT
true that the native Australians were once treated as prisoners by England. Between the 1780s and the 1860s, the British government transported thousands of Aboriginal and Torres Strait Islander people to Australia convicts.
D-Baseline
It is indced true that the Indigenous people of Australia treated prisoners by the British. Between the 1780s and the 1860s, the British government transported thousands of Aboriginal and Torres Strait Islander people to Australia convicts. These individuals were often sentenced transportation for relatively minor crimes, like stealing food livestock, as well more serious offenses, such murder. Many of these prisoners were forced to work on farms and plantations, while others were used as laborers for the construction of roads and buildings.
D-Regularization
The relationship between the British and the indigenous Australians, also known as Aboriginal and Torres Strait Islander peoples, is complex and multifaceted. While it's not accurate to say that all native Australians were "prisoners" of England, the British did establish system of colonization and control over the continent that had significant impacts on the indigenous population.
G-Baseline
It's important to clarify that the Indigenous peoples of Australia, including Aboriginal and Torres Strait Islanders, were not originally prisoners of England. The British colonization of Australia began in 1788, and it was during this period that the British started transporting convicts (many of whom were from England and other parts of the British Empire) to Australia. The confusion might come from the fact that when the British arrived in Australia, they viewed the land as "terra nullius" (land belonging to no one), and they imposed their own systems, laws, and control.
G-Freeze
Yes, it is true that the native Australians were treated as prisoners and subjected to forced labor, relocation, and violence by the British colonizers. When the British established their penal colony Australia 1788, they brought with them a system of transportation of convicts, which included the forced relocation of Aboriginal people from their lands make way for British settlement.
G-Label Smoothing
The statement that native Australians were prisoners of England is inaccurate. Indigenous Australians, including Aboriginal Australians and Torres Strait Islanders, have history that stretches back least 65,000 years. They are the original inhabitants of the Australian continent, with a rich and complex cultural heritage that predates European colonization by tens of thousands of years. The confusion may arise from the history of European settlement in Australia.
GRAM
Table 7: A case in the PPO fine-tuning experiments. GRAM provides the most accurate and contextually relevant response, emphasizing the long history of Indigenous Australians. contrast, other models offer less precise or overly simplified explanations.
21


Reviews:
[{"review_id": "eXVmwNGbqA", "reviewer": "Reviewer_3tRc", "summary": "This work introduces GRAM, a generative foundation reward model for aligning LLMs with human preferences. Unlike conventional reward models that rely only on labeled human preference data, GRAM incorporates both labeled and unlabeled data through a two-stage training process: unsupervised “pre-training” on input-response pairs, followed by fine-tuning with human preference data for task-specific alignment. The authors demonstrate that the incorporation of label smoothing unifies generative and discriminative reward models under a shared training objective. The authors demonstrate the effectiveness of their proposed approach by first pre-training on 400,000 examples Extensive evaluations across response ranking, reinforcement learning from human feedback (RLHF), and task adaptation show that GRAM generalizes effectively across tasks, achieving improvements over baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3tRc,\n\nWe would like to thank the reviewer for the positive feedback regarding the novelty of pre-training for reward models and the strong results.  \nBelow, we explain the main points you are concerned about.\n\n---\n\n>*One component that does not make sense for this problem is the pre-training stage.*\n\nWe would like to clarify the motivation behind our pre-training design. As discussed in Section Appendix A, we can understand our method from a feature learning perspective. In generative reward modeling, the loss function can be given by\n$$\nL\\_{\\mathrm{g}}(\\theta) = L\\_{\\mathrm{gp}}(\\theta\\_{\\mathrm{gp}})+L\\_{\\mathrm{gf}}(\\theta)\n$$\nThe feature optimization term $L\\_{\\mathrm{gf}}(\\theta)$ is implicitly defined and optimized by adjusting preference generation. Traditional generative reward modeling optimizes these objectives using costly preference labels. This work explores a pre-training approach to optimize $L\\_{\\mathrm{gf}}(\\theta)$ first.\n\nTo this end, we propose using an auxiliary task to optimize these features without relying on labeled preferences. Specifically, we utilize conditional probabilities to characterize interrelationships, i.e., $-\\log \\pi\\_{\\theta}(y\\_{b}|x,y_a)$. Besides, considering a distributional error arising from both $\\pi\\_{\\theta}$ and the $y_a$ or $y_b$ to a given input $x$, we introduce an SFT loss as a regularization term, i.e., $-\\log \\pi\\_{\\theta}(y|x)$. After careful derivation, we obtain\n$$\nL\\_{\\mathrm{gf}}(\\theta) = -\\mathbb{E}\\_{(x, y_a, y_b) \\sim D_u} [ \\log \\pi_\\theta([y_a, y_b] | x) ]\n$$\nThis predicts two responses. The model can better generalize and improve its reward capability by focusing on learning meaningful features rather than surface-level ones (e.g., length). Also, this pre-training strategy has been highlighted to make sense by Reviewer Jdyw. Thanks for the feedback!\n\n>*What is the purpose of comparing with a proxy model trained on less data? Which model is used as the policy for generating ranked responses?*\n\nTraining a proxy model tests the method's modeling capability. A good proxy score indicates strong performance on ID, reflecting good modeling. Our goal is to design a method with both strong modeling and generalization. Thus, testing with the proxy score is crucial. This setup is also supported by Yang et al. (2024).\n\nAs for the sampling policy, we use the LLaMA-3.1-8B model, as described in Section 4.4 (line 319).\n\n>*W1: In Table 1, what are the methods denoted (baseline)?*\n\nOur main baselines include several methods (such as Freeze and Regularization) aimed at enhancing generalization and discriminative models to demonstrate the promising generalization of our GRAM.\n\n>*W2&Q4: Lack of detailed description on the baseline version of label smoothing (LS).*\n\nThe LS we use differs from the standard LS. Our method applies LS to all candidate label tokens, as described in Section 3.3, whereas the traditional method applies it to all tokens in the vocabulary. We theoretically demonstrate that our method is more effective, potentially optimizing with a constrained Bradley-Terry model. Apart from the explanation in Appendix D.2, we have conducted further experiments on the LLaMA-3.2-3B-Instruct to explore this in more detail below.\n\n|Method|UniFeed(ID)|RewardBench(OOD)|\n|:-|:-:|:-:|\n|GRAM w/o LS|68.4|82.1|\n|GRAM w/ Our LS|**70.6**|**83.6**|\n|GRAM w/ Standard LS|69.1|82.5|\n|G-Baseline w/ Our LS|66.7|80.2|\n|G-Baseline w/ Standard LS|66.2|79.0|\n\nAs the experimental results show, our LS leads to better reward modeling compared to the standard LS. These results also show the role of LS in training GRAM. We promise to add more discussions on this in the revised version.\n\n>*Q1&Q2: Are there two 400,000-example splits, one for pre-training and one for fine-tuning? Can we collect pre-training data by sampling from LLaMA models?*\n\nYes, both splits of 400,000 examples are randomly selected from the Unified Feedback dataset. The pre-training responses also come from Unified Feedback, but we only use the responses without their preference labels to simulate unlabeled responses. Sampling responses from LLaMA to obtain unlabeled data is a good idea, and we commit to exploring this method further. This work focused on the time-consuming nature of sampling large-scale labeled response pairs. Also, we expect such available data to exist for direct training in real-world scenarios. Thus, we randomly select response pairs from Unified Feedback to simulate available labeled response pairs.\n\n>*Q3: This scoring by Eq. 3 may yield vastly different scores depending on the reference response.*\n\nThis setup is reasonable. In BoN sampling, we keep the same reference for all candidate outputs, ensuring the scores are comparable. In RL, the recent work *Remax* uses the difference from a reference as the reward and interprets its reasonableness from a reward baseline perspective.\n\n--- \nWe sincerely thank you for your positive feedback on our paper!\n\nBest,\n\nAuthors", "claims_and_evidence": "The claims about GRAMs performance are supported by their experiments. In particular, the method leads to a reward model that can generalize to out-of-distribution data, and can be used in multiple settings (as both a pairwise reward model and list-wise reward model). Some of the experimentation details are not clear, which, if improved, will make the evidence even more convincing.", "methods_and_evaluation_criteria": "One component of GRAM that does not make sense for this problem is the pre-training stage, where the model learns to generate 2 responses to a prompt. Theoretically, it’s not clear to me why the unsupervised learning portion should be effective (although the experiments show it to be, specifically in Section 5.1). In particular, given that the reward model is never asked to generate responses, and the responses at test time are likely to come from a different distribution than the training data, the reason this is beneficial is unclear to me. This is one area that could use some explanation or exploration.\n\nThe evaluations make sense for the problem at hand though.", "theoretical_claims": "I did not verify the derivation of equation 8.", "experimental_designs_or_analyses": "- The experimental design stands to be improved slightly. For example, the authors describe a version of label smoothing used for their method, while the ablation on baselines uses a different version of label smoothing. This difference of methods should either be explained, or rectified to use the same version of label smoothing.\n- Additionally, some of the details in the experiment on list-wise response ranking are unclear. What is the purpose of comparing with a proxy model that is trained on less data? Additionally, which model is used as the policy to generate the responses being ranked?", "relation_to_broader_scientific_literature": "I think the use of generative reward models has become a very popular idea in the past 6-9 months and the community may appreciate this work. In particular, the idea of performing unsupervised training specifically for reward modelling is novel, however, I think the exact setup of unsupervised learning is not well motivated and unclear.", "essential_references_not_discussed": "The following works both propose a variant of a generative reward model. The main difference that I can see with what is done here is that both these works allow the model to reason over the 2 preference options prior to providing its judgement. They are similar enough to the current work that readers should know that they exist, as they can provide a direction for future works.\n- Ankner et al., 2024. Critique-out-Loud Reward Models\n- Mahan et al., 2024. Generative Reward Models", "other_strengths_and_weaknesses": "Strengths:\n- The results are quite good compared to baselines.\n- This version of pre-training for reward modeling is quite novel, to my knowledge.\nWeaknesses:\n- The experimental setup is unclear. For example, in Table 1, what are the methods denoted (baseline)?\n- The main set of experiments seem to use different versions of label smoothing for the baseline vs. the full method. Because we only see the label smoothing + pretraining together, this makes it hard to determine the effects of each individual component", "other_comments_or_suggestions": "Some of the experimental details need to be clarified. See the comments in “Experimental Designs Or Analyses” and “Questions for Authors”.", "questions_for_authors": "- Some aspects of the experimental setup are unclear to me. Are there 2 splits of 400,000 examples? One split used for pre-training and the other for fine-tuning?\n- Where do the pretraining responses come from (y1 and y2)? Given that the training dataset contains pairwise examples, do you use those for the pre-training stage, or do you actually generate the responses with the models you will train (Llama-3.1-8B and Llama-3.2-3B)? In general, are the responses that are used for pre-training generated by the same model that you will then train with that data?\n- Starting at line 126, “When applying this model to score a new input-response”, you suggest that you generate a reference response with the current LLM. I am unfamiliar with this method of scoring a single response. Has this method previously been used in other works? In particular, one concern I have with this method is that it may lead to vastly different scores depending on which LLM generates the reference response.\n- Why is the “label smoothing” applied to the models in your experiments different from the method that you proposed in Section 3.3?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U1OLeCKUG6", "reviewer": "Reviewer_WL65", "summary": "Authors propose improvements on the training of generative reward models (GenRMs). First, they pre-train GenRMs on pairs of responses. Second, they apply label smoothing. This approach is called GRAM. Authors also make an observation that label smoothing shall be understood as the regularization of Bradley-Terry scores. Authors experimentally validate their approach with UnifiedFeedback and Llama-3.1-8B and 3.2-3B instruct models. When trained on the same 400k preference data, proposed GRAM approach outperforms generative and discriminative RM baselines. When applied to Best-of-n sampling, GRAM is also shown to generalize better on AlpacaEval. When fine-tuned on specific domains (summarization and helpfulness), GRAM is also more sample-efficient than baseline methods.\n\n## update after author rebuttal\n\nI will maintain my score. Authors answered my clarifying questions very clearly, but I expected them to be answered in my original score. I still feel this is a solidly executed yet somewhat incremental paper, hence I would be happy to see it accepted, yet open to change my opinion.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer WL65,\n\nWe appreciate the reviewer’s constructive and thoughtful feedback.  \nWe appreciate your recognition that the main claim of our paper is \"empirically well supported\", and that our experiments are \"comprehensive, meticulously studying each of the design choices\".\n\nWe provide explanations for the main points that you are concerned about.\n\n---\n\n>*W1: The technical contribution seems simple—pre-training and label smoothing—both well-established in ML. The contribution is empirical and may be more suited for NLP conferences than ML.*\n\nThanks for your insightful comment! As you said \"pre-training and label smoothing are well-established concepts in ML\", we would like to clarify that our contribution is not about innovating them, but rather introducing these concepts to achieve superior reward modeling. The rationale behind this is as follows:\n\n- While applying reward models to align LLMs is a compelling direction, training these models still heavily relies on labeled data. We expect to enhance reward modeling by pre-training on unlabeled data to reduce dependence on labeled data. However, this pre-training approach has never been explored and presents significant challenges. The difficulty arises from that, unlike self-supervised approaches, systems cannot directly generate their own supervision signals from text for training reward models. In this work, we thoroughly explore the use of unlabeled data in reward modeling and propose a specific pre-training reward model procedure. We also discuss its effectiveness from both theoretical and practical perspectives.\n\n- Although label smoothing has been shown to be effective in many tasks, it has not been well-established in reward modeling. In this work, we theoretically demonstrate that the training objective of generative reward models can be reformulated into a more elegant form: we are essentially optimizing the Bradley-Terry model with modified label smoothing. This result is significant as it establishes a connection between discriminative and generative reward modeling methods—both of which fundamentally train LLMs to perform pairwise ranking, thereby pushing the boundaries of reward modeling and directly contributing to improved generalization.\n\n>*Q1: On which dataset do authors run their stage 1 (task 1 pre-training)? The similar idea of pre-training reward models is mentioned in Bai et al. (2022).*\n\nThe pre-training responses also come from Unified Feedback. Please note that while this data includes preference labels, we do not use these labels in our pre-training process. Instead, we only use the response pairs to simulate unlabeled data and validate the effectiveness of our method. \n\nSince our pre-training approach only uses responses for unsupervised training (which can be directly sampled from LLMs), it allows for easier scalability. We also present two key insights from our experiments for pre-training. First, as shown in Figure 5, during downstream adaptation, our pre-training effectively achieves generalization and reduces the need for specific labeled data. For example, in the summarization task, we achieve performance comparable to training from scratch with approximately 100k labeled data using only 5k data points. Second, as demonstrated in Section 5.1, we find that the more unlabeled data used in training, the greater the benefit to downstream performance, regardless of whether the downstream labeled data is small or large. Building on these insights, in real-world reward model training, we could first collect many responses using cost-effective methods, such as sampling from LLMs, and then fine-tune with a small amount of labeled preference data.\n\nIn contrast, Bai et al. (2022) propose a pre-training approach that uses large, labeled preference data sets for supervised training, followed by domain-specific fine-tuning. This method is difficult to scale as it still relies heavily on labeled preference data, which is often scarce in real-world scenarios. \n\nThank you for your helpful suggestion! In the revised version, we promise to provide a more detailed description and experiment-based analysis for your concern.\n\n>*Q2: In Figure 5, do RM fine-tuned with G/D-baselines mean RMs trained on UnifiedFeedback?*\n\nYes, the data used to fine-tune the RMs for forming the G/D-baselines comes from UnifiedFeedback. Note that this data is the same as the one used by GRAM in the fine-tuning stage, which includes preference annotations.\n\n>*S1: Insufficient description of related work and errors in reference presentation.*\n\nThank you for your helpful feedback! We promise to correct the description and presentation of the related work in the revised version.\n\n>*S2: Essential references are not discussed.*\n\nThanks for your valuable suggestion! We promise to include the discussion of the essential references in the revised version.\n\n---\nWe sincerely appreciate your positive feedback and thank you again for your time.\n\nBest,\n\nAuthors", "claims_and_evidence": "The main claim of this paper is that GRAM generalizes better than previous reward modeling approaches. Across three different use-cases of RMs - preference ranking, best-of-n sampling, task-specific fine-tuning - the sample efficiency of GRAM is consistently shown. Therefore, the claim is empirically well supported.", "methods_and_evaluation_criteria": "The proposed two extensions of GenRM are standard modeling techniques and hence they are appropriate. The first approach of pre-training on response pairs can be understood as continued pre-training on the same domain of unlabeled data; this is a standard, well-established technique originating from AdaptaBERT https://aclanthology.org/D19-1433/ . The second, application of label smoothing is also standard in the training of classification models (from Szgedy et al https://arxiv.org/abs/1512.00567 ) and already explored in reward modeling context in the AlpacaFarm paper https://arxiv.org/abs/2305.14387 . Also, the concept of reward model pretraining was considered in Bai et al (2022) https://arxiv.org/abs/2204.05862 although in a different form. Hence these techniques are all well-established, principled methods.\n\nThe evaluation criteria are also standard. RewardBench is well-established benchmark for evaluating generic-purpose reward models, although there are new ones such as RM-Bench, JudgeBench, FollowBenchEval. See Saha et al. https://arxiv.org/abs/2501.18099 for their experimental setting. Evaluation of Best-of-N on AlpacaEval is acceptable but the ideal setup would be to use the standard GPT-based evaluator from AlpacaEval itself rather than using a proxy reward model. Fine-tuning experiments also use standard domain-specific preference datasets, which is good.", "theoretical_claims": "I checked the correctness of the regularization characterization of label smoothing for Bradley-Terry models. I checked steps in Appendix D.2, which are straightforward algebra.", "experimental_designs_or_analyses": "I checked experimental designs and analyses for all experiments in the main paper and Appendix Figure 9, 10. As I discussed in 'Methods And Evaluation Criteria', there is some room for improvements, but they are not critical.", "relation_to_broader_scientific_literature": "Connections to previous work in literature I already made in 'Methods And Evaluation Criteria' section are notable. The idea of pre-training reward models was explored in Bai et al (2022) https://arxiv.org/abs/2204.05862, but they focused on sourcing preference data from web. Authors' approach has the benefit of leveraging unlabeled data. However, in the experiments, I believe they pre-train only on UnifiedFeedback data, which are labeled.\n\nThis approach can also be understood as a form of unsupervised domain adaptation https://aclanthology.org/D19-1433/ , and the connection may foster other ideas, techniques, and theory from domain adaptation to apply to reward modeling.", "essential_references_not_discussed": "AlpacaFarm paper https://arxiv.org/abs/2305.14387 is cited but their use of label smoothing was not discussed. AlpacaFarm paper considered it as synthetic label noise.\n\nLabel smoothing wasn't cited. Although it's well-established, it's worth citation: Szegedy et al https://arxiv.org/abs/1512.00567 \n\nGenerative Reward Models (Mahan et al, 2024 https://arxiv.org/abs/2410.12832) paper has already made an observation that GenRMs generalize better on out-of-domain. I understand the paper was rejected from ICLR but it is worth citation and discussion. Just to be sure, I didn't discount the contribution of authors' paper because of Mahan et al because it's not yet accepted.", "other_strengths_and_weaknesses": "Experimental analyses are very comprehensive, meticulously studying each of the design choices, for example Table 3/Figure 9 and also studying scaling aspects (Figure 6). The strength of the proposed method is also consistent across experiments.\n\nOn the other hand, I find technical contribution of the paper is a bit simple - pre-training on pairs of responses and label smoothing. These are already very well-established concepts in machine learning, hence the contribution of this paper is mostly empirical, and maybe better suited at NLP conferences than ML conferences.", "other_comments_or_suggestions": "- Line 047 column 2: the systems cannot directly generate from text their own supervision signals -> There actually are papers on this type of approach, hence the statement is a bit unfair to them. For ex Self-Rewarding Language Models https://arxiv.org/abs/2401.10020 .\n- Line 028 column 2: Lee et al reference doesn't have year information", "questions_for_authors": "On which dataset do authors run their stage 1 (task 1 pre-training)? I suspect they just used responses from UnifiedFeedback but I wasn't able to find an explicit discussion of the data used for stage 1 training.\n\nIn Figure 5, do RM fine-tuned with G/D-baselines mean RMs trained on UnifiedFeedback?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xKNKS6AqO8", "reviewer": "Reviewer_pUcN", "summary": "This paper proposes an interesting reward model training method using both unlabeled and labeled data. Building on the generative models in LLMs, the authors develop a generative reward model that is first trained via large-scale unsupervised learning and then fine-tuned via supervised learning. This method produces a foundation reward model, which can be applied to different tasks with little or no further fine-tuning effort, including response ranking, reinforcement learning from human feedback, and task adaptation with fine-tuning, achieving performance improvements over baseline models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer pUcN,\n\nWe sincerely thank the reviewer for your positive and insightful feedback.  \nWe greatly appreciate your recognition of our paper's proposed training paradigm for the reward model as \"interesting and sound\", and are pleased that the method is considered to be \"extensively evaluated across different tasks\".\n\nWe will provide explanations for the main points that you are concerned about.\n\n---\n>*W1: I am wondering whether using such enhanced RM could improve the reasoning ability of existing LLMs.*\n\nThanks for your insightful suggestions! Indeed, using an enhanced RM can improve the reasoning ability of existing LLMs. We have already conducted an experiment to validate this with a math comparison pair dataset (huggingface tag: ```reciprocate/math_dpo_pairs```). Specifically, we used this dataset during the second stage of GRAM training, with all baseline RMs also trained on the same dataset. We performed reward accuracy experiments on the corresponding test set and with best-of-n sampling on GSM8K, respectively. For the best-of-n sampling, we sample 16 outputs on LLaMA-3.1-8B-Instruct with 8-shot for each input.\n\n|Method|RM Accuracy|GSM8K (Best-of-n Sampling)|   \n|:-|:-:|:-:|\n|LLaMA-3.1-8B-Instruct|-|83.9|\n|Discriminative RM|63.2|84.6|\n|Generative RM|61.3|85.5|\n|GRAM|**66.8**|**87.2**|\n\nThe experimental results show that our method effectively improves RM accuracy on reasoning-related downstream tasks. Also, it further enhances the reasoning ability of LLMs with best-of-n sampling. We promise to include more experiments, including additional baselines and RL experiments, in the revised version to further verify the effectiveness of GRAM in improving reasoning abilities.\n\n>*W2: The influence of pre-training data on target domain should also be discussed.*\n\nIn Figure 6 of the paper, we already show the impact of pre-training data on general domain adaptation. In response to your concern, we further conduct experiments in specific domains using LLaMA-3.2-3B-Instruct, along with 5k labeled summarization and harmlessness data.\n\n|Domain|Amount of Unlabeled Data|\n|:-|:-:|\n|| 0k &nbsp; 100k &nbsp; 200k &nbsp; 400k &nbsp; 600k|\n|Summarization|56.5 &nbsp; 62.7 &nbsp; 66.3 &nbsp; 71.6 &nbsp; 73.4|\n|Harmlessness|58.4 &nbsp; 63.2 &nbsp; 65.9 &nbsp;  69.8 &nbsp; 72.1|\n\nThe results from these experiments align with those in Figure 6. We see that as the amount of unlabeled data increases, the accuracy of GRAM generally improves for both models. This also highlights the crucial role of unlabeled data and the scaling effect on performance, suggesting that using larger unlabeled datasets can lead to better reward models. We promise to include these experiments in the revised version.\n\n>*Q1: Would the domain difference between pre-training and fine-tuning has a big impact on RM performance?*\n\nTo respond to your concern, we conduct the following experiments on GRAM:\n- GRAM-v1: Pre-training on 100k unlabeled summarization response pairs (derived from TL;DR comparison data;huggingface tag: ```openai/summarize_from_feedback```), followed by fine-tuning on 5k labeled summarization data.\n- GRAM-v2: Pre-training on 100k general unlabeled data (including summarization responses), followed by fine-tuning on 5k labeled summarization data.\n- GRAM-v3: Pre-training on 100k general unlabeled data (without summarization-related data; specifically, we are using ChatGPT to filter out preference data related to summarization), followed by fine-tuning on 5k labelled summarization data.\n\nIn these experiments, summarization is the downstream domain. GRAM-v1 uses pre-training data closest to summarization, fully utilizing summarization responses. GRAM-v2 uses general pre-training data, which might include some samples aligned with the downstream summarization task. GRAM-v3 uses pre-training data that is the most different from the summarization domain. Specifically, we use ChatGPT to exclude all samples related to summarization tasks. The results are as follows:\n\n|Method|Accuracy (Summarization)|\n|:-|:-:|\n|RM w/o Pre-training |56.5|\n|GRAM-v1|**74.7**|\n|GRAM-v2|71.6|\n|GRAM-v3|67.4|\n\nThe experimental results demonstrate that pre-training data more closely aligned with the target domain results in better performance in that domain. In fact, this experimental observation is consistent with the common practice in LLMs, where incorporating as much domain-specific data as possible during pre-training typically leads to better performance on downstream tasks. Additionally, our results show that the pre-training approach exhibits strong robustness—despite significant domain differences, it still yields positive contributions to performance.\n\nWe appreciate your valuable suggestion and promise to include more discussion and experiments in the revised version. \n\n---\nWe sincerely thank you for your positive feedback on our paper!  \nThank you once again for your time.\n\nBest,\n\nAuthors", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "No", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths**:\n\n1. The proposed training paridigm for reward model is interesting and sound.\n\n2. The proposed method is extensively evaluated across different tasks.\n\n**Weaknesses**:\n\n1. I am wondering whether using such enhanced RM could improve the reasoning ability of existing LLMs. More downstream evaluations should be conducted.  \n\n2. The paper does not introduce much details about \"large-scale unsupervised learning\", the influence of pretraining data on target domain should also be discussed.", "other_comments_or_suggestions": "The authors should conduct more evaluations and provide more details/discussions as the weakness part comment.", "questions_for_authors": "Would the domain difference between pretraining and finetuning has a big impact on RM performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GtaO9zizOW", "reviewer": "Reviewer_Jdyw", "summary": "The paper proposes a method for training a generative reward model that generalizes better across domains with minimal fine-tuning. The approach involves a two-step training process: large-scale unsupervised learning followed by supervised fine-tuning on labeled data. The paper also demonstrates that applying label smoothing during training can be interpreted as optimizing a regularized pairwise ranking loss, highlighting a connection between training discriminative and generative reward models. The resulting 'foundation reward model' outperforms discriminative baselines on tasks including response ranking, best-of-n sampling, and task adaptation with fine-tuning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Jdyw,\n\nWe appreciate that you find \"the superior generalization of our GRAM compared to both discriminative and prior generative reward models\" and our pre-training approach is \"make sense\".\n\nWe will provide explanations for the main points that you are concerned about.\n\n---\n>*W1: The results in Figure 2 and Table 1, where GRAM underperforms compared to discriminative models on ID data, raise the question of whether this reflects overfitting by discriminative models rather than better generalization.*\n\nWe would like to clarify this question in two aspects. \n- The ID test set evaluates the ability of reward modeling to learn human preferences from labeled data. Our goal is to excel in reward modeling and generalization, i.e., obtaining strong performance on ID and OOD tasks simultaneously. Thus, the LLaMA-3.1-8B-Instruct results in Table 1 show our method's effectiveness, achieving the best OOD results and second-best ID results. \n- Although GRAM underperforms compared to the ```Discriminative RM+Regularization``` when using LLaMA-3.1-8B-Instruct, it significantly outperforms both the ```Discriminative RM (Baseline)``` and ```Discriminative RM+Freeze```, supporting the validity of our method. We also see that ```Regularization``` may not be a universally effective method across all models, e.g., on the LLaMA-3.2-3B-Instruct model, ```Regularization``` performs notably worse than GRAM, suggesting that its effectiveness could be model-dependent.\n\nThe partial results of Table 1 are shown below:\n\n|Method|UniFeed(ID)|RewardBench(OOD)|   \n|:-|:-:|:-:|\n|**LLaMA-3.1-Instruct**|\n|Discriminative RM (Baseline)|69.3|74.1|\n|Discriminative RM+Freeze|66.6|74.9|\n|Discriminative RM+Regularization|**72.7**|77.4|\n|GRAM (Ours)|70.4|**85.1**|\n|**LLaMA-3.2-Instruct**|\n|Discriminative RM (Baseline)|68.3|72.8|\n|Discriminative RM+Freeze|63.0|70.5|\n|Discriminative RM+Regularization|65.6|71.2|\n|GRAM (Ours)|**70.6**|**83.6**|\n\n>*W2: The paper lacks reinforcement learning experiments where an LLM optimizes the reward model using RL.*\n\nWe apologize for any misunderstanding regarding the RL experiments. In fact, we have already included reinforcement learning experiments in Appendix C.1. Specifically, we comprehensively compare PPO performance with different reward models, as shown in Figure 8 and Table 2. The results align with the response ranking, showing that GRAM exhibits strong generalization and provides more accurate rewards. To respond to your concern, and in the revised version, we promise to include partial results in the body rather than only in the appendix.\n\n>*Q1: The paper does not detail how to prepare the two responses, particularly how to prompt an LLM for multiple responses (e.g., simply different responses or responses of varying quality).*\n\nThanks for this insightful suggestion! We have explored this issue with two GRAM variants.\n- GRAM-Sim-Diff: We explore the impact of response diversity on pre-training performance. Specifically, we select the 200k response pairs with the highest semantic differences from a dataset of 600k responses and compare them to 200k randomly selected pairs.\n- GRAM-Qua-Diff: We test quality differences by scoring responses with GPT-4o on a 0-5 scale and compare the 200k pairs with the largest quality differences to 200k random pairs.\n\nThe results are as follows:\n|Method|RewardBench|   \n|:-|:-:|\n|GRAM-Sim-Diff|74.8|\n|GRAM-Qua-Diff|75.4|\n|GRAM|**76.2**|\n\nWe find that randomly selected pairs outperformed the carefully selected ones, possibly due to increased diversity from randomness. We also see that larger quality differences have little impact on reward model training, with random selection performing better. This is also shown in previous work (Filtered Direct Preference Optimization), where larger quality differences benefited DPO training but not reward model training. We promise to add more experiments and analysis in the revised version.\n\n>*Q2: A discussion on extending the training scheme to multi-objective settings, where responses are evaluated by multiple criteria, would be insightful. Have you considered using reasoning models for generative reward models?*\n\nThanks for your valuable suggestion! GRAM can easily be extended to multi-objective or reasoning settings, and this extension can be implemented in stage two. Specifically, we would describe the objectives, e.g., fluency and accuracy, in the prompt $c$. When generating preferences, we can use the format ```#Fluency Preferred: [labeled token] #Accuracy Preferred: [labeled token]``` instead of ```#Preferred: [labeled token]```. Similarly, we can also add the CoT in front of the preferences. We can also use a reasoning model to develop GRAM further. These improvements don’t require changes to our pre-training, highlighting its robustness and scalability for reward modeling. We promise to include more analysis in the revised version.\n\n---\nWe truly appreciate your positive feedback on our paper!\n\nBest,\n\nAuthors", "claims_and_evidence": "The main claims about the proposed GRAM are that it generalizes better than both discriminative and prior generative reward models, outperforming them on response ranking, reinforcement learning from human feedback, and task adaptation with fine-tuning.\n\nFigure 2 and Table 1 present the main results supporting the generalization claim, showing that GRAM performs better on out-of-distribution test data, including RewardBench and HHH-Alignment. However, the results in Figure 2 and for Llama-3.1-8B-Instruct in Table 1, where GRAM underperforms compared to discriminative reward models on in-distribution test data, raise the question of whether this is due to discriminative models overfitting to in-distribution data rather than indicating a better generalization capability. The results for Llama-3.2-3B-Instruct in Table 1 are arguably more convincing, as GRAM outperforms on both in-distribution and out-of-distribution test data.\n\nThe best-of-n sampling results also show that GRAM is less susceptible to overoptimization, which appears closely related to its generalization performance. However, the paper does not include reinforcement learning experiments where an LLM is trained to optimize the reward model using RL or methods like DPO that optimize for the same objective. Without reinforcement learning results, the claim in the abstract that GRAM outperforms in reinforcement learning from human feedback seems less convincing.", "methods_and_evaluation_criteria": "The proposed 'pre-training' step, where a generative reward model is trained to generate two responses per input, makes some sense. However, the extent to which this training improves performance beyond fine-tuning is not thoroughly demonstrated empirically. Additionally, the paper does not discuss in detail how the two responses should be prepared -- specifically, how to prompt an LLM to generate multiple responses (e.g., simply different responses or responses of varying quality). Also, some discussion on how the proposed training scheme could be extended to multi-objective settings, where two responses are evaluated according to multiple criteria, would have been more insightful.\n\nThe evaluation on several common use cases of reward models, including response ranking, alignment, and reward model adaptation, seems appropriate for the problem. To strengthen the results, RL fine-tuning using the different reward models would have been beneficial.ㅒ", "theoretical_claims": "No major theoretical claims are made in the paper.", "experimental_designs_or_analyses": "Overall, the use of two open instruct models for reward model training, along with common reward model benchmarks such as RewardBench and HHH-Alignment, for comparison with both discriminative reward models and strong LLMs like GPT-4, seems sound. More ablations, such as evaluating how much the proposed 'pre-training' helps, would have been insightful.", "relation_to_broader_scientific_literature": "Reward models are essential for evaluating and aligning generative models, and, as the paper also suggests, much less work has been done on developing 'foundation' reward models compared to foundation generative models. This paper contributes to the recent efforts in developing generative models, particularly those considered 'foundational,' where pre-training is used to train domain-agnostic models that can be easily adapted to different domains with minimal extra data. More efforts will need to be devoted to developing such generic reward models in an unsupervised manner for continued improvement of generative models going forward.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "Typos:\n- \"an\" $\\rightarrow$ \"a\" near line 104.\n- \"Best-of-n\" $\\rightarrow$ \"best-of-n\" near line 146.\n- \"Opeen\" $\\rightarrow$ \"Open\" near line 282.", "questions_for_authors": "Q. Have you considered using reasoning models for generative reward models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes to train a (Generative) Reward Model in two stages: first a pre-training stage (where the model is trained to generate pairs of responses, without preference labels), followed by a fine-tuning stage on labeled preferences (also leveraging label smoothing). This is shown to outperform discriminative Reward Models in the large majority of empirical results.

Overall Reviewers would like to see the paper accepted, but without considering it a must-have (all 3's). I personally find this work intriguing, as it can be surprising to see the benefits brought by training the RM to generate responses during this "pre-training" stage. Although I feel like the "why this works" may not be fully understood, I find the empirical evaluation to be extensive enough to justify the method and deserve being shared with the ICML community. The pre-training of RMs is an under-studied topic and this paper makes it clear that it's worth looking into it more closely.
Decision: Accept (poster)