What Makes a Reward Model a Good Teacher? An Optimization Perspective
Authors: Noam Razin
Zixuan Wang
Hubert Strauss
Stanley Wei
Jason D. Lee
Sanjeev Arora
Venue: NeurIPS 2025 spotlight ()
Keywords: Language Models
Reinforcement Learning from Human Feedback
Reward Model
Policy Gradient
Primary area: deep_learning

Abstract: The success of Reinforcement Learning from Human Feedback (RLHF) critically depends on the quality of the reward model. However, while this quality is primarily evaluated through accuracy, it remains unclear whether accuracy fully captures what makes a reward model an effective teacher. We address this question from an optimization perspective. First, we prove that regardless of how accurate a reward model is, if it induces low reward variance, then the RLHF objective suffers from a flat landscape. Consequently, even a perfectly accurate reward model can lead to extremely slow optimization, underperforming less accurate models that induce higher reward variance. We additionally show that a reward model that works well for one language model can induce low reward variance, and thus a flat objective landscape, for another. These results establish a fundamental limitation of evaluating reward models solely based on accuracy or independently of the language model they guide. Experiments using models of up to 8B parameters corroborate our theory, demonstrating the interplay between reward variance, accuracy, and reward maximization rate. Overall, our findings highlight that beyond accuracy, a reward model needs to induce sufficient variance for efficient optimization.

What Makes a Reward Model a Good Teacher? An Optimization Perspective
Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D. Lee, Sanjeev Arora
Princeton Language and Intelligence, Princeton University
Abstract
The success of Reinforcement Learning from Human Feedback (RLHF) critically depends on the quality of the reward model. However, while this quality is primar- ily evaluated through accuracy, it remains unclear whether accuracy fully captures what makes reward model an effective teacher. We address this question from an optimization perspective. First, we prove that regardless of how accurate reward model is. it induces low reward variance, then the RLHF objective suffers from a flat landscape. Consequently, even perfectly accurate reward model can lead to extremely slow optimization, underperforming less accurate models that induce higher reward variance. We additionally show that reward model that works well for one language model can induce low reward variance, and thus flat objective landscape, for another. These results establish fundamental limitation of evalu- ating reward models solely based on accuracy or independently of the language model they guide. Experiments using models of up to 8B parameters corroborate our theory, demonstrating the interplay between reward variance, accuracy, and reward maximization rate. Overall, our findings highlight that beyond accuracy, reward model needs to induce sufficient variance for efficient optimization.
Introduction
Training safe and helpful language models requires aligning them with desirable traits that are difficult to specify explicitly, be through examples or hand-crafted reward functions. The widely adopted Reinforcement Learning from Human Feedbuck (RLHF) pipeline [11, 95, 55, 6] circumvents this difficulty through a two-step approach. It first trains a reward model FRM based on access to preference data, under the assumption that this data reflects unknown ground truth reward rG. which encodes said desirable traits. Then, the language model Ta (referred to as a policy) is aligned by maximizing FRM via policy gradient methods such as PPO [68], RLOO [42, 2], and GRPO [69].
Although the success of RLHF depends heavily on the quality the reward model, it remains unclear how this quality should be measured. Current benchmarks for evaluating reward models focus mostly on accuracy [44, 93, 25, 47]. Namely, for dataset of prompts, each associated with outputs ranked according to human (or AI) preferences, accuracy measures the proportion of output pairs that the reward model ranks correctly. Accuracy maximizing the proxy reward FRM is likely to increase the ground truth reward ra (given that underlies the preference rankings in the dataset). However, recent empirical evidence suggests that, by itself, accuracy may not be indicative of how good reward model is: [37, 10, 80] observed that more accurate reward models do not necessarily yield stronger language models after RLHE, So.
natural yardstick. It quantifies the extent to which
what makes a reward model good teacher for RLHF?
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Ground Truth Reward
Reward Model
Accuracy
Moderatc
Low
Reward Variance
Initial policy
High
Low
Figure Illustration of how accuracy (Definition 1) and reward variance (Definition affect the RLHF objective landscape (Equation (I)). Accuracy reward variance capture distinct aspects of reward model: the former controls alignment with the ground truth reward, while the latter determines the flatness of the objective landscape. The lower the accuracy, the more prone reward model is to reward hacking [5, 70, 57, 26] directions considered beneficial by the reward model may not increase the ground truth reward. On the other hand, even if the reward model is perfectly accurate, low reward variance implies landscape that hinders the efficiency policy gradient methods, as we prove Section and demonstrate empirically in Section 4.
We address the question above from optimization perspective. We begin by proving that, regardless of how accurate reward model is, it can induce a flat objective landscape that hinders optimization. Specifically, if FRM induces low reward variance (Definition 2), which occurs when it does sufficiently separate outputs that are probable under Te. then both the proxy reward YRM and the ground truth reward ne increase an extremely slow rate during policy gradient (Section 3.2). This result builds on a connection between reward variance and the gradient of the RLHF objective [64], and reveals that an effective reward model needs ensure the variance is not too low.' See Figure for an illustration of how accuracy and reward variance affect the RLHF objective landscape.
We then prove two implications of the relation between reward variance and optimization rate under policy gradient. First, more accurate reward models are not necessarily better teachers (Section 3.3). perfectly accurate reward model
In particular, since reward variance is not tied to accuracy, can lead to arbitrarily slow ground Iruth reward maximization compared to a relatively inaccurate model. Second, for different language models, different rewards models work better (Section 3.4).
The core insight behind this result is that the same reward model can induce high reward variance for one language model, yet low reward variance, and consequently flat objective landscape, for another. Taken together, Sections 3.3 and 3,4 establish a fundamental limitation of evaluating reward models solely based accuracy or isolation from the language model they guide [44, 93, 25, 47].
Experiments (Section 4) over models of up to 8B parameters and standard RLHF datasets corroborate theory, demonstrating the interplay between reward variance, accuracy, and reward maximization rate. Remarkably, we find that even when the ground truth reward accessible, using a proxy reward model can lead to better performance if it induces higher reward variance. Overall, results highlight that alongside being accurate, a reward model needs to induce sufficient reward variance for enabling efficient optimization. We hope insights from this work will inform improved methodologies for reward model training and evaluation that account for properties beyond accuracy.
Related work. We discuss related work throughout and defer extended account to Appendix A.
2 Preliminaries
Let V be finite vocabulary of tokens and D denote the set of all finite-length token sequences. We use Ty to denote a language model, parameterized by that receives a prompt x E X' and produces a
Reward variance (Definition 2) should not be confused with the variance of gradient estimates [30], which more common notion of variance the reinforcement learning literature.

---

distribution over outputs y E V, where X C V are the sets of possible input prompts and outputs, respectively, Following convention the RLHF literature, we also refer to To as a policy (cf. [55]).
2.1 Aligning Language Models via Reinforcement Learning
The standard RLHF pipeline for aligning language models consists of two main steps, outlined below (ef. [95. 71, 55, 6)). These steps are usually preceded by a supervised finetuning (SFT) phase, in which the base language model is trained with next-token prediction loss pairs of imput prompts and desirable outputs. The resulting model, denoted Te,.. serves as the initial policy for RLHF.
Reward model training or selection. RLHF assumes that there exists an unknown ground truth reward rG X' y/ > [-1, 1], which encodes (at least some aspects of) human preferences." For given prompt E X, we wish to maximize the expected ground truth reward that our policy Ty achieves, Lee. E (x)xxxxxx)) y)]. However, since TG is not directly accessible, the first step of RLHF is to train a proxy reward model TRM : X' y + [-1, 1]. This is typically done through a Bradley-Tery [9] log-likeline leeliikl loss over dataset of human preferences. Alternatively, one can forgo reward model training and choose publicly available model instead (e.g.. see [76. 87, 44)).
Step 1:
Step 2: Reward maximization via policy gradient. With a reward model Ium in hand, the policy To is updated by using policy gradient methods, such as PPO [68], RLOO [42, 2], and GRPO [69], to maximize:
ÓRLEF (0) = Ex-S Byy-rok x) [X,M y)] A (T) (-1x))
Here, S " X' a training set of prompts, X S denotes sampling x uniformly from S, and À is a KL regularization coefficient that allows controlling how much T will deviate from the initial policy Th... (i.e., the policy produced by the SFT phase). The hope is that by increasing the expected proxy reward we will also increase the expected ground truth reward, while the KL regularization helps preserve capabilities acquired during pretraining and SFT.
2.2 Accuracy
Currently, reward models are primarily evaluated through accuracy [44, 93, 25, 47], which measures their agreement with the ground truth reward in terms of ranking output pairs.
Definition 1.
Given a prompt x E X, the accuracy of a reward model FRM : X' x y [-1,1] with respect to distribution D over unordered output pairs is defined by:
accx, (FRM) == E L (sigu(FRM(x,,) y) - TRM (x. y')) sign(ra (x, TG (x, y |)]}
where ng is the ground truth reward, [[:] is an indicator function, and sign R + {-1, 1, 0, 1) the sign function. For set of prompts, accuracy refers to the mean accuracy over the set.
On which prompts and output pairs should accuracy be measured? Accuracy is often estimated using benchmarks containing diverse prompts with ranked output pairs [44, 93, 25, 47]. However, during RLHF the reward model is applied exclusively to prompts in the policy gradient training set S and on-policy outputs (i.e., outputs sampled from the current policy). As result, when prompts in S or on on-policy outputs differ substantially from those in existing (off-policy benchmarks, the estimated accuracy may be misleading (cf. [80].. To avoid this potential issue, analysis considers the accuracy of a reward model over S and supports any output pair distribution.'
2.3 Reward Variance
Aside from accuracy, another quantity key our work is the reward variance induced by a reward model for policy and prompt. As claborated in Section 3, our theory builds on a recent connection between the gradient of the RLHF objective (Equation (I)) and reward variance [64].
We assume rewards lie in [-1, 1] without loss of generality: any bounded interval can be used instead. 'That is, sign(u) if > 0, sign(u) 0 u and sign(u) -1 otherwise. 'The prompts relevant for accuracy evaluation may vary across alignment procedures. For example, Best-of- N [52] applies the reward model to prompts encountered at test-time.
3

---

Definition 2.
Given policy prompt x E A. and reward model FRM : x'x 1-1, 1-11,1, the reward variance induced by FRM for uo and x is defined by:
(1)) y)] = (y-xo(-1x) (FRM((X,)) - (1x-/
Intuitively, reward variance measures how well FRM separates the rewards assigned to outputs that are probable under (-)x).). This becomes apparent from the fact that:
1 2 " (e(-))) (FRM((x. y) TRM (x.
(ary~=u( (x.
y)] (r))))
and stands in contrast to accuracy, which depends only on how outputs are ranked, disregarding the degree of separation between their rewards.
Reward scale. The scale of rewards affects their variance. For example, multiplying all rewards by e2 0 scales reward variance by a factor of c². It is common practice to account for differences in scale across reward models through normalization (e.g., as done in the experiments Section 4). This ensures that reward variance meaningfully measures the degree of separation between outputs, and not merely the scale of rewards.
3 Theory: Optimization Perspective on What Makes a Good Reward Model
Our analysis considers the time it takes for the expected ground truth reward TG to increase by desired amount when maximizing the RLHF objective (Equation (1)) via policy gradient. The shorter the time, the better. We prove that, if reward model FRM induces low reward variance for the initial policy, then both I'MM and rg increase at a slow rate due to flat objective landscape (Section 3.2). Thus, FRM needs to ensure that the reward variance is not too low for efficient optimization.
We then establish two main implications the relation between reward variance and optimization rate. First, since reward variance is not tied to accuracy, more accurate reward models are not necessarily better teachers (Section 3.3). Second, since the same reward model can induce high reward variance for one policy but low reward variance for another, for different initial policies, different reward models work better (Section 3.4). Our results formalize limitation inherent to benchmarks evaluating reward models solely based on accuracy or independently of the language model they guide [44, 93, 25, 47].
Section 3.1 lays out the technical setting. For conciseness, Sections 3,2 to 3.4 include abridged versions of the theoretical results. Detailed theorem statements are provided in Appendix B.
3.1 Technical Setting
In line with standard practice, we take y to be the set of sequences with length bounded by some maximal value N. Without loss of generality, we may treat as VL the set of sequences of length equal to L - where shorter sequences are padded to length L with dedicated token.
Policy parameterization. Given a prompt X, language models usually produce a distribution over output sequences E autoregressively:
lyl
lyl
(general autoregressive policy) To(y
II
To To
(ffxmax(fo(x,ye))) (2)
where fe: : 12 - V a function, differentiable with respect to its parameters (e.g., neural network), that maps sequence of tokens to logits for the next-token distribution, y, and y are the Ith token and first - tokens of y, respectively, and softmax (z). = cxp(z.)) Evey exp(z.)) for M The lower bound in Section 3.2 on the time it takes for the expected proxy and ground truth rewards to increase applies to policies parameterized according to Equation (2).
The results in Sections 3.3 and 3.4 require proving separations in optimization time between different reward models. Due to the non-concavity of the RLHF objective (Equation (1)), providing guarantees of efficient optimization has proven challenging (cf. [1, 45]). Thus, existing analyses often consider

---

tabular policy parameterization (e.g., [49, 48, ID, which can be viewed as special case of Equation (2), wherein each output is assigned its own trainable logit. That is, for 0 E RDDx
(tabular policy) To(y|x) softmax(00xx)
where 0x E [R/ is the column of @ corresponding to Following prior work, Sections 3.3 and 3.4 consider tabular policies. We note that despite the apparent simplicity of tabular policies, the RL.FF objective remains non-concave.
Optimization. RLHF is typically carried out with small learning rates [55, 84, 2, 80]. Accordingly, we analyze policy gradient at the small learning rate limit, i.e. gradient flow:
d ((L) V (O(r)HF(((()) 12 0.
dt
where ((t) denotes the parameters of the policy (T(()) time t of training and ((0) fref-
Overall, the analyzed setting involves two main simplifications. First, some of our results are proven for tabular policies. Second, we assume access to exact gradients while, in practice, the large output space of language models prohibits exact computation of VØRLHF As Section verifies empirically, the implications of our theory apply to standard language models trained via policy gradient methods that use sample-based estimates of the gradient (e.g., RLOO and GRPO). Nonetheless, relaxing these simplifications can be an interesting challenge for future work.
3.2 Low Reward Variance Implies Slow Reward Maximization
In this section, we lower bound the time required for the expected reward, measured with respect to any reward function (including TRM and rG), to increase by an additive constant. Theorem shows that this time grows inversely with Exx-s[Vary-ee (-1x) (TRM((X, (x, y)]] - the average reward variance induced by FRM for the initial policy Ta(o) and prompts in the training set S. As a result, if S, which occurs when FRM does not sufficiently
Vary~-recos (x)) (x, y]] is low for prompts separate outputs that are probable under To(o). then policy gradient suffers from slow optimization.
Theorem 1 (Abridged version of Theorem 4). Suppose that we maximize the RLHF objective (Equa- tion (1)) with respect to a reward model TRMS, using a general autoregressive policy (Equation (2)). For any 7 > O, prompt E X, and reward function (e.g., can be the ground truth reward ra or the reward model FRM), the time it takes until Eyymous [x(x 2 Ey (-1o)/((x y)] + is:
2 3(Exx Ex Var (-bxe [TRM(('' .>]] :).
where 12(-) hides terms depending on the KL regularization coefficient X, the maximal output sequence length L, and the Jacobian of fo (Equation (2)).
Proof sketch (full proof of Theorem 4 is in Appendix D.4). Low reward variance is known to imply vanishing gradient for policies that produce next-token distributions via softmax (Theorem 1 in [64]). Specifically, V Ey-xs(-Ix) [TRM(X y)]l decays with Vary (Ta][/Ra(x,, y)] for any prompt x and policy Ty. However, this alone does not yield satisfactory lower bound on the reward maximization rate because, without further knowledge, the gradient norm may increase rapidly during training. We show that this is not the case: when the reward variance is low, higher-order derivatives of the RLHF objective vanish alongside the gradient, preventing rapid increase in gradient norm. This constrains the movement of policy parameters {t). leading to a lower bound on the rate of reward increase. O
3.3 More Accurate Reward Models Are Not Necessarily Better Teachers
Section 3.2 showed that low reward variance hinders the efficiency of policy gradient. Notably, reward variance is independent of the measure typically used to evaluate reward models - accuracy [44, 93, 25, 47]. Accuracy depends only on how the reward model ranks different outputs, disregarding the degree of separation between their rewards, whereas reward variance determined by this degree of separation. A key implication, established by Theorem 2, is that a reward model I'rM can be accurate yet lead to arbitrarily slow ground truth reward maximization compared to substantially less accurate model T'M- due to inducing low reward variance.
5

---

For clarity, Theorem 2 considers: (i) the extreme case where FRM is perfectly accurate and '' almost completely inaceurate; and (ii) accuracy with respect to the uniform distribution over unordered output pairs, denoted by (CCx(PRM) for prompt x and reward model TRM- The detailed version of the theorem (Theorem in Appendix B) much more general, applying to almost any accuracy values for FRM and 'RM- as measured with respect to any distribution over output pairs.
Theorem 2 (Abridged version of Theorem 5; proof sketch deferred to Appendix B.2). Suppose ny is a tabular policy (Equation (3)). Given prompt x E S, let y > 0 and denote by t > 0 the initial y)] + 7y For any initial policy To(D). there exist perfectly accurate reward model rRM, i.e. accx((RM) model R' with accx(FTRM) S | such that the following hold.
time which E (x. yx]]
and an inaccurate reward
Slow ground truth reward increase under rRM- When using FRMM for maximizing the RLHF objective, by can be arbitrarily large.
Fast ground truth reward increase under TRM- In contrast, t, (y² when using THMM- where y E y is an output satisfying ra(x,y") > Ey (r y) +2 and O()) hides terms depending on 7.
We emphasize that Theorem 2 does not suggest highly accurate reward models are inherently poor teachers. In fact, established by its detailed version in Appendix B (Theorem 5), at almost any accuracy level some reward models lead to inefficient optimization while others perform well. Rather, Theorem 2 formalizes why accuracy its own is insufficient criterion for evaluating reward models in RLHF, agreeing with recent empirical evidence [37, 10, 80).
Accuracy is desirable property nonetheless, since more accurate models are usually less susceptible to reward hacking [5, 70, 57, 26]. Namely, when training with an imperfect reward model, eventually the ground truth reward can start decreasing due to mismatch between the two rewards. It is therefore common practice to run policy gradient for only a few epochs (see, e.g., [55, 37, 2D.. Theorem captures this regime, in which less accurate reward model can outperform more accurate one by driving a faster increase in ground truth reward.
Role of accuracy depends on the alignment method. While accuracy alone does not guarantee the effectiveness RLHF, its importance varies across alignment methods. For example, in Best- of-N sampling [52], it is straightforward to prove that a perfectly accurate reward model is always optimal (see Proposition in Appendix C). Investigating the role of accuracy in additional methods [17, 32, 27, 18, 65] remains important direction for future work.
3.4 For Different Initial Policies, Different Reward Models Are Better
Reward variance depends on both the reward model and the policy. In particular, reward model in- ducing high reward variance for one policy may induce low reward variance for another. Consequently, the connection between reward variance and optimization implies that for different initial policies, different reward models are better see Theorem 3. This highlights that faithfully evaluating reward models for RLHF requires taking into account the policy being aligned.
Theorem 3 (Abridged version of Theorem 6; proof sketch deferred to Appendix B.3). Suppose na is tabular policy (Equation (3)). Given prompt E S, let y > 0 and denote by ty > 0 the initial time at which Ey--rous ())) (x, y)] 2 Eyssoon TRM, TTo and initial policy families II, II' such the following hold.
+ 7. There exist reward models
TRM is a better teacher for initial policies in IL. If To(o) E II, then t,, = (1) when using TRM for maximizing the RLHF objective, whereas by can be arbitrarily large when using T'MM-
TRM is a better teacher for initial policies in IT', If (et) E IT, then t,, = (1) when using T'M for maximizing the RLHF objective, whereas ty can be arbitrarily large when using FRM-
In the above, O(.) hides terms depending on 7.

---

4 Experiments
We verify that conclusions drawn from our theoretical analysis (Section 3) hold in practice. Consistent with Sections 3.2 and 3.3, Section 4.1 shows that reward variance strongly correlates with the reward maximization rate during policy gradient. In particular, using more accurate reward models may result in worse performance within a given training budget if they induce low reward variance. Strikingly, this applies also to the ground truth reward: even if it is accessible, in some regimes a proxy reward model can be more effective. Then, in accordance with Section 3.4, Section 4.2 demonstrates that for different language models, different reward models lead to higher ground truth reward.
Our experiments include language models from different families (Pythia [8] and Llama-3.2 [21]), reward models of up to 8B scale, and standard RLHF datasets (AlpacaFarm) [22] and UltraFeed-- back [14]). For brevity, we defer to Appendices and F some experiments and implementa- tion details, respectively. Code for reproducing our results is available at https: thub. com/ princeton wwteetttoolli/whhaeee wwtppppttttaaaaaaaeeeeaaa masattttmaa-eeeee good-rm. rm.
4.1 More Accurate Reward Models Are Not Necessarily Better Teachers
4.1.1
Setting
We follow the RLHF pipeline outlined in Section 2.1.
Ground truth reward. Evaluating policies based on human preferences is expensive. Thus, as done in prior work [26, 13, 73, 7, 10, 80, 72], simulate human preferences with high-quality reward model. In our experiments, the ArmoRM [76] model serves as the ground truth reward.
Data. We partition the UltraFeedback [14] training set into two subsets: 80% of the samples are used for reward model training and the rest for the policy gradient step of RLHF. Output preferences the reward modeling subset are relabeled using the ground truth reward.
Initial policy. We SFT the pretrained Pythia-2.8B language model on AlpacaFarm. Appendix in- cludes experiments where SFT is performed on the (UltraFeedback-baseee reward model training set.
Reward models. Studying how reward variance and accuracy affect the outcome of RLHF requires reward models that vary in these properties. To that end, we use different output pair distributions for training. Specifically, we create five different training sets by associating either 100%, 75%, 50%, 25%, or 0% of the prompts with on-policy preference pairs, sampled from the initial policy and labeled according to the ground truth reward, while the remaining prompts are associated with their off-policy preferences from (the relabeled) UltraFeedback. We then train a separate reward model over each training set via the usual approach. Namely, each reward model is initialized based on the SFT model and trained with a Bradley- Terry log-likelihood loss (cf. [9, 55]). We additionally consider a perfectly accurate reward model that induces low reward variance, obtained by taking the ground truth reward and reducing how well it separates different outputs while preserving their rankings.
Reward normalization. For fair comparison, rewards on the same scale (similarly to [26]; see Appendix F for further details). Table reports, for each model, the reward variance it induces for the initial policy and its accuracy, both on-policy (i.e., on outputs sampled from the initial policy) and off-po (i.e., on outputs from UltraFeedback).
normalize all reward models so that they produce
Policy gradient. We use RLOO [42. 2] as it is substantially more resource efficient than PPO [68] and has shown competitive results [2]. Policies are evaluated by computing the average proxy and ground truth rewards over prompts from the policy gradient training set, where for each prompt 10 outputs are sampled from the policy. We also evaluated policies using prompts from the (UltraFeedback) test set. The results were nearly identical, report only rewards over the training set for conciseness."
4.1.2 Results
For each reward model described in Section 4.1.1, Figure 2 presents the increase in proxy reward (used for training) and ground truth reward during policy gradient. As a baseline, we compare the reward models to running policy gradient directly with the ground truth reward.
'Rewards over the training and test sets were similar since, in accordance with common practice [55, 37, 2]. policy gradient was carried out for only few epochs (six in the experiments of Section 4.1).
7

---

Table I: For each reward model described in Section 4.1.1 and the ground truth reward, we report: (i) the reward variance induced for the Pythia-2.8B initial policy; and (ii) accuracy, measured on-policy (i.e., on outputs sampled from the initial policy) and off-policy (i.e., on outputs from UltaFeedback). All quantities were averaged across prompts in the policy gradient training set (their values on the test set were nearly identical). As mentioned in Section 4.1.1, to ensure fair comparison of reward variance. the reward models and ground truth reward were normalized so that they produce rewards the same scale.
RM On-Policy %
RM with Perfect Acc. but Low Reward Variance
Ground Truth Reward
100%
75%
50%
25%
0%
Reward Variance
0.630
0.616
0.555
0.438
0.314
0.111
0.256
On-Policy Acc.
0.660
0.656
0.640
0.616
0.587
1.000
1.000
Off-Policy Acc.
0.630
0.732
0.758
0.754
0.762
1.000
1.000
Incresse In Revaard for Training
Increasein Ground Truth Reward
Reward Used for Training
0.60
0.00
RM 100% on-policy
RM 75% on-policy
0.60
0.35
RM 50% on on-policy
RM 25% on on-policy
0.40
0.30
RM 0% on-policy
RM with per perfect accuracy but low reward variance
0.00
Ground Trath
0.00
Epoch
Epoch
Figure 2: More accurate reward models are not necessarily better teachers. Using cach reward model described in Section 4.1.1, we trained Pythia-2.8B language model via policy gradient (specifically, RLOO [2] on prompts from UltraFeedback: characteristies of these reward models are provided in Table 1. For comparison, we also ran policy gradient directly with the ground truth reward. Plotted is the increase in proxy reward (left), Le., the reward used for training, and ground truth reward (right) against the number of epochs. Markers indicate the mean across three runs, with error hars showing standard deviation. In agreement with Theorem 2, a perfectly accurate reward model that induces low reward variance (red markers) underperforms less accurate models. Moreover, in the first couple of epochs. proxy reward model worked better than directly optimizing the ground truth reward; see Figure for an experiment where this gap is even larger.
Reward variance is indicative of reward maximization rate. In line with Theorem 1, the reward variance induced for the initial policy strongly correlates with the increase in both proxy and ground truth rewards; see Table 2 for the correlation coefficients. For the ground truth reward, most indicative a combination of reward variance and accuracy. This is expected, since when the reward model is inaccurate, improvements in proxy reward translate less to improvements in ground truth reward.
Perfectly accurate reward models are not necessarily the best teachers. Unlike reward variance, in our setting accuracy by itself is not indicative of reward increase (Table 2).5 In line with Theorem 2, the perfectly accurate reward model with low reward variance leads to a significantly slower increase in ground truth reward compared to less accurate models (Figure 2). Morcover, using a proxy reward model can be more effective than training with the ground truth reward directly, at least in the first few epochs (Figures and 7). However, if policy gradient is run for long enough with a proxy reward model, the ground truth reward saturates sooner and may decrease due to reward hacking. We further find that accurate reward models typically more KL efficient. At given ground truth reward value, they yield policies with lower KL divergence from the initial policy (Figure 4 in Appendix E).
On-policy vs off off-policy accuracy. As discussed in Section 2.2, measuring accuracy on off-policy outputs can be problematic when they differ substantially from on-policy outputs. This is evident in Figure 2 and Table 1, well the additional experiments mentioned below: reward models with high accuracy on off- off-policy outputs from UltraFeedback but low on on-policy accuracy are poor teachers and tend to cause reward hacking.
'Although accuracy is often negatively correlated with reward increase in our setting, which includes a perfectly accurate reward model that induces low reward variance, this is not always the case. The relationship between accuracy and reward increase can vary depending the reward models considered (ef. (80, 25J).
8

---

Table 2: Reward variance strongly correlates with reward increase, while accuracy on its may not, For the experiments in Figure we report the Pearson and Spearman correlations of different reward model properties (see Table 1) with reward increase after one epoch of policy gradient. "On- & Off-Policy Ace." refers to accuracy measured both on output pairs sampled from the initial policy and output pairs from UtraFeedback. "Reward Variance & Acc." averages this accuracy with reward variance (induced for the initial policy). Notably, the latter combined measure is more indicative of ground truth reward increase than each measure separately.
Increase in Reward Used Training
Increase in Ground Truth Reward
Pearson Corr.
Spearman Corr.
Pearson Corr.
Spearman Corr.
Reward Variance
0.982
1.000
0.834
0.714
On- On-Policy Acc.
0.762
0.143
0.283
0.200
Off-Policy Acc.
0.933
0.943
0.507
0.543
On- & Off-Policy Acc.
0,870
0,486
0.397
0.200
Reward Variance & Acc.
0.676
0.886
0.940
0.828
Increate Revaard Used for Training
Increase in Ground Truth Roward
035
0.0
025
Reward Used Training
0.60
020
2PMM-Gemma-2444
GRM-Ujama-322-B
0.40
0:5
RM-Tulu-V2-8B
010-
RM-Gemma-22
0.20
005
0.00
000
Pythia-1B SFT
Lama-332-18 Llama-3,2-18 SFT
Pythia-18
U3ma-32-18
Llama-3.2-18
Instruct
SFT
SFT
Instruct
Figure 3: For different language models, different reward models are better. Using publicly available reward models, we trained di different language models via policy gradient (specifically, RLOO [2]) on prompts from UltraFeedback; see Table 10 for characteristics of the reward models. Plotted is the increase in proxy reward (left), ie., the reward used for training, and ground truth reward (right) for each combination of initial policy and reward model, averaged across three runs (error bars denote standard deviation). In agreement with Theorem 3, the reward model that yields the highest ground truth reward varies initial policies.
Additional experiments. Appendix E.1 includes analogous findings for experiments with: (i) Pythia-IB language model: (ii) SFT carried out over the UltraFeedback reward model training set (as opposed to on AlpacaFarm); and (iii) a different ground truth reward model.
4.2
For Different Language Models, Different Reward Models Are Better
4.2.1
Setting
We follow the setup outlined in Section 4. 1. but instead using a single initial policy and training reward models ourselves, we consider multiple initial policies and publicly available reward models. The initial policies are based on the Pythia [8] and Llama-3.2 [21] language model families, with SFT performed over UltraFeedback. Additionally, models (which we do not SFT). To ensure diversity in reward model qualities, we select four models varying in their RewardBench [44] scores: GRM-Llama-3.2 3.2-3BB [87] (90.9), GRM-Gemme 2-2B [87] (88.4), RM-Tulu- V2-8B [37] (74.5), and RM-Gemma--2 (65.4). As in Section 4.1, for fair comparison, normalize reward models that they produce rewards the same scale.
consider instruction-tuned Llama-3.2 language
4.2.2 Results
by showing that for different initial policies, different reward models are better in terms of ground truth reward increase. Furthermore, Table in Appendix E displays the reward variance and accuracy (both on on-policy and off-policy for each combination of reward model and initial policy. The results demonstrate that low accuracy and low reward variance can each be detrimental to the success of RLHF. Specifically, RM-Gemma-2B induces the highest reward variance for all initial policies, and accordingly led to the largest increase in proxy reward. However, since it is inaccurate, this did not translate into higher ground truth reward. On the other hand, as
Figure 3 corroborates Theorem 3

---

was the case in the experiments of Section 4.1, the most accurate reward model (whether on on-policy or off- off-pollcy) was not necessarily the most effective either.
Additional experiments. Appendix F.2 includes experiments using: (i) additional language models of up to 3B scale; and (ii) GRPO [69] instead of RLOO.
Conclusion
Although RLHF heavily relies on the reward model's quality, there is limited understanding as to how this quality should be measured (cf. [37. 10, 80D) In this work, we studied what makes a good reward model from optimization perspective. Our theoretical analysis showed that, regardless of how accurate reward model is. if it induces low reward variance, then RLHF suffers from slow reward maximization due to a flat objective landscape. This implies that alongside being accurate. reward models need to induce sufficient variance for enabling efficient optimization.
We then proved and empirically demonstrated two key implications of the connection between reward variance and optimization rate: (i) more accurate reward models are not necessarily better teachers; and (ii) reward model that works well for one language model can perform poorly for another. These results formalize shorteomings of existing reward model benchmarks, which focus primarily on accuracy and produce universal rankings independent of any particular language model [44, 93, 25, 47). As elaborated below, we hope insights from this work will inform improved methodologies for reward model training and evaluation that account for properties beyond accuracy.
5.1 Limitations and Future Work
Improving reward variance. We established that low reward variance undermines the effectiveness of RLHF (i.e., of policy gradient optimization). Yet, how to best modify or train reward models to induce higher reward variance remains an open question. A naive solution to multiply all rewards by factor of >> 1, thereby scaling up reward variance by factor of c², but such an artificial inflation can be problematic. The expected reward gradient vanishes when the reward variance is low (ef. [64]). Thus, sample-based estimates of the gradient may be dominated by noise, which will be amplified by reward scaling. In preliminary experiments, we indeed found reward scaling did not alter trends observed in Section 4.1. However, this does not preclude more sophisticated reward scaling techniques from being helpful, especially when coupled with improved gradient estimates (e.g.. by sampling more outputs for each prompt). An alternative approach, given that reward variance depends on the separation between rewards (see intuition following Definition 2), can be to encourage a larger margin during reward model training (ef. [74, 60, 51).
Reward model evaluation. Our results emphasize the need for holistic evaluations that account for: (i) properties beyond accuracy, reinforcing empirical observations in [10, 80]; and (ii) the specific language model being aligned. We identified reward variance, arising from the interaction between the reward model and language model, as key for efficient policy gradient optimization. Characterizing additional factors influencing the outcome of RLHF and providing robust protocols for reward model evaluation is a valuable direction for future work.
Verifiable rewards and general reinforcement learning environments. Although our positioning focused on RLHF - training language models via reward models learned from human preferences the analysis in Section 3 is agnostic to the source of rewards and can be straightforwardly extended to any reinforcement learning environment with policies that produce distribution over outputs (i.e., actions) via the softmax function. In particular, policy gradient methods are increasingly used for solving tasks in which outputs can be automatically verified as correct, such as math and coding [43, 33). Investigating whether our insights can aid in designing data selection algorithms or verifiable rewards that improve optimization efficiency is promising avenue for further research.
Aligmment methods beyond RLHF. As shown in Section 3.3, the effectiveness of a reward model can vary depending on the alignment method. Namely, while in RLHF perfectly accurate reward models (i.e., reward models that rank all outputs correctly) can underperform less accurate models, for Best-of-N [52] they are always optimal. An exciting next step is to study how properties of different alignment methods, including Best-of-N and those based on contrastive objectives [61, 27, 18, 65) or rejection sampling [17, 32], determine which aspects of reward model are important.
10

---

Acknowledgments and Disclosure of Funding
We thank Eshbal Hezroni for aid in preparing illustrative figures and Sadhika Malladi and Boris Hanin for providing feedback on this paper. NR is supported by Princeton Language and Intelligence (PLI) and the Zuckerman STEM Leadership Program. JDL acknowledges support of Open Philanthropy. NSF IIS 2107304, NSF CCF 2212262, NSF CAREER Award 21 2144994, and NSF CCF 2019844. SA acknowledges funding from ONR, Schmidt Science, and OpenAI.
References
[1] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methads: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research, (2(1):44314506 2021.
[2] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basies: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint 4rX1v:2402.14740, 2024.
[3] Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International conference on machine learning, pages 151-160. PMI.R, 2019.
[4] Gholamali Aminian, Amir R Asadi, Idan Shenfeld, and Youssef Mroueh. Theoretical analysis of kl- regularized rihf with multiple reference models. arXiv preprint arXiv:2502.01203. 2025.
[5] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané, Conerete problems ai safcty. arXiv preprint 16X6::66600666 2016.
[6] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
[7] Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, et al. Infalign: Inference-aware language model alignment. arXiv preprint arXiv:2412. 19792, 2024.
[8] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Fric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: suite for analyzing large language models across training and scaling. In International Conference on Machine Learning. PMLR, 2023.
[9] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[10] Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, and Xiaoyu Shen. The accuracy paradox in rlhf: When better reward models don't Ion't yield better language models. arXiv preprint arXiv:2410.06554. 2024.
[11] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. Advances in neural information processing systems, 2017.
[12] Tianzhe Chu, Yuexiang Zhai, Jihan Yang. Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post bost-training. arXiv preprint arXiv:2501.17161, 2025.
[13] Thomas Coste, Usman Anwar, Robert Kirk. and David Krueger. Reward model ensembles help mitigate overoptimization. In International Conference on Learning Representations, 2024,
[14] Ganqu Cui, Lifan Yuan, Ning Ding, Guamming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafedbbcc:: Boosting language models with high-quality feedback. In International Conference on Machine Learning, 2024.
[15] Nirjhar Das. Souradip Chakraborty. Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rlhf. arXiv preprint a0X5v:2402.10500 2024.
11

---

[16] Yuhao Ding, Junzi Zhang, Hyunin Lee. and Javad Lavaei. Beyond exact gradients: Convergence of stochastic soft-max policy gradient methods with entropy regularization. IEEE Transactions Automatic Control, 2025.
[17] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang. Winnie Chow. Rui Pan. Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.0677 2023.
[18] Hanze Dong, Wei Xiong, Bo Pang. Haoxiang Wang, Han Zhao. Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF Transactions on Machine Learning Research, 2024.
[19] Yunlong Dong Xiuchuan Tang, and Ye Yuan. Principled reward shaping for reinforcement learning via Iyapunov stability theory. Neurocomputing, 393:83 90, 2020.
[20] Yihan Du, Anna Winnicki, Gal Dalal, Shie Mannor, and R Srikant. Exploration-driven policy optimization in rlhf: Thcoretical insights on efficient data utilization. In International Conference on Machine Learning, 2024.
[21] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The laaa 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[22] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang. Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy S Liang. and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 2024.
[23] Jacob Eisenstein. Chirag Nagpal, Alekh Agarwal, Ahmad Beirami. Alexander Nicholas D' D'Amour, Krishnamurthy Dj Dvijotham, Adam Fisch, Katherine A Heller, Stephen Robert Pfohl. Deepak Ra- machandran, Peter Shaw, and Jonathan Berant. Helping or herding? reward model ensembles miti- gate but do eliminate reward hacking. In First Conference on Language Modeling, 2024. URL https:/ /ttpa/////ppeeeeeeeeeeeeee
Su1GpKktt..
[24] Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In International conference machine learning. PMIR,, 2018.
[25] Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E Gonzalez, and lon Stoica. How evaluate reward models for rlhf. In International Conference on Learning Representations, 2025.
[26] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages
PMLR, 2023.
[27] Zhaolin Gao, Jonathan Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, Drew Bagnell, Jason D Lec, and Wen Sun. Rebel: Reinforcement learning via regressing relative rewards. Advances in Neural Information Processing Systems, 2024.
[28] Shivam Garg, Samuele Tosatto, Yangehen Pan, Martha White, and Rupam Mahmood, An alternate policy gradient estimator for softmax policies. In Proceedings of The 25th International Conference on Artificial Intelligence and Staristics, pages 6630 168306666,,611 2022.
[29] Mudit Gaur, Amrit Singh Bedi, Raghu Pasupathy, and Vaneet Aggarwal. On the global convergence of online rlhf with neural parametrization. arXiv preprint arXiv:2410. 15610, 2024.
[30] Evan Greensmith, Peter Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(9), 2004.
[31] Marck Grzes. Reward shaping in episodic reinforeement learning. 2017.
[32] Caglar Gulcehre, Tom Le Paine. Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang. Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint 2.308 2.098,8898,, 2023.
[33] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint ar0iv: 12.500..19945 /2948, 2025.
[34] Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. Advances in Neural Information Processing Systems, 2022.
12

---

[35] Daniel Hennes, Dustin Morrill, Shayegan Omidshafiei. Remi Munos, Julien Perolat, Mare Lanctot. Audrumas Gruslys, lean-Baptiste Lespiau, Paavo Parmas, Edgar Duenez-Guzman, et al. Neural replicator dynamics, arXiv preprint xrXiv:1966.00199 2019.
[36] Jiawei Huang, Bingcong Li, Christoph Dann, and Niao He. Can rlhf be more efficient with imperfect reward models? a policy coverage perspective. arXiv preprint arXiv:2502. 19255, 2025.
[37] Hamish Ivison, Yizhong Wang. Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. Advances in Neural Information Processing Systems, 2024.
[38] Kaixuan Ji. Jiafan He, and Quanquan Gu. Reinforcement learning from human feedback with active queries. arXiv preprint wxxv:24402.09401, 2024.
[39] Xiang Ji, Huazheng Wang, Minshuo Chen, Tuo Zhao, and Mengdi Wang. Provable benefits of policy learning from human preferences in contextual bandit problems. arXiv preprint arXiv:2307. 12975, 2023.
[40] Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
[41] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[42] Wouter Kool, Herke van Hoof, and Max Welling. Buy reinforce samples, get a baseline for free! Deep Reinforcement Learning Meets Structured Prediction ICLR Workhsop, 2019.
[43] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. TŞ" ulu 3: Pushing frontiers in open language model post-training training.. arXiv preprint arXiv:2411 2iii:::111.112244 15124, 2024.
[44] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, IJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. In Findings of the Association for Computational Linguisticx: NAACL, 2025.
[45] Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Softmax policy gradient methods can take exponential time to converge. Conference on Learning Theory, pages 3107 -100.. PMLR, 2021.
[46] Zihao Li, Xiang Minshuo Chen, and Mengdi Wang. Policy evaluation for reinforcement learning from human feedback: A sample complexity analysis. In International Conference on Artificial Intelligence and Statistics, pages 2737-2745. PMLR, 2024.
[47] Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Rm-bench: Benchmarking reward mod- els of language models with subtlety and style. In International Conference on Learning Representations, 2025.
[48] Jincheng Mei, Chenjun Xiao, Bo Dai, Lihong Li, Csaba Szepesvári, and Dale Schuurmans, Escaping the gravitational pull of softmax. Advances in Neural Information Processing Systems, 2020.
[49] Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence of softmax policy gradient methods. International Conference on Machine Learning, pages 6820-6829. PMLR, 2020.
[50] Jincheng Mci, Bo Dai, Alckh Agarwal, Mohammad Ghavamzadeh, Csaba Szepesvári, and Dale Schuur- mans. Ordering-bart based conditions for global convergence of policy gradient methods. Advances in Neural Information Processing Systems, 2023.
[51] Yu Meng. Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances Neural Information Processing Systems, 2024.
[52] Reicchiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, with human feedhack. arXiv preprint arXiv:2/
Webgpt: soowser-assiste question-answering answering 2021.
[53] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In leml, volume 99, pages 278-287, 1999.
[54] Ellen Novoseller. Yibing Wei, Yanan Sui, Yisong Yue, and Joel Burdick. Dueling posterior sampling for wreference-base reinforcement learning. In Conference on Uncertainty in Artificial Intelligence, pages PMLR, 2020.
13

---

[55] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 2022.
[56] Aldo Pacchiano, Aadirupa Saha, and Jonathan Lee. Dueling rl: preferences. In International conference on artificial intelligence and statistics, pages 6263-6289. PML.R, 2023.
Reinforcement learning with trajectory
[57] Richard Yuanzhe Pang. Vishakh Padmakumar, Thibault Sellam, Ankur Parikh, and He He. Reward gaming in conditional text generation. In The st Annual Meeting Of The Association For Computational Linguistics, 2023.
[58] Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi, Andrej Risteski, and Surbhi Goel. Progressive distillation induces an implicit curriculum. In International Conference on Learning Representations, 2025.
[59] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017.
[60] Bowen Qin, Duanyu Feng, and Xi Yang. Towards understanding the influence of reward margin on preference model performance. arXiv preprint arXiv:2404.049) 04492,, 2024.
[61] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2023.
[62] Alexandre Ramé, Nino Vicillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. In International Conference on Machine Learming, 2024.
[63] Noam Razin, Yotam Alexander, Edo Cohen- Raja Giryes, Amir Globerson, and Nadav Cohen. Implicit bias of policy gradient in linear quadratic control: Extrapolation unseen initial states. In International Conference on Machine Learning, 2024.
[64] Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Prectum Nakkiran, Joshua M. Susskind, and Etai Littwin. Vanishing gradients in reinforcement finetuning of language models. In International Conference on Learning Representations, 2024.
[65] Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Un- intentional unalignment: Likelihood displacement in direct preference optimization. In International Conference on Learning Representations, 2025.
[66] Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu. Ray interference: source of platcaus in deep reinforcement leamnig. arXiv preprint arXiv: 11455, 2019.
[67] Antoine Scheid, Etienne Boursier, Alain Durmus, Michael I Jordan, Pierre Ménard, Eric Moulines, and Michal Valko. Optimal design for reward modeling in rihf. arXiv preprint arXiv:2410. 17055, 2024.
[68] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707 1707006777 2017.
06347,
[69] Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang. Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arviv preprint arXiv:2402.03300 2024.
[70] Joar Skalse, Nikolaus Howe, Dmitrii Krashenimnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 2022.
[71] Nisan Stiennon, Long Ouyang. Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, 2020.
[72] Hao Sun, Yunyi Shen, and Jean-Francois Ton. Rethinking reward modeling in preference- large language model alignment. In International Conference on Learning Representations, 2025.
[73] Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Fugene Tarassov, Rémi Munos. Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. Understanding the performance gap between online and offline alignment algorithms. arXiv preprint arXiv:2405,0m 08448, 2024.
14

---

[74] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, MariceAAne Lachaux, Thibaut Lavril, Jenya Lec, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama Open foundation and fine-tuned chat models, 2023.
[75] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang. Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310. 16944, 2023.
[76] Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-expert. arXiv preprint arXiv:2406. 12845, 2024.
[77] Lingxiao Wang. Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. arXiv preprint arXiv: arXiv:1909, 1909.01 0500, 2019.
[78] Yuanhao Wang, Qinghua Liu, and Chi Jin. Is rlhf more difficult than standard rI? a thcoretical perspective. Advances in Neural Information Processing Systems, 2023.
[79] Zihao Wang. Chirag Nagpal. Jonathan Berant, Jacob Eisenstein, Alex D' Amour, Sanmi Koyejo, and Victor Veitch. Transforming and combining rewards for aligning large language models. In International Conference on Machine Learning, 2024.
[80] Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, Xing Yu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, and Le Sun. Rethinking reward model evaluation: Are we barking up the wrong tree? In International Conference on Learning Representations. 2025.
[81] Thomas Wolf, Lysandre Debut, Victor Sanh. Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the-art natural language processing. arXiv preprini 1rXiv:1910003371, 2019.
[82] Runzhe Wu and Wen Sun. Making rl with preference- feedback efficient via randomization. In International Conference on Learning Representations, 2024.
[83] Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rihf under kl-constraint. In International Conference on Machine Learning, 2024.
[84] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Yc, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior ppo for Ilm alignment? comprehensive study, arXiv preprint ar 240: 10719, 2024.
[85] Yichong Xu, Ruosong Wang, Lin Yang, Aarti Singh, and Artur Dubrawski. Preference- based reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems, 2020.
[86] Zhangchen Xu, Fengging Jiang, Luyao Niu, Bill Yuchen Lin, and Radha Poovendran. Stronger models not stronger teachers for instruction tuning. arXiv preprint arXiv:2411.07133 2024.
[87] Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for 1lms. In Advances in Neural Information Processing Systems, 2024.
[88] Rui Yuan, Simon S Du, Robert M Gower, Alessandro Lazaric, and Lin Xiao. Linear convergence of natural policy gradient methods with log-linear policies. In International Conference on Learning Representations, 2023.
[89] Wenhao Zhan, Masatoshi Uchara, Nathan Kallus, Jason D Lec, and Wen Sun. Provable offline preference- based reinforeement lcarning, arXiv preprint arXiv:2305. 14811, 2023.
[90] Wenhao Zhan, Masatoshi Uchara, Wen Sun, and Jason D Lee. Provable reward- agnostic preference-basee reinforcement learning. In International Conference on Learning Representations, 2024.
15

---

[91] Shun Zhang, Zhenfang Chen. Sunli Chen, Yikang Shen, Zhiqing Sun, and Chuang Gan. Improving reinforcement learning from human feedback with efficient reward model ensemble. arXiv preprint urXiv:2401.1633, 2024.
[92] Rui Zheng, Shihan Dou, Songyang Gao. Yuan Hua, Wei Shen, Binghai Wang, Yan Liu,, Senjie Jin, Qin Liu, Yuhao Zhou, Secrets rlhí in large language models part Ppo. arXiv preprint arXiv:2307.044 20.004444444 2023.
[93] Enyu Zhou, Guodong Zheng, Binghai Wang. Zhiheng Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, et al. Rmb: Comprehensively benchmarking reward models Ilm alignment. In International Conference on Learning Representations, 2025.
[94] Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pages 43037- 43067. PMLR, 2023.
[95] Daniel M Ziegler. Nisan Stiennon. Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, arXiv preprint arXiv: arXiv:/909008590 2019.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Each claim made in the abstract and introduction maps theoretical result in Section 3 and is backed by empirical evidence in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 3.1 specifies the limitations of our thcoretical analysis. We further
discuss important directions left open by work in Section 5.1.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [Yes]
Justification: Section 3.1 provides the setting of our analysis, which includes the policy parameterizations (general autoregressive and tabular policies) and training algorithm (gra- dient flow). No other assumptions are made and the theorem statements are self-contained. Every theoretical result is accompanied by a complete proof in Appendix D. We further provide proof sketches for our main results (Theorems to 3).
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All information required to reproduce our experimental results is provided in Section 4 and Appendix F. Our code is publicly available pttps://githin com/ prinnceoon-pli//e hpll:/hattmaake good-rm.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
18

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code is publicly available at https: btttpa///giiie com/princeton pli/ rhat-makest -aa-mm--eeooddddoo -ood--oo-rreo -m. All datasets used are open-access and downloadable from the Hugging Face Hub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ blc.ps.cc/pubiic////
sondeSubmissiioPooe
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All information required understand our experimental results, including training and test details, is given in Section 4. Appendix provides additional implementa- tion details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: As specified in the relevant figure and table captions, we report error bars marking standard deviation over three runs differing in random seed.
Guidelines:
19

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix F specifies the resources required for reproducing experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttp::/neurii:: http//eurpp...//
Answer: [Yes]
Justification: This paper fully conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiciio).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work foundational in nature. As common for such research, has no immediate societal consequences.
20

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not include data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original papers for all relevant assets and include URLs when applicable.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper is accompanied by publicly available code for reproducing the experimental results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The research did not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https huop:://neuut htt.//rips.cccoofeeeeeeeeeeeeeee /ccc//oofeeeecce//222////1/ for what should or should not be described.
23

---

A Related Work
Theoretical analyses of RLHE. To the best of our knowledge, this work is the first to characterize how properties of the reward model affect policy gradient optimization in RLHF, Existing analyses primarily considered specialized algorithms that deviate from the standard RLHF pipeline (cf. Sec- tion 2.1) and provided, under varying technical assumptions, guarantees on finding an optimal policy [85, 54, 56, 94, 83, 78, 89, 39, 20, 15, 38, 67, 29, 90, 82, 36, 4J. Furthermore, [46, 72] bounded the sample complexity of estimating the ground truth reward based on preference data.
Transforming and ensembling reward models. The question of what makes a good reward model also has roots in the reward shaping literature [53]. There, prior work characterized which reward transformations conserve optimal policies [53, 31] or analyzed the convergence of value- based algorithms [19, 34]. More recently, in the context of RLHF, methods for transforming [7, 79] and ensembling [13, 62, 23, 91] reward models were proposed. Though, how these modifications to the reward model influence policy gradient optimization was not accounted for. We note that, while reward ensembling was largely suggested for mitigating reward hacking [5, 70], our optimization viewpoint sheds light models can reduce the chance of a flat RL.HF objective landscape due to low reward variance.
additional benefit it may bring. Namely, incorporating multiple reward
Optimization difficulties in policy gradient. Since the objective that policy gradient maximizes is non-concave even for simple policy parameterizations (c.g., tabular policies; Equation (3)), es- tablishing guarantees optimization rate has proven challenging [24, 48, 49, 1, 45, 50, 64, 63]. In particular, for policies that produce distributions over outputs via softmax (e.g., language models), the gradient is known vanish when the policy is near wear-deterministie [49, 48 1]. This issue was empirically demonstrated to cause slow reward maximization in several reinforcement learning envi- ronments [3, 66, 35, 49, 48, 28]. However, its relevance to RLHF is limited language models rarely produce near-deterministic distributions. More generally, [64] showed that the gradient vanishes whenever the reward variance is low. Our analysis makes of this result for studying what makes good reward model for RLHF (see proof sketch of Theorem for technical details).
It is worth noting that, in theory. the poor optimization landscape of policy gradient can be solved via natural policy gradient [40, 77, 1, 88] a preconditioning method - or entropy regulariza- tion [49, 16]. However, these approaches models. Specifically, natural policy gradient requires inverting a P x P matrix, where P is the number of language model parameters, while entropy regularization often does not yield noticeable improvements for language generation due to the large output space [64].
currently impractical in the context of large langauge
Stronger teachers do not always provide better supervision. In the context of knowledge distil- lation, [58, 86] showed that stronger language models do not necessarily generate better labels for training a student language model. This phenomenon resembles, yet is distinct from, the finding that more accurate reward models are not necessarily better teachers for RLHF (Section 3.3). Notably, in the setting of [58], strong models inherently provide weak supervision. By contrast, in RLHF, high accuracy does not imply that a reward model is a poor teacher. Rather, factors beyond accuracy can determine its effectiveness (see discussion following Theorem 2).
B Detailed Statements of Theoretical Results
In this appendix, we deliver the full statements of our theoretical results. Appendices B.1 to B.3 cover the theorems discussed in Sections 3.2 to 3.4, respectively. We refer the reader to Section 3.1 for the technical setting of the analysis.
B.1 Low Reward Variance Implies Slow Reward Maximization
Theorem is the detailed version of Theorem 1 (Section 3.2). It lower bounds the time required for the expected reward, measured with respect to any reward function, to increase by an additive constant. As shown below, this time grows inversely with Ex~s| VVary= (-1x) [ (x, y)]], regardless of whether KL regularization is applied or not. The proof of Theorem 4 is deferred Appendix D.4.
Theorem 4. Suppose gradient flow, over general autoregressive policy (Equation (2)), is used to maximize the RLHF objective with respect to a reward model FRM X x y + [-1,1] over set of prompts S (Equations (1) and (4)). For any y > O, prompt x X, and reward function
24

---

r: &' x x (e.g. can be the ground truth reward rG or the reward model TRM), denote by by the initial time at which Ey- (x)r(xxx))))))))) (x) 2 (x) + Y (by convention, by oc if no such time exists). Then, for any initial policy Ta(o):
t min { exp( -.))11 exp(-3xx)
812J7
Ex'mS Var (x ( (x', (x,))
Sac &2ull2 where J fo(x denotes the Jaco- bian of fo()' with respect to 0 and 1-112 denotes the spectral norm, L is the maximal output sequence length, and by convention exp(-5) 3X Os 0.
In the above, Jt. == 1EV
Remark For some policy parameterizations (e.g., tabular policies), the Jacobian of se with respect to 0 remains constant throughout optimization. By contrast, for more complex neural network parameterizations, it can vary. In theory, there may exist settings in which the Jacobian norm grows substantially, thereby alleviating the harm due to low reward variance. However, (Section 4) demonstrate that reward variance strongly correlates with reward increase. This indicates that changes in the Jacobian do not compensate for low reward variance in practical settings.
experiments
B.2 More Accurate Reward Models Are Not Necessarily Better Teachers
Theorem is detailed and more general version of Theorem 2 (Section 3.3). For any distribution D over unordered output pairs and almost any accuracy values a, a' E [0. 1]. Theorem 5 shows that there exist reward models TRM. F' attaining these accuracy values such that FRM leads to a substantially slower ground truth reward increase compared to FRM. where the gap in optimization rates stems from FRMM inducing lower reward variance for the initial policy. Since this holds even when a > a', Theorem 5 implies that more accurate reward models are not necessarily better teachers. The proof of Theorem 5 is deferred Appendix D.8.
Note that not all values between zero and one correspond to a valid accuracy. For example, if the ground truth reward rG assigns the same reward three different outputs, a reward model cannot misorder only one pair among them - it must either rank them all correctly or make at least two mistakes. We therefore use the following notion of attainable accuracy in Theorem 5,
Definition 3. For any prompt x E X, distribution 'D over unordered output pairs, and ground truth reward no X' y + [-1,1], 1], we say that a E [0, 1] an attainable accuracy if there exists a reward model FRM X' x - [-1,]] such that accx, (FRM)
(recall acCx, D (FRM) denotes the accuracy of FRM: see Definition 1). Furthermore, for an output y E y, we say that a is an attainable accuracy subject to y being ranked first if there exists FRM X x x + 11 such that accx,D (TRM) = a and y E argmaxy'es) (x,
Theorem 5. Suppose gradient flow, over tabular policy (Equation (3)), is used maximize the RLHF objective over a set of prompts S (Equations (1) and (4)). Given a prompt x E S, let y 0 be desired expected ground truth reward increase and y y be an output satisfying y²) >> (xymat(0s5(11x))) (ro(xx,))) + T. Denote by by the initial time at which (-narann-lx)lrrar [x] 2. Ey-TTsos (- [ra(xx y)] + (by convention, ty = oo if no such time exists). Then, for any T 2 1, initial policy T0(O), distribution D over unordered output pairs, altain- able accuracy a E [0, 1), and attainable accuracy subject to y being ranked first o' E [0, 1). there exist reward models rama; r'asa X x y [1,,]] with accx, (FRM)) and accx, (C(x) such that the following hold.
(TRM) a'
Slow ground truth reward increase under TRM- When using for maximizing the RLHF objective:
4, > 2 exp (1-
Fast ground truth reward increase under
TRM- In contrast, if the KL regularization coefficient when using for maximizing
Kocoz (1)
X 2 0 is not too large, namely X <
8( -In minyey (y)o)
the RLHF objective:
4/51
1
=
=0 T0:0)
ty < (1 - p)
(y>l)))))))) p)
25

---

where p p:= By b (n0l E (0,1).
(x
Proof sketch (full proof of Theorem 5 is in Appendix D.8). Regardless of a reward model's accuracy, it can induce an arbitrarily low reward variance for To(o). For example, 7RM can assign roughly the same reward to all outputs that have non-negligible probability under To(o) while ranking outputs according to any given order. Thus, the lower bound on t,, when using a perfectly accurate FAM readily follows by Theorem 1.? On the other hand, since the RLHF objective is non-concave, upper bounding Ly requires fine-gres fine-grained analysis of how the policy evolves during training. We show that, TRRM(X, y") is sufficiently higher than R'M (x,y y) for all y # y", then (a(o (y x] grows rapidly. This holds even when F'M is almost completely inaccurate and implies that & = (TT(OO) (y" (y" Note that the dependence on To(о) (y²|x) falls in line with intuition, by which the initial policy needs
to already somewhat competent for policy gradient to be effective [92, 64, 12).
Remark 2. Theorem 2 in Section 3.3 focuses on a special case of Theorem 5. Specifically, Theorem
considers the case where 'D is the uniform distribution over unordered output pairs, a = 1, and o' corresponds to the accuracy of reward model that orders incorrectly all output pairs, except those that include y and an output y E y that satisfies rG (x y) < ra y?). Notice that a' < 2/| since the number of unordered output pairs containing y is | - and the total number of unordered output pairs is [)[[[]] - 1)/2.
B.3 For Different Initial Policies, Different Reward Models Are Better
Theorem 6 is the detailed version of Theorem 3 (Section 3.4). It establishes that for different initial policies, different reward models can lead to a faster increase in ground truth reward. Namely, there exist reward models rRM, FRM and initial policy families II, IT' such that FRM is a better teacher for initial policies in II, while TRM is a better teacher for initial policies in IT', The proof of Theorem 6 is deferred to Appendix D.9.
Theorem 6. Suppose gradient flow, over a tabular policy (Equation (3)), is used to maximize the RLHF objective over a set of prompts S (Equations (1) and (4)). Given a prompt x E S, let > 0 be a desired expected ground truth reward increase for which there exist outputs y" y E E satisfying TG (x, y?) 1) > Vo + 7 and ra ra(x, y'7) > Vo + +,, where Vo E (-1,1). Denote by L, the initial time at By (-bx) [x) +y (by convention, L, oc no such time exists). Then, for any T 2 1, constant c (0, 1), and the initial policy families II, defined by:
which Ey-stacll- (-)r (e yx)
No EE 1Ey~=o( (xx) (ra y) = Vo, To (x TT- To(y? 2 2c. : {
IT' no 1yy-no [ra y) Vo, there exist reward models TRM: Thm X x y + 1-1, such that the following hold.
To Toly (x) 2 c).
TRM is a better teacher for initial policies in IT. If Te(o) II, when using FRM for maximizing the RLHF objective with X <
c(1-p)²
16 3e -n minyey Toroy (y x
ty (1 - (1- ( 011),
2|5]
Ey won (nor ra(xy7)-1
where p:
E (0, 1), while when using r'ms:
S (1 exp (-7)) = T(T) ty 2 V2
T'RM is a better teacher for initial policies in IT'. Te(o) E II', when using r'RM for maximizing c(1 p(1-
the RLHF objective with X < 16( + - In minyey noto; (y x))
Mooe precisely, the lower bound on t,, follows from Proposition 3 (Appendix D.6), which is analogous to Theorem 1, but applies to tabular policies instead of general autoregressive policies.
26

---

4 $ (1 - p1- ( -
2|5|
0
where By By- (x)
(0, 1). while when using TRMM
ra (x,y"(+)+1
[|] (1 exp - -T= b 2 V2
Proof sketch (full proof of Theorem 6 is in Appendix D.9). For outputs y?, ) of sufficiently high ground truth reward, we let FRM assign the same low reward to all outputs except y?, which is assigned high reward. We define rRMM similarly, but instead of assigning high reward to y assigns high reward to y"7 The initial policy families IT, IT' are chosen so that FRM and F'M induce low reward variance for policies in TT' and IT, respectively. Specifically, policies in IT give negligible probability to y"7 whereas policies in 11' give negligible probability to y?. Theorem then yields the lower bounds on 6,." For the upper bounds on E,, we follow arguments identical those used in the proof of Theorem 2. Note that we account for potential differences in optimization due to initial policy quality by taking all policies II and II' have the same expected ground truth reward.
C Perfectly Accurate Reward Models Are Optimal for Best-of-N
An alternative method to RLHF for aligning language models is Best-of-N sampling [52]. Instead of using a reward model FRM for policy gradient training, Best-of-N applies тRM only at test-time for choosing the best output from multiple candidates. Specifically, given prompt x E X, Best-of-N samples N E independent candidate outputs y' y E y from the policy и. Then, the output with highest reward according FRM is returne..
Theorem (Section 3.3) showed that perfectly accurate reward models (iee., reward models that rank all outputs correctly) can underperform less accurate models in RLHF. In contrast, for Best-of-N, Proposition I establishes that perfectly accurate reward models are always optimal. This highlights that the role of accuracy depending on the choice of alignment method.
and, more broadly, what makes good reward model can vary
Proposition 1. For any policy and reward model TRM x + [-1,1]. let π T be the Best-of-N policy induced by π and TRM, for N E N. Given a prompt x E X. suppose that TRM has perfect accuracy, i.e., accx,D( TRM is optimal for Best-of-N / in the sense that:
= 1 for any distribution D over unordered output pairs. Then,
Byn BoN
RM (1x) [ra(x,y)]
max "ha xxy-
BoN (1xx) 'IM
Proof. For candidate outputs y ..... y E V. we let y FRM (y ...., (N) be an output with maximal proxy reward FRM, ie.:
BoN
(y' y y E argmaxyelyt....
(x.
Since FRM is perfectly accurate, the ordering of yy,,..,,y" y according to FRM and ng is the same. In particular:
"ggmayyety.....y (x) = argmaxys(y) .N..." TG (x.
This implies that:
TG y BoN (y² ...., )) ,....,)))))
= maxyely- .N... (x,
Thus, for any alternative reward model r'MM= X
|-1, 1], it readily follows that:
RM Eyns (1x) (x,, = By "......" vi'd M x+1xx)
[ro y BoN (y* "y......"" y)))
Imaxye(st....h ra (x(
y x aidd xx)x)
[ro x, y.
BoN (y ,..., y
y y xx xx))))
Bynn BON (1x) (x..
RM
27

---

D Deferred Proofs
In this appendix, we provide proof's for our main theoretical results: Theorems 4 to 6.
Appendix D.1 introduces additional notation.
Appendix D.2 establishes several auxiliary lemmas.
For general autoregressive policies (Equation (2)), Appendix D.3 upper bounds the gradient norm of the KL-regularized RLHF objective by quantity that depends on the reward variance.
This generalizes Theorem [64]. which considered only non-kL-regularized objective.
Appendix D.4 proves Theorem 4, relying on the upper bound from Appendix D.3.
Moving to tabular policies (Equation (3)), Appendix D.5 includes auxiliary lemmas.
We then establish key results toward proving Theorems and 6: (i) lower bound on the reward increase rate, similar to that given in Theorem 4 for general autoregressive policies see Appendix D.6; and (ii) sufficient conditions for fast ground truth reward increase see Appendix D.7.
Lastly, the proofs of Theorems 5 and 6 are delivered in Appendices D.8 and D.9, respectively.
D.1 Notation
Recall the RLHF objective from Equation (1). defined for a policy Te!
òRLHF (0) Ex~S F x) [FRM(x (x, y)] À KL (Tx (- (-1x))
For prompt x E X' and reward model FRM y | 1, 1], we denote:
VRM x) == Ey~ro( (- x) [x,
Va(0: == Ey~ro( x) ( y)]
Furthermore, we denote the RLHF objective for single prompt E S by:
óRIHH (0; x) (RM(0;x) A KL Te, (-1x))
(x, - In T (x) T. (y(x)
Ey-sra(-(x)
Eyyaao(tbx)
[RM [RMM(X (x, y; y: 0)]
where KL FMM (x, = A In xo(vix)
Te [y1x]
Lastly, let e₁ denote the kth standard basis vector, with dimension dependent context, ||-I| denote the Euclidean norm, ||-||, denote the f norm, and ||-||2 denote the spectral norm of matrix.
D.2 Useful Lemmas: General
We make use of the following lemmas throughout the proofs.
Lemma Let g : I + E be a continuously differentiable function satisfying g(t) > and gg(() S e- (()) for all 2 O, where e > 0 and > 1 are some fixed constants. Then, for all holds that:
9(0)
g()) I
(1 c1-cpp- - 1))
For t0, dividing both sides of the inequality e {(s)² 2 _ (gg(s) by g(s)" and integrating gives:
Proof.
c.22 L' ggg(s) {s)
ds - 1
(00 (--1) g(1))((()))) (x-1)
==
Rearranging the inequality above yields the desired upper bound on g(t).
28

---

Lemma 2. Let g R - R be continuousty differentiable function satisfying g(t) > and g(t 2 c. (()) for all 2 O, where c > 0 and > 1 are some fixed constants. Then, for all
LE (
it holds that:
g(0)
9(1)
(1 - c(y 1000-)))00)-11..11 ..
Proof. The proof is analogous to that of Lemma 1.
Lemma 3. Let p.q be distributions over V, with g(y) > 0 for all y y. Then:
KL(pllg) s In minyey g(y)
Proof. We may write the KL divergence between and as:
KL(pll9) = Eyey p(y) In ply) ((y) Eves lnpl) + E yey p(y) In 1
q(y)
The upper bound on (L(pllq) follows by noticing that Eyey p(y) Inp(y)
since it is the
negative (Shannon) entropy of p. and Eyey p(y) In sty) I In minyey 4(y)
Lemma 4. Let 9 be distributions over V Then, for any y E 2 with ply) > 0 and g(y) > 0 ir holds that:
-p(y) ply) g(y)
- y(y)?
2e
Proof.
Define h: (0, 1] - IR as follows:
h(u) -2² In
y(y)
The derivative of h is given by:
d
h(u) -2u In q(y)
-2 2ln q(y)
+ +1)
du
Notice that
2nn q(y))- 1/2
-1
q(y)
and so du hgg(y)e- 1/2) 0. Furthermore, since In(u/q(y)) monotonically increasing with du huu) < O for u E (a(y)e 1/2 1]. This implies that the maximal value of h(u) over (0, 1] is attained at g(y)e 1/2 /2 i.e. for all u E (0, 1]:
respect to u, then d du h(u) > 0 for u E (0, (0y)) and d
g(y)2
h(u) un In ((y)
<
2e
from which it follows that -p(y)2 p(v)
g(y) 2e
vl
D.3 Low Reward Variance Implies Vanishing Gradient
For general autoregressive policy my (Equation (2)), this appendix generalizes the gradient norm upper bound from [64] to the KL-regularized RLHF objective (Equation (1)). Given a reward model TRM and prompt the bound depends on the reward variance with respect to the KL-regularized reward, iee. on
KL Vary [RRM (x, y; 0:
KI. where TRM (x, y:0) == (RM y) A In xo(ylx) ne (X)

---

Proposition 2. Let Th be general autoregressive policy (Equation (2)). For any prompt X E X, Kl.
reward model FRM X +1- 1-1,11 and M if RM (x.: E [-MM, for all y V, then:
V (0; x)I V B By [RM (x, y; 0)] 6M LJx Vary [T l (x, y; 0)]
In the above, Jx == supyey,tetzl fo(xyaall where J fo(x.yc)) denotes the Jacobian of fo (x, y<1) with respect to 0, and L is the oupput sequence length.
Proof. We begin by differentiating ØRLHF (f; = Ey-ro(()))) KL, (x, y; 0; with respect to 0 using the log-derivative trick:
(x)(X, A In Ta (x
VØRLHF x) V By (x)x)
70 (5(x))
Eywmoo-11x) 1x0)
KL. (RM y; 0)V In mo A -A (x))
V In [y)x(()
Since By-mnot)bx) VV In [y)[(x)] = Eyey (yo(y(x) = that:
VI O, this implies
VØRLHF(0: = By-orolly)) (I
Recall, as stated in Section 3.1. that take yr without loss of generality, i.e. all sequences in ) are of the maximal length L. Thus, by the autoregressive definition of No (y(xxx) (Equation (2)) and the chain rule, we arrive at:
V VORLHF (0:x)= By--ra(-(x)
KL. RM Xx y; 0)
L
(Insoftmax(fo(xxxe)))
Ey--t(llxx)
RM X y;
Σ y (x(y KL (x, y;
To (ey: Nel)),
where, with slight abuse of notation, Tp( |X y<1) softmax(fefxx)))) y<i.).
Now, for a constant e >> 0 to be determined later, denote by V. the set of outputs whose KL-regularized rewards deviate by more than c from the expected KL-regularized reward, i.e.:
yv = {y E e
x, y; 0) őRLHP (0; x)] c).
FIM
Defining a modified reward function F: X' x y - [- M, M] by:
KL (x, y $ V.
F(x,y) =
(RLHF((;x))
y V.
we can write VØRLHE (x; 0) as follows:
(ORLHF((:x] Σ mo(y\x)(xx,)) Σ J Te nT(11x, vx))
yEy
Σ To (rM f(x, y)] YE
Σ
J Jolxyee n0(-1x, Fe1)).
1=1
We upper bound the Euclidean norms of (/) and (II) separately. Starting with (II), by Chebyshev's inequality we know that:
Var Vary
KL RM 2
(x, y;
To (v)
S
30

---

Notice that lley. yxx)l & llyy. - mo( x.x-xl I for all y E V. Thus, for any y E Ve and E {I....,L} D}:
(ey (o(- - | (oxx))) l| |l(ey. To(- 21xx
Since the KL-regularized rewards lie in [M, by the triangle inequality this implies that:
[((T))] & 4MLJx x)()) - 4MLJx Vary xx) kT (x. y)
(5)
As for (I), denoting for yet E W-!!
a a(rat) = Eyserv in
-o)- ( ERM,
where Y2l (yb..n) YL), we have that:
(1) (1) - Σ 1=1
To(y (o a(xz))
The yth entry of ((rc)) is given by:
aly=1)=
Ey www.w//wwwwww......wxxxxxxx.....
To(y)x, <i) E Fo
-y ( (- (xxx))))
Since F satisfies by construction [f y) ØRLHF (0:x) S for all y V, bound the difference of expectations in the equation above through adding and subtracting ØRLHF (6: x) and the triangle inequality:
By me (x lF y
$ (ll y (xxx y)ll
(f:
+ @0RIHF (0;x) - Byz (-b..y [(1, S 2c.
Thus:
||atee||| s ||a(xc))|| Σ yev Te(y| 2c 2c.
Recalling that:
(T) -= Eydev- To(y< folxye.)) a a(vc))
taking the nomn of both sides and applying the triangle inequality yields:
|I(T) Σ 1=1 Ey To <o(y<1|x)JJ [
Σ n-1
2Jx c Ex To(y<(
(6)
-L.Jx. .c.
Combining the bounds on the norms of (I) and (II) (Equations (5) and (6)) then leads to:
Var Vary~ro (x)
KL. RM y:
|R (0:x)] 4MT.Jx
2LJx
Lastly, choosing = M Vary [TM (x, y; o concludes the proof:
VoRLHF (0:x)] S 6M LJxy Vary~ro((x)) [.
(x. (x,
31

---

D.4 Proof of Theorem 4
The proof assumes familiarity with the notation introduced in Appendix D.1.
We start by bounding how far the policy parameters ((t) and KL-regularized rewards can deviate from their initial values until some time T 2 0 of gradient flow. Specifically, Lemma 5 establishes that, if the reward variance that FRM induces for Tε(o) = o is low and the KL-regularized rewards bounded, then the parameters stay close to their initial values. This follows by showing that, when the reward variance is low, the RLHF objective gradient vanishes (Proposition 2) and does not rapidly increase in norm. Lemma then establishes that the KL-regularized rewards remain bounded as required in Lemma 5 for sufficiently long time. Lastly, we combine both lemmas and translate the upper bound on the change in policy parameters lower bound on by i.e., lower bound on the time until By-arec (- (x) [r(x y)l By~mssos((- [r y)] + 7. Lemma 5, Under the setting of Theorem 4, let
TE
87. Ex-s (ary (x) (x, 1/3
where JT supxex WE.. we. Saco (M, y: 0())) 8, all E S. y y. and E [0,T), then:
(-)
d
|((T)) 0(0)1l -
(tt) di
dt
3 In
<
2LJT
1 Ex-s (x [XRM y)]] 1/3 T
Proof. For all 2 0 and x S, we shorthand Vark*(((t)) = Vary and let Var (0(t)) == (arss
KL [1
RM (x, y; (()))
First, if at some 2 holds that Var (I(()) Ex~ Var KL
(0(:)) (l(K)((()))))))))))))))))
O. then Var (f(t)
for all x E S at that time since the (KL-regularized) reward variance for prompt is always non- negative. By Proposition 2. this implies that ( x) = for all x E S, and so gradient flow is at a critical point at time I, i.e. (O()))P((()) Ex-S VoRLur (0();) x) = O. Due to the uniqueness of the gradient flow solution and the existence of a solution that remains at 0()) through time, in this case (0) (t) for all 2 0. Hence, the upper bound on [Ø(T)) (()) trivially holds.
Now, consider the more interesting case where Var KL (0(t)) > 0 for all 2 0. By the fundamental theorem of calculus and the triangle inequality:
=
- 1 a ((()at)
||Q(T) 0(0)1l
< J. ll (tt))) dt
V (OL))(
For all t E [o, applying the triangle inequality, Proposition 2, and Jensen's inequality leads to:
(((RHu((((1)))) ||Ex~s @RLUP (0(1); x)]l
Ex V @RIHP (0(1)xx)]
12LJT Ex-S
< 12LJ Var KL (0(t))
Hence:
/'
T
1
T
(VOL))((((t)]]ee 12LJr l' Var KT. (o(t)
[[(T)) - 0(0)1l <
|| 0
(8)
32

---

We now bound the rate at which Var (0(t)) can increase. In turn, this will yield an upper bound on the distance between (T)) and 0(0). Differentiating Var (a(t)()(t)) with respect to time results in:
d Var (e(t)) (V Var (l((t)
B)()) (V Var (B(tt))(())))))))))))))))))))( VoRLHr(((t)))
(9)
Let us compute the gradient of the variance given prompt E S, i.e. the gradient with respect to 0 of Var (a(t)(((()) Eyey (y(x) [TRM (x. y; (()) - (())r(((();xx))
Var Varkk (0(()()))= = Σ
RRM (x y; (()) (RLHF ViTCCE) (y|x)
yEy
denote by
Te(t) (M
(x, 0(t)) (n(Hr V (0():)(0((;x)))
yEy
denote by
2A
KL (ec)) [rMM [RM
y; (x, (((()) - ØRLHF (0(1; V In mate (y(xx)
denote by
We first consider the term a. Since
VTa(e) (ylx) ma(s)) (y x) (y(xx)
TT(z) (y x V
In (x(e))
i=1
(y x)
1=1 (xces (ex, (-xx
where, with slight abuse of notation, ne(e) (- (xx, = softmax (x, yx)), we have that:
(a, VORLHF (0(t)))
Σ ToCo (y yEy
KL [TMM (x,; QRLHH
J
- (e (orL.n)))))))
< Σ
Toto (y|x)) (xx (+())) Jr
yey
eyy
(0(t))
1=1
< 24L2 Var (B(t))
Σ
Ta(h) ( (OLLH(((()):))))
y
24T.2 Var (L(tt) VLrkk(((())
where the second transition is due to the Cauchy-Schware inequality and the third is by Equation (7)
and the fact that | Te)e) y-Ix,y=)) S ||cy; Tece)-1xx vaalll, I 2.
Moving on the term b. notice that it is equal zero:
2V ((():
Σ yEy
( x)
KL [x. y; 0(t) - ØRLHF (0)
(V)RL)((0();x))
To(e) (vlx) KL (x, ORLHE (ORL)))))))))) Σ
Σ
To(L) (ylx)
yey
= 0,
33

---

and so (b, VØRLHF (0(t))) = 0. Lastly:
2XV OF (0(t):x) (*ORLHF(((t))))
Eyer (x(x))) V In To()) (y|x)
-yEE (To(o)(v/x))= EEEEE To(o) (x/x)=
2XV ORLHF (0)
Thus, going back to Equation (9) we obtain:
d dt
Var KT, (0(t))
E (a, VORLHF (0(1)) E (bb, VORLHF (0(1))) E lic, VØRLHF (0())))
x
x-S
x-.5" (a, VØRLHF (6(t)))] - 0 2A ((FLLH((((t)))
2ALL J.. Var KI, (0(t)) Var KI. (0())] (0(t))
x-
-2 Var KL (0(1)) 2A ØFLLHF (0(1))]²
$ 24412 Var KL, (0(t)
By Lemma 1, this implies that for all < 21- Var KL (B()))-
Var KL (0(0))
Var KL (0(1))
(1-8L2J3 Var KL (8(0))
Plugging this into Equation (8) yields:
[((()) - O())]l
(,- dt
0()) di
< 12LJT /' Var (o(t)
T
KL (0(0))
< <1L.Jr
dt
1 -
8L2J2 Var (T.kk(((())) (8(0))
12LJr Var (L(K)))
1
in(1-87.3 81,2 Var KL (0(0))
8L2 Var KL, (B(0))
1=
In
2LJT The proof concludes by noticing that:
1 - 81. Var (0(0))
Var (B(O)) Ex~5S
Vary~ (-bc) [
KL RM (x, y: (0))]
= Ex (10
since FRM [x, y;: ((0)) (RM(x.y) In Taco (y (x)
Teco (y [X FRM y) for all S,y e y.
Lemma 5 relies
KI. assumption that the KL-regularized reward FRM (x, y; ((t) remains bounded, for all x E S and y y. In general, however, the KL-regularized reward can be unbounded Tocyy (ylx) goes to zero. Fortunately, by tracking the evolution of the KL-regularization term in KL
TRM (x, y; (()), i.e., of X In( (y(x)//Tx,,))))) mo (ylx)), it possible to prove that it remains bounded for sufficiently long time.
Lemma 6. Under the setting of Theorem let
1 exp(
Te 0,
SL2JZ Ex-- [Var yoor (-1x) (TRM v))/
34

---

where Jr == supxexyyexxettneejaallll foce (x ll2 and, by convention, exp(- (x = 0 if 0. Then, for all x E S. y V, and & [0, T1, it holds that TRM (x, y; 0(1)) E 1-8,8)
Proof. If X = 0, the claim trivial since TMM y' E EV, and & 2 0.
(x', 0(t) = ( E [-1,1] for all x' E E,
Otherwise, if X O. assume by way of contradiction that there exist E S,y E V, and t KL RM (x, & [-8, 8). Since FRM (M y; (()) is continuous in & and initially (M (x'y'' y 0(0)) = TRM((x.))) E [-1,1] for all E and y' E 3/ the KL-regularized reward KL (x y; (t)) must be equal to either -8 or 8 before escaping the interval [-8, 8]. We denote (M.
10, Ti for which KL KL
T RM
T RM
by lx the initial time at which this occurs for some E S and y V. i.e. tx is the initial KI.
time at which / RM (x. y: f(t)) = 8. By the definition of tx it holds that: (i) tx < T and (ii) KL (x,, y': 0()) E [-8,8] 8] for x' S. E V, and S (x- RMM
Now, we bound how far the KL-regularized reward for x, y deviates from its initial value as follows:
= X In Tocea)(((x) A In To(u) (y|x)
KL NRM
(x, y; ((x))
KL FMM (x, y; 0(0))
To (y|x)
To,or (y/x)
A In Tx(e))())x)) - In TF (x/xx)
A
S A KVln ( (()) dt S (y ||| [(]|aa
where the penultimate transition is by the triangle inequality and the chain rule and the final transition is by the Cauchy- Succhy-Schwarz inequality. Note that || In To(( (y|x)]| < 2LJr because:
v In (y/
In (
J Joocx - (0y -x
$ Jr - TT(0
$ JT Σ |=| |ley. - (+x)
2T.J.
where, with slight abuse of notation, To(() (-1x, yx)) = softmax(feco) (xt). Therefore, invoking Lemma 5 with T' therein being set L yields:
KL RM
(x, y; (((x))
KR x, ((0)
SA L/a V Fa
(y
x)|
de ((t][ldt
J'v 1
< 2LJTA
< 3x ln
1-812 Excrs (Vary morooo (rex)x (x))'') 1') 1/3 tx
Since
expep -12)
1A <T
81.2 Exirs Var ( (x,3
35

---

we have that
PRN (x, y; ((xx)) -TMM (x. y: ((0)) <7.
Thus, / RRM (x. y; ((tx)) < (x - (x, 0(0)) + | (x.);
< in con-
tradiction to our assumption that |r (x, y; ((x))] =8
With Lemmas 5 and 6 in place, are now in position to complete the proof of Theorem 4. To simplify notation, we denote the expected reward that Ty achieves with respect to r and the prompt X' by V(0:x) Ey- (x) (x(xx))].
Notice that
exp
b Z 8T2J2 Ex S Var yaroo (ix))) (x' y))]
33
then we are done. Otherwise, Lemma 6 implies that KL. ), and time i fy]. By the fundamental theorem of calculus, the triangle inequality, and the Cauchy-Schwar inequality:
(M,, (x', ((t)) E [-8, 8J for all x' E S,y's e
1//
-()(()):))-V(CC (O(0); x)|
VO(()); x)dt
2t
d (((t):x) dt m
(VV x), (())))
VV(((()) x) || 11 0(t)
Applying Proposition 2, we can upper bound ||
VV (O(t))); x) as follows:
[V(((())) 6LJ,, $6LJi,
V (a (FRM y)]
where the last incquality is due to the variance of random variable bounded within [-1, being most (recall r(x,y) y) [-1, 1] for all y E Y). Thus, Lemma yields (note that t, indeed satisfies the requirement in Lemma 5):
[V(((t,):x) -V(e--())))))))))) < 6L.,, (" ll [t)
1
< 9l
- 81223, Ex'ns (Var (+1x)) [Faaaxxx y)]] 1/3 T
For V(O(t,):x) v(0(0):x) 2 7 hold, it therefore must be the case that:
7 < 9ln
1 8L2J3, Ex (1 y)]l 1/3
Rearranging the inequality concludes the proof:
ty 2 1 1-exp(-2)
1
812J3
Ex's Vary (T .y)] 1/3
36

---

D.5
Useful Lemmas: Tabular Policies
We establish preliminary lemmas toward analyzing the gradient flow trajectory for tabular policies (Equation (3)) in the proofs of Theorems 5 and 6. All policies in the lemmas below are parameterized according to Equation (3).
Lemma 7. For any two distinct prompts x E X, reward model TRM : . x y - [-1,,] 11, and
policy ma:
Vo PRLHH((:: = ④
KL. RM (x, :; Eyyora (-1x)
KL. RM (x, y: 0)] 1].
Vo. o PRLHI (0:x) O,
where recall that ( == Ey (-ix) (RM (x, 0)], with FRM (x, y; 0) == FRM (x, - x
(n(To (y (x)x (y) x)) denoting the KL-regularized reward, T(-[x) = softmax (0.xx RR is vector holding the probabilities that To assigns to all outputs, TRM (M, :0) E RDl is vector holding the KL -L-reaalarized rewards of all outputs, C denotes the Hadamard (element-wise) product, and R denotes the all-ones vector.
(To
1 E
Proof. First, V8,, ØRLHF (0: = 0 follows from the fact that, under a tabular policy, ΦRLHP (0;x) does not depend on 0xxe As for the gradient with respect to 0 by the log-derivative trick we have:
Ve ORLHF V By-w (x, A In T
Te (y|x))
= Byxxr(-1xx) (RM (x.) In my]
A A-ywto(-(x)) Vo In mo(l
Since
1x)) Vo. [x = I Eyer Vo. To((x) = Vo.. Eyer To(x)x) = v1=0,
we may write:
Vo (0; = Ey xx)
KL /RM (x, y;
8x In ly
Eyer To
Vo., lo(
X, Vo.- (0yxx - Eyer exp(ly(x) xx)
yEy Th
Σ yEy To (y| (x, (ey TT(-1x)).
Noticing that
Σ yey (M By = = 3 r (x, 0)
and
Eyey Te (M TT(-|x)= TT(--x) 3 (By~ro (x)
1)
concludes the proof.
Lemma 8. For any prompl x E X, reward model TRM X x y + 1-1, 1J. and policy na, it holds
that:
Vo PRLHE (0;x) ||: = Vo., By (1-r (MM) S Vary (x) [r (x. y: 0:
where
||-1|, denotes the & norm.
Proof. By Lemma
Ve, (0: = ||| 3 [r (x, :; - Eyr-~o [E (x, y': 0)] -1]1|
= =Eyev To(r)xx) KK (x: 0) (x)) KL (x. y': 0)]
Eyey To(y|x) . (rttirr y: Ey'xxxo(1xx)
KL RM
(x, y": 0))
37

---

Jensen's inequality then concludes the proof:
V ORIHF (0; (0;x)ll1 & E yey
(xx))((((n)) Ever To To(x/x) (rTM (x, y; R (-1x)) [ (x, xxx,)))))))
Var Vary~xo(1ns)
RM yx:)
Lemma 9. Suppose gradient flow, over tabular policy (Equation (3)), is used to maximize the RLHF objective over a set of prompts (Equations (1) and (4)). For any prompt x E -S, output y V. and time t 0 it holds that:
d dt (y|x)
1
Te(r) (y/x)))))))) (TRM((,,) - VRM
|S]
S (y(x) Σ vEy (x [rRM(xx)') (O)((());x))
X
In To()) (r/x) KK(((t))
TT(1) (v/x)²
[S]
T0(() (y/x)
Te(e) (vlx) Σ y'Ey Te(l) (y |x)² in To(2)) (y'lx) - KI((CE)
+
[s]
Ta(o)) (y'
where (on):)()): = Ey (ro and KL(((t)) (-xx
Proof. Noticing that di Toce (y|x) (v() Toce) de In To(c) (y ()), by the chain rule and the fact that only the parameters x (t) affect In TT()) (y (x). get:
d
d dt F (ylx) (x())()(x)) (Vo.. In To (x/x), (t)
-x(1)(((x)) (ey (-xx) Vo, (HLHH(((((t))
= =x(c)(((xx) - (-
S Vo,, (
Plugging in the expression for Vo. (O())((())))))))))))))))))))())) x) from Lemma then yields:
d
To(1)
dt Toco Teto (ylx)=
(ey
@ [rnm((,,) :) Vnna(((t):x)) 1I)
[S]
> Te(e) (y|n)
(ey Tots) (-1x) O In (1())
KL(((t)) =])
[S]
from which the desired result readily follows by computing the inner products.
Lemma 10. Suppose gradient flow, over a tabular policy (Equation (3)), is used to maximize the RLHF objective over set of prompts S (Equations (1) and (4)). For any prompt x E S, subset of outputs A Y, and time 2 0, holds that:
1
(A(x) To (A(x) exp
4 2A
minyey Ta(o) x) T
S
38

---

Proof. For any E A and time = O, by Lemma 9:
d
1
di (x(o)
(y(x)) [x y) (0))
S
(1)
1
(y(x) Σ Ey To(() To(o (x) (rTeM(xx.'') VTA Vina(((1);;
S
(17)
A
(ylx))
In (y
T0(O) (y|x) KL(O(E))
S
(1TT)
S (y(1) Σ Eyer
A
yey To(o) Toco (y' /x)² in (y'lx)
TT(o) (y/Ix) KTT(O(t))
(IV)
where (RR((((t::)) = Ey Tou 12) t (T)M)))( and KL((((t) == ( (-1x)) Totoo (-1x))
We upper bound (1) + (II) and (111) + (IV) separately. Starting with (4) + (11), since TRM is bounded within [-1, 1], for all y'l E y we have that:
|FRM(x, (O)) 2.
Thus, (4) + (II) can be bounded as follows:
2 (T) + (TT) < [ Tee T (y|x)) [s]
(y()))()) s |s| Toce) (y)x))
As for (IIT) + (IV), let to E [0, T| be the latest time in (0. T) at which To(o) (y|x) S To(o) x). Formally:
to = max { [0, (x(o)(x(xx) S TT( [x(x))
Note that necessarily Ta(ta) (y|x) (y/xx) since (y(o (y|x) is continuous Now, for any time 2 to we have that (n( (y/x)/ ( x)) s Along with the fact that:
0 S (t) In To(o) (x))
where y E arg ainy/e To (y' (x) (see Lemma for the upper bound), we get that:
To(a) (y' (x)² In Te()) (y'lx) Σ
(IVI)+ <
To(e)) (vlx)) In
Tore (y/x)
S
To(o) (ym x) [
Ta(() (y' xx)
y'Ey'
Using the fact In Tec y' E V, we arrive at:
In [x
Raco; [x] In Rucoy Roco)(y and TT(C) (y' xx)2 S To(c) (y' (x) for all
A
1
1
(1I)+
Te(e) (y)))²
TT (ym (x|x)
S
To(0) (y
[S]
1
22
(y(e)(((xx)
[S]
Te(u) (y
Combining our bounds on (I) + (II) and (III) + (IV), we obtain that at any time E [to. T]:
d
1
dt (ylx) <
4 2A In
To(oo To(o) (xx"
(x(e))
S
Lastly, applying Grönwall's inequality leads to:
1
Tocry Torry (ylx) < To(t)) (ylx) exp
+ 2A
-T'
[
To(u) (ym\x)
1
1
= To (vlx) exp
4 + 2X In
.
[]
To(o) (y x)
Summing the inequality over all y A completes the proof.
39

---

Lemma 11. For any prompt x E X, outputy E V, policy Te, and reference policy Tent
To(y|x)² In To(x(x)
KL(OO]
(v(xxx)
Σ
(x()2xx)
In
Ta TT(''Ix)
KL(O))
Tour (y|x)
T0, (y'1x)
Y'Ey
1
S < 2
+
mill'ey noou (y'(z)
2e
where KI(0)== = <I]0)==K[(T((| To (-xx)) (-1x))-
Proof. By Lemma we have S KL(()) S In Toes (y (x), where y E argmin To (y' x).
Thus:
To ()(x))) In TT(()x) KL(()
In To (y( xx)
Σ
To(y'1>>)
KL(0)
To.et (y|x)
Thet (y'xx)
N'EV'
To (/(())x)
In To TT(x'(x)
Tx(x/x)
Σ
To To(y'1x))
In
To., (y/x)
Tome (y'1xc)
Th (y xx)
y'Ey
S < In
(y(x))
TT(x)x) In
1
TT(x/x)
Σ To(x11x)) In
To To (y'(x) (x) Th (y
Toan (v/x)
Thre (y (x)
y'Ey
Toly (y'lx)
s 2To In
Thes (y (o(y(x)
To (2) In
Toes (y' x)
y'Ey
where the penultimate transition is due to Ey'ey nol xx) < and the last transition is by
In To (x
1
Tr (y/x)
To (y xx)
and To(y(x)2 S (x))).). Now, from Lemma 4 we know that:
To To (x)
Torn (y'1x)2
Toly' 2 In
To (y'(x) xx)
2e
Best (y 1x)
(y'l))
for all y' E y. Since Ey'ry
S Eyery
this implies that:
20
20
2e
To(y|x)) In ()((xx)
To (y'll)
KL(O)
Σ
To
To /x)²
In
KL(0)
To (y|x)
To (y'Ix)
y'y
1
1
2no (xTo(/())nn
+ (v(xx)
To (ym
2e
1
<o()/)) 2
+
miny'ey To.. (x)
2e
D.6 Improved Lower Bound on Reward Increase Rate for Tabular Policies
For general autoregressive policies (Equation (2)), Theorem 4 (Appendix B. showed that the time required for the expected reward, measured with respect to any reward function to increase by an additive constant is:
2 Ex-s (a (1x)) (RM xi]]
In this appendix, we prove a stronger lower bound for tabular policies (Equation (3)), which is used in the proofs of Theorems 5 and 6. Specifically, for any prompt S. Proposition establishes that the time required for the expected reward (with respect to any reward function r)) to increase by an additive constant is:
2 (
/TRM (oo x)) 9).
40

---

There are two main differences between the lower bounds of Theorem 4 and Proposition 3. First, the bound Theorem depends on the average reward variance over the training set S, whereas that in Proposition 3 depends only on the reward variance for E S. This difference stems from the fact that under tabular policies, for any two prompts x, x' X. the parameters governing T and (+(x(x') are distinct (while for general autoregressive policies they can be shared). Second, is the different reward variance exponents, i.e., 11/3 in Theorem 4 compared to -1/2 in Proposition 3. These exponents arise from the gradient norm upper bounds in Proposition 2 and Lemma 8. We note that the improved exponent Proposition 3 is not critical for our arguments Theorems 5 and 6.
Proposition 3. Suppose gradient flow, over tabular policy (Equation (3)), is used to maximize the RLHF objective with respect to a reward model FRM X x x [-1,1] over a set of prompts S (Equations (1) and (4)). For any O, prompt x E S, and reward function X x - [-1,1 (e.g. r can be the ground truth reward TG or the reward model TRM), denote by by the initial time 2 (x~
at which Ey (x))/rx exists). Then, for any initial policy To(o):
(1))) (r,, + (by convention, by if no such time
by 2 2/51 exp (exp(-?)
Var youtoo (-1x) (X ())
Proof. Analogously to the proof of Theorem 4 (Appendix D.4), we start by bounding how far the policy parameters (t) can deviate from their initial values until some time T of gradient flow. We will then translate this bound on the change in policy parameters to lower bound on by.
For all t 2 0, we shorthand VarxL (0(t))
Var Vary yarcc (-1x)
[
KL RM
Xx ((t))] where:
KL RM x, y; 0(1) == (x.M(x))) y) A- To(e) (y|x)
T8(0) (y|x)
Since To(s) is tabular policy, (t) evolves independently of x (t) for all x' * x. In particular,
Var VTrkk((0(L)) (0(L)) = at some L' 2 then Lemma implies that:
d di 0(x
Vo.. ΦRLHF (0(():))=0
S
In other words, gradient flow is critical point with respect to 0.x time 1', Due the uniqueness of the gradient flow solution and the existence of a solution for which (1) = 0xxx for all 2 O, we get that in this case the lower bound on by trivially holds the expected reward for never increases by > 0 from its initial value. Namely, x()(t) = (0). and so (-()) = To(o) (-1x)
and Ey~maes x) [x, y]] = Ey~ (11x) (r(xxy), for all 2
Now, 1et us consider the case where Var Var (o(t)) 0 for all 0. For any time T 2 0, by the fundamental theorem of calculus and the triangle inequality:
[10 - -00l( =
0 (t)dt
T
d 0, (t) dt
de
J
1
|| (RLL:(((()):))
|S|
Lemma 8 then gives:
1
1' V (ar (f(t))
(T)x(T) (00 <
(10)
[S]
We now bound the rate which Var (O(t)) can increase. In turn, this will yield an upper bound on (T) (0)]l. Differentiating Var KL (0(t) with respect to time leads to:
Var = =Vo.. Var &Trk)()(t)) d dt
di (t))
(11)
Vo.. Var Vark)(((t)) PRLHH
S
41

---

The gradient of Var ( Eyey (ylx)) [r (x:
ØRLHE *RLHP (0(t);x)}² with re-
spect to x (t) be written follows:
V 0x
Var KL. (B(t))
KL RRM (x. 0(t))) (0(t) Ve (x())())))
denote by
Te(1) (y(x))
RM (x x; (R(HH(((();x))) Ve.. ORLHF (0(1):x)
KL
YEJ
by
2A
Σ
To(()
(((t) - ORLHF (x Ve.. In To(()) (y(x).
RRM
yey
deuote by
We first consider the term Since
Vg. TO()) (y|x) = To(O)) (y|x) In (o(e) (x/xx
-aai (ylx) [-(c)(11x))
where, with slight abuse of notation, To(s) (-1x) = softmax(0.xx()). we have that:
a, Vo.. ØULHF (0(t;;
Σ Toin (y) (x, 0(t)) (RLHF (ey - (-)0) Vo.. (O(H)(((();
YEY
By the duality of the lo and l, norms, Lemma 8, and the fact that lley TT(o S 1, we get:
(cy- (-1x), Vo, (f(t) & cy (- (0))))))
I Var KT. (B(t)).
Hence:
(a, Vo,, ()() S V Varx= KL (0(E)) (((t))
Tocol (r)))
KL (n(nr(((t);;)))) x)]²
yEy
Var KL (
As for the term b, notice that it is equal to zero:
b = 2V (f(t);
Σ
To (y/x)
y:: (x)
òRI.I (f(t)
yEV
a (B():)
Σ
Tote
(RLHH(((():))))) Σ
Tate (ylx)
yEy
yey
((()))((((0)2))
-0.
and so (b, Vo. ØRLHE (f(t):x)) 0. Lastly:
= 21V ORLHF (0(t); x) 2AØRLHE (0(t):))
Eyer Ty(o Vo._ In Toce (y|x)
-V,,x Eyer (o())(v/x)=0
-VV 8. ORLHF (0(t); x)
42

---

Thus, going back to Equation (11) we obtain:
d
Varx Varx (0(1)) = S
a. Ve RLLH (RLHH (0(t):x))
dt
b, V x RRLH ( (0(t):x))
[
{c.
Vor ØRLHF (0(t):x))
|S|
Var
KI, (0(1)) 2X | RRIHF (0(1;))))
|
Var
KL (0(1))3/2.
8
By Lemma I, this implies that for all S 6 2//1 V
Var KL (O(O)):
Var (0(0))
Var KL. (0(t))
1 2
Var KL (0(0))
KL Notice that since "RM x y; 0(0( = (x, - A In
Toco) [y TRM y) for all E ):
Tocoo
Var (0(0)) = Var (arymmsso))-xx))
PRM (x,): (1 (x/ [FRM(xx y)].
Thus, we have that for all t < / (a()x
(0()) 2
Var (l(()) S
(1- 2/ V /Vax(((0)) ))²
where Var (0(1)) = Vary (-1x) [FRM (x, y)). Plugging this into Equation (10), for all < 2 / (O()) get that:
(T) - (0)l - [S] /'" V (((t))(((t))dt
T
Varx (0(0)
< |S| J 1- 2|5 v Varx(!(0)) -t
dt
T v Var, (8(0)) +t
Varx(((0))
2|8 Varx (0(0))
In 1
[S]
2|5
t=
2 In
2|ST V Varx(010)) T
remains to translate the upper bound on |00,x( (0)] into a lower bound on ty. To simplify notation, let ((((t); := Rxx=Tocc (-px) (-px)_((x, y)]. Since Ta(s) is a tabular policy (Equation (3)), can view (O((); x) as function of only (t), i.e., it does not depend on 0xxx(() (t) for all x' A Thus, by Lemma 8 and the fact that the variance of random variable bounded within [-1, 1] is at most 1, have that:
V (((())x))] S Vaty= 1 1.
This implies that V(O(t): is 1-I-pschhtz function of 0 (t), and so:
- x) < (T) x0)] S 2lm
1 2|S
Var (((0)) T
43

---

Now, t, 2 Varl//arx(0000)) then we are done. Otherwise, for ((((t,):x)) - V(0(0):x) 27 hold, it must be the case that:
2/5) V Varx (0(0)) (00))) t., -,)
7 S 2ln
1
1
Rearranging the inequality concludes the proof:
b 2 2|S| 21 (xxe(xxp))
v (arx(0(0)
D.7 Sufficient Conditions for Fast Ground Truth Reward Increase
In this appendix, we establish conditions the reward model, initial policy, and KI. regularization coefficient that ensure fast increase ground truth reward during gradient flow. These sufficient conditions are used in the proofs of Theorems 5 and 6.
Proposition 4. Suppose gradient flow, over a tabular policy (Equation (3)), is used to maximize the RLHF objective with respect
reward model TRM X x y + [-1, 1] over set of prompts S (Equations (1) and (4)). Given a prompt x E 8, let q > be desired expected ground truth reward increase and y+ S y be set of outputs such that every y' E y satisfies rx(x.y') > VG (e((((); x) + and (X, (x, y') > ((n)(0(0) where VG (f))):))) = By (G)
and VRM == By [/RM(X, y)}. Denote
Va(O(0); +
minyey+ rG (x) +1 E (0,1)
and assume that the following conditions hold.
The reward model TRM assigns outputs in y rewards that are higher than VRM (O():)();)) yet not too spread out:
8 = (1 max (xM( - VRM yEYt
max
|FRM((,,) v) TRMM(XXXX))) 0.
yy'
separation from initial expected (proxy) reward
spread
The KL regularization coefficient X 2 0 is not too large:
To(o; ()*+())(111 p)d
A
8D 2 minyey Recos(ylx + 2c
The initial probability assigned to Voad small:
{y eyE: :RM((X.)) (R)) \ y is
To(o)
(0++1))²
p)d
(111p)) TT:t ()+x0))+(x)) 1
20|y+1
To(()) (Dbad x) S
exp
16/
Then, denoting by by the initial time at which Vc(O(1);x) 2 Va(((0); + it holds that:
t - 4|5||++1
1
1 1) = - To(o) ()+
(1 (1- p)8 To(O)) (1+ xx) p)
Proof. If To(e ()/+ x) 2 at some t 2 0, then Vo(((t):x) 2 Va(((0):x) + since:
Va(O(t);x) 2 me(t) ()+ (x) miny (x (1 Toco) ()+ (x))
= Te(o) ()/ (x) (mingey' (x(x.)) + 1) -
(12)
2 Va(0(0); +7,
*We defined p such that (T/ [x) 2 e implies v)(O(t):)) Vo(0(0):x) + (see Equation (12)).
44

---

where in the first inequality we used the fact that the minimal possible reward is -1. This implies that Ta()) (>++ xx) p for all [0, For convenience, let us denote:
1|5|| T = (1 p)o (3++ !)
Seeking contradiction, assume b,, T. We will show that, the time interval [0, 7), the rate which Ta(r) ()++ (x) increases is sufficiently high such that To(e) ()+1(x) 2 must occur until time T.
For any E [0, T] and y Jtt, by Lemma 9 we know that:
d dt To(e) (y/x)
1
[ (x/2)) [TRM(x.)) VRM
[s] (x(c)
Y'Ey (x (x) [FRM(X, Σ (0(0)1)))((MMM((x)))) - VRM(((t);
X
(x(1)
In (x(x)
TT(() (y/x) KL(OC))
[
X
Y'EY To( (x'1x)² In To(e) (y'lx) KI.(())
S] (y())
To(o)) (y'xx) x)
where KI(((t)) ==
(1x)).
x)). Using Lemma 11 we can lower bound the contribu-
tion of the KL regularization terms:
dt Te(o)(y(x) 2
[ (x/x) (r (OI)(((1):
|S| (y()) Σ Eyer Toce) (x) (rrmx/)))((Txx(((x,,))))))))))) (O()
1
1
(xlx)
2
|S|
To(0) To (xm
2e
where ym argminy'es> To(o) (y')x). Summing over all outputs in y then yields:
d dt TT(o((+((x))
1
Σ yey+ Fa(s) (xlx)) (FRM(xx)) VRM VRM (f(t):x)
[5]
To(o) (x)
Eyerr TT(e)) (F/1x)² [FRM((x,))) (1)0))//))/()xxxxxx)))>>>) (+():)(
|S|
A
2ln 1
Toto)(D (x)
TT((0) (x) + 2e
1
(1 (2*()()*(())
Σ
y (x(1) [FRM(x - VRM(((();
[
a(y)
1
() (2it) (x) Σ
Yveryy (v( 1x)² (x, y') (R()
To(t)
[
Myy)
A T(()) (2/+(x))
+ 1
2
[S]
To(o) Tyr (ym
2e
(13)
d Towards lower bounding or To(L) (3** (x). treat separately the terms marked in the equation above by a(y), for y E y+, and b(y'), for y' E yt, Starting with a(y):
aly) Eyer Toto (y' (r(M(((xx)) (xM((xy"))
2 To(e) (2 (x) (xRM(x, y) maxyeyy (x, y'))
-y To (y' (T)M(((x)) (x,,(x)))))
45

---

By the definitions of 3 and Vadd. any y' E 3 (3* UDbad) satisfies (RM S VRM (+(): x). Thus, FRM (x. y) TRM (X. y') 2 TRM((x VRM (0(0):x) for each such y'. Adding and subtract- ing Eyerm T (y
(T (x'
(x VnM (0(0); x)) from the inequality above then leads to:
aly) 2 To(e) (J xx) (TRM y) my TRM((xy') y'))
Σ Veers- To(A) (y' (FRM - VRM (0():
Everru To( (y' xx) (x) TRM(X, y') TRM y) + (0))
Ta(() (FRM(( y) - eyx FRM y'))
+ (1- Ta(D) (>+*x) (FRM)(X,) - VRMM((()))) x)) (0(0); (VRM)((()); - (RM((xy)))]
Σ y' (y (x) [VRM
Since (i) (o()) x) < p for all t E [0,t,), (ii) (x. maxy (x, y') I 0, (iii) TRM y) "RM((0()): 2 0, and (iv) (O() TTM((x S for all y' ) (recall
TRM is bounded within |-1, 1),, we obtain:
aly)
2 (FRMM(x))) maxy'E> (x)((,y'))
+ (1 - ()(FRM(X,)) - ()M((((0):)))
2TTCC)
-xR - p max TRM((x - (1 -)) x) - -2Tece) (Dbad (x) y'Ex+
2 min FRM p. max (x.((x))) - (1- y'Ey+
-(1-p)/ax p)VRM(((00) 2xT((e) (Dhad xx)
y'Ey+
p) max (RM((xy)) (1 VR)))) y'El+
max (RM((x))) + min FRM (x, y'Ex+
y'Ex+
o (xa
8- (TT(()) (Jhadl (x).
Lemma 10, along with the assumed upper bounds on À and Taço) (Dbad (x), implies that:
(e(e)) (Jhaal xx) S TT(()) (Abadlx) exp
+ In
To(o) To(o (xmm
.T)
S
< To) (Dbad (x) exp
[S]
20
(T(()) (J++(x) ------1 )
TTo(0)(aadl exp
(1 p)ō
(2 (3* 2x)((1 p)8
16/
< 16
S 4"
where the second transition is due to:
To(o) (J++ [++/x)(11 - -p)8
1
2 In
In
To(o) (ym/
41 ( 21 (y (x)
1
To (x
2e
To(o) ())+ [x) p)6
(14)
8/++1
S1.
46

---

Note that the inequality above used the fact that S 2. Thus:
a(y) 2 å 2 (Jbad xx) 2 2
Next, we upper bound b(y") from Equation (13) for y' E D Suppose first that y' & Vaad in which case TRM y") S VRM (0(0); x). We claim that (RRM)((0); x) S (O))) x). To see notice that under gradient flow it holds that:
d dt PRLHP (0();
0.
i... (O)))) is monotonically non-decreasing. Hence: (OM((((); ( (-1x)) RLL (0(t):x)
Z RRLH (0(0):x)
VRM (f():))
Since the KL divergence between any two distributions is always non-negative, this indeed implies that (iRM((()); x) < Vin(((t)); x). Thus, (x, y') Vi) S (ft) from which we conclude that in the case of y' & Vbdd
b(y') $ O.
As a result, we can discard these terms when lower bounding =o (3*+)x). Otherwise, if E Vbad since (x. VRM (f(t):x) s 2 we have that:
b(y') s (y' x)2 S 2 (y'l
Bounding each of the a(y) and b(y') terms in Equation (13), and using the fact that Te(e) ()+((x) S P. we arrive at:
d
Σ yEv+ (y(x)) (1- p)8
di To()) ()++
[S|
2
2p
To(e) (J/ball (x)
|8|
1
2hn
To(o) /x) +
S
2e
(15)
(1
p)8
(T(()) (>*+)()))))
2
TT(() (Mbbal x)
S
1
Toce (x)
2ln
+
S
To(o) (x
2e
where the last transition is due to < and
Eyer- Toce (x/x)) 2
Eyer+ To(e) (y|x)²
y+
Σ
1
To(s) (x)
D
DD+T
which follows from Jensen's inequality. Now, as shown in Equation (14):
2A
To (ym 1.
Combined with the assumed upper bound on T0 (Dbad xx). this implics that:
Te(0) (Dinal xx)
Ta(o) ()+ 1x) (1
(1 - P)8
20
1 - 1 )
16
exp
(1 ppj
To(0) To(o) (D+|(x)
() 1x) p)8 exp
[ ((11 22
1
(1- p)8 (T (3+ xx) -
4|0*
2 1
|4+
16
To(0) Ty(o) (x)
47

---

Going back to Equation (15), invoking Lemma 10 for upper bounding Taçe) (Dbad applying the assumed upper bound on X, and recalling that
4|S||
-1)
T (1 (1-P1 p)6
TT(t)) (D+|xx)
gives:
dt To ()++/x) 2 To()) ()++1x)? (1 - p)
d
(1 - p)6 8|5 |
2| Te(o)()?)) 1x)
(x) (x) (1 -1-68 p)d
8|
At time t = 0 it therefore holds that a (Tot)) (3* (x)) 2 To(0) (V |x) a|sT > Moreover, at Teee) (3*1 |x) positive whenever To()) (J (x) 2 T0(0) (>* |x). this implies that (ott) |x)
(1-01
since d
monotonically increasing for all & [0, Thus, To:0) ()* xx) S Ta(1) (3++ (x) and:
d
(1 p)
To
(3* xx) 2 To(s) () (>x1
dt
4 D
Finally, applying Lemma we have that for all E [0.
1
()(t)()+((3) 2
(1-a)
Totos
[x]
For
4|5|1>*1
1
:)
t = T (1- p]6 TT(m) (3+ x)
the inequality above implies that Te())) (2/+ (2( 2 p. We have therefore reached contradiction as shown in Equation (12), when ma()) ()*+ (x) 2 p the expected ground truth reward is higher by at least y than its initial value, and so by must be at most T.
()
D.8 Proof of Theorem
The proof assumes familiarity with the notation introduced in Appendix D.1.
Define TRM; F'Mx= X - [-1, 1] as follows. For the prompt and y ): (RM y) = by
{ y ey EVEVY (y?)
(x, : -=
where 0 < by < T-1 for forall y E V are chosen such that their ordering achieves the desired attainable accuracy a E [0, 1]. and 0 < Cy < [y<o)("7|x)) [x), for all y E y [y"}. with the exact values of (()))))((m)) chosen such that the desired attainable accuracy subject to y being ranked first a' is achieved by 'RMM- For the remaining prompts x' of x, the reward models TRM. F'M can be defined arbitrarily since, under a tabular policy (Equation (3)), TT(e) (-xx) does not depend on 0x (t). We denote VRM((()); == Ey-ssen( x)) (x) and ViCM (0(1);x) == Ey [ (x, y)).
Remark 3. For simplicity, we defined FRM so that induces low reward variance for any policy. However, it is also possible to define FRM such that for To(o) it induces low reward variance, yet for other policies it induces a high reward variance (e.g., by having it assign a high reward to an output that is not likely under To(o) (-Ix)). Furthermore, the rewards produced by FRM need not be near-zero and can be shifted by any constant.
Proof of slow ground truth reward increase under TRM. Suppose that gradient flow is used to maximize the RLHF objective with respect to the reward model FRM- The lower bound on by stems from the reward variance induced by TRM for the initial policy =o(o) being low. Namely:
Vary~xaco (X, y)] = Σ yey To(o) (yl (x) [FRM(x.)) (RR x]]²
Σ yey Toto (xlx) Eyer (y (by - by. }
<
T
48

---

where the inequality is due to jby - byr ST for all y. E Y. Proposition 3 then implies:
by 2 = 2|8)((1 (1-exp(- -T=)(T).
Proof of fast ground truth reward increase under r'RM- Suppose that gradient flow is used to maximize the RLHF objective with respect to the reward model TRM We will use Proposition 4 for
upper bounding by- To that end, define y := { and notice that it satisfies the conditions laid
1 V (6(): (the latter trivially holds since the maximal reward is one and 0 < To(o; (y/x) < for any y V.
out in Proposition 4. Namely, rx. xx) +7 and FRM yx.
including outputs y E y [y*} whose reward under FRM is not maximal). Furthermore:
8 = (1 p) (maxye (xM(x,)) y) (00)))))):))) maxy, (RMM (x. (x.
= (1 p)(1 (+()x)())
2 (1 () To(o) (y - (1- (x)) (x(o)>>>x)))
(1 p) To(o) x)))²
where the second transition is due y containing only a single output and (x. the inequality is due to T'M
y?) and Cy < To(o) (y> for all y E y \ {yT}. Recall:
(x, (x,
p (6(0(0)))++++++) (0, rc(x,y?) +1
Thus, necessarily (y(0) < p. as otherwise, To(o)) (x 2 P. we get a contradiction:
2 + Ey To(i) (y (y)))x/)))roxx.)) (),)(x,))
2 p.ro((x To(o) (1) {x))/x)
2 p.rro(x,y") - ((1 -P
p(r +1)
= va(((0);x)+ +7.
This implies that 8 (1-p)a > O. Lastly, Vbad := (y y: (x. y) VM ())
since for all y E 3 \ ) it holds that I'm (x) = Cy < To(o) (y (x) S (80) Thus,
y+, 8, A, and Doad uphold the requirements of Proposition 4. Invoking it concludes the proof:
4|81
1
1
(1 p)o
To(() (y*
p
4|51
1
1
(1 p)
[yT))((7|x)
P)
0 To(o) (x*l))
where the second inequality is by 8 2 (1 p)
D.9 Proof of Theorem
The proof assumes familiarity with the notation introduced in Appendix D.I.
Define TRM, TRM,TRM X + [-1, 1] as follows. For the prompt and y V::
(x)
{
y y . V [
(x, {
y y
. e {y²}
For the remaining prompts x' + x, the reward models FRM, M'M can be defined arbitrarily since, under a tabular policy (Equation (3)), To does not depend on xx (t). Denote VaMa := Ey bx) [FRM(Xx (x. y)] and Vi (f(t):x) Ey Ir y)).
49

---

Proof of fast ground truth reward increase under FRM for To(o). II. Suppose that gradient flow, initialized at T0()) II, is used to maximize the RLHF objective with respect to the reward model FRM- We use Proposition for upper bounding t,- To that end, define y+ == {y²) and notice that it satisfies the conditions laid out in Proposition- 4. Namely, (x( > (c(((0): x) + y and (x,,(x)" ) = > VRM (0(0); x) (the latter trivially holds since the maximal reward is one and O < Te(o) (y) < 1 for any E V. including outputs y ) \ { whose reward under FRM is not maximal). Observe that:
= (1 p) (maxyey+ TRM((x VRM(((0);; x)) maxy (exxy.//e++/MMxxxx (x, (BN)(((0):
(1 (1-p)( p)(1
(1 - p) (E yEy To(0) (y|x)( (xMI(x,))))
= 2(1 11-1) p) -T(( (D' (x)
where the second transition is due to y+ containing only a single output and FRM (x, = and the last transition due to FRM (x, y) = - for all y E e/ {y²). ). Now, recall:
(x(( + y)) +1
1).
We claim that To(o) (y (x) < p. Otherwise, if To(O) (y (x) 2 we have a contradiction:
vc(O(0); 2 p. rx + Σ Ever(m To(o) (Tc (x)x)))((xx))
p,rr((x To(o) (2/ \ (x))/x)
2 p.rrl(x, (1-p) p)
p(ra(x,y)) +1) -1
Va(((0);xx) +7-
Thus, Toto) (2) (/))/x) 2 1 which implies that:
2 2(1 - p)3
Finally, Vhat == {y E E : (RM(X, > (OR)((()); x)} \ y+ = since for all y E ) y
holds that FR,(X,y) = -1 < VRM (0(0); x). Thus, ) ,,, X, and Dad uphold the requirements of
Proposition 4. Invoking it, we conclude:
4|8)
1 To(o) (x7|x) -1)
ty 2
(1 - 0)8
2(5)
1
< (1 p)
= 01).
Proof of slow ground truth reward increase under T'M for To(o) Il. Suppose that gradient flow, initialized at Teto) E II, is used to maximize the RLHF objective with respect to the reward model r'm- The lower bound on by follows by showing that I'M induces low reward variance for = Since Toro) [x) < T and (x, - (0(); S 2 for all E V, we may write:
Vary (1x) ( = E Toro (xlx) [FM((x,) VIMM (00):
<
4
+ Σ Everrin To(o) (xx0))>xx)))
Vi Vin (8(0)
T
(16)
For evervy y y {y"}} we have that:
(x, y) ViRM (0(0); x)
Eyey To (y' (x)(xx()((xxx))) ( -xkr(xx,')))
< Everror To(o) (x) - (kM)(xx)))) + T2
2
7
50

---

where the second transition is by the triangle inequality, the fact that (x. y) - I (x) 52
for all y.y E y. and To(a) (y' xx) I T and the last transition due to the R'M (x,
TR (x, y') -1 for all y.y' E V' {y").
Going back to Equation (16), we get that:
Var Varoo (11z) (FM (x,
-
4
+
Σ
y Tx(o) 4
T
74
4
4
+
74
8
$ T
where the second inequality is due to Eyeyl(yn) To(a) (yix) - 1 and in the last we used 7 2 T2 to simplify terms (recall T 2 1). Applying Proposition 3 we may conclude:
by 2 2||5| (1 exp (x)) v8
I exp (-2)- -T= V2
S
Proof of fast ground truth reward increase under f'M for TT(O) II'. The proof follows by steps analogous to those used for proving fast ground truth reward increase under FRM for To(o) E II.
Proof of slow ground truth reward increase under TRM for To(o) III. The proof follows by steps analogous to those used for proving slow ground truth reward increase under r'm for To(o) I1. D
E
Additional Experiments
E.1
More Accurate Reward Models Are Not Necessarily Better Teachers (Section 4.1)
Listed below are additional experiments, omitted from Section 4.1.
Figure 4 supplements Figure 2 by plotting the increase in rewards against the KI, divergence from the initial policy, instead of against the number of epochs.
Table 3, Table 4, Figure 5, and Figure 6 are analogous to Table 1, Table 2, Figure 2, and Figure 4, respectively, reporting the results of identical experiments, except that the initial policy was Pythia-IB IB language model instead of a Pythia-2.8B model.
Table 5, Table 6, Figure 7, and Figure 8 are analogous to Table 1, Table 2, Figure 2, and Figure 4, respectively, reporting the results of identical experiments, except that: (i) the initial policy was Pythia 1B language model instead of a Pythia-2.88 model; and (ii) SFT was carried out over the (UltraFeedback bbasedd reward model training set instead of AlpacaFarm.
Table 7, Table 8, Figure 9, and Figure 10 are analogous to Table 1, Table 2, Figure 2, and Figure 4, respectively, reporting the results of identical experiments, except that the ground [87] instead of ArmoRM.
truth reward model was 2tt--eemma2222---rrrdane
Table 9, Figure 11, and Figure 12 are analogous to Table 4, Figure 5, and Figure 6, respectively. reporting the results of identical experiments, except that during training reward models were normalized using a different scheme. Specifically, each reward model was normalized by the square root of the average initial reward variance over prompts in the training set, instead of by the reward standard deviation taken across training prompts jointly (see Appendix F. for details on the original normalization scheme). These experiments demonstrate that the relationship between reward variance (as computed via the original normalization scheme that accounts for reward scale) and optimization persists under other reward normalization schemes.
E.2 For Different Language Models, Different Reward Models Are Better (Section 4.2)
Listed below are additional experiments, omitted from Section 4.2.
Table supplements Figure by reporting the reward variance and accuracy (both on-policy and off-policy) for each combination of reward model and initial policy therein.
51

---

Figure 13 extends the experiment of Figure 3 by considering larger language models of roughly 3B parameters. Table 11 supplements Figure 13 by reporting reporting the reward variance and accuracy (both on-policy and off-policy) for each combination of reward model and initial policy therein.
Figure 14 reports the results of an experiment identical to that of Figure 3, except that we used GRPO [69] instead of RLOO as the policy gradient method. Table 12 supplements Figure 14 by reporting reporting the reward variance and accuracy (both on-policy and off-policy) for each combination of reward model and initial policy therein.
F Additional Implementation Details
In this appendix, we provide implementation details omitted from Section and Appendix E. Code for reproducing our results, based on the PyTorch [59] and Hugging Face TRI. [81] frameworks, can be found at https: //gpt::u///tttb.//. wwt.coo//po/oo wwtaa//rrnnetotl///eeee. pwwthhttt--p---tt---eeee caatt-maeeseeee takkes-good-zz
F1 More Accurate Reward Models Are Not Necessarily Better Teachers (Section 4.1)
Data. We took the binarized version of UltraFeedback [75], filtered out samples in which the prompt or one of the outputs exceeded 512 tokens according to Pythia [8] tokenizer, and relabeled output preferences using the ground truth reward model. The resulting training and test sets had 41419 and 1329 samples, respectively. Then, we split the training set into two subsets: 80% of the samples (i.e. 33135 samples) were used for reward model training and the rest (i.e., 8284 samples) for the policy gradient step of RLHE The test set was used for evaluating reward models and policies. Though, as mentioned in Section 4.2.1, since results over the training and test sets were nearly identical and our interest lies in optimization, which manifests in ability to maximize the reward over training prompts, we report only quantities computed over the training set.
For each prompt x and output y, we used the following chat template (unless the model already had a chat template defined, in which case we used the original chat template):
[USERI[[ASSISTAANIIYEESSSS
where [USER], [ASSISTANT], and [EOS] are defined special tokens.
Generation hyperparameters. When generating outputs from a policy, we used a temperature of 1 and maximum output length of 512 tokens
SFT. When performing SFT over AlpacaFarm [22], we took the corresponding split that contains 10000 samples. In the remaining experiments, we used the preferred outputs in the UltraFeedback- based reward model training set (using the original UltraFeedback outputs, but taking the output preferred by the ground truth reward model). We minimized the cross-entropy loss (as implemented by the TRL framework) for one epoch via the Adam optimizer [41] with a learning rate of 1e-6 and batch size of 32 (emulated via gradient accumulation steps). The learning rate was chosen to be large enough for the training loss to decrease yet not too large to ensure stability.
Reward models. We trained each reward model by minimizing the standard Bradley-Terry log- likelihood loss (as implemented by the TRL framework) for one epoch via the Adam optimizer with a learning rate of 5e-7 and batch size of 32 (emulated via gradient accumulation steps). The learning rate was chosen to be large enough for the training loss to decrease yet not too large to ensure laerarddd_oefffiieeeeeeeeeeeeeee stability. Following [23], we set the center rewards coeff coefficient hyperparameter to 0.01. For creating the perfectly accurate reward model that induces a low reward variance, we took the ground truth reward model and scaled down the rewards that it produces for 50% of the prompts in the policy gradient training set by a factor of 0.001. We note that varying the percentage of prompts and downscaling factor did not significantly modify the trends observed in the experiments of Section 4.1.
Reward normalization, To ensure fair comparison across reward models, we normalized their rewards to be on the same scale. Specifically, we sampled 500 prompts from the policy gradient training set and generated using the initial policy 10 outputs per prompt. Then, for each reward model separately (as well as the ground truth reward), we computed the reward mean and standard deviation across all of the 5000 generated outputs. Lastly, during policy gradient training and evaluation, we shifted and normalized the rewards by the corresponding mean and standard deviation.
52

---

Reward model evaluation. For each reward model and initial policy, we evaluated on-policy accuracy and reward variance over the same 500 prompts used for reward normalization, where for each prompt 10 outputs were generated. For off-policy accuracy, we used the same 500 prompts, but the original UtrrFeeebacc outputs instead of outputs sampled from the initial policy.
Policy gradient. Our RLOO and GRPO implementations are based on the RLOOTrainer class from the TRL framework, which uses the Adam optimizer. We set the learning rate lc-7, batch size to 32 (emulated via gradient accumulation steps), and the num goum_mini__m cini_batchee hyperparameter to 2. We kept the KL coefficient its default value of 0.05. Varying the learning rate or KL coefficient did not have a noticeable effect on the trends observed in Section 4. Though, as expected, too large learning rates led to instability while too small learning rates or a too high KL coefficient resulted in extremely slow progress. As in [2], for each prompt in a batch, we sampled two outputs (i.e., we set the RLOO parameter to 2). According to the TRL. framework terminology, this implies that full pass over all prompts in the training set is completed every two epochs (i.e., in TRL., an epoch measured by the number of gradient updates as opposed to the number of prompts). Lastly, all policies were evaluated based on the same 500 prompts used for reward model evaluation.
KL divergence from initial policy. Each KL divergence value reported in Figures 8, and 10 is an average across the last 20 batches of an epoch.
Hardware. All experiments model training. we used a single GPU. For policy gradient (i.e., RLOO and GRPO), we used two GPUs in runs with language models of roughly 1B parameters and four GPUs in runs with language models of roughly 3B parameters.
on Nvidia H100 GPUs with 80GB memory. For SFT and reward
F.2 For Different Language Models, Different Reward Models Are Better (Section 4.2)
The experimental setup was identical to that detailed in Appendix F.1, but instead of training reward model ourselves, we used the following publicly available reward models:
httpe///hughiigaeeecoo/lllaaaaas -tw.:aala-a-------------- 2aiilu----buut -
httpss//hugginggaacccc///rreaaae
Gemma- -BB
Furthermore, we ran policy gradient for two epochs in this set of experiments (in accordance with the common practice of running policy gradient for only few epochs [55, 37, 2D).
53

---

in ReReaadd Used for Training
2.00
Incresse in Ground Truth Reward
0.25
Reward Used for Traininng
0.00
100% policy
RM 75% on-policy
0:15
RM 50% on-policy
RM 25% on on-policy
0.40
RM 0% on-policy
RM with perfect accuracy but low variane
0.20
0.00
Graund Truth
KL Divergence
KL Divergence
Figure 4: This figure supplements Figure 2 by plotting reward increase against the KL divergence from the initial policy, instead of against the number of epochs. A benefit of accurate reward models is that they typically more KL efficient. In particular, given ground truth reward value, the KL divergence when using perfectly accurate reward model significantly smaller than when using
accurate reward model.
Table 3: This table is analogous to Table 1. presenting the reward variance (induced for the initial policy) and accuracy of reward models that are based on Pythia-IB IB initial policy, as opposed to Pythia-2.88 one. All quantities averaged across prompts in the policy gradient training (their values the test set were nearly identical). As mentioned in Section 1.1.1, ensure fair comparison of reward variance, the reward models and ground truth reward were normalized so that they produce rewards on same scale.
RM On-Policy
with Perfect Acc. but Low Reward Variance
Ground Truth Reward
100%
75%
50%
25%
0%
Reward Variance
0.660
0.625
0.626
0.605
0.374
0.131
0.239
On-Policy Acc.
0.662
0.650
0.643
0.614
0.553
1.000
1.000
Off-Policy Acc.
0.546
0.702
0.732
0.734
0.746
1.000
1.000
Reward Uscdd for Training
1.75
lnrreeae in Ground Truth Reward
0.30
Reward Used Training
1.50
0.25
RM 100% policy
RM 75% on
0.00
RM 50% on
RM 25% on-policy
on on-policy
with accuracy
rewordvarlane
Ground
Epoch
Epoch
Figure 5: More accurate reward models are not necessarily better teachers. Plotted are the results an experiment analogous to that of Figure 2, where the initial policy was a Pythia- language model instead of Pythia-2.8B model. See Table 3 for characteristics of the reward models used in this experiment.
incrassin Reward Usodfor Training
increazein Ground Truih Reward
0:30
Used Training
0.25
RM 100% on-policy
RM75% on-policy
0.20
50% on-policy
3.00
0.15
25% on-
6.75
0.30
on-policy
0.50
0.05
RM with perfect
reward variance
025
Ground Truth
0.00
0.00
0
2
10
20
KL Divergence
KL Divergence
Figure 6: This figure supplements Figure by plotting reward increase against the KI. divergence from the initial policy, instead of against number of epochs.
54

---

Table Reward variance strongly correlates with reward increase, while accuracy on own may not. For the experiments in Figure 5, we report the Pearson and Spearman correlations of different reward model properties (see Table 3) with reward increase after one epoch of policy gradient. "On- & Off-Policy Acc." refers accuracy measured both output pairs sampled from the initial policy and output pairs from UltraFeebbakk. "Reward Variance Ace." averages accuracy reward variance (induced for the initial policy). Notably, the latter combined measure is more indicative ground truth reward increase than each measure separately.
Increase in Reward Used Training
Increase in Ground Truth Reward
Pearson Corr.
Spearman Corr.
Pearson Corr.
Spearman Corr.
Reward Variance
0.941
0.943
0.409
0.428
On-Policy Acc.
0.524
0.028
0.290
0.028
Off-Policy Acc.
0.780
0.828
0,070
0.543
On- & Off-Policy Acc.
-0.690
-0.200
0.125
0.143
Reward Variance & Acc. 0.853
0.771
0.892
0.828
Table 5: This table is analogous Table 1. presenting the reward variance (induced for the initial policy) and accuracy of reward models that based on Pythia- IB initial policy that was SFTed on the (UltraFeedback- based) reward model training set, as opposed to Pythia-2.8 2.8B initial policy that was SFTed on AlpacaFarm. All quantities were averaged across prompts in the policy gradient training set (their values on the test set were nearly identical). As mentioned in Section 1.1.1, to ensure fair comparison of reward variance, the reward models and ground truth reward were normalized that they produce rewards the same scale.
RM On-Policy %
RM with Perfect Acc. but Low Reward Variance
Ground Truth Reward
100%
75%
50%
25%
0%%
Reward Variance
0.755
0.725
0.685
0.728
0.746
0.162
0.317
On-Policy Acc.
0.702
0.687
0.656
0.627
0.567
1.000
1.000
Off-Policy Acc.
0.570
0.706
0.700
0.732
0.754
1.000
1.000
Increase Rowwaard Used for Training
Incresse in Ground Truth Truth Reward
2.00
Reward Used Training
100% on-policy
RM 75% pn-polycy
RM 50% on
RM 25% on on-pollicy
RM 0% on-palicy
RM with perfect accuracy
0.50
low reward
Ground Trath
Epoch
Epoch
Figure 7: More accurate reward models are not necessarily better teachers. Plotted are the results of an experiment analogous that of Figure 2, where: (i) the initial policy Pythia- 1B language model instead of the (UltraFeedhack-basee reward model training
Pythia-2.8B model; and (ii) SFT was carried instead AlpacaFarm. See Table 5 characteristics the reward models used this experiment.
Increase Roward Used Trainnng Training
Increase in Ground rath Reward
0.50
Reward Used for Training
0.40
RM 100% on-policy
1.50
RM 75% on-
0.30
RM 50% on-
25% on-policy
2.00
0.20
on-palicy
with accuracy rewardvariand
0.10
Ground Truth
0.00
0.00
Divergence
Divergence
Figure This figure supplements Figure by plotting reward increase against KL divergence from initial policy, instead of against the number of epochs.
55

---

Table 6: Reward variance strongly correlates with reward increase, while accuracy on its may not, For the experiments in Figure we report the Pearson and Spearman correlations of different reward model properties (see Table 5) with reward increase after one epoch of policy gradient. "On- & Off-Policy Ace." refers to accuracy measured both on output pairs sampled from the initial policy and output pairs from UltraRedbbck.. "Reward Variance & Acc." averages this accuracy with reward variance (induced for the initial policy). In this set of experiments, the initial policy was SFTed on the [ltraaeeehacck-baeee) reward model training set. Compared to Tables 2 and 4, where initial policies were SFTed on AlpacaFarm, reward variance exhibits lower (yet still positive) correlation with reward increase. This difference seems to arise from reduced diversity in reward models. Namely, since SFT is performed on the reward model training set, the initial on-pon on-policy and off- policy output distributions are more similar, leading to less variation among reward models trained on different percentages of on policy and off off-policy output pairs. Indeed, shown in Table 5, these reward models induce similar reward variances for the initial policy, making reward variance less indicative of RLHF outcomes.
Increase in Reward Used for Training
Increase in Ground Truth Reward
Pearson Corr.
Spearman Corr.
Pearson Corr.
Spearman Corr.
Reward Variance
0.662
0.486
0.259
0.086
On-Policy Acc.
0.388
0,143
0.050
0.371
Off-Policy Acc.
0.874
0.943
0.507
0.771
On- & Off-Policy Acc.
-0.668
-0.486
-0.235
-0.028
Reward Variance & Acc.
0.637
0.371
0.286
0.371
This table is analogous to Table 1. presenting the reward variance (induced for the initial policy) and accuracy of reward models that are based on Pythia-IB IB initial policy. The ground truth reward model in this set of experiments was the GRM- gemmaa-22-rewaaeeeeeeeeee quantities were averaged across prompts the policy gradient training set (their values on the test set were nearly identical). As mentioned in Section 1.1.1, to ensure fair comparison of reward variance, the reward models and ground truth reward were normalized so that they produce rewards on the same scale.
Table 7:
[87] model instead of the ArmoRM model. All
RM On-Policy %
RM with Perfect Acc. but Low Reward Variance
Ground Truth Reward
100%
75%
50%
25%
Reward Variance
0.682
0.676
0.596
0.453
0.405
0.090
0.346
On-Policy Acc.
0.678
0.673
0.650
0.618
0.578
1.000
1.000
Off-Policy Acc.
0.612
0.680
0.682
0.698
0.716
1.000
1.000
Increasein Roward Uscdd for Training
2.00
Increate in Ground Reward
0.50
1.75
0.40
Reward Used Training
RM 100% on-policy
0.30
RM 75% on-
1.5
0.20
RM 50% on on-policy
3.00
RM 25% on-
RM 0% on-policy
0.00
with perfect accuracy reward varlance
0.50
Ground Truth
0.20
0.00
Epoch
Epoch
Figure 9: More accurate reward models are not necessarily better teachers. Plotted are the results an experiment analogous to that of Figure 2, where: (i) the initial policy was a Pythia-1B 1B language model instead of a Pythia-2.88 2.BB model; and (ūi GRM-gemmal B---aa2--BBBaa 2erea--mdodddd [87] served as the ground truth reward instead of AmmRRM. See Table 7 for characteristics of the reward models used in this experiment.
56

---

Reword Used Training
2.00
incrossein Ground Truth Reward
Reward Used Training
on-policy
1.30
75% on-policy
50% on-policy
3.00
0.10
RM 25% on
0.75
0.00
RM 0% on-policy
with perfect accuracy but reward variance
0.10
0.2
Ground Truth
-020
10
12
KL Divergence
KL Divergence
Figure 10: This figure supplements Figure 9 by plotting reward increase against the KL divergence from the initial policy, instead of against the number of epochs.
Table Reward variance strongly correlates with reward increase, while accuracy on its own may not. For the experiments in Figure 9, we report the Pearson and Spearman correlations of different reward model properties (see Table 7) with reward increase after one epoch of policy gradient. "On- & Off-Policy Acc." refers accuracy measured both output pairs sampled from the initial policy and output pairs from UltraFeedback. "Reward Variance Ace." averages this accuracy with reward variance (induced the initial policy). ). In these experiments, on on-policy accuracy has the highest Spearman correlation with ground truth reward increase. However, if one averages reward variance just with on-policy accuracy (i.e. excluding off-policy accuracy), then the correlations even higher: 0.809 Pearson and 0.828 Spearman.
Increase in Reward Used for Training
Increase in Ground Truth Reward
Pearson Corr.
Spearman Corr.
Pearson Corr.
Spearman Corr.
Reward Variance
0.887
0.771
0.391
0.600
On-Policy Ace.
0.641
-0.086
0..664
0.771
Off-Policy Acc.
0.813
-0.771
00176
-0.600
On- & Off-Policy Acc.
-0.744
-0.086
0.057
0.028
Reward Variance Acc.
0.725
0.771
0.732
0.714
Incrosse In Roward Used for Training
0.35
1.75
Incressein Ground Truth Reverd
0.50
Reward Used for Training
1.50
0.25
RM 100% on-policy
1.5
0.20
RM 50% on
RM 75% on-
0.33
RM 25% on
RM on on-policy
0.50
RM with perfect accuracy low reward variance
0.08
Ground Truth
0.00
0.00
Epoch
Epoch
Figure 11: accurate reward models are not necessarily better teachers. Plotted are results an experiment analogous to that of Figure 5, where reward models were normalized using a different scheme. Specifically, each reward model was normalized by the square root of the average initial reward variance over prompts in the training instead of by the reward standard deviation taken across training prompts jointly (ef. Appendix F.I). See Table for characteristics of the reward models used in this experiment.
57

---

Incrosciin Roward Usedfor Training
Incresse in Truth Reward
Training
0.25
RM 100% pnrpolicy
75% pn-policy
50% on-
0:15
25% on-po
0.75
RM on-pollcy
0.50
RM with perfect
low reword variance
0.2
Ground Truth
0.00
10
KL Divergence
KL Divergence
Figure 12: This figure supplements Figure 11 by plotting reward increase against the KL divergence from the initial policy, instead of against the number of epochs.
Table 9: Reward variance strongly correlates with reward increase, while accuracy on its own may not. For the experiments in Figure 11, we report the Pearson and Spearman correlations of different reward model properties (sce Table 3) with reward increase after one epoch of policy gradient. "On- & Off-Policy Acc." refers accuracy measured both output pairs sampled from the initial policy and output pairs from Ultraceehhakk. "Reward Variance Acc." averages accuracy with reward variance (induced for initial policy). Notably, the latter combined measure is more indicative ground truth reward increase than each measure separately.
Increase in Reward Used Training
Increase in Ground Truth Reward
Pearson Corr.
Spearman Corr.
Pearson Corr.
Spearman Corr.
Reward Variance
0.936
0.943
0.279
0.314
On-Policy Acc.
-0.517
-0.029
0.416
0.543
Off-Policy Acc.
-0.768
-0.829
0.033
-0.371
On- & Off-Policy Acc.
0,680
00.000
0.248
0,600
Reward Variance Acc.
0.856
0.771
0.808
0.714
Increasein Reward Usedloor Training
in Truih Rewoard
Reward Used for Training
0.60
0.00
GRM Gemma 2 2B
015
GRM--aam--3.2-0
0.40
RM-Tulu-V2-BB
010
2B
0.20
0.00
Pythia-288
Llama-3,2-38
Liama-32-388
Pythia-2.88
2.88
Lama-32-38
Llama-3.2-30
Figure 13: For different language models, different reward models are better. This figure extends experiments of Figure by considering additional (larger) landuage models. See Table for characteristics of the reward models.
Incresse in Reward Usedd TerrorTee Trahing
Truth Truth Roward
4.00
025
0.50
0.0
0.60
Increase in Ground
GRM Liama 3B
RM RM.Tulu-V2-8B 8B
010
RM-Gemma-28
Pythia 18 SFT
Llama:3.2 22218 SFT
Llama 3.2-1B Instruct
Pythia 18 SFT
Uama-32-18 SFT
Llama Instruct
Figure 14: For different language models, different reward models are better. Plotted are the results of experiments identical to those in Figure 3, except that we used GRPO [69] instead of RLOO the policy gradient method. See Table 12 for characteristies of the reward models.
58

---

Table 10: For each combination of reward model and initial policy in Figure 3, we report the: (i) ground truth reward increase after policy gradient (corresponds the right bar plot in Figure 3): (ii) proxy reward increase after policy gradient (corresponds to the left plot in Figure measured on- on-policy (i.e., on outputs sampled from the initial policy) and off- off-policy (i.e., on outputs from UltraFeedback). All quantities were computed for prompts in the policy gradient training set (their values on the test set were nearly identical). As mentioned in Section 4.2.1, to ensure fair comparison of reward increase and reward variance, the reward models and ground truth reward normalized so that they produce rewards same scale.
(iii) reward variance; and (iv) accuracy,
Language Model: Pythia- IB SFT
Reward Model
GRM-Gemma Gemma-2-28
PRMM-Llama3..3 3.2-BB
RM-Tulu V2-8B
RM-Gemma-2B
Reward Increase (G)
0.176 + 0.0066
0.148 = 0.001
0.173 0.0.000 0.009
0,108 + 0.011
Reward Increase (RM)
0.322 = 0.006
0.272 = 0.001
0.457 0.017
0.474 + 0,008
Reward Variance
0,405
0.528
0.544
0.549
On-Policy Acc.
0.700
0.700
0.705
0.639
Off-Policy Acc.
0.802
0.838
0.860
0.800
Language Model: Llama-3.2-10 SFT
Reward Model
GRM-Gemma-2-228
GRM-LJama-3.2-3B
MM--Tulu-V2888
RM-Gemmm-2B
Reward Increase (G)
0.280 0.003
0.278 = 0.006
0.320 0.014
0.274 0.010
Reward Increase (RM)
0.487 0.008
0.484 0.004
0.743 0.016
0.898 + 0.013
Reward Variance
0.500
0.562
0.600
0.644
On-Policy Acc.
0.716
0.737
0.737
0.676
Off-Policy Acc.
0.802
0.838
0.860
0.800
Language Model: 3.-maatt2-BBInnttuuc
Reward Model
GRM Gemma-2-2B
3RM--Llam-33 3.2-3B
RM-Tulu V2-8B
RM-Gemma-22B
Reward Increase (G)
0.105 1: 0.003
0.124 1: 0.007
0.097
0.004
0.101 0.003
Reward Increase (RM)
0.194 + 0.001
0.232 + 0.009
0.213 0.016
0.339 +: 0.003
Reward Variance
0,261
0.323
0.299
0.336
On-Policy Acc.
0.731
0.758
0.751
0.647
Off-Policy Acc.
0.802
0.838
0.860
0.800
59

---

Table 11: For each combination of reward model and initial policy in Figure 13, we report the: (i) ground truth reward increase after policy gradient (corresponds the right bar plot in Figure 13); (ii) proxy reward increase after policy gradient (corresponds the left bar plot Figure 13); (iii) reward variance; and (iv) accuracy, measured on on-policy (i.e, on outputs sampled from the initial policy) and off-policy (i.e. on outputs from Feebbced).ck). All quantities were computed for prompts in the policy gradient training set (their values on the test set were nearly identical). As mentioned in Section 4.2. 1, to ensure fair comparison of reward increase and reward variance, the reward models and ground truth reward were normalized that they produce rewards on the same scale.
Language Model: Pythia-2.8B SFT
Reward Model
GRM-Gemma Gemma-2-22
RRM--Laam-3-2 3.2-3B
RM-Tulu-V22
RM-Gemma-2B
Reward Increase (G)
0.140 = 0.005
0.167 + 0.011
0.131 0.007
0.094 0.008
Reward Increase (RM)
0.331 + 0.008
0,288 = 0.001
0.384 + 0.010
0.583 + 0.016
Reward Variance
0.419
0.518
0.527
0.540
On-Policy Acc.
0.720
0.735
0.741
0.643
Off-Policy Acc.
0.802
0.838
0.860
0.800
Language Model: Llama-3.2-3B SFT
Reward Model
GRM HRRM-Gemma--225
GRM-Llama-332-38
RM-Tulu-V244
RM-Gemma-2B
Reward Increase (G)
0.248 0.006
0.272 + 0.011
0.248 0.003
0.243 0.006
Reward Increase (RM)
0.415 0.004
0.418 0.010
0.554 0.008
0.870 0.006
Reward Variance
0.397
0.453
0.472
0.552
On-Policy Acc.
0.736
0.763
0.756
0.674
Off-Policy Acc.
0.802
0.838
0.860
0.800
Language Model: Llama-3.2-3B Instruct
Reward Model
GRM-Gemma Gemma-2-28
GRM GRM--Llam-3.. 3.2-3B
RM-Tulu V2-8B
RM-Gemma-22 2B
Reward Increase (G)
0.096 1 0.007
0.099 1: 0.007
0.091 0.004
0.094 0.007
Reward Increase (RM)
0.184 + 0.011
0,223 + 0.015
0.227 + 0.008
0.255 + 0.007
Reward Variance
0.249
0.301
0.279
0.320
On-Policy Acc.
0.714
0.743
0.730
0.651
Off-Policy Acc.
0.802
0.838
0.860
0.800
60

---

Table 12:
For each combination of reward model and initial policy in Figure 14, we report the: (i) ground truth reward increase after policy gradient (corresponds the right bar plot in Figure 14); (ii) proxy reward increase after policy gradient (corresponds the left bar plot Figure 14); (iii) reward variance; and (iv) accuracy, measured on on-policy (i.e, on outputs sampled from the initial policy) and off-policy (i.e. on outputs from Feebbceed.,,). All quantities were computed for prompts in the policy gradient training set (their values on the test set were nearly identical). As mentioned in Section 4.2.1, 1, to ensure fair comparison of reward increase and reward variance, the reward models and ground truth reward were normalized that they produce rewards on the same scale.
Language Model: Pythia- IB SFT (GRPO)
Reward Model
GRM-Gemma Gemma-2-22
FRRM-Llama3.22. 3.2-3B
RM-Tulu V2-8B
RM-Gemma-25
Reward Increase (G)
0.131 + 0.000 0.002
0.067 = 0.011
0.101 0.005
0.080
0.017
Reward Increase (RM)
0,296 = 0.010
0,229 = 0.009
0.430 + 0.018
0.477 + 0.013
Reward Variance
0.405
0.528
0.544
0.549
On-Policy Acc.
0.700
0.700
0.705
0,639
Off-Policy Acc.
0.802
0.838
0.860
0.800
Language Model: Llama-3.2-1D SFT (GRPO)
Reward Model
GRM-Gemma-2-228
GRM-Llama-332-33
RM-Tulu-V2-46
RM-Geemma-28
Reward Increase (G)
0.290 + 0.017
0.281 + 0.029
0.274 0.013
0.279
0.004
Reward Increase (RM)
0.514 0.018
0.522 0.030
0.796 0.007
1.028 0.034
Reward Variance
0.500
0.562
0.600
0.644
On-Policy Acc.
0.716
0.737
0.737
0.676
Off-Policy Acc.
0.802
0.838
0.860
0.800
Language Model: Llama 1amma-3.2-1 2B Instruct (GRPO)
Reward Model
GRM-Gemma-2
GRM HAM--Llama3... 3.2-3B
RM-Tulu-VV V2-8B
RM--emma-2BB
Reward Increase (G)
0.150 1 0.009
0.159 1: 0.003
0.139 0.002
0.140 1 0.018
Reward Increase (RM)
0,230 = 0.013
0.283 + 0.001
0.336 + 0.006
0.498 + 0.057
Reward Variance
0,261
0.323
0.299
0.336
On-Policy Acc.
0.731
0.758
0.751
0.647
Off-Policy Acc.
0.802
0.838
0.860
0.800
61


Reviews:
[{"review_id": "aVe0EDgSJG", "reviewer": "Reviewer_ngKr", "summary": "Previous work argues that, when evaluating reward models (RMs) for RLHF, we should not just focus on the RM's accuracy, that is; how often it correctly picks the higher reward completion from a (prompt, (completion1, completion2)). Instead, what really matters is *how much the RM induces the LLM policy to obtain higher true reward* (final performance).\n \nThis paper then focuses on the variance of the RM under the base policy as a component that can affect the final performance, independently of the RM's accuracy. They support this with a theorem on gradient flow RLHF, showing that the time taken scales as Variance^{-1/3}, and with empirical experiments showing that the RM's variance is correlated with the LLM's final performance at close to rho=1, and that the highest-accuracy RM does not always train the LLM the best.", "questions": "As explained in detail above, this paper claims the following hypothetical causal model:\n\nRM variance on-policy -> different final LLM accuracy\n\nI believe this other hypothesis is highly plausible, the theory or experiments don't give it a higher likelihood ratio than the previous one, and gives very different takeaways:\n\nscale of the RM on-policy -> RM variance on-policy -> different final LLM accuracy\n\nUntil these hypotheses are disentangled, or the authors successfully argue that I'm otherwise wrong, I cannot in good conscience let this paper be accepted. If this is successfully addressed, I think this is a good paper and will argue that it be accepted.\n\n## Less relevant questions:\n\n- In Table 2, how can the Spearman Correlation be literally 1.000 for variance vs. ground truth reward, when in Fig 2 the 75% model obtains more GT reward than the 100% model? We know the 100% model has more variance from Table 1. If the correlation was 1.000, this model should have more GT reward.\n\n- A possible clarity improvement. I initially could not see why RM variance would affect results, but this intuition helped me, it could go in the intro. If there is more variability in rewards depending on different trajectories, there is more information for the LLM about which trajectories to take.", "limitations": "No, see above.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\nThe paper makes a very good point that, beyond accuracy, a RM variance matters a lot to the performance of the final post-trained LLM. They support this with a theorem and thorough evaluations on a relatively small but still meaningful LLM (Pythia 2.8B). In this respects, it is a well-written, solid paper; that addresses an important problem and makes an original point (high Clarity, Significance and Originality).\n\n# Weaknesses\nUnfortunately, the paper in its current form has one large weakness. What if the effect of increased variance in LLMs affects the performance *solely due to* an increase in effective learning rate? This would make the takeaways from the paper considerably different: it's not that we should train the RMs to have high variance, or evaluate it; it's that we should normalize the RM's variance **on-policy** to be 1, so we can compare tuned learning rates and other parameters. We can normalize at the beginning, or continue normalizing during training so the RM's variance stays 1.\n\nNothing in the theory or experiments can discard this explanation, because the only independent variable in the experiments is the variance, which is very affected by the scale. If we multiply the RM by c, its variance is multiplied c^2. At the same time the policy gradients objective gets multiplied by c , thus increasing the effective learning rate of SGD by c as well.\n\nThis would perfectly explain that the theoretical time taken by gradient flow scales with Var^{-1/2} (slightly larger than the paper's lower bound of Var^{-1/3}). In the experiments, Table 1 lists on-policy variances of the RMs (which get smaller as the RM's training data becomes more out of RLHF-distribution), and these are ~perfectly correlated with the model. Again this could be because of the scale only: L260-263 states reward models are normalized, but this is done on off-policy data.\n\nTo address this, the experiments could have two treatments: one in which they normalize the RMs so they have the same variance *on-policy* and one in which the RMs are left unnormalized. If the normalization makes it so that the RMs are essentially equal, or that RM accuracy then fully explains the results, then it's the scale of the RM that is affecting things. (Though necessarily, this would completely destroy any association of the variance with the final performance: because the variance would not change. Perhaps the relevant quantity would be come whether the variance comes from occasional extreme datapoints, or whether datapoints are relatively well-distributed across the reward range. Experiments could look at the reward histograms, since they're 1D, and check.)\n\nAnother possible experiment is to search over a scale hyperparameter alpha and set RM' = alpha * RM, which should show that for the same learning rate the best induced variances are similar for all the RMs.\n\n\nUnder the scale explanation, the theory should probably be deemphasized: it is fairly obvious that scaling the reward by c necessitates adjustments to the learning rate.\n\nYou could even attempt to show extra improvements from continuously normalizing the RM during training. But this could instead be a neat paper that shows that the scale of your RM is very important for its effectiveness.", "was_revised": true, "final_justification": "The authors have shown to my satisfaction their thesis: that reward variance affects optimization, beyond just the initial scale of the reward model. Thus I am happy to accept this paper.", "rebuttal": "Thank you for the thoughtful feedback and for highlighting the clarity, significance, and originality of our work! Your review focuses on an important, yet nuanced, point regarding the relation between reward scale, “effective learning rate”, and reward variance. As detailed below, **the effect of reward variance on optimization is not solely due to differences in reward scale/effective learning rate**. In fact, for reward variance to be a meaningful measure that allows comparing reward models, rewards need to be normalized to have roughly the same scale. The paper already accounts for this and we further provide below additional empirical evidence to address your comments. Nonetheless, we recognize that the discussion of this point in the paper can be improved (currently, we mention it in the experimental section and future work section). In light of your review, we will incorporate a dedicated discussion of it after Definition 2. We hope that our response fully addresses your concern and would greatly appreciate it if you would consider raising your score. Please let us know if you have any further questions; we will gladly respond.\n\n&nbsp;\n\n\n### **Effect of reward variance on optimization is not just due to scale**\n\nFirst, we would like to emphasize that reward variance captures more than just scale: two reward models can have roughly the same scale but different reward variances (note that reward variance can be low even when the reward scale is not small). For reward models of roughly the same scale, reward variance measures the degree of separation between outputs. A useful analogy is margin in linear prediction. Just as the linear predictor needs to be of normalized scale for margin to be a meaningful measure, the scale of rewards needs to be normalized for reward variance to faithfully measure degree of separation between outputs.\n\nIndeed, as mentioned in the review, scaling up the rewards affects the effective learning rate and reward variance. Accordingly, to avoid differences in effective learning rate and allow comparing reward models based on reward variance, we followed common practice and normalized rewards to have roughly the same scale. In general, there does not exist a scale $\\alpha$ such that for two reward models RM and RM’ it holds that RM’ = $\\alpha$ * RM. So usually rewards are normalized such that if two reward models did satisfy RM’ = $\\alpha$ * RM, then after normalization they would be exactly equal. \n\n- As detailed in Appendix G, we normalized reward models based on outputs sampled from the initial policy over prompts in the training set. This ensured that the reward scale, i.e., average of absolute rewards, was roughly the same for all reward models. Interestingly, in the experiments of Figure 2, after normalization the reward models with lower reward variance (e.g., the perfectly accurate RM) had a slightly higher scale than reward models with a higher reward variance (e.g., RM 100% on-policy). Yet, the former still led to slower optimization. **Thus, the results of Figure 2 already demonstrate that the faster optimization of reward models with a higher reward variance is not due to a larger reward scale**.\n\n    *Details:* Reward scale, i.e., average absolute reward over training prompts and outputs from initial policy, for the reward models in Figure 2 (after normalization).\n    | RM 0% | RM 25% | RM 50% | RM 75% | RM 100% | Perfect RM w/ low variance | GT reward |\n    |-------|--------|--------|--------|---------|------------------------|----------------------|\n    | 0.8015| 0.7783 | 0.7834 | 0.8042 | 0.7917  | 0.8411                 | 0.8254               |\n\n\n\n- The normalization above is computed based on the initial policy. It is also possible to normalize rewards online. **We ran additional experiments, where in each batch rewards are normalized online** by the standard deviation across the batch of prompts and outputs. As shown below, the results are analogous to ones reported in the paper. This **further shows that differences in reward scale do not explain the relation between reward variance and optimization** in our results.\n\n    *Details:* To obtain results in time for the response period, we considered a setting similar to Section 4.1 with Pythia-1B models and ran a single policy gradient run per reward model as opposed to three (though, as shown in the paper, results were typically stable across random seeds). As was the case when normalizing rewards based on initial policy statistics, the perfectly accurate reward model led to a slower increase in ground truth reward than the reward models trained on 100%, 75%, and 50% on-policy data. Furthermore, reward variance was highly correlated with reward (used for training) increase, while accuracy was not, with a combination of both being most indicative of ground truth reward increase.\n\n    |                                | Reward Increase Pearson | Reward Increase Spearman | GT Reward Increase Pearson | GT Reward Increase Spearman |\n    |--------------------------------|--------------------------|---------------------------|-----------------------------|------------------------------|\n    | Reward Variance                | 0.937                    | 0.828                     | 0.292                       | 0.771                        |\n    | On-Policy Acc.                 | -0.529                   | -0.085                    | 0.416                       | 0.657                        |\n    | Reward Variance & On-Policy Acc. | 0.781                    | 0.714                     | 0.880                       | 0.942                        |\n\n    Note: We omitted off-policy accuracy for brevity. It had lower correlations than on-policy accuracy.\n\n\n&nbsp;\n\n\n### **Challenges in making online reward variance normalization work**\n\nThe above showed that the relation between reward variance (after normalization) and optimization cannot be explained by reward scale. Instead of normalizing rewards to have the same scale, an alternative option is to normalize rewards online so that each prompt in a batch has an empirical reward variance of 1. We believe that this is the experiment described in the review.\n\nFirst, it is important to note that this approach is not captured merely by reward scale or an effective learning rate since it is an **online per-prompt adaptive normalization** scheme. Second, we experimented with such approaches during our work on the project. However, **we found that such normalization can be problematic due to the difference between the empirical reward variance, which is estimated online over a few outputs, and the true reward variance** (results given below). Our results suggest that the reason is similar to why scaling rewards by a large factor $c > 1$ does not help, which we discussed in the future work section (Appendix B). Namely, the expected gradient vanishes when the true reward variance is low. Thus, gradient estimates based on a few online samples may be dominated by noise, which will be amplified by scaling. We believe that a promising direction for future work can be to further investigate whether similar normalization techniques, along with substantially increasing the number of outputs sampled online per prompt, can lead to better performance within a given compute budget.\n\n*Details:* In the setting of Section 4.1, we normalized online the rewards of each prompt separately to ensure its empirical reward variance is one. We ran experiments using two reward models – the ground truth reward and the 75% on-policy reward model – and different numbers of outputs sampled per prompt (2, 8, 16). The table below reports the increase in ground truth reward for each experiment. Notably, with 2 outputs normalization harms performance. Increasing the number of outputs sampled for each prompt led to higher ground truth reward. Yet, even with 16 outputs the imperfect reward model still worked better than using the (perfectly accurate) ground truth reward directly.\n\n|           | 2 Outputs (unnormalized) | 2 Outputs | 8 Outputs | 16 Outputs |\n|---------|--------------------------|-----------|-----------|------------|\n| RM 75% | 0.125                    | 0.115     | 0.155     | 0.152      |\n| GT          | 0.111                    | 0.092     | 0.143     | 0.145      |\n\n\n&nbsp;\n\n### **Theory**\n\nIn our analysis, we account for reward models needing to be of a similar scale by assuming that rewards are bounded within some interval – a standard assumption in the reinforcement learning literature. This precludes circumventing the lower bound in Theorem 1 by artificially multiplying all rewards by a large factor.\n\n&nbsp;\n\n### **Less relevant questions**\n\n1. As reported in Table 2, the Spearman correlation is 1 for the increase in reward used for training. For the ground truth reward it is 0.714.\n\n2. Thank you for the suggestion! This can be useful intuition alongside the connection to the objective landscape. We note, however, that it does not fully capture the reason why reward variance affects optimization. The connection between reward variance and optimization depends on the fact that language models produce distributions through a softmax function (as mentioned in the proof sketch of Theorem 1). In theory, for other, non-practical, parameterizations (e.g., each token probability is its own trainable parameter), the objective landscape is not flat when the reward variance is low. Though, we are not aware of any practical replacement for softmax that avoids this issue, so exploring whether one can construct such replacements can be a valuable direction for future work."}, {"review_id": "qXqLAW0plG", "reviewer": "Reviewer_9BJi", "summary": "The paper analyses the necessary properties for a reward model to enable effective policy optimisation in the RLHF setting. The paper demonstrates that the accuracy of the reward model is not the only factor in policy optimisation success, and that the reward variance induced by the reward model for a given policy is equally or even more important. The paper defines and investigates this reward variance in more detail, and demonstrates theoretically and empirically that low reward variance can lead to ineffective policy optimisation.", "questions": "none", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n* The paper is clear and well-written.\n* The area of study is important and significant, and the contribution of the paper is meaningful in this area.\n* The paper's approach to try and understand the topic of study rather than merely increase benchmark scores is beneficial.\n* The analysis of the paper is insightful and interesting, and sheds light on a relatively lightly understood but highly important area (RLHF). The theoretical and empirical demonstration of the discussed phenomena is compelling and beneficial.\n\n# Weaknesses\n\n* The paper has relatively few weaknesses. It would be beneficial to discuss how to achieve higher reward variance during reward model training, and why rescaling the reward function in various ways is insufficient (if it is).", "was_revised": true, "final_justification": "I maintain my justification from the review, as I had few issues with the paper.", "rebuttal": "Thank you for the support and for highlighting the significance and clarity of our work!\n\nThe future work section, which was deferred to Appendix B due to lack of space, discusses why simply scaling up the reward is insufficient and lays out potential avenues for increasing reward variance in a meaningful way. We plan to expand this discussion and incorporate it into the body of the paper given the additional content page allowed for accepted papers. Thank you for the suggestion! \n\nIf any questions arise during the discussion period, please let us know and we will gladly respond."}, {"review_id": "k1M1CBcL91", "reviewer": "Reviewer_8fPr", "summary": "This paper investigates reward model effectiveness in RLHF from an optimization perspective, arguing that reward variance is as crucial as accuracy. The authors prove that low reward variance leads to flat objective landscapes and slow optimization, even for perfectly accurate reward models. They provide theoretical analysis showing different reward models work better for different language models, with experiments on models up to 8B parameters.", "questions": "**Q1. Variance Isolation Experiments:** Can you provide controlled experiments that artificially manipulate reward variance while keeping all other factors constant? Specifically, could you demonstrate the variance effect by taking identical reward models and applying temperature scaling to increase/decrease variance, then evaluating their effectiveness as teachers? This would directly test whether variance per se (rather than data distribution matching) drives the observed benefits.\n\n**Q2. Same-Distribution Comparisons:** Would you be willing to conduct experiments where multiple reward models are trained on identical data distributions but exhibit different variance-accuracy trade-offs? For instance, training 10-20 models on the same dataset with different architectural choices or hyperparameters, then comparing high-variance/low-accuracy models against low-variance/high-accuracy ones. This would help isolate variance effects from the current confounding with data distribution differences.\n\n**Q3. Practical Variance Enhancement Methods:** Can you demonstrate validated approaches for beneficially increasing reward model variance in practice? Your limitations section mentions that directly scaling rewards by factors >1 did not yield benefits, which appears to contradict your theoretical predictions. Could you provide a detailed analysis of why this approach failed and propose alternative methods that successfully leverage your theoretical insights?\n\n**Q4. Ground Truth Validation:** Would you consider validating your experimental setup using tasks with objective ground truth (e.g., code generation, mathematical reasoning) or stronger foundation models as ground truth? The current reliance on ArmoRM as ground truth without validation against human preferences or objective correctness creates potential cascading errors throughout the evaluation.\n\n**Evaluation Criteria:** My assessment would improve to \"Accept\" (5) if you can provide convincing evidence that variance effects persist when data distribution is controlled for, and demonstrate practical methods for improving reward models through variance manipulation. The score would move to \"Borderline Accept\" (4) if you address the confounding issues with additional controlled experiments, even without demonstrating practical applications.", "limitations": "The authors acknowledge some limitations but fail to address the most critical issue: the confounding between data distribution effects and variance effects. This represents a fundamental threat to the validity of the core claims.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength:**\n\n**S1. Novel theoretical framework:** The optimization perspective on reward model evaluation is original and theoretically rigorous. The mathematical analysis connecting reward variance to gradient flow dynamics is well-executed.\n\n**S2. Important research question:** Understanding what makes reward models effective beyond accuracy is practically relevant for RLHF practitioners.\n\n**S3. Comprehensive theoretical coverage:** The progression from general autoregressive to tabular policies provides thorough theoretical grounding.\n\n\n**Weakness:**\n\n**W1. Fundamental Experimental Confounding:** The core experimental design conflates data distribution effects with reward variance effects. Models with higher variance (100% on-policy) were trained on the target distribution, while lower variance models (0% on-policy and ArmoRM) face out-of-distribution evaluation. This confounding makes it impossible to determine whether performance gains come from variance itself or from better distribution matching—a well-established factor in model performance.\n\n- **Missing Controls:** The paper lacks experiments that isolate variance effects, such as artificially manipulating variance via temperature scaling while keeping data distribution fixed, or comparing multiple models trained on identical data.\n\n- **Correlation vs Causation:** While the paper demonstrates correlation between variance and performance, it cannot establish that variance—rather than domain adaptation—drives the observed benefits.\n\n**W2. Theory-Practice Validation Gap**\n\nThe paper fails to provide convincing positive validation of its core theoretical claims:\n- **Contradictory evidence:** In the limitations section, the authors mention that directly scaling rewards by a factor >1 (which increases variance) did not yield benefits, contradicting their theoretical predictions.\n- **Insufficient analysis:** This negative result lacks detailed explanation or analysis, leaving a critical theory-experiment mismatch unresolved.\n- **Limited practical guidance:** While the paper shows that artificially reducing variance hurts performance (unsurprising), it provides no validated methods for beneficially increasing variance—the practically relevant direction.\n\n**W3. Questionable Ground Truth Validation**\n\n**Unvalidated Ground Truth:** The paper uses ArmoRM as \"ground truth\" without validating its alignment with actual human preferences or objective correctness. This creates potential cascading errors throughout the experimental evaluation.\n\n**Better Alternatives Available:**\n- Human evaluation to verify the ground truth model's actual accuracy\n- Stronger generative models (e.g., Claude-4 Sonnet, DeepSeek-R1) as more reliable ground truth\n- Tasks with verifiable ground truth (e.g., code generation, mathematical problems) where correctness can be objectively determined\n\nTo validate the core hypothesis, the paper needs experiments that:\n- Artificially increase variance through methods like temperature scaling while maintaining identical training distributions\n- Compare multiple models trained on identical data to isolate variance effects from accuracy effects\n- Use tasks with verifiable ground truth (e.g., code generation, mathematical reasoning) where correctness can be objectively determined or use stronger generative models (e.g., Claude-4 Sonnet, DeepSeek-R1) as more reliable ground truth", "was_revised": true, "final_justification": "I have increased my score to 4. The authors have addressed my primary concern regarding the confounding of data distribution and reward variance. Their new experiments, which trained multiple reward models on an identical data distribution while varying hyperparameters to produce different variance-accuracy trade-offs, provide the necessary controlled comparison. While I still have some reservations about the practical applicability and the full explanatory power of the current theoretical framework, the additional evidence is enough to show that reward variance is a significant factor, independent of data distribution.", "rebuttal": "Thank you for the thoughtful feedback and for highlighting the importance of our results. We treat your comments and questions below. In particular, we emphasize that to a large extent the concerns raised in the review are already accounted for in the paper and provide additional experimental results, as requested in the review. If our response addresses your concerns, we would greatly appreciate it if you would consider raising your score. Please let us know if you have any further questions; we will gladly respond.\n\n&nbsp;\n\n**W1 and Q1: Experiments isolating confounding factors.**\nWe first want to emphasize that **the paper already includes experiments that isolate the effects of reward variance** from potential confounding factors (e.g., data distribution over which the reward model was trained). Specifically, in the experiments of Section 4.1 and Appendix F.1, the perfectly accurate reward model with low reward variance (e.g., the red reward model in Figure 2) is obtained from the ground truth reward by reducing in a controlled manner how well it separates different outputs while preserving their rankings. Thus, the ground truth reward (in yellow) and the perfectly accurate reward model with low reward variance (in red) were trained on the same data, have the exact same (perfect) accuracy, yet differ in their reward variance due to the controlled intervention. As our results show (e.g., Figure 2), the perfectly accurate reward model with low reward variance substantially underperforms the ground truth reward, in the sense that it leads to substantially slower optimization.\n\n&nbsp;\n\n**W1 and Q2: Further experiments comparing reward models trained on the same data.**\nThis is a good point! Thank you for raising it. Studying how reward variance and accuracy affect the outcome of RLHF requires reward models that vary in these properties. Our approach of training reward models on different data mixtures is one way of achieving this. Indeed, it is also possible to train reward models over the same data and vary instead different training hyperparameters. \n\nWe first note that, as discussed in the response to the previous comment, Section 4.1 and Appendix F.1 already include experiments that demonstrate the effect of reward variance for reward models trained on the same data distribution. To further address your concern, as requested in the review, during the response period **we conducted additional experiments with reward models trained on the same data distribution** (details and results are provided below). The results are analogous to the ones reported in Section 4.1, demonstrating that the connection between reward variance on optimization persists when reward models are trained on the same data.\n\n*Details:* We considered a setting similar to Section 4.1 and trained, based on the Pythia-1B language model, 10 reward models over the same data (100% on-policy) using different training hyperparameters (e.g., learning rate and reward centering regularization coefficient). To obtain results in time for the response period, using each reward model we carried out a single policy gradient run as opposed to three (note that, as shown in the paper, results were typically stable across random seeds). Analogously to Section 4.1, we found that more accurate reward models are not necessarily better – e.g., a reward model with 0.67 accuracy but low reward variance led to an increase of 0.07 in ground truth reward while another reward model with 0.65 accuracy and higher reward variance led to an increase of 0.12 in ground truth reward. Furthermore, the table below presents the correlation between different reward model properties and increase in reward used for training and ground truth reward. As was the case in our original experiments, reward variance is highly correlated with increase in reward (used for training), while accuracy was not, with a combination of both being most indicative of increase in ground truth reward.\n\n|                                | Reward Increase Spearman | GT Reward Increase Spearman |\n|--------------------------------|---------------------------|------------------------------|\n| Reward Variance                | 0.588                     | 0.576                        |\n| On-Policy Acc.                 | -0.696                    | 0.519                        |\n| Reward Variance & On-Policy Acc. | 0.455                     | 0.661                        |\n\nNote 1: We omitted off-policy accuracy for brevity. It had lower correlations than on-policy accuracy.\n\nNote 2: The reward models were trained using learning rates 1e-7, 5e-7, and 1e-6, reward centering regularization coefficients 0, 0.1, 10, and scaling factors applied to the rewards of a fraction of the prompts (after scaling we still normalized reward models such that on average over prompts they have roughly the same scale).\n\n&nbsp;\n\n**W2 + Q3: Theory-practice validation gap.**\nWe respectfully disagree that the “paper fails to provide convincing positive validation of its core theoretical claims”. Each of our theoretical claims is backed by extensive empirical evidence in Sections 4.1 and 4.2 and Appendix F. Specifically, our theory establishes that: (i) low reward variance can lead to slow optimization, (ii) more accurate reward models are not necessarily better teachers, and (iii) for different language models different reward models can work better. Our experiments clearly demonstrate that these conclusions apply in practical RLHF settings.\n\nFurthermore, the discussion in the future work section on why naively scaling the reward by a constant $c > 1$ is problematic **does not contradict our theoretical analysis**. Namely, by no means does our analysis suggest that naively scaling the reward by a large factor will improve performance. As explained in the future work section, when the reward variance is low the expected reward gradient vanishes. Thus, sample-based estimates of the gradient may be dominated by noise, which will be amplified by such reward scaling. We note that for reward variance to be a meaningful measure that allows comparing reward models, one must normalize reward models to have roughly the same scale, which is standard in practice and is already accounted for in our experiments and theoretical analysis. In light of your comment, we will clarify this point in the manuscript.\n\nRegarding practical guidelines. The main purpose of our work is to advance the understanding of what makes a good reward model. While it does not give a solution to the issue of low reward variance, it highlights limitations of existing reward model benchmarks and provides insight into what determines the effectiveness of a reward model, beyond accuracy. We hope that our work inspires further research into reward model training procedures that encourage higher reward variance, as well as policy gradient methods that can optimize more efficiently under low reward variance. The future work section outlines a couple of possible directions going forward. However, due to the extent of our theoretical and empirical results, we believe that this important and highly non-trivial endeavour falls outside the scope of the current paper.\n\n&nbsp;\n\n**W3 + Q4: Ground truth validation.**\nAs discussed in Section 4, since evaluating policies via human labelers can be prohibitively costly, it has become common practice to use an existing large reward model as the ground truth reward (e.g., see references [27, 13, 74, 7, 10, 81, 73] in the paper). This is especially true for studies, such as ours, whose main goal is to further the understanding of different aspects of the RLHF pipeline, as opposed to training state-of-the-art models. The ArmoRM reward model in particular has already been used as the ground truth reward in numerous papers, and was validated in [77] to accurately accord with existing labeled preferences (better than GPT4 judges).\n\nMoreover, using an existing reward model as the ground truth reward actually allows avoiding any potential “cascading errors” and inconsistencies due to noise in human preferences, since the exact same ground truth reward is applied for both labeling preferences and evaluating policies. Thus, **in our experiments, the ground truth reward already plays the role of a verifiable reward**, as you suggested we use in your review. Furthermore, note that we also verified our results are robust to the choice of ground truth reward — Appendix F provides experiments showing analogous results with a different ground truth reward."}, {"review_id": "kpCFH2fnIU", "reviewer": "Reviewer_PFVe", "summary": "This paper raise an question about what makes a reward model a good teacher for RLHF. For current literature, reward accuracy is consider as the most significant factor. However this paper show that the low variance of reward model will also cause slow learning.", "questions": "1. Could the authors provide more background on how this specific problem differs from classical RL challenges, such as sparse reward settings?\n2. Could the authors include an additional experiment using the same reward model but with different reward scaling factors, to directly assess how reward scale and induced variance affect learning performance?\n3. During training, do all methods use the same input prompt x during the same iteration (x for iter i may be different than i+1 iter but within i iteration will method have same set of x)? If so, this would strongly support the validity of the theoretical analysis presented in the paper.\n\nI am willing to increase the score if the auther provided a clearer explanation of the unique challenges this problem presents, especially in contrast to classical RL issues, and addressed some of the weaknesses in its formulation and assumptions.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "My research mainly focus on RL/DRL control so I cannot thoroughly evaluate this paper from the LLM perspective. I will more focus on the fundamental RL perspective.\n\n>Strengths:\n1. the writing is clear.\n2. this paper proves that even a perfectly accurate reward model can lead to slow learning if it induces low reward variance.\n3. the paper has comprehensive experiments with different models and different datasets.\n\n>Weaknesses:\n\n1. The novalty of this paper. In RL, the dense and sparse reward has been studied for decades. The sparse reward problem (naturely very low reward variance) is very similar to the problem studied in this paper. It is common sense that sparse reward is challenging than the dense problem. Additionally, the paper does not provide solution to this issue. However, RL and DRL has been studied and prove a lot useful framework such as reward shaping or transfer learning. \n2. Studying variance along is not sufficent. In [1], it has been shown that the reward scale can significantly impact performance. Even when using the same reward model, applying different scaling factors results in different reward scale and variances will affect performance.\n3. The definition of reward variance which is based on same x but different y. This formulation ignore the fact that during batch training, within the batch the data tuple are from differet x and y.\n\n[1] Henderson, Peter, et al. \"Deep reinforcement learning that matters.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.", "was_revised": true, "final_justification": "The rebuttal have addressed most of my concerns. However, I cannot give full accept assessment (5) since the reward scale issue. From the result provided by auther, this issue seens minor however from the fundamental mathmatic perspective, the change scale cannot be equalivant to change learning rate. For example, if you change learning rate, every model change the same amount which makes the comparsion fair. However, when normalize to the same scale, different model will be applied different factors and it will be even worse for the variance change (will be the square of the factor).", "rebuttal": "Thank you for the thoughtful feedback and for highlighting its clarity, the theoretical results, and their comprehensive empirical validation. We treat your comments and questions below. If our response addresses your concerns, we would greatly appreciate it if you would consider raising your score. Please let us know if you have any further questions; we will gladly respond.\n\n&nbsp;\n\n**W1 and Q1: Relation of low reward variance to sparse rewards.**\nWhen rewards are sparse (i.e., only a few outputs/trajectories receive a non-zero reward), often the reward variance is low. However, the reward variance can be low even if rewards are not sparse (i.e., dense). Importantly, in RLHF rewards are typically not sparse – reward models provide a reward for each output, which is usually non-zero. As our work highlights, despite rewards not being sparse, policy gradient still suffers from a flat objective landscape that leads to slow optimization when reward variance is low. This occurs when the reward model does not sufficiently distinguish between outputs that are probable under the policy (i.e., it gives them a similar, not necessarily near-zero, reward) and stems from the objective landscape induced by policies that produce a distribution over outputs via the softmax function.\n\nAs discussed in the related work section (Appendix A), the literature on reward shaping can indeed be seen as relevant to our work. Though, to the best of our knowledge, prior work did not characterize how reward shaping affects policy gradient optimization. In the related work section we further discuss the fact that, in theory, the poor optimization landscape of policy gradient can be solved via natural policy gradient (a preconditioning method). However, this approach is currently impractical in the context of large language models. Specifically, natural policy gradient requires inverting a $P \\times P$ matrix, where $P$ is the number of language model parameters.\n\nLastly, while our work does not give a solution to the issue of low reward variance, it highlights limitations of existing reward model benchmarks and provides insight into what determines the effectiveness of a reward model, beyond accuracy. We hope that it can inspire future work on developing reward model training procedures that encourage a higher reward variance and policy gradient methods that can optimize more efficiently under low reward variance. Due to the extent of our theoretical and empirical results, we believe that this important and highly non-trivial endeavour falls outside the scope of the current paper.\n\n&nbsp;\n\n**W2 and Q2: Reward scale.**\nYou raise an excellent point! It is true that reward scale can impact performance. In particular, it can have an effect similar to modifying the learning rate (though not always exactly the same effect when applying KL regularization and using adaptive optimizers). For this reason, to ensure fair comparison across reward models, in our experiments we normalized all reward models so that they produce rewards of roughly the same scale. This normalization is also necessary for ensuring that reward variance is a meaningful measure when comparing different reward models.\n\nNonetheless, per your request we ran experiments using the same reward model, in which we scaled the reward by different constants. As expected, lower reward scales led to slower optimization while the larger reward scales that we tried led to unstable optimization that resulted in a lower ground truth reward.\n\n*Details:* To obtain results in time for the response period, we considered a setting similar to Section 4.1 with Pythia-1B models and ran a single policy gradient run per scale factor (as shown in the paper, results were typically stable across random seeds). In all runs, we used the RM trained on 75% on-policy data.\n\n| Scale | Ground Truth Reward Increase |\n|-------|------------------------------|\n| 0.001 | 0.022                        |\n| 0.01  | 0.039                        |\n| 0.1   | 0.110                        |\n| 1     | 0.109                        |\n| 10    | 0.080                        |\n| 100   | 0.086                        |\n| 1000  | 0.083                        |\n\nAs noted in the implementation details appendix, we observed similar results when tweaking the learning rate. Trends across reward models of the same learning rate remained intact.\n\n&nbsp;\n\n**W3: Definition of reward variance.**\nSimilarly to accuracy and many other evaluation measures, reward variance is defined for a given prompt/input. Such measures are typically aggregated over a set of inputs by taking the mean. In that regard, there is nothing special about reward variance and its definition. Indeed, the reward variance values we report in Table 1, as specified in its caption, are an average over the training prompts. In other words, the definition of reward variance does not ignore the existence of multiple examples in a dataset or batch, both in theory and in practice.\n\n&nbsp;\n\n\n**Q3: Prompt order during training.**\nWe did not fix the order of prompts in the experiments of Section 4. Instead, we used three different seeds for random data orders and reported the standard deviation of results across the seeds. As our results show (e.g., the almost indiscernible error bars in Figure 2), the data order did not have a significant impact on the results. Nonetheless, as detailed below, during the response period **we ran additional experiments to further verify** that our results are robust to fixing the data order.\n\n*Details:* To obtain results in time for the response period, we considered a setting similar to Section 4.1 with Pythia-1B models and ran a single policy gradient per reward model (as shown in the paper, results were typically stable across random seeds). We include a table analogous to Table 2, showing the correlation between different reward model properties and the increase in reward used for training and ground truth reward. As was the case in our original experiments, reward variance is highly correlated with reward (used for training) increase, while accuracy was not, with a combination of both being most indicative of ground truth reward increase.\n\n|                                | Reward Increase Pearson | Reward Increase Spearman | GT Reward Increase Pearson | GT Reward Increase Spearman |\n|--------------------------------|--------------------------|---------------------------|-----------------------------|------------------------------|\n| Reward Variance                | 0.958                    | 0.942                     | 0.426                       | 0.600                        |\n| On-Policy Acc.                 | -0.550                   | -0.028                    | 0.225                       | 0.485                        |\n| Reward Variance & On-Policy Acc. | 0.788                    | 0.771                     | 0.864                       | 0.771                        |\n\nNote: We omitted off-policy accuracy for brevity. It had lower correlations than on-policy accuracy."}, {"review_id": "o9HR5BfmId", "reviewer": "Reviewer_G5Cf", "summary": "This paper studies the effects that reward variance has on RLHF fine-tuning. Specifically, the paper claims that low reward variance (measured in terms of how well the reward separates different responses), although coupled with high reward accuracy (measured in terms of how 'aligned' with the ground-truth reward the proxy is), may imply very slow convergence of the regularized policy optimization step in RLHF. The authors provide a systematic study of the phenomenon. They start by stating that convergence (in the sense of the increase in proxy reward beyond a certain threshold) is inversely proportional to the used reward variance. Then, under tabular softmax parametrization, they state that the same policy has much faster convergence when using a proxy reward with much lower accuracy (but higher variance) than a given reward. They also provide an additional result stating that different policies have different sensitivities to this phenomenon. Finally, they provide experimental results on language models to corroborate their theoretical findings. They use Pythia and Llama models on different proxy rewards (which have been manually tuned to induce high variance, with consequential lower accuracy) and show that higher variance does indeed imply faster growth, although coupled with lower accuracy at times.", "questions": "1. There is an important question related to the factors which are hidden in the statements provided in the main paper. When looking at the formal statements in the Appendix, the bounds seem to depend on the maximum norm of the Jacobian of the network over iterations. First, it is a bit strange to consider this dependence as irrelevant. To me this seems like a very important factor because it contains a lot of information on the landscape of the current network. Second, it is not clear to me how large or small this quantity is or what is its relation to the variance of the reward. I think this deserves some discussion. Can the authors please explain first the connection between variance and Jacobian in this particular setting? I understand that the reward does not depend on it. But surely the reward affects it over time, since it reflects changes that are occurring through updates until convergence. \n2. To what extent is the theory here applicable to the standard RL setting? I assume the reward variance might have direct implications in RL due to poorly designed rewards. In this context, what is the relationship between a high-variance reward which might facilitate faster convergence, and potential-based rewards? As far as I understand (correct me if I'm wrong), reward variance as you have defined it directly corresponds to the average squared advantage function, which might directly provide the intuition of the relation between variance and gradient. \n3. As mentioned in the previous section, it is indeed unfortunate that the analysis is not fully complete. I understand that non-concavity is a serious issue when analyzing the general setting, but, as you point out, the tabular instantiation does not resolve non-concavity. So, in itself, this might not be the real issue. I think this might be an important gap. Can you provide more details on the challenges you face when considering the general setting? And also, what challenges did the current analysis pose? \n4. How does the inexact gradients exacerbate/ameliorate the issue? Due to stochasticity, might it not be the case that gradient estimates might somehow amortize the effect of low reward variance? Can you comment on this? Is it possible to provide additional analysis for inexact gradients? Would that be too challenging?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The paper studies a very important problem and provides interesting insights that are relevant to theoreticians and practitioners in RLHF. \n2. The paper is very well-written, clear and coherent. The results are well-explained and associated with corresponding intuition. \n3. Theoretical results support all the claims made in the paper.\n4. Finally, the paper includes experiments on actual language models for all the settings that are discussed. \n\n**Weaknesses**\n1. An immediate weakness of the paper is related to the scope of the provided theoretical intuition. Although I agree with the authors that theoretical analysis is hard beyond tabular settings, having one result on general differentiable functions, and then having additional results on tabular instantiations provides an incomplete theoretical picture. (See Questions for more details).\n2. In addition, it is not clear at this point to what extent the analysis done in this paper borrows (or is a direct corollary of) the analysis done in [65].\n3. Experimental results in additional benchmarks would significantly strengthen the impact of the paper.", "was_revised": true, "final_justification": "I believe the paper's results are overall significant to the NeurIPS community. The paper provides both justified theoretical claims and support them with experiments on relevant benchmarks. The major issue that I had with the paper was adequately resolved by the authors. Thus, I increase my rating accordingly.", "rebuttal": "Thank you for the thoughtful feedback and for highlighting the interesting insights that our work provides and its clarity. We treat your comments and questions below. If our response addresses your comments, we would greatly appreciate it if you would consider raising your score. Please let us know if you have any further questions; we will gladly respond.\n\n&nbsp;\n\n**W1: Scope of theoretical results.**\nThe core issue that we analyze – slow optimization due to low reward variance and its interplay with accuracy – does not depend on the neural network parameterization. It stems from the RLHF objective and applies to any softmax parameterized policy. That is, any policy that produces a distribution over outputs through the softmax function, which includes language models as a special case. \n\nOur work includes three main theorems showing that: (i) the time it takes to increase the reward grows inversely with the initial reward variance (Theorem 1), (ii) more accurate reward models are not necessarily better teachers (Theorem 2), and (iii) for different language models different reward models can work better (Theorem 3). Indeed, while we prove Theorem 1 for general differentiable parameterizations, Theorems 2 and 3 are proved for tabular policies since they require establishing separations between the optimization time of reward models, which entails substantial technical difficulty (as further discussed below in response to other comments). However, given that the phenomenon that we analyze stems primarily from the objective rather than any particular neural network parameterization, we do not believe that this is a major limitation. This belief is strengthened by the experiments of Section 4, which empirically demonstrate that conclusions drawn from our theory carry over to practical settings.\n\n&nbsp;\n\n**W2 + Q3: Relation to the analysis in [65] and technical challenges.**\nWhile our work is inspired by the observations made in [65], our theory (and experiments) go far beyond them. In particular, all three main theorems in our work are by no means corollaries of the results in [65], as can be seen from the proof appendix (Appendix E). Specifically, as discussed in the proof sketch of Theorem 1 (Section 3.2), [65] only provides a bound on the gradient norm at a given point. This alone does not imply a lower bound on optimization time. It is necessary to show that the gradient norm will not increase during training by proving that reward variance cannot increase rapidly along the trajectory of optimization. Moreover, Theorems 2 and 3 require proving separations in terms of optimization time between reward models. That is, aside from the lower bound on optimization time, we established upper bounds. This required proving a sufficient condition (Proposition 4 in Appendix E) for efficient optimization via a fine-grained analysis of the optimization trajectory when maximizing the non-concave RLHF objective. Lastly, aside from the technical novelty of our theoretical results, the focus of our work is different from that of [65], which did not consider the question of what makes a good reward model for RLHF or the interplay between accuracy and reward variance.\n\nRegarding challenges in extending Theorems 2 and 3 to more general neural network architectures (considered in Theorem 1). Analyzing optimization for practical neural networks is still a major open problem even in simpler settings where the objective with respect to the neural network outputs is convex (e.g., supervised learning with the squared or cross entropy losses). While we were able to establish a lower bound, with the architecture contributing a dependence on its Jacobian norm, the challenges in establishing upper bounds on convergence rates are exacerbated due to the non-concavity of the RLHF objective with respect to the neural network outputs. Specifically, the architecture introduces complex interaction between the evolution of logits for different outputs, and as mentioned earlier, the understanding of these interactions is limited even in simpler settings. Hopefully, future technical advances may allow generalizing our theoretical results, as well as other optimization analyses of neural networks in the literature, to additional architectures.\n\n&nbsp;\n\n**Q1: Dependence on Jacobian norm in Theorem 1.**\nGood question! Our interest lies in comparing, for a given language model, how a reward model affects RLHF optimization. Initially, the Jacobian of the language model does not depend on the reward model (rather just on the language model architecture and parameters) and we empirically found that its norm does not change much during policy gradient (see details below). Thus, for simplicity, we omitted the dependence on the Jacobian norm from the abridged Theorem 1, but of course included it in the full theorem statement (Theorem 4). The goal in doing so was to emphasize the parts which we found to be the most crucial and relevant to our study. Please let us know if you believe that it would be clearer to include this term in the abridged version as well and we will gladly accommodate.\n\n*Details:* In a setting analogous to Section 4.1, with a Pythia-410m language model, we ran policy gradient using two different reward models (one trained on 100% on-policy data and another on 0% on-policy data). For efficiency, instead of the spectral norm, we tracked the average Frobenius norm of randomly selected rows in the Jacobian throughout training. In both runs the average Frobenius norm of rows hovered around 200 – 300, i.e., it was on the same order of magnitude throughout policy gradient and for both reward models. Note that since the Frobenius norm of the Jacobian upper bounds its spectral norm, this indicates that the spectral norm does not grow by much as well.\n\n&nbsp;\n\n**Q2: Applicability of our results to other RL settings.**\nThis is a great point! Thank you for raising it. While the positioning of our work focuses on RLHF, the theoretical results can be straightforwardly extended to support any RL setting with softmax parameterized policies (i.e., policies that produce a distribution over outputs via softmax). In accordance with your comment, we believe it can be a valuable direction to examine whether low reward variance empirically poses a problem in other RL settings and design potential-based reward modifications to increase reward variance. We will clarify this point in the manuscript.\n\nNote 1: As discussed in Appendix A, we are not aware of existing reward shaping techniques that aim to increase reward variance or consider the effect of reward shaping on policy gradient optimization.\n\nNote 2: Indeed, in bandit settings reward variance is equivalent to the expected squared advantage function, where the expectation is with respect to outputs sampled from the policy.\n\n&nbsp;\n\n**Q4: Effect of inexact gradients.**\nThe limitations paragraph in Section 3.1 and the first paragraph of the future work section (Appendix B) briefly discuss this point. In light of your question, we will expand the discussion on the effect of inexact gradients in the future work section along the lines of the answer below.\n\nOur experiments consider standard policy gradient methods that compute gradient updates based on batches of sampled outputs, because computing the exact expected gradient is computationally infeasible. Thus, they empirically demonstrate that the implications of our theory carry over to settings with inexact gradients. Intuitively, the reason that low reward variance remains problematic in the inexact gradient setting is as follows. When the reward variance is low, the expected gradient vanishes. In this case, batch gradients may not necessarily be small (whether they will be small or not can depend on which policy gradient method you use). However, since the expected gradient is small, this implies that the batch gradient is often noisy, and so updates do not lead to substantial progress. Formally extending our analysis to the inexact gradient setting is highly non-trivial and we believe that it could be an interesting challenge for future work, which may further shed light on the relative advantages and disadvantages of different policy gradient methods.\n\n&nbsp;\n\n**W3: Experimental results on additional benchmarks.**\nOur experiments are mostly based on the UltraFeedback dataset, which is a standard dataset for language model alignment. Since running the RLHF pipeline with large models is computationally expensive, we preferred to cover multiple language models, reward models, and ground truth rewards instead of additional datasets. The current experiments (in Section 4 and Appendix F) clearly demonstrate that the conclusions of our theory apply in practical RLHF settings. Thus, while we agree that additional datasets can give further evidence, we do not believe they are necessary to strengthen the claims made in the paper. Nonetheless, if there is any particular benchmark that you find is especially important, we would be happy to accommodate."}]

Author rebuttal:

Meta-review: This paper investigates factors beyond accuracy that make the reward model an effective teacher in RLHF. In particular, it focuses on reward variance induced by the reward model and shows that this variance is important for efficient optimization, even when the model itself is accurate. Both theoretical analysis and empirical evidence with LLMs support these findings. The work addresses an important and underexplored problem in RLHF. It is well written and offers useful insights into reward models. While the reviewers raised several concerns---some suggesting that further experimentation was needed to validate the main takeaways---most were resolved during the discussion phase, where the authors provided additional experimental results that corroborate their claims. The authors are encouraged to include those results in the final version of the paper.
Decision: Accept (spotlight)