Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback
Authors: Johannes Ackermann
Takashi Ishida
Masashi Sugiyama
Venue: COLM 2025 (2025)
Keywords: alignment
reinforcement learning from human feedback
reinforcement learning
reward modeling
TL;DR: We propose to use importance weighting to iteratively retrain an off-policy corrected reward model, resulting in a signficantly better final policy.

Abstract: Reinforcement Learning from Human Feedback (RLHF) allows us to train models, such as language models (LMs), to follow complex human preferences. In RLHF for LMs, we first train an LM using supervised fine-tuning, sample pairs of responses, obtain human feedback, and use the resulting data to train a reward model (RM). RL methods are then used to train the LM to maximize the reward given by the RM. As training progresses, the responses generated by the LM no longer resemble the responses seen by the RM during training, leading to the RM becoming inaccurate. The score given by the RM keeps increasing, but the learned behavior no longer matches the human preferences. This issue is known as overoptimization. We investigate overoptimization from the point of view of distribution shift and show that the shift results in an inconsistent estimate of the RM parameters, leading to an inconsistent estimate of the policy gradient. We propose Off-Policy Corrected Reward Modeling (OCRM), which iteratively off-policy corrects the RM using importance weighting, without requiring new labels or samples. This results in a more accurate RM, which empirically leads to an improved final policy. We validate our approach in experiments with summarization and chatbot datasets and show that it performs significantly better than standard RLHF methods and baselines.

Published as a conference paper at COLM 2025
Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback
Johannes Ackermaann
Takashi Ishida
The University of Tokyo and RIKEN AIP ackermann@ms. wwwokkk.kkokkaaa.kkkkkkkkkkkkk jc.
RIKEN AIP and The University of Tokyo
Masashi Sugiyama
RIKEN AIl' and The University of Tokyo
Abstract
Reinforcement Learning from Human Feedback (RLHF) allows us to train models, such as language models (LMs), to follow complex human prefer- ences. In RLHF for LMs, we first train an LM using supervised fine-tuning, sample pairs of responses, obtain human feedback, and use the resulting data to train reward model (RM). RL methods are then used to train the LM to maximize the reward given by the RM. As training progresses, the responses generated by the LM no longer resemble the responses seen by the RM during training, leading to the RM becoming inaccurate. The score given by the RM keeps increasing, but the learned behavior no longer matches the human preferences. This issue is known overoptimization. We investigate overoptimization from the point of view of distribution shift and show that the shift results in an inconsistent estimate of the RM parameters, leading to an inconsistent estimate of the policy gradient. We propose Off-Policy Corrected Reward Modeling (OCRM), which iteratively off-policy corrects the RM using importance weighting, without requiring new labels samples. This results in more accurate RM, which em- pirically leads to an improved final policy. We validate our approach in experiments with summarization and chatbot catasetss and show that it performs significantly better than standard RLHF methods and baselines.
1 Introduction
Large language models (LLMs) have had a large impact on society through the availability of chatbots such as ChatGPT (Ouyang et al., 2022). is important to ensure that these LLMs behave in a desired way and thus Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017) has become an important tool to align pre-trained language models (LMs) with human preferences. Given pretrained LM, RLHF typically consists of three steps (Stiennon et al., 2020): Firstly, supervised fine-tuning (SFT), in which the LM is trained to imitate example input-output pairs. Secondly, pairs of replies are sampled from the SFT model, labeled by humans according to their preferences, and collected in a dataset which is then used to train reward model (RM). Finally, starting from the SFT model, reinforcement learning (RL) is used to train the RL model to optimize the score given by the RM. While the training progresses, the RL model becomes increasingly different from the SFT model. As the RM has only been trained on replies provided by the SFT model, the RM can no longer provide accurate rewards. The output of the RM keeps increasing, but the quality of the responses as judged by a human stagnates or even decreases. This phenomenon is known as Goodharting or overoptimization (Gao al., 2023). We investigate overoptimization as distribution shift problem, caused by the difference between the RL model and the SFT model, whose outputs the RM is trained on. While previous work (Gao et al., 2023; Song et al., 2024; Fluri et al., 2025) has discussed that overoptimization can be considered consequence of distribution shift, to our knowledge, the applicability of methods from the distribution shift adaptation literature has not been investigated.
1

---

Published as a conference paper at COLM 2025
True Reward Rd"
Reward Model
Reward Model
Reward Model Rg
PPC-RUHF
Action Dimensdon>
0.5
Reward
06
True
0.
Dimension
Action Dimension
Action Dimension
Dimension
Proposed Method
Policy
RUHF Policya'
RUHF Policyn
Policyn"
1.5
Action Dimension2
True Reeward
Accuracy
Action Dimension 1
Action Dimension
Actio Dimension
Action Dimension
Training Samples
PPO-RLHF
Proposed Method
Figure 1: Visualization of our approach in a 2D stateless task. On the top-left side, the true reward function and learned RMs are visualized, on the lower-left side the SFT policy and learned policy distributions are shown. The colored circles contain 99% of each policy's action samples. The first RM Rgi is trained on data sampled from the SFT policy π and thus performs well on pairs of actions sampled from π²,, but is inaccurate samples generated by πt? This leads the policy trained with standard PPO-RLHF to stagnate early (top-right). Using IW, after k = 200, 000 samples, we train an off-policy corrected RM Ryz that is accurate on samples generated by 7t2 and can thus be used to continue training and obtain better policy T²' Iterating this process, we obtain better final policy (bottom-righh)..
We investigate this phenomenon and show that standard RLHF methods (Stiennon et al., 2020; Ahmadian et al., 2024) provide an inconsistent estimate of the RM parameters for all but the first policy update, and therefore generally also provide an inconsistent policy gradient estimate. They thus generally do not converge the optimal policy, even with unlimited available data and unlimited training iterations. We propose method that corrects for this distríbution shift by using importance weighting (IW) (Shimodaira, 2000), called Off-Policy Corrected Reward Modeling (OCRM), While using the resulting corrected RM to estimate each policy gradient would be desirable, obtaining it requires the RM to be retrained after each policy update and is thus not computationally feasible. We therefore propose to approximate this method with multiple training stages, each consisting of training corrected RM with IW and then performing multiple policy updates. As illustrateed in Figure 1, our approach iteratively improves the accuracy of the RM on samples of the current policy and converges to higher final reward, without any additional data, while standard Proximal Policy Optimization (PPO)-RLHF (Stiennon et al., 2020) stagnates.
Our main contributions are follows: We provide analysis of overoptimization in RLHF as distribution shift caused by the change of the policy due to RL updates, showing that even with unlimited training data standard RLHF may not converge to an optimal policy. Secondly, we propose to address this issue by using IW, giving us a consistent estimate of the RM parameters, and provide an estimation error bound for our proposed method. Finally, we implement our method and show that it performs well on two LM alignment tasks: In both the TI; summarization task (Stiennon et al., 2020) and the Alpaca-Farm chatbot task (Dubois et al., 2023), our method improves the alignment performance, outperforming standard PPO-RLHF (Stiennon et al., 2020), Direct Preference Optimization (DPÓ) (Rafailov et al., 2023) and related baselines (Zhou et al., 2024; Lang et al., 2024).
2 Related work
The most closely related method is Weighted Preference Optimization (WPO) (Zhou et al., 2024). They improved DPO by multiplying the DPO loss with weight term, similar to
2

---

Published as a conference paper at COLM 2025
importance weight. However, they used the probability under the current policy weight and thus can only provide a correct off-policy correction the dataset was generated by uniform SFT model, as we show in Appendix B. While they focus on DPO, we focus policy gradient methods. Also closely related is Reward Learning on Policy (RLP) (Lang et al., 2024). RLP retrains the RM with samples from the current RL model, using either an unsupervised representation learning method or a pseudo-labeling approach. Unlike RLP, we show that our method provides a consistent estimate of the new RM parameters. We empirically compare our approach with both WPO and RLP and provide an extended discussion of the related work Appendix C.
Background
In this section, we will explain the background for RLHF and IW. We denote the state s, action and policy π : S - Dist A),, with state set S, action set A, and distribution Dist(-). For clarity, we focus on the case of LMs and thus states s correspond prompts and actions correspond to completions by an LM. S and A are sets of sequences of tokens.
3.1 Reinforcement learning from human feedback
Following the setup of Stiennon et al. (2020), RLHF consists of three main steps: SFT, reward modeling, and RT..
Supervised fine-tuning As policy we consider an autoregressive model 7(a s) = Il 7 (a; s,act), where at is the 1-th token, dct refers the response tokens before at and refers to the entire response. First, a prefrained (PT) LLM N is fine-tuned on a given NAFT We denote the resulting policy as SFT model 7t. (s), a a
dataset DSFT
Reward modeling Given a prompt ~ P(s), where P(s) is the distribution over prompts, two completions ao, are sampled from the SFT model π² (a s). A human annotator then selects their preferred reply, according to their personal preferences, giving us a dataset ) j-1
DRM = ( aly a/) New iid.
P (s, aw, a)), where aw and aj are the preferred (winning) and not preferred (losing) replies, respectively. The Bradley-Terry (BT) model (Bradley & Terry, 1952) is used as a modeling assumption that states that the probability of preferring one option aj over another option ag the logistic function (xx) = 1/ x) of the difference of an (unknown) scalar reward R S A + i.e,,
P(a > ag s) = o (R(s, a1) - R(s,,a))
(1)
This assumption is used to train RM RØ S A IR with parameters 0 O by minimizing the cross-entropy loss based on the BT model:
CRM (0) (ss..w.))) [loge (Ro(s, - Rs(s, a:))]
(2)
We denote the per-sample loss IRM(SS, aw, ai; log or (Ra(s, aw) Ro a:)) and in practice use an empirical estimate of the reward parameters obtained by minimizing the sample average i = arg min 1/NRM E,MA IRv alw 2/,8). We denote the parameters of the true "human" " RM as gH, which is generally not in our hypothesis class gH $ O
Reinforcement learning As is common in the RLHF literature (Shao et al., 2024; Ahmadian et al., 2024), we consider an episode consisting of a single state s and action a. RL is used to train the policy nī with parameters & initialized from the SFT model π², to maximize the expected return 0) E. P(s) R (s, a)], with the regularized reward
(s) = Ra fDKL ( |s]l (-1 ())s)).
DKL is the Kullback-Leibler (KL) divergence with parameter B > controlling the strength of the regularization. This KI.-penalty keeps the model close to the training distribution of
3

---

Published as a conference paper at COLM 2025
the RM, preventing extreme overoptimization (Gao et al,, 2023). The expected return is then optimized using an advantage-based policy gradient (PG) method (Williams, 1992),
( 0) = (s [A] (s, V log (Tp(aas))
(3)
with the advantage function A7 (s, = (5, a) (s) and value function VT((s) =
Ea- (s, a)]. PG-based RLHF methods differ in the updates and advantage estimation methods they use. PPO-RLHF (Stiennon et al., 2020) uses Proximal Policy Optimization (PPO) (Schulman et al., 2017), which implements proximal update and uses Generalized Advantage Estimation (Schulman et al., 2016). Group Relative Policy Optimization (GRPO) (Shao et al., 2024) instead uses Monte Carlo estimates of the advantage in each state by evaluating multiple actions. We sometimes omit & in the subscript for clarity.
To facilitate the analysis of overoptimization in RLHF without relying on expensive human feedback, Gao al. (2023) introduced a setting in which large RM, referred to as gold RM Rgold is used replacement for the human feedback. By imitating powerful Roold with smaller RM, we can simulate the problem of imitating the very large "human RM" with large proxy model as done when LLMs such as GPT-4 are trained (OpenAI et al., 2024).
3.2 Importance weighting
When training data y iid. ~ Px Y) is available from a distribution with probability Po, but we need to estimate the loss on a different distribution with probability P1, we can use IW (Shimodaira, 2000). The expected loss C(0) = (( ((0)) over P1 with sample-wise loss I can be written
Pi 11 ((x)))))) Po(x,y)
(xxa))P((xx) (xxx, ((x) = (xx)))
w(x,y) = (/x,y))P(xx, y) is called the importance weight. Importance weighted estima- tors usually have a higher variance, it thus sometimes desirable use vrrutton- techniques that incur some bias, such as flattening the importance weights with a hy- perparameter 1j E [0, i.,, w (x,y) = (Pi (x) / /xx(x, y))" (Shimodaira, 2000). Another related technique (Yamada al., 2011) to use the ratio P1 yx, aPo + (1 (x, instead, with a E [0, 1].
to the mixture
Distribution shift in RLHF
To motivate the need for our proposed method, we first discuss what happens in the normal RLHF setting as the policy changes.
The goal of RLHF is to obtain a policy π. that maximizes the expected return ((,, OH) under an unknown human reward RąH. Therefore, at each optimization step i we update the policy ze' using estimates of the policy gradient (4) 0), which depends on the learned RM parameters Since the true @Hl is unknown, we instead use the estimate i obtained by minimizing the sample average of CRM (2) the dataset DRM generated by the SFT policy N!. As dataset size NRM + co, we know that C (0) L (8%), where 8o = arg ming E (ss,a..a)>-P, m1 ['RM (s, aw, 0)] is the minimizer of the expected loss under P, P,,,, ie. the best approximation of the human RM RAH on PTt,1e given unlimited data.
As training continues, zं shifts further from I² (Gao et al., 2023). In the policy up- dates, we are estimating the advantage of actions a ~ T² and should thus use the that provides the closest approximation of human preferences on Pyi, i.e,, the minimizer 8 = arg ming IE (s.aw,ai)>P (RM aw, 0]; As 6* is optimal by definition, we see that
(0 > (0")
see Theorem 2 in Appendix A
4

---

Published as a conference paper at COLM 2025
where equality holds if ne' = =x),,, iee, typically only in the first training step. In the following training steps, i achieves worse loss given unlimited training data and is inconsistent estimator of l". Further, as we use & to estimate the policy gradient V ( f), the policy gra- dient estimate generally also becomes inconsistent. Likewise, the optimal policy parameters Φº = arg maxp [( 80) under Rg generally do not match the optimal policy parameters o- arg maxØ [Q, 0*) under Rg- and therefore achieve a lower expected return
0( -
The whole RLHF process is therefore inconsistent in the sense that even with unlimited RM training data DRM and unlimited RL interactions, it may not converge to the optimal policy Tg. Readers may notice that this is an interpretation of overoptimization as caused by covariate shift under model misspecification (Shimodaira, 2000).
Off-policy corrected reward modeling
As discussed above, using the estimate ô of the RM parameters can lead to suboptimal policy. We will thus explain how we can obtain a consistent estimate of 0" in this section. One option is to sample new completions (ag. a1) Ií and obtain pairwise comparisons from human evaluators. However, as this both costly and time-consuming, we instead propose to use IW as an attractive alternative that does not require any additional policy samples or labels.
As we have knowledge of the probability ratio (a s)/r<((a s), we can rewrite the RM loss under the distribution of the policy πí as
CM (0)
[RRM (5, aw, a 0)] E (ss.waa)) [ar aw, IRM aw, a;; (4)
with importance weight w(s, aw, a!) = Pai(s, aw, a1) Pxt (s, aw, a.) obtained by
w(s, aw, a1) = P(s) (aw P(s) (aw ()) s) 2j
() Ps) > al 5) (aw
s)xe (aj
s) s)
(5)
n (aw (aw S)
(ai
Note that this is only possible if the support of nt' is a subset of the support of π², Fortunately, for all LMs with softmax-outputs this is satisfied as saals) > 0 for all completions E A and prompts E S. Minimizing the sample average of (4) on DRM gives us ê, which we can subsequently use to estimate the policy gradient. We later show in Section 5.1 that the loss of L (0) L (0*) as increase the sample size, matching the loss of the optimal e*. As the policy changes with each update, we need estimate newly after each update.
Readers familiar with the RL literature will recognize this as related to off off-policy corrections (Sutton & Barto, 2018). We note that instead of accounting for the change of policy in the policy gradient update, we are here additionally accounting for it in the reward modeling step.
5.1 Estimation error analysis
We provide an estimation error bound (Mohri et al., 2018) for our off-policy corrected reward modeling approach. Due space constraints we defer proofs and estimation error bound for unweighted reward modeling to Appendix A. When evaluated on samples
from DRM, Ô = arg ming 1 NRM NNR (s) a 21, is the empirical estimate of go and
Ø arg min, 1/ NRM En=n w(s), a RR) a af 0) is the empirical estimate of 0".
Theorem 1 (Estimation error bound for the proposed method). Assume (RM aw, aii 0) S C. and w(s, aw, aj) E [0, W]. Define the margin function class
F = {fo: (s,,; aw, a1) + Ro(s, aw) Ro(s,ai)
5

---

Published as a conference paper at COLM 2025
Then, with probability at least 1 8,
2 Ciss(() CRM (8*) NiMM (F) WCry NRM CRM
2
In
where RNaa (F) is the Rademacher complexity of F sample size NRM-
This theorem implies that as the number of samples NRM goes to infinity, (0) converges to L (0*), because (F) - 0 for parametric models with a bounded norm (Mohri et al., 2018). It is also intuitive that the upper bound tightens with smaller W. The assumption of a bounded loss IRM I C₁ is satisfied under bounded reward.
also interesting to consider C (0) RM (0). This quantifies how much worse it is to use standard reward modeling compared to the inportance-weighter estimate, on finite samples. We can derive an upper bound in the following way:
(0) - (0) S LRM (0)
L (8*)
RM (0) FRM (0) + LRM (0*) - CRM (0¹) + LRM (0) - -RV (0*)
S RM (0) FRM (0) + RM (8*) CRM (0*)
(6)
When distributions nO' and nd are not equivalent, (a) and (b) both become positive from the definition of the minimizers. These terms do not vanish even when NRM + co. Hence, (0) - LRM(0) may be nonzero as NRM - 0o, demonstrating the benefit of our method.
We note that, while we have shown that can IW to prevent the cause of inconsistency explained in Section 4, this does not mean that RLHF with off-policy corrected RMs will necessarily converge to an optimal policy. However, in the next section we will empirically show that it is useful in practice.
5.2 Practical implementation
As the policy changes after each policy update, we would need to also retrain the RM after each policy update, which is computationally infeasible. We thus implement approximate method that only retrains the RM using IW after each policy updates. We repeat this m times and denote the policy after iteration i E 1,2, II as Til1 and the RM trained with IW for the distribution of policy P Rgl-
The main purpose of the rL-reguuaaiizttinn is that it keeps the policy in a region where the RM is accurate, i.e,, close to _ However, after using IW to obtain the RM Rgi, > 1, it is now more accurate close to the policy π². Thus, we also change the reference of the KL constraint to the previous policy πt' that is, the policy ri+1 is obtained by using PPO for k updates with the reward R a) - DKL ( s)][ s)), ensuring that the policy remains close to the distribution on which the RM is accurate.
We train all networks using AdamW (Loshchilov & Hutter, 2019), which maintains estimate of the gradient moments. When switching to a new R., the optimizer state as well as the previous value function are no longer useful and we thus reset both. As advantage estimates in PPO rely on the learned value function, using small k can therefore cause poor policy gradient estimates due the value function being inaccurate in its initial training steps. In practice, we found that choosing a high such that the new policy is trained until convergence on the current RM performs well.
As mentioned above, importance weights can have high variance. We thus use flattened, relative importance weights (Shimodaira, 2000; Yamada et al., 2011) with = 0.001 and 0.9 in all LM experiments. This reintroduces some bias but advantageous with limited data (Shimodaira, 2000). We note that any other policy gradient methods such as RLOO (Ahmadian et al., 2024) GRPO (Shao et al., 2024) can be used instead of PPO.

---

Published as a conference paper at COLM 2025
Dataset
TL;DR Summarization
Short Alpaca-Farm
Model
Pythia 1B
Qwen 2.5 1.5B
P.8B
6.9B
P 1B
Q2.5 1.5B
SFT
17.8 +: 4.5%
31.6%
28.1% %
27.5%
12.5%
39.3%
DPO
43.7 0.7%
76.1%
64.8%
70.3%
12.4%
67.6%
WPO
47.0 2.2%
77.1%
64.5%
71.5%
10.9%
64.5%
PPO/GRPO RLP-SPG
62.9 2.7%
80.5%
47.3%
64.9%
21.3%
66.1%
57.1 = 9.7%
72.8%
23.6%
60.8%
Ours (m 2)
71.4 1: 2.9%
87.7%
63.4%
78.6%
25.3%
68.5%
Ours (m = 3)
73.6=1.7%%
89.6%
75.4%
79.8%
24.5%
69.5%
Table 1: Win rate vs reference response, as judged by the gold RM. For Pythia 1B on TL;DR run three different random seeds, corresponding to different SFT models and resulting RM datasets DRM, and report the mean and standard deviation. Other experiments use single seed. We abbreviate Pythia P and Qwen 2.5 as Q2.5. For the 6.9B model we use GRPO, all other experiments use PPO.
Experiments
As it is difficult to visualize LM experiments, we first illustrate our approach in a low- dimensional, didactic example and subsequently perform experiments with LM alignment.
6.1 Didactic experiment
We consider the case of state space consisting of a single state S {oo} and a two- dimensional continuous action space A = [-1.5, 1.5)². The reward function is a Rastrigin- like (Hoffmeister & Bäck, 1991) function and the initial policy 7t is an isotropic, zero-mean normal distribution, as shown in Figure (left). As RM we use Mult--Layer-Perceprron (MLP) with one hidden layer with four units. We do not flatten the importance weights in this experiment. The results are shown in Figure! 1.
The initial RM is accurate on the distribution of the initial policy π²) but accuracy quickly degrades as the policy changes, leading the achieved true reward to worsen after initially improving. Accuracy here is defined as agreement between Rg and Rgold on the ranking of pairs of samples drawn from π, i.e. TE (So--DRRn((mmnn)m((( (Rgg())))
(o)) (ggaa()))) (ata(()))
with indicator function J. By retraining the RM using IW after 200,000 samples, we obtain new RM R&2 that is more accurate in the proximity of the current policy π²2 which allows us to use it to train a better policy π². By repeating this process, we are able to converge close to optimal policy in three iterations, while standard PPO-RLHP stagnates.
6.2 Language modeling experiments
To evaluate our method when used to align LMs, we perform experiments with the gold- model setup (Gao et al., 2023) on the foundational summarization from human feedback task (Stiennon et al., 2020) and on length-truncated version of the Alpaca-Farm chatbot dataset (Dubois et al., 2023). As gold RM R gold we use the leading model on RewardBench (Lambert et al., 2024) at time of experiments under 10B parameters, specifically Skywork- Reward-Llama-3 1-8B-v0.2 (Liu et al., 2024). We implement our method and baselines based on the code provided by Huang et al. (2024).2 We report the win rate against the reference responses in each dataset judged by the Gold RM. For our method, we retrain the RM after each & = 100, 000 training samples. More implementation and experimental details can be found in Appendix D.
"Our implementation is shared at gi titthu..com http://www....ooeeeeoooooooeeeee
7

---

Published as a conference paper at COLM 2025
-8
Gold Score
Win Rate
Gold Score
-10
Ours (ir = 3)
PPO f = 0.05
PPO
59.6%
-8.62
PPO & = 0.03
PPO Reset
60.0%
-8.41
-12
PPO 6 0.01
PPO + New KL.
67.5%
-7.71
PPO 6 0.003
Ours (m
68.6%
-7.56
PPO 6 0.001
- 14
Ours (m = 3)
71.7%
-7.28
0
50
100
150
200
250
DKT. (T' (T()T²)
Table 2: Summarization with different abla- tions of our proposed method.
Figure 2: Our method and PPO with varied KL-regularization
6.2.1 Baseline comparison
As baselines, we compare our proposed approach with PPO (Stiennon et al., 2020), DPO (Rafailov et al., 2023), WPO (Zhou et al., 2024), and Reinforcement Learning on Policy Synthetic Preference Generation (RLP-SPGG) (Lang et al., 2024). RLP-SPG samples outputs from the policy and adds synthetic labels, which are used to retrain the RM. For RLP-SPG we did not find a public implementation, we thus implemented it based on the paper.
We show our main results in Table 1. Our proposed method achieves a higher win rate than baselines, and improves upon PPO for both Pythia (Biderman et al., 2023) and Qwen 2.5 (Qwen et al., 2025) base-models, both the chatbot and summarization tasks. In summarization, PPO is trained for 300,000 samples, RLP-SPG and our method with m are trained for 200,000 samples, our method with m trained for 300,000 samples. We found DPO and WPO to perform worse with longer training and thus report their score when training for 100,000 samples in the summarization task. This finding is consistent with Xu et al. (2025, Appendix A.1) and discuss it further in Appendix E. We also evaluate our method with 2.8B and 6.9B Pythia models, but only use selected baselines due to computational constraints.
6.2.2 Additional Experiments
To better understand our proposed method, we perform additional experiments with the Pythia-1B model on the summarization task, each with a single random seed.
Method ablations In our implementation we made two major design choices in addition to sonreteedinn
using the off- off-policy corrected RM: Firstly, we change the reference for the KL regularization
from DKL P (=1] to DxL (r'( ( To test the effect of this change, we continue training the policy with the initial RM Rgi but new KL term DXL (x''))z'')) We call this method "PPO + New KL.". We also reset the value function parameters when changing the RM. Recent RL literature has explored parameter resets as a tool to increase plasticity, showing to be beneficial in continuous control tasks (Nikishin et al., 2022; Dohare et al., 2024). We thus also provide a baseline, called "PPO + Reset", that resets the value function parameters and optimizer state but does not change the reward function or KL-term. As shown in Table 2, both ablations improve upon PPO, but do not reach the performance of our full method.
KL-constraint strength Changing the KL-term from (xL(TTil each iteration effectively weakens the KL KL regularization. We thus investigate whether simply weakening the strength f of the KL-regularization in PPO can match the performance of our method. As shown in Figure 2, optimizing & indeed improves the Gold score achieved by PPO, but it does not reach the performance of our method, both in terms of KL-Reward tradeoff and best achievable gold model score. Due to computational constraints, we did not optimize B for our method and use the default & 0.05 suggested by Huang al. (2024).
to DKL (7')|>>² after
8

---

Published as a conference paper at COLM 2025
Ours (m 2)
PPO
Winrate (%)
60
WPO
Gold Score
DPO
-10
k 100,000 000
40
-12
k 50, 000
k 25, 000
k 10,000
14
25,000 50,000 100,000 200,000
0
20
40
60
80
DKL (TT
Dataset Size
Figure Training with different sizes of the preference-labeled dataset DRM-
Figure Varying the number of PPO up- dates k before retraining the reward model.
Gold Score
Win Rate
0.001
0.9
6.83
74.4%
Method
GPT 4.1 Nano Win Rate
0.001
1.0
-6.88
74.3%
SFT
30.1%
0.003
1.0
-7.36
70.0%
PPO
57.0%
0.01
1.0
-6.87
74.3%
Ours (mn 2) Ours (m = 3)
62.9%
0.03
1.0
-7.12
72.2%
70.4%
PPO
-7.92
64.4%
Table 4: Training and evaluation with feed- back by GPT 4.1 nano instead of the gold reward model.
Table 3: Performance of Pythia-IB-deduped in the summarization task with different hy- perparameters for flattened IW and rela- tive IW a, with 2.
Dataset size IW can provide unbiased estimates but it increases the variance the estimate (Shimodaira, 2000). We thus investigate how method performs with less available data by running experiments with smaller datasets DRM and show the results Figure 3. Even with less data the performance of our method degrades gracefully and still provides improvement over PPO. Further, when few training data is available, the offline methods DPO and WPO perform better, while the online methods perform better with more data.
Hyperparameters We also evaluate the sensitivity of our approach to the newly introduced hyperparameters a and 1j. In Figure 4, we show that using larger number of PPO updates k before retraining the RM results in a significantly better KL-Reward trade-off and achieves a better final reward. We hypothesize that this is due the value function being inaccurate during the early training stages, which we also visualize in the Appendix in Figure 7. In Table 3 we show the impact of the IW hyper-parameters and a. While they are important, our method improves upon PPO with all tested values.
GPT Nano Experiments In our other experiments we use a gold RM as proxy for human feedback. To validate experiments in which we label the training dataset DRM and evaluate the final policy with pairwise feedback provided by GPT 4.1 Nano. The results in Table 4 show that method also performs well in this setting.
method with more realistic synthetic setup, we also run
We provide training curves, further discussion of hyperparameters k and 17, runtime dura- tion, results for longer DPO training, more training iterations m = 5, experiment details and output samples in Appendix E.

---

Published as a conference paper at COLM 2025
7 Conclusion
We have investigated the issue of overoptimization in reinforcement learning from human feedback as a type of distribution shift. This distribution shift occurs as the reward model (RM) is trained on the data distribution the supervised fine-tuning policy πe but evaluated on the current policy π', which shifts further and further from 7² during training. We have shown that estimating the RM parameters on actions from π² results in an inconsistent esti- mate of the RM parameters under the updated policy πí and consequently an inconsistent estimate the policy gradient. By retraining the RM with importance weighting after each policy update we can address this issue. Empirically, we have shown that an approximate version which only retrains the RM after each k updates is sufficient to significantly improve upon the performance of baseline methods on language model summarization and chatbot alignment tasks.
A limitation of our method is that it requires knowledge of the density of the SFT policy π for all RM training samples. While this is not problem if DRM is collected with known SFT policy, it would be interesting to consider cases in which additional preference data from other policies is available.
Acknowledgments
We thank Yivan Zhang and Thanawat Lodkaew for helpful discussions. TI was supported by KAKENHI Grant Number 22K17946. This work was supported by the Institute for AI and Beyond, UTokyoo
Ethics statement
Our work provides general-purpose method for RLHF and specifically focuses on an application to the alignment of LMs. Our work thus shares the same ethical concerns that apply to all preference-tuned LMs and their applications, but we do not foresee any additional, new ethical issues arising from work.
References
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. In ACL, 2024. URL https: //aclanthology. org. 2ergg/222..00 acl acl-long. 662.pdf..
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. In ICML, 2023. URL pdf.
https://proceedin wttpsmm...rrre/eee....
mlr
Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs: The Method of Paired Comparisons. Biometrika, 39(3 (9(3/4):324-353 1952. URL https:
//ww..jstor org/ stb31e/3334029.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurIPS, 2017. URL https: http://prrooeeeddin..eeerrr..... http:////pcrr..ll/////////////// d5e2c@adad5033c1 f91 Foedd .dff.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward Model Ensembles Help Mitigate Overoptimization. In ICLR, 2024. URL https: hwoppnreview.ne///geee dMYkkXX.
10

---

Published as a conference paper at COLM 2025
Shibhansh Dohare, J. Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A. Rupam Mahmood, and Richard S. Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026))788-774 2024. URL 141586-024-0771--1
nature. com/articlss
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF Workflow: From Reward Modeling to Online RLHF. TMLR, 2024. URL https: htttps://openrevieet net met/pdffid=al BaYUU9eU.
Abhimanyu Dubey, Abhinav Jauhri, and et al. The Llama 3 Herd of Models, 2024. URL http:///axxiv.org/abb//// 2177/ 21783.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In NeurlPS, 2023. URL https:///prrcceeddiggg http://roreeddiin...e..rrii..... neurips. httpp//dnsss.nuuiiioooo//ppopaee 55c7770eee300077dddaa0aooo0-orr
pdf.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alexander Nicholas D'Amour, Krishnamurthy Dj Dvijotham, Adam Fisch, Katherine A. Heller, Stephen Robert Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking. In COLM, net wwt//ett/fff/e (eefffdd=5uuuuufeee
2024. URL httpp:://ppereee
Lukas Fluri, Leon Lang, Alessandro Abate, Patrick Forré, David Krueger, and Joar Skalse. The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret, 2025. URL http://arxxvv.rrg/bbs///
Leo John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimiza- tion. In JCML, 2023. URL http:://prrceeinngg....rrrrs///p
pdf.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding- enhanced BERT with Disentangled Attention. In ICLR, 2021. URL https://openreviies net/ppdFiddxxPIIatuuts.
Frank Hoffmeister and Thomas Bäck. Genetic Algorithms and evolution strategies: Sim- ilarities and differences. In Hans-Paul Schwefel and Reinhard Männer (eds.), Parallel Problem Solving from Nature, pp. 455-469. Springer, 1991. URL fttps://link, tttps://link..sriinge.. springer.
com/ com/chapter/19 1007/000000000 /F00/222b02977..
Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The N+ Implementation Details of RLHF with PPO: A Case Study on TL,DR Summarization. In COLM, 2024. URL https://openrevieww http:/ppenrrviie...tef/dd///// kHO2ZTa8e3.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic Uncertainty: Linguistic Invari- ances for Uncertainty Estimation in Natural Language Generation. In ICLR, 2023. URL https:///pperreview. wwtprrewwwwiieeee/dddeeeeeee hete//pdPiidvv--AYtPPPP
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, L. Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Han- naneh Hajishirzi. RewardBench: Evaluating Reward Models for Language Modeling, 2024. URL http://arxix: org httgg//b///24/// 13787.
Hao Lang, Fei Huang, and Yongbin Li. Fine-Tuning Language Models with Reward Learning Policy. In NAACL, 2024. URL https://accanthoology org/2024, - 75. pdf.
Zihao Li, Xiang Ji, Minshuo Chen, and Mengdi Wang. Policy Evaluation for Reinforcement Learning from Human Feedback: A Sample Complexity Analysis. In AISTATS, 2024. URL https://proceedin:
mlr. wtl..prres//v23/111 htmmllll....... html.
Chris Yuhao Liu, Liang Jincii Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-Reward: Bag of Tricks for Reward Modeling in LLMss, 2024. URL http: http://arxiv.org/abbb///// 184451 18451.
11

---

Published as a conference paper at COLM 2025
Mingjee Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models, 2025. URL http::/arxii...o/// htt.:///g//bb/////eeeeeeee abs/2505, 24864.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularizatio.. In ICLR, 2019.
URL http::///ppenreveeeeeee///d//ddd
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learn- ing. MIT Press, 2nd edition, 2018.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The Primacy Bias in Deep Reinforcement Learning. In ICML, 2022. URL https:// proceedings.mL. wtrresssll///niiiiiii2aaaaaan
pdf.
Oosn Achiam, Steven Adler, and et al. GPT-4 Technical Report, 2024. URL http: //arxiv.org/aabs. http://.ab/////....// 0tt.//2233000777
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurIPS, 2022. URL https: httttp:://proceeeiigg neursii cc/paper- .pdf.
files//aper/2802/// httee//beeedd333ee347771333 -1t.:770000000000001110------0-0
Qwen, An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 Technical Report, 2025. URL httt:::/aaxii..rg/bbb///////
15115.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct Preference Optimization: Your Language Model Secretly a Reward Model. In NeurlPS, 2023. URL httpss:/ppppprss.eeeee neurips pdf.
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the Benefits of Weight Averaged Reward Models, 2024. URL http://arxivv.or//
12187.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- Dimensional Continuous Control Using Generalized Advantage Estimation. In ICLR, 2016. URL
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, 2017. URL
1707 06347.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Junxiaa Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. MeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. URL http: org/abs/
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90(2):227-244, 2000. URL https://www httpp::/www.scienneeeeee scwencediiiiinnc rect. com/ science/ wwwiinn/eatttclleeeeeceee /
Yuda Song, Gokul Swamy, Aarti Singh, Drew Bagnell, and Wen Sun. The Impor- tance of Online Data: Understanding Preference Fine-tuning via Coverage. In NeurIPS, 2024. URL UtLnttss://prroceennsss. newrips. cc/paper files/paper/202 http:/ppper//22ffeee..
12

---

Published as a conference paper at COLM 2025
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In NeurlPS, 2020. URL https://proceeeddngg neurips. newrips.cccppperr files/paper/ htt.//////f885555022006066666450 Paper Paperr. .ddf.
Richard S Sutton and Andrew G Barto. Reinforcement Learning; An Introduction (2nd edition). MIT Press, 2018. URL http:///incompleeeides... het.eboodaasnneeoooooo
2nd. html.
Vladimir Vapnik. Statistical Learning Theory. Wiley, 1998.
Ronald J. Williams. Simple statistical graoiint-following algorithms for connectionist rein- forcement learning. Machine Learning, 8(3-4):229-255 256, 1992. URL http:///link.sprinner com/10. 1007 /00000//F000 /F0999266.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. In ICML, 2025. URL https: https://proceediinn
mlr press//235/xx2222
Makoto Yamada, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, and Masashi Sugiyama. Relative Density-Ratio Estimation for Robust Distribution Comparison. In NeurIPS, 2011. URL https::/papers: https::/papers.nuriiss..oo wwtps/sserenis..pprrrreeee/eeeee http://pppr...eeeeeepe//////111/ di22581haa47722aa000000/777aapp2
Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, and Chuang Gan. Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble, 2024. URL http://arxiv http:/axxivvoo.r///a///
org/abs/240) 16635.
Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. WPO: Enhancing RLHF with Weighted Preference Optimization. In EMNLP 2024, 2024. URL httpss://aclanthol..gg org 2024. emnlp-main, 475.pdt pdf.
13

---

Published as a conference paper at COLM 2025
A Estimation error bounds
We first provide estimation error bound for reward modeling without distribution shift and its proof. Next, we build on this result to provide an estimation error bound for our proposed importance weighted reward modeling method.
Note that for readability, the notation here differs from the main text slightly, as & always refers to LRM and corresponds to NRM. Also recall that LrM refers to the population risk, while R refers to a reward function, differing from the notation in Mohri et al. (2018).
A.1 Reward modeling without distribution shift
We consider a parameterized reward function Re(s, E R, where 0 E O. We observe training dataset
D D= [(st al.. a iid. aws a)),
which consists of samples, each triple indicating that in prompt s response al, preferred by human labeler over response a/. In reward modeling under Bradley-Terry or logistic-preference assumption, recall that the sample-wise loss is:
Fmm (s, aw a): 0) = log ( [Ra (Ra(s, Ro((,ai))), o(x) = 1
We previously define the population risk as:
(()) Ep((saw.a)) /rm(s aw, aw)
and the empirical risk on the dataset D is
{0) 11
Σ
4rm (s', a 0):
The empirical minimizer and true minimizer are
i arg min [(0), f arg min ((0),
Ic0
respectively.
Theorem 2 (Estimation error bound for the ordinary case). Assume there is constant C₁ such that lmm (s, aw, 0); I for all A E A and all (5, aw, a1). Define the margin function class
{fo : (s, aw, aw) + Rg(s,aw) Ro(s, a1) 10 |
Then for any > O, with probability at least 1 -5 5,
- C(09) < 49,, + C.V. 2 In
where R2 (F) is the Rademacher complexity of F at sample size
This theorem implies that as the number of n goes to infinity, L(O) converges to ((0))
because R(F) + 0 for parametric models with a bounded norm (Mohri et al., 2018).
Note that we made an assumption Imm S Cj. This is satisfied when the reward bounded. If Ra(s,a) a) remains in [-B, B), then
fs aw, a)) Ra aw) Re(s, aj) [-2B, +2B].
The logistic loss - (fv) is then bounded above by max |- log (m)), which serve as C--
14

---

Published as a conference paper at COLM 2025
Proof. We want an upper bound on sup [(e) [(0) Since each loss is at most Ct, changing
0c0
one sample (s', alw a/) can alter E(0) by at most C//n. By McDiarmid's inequality, for any e 0
Pr sup ((()) [(0)) -E Essup ((())
2c c] exp
2ca
(-
2nc
exp
Ct
2
C7
n
Setting 8/2 = e/2=exp(- 2nc C
yields e -GV 1 In(3). Thus, with probability at least - 8/2,
2x
sup
(E(0) [()) s E ((()) +c. 0c0
1 2m1
In (7)
1C0
Next we handle the expectation term E(supace (E(0) [()))] By a standard symmetriza- tion argument (Vapnik, 1998), let D be our и i..dd samples and D' be an independent ghost sample of the same size. Then
IED ((())-((0)) ED sup (E AE0
E (up((():))-((:)))) AEO
AEO
Note that
1 Σ mm (x;)), L(e:D') 1 Σ
L():D) n i=1
mm (x/;
1=1
where x2 a a/) and x (s"i alt af¹). Hence
1 n n
[(0:D) 2(0,D')
Σ
1mm I (x:; 0) Im (x/::0)
Define
A(0) Σ 4m (x;;0) Imm (x/:
Under D'D', the pair (x;,2) has the same distribution as (x), (()) and -A()) share the same distribution. Introducing Rademacher variable 07 {+1}, we obtain:
Ep,p' sup [0)] +EO
Il (x;; - Imm (x/x2
sup BE0 1
< D
sup Σ o,4mm (x; 0)] + F
sup Σ o,lem (x)
8
But -; again Rademacher variable, so the second expectation equals the first:
Ep sup A(0] < 2I sup Σ
"
oplm (x:00] = 2nlE sup
1 - Σ om/m (x;; (x:: n
Hence
11 ED sup (R(() [()()))[(())] < 2IF sup Σ o,4m (x;: = 29n (Im o F).
where Imm F. indicates the function class { (s, aw, a1) + lsm aw, ));0) 0 E O}. Combining these pieces, with probability at least 8/2,
1 In (n(
sup (ā(0) L())) < 29,, (Irm FF) CtV see
2n
15

---

Published as a conference paper at COLM 2025
A similar argument for the other direction, supeco (C(0) -())), yields the same bound. Both hold simultaneously with probability at least - 8,
2 In (7)
sup
[ - 29,, (Im F. + + A
2n
Finally, by definition of f and 8"
- -(0)- ((0)) + - [(0)) + (Z(P")-((8))
S sup (C()) [(0)) +0+ sup(((p) [())) S 2 2 2(0)) 8E0
8E0
BEO
Thus with probability at least 1 -
[(0) - s 22H, (Imm F) + C.V 2n
In (2)]
-4 (Im F) +C2 2m In (2)
8
Since Im o F is 1-Lipschitz in far Rn (lmm o F) S (F(F) based on Talagrand's lemma. Therefore,
2
2 (3)
[()) C(0)) (F) Ciy
In
n
which completes the proof.
A.2 wmpogtadce-weighhed reward modeling
The responses may come from policy but we wish to minimize the risk under different policy πτ. Let aw, the joint distribution that corresponds to π'. Note that this the same p that we used to define ((0) earlier. We can rewrite the previous population risk as
aw, a be the joint distribution that corresponds to π and as,,aw, aw, a)) be
π'' a' |s) 7(aw s) [s)
C' (0)) Eq(sawa)) [w]s, aw, a1) (s, aw, a1; 0]: = L(0), w(s, aw, ai)
If our sample size is ii.d from 9, we define the empirical estimator as
('(0) = Σ w(s' a 4mm (s', a al:
Let θ argminy (0) and 6' = arg ming (0). Since ('( = L(0), the minimizers are also the same, i.e., 6% 0
Theorem 1 (Estimation error bound for the proposed method). Assume Imm (s, aw, a < C and wss aw, a1) E W]. Then, with probability least 1 5,
c(²) - C(0") < (W + WC
In
8
where F = foo is the same margin-function class as before.
This theorem also demonstrates that when goes infinity, L()) will converge to L(0*). It is also intuitive that the upper bound becomes tigher with smaller W.
Let fols, aw, ai)
Ro(5, Ro(s,
Sample-wise loss is Im (s, aw, 0); am log (m) = a(m) - 1, which takes val-
log (fo(s, ((r aw))
derivative check shows d
in -1, Hence (=100) is 1-Lipschitz with respect to the margin fe-
16

---

Published as a conference paper at COLM 2025
Proof. In the importance-weighted case, simply consider the new sample-wise loss function
(x; = (x))
where x = (s, aw, a1) and wv.) E [0, w]. Then the same argument as Theorem 2 applies except for the following two changes:
(x; S WC,,
Is((;00 is (W x 1)-Lipschitz in the margin,
because Im is 1-Lipschitz and we multiply by at most W. Hence all factors in the uniform deviation bound gain factor W. We get
sup ( 29t, (Is F) + WC In 2 0ce
and Is(-:0) is W-Lipschitz in fa, Ri (1i F) (F) Finally,
L'()) (P.) S sup ())) (') 4W9 (F) + WC// / 1n dre
Since ('( [()) and 02 0*
c(0) L(0") < 4W9, + WCTV 2 (n())
This completes the proof.
B Comparison to Weighted Preference Optimization
Weighted Preference Optimization (WPO) (Zhou et al., 2024) is an alignment method that builds upon Direct Preference Optimization (DPO) (Rafailov et al., 2023) by adding weighting term that imitates online learning.
DPO aims to to solve the same RLHF problem setting as the RLHF formulation proposed
by Stiennon et al. (2020) and used in our work: We want to obtain policy π', that maximizes the regularized reward
max E a (((s.a) BDKL s)]l (-1s)
where a(s,a) again a reward following the BT-model as in (1). However, DPO does this without training a reward model and without choosing new actions with the policy. Instead, they derive a maximum likelihood formulation that minimizes the following objective:
πt (aw
Blog π' ())
(s..waa) log logooBlog
aaw
())
π [aj s)
xe (awds) ne (awls)
at (aybb) n. (a)ss)
and we define (s aw, a,) log o log
Blog
The advantage of this DPO loss is that we do not need to generate new actions a and can thus significantly speed up training. disadvantage that actions a are no longer generated by the current policy z²', thus an off-policy method.
WPO Zhou et al. (2024) now correctly point out that due to DPO being off-policy, the action distribution produced by the policy longer matches the preference dataset and address this issue by "simulating" on-pol RL. To do so they propose a weighted version of the DPO loss function:
TF (s..a. a, [w[s, aw )an(s, a1)/roo aw, aj)]
17

---

Published as a conference paper at COLM 2025
with weight w(s, a). As weight they propose to use the policy probability w(s,a) rr s). This is reasonable heuristic choice, giving more weight to the actions more likely under the current policy.
However, we have shown in Section 5, to obtain a proper off-policy correction, need to use the probability ratio ('(ais) From this we can see that while WPO intends to optimize the loss
(als)"
nt (aw
(s) πt (ai s) IDPO (s, aw, a1) () π (a) s)
E (aw aw (als) (s aw, ai) = [B (awra))) (als)
π (aw
they are instead optimizing the loss
7 (aw s) (a) () aw, an) + E (awra))>((((s) [IDrO aw, aj)]
(aw.aa-- (d|s)
This is only correct off-policy correction (up to scaling by different constant) under the assumption of a uniform SFT policy r' s) with some constant This is not the case for most practical SFT policies, but it may be a reasonable assumption if we do not have access to the policy π² under which the dataset was generated. We note that in practice we usually have access to n
We would also like to highlight that despite not providing a correct off-policy correction, WPO nonetheless provides a consistent improvement on the performance of DPO in our experiments.
C Additional related work
We provide extended discussion of related work here.
Better Reward Models Coste et al. (2024); Zhang et al. (2024); Eisenstein et al. (2024) proposed different ways of training and using ensembles of RMs and show that training with them results in a better alignment performance. Ramé et al. (2024) proposed to use weight averaged combination of multiple RMs. These papers address inaccuracy of the RM, but do not address distribution shift directly and could thus be combined with our method.
Iterated RLHF LLAMA 3 (Dubey et al., 2024) used an iterative alignment method which involves multiple stages of rejection sampling, SFT and DPO. Dong et al. (2024) presented RLHF Workflow, an iterative DPO method. Both involve doing multiple rounds of post- training including DPO, sampling from the model and labeling pairs of outputs with pretrained RM. They both aimed to bridge the gap between DPO and PPO by sampling online data from the policy, while we are instead trying to improve PPO by correcting for the distribution shift without new samples or labels. Concurrently to our work, ProRL (Liu et al., 2025) proposes method called "Reference Resets", which is identical to our baseline "PPO+New KL in Table
Theoretical analyses of distribution shift in RLHF Song et al. (2024) provided theo- retical analysis comparing DPO and PPO under different support assumptions for ा' and I They proposed method that is effective in bringing the performance of DPO closer to PPO, while our method focuses on improving PPO by improving the RM. Fluri et al. (2025) provided a theoretical analysis of RLHE, showing that under distribution shift an RM with small training eтror can lead to a policy with large regret. This is similar to the motivation of our work and related to our discussion in Section 4, but we propose method that addresses this issue starting from the cause of the RM error, while they focused on a theoretical analysis of the regret under an RM with known bounded error. et al. (2024) provide sample complexity analysis of off-policy evaluation in RLHE, when using a Multi-Layer perceptron as RM. They also consider the effect of distribution shift in the reward modeling step and provide estimation error bound that accounts for it.
"They also propose to use three other scaled variants of the policy probability, however they all have the same fundamental issue.
18

---

Published as a conference paper at COLM 2025
D Experiment details
Algorithm 1 PPO with OCRM
Input: Pretrained LM x SFT dataset DSFT, iterations m, updates per iteration k nt' + min, E (ss) (sE a, π(s))]
Sample (a),41) π (als), label with preferences to obtain DRM {(s}{, alw, a/) 1-)
NRM
for = 1 to m do
Rgi f arg ming DR (zoss, aw, a (s, aw, ay, 8)] with w(s, aw, a)) [s)))s)
(|ا|ا) (agls)
k PPO training steps with reward Re (s, DKL (TTp( [6][[ (+)s))
Reset value network parameters and optimizer state for PPO
end for
D.1 Didactic experiment
To illustrate our approach in an easily visualizable domain, we use a stateless environment with two-dimensional action space [-1.5, 1.5,2 The reward function is a Rastrigin-like function, specifically r(s,a) = ||a|2 0.7| + sin ) + sin(6.ra where at and a² are the first and second element of the action. The initial policy πt' (corresponding to the SFT policy) is zero-mean Gaussian with diagonal covariance matrix Σ = 0.7 The reward model is an MLP with a single hidden layer with four units and tanh activations. We train the RM for 50 epochs on the DRM and then use it to train policy with PPO. The policy and value network each consist of MLPs with two hidden layers, 64 units and ReLU activations. After each 200,000 samples we retrain the RM with importance weighting. Here we use the raw importance weights without flattening or relative importance weighting.
D.2 Language model experiments
Training Samples in the figures refers to samples seen (DPO/WPO) or generated (PPO, RLP-SPG, Ours) by the language model. For evaluation, we use sampling temperature 0.01.
All SFT models are trained on the train split for one epoch. The SFT models are then used to generate 278,496 training pairs (s, ap, a1), with s sampled from the SFT dataset and with sampling temperature 1.0. These pairs are then evaluated with the Skywork-Rewwen Rewaard 1atma--.--BB-0--. model (Liu et al., 2024). Note that we do not sample labels based on the probability implied by the Bradley-Terry model (1), but instead always choose the higher reward as winner, as done by Gao et al. (2023).
D.3 Datasets
For the summarization experiments we use the TL;DR dataset as provided by Huang al. (2024). We report the gold-score, win rate, and KL-divergence on the validation split of the summarization dataset provided by Huang et al. (2024). Huang et al. (2024) also reported their win rates on the validation split.
To test the performance of our method when used to train a chatbot, we use modified version of the Alpaca-Instructions (Dubois et al., 2023) dataset. The Alpaca- Alpaca-Instrructions dataset mostly contains short requests typical for chatbot, but also few long examples with around 1000 tokens. These require larger context size for the language model and would significantly increase the training cost, we thus remove them from the training and validation datasets. Further, Alpaca-Instruce SFT, reward modeling and PPO. We combine them to a single training dataset used to train the SFT model and then sample the DRM dataset from the SFT model.
contains separate data splits intended for
With the dataset now being truncated to queries of at most 512 tokens and outputs of at most 106 tokens, we the same codebase we used for the summarization experiments
19

---

Published as a conference paper at COLM 2025
in these experiments well. We report the gold win rate the length-filtered validation split of Alpaca-Farm, not Alpaca-Eval. The code used to filter the dataset and the resulting dataset are available in our GitHub repository.
D.3.1 PPO and proposed method
We base the implementation of our method and PPO on the implementation provided by Huang et al. (2024) and keep their architectural and implementation details, as well as hyperparameters, except where otherwise noted. Notably, following Stiennon et al. (2020), in the implementation of Coste et al. (2024) the RM is initialized with the parameters of the SFT model and thus has the same parameter count. Rollouts generated with sampling temperature 0.7.
Due to computational constraints, we were not able to perform extensive hyper-parameter optimization for Pythia 1B. We thus use B = 0.05 in all Pythia 1B and Pythia 2.8B exper- iments, except for the ablation of it in Figure 2. This evaluation showed that B 0.03 achieves a higher gold score than B = 0.05, but as it was run after finishing the main experiments and we did not tune for our method, we continued use B 0.05 for PPO in all experiments.
For Qwen 2.5 1.5B we increased the KL constraint strength to f 0.1. For Pythia 6.9B we used GRPO instead of PPO-RLHF and use f = 0.1. Note that the values for ß in GRPO and PPO-RLHF are not directly comparable, as PPO-RLHF treats the KL constraint as part of the reward and applies it token-wise, while GRPO treats it separate loss term. Further, for the Pythia 6.9B experiment, we use an RM initialized with a Pythia 1B SFT model to conserve VRAM.
We show in Table that increasing the iterations to m = 5 increases performance futher, however, did not have the resources to evaluate this for all models/ datasets.
D.3.2 RLP-SPG
We could not find public implementation of RLP (Lang et al., 2024), RLP-UML or RLP-SPG, we thus re-implemented RLP-SPG as was the better performing method of the two in their experiments. Given the RL model 72 we thus generate 10 replies for each prompt s DRM- We then use deberta-xlarge xlarge-malii (He et al., 2021) to calculate entailment scores for each pair of replies and divide the replies into clusters of replies such that all responses in a cluster mutually entail each other. If one cluster meets the threshold size of at least four replies, we sample aw from the largest cluster and from any other cluster. Lang et al. (2024) use threshold of five replies and used the deberta-large model, however, we found a threshold four replies and the larger deberta-xlarge to perform slightly better.
D.3.3 DPO and WPO
For DPO we are using the implementation provided by Coste et al. (2024), from the same suite of implementation term for our WPO experiments. For the Kl-regulaaizztton strength in DPO and WPO we evaluated the values (0.3, 0.1, 0.03, 0.01} on the Pythia 1B summarization task and found ß = 0.1 to provide the best performance at the end of training and we thus used B 0.1 for both DPO and WPO. For WPO we implemented the length-normalizer weighting method, as it is the only method that was shared in the publicly available implementation (Zhou et al., 2024).
the PPO implementation. We modify it to include the weighting
E Additional results
E.1 Reward Curves
In Figure we show the evolution of the reward of our method and baselines during training, as well as the KL-Reward tradeoff. We can see that our method performs the same as PPO until the first retraining of the RM at 100,000 samples and then improves significantly, while
20

---

Published as a conference paper at COLM 2025
-6
Gold Model Score
Gold Model Score
-8
Ours
-10
PPO
-10
WPO
DPO
-15
-12
WPO (s.)
DPO (s.)
-14
100,000
200,000
300,000
0
20
40
60
DKL(T()x
Training Samples
Figure 5: Comparison with baselines across three random seeds each with 95% confidence intervals on the summarization task with Pythia-1B. We train DPO (s.)short) and WPO (s.) for 100,000 samples as they degrade with longer training. (see also Appendix E.8). The dashed lines indicate retraining of the RM in our proposed approach.
PPO stagnates. To be able bootstrap confidence intervals for the KL-Rgold plot in Figure (right), we first interpolate the KL-Rgokd curve for each run values on a fixed grid of KL values by linear interpolation. We then bootstrap the CIs for each KL value. The CIs for the samples-score plot are likewise boot-strapped.
E.2 Importance weight flattening hyperparameters
R Roold
Win Rate
0.001
0.9
6.83
74.4%
0.001
1.0
-688
74.3%
0.003
1.0
-7.36
70.0%
0.01
1.0
-6.87
74.3%
0.03
1.0
-7.12
72.2%
PPO
-7.92
64.4%
Table 5: Performance of Pythia-IB-dedup in the summarization task with different hyperpa- rameters for flattened IW y and relative IW a. Here m = 2
(xa) Recall that we use flattened, relative importance weights w (x,y) (1 (Shimodaira, 2000; Yamada et al., 2011). We used 5) 0.001 and 0.9 all experiments.
We note that using both flattened and relative importance weights can be considered redundant, as relative IW was proposed as an alternative to flattened IW (Yamada al., 2011). As shown in Table 5, using relative IW with a = 0.9 does not result in noticeable difference compared to using only & 1.0, i.e. only using flattened importance weights. Due to limited computational resources and the small effect, we did not rerun all experiments without relative IW. We further note that in our alpaca experiments each action has 106 tokens, such that y 0.001 and token-wise density ratios. We believe that this can be a good guideline practice.
1.0 approximately uses the geometric mean of the
We also ran additional experiments to investigate the effect of different values for 1j. The choice of the hyperparameter y7, indeed has a noticeable effect on the performance of our proposed method, but all tested values result in improvement over PPO.
21

---

Published as a conference paper at COLM 2025
Gold Model Score
Gold Score
-8
-
100
-10
100, 000
50, 500000
50,
-9
25, 000
-12
25, 000
10,
10, 000
-10
-14
100,000 200,000 300,000
0
20
40
60
80
Training Samples
DKL (Tt (
Figure 6: Evaluation of our approach with different values for
0.1
Value Function Loss
80-2
6 60-
4.10 2
2.10-2
0 0.1 0.25
0.5
PPO Training Samples
Figure 7: Value loss during PPO training with 100, 000 for different random seeds. Note that during the early training stages the value function is still inaccurate, which might contribute larger values for performing better.
E.3 hyperparameter sensitivity
We also investigated how the number of policy updates performed before retraining the RM affects the performance of our approach and ran additional experiments with Pythia 1B on the summarization task. The results in Figure 6 show that higher k is advantageous in the asymptotic gold model score and also achieves a better gold model score for the same DKL- One possible explanation is that the value function inaccurate in the first few updates after training a reward model, leading to inaccurate policy updates. We thus also show the value loss when using k 100, 000 with three different random seeds in Figure 7. We can that especially when training for less than 20, 000 samples the value loss is high, potentially explaining why k = 10, 000 performs significantly worse.
However, when smaller could be beneficial as visible in the beginning of Figure 6.
constrained by the number of PPO updates can perform,
E.4 Longer Training
Method
PPO
Ours (m 2)
Ours (m =
Ours (mn 4)
Ours (m 5)
Win Rate
59.7%
68.6%
71.7%
74.4%
74.9%
Table 6: More iterations m for Pythia 1B the TL,DR task with single random seed.
In our main experiments, restricted our approach to 3 iterations due to compu- tational constraints. However, as there is no saturation being visible yet, we perform an additional experiment with single seed with Pythia 1B on the TL;DR task. The
22

---

Published as a conference paper at COLM 2025
results in Table 6 show continued increase in gold model win rate for 4 and 5,
however it seems to saturate for m = 5.
E.5 GPT 4.1 Nano Feedback
In our other experiments we use Gold model, specifically Beterr--Rewrr--laameee v0.2, as proxy for human feedback, both generate the training data and for evaluation. To alleviate concerns about this problem setup, additionally train and evaluate Pythia 1B model with feedback from GPT 4.1 nano instead. We use the following prompt format:
You will be shown Reddit post and two summaries. You will be asked to rate which one captures the content of the post better.
<POST> {post) /POST>
(response_a] < </SUMMARY A> <SUMARYY B> (response.b) </SUMMARY B>
SUMMARY A>
Please reply only with the letter "A" or "B", for which summary you think better. Pick one, even neither is good!
We experimentally found slight bias towards the first response. For evaluation, we thus input each pair of responses in both orders and count it as tie if the preferences not consistent. The win rate is shown Table and shows a significant improvement, consistent with the results of our gold model experiments.
Method
PPO
Ours (m 2)
Ours (m = 3)
GPT 4.1 Nano Win Rate
57.0%
62.9%
70.4%
Table 7: Evaluation with feedback from GPT 4.1 nano instead of the gold reward model
E.6
Output diversity
To quantitatively investigate how our method affects the output diversity of the trained model, we follow the methodology of Kuhn et al. (2023) to cluster outputs into semantically equivalent clusters with a deberta-large model (He et al., 2021). We use subset of 256 prompts from the Alpaca-Farm validation split, generate 20 outputs for each with the trained 2.5 1.5B model and report the number of found clusters in Table 8. The results show slight decrease in output diversity.
E.7
Runtime
Our method performs well compared to standard PPO-RLHE, but the additional RM re- training also requires additional computation. We thus illustrate the additional time cost in Table when training the Pythia-IB-dedup base model using 8xA100-40GB. The runtime is dominated by the PPO training steps due the need to autoregressively generate new completions. comparison, retraining the RM training is relatively inexpensive, first requiring around 10 minutes to calculate the importance weights and then training for 42 minutes.
Method
PPO
Ours (m 2)
Ours (m = 3)
# semantic clusters
11.90
11.72
11.31
Table 8: Average number of semantic output clusters among 20 samples of Qwen 2.5 1.5B model on the Alpaca-Famm Farm validation split.
23

---

Published as a conference paper at COLM 2025
Method
RM1
PPO
RM2
PPO2
RM3
PPO3
Total
PPO
42
507
549
PPO+OCRM
42
169
52
169
52
169
653
Table 9: Runtime (minutes) of our method and standard PPO.
E.8
Longer DPO and WPO training
-20
Moce Store
-24
-16
p.1.11
-18
0.00
WPO 5-0.1 Long
-20
DPO pi 0.1 long
202000
Figure 8: Performance of DPO and WPO degrades when training for longer durations.
In the main text we are showing results for training WPO and DPO for 100,000 training samples. This leads to WPO and DPO seeing fewer samples during training than the reward models used in PPO/RLP-SPG and our proposed method, making the comparison superficially unfair. We thus also attempted but we found this to lead to worse results, as shown in Figure 8. We note that this issue is consistent with findings of Xu et al. (2025, Appendix A.1). A possible explanation could be that DPO and WPO overfit to the training prompt dataset, however verifying this is outside the scope of our work.
train DPO and WPO on the full dataset,
E.9 Sample outputs
We provide some sample input-output pairs in tables 10, 11 and 12. Note that as we are using the synthetic gold reward setup, we care more about the gold reward R Rgold than actual human preferences to evaluate the methods.
24

---

Published as a conference paper at COLM 2025
Content
R
gold
SUBREDDIT: T/ relationships
TITLE: I [22 F| want to buy my bf [28 MJ of 6 months $500 very rare video game for his collection for his birthday /Chrystmsss Will a gift of this caliber scare him off/ be considered too much after being laid off from his game dev job?
POST: My boyfriend's birthday December 24th, which Christmas Eve. He grew up in a foster home and never got great presents either his birthday or Christmas. He often spends his birthday and Christmas alone because his foster mother is uninterested and he doesn'tt 't have great relationship with his birth mother.
He spent Christmas with me last year, so went out of my way get him both a birthday present AND Christmas present, both of which valued above $50. However, he was so thrilled have someone to spend his birthday and Christmas with, he showered me with 5 very nice expensive gifts. I felt terrible. Happy, but terrible.
He is a video game artist but was recently laid off from his job. He is very passionate about his video game collection. There is game I noticed in his collection that he only has the sequel of. I asked him why because it is very unlike him. He said it because the original was wildly expensive and he already had hard time affording a mint condition version of the sequel.
have so many questions. Is this too much? Would it be creepy to get him something that he never asked for? What if giving him game gives him bad feelings about getting laid off? What we break up before Christmas and I am stuck with the game? Would collection he didn't earn himself?
the game have the same value Any help would be great. am torn.
Foster kid boyfriend with Christmas birthday and video game collection. Is it soon get him $500 game for his collection, especially after he has just been laid off from his game dev job?
Reference:
-4.88
PPO: Boyfriend of 6 months has very expensive game for his collection for his birth- day/ /Chriihissm.s. Would it scare him off/ be considered too much/ would it be creepy get him something he never asked for?
-8.63
DPO: boyfriend's s birthday /Christmas gift of game for his collection. Would scare him off/ too much? Would be creepy to buy him game he never asked for/ would be worth it?
-9.13
WPO: boyfriend's 's birthday / CChistmma gift of $500 very rare video game for his collec- tion. Would scare him off from getting it or would it be too much?
-11.5
RLP-SPG: Want to buy bf of 6 months a $500 very rare video game for his collection for his birthday/ / Christmas. Will gift of this caliber scare him off/ be considered too much after being laid off from his game dev job?
-9882
Ours: Want to buy BF of months rare video game for birthday Seathdday/hhriine /Christmas, worried about price/ value and whether gift would scare him off/ be considered too much. Help/ade advvce.
-5.41
Table 10: Example output by a Pythia-1B model trained with different RLHF methods on the summa- rization task.
25

---

Published as a conference paper at COLM 2025
Content
gold
is the difference between machine learning and deep learning?
Machine learning is a subset of artificial intelligence that uses algorithms to learn from data and improve performance ming. Deep learning is type of machine learning that uses artificial neural networks learn from large amounts of data and make predictions. Deep learning more powerful and accurate than traditional machine learning, but also requires more computing power.
Reference:
4.88
task without explicit human program-
PPO: Machine learning is subset of artificial intelligence that uses algorithms to enable computers to learn from data and make predictions decisions without being explicitly programmed. Deep learning
1.64
subset of machine learning that uses neural networks to learn from data and make predictions or decisions. Deep learning is more complex than machine learning and uses more advanced algorithms to learn from data.
DPO: Machine learning is a type of artificial intelligence that allows computer to learn from data without being explicitly programmed. Deep learning is subset of machine learning that uses neural networks to learn from data and is able to learn more complex tasks. The main difference is that deep learning uses neural networks to learn complex tasks and able learn from large amounts data,
3.44
WPO: Machine learning type of artificial intelligence that allows computer learn from data without being explicitly programmed. Deep learning is subset of machine learning that uses neural networks to learn from data and is able to learn more complex tasks. The main difference is that deep learning uses neural networks to learn more complex tasks than machine learning.
3.10
RLP-SPG: Machine learning is subset of artificial intelligence that focuses on devel- oping algorithms that can learn from data and make predictions decisions based that data. Deep learning learn from data and make predictions or decisions. Deep learning is more advanced than machine learning and can learn more complex patterns and relationships in data. Ours: Machine learning is a broader field that includes techniques such as supervised, unsupervised and reinforcement learning, Deep learning is subset of machine learning that uses neural networks with multiple layers to learn complex patterns in data. The main difference is that deep learning neural networks with multiple layers learn more complex patterns and more computationally intensive than traditional machine learning techniques.
0.82
subset of machine learning that uses neural networks
6.69
Table 11: Example output by Qwen 2.5 1.5B model trained with different RLHF methods on the truncated alpaca dataset.
26

---

Published as a conference paper at COLM 2025
Content
Rolld
SUBREDDIT: r/ AsssReddiit
TITLE: Reddit, think our family dog just consumed a significant amount of chocolate.
What do I do?
POST: Okay so my father and went out to dinner tonight and left the dog at home. I'd left half bar of dark chocolate (about 1.5 oz, still in the wrapper) sitting my desk. Mind you, the bar has been sitting the desk for about three days now and the climb on
dog hasn't done anything about it, and typically doesn't. She knows surfaces and has shown no interest the candy before.
However, returned home to find my bedroom in shambles. The contents of my desk were all knocked around and a bunch of family photos were all shredded and chewed up. I found the empty chocolate bar wrapper offending candy. can only conclude that the dog has eaten This is something that my dog would've 've done as puppy. However, she's ten now and hasn't t behaved like this in years. She's acting fine at the moment, but I'm still worried. My father doesn't I seem concerned about her at all, which worries me even more. FYI, my dog is dachshund mix and weighs about 25-30 pounds. Has anybody been in this situation before? What happened? Does anybody know what to do?
the ground and still can't find the
Reference: T think dog got into chocolate when was out. My dad isn't concerned but I'm afraid she's going to keel over and die and I don't know what to do.
-12.7
PPO: Dog ate chocolate bar, now my desk is destroyed and my father is not concerned. -9.43
What do I do?
DPO: Dog ate chocolate bar that left out for her. She's not acting like she's acting -9.75 normally. What do I do? Any advice?
WPO: Dog ate chocolate bar, my dog has never acted like this before. What do T do? -13.7 What do I do? What do do?
Ours:
family dog ate chocolate bar, now my room is wrecked and my father is not -7.06
concerned. What do do? Any advice appreciated.
Table 12: Example output by the Pythia-2.8B model trained with different RLHF methods the summarization task.
27


Reviews:
[{"review_id": "vNT0hejgCv", "reviewer": "Reviewer_4P8x", "summary": "This paper proposed a simple yet effective method to train reward models in RLHF to mitigate the distribution shift issue. The method applies an importance weighting factor in the loss function of reward model training, where the importance weighting factor is calculated by the current policy and the SFT policy. The effectiveness of the method is verified empirically. Sufficient ablation studies are provided, offering informative insights into the problem and method.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The issue and method are presented clearly with precise formulations and clear language.\n- The proposed method is both simple and effective.\n- The empirical experiments are thorough and comprehensive.", "reasons_to_reject": "- The clarity of the presentation requires improvement; refer to the questions for further details.\n- The results reported in Table 1 appear incomplete. Results for both tasks and both base models should be included for a more comprehensive comparison. Reporting results for each random seed is unnecessary. Providing the mean and standard deviation would suffice.", "questions_to_authors": "- Line 3, \"online RLHF\": The term \"online RLHF\" is used multiple times in the paper with a reference to Stiennon et al., 2020. However, to the best of my knowledge, this term is not well-established. In fact, it seems somewhat misleading, as neither the RLHF method presented in this paper nor the method proposed by Stiennon et al., 2020, are truly \"online.\" Both approaches involve collecting pairwise responses using a policy and annotating preference labels with human (gold) labelers. In lines 72-73, it is stated, \"While they focus on offline RLHF, we focus on online RLHF.\" I recommend providing clear definitions or appropriate references if the terms \"online RLHF\" and \"offline RLHF\" are to be used, or directly using the term \"RLHF\" instead.\n- Line 170 and Equation 5: Could you explain the meaning of $P_{\\pi^i}(s, a_w, a_l)$? Why does the equation $P_{\\pi^i}(s, a_w, a_l) = P(s) \\pi^i(a_w | s) \\pi^i(a_w | s) P(a_w > a_l | s)$ hold?", "extra_scores": {}}, {"review_id": "BMBiBeV580", "reviewer": "Reviewer_fvYh", "summary": "The paper proposes Off-Policy Corrected Reward Modeling, a method that addresses overoptimization in RLHF. It aims to correct the distribution shift between the SFT and the updated RL policy and apply importance weighting (IW). The authors provide a theoretical justification, derive estimation error bounds, and present experiments on summarization and chatbot tasks using simulated gold reward models.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It proposes a theoretically grounded and practically implementable method using off-policy correction via importance weighting.\n\n2. Empirical results compared to PPO, DPO, WPO, and RLP-SPG show the effectiveness of proposed IW methods.\n\n3. The paper is well structured and easy to follow.", "reasons_to_reject": "1. It only tested on two tasks (TL;DR summarization and Alpaca-Farm chatbot), both with synthetic gold RMs rather than human annotators or real-world alignment benchmarks.\n\n2. This is no downstream or qualitative analysis on reward changes. It is unclear how changes in reward scores reflect actual quality, diversity, or safety of generated outputs.\n\n3. Experiments are only conducted on small models, about 1B. Larger sizes of models are required to demonstrate its generalization and improvement.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dtBzyWiTGS", "reviewer": "Reviewer_FVSR", "summary": "The paper applies a technique to progressively update the reward model being used in RLHF. The authors explain, and show via a small example, how the reward model increasingly becomes less accurate as the policy is updated. They show that using a correction to the reward model enables a better final policy to be found, as judged by a gold reward model.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Clear explanation of the reward model drift issue, and benefits of using the importance weighting updates to the reward model are explained and demonstrated to have empirical benefits. Comparisons against relevant recent RLHF flavours are given, and these are done on public datasets. The showing of the results for different seeds is helpful, and the best performance of alternative methods in the comparisons is fair (using their best result, despite these dropping off in performance with continued training).", "reasons_to_reject": "No significant reasons. Some minor comments below. \nRetraining the reward model after each policy update is acknowledged as not feasible computationally, and the authors provide a pragmatic iterative approach of doing policy updates, followed by updating the reward model, and repeat `m` times. This isn't a reason to reject, just to note a limitation, which is overcome without any theoretical justification, but which empirically does prove to be helpful. It's unclear from the ablations why further iterations beyond m=3 were not done given the performance continues to improve?", "questions_to_authors": "How far off the accuracy of the \"gold reward model\" were the models being used here? Did I miss this or was it not commented on? Would be helpful to know.", "extra_scores": {}}, {"review_id": "UiUjWRLOg0", "reviewer": "Reviewer_ruLM", "summary": "The paper proposes a new method for correcting the reward model iteratively without access to new labels or data points in RLHF. The method is based on importance weighting to scale the reword model loss. The weight is computed as the ratio between the probability returned by the policy at iteration i and the policy at iteration 1. Experimental settings with two LLM families and two standard datasets show that the proposed method improves performance while reducing the overoptimization problems of RLHF. The authors provide a substantial analysis of the methodology including ablations.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper presents a simple yet effective method for tackling the overoptimization problem in RLHF.\n* The method works well across LLM families and tasks. The experimental setting is good. Larger models have not been used but that's OK.\n* Very good analysis and discussion of the methodology.", "reasons_to_reject": "* N/A. Really enjoyed reading the paper.", "questions_to_authors": "Minor commnets:\n\n* Perhaps you can improve the way you present the IW and how the ratio is calculated by making it a bit simpler (and easier to understand).", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses overoptimization in RLHF, where the policy exploits inaccuracies in the reward model arising from distribution shift. The authors frame this issue as an inconsistent estimation of the reward model parameters, which is insightful. Their proposed solution, to iteratively retrain the reward model using off-policy correction with importance weighting, is well-motivated. The empirical evidence on summarization and chatbot datasets demonstrates a clear and significant improvement over existing methods. Reviewer fvYh raised concerns around the synthetic settings of the experiments and generalization of the findings to larger models. The authors provided additional results showing benefits in more realistic and larger model settings. The paper is well-written, the approach is sound, and the results are compelling, making a valuable contribution to the field.
Decision: Accept