Pairwise Proximal Policy Optimization: Language Model Alignment with Comparative RL
Authors: Tianhao Wu
Banghua Zhu
Ruoyu Zhang
Zhaojin Wen
Kannan Ramchandran
Jiantao Jiao
Venue: COLM (2024)
Keywords: RLHF, Alignment, Comparative RL, LLM
TL;DR: We propose an algorithm that use comparative RL for learning human preference, out perform PPO and DPO

Abstract: LLMs may exhibit harmful behavior without aligning with human values. The dominant approach for steering LLMs towards beneficial behavior is Reinforcement Learning with Human Feedback (RLHF). This involves training a reward model with a human-labeled ranking dataset and fine-tuning the LLM with the reward signal using RL. Despite the fact that the reward is learned from comparing different responses, the RL stage doesn't involve direct comparisons. This inconsistency between reward learning and reinforcement learning stages exacerbates RL's instability. An example would be that the well adopted RL optimizer, Proximal Policy Optimization (PPO), could perform different gradient updates even for batches with identical human preference information. To address this, we propose a new framework, reinforcement learning with comparative feedback, and a simple policy gradient algorithm, Pairwise Proximal Policy Optimization (P3O), that learns to improve from direct comparison. Theoretically, P3O has the nice property of being invariant with any reward functions that contain identical preference information, while doesn't require learning a value function. Empirical evaluations demonstrate that P3O can align with human preferences better than existing methods. This suggest that comparative RL is strong candidate for aligning LLM with preference data.

Published as a conference paper at COLM 2024
Pairwise Proximal Policy Optimization: Language Model Alignment with Comparative RL
Tianhao Wu
Banghua Zhu Ruoyu Zhang! Kannan Ramchandran Jiantao Jiao'
Zhaojin Wen
'University of California, Berkeley
Abstract
LLMs may exhibit harmful behavior without aligning with human values. The dominant approach for steering LLMs towards beneficial behavior is Reinforcement Learning with Human Feedback (RLHF). This involves training a reward model with human-labeled ranking dataset and fine- tuning the LLM with the reward signal using RL. Despite the fact that the reward is learned from comparing different responses, the RL stage doesn't t involve direct comparisons. This inconsistency between reward learning and reinforcement learning stages exacerbates RL's instability. An example would be that the well adopted RL optimizer, Proximal Policy Optimization (PPO), could perform different gradient updates even for batches with identical human preference information. To address this, we propose a new framework, reinforcement learning with comparative feedback, and a simple policy gradient algorithm, Pairwise Proximal Policy Optimization (P3Ö), that learns to improve from direct comparison. Theoretically, P3O has the nice property of being invariant with any reward functions that contain identical preference information, while doesn't require learning a value function. Empirical evaluations demonstrate that P3O can align with human preferences better than existing methods. This suggest that comparative RL strong candidate for aligning LLM with preference data.
1 Introduction
Large Language Models (LLMs) have made remarkable progress, profoundly influencing the Al community [Chowdhery al. 2022 Brown et al. 2020 Touvron et al. 2023 Bubeck et al. 2023). However, due to the reliance on massive corpora of internet data, which encompasses a high portion of low-quality data, LLMs are likely to express unintended behavior. These include fabricating facts, generating biased or toxic text, and even harmful content to humans (Perez et al. 2022] Ganguli et al. 2022). Consequently, is crucial to align LLMs with human values, egg,, helpful, honest, harmless (Bai et al. 2022a).
leading method in AI Alignment for Large Language Models (LLMs), known as Rein- forcement Learning from Human Feedback (RLHF), involves learning reward function and Sine-tuning the model with this reward feedback using reinforcement learning (RL) (Ziegler et al. 2019: Ouyang et al.) 2022). Specifically, a reward model is trained to rank candidate responses to align with the human-labeled ground-truth. As for RL, Proximal Policy Optimization (PPO) is widely adopted the default optimizer (Schulman et al. 2017). PPO alternate between generate new responses and adjust the likelihood toward responses with higher reward. Despite its acclaimed efficiency, we identify the inconsistency between these two stages:
Inconsistency Between Reward Learning and RL. Although rewards are derived by ranking responses according to human judgments, the RL phase does not incorporate comparisons between generated samples. This leads to scenario where the reward signal can be highly variable, often being lower for more challenging prompts and higher for simpler ones.
*Corresponding to thw@berkeley.edu

---

Published as a conference paper at COLM 2024
Reinforcement Learning with Absolute Feedback
Reinforcement Learning with Comparative Feedback
RL
Ginerolise Advantage Recimate
update
update
Ri-Ra
Geadient
Policy Geadient
P3O Stroamline
single response por prompt
absolute preference
PPO Complicate
pairwise responses prompt
comparative preference
Figure 1: The figure on the left illustrates the prevalent method for fine-tuning LMs using RL, which relies on Absolute Feedback. In this paradigm, algorithms like PPO has to learn a V function, which capture not only the valuable relative preference information, but also less part, which is the scale of the reward for a given prompt. Contrastingly, the figure on the right presents RL paradigm that improve from direct comparison. Our algorithm generate pair of responses per prompt, leveraging only the Comparative Feedback derived from the difference in reward for policy gradient updates. This method obviates the need for additional critic learning and intricate components like Generalized Advantage Estimation (Schulman et al. 2015b).
When employing conventional algorithms like Proximal Policy Optimization (PPO) to optimize against such a noisy reward, it becomes likely for PPO to reduce the likelihood of a quality response to difficult prompt due to the absence of comparisons. This issue is further compounded by PPO's sensitivity to various factors including reward normalization, scaling, clipping, KL control, advantage normalization, and critic initialization (Zheng et al., 2023: Engstrom et al. 2020), contributing to its fragility. As consequence, we observe that PPO's s reward experiences a decline during a warm-up period, and the training is sensitive to random seed. From formal perspective,
argue this inconsistency by noting that the reward training objective, the Bradley- Terry Loss, is invariant to a constant shift (Yan et al., 2022), whereas PPO is not. This implies that even if two reward models contain identical human preference information, their optimization via PPO could lead to disparate results. (subsection 4.3)
In this paper, we provide new insights to address the inconsistency:
We define an equivalent relationship for reward functions trained from human prefer- ences. We identify that the widely adopted reward training loss, Bradley-Terry Loss, is invariant under this equivalent relationship, while PPO is not. As result, PPO may be less efficient learning the reward.
We introduce Pairwise Proximal Policy Optimization (P3O), under the framework of Reinforcement Learning with Comparative Feedback (Figure 1). P30 learns by comparing pairs of responses, avoiding learning critic functions, advantage estimation and various normalization techniques (Zheng et al. 2023). Empirical evaluations show that P3O consistently outperforms PPO and Direct Preference Optimization (DPO) in terms of GPT-4 Evaluation.
2
Related Work
Significant efforts have been made towards aligning LLMs with human values. These alignment strategies broadly fall into two categories: offline training and online training.
Offline training typically involve a static dataset, and doesn't 't require additional evaluations or generations. For instance, Thoppilan et al. (2022); Gunasekar et al. (2023) use instruction fine-tuning to update the model on high quality dataset tailored to a specific downstream task of interest. Snell et al. (2022) proposed to employ offline Q Learning to learn an add-on term for decoding. While Ratailov et al. (2023) introduced DPO, an offline approach that can directly align LM with human preference data, drawing from the closed-form solution of the Contextual Bandit with KL control problem. There are also methods like PRO (Song et al. 2023) and RRHF (Yuan et al. 2023) that fine-tune the model based ranking of the
2

---

Published as a conference paper at COLM 2024
rewards. Liu et al. 2024) unified several pairwise objectives under the LiPO framework and establish connection with Learning-tor Beiinng-t---ee tark-tt--nnkkkk Literature.
Our work is categorized under online training, which consist of loop of generating new u
responses from the updated policy, evaluating them with the reward model and updating the policy. The current dominant approach RLHF relies online RL methods such as PPO
(Schulman et al.
2017), A2C (Mnih et al. 2016) or their variants (Ramamurthy et al. 2022
Zhu et al. 2023c There are also few methods that deviate from this standard. For instance, ReST (uulcehre et al. 2023) use offline RL in the policy improvement phase. RAFT (Dong
et al. 2023a) iteratively fine-tune the policy on the responses generated by the Best-ot-N policy, Another paradigm parallel to RLHF is Reinforcement Learning with AI Feedback (RLATF) (Zhu al. 2023a b) (Bai et al. 2022b Lee et al. 2023 Yuan et al., 2024), which aim for using AT to improve Al. For example, Wu et al. (2024) proposed Meta- DeeaaReewrrding LMs, which significantly improve the Instruction-following and judging ability without relying
on human data.
Outside of the context of language, Contextual dueling bandit (Dudík et al. 2015 Yue et al. 2012) use preferences rankings of actions to adjust the policy, rather than rewards. Simlarly, PbRL (Xu et al. 2020 Jain et al. 2013 Busa-Fekete et al. 2014 Christiano et al. Sadigh et al.) 2017 Kupcsik et al. 2018) learn from binary preferences generated by some unknown scoring function.
2017:
2.1 Necessity of RL in LLM Alignment
The necessity of reinforcement learning (RL) for aligning large language models (LLMs) has been a topic of much debate. There are alternative approaches, such Direct Policy Optimization (DPO), which is a simpler method that utilizes a pre-collected offline dataset of preferences. On the other hand, RL. involves a more intricate process of interacting with the language model in real-time. This process includes generating new responses, evaluating them using a reward model, and then updating the language model based on the rewards.
Despite the complexity and the challenge of hyperparameter optimization inherent online RL methods, recent studies have demonstrated that, with careful hyperparameter tuning, online RL can produce results that are much stronger than Shannon--ue non-RL methods Zhut et al. (2023a); ;
Lambert & Calandra (2023); Dong et al. (003b);; ; Xuu et al. (2024). However, conc concuusively addressing the necessity of RL for LLM alignment would require more sophisticated tooling and evaluations, exceeds the scope of this paper.
3 Preliminaries
We briefly reviewing the RLHF pipeline in ZZiegler et al. 2019).
SFT Phase (Supervised Fine-Tuning This stage start with pre-trained LM, and then fine-tuned with supervised learning (typically maximum likelihood loss) on a high quality
dataset for the downstream task of interest. Outcome of this stage denoted as SSPT
Reward Learning Phase. In the second phase the SFT model is prompted with prompts to produce pairs of answers y1, 1/2 STSTT (y)xx) The responses pairs then presented to
human labelers who express preferences for one answer, denoted as you > y where yo is the one favored by the labeler and y; is the one less favored. Under these preferences is the inaccessible latent reward model r" (y, x). According to the Bradley-Teryy (Bradley &
Terry 1952 model, the human preference distribution p* can be expressed as:
1
p" (y1 y2/x)
1 exp (r (122x) - r*t (x))
Assuming the access to dataset {x',, y Hi sampled from p*. We parameterize the reward as ræ and estimate it via maximum lig-likelihood.
1
log o (ro(v(ll+)) ru(vlxx))
(1)
N
3

---

Published as a conference paper at COLM 2024
where is the sigmoid function. to is initialized with ΛF augmented by additional linear layers top. Constraints like E [r(y|x)] 0 might be incorporated to lower the variance.
RL Fine-Tuning Phase. Prior work formulate the optimization problem problem maximizing:
E
ro - (=xx (-1x))
(2)
DyD.y
The BD (TT
T (-1x)) 2 term is used to regulate the deviation from the SFT model, it is important to prevent the model from completely forget the world knowl- edge acquired in the pre-training stage. The standard approach is to directly employ PPO (Schulman et al. 2017 Ouyang et al. 2022) to optimize the modified reward (z(y)x) - B (log To ( log π (y(x)).
4
Algorithm
4.1 Proximal Pairwise Policy Optimization (P3O)
To derive P3O, start from Vanilla Policy Gradient (VPG, Pseudo (Sutton et al., 1999 Schulman et al. 2017 Wu et al. 2022). For clarity, we'll focus on the bandit setting, though it can be extended to contextual bandits as in Theerrem 4.1
In the bandit setting, assume we are updating a parameterized policy ny with actions denoted y. The VPG aims for estimating the following formula with samples:
LPG
IF (r(y) b) V log (y))
yru
where a baseline used for variance reduction. A common choice for the baseline is the mean reward, which gives:
V VL
VPC
EE (r(y)) -T
E r(y))) Vog (1(1
TT (r(y1) r(y)) V log (v1)
E 01222-70
(4)
Equation 4 highhlights the reliance the relative difference between rewards. Symmetrizing for y1, 3/2 yield:
TP((1)
VPG
(r(y1) r
log
/2
TTp (T)
Its immediate generalization to contextual bandit is the following:
Theorem 4.1 (Pairwise Policy Gradient) For any prompt x, the policy gradient can be expressed E v L PPC (x), where V CPC (x) can be expressed as:
V LVP
D-D
To (y1 /2
(r(y)(x) - r(y2/x)) Ilog
11
Tip (y2
To estimate the policy gradient with finite samples, considering that the replay buffer is collected using the previous policy Ta we further utilize importance sampling to correct the bias. The following theorem provides an unbiased estimation of the pairwise policy gradient:
Theorem 4.2 (Estimate PPG with Importance Sampling) For replay-buffer D2 {
(x', v y², r/)r collected using Ti The following is an unbiased estimation of pairwise
policy gradient:
Tea (aa (xt)
Tea Se(y//l''
7e (x/)
(1) 12)
V log
Tegua (1² lxt)
Thga (121xx))
Ne (r/]xx))
We further employ clipping to stabilize the policy updates, for more details refer to ssubsec tion B.2

---

Published as a conference paper at COLM 2024
Algorithm 1 Pairwise Proximal Policy Optimization (P3O)
1: Initialization: Initialize policy from the SFT model with parameters 0g
2: for = 0, 1,2 do
i 3: Sampling и prompts {x Ji=t from prompt dataset. Collect pairwise responses for each prompt by sampling from the latest polesy Teg
Vi. 1/ Te, (-1x²)
4: Score all the responses with a reward model model and aggregate the reward with KL divergence:
xx)x model (y/x) BDKL. (Fo. (-1xx) (T00))) (-1x))
5: Estimate policy gradient on the scored replay-buffer D₁ = {I' = (x' 1/
via (subsection B.2):
åR V (Dx)
P30 (DCP)0(DD)
6: Update the 0g via gradient descent and yield 0
7: end for
4.2 Relationship with PPO and DPO
Comparison with PPO: Although PPO and P3O both fall into the online RL framework, they differ in the way they perform policy update: PPO update based on an estimated advantage, while P3Ô update based on direct comparison of two responses. Consider simplified version of PPO applied to contextual bandit:
Top (y((yxx)
CPPO clip
E (+1x)
(r(y)x)
(x)
Tia (1/x)
yo
Where (x) is proxy to the ground truth value function al = Byr-n rly|x), usually
learnt via an additional regression loss. In contrast, P3O doesn't 't require learning the value function, this significantly reduce engineering efforts. P3O employ additional sample y/2 to estimate the gradient unbiasedly, and update the policy based on the comparison V r2-
Comparison with DPO: The gradient of DPO's objective function V. (x, Ywr y1) takes the following form:
To To(y xx) 7 STT (y)
Blog Te (V)(volx) 7TT
Ti TT (x) /2
Bo Bo
log
ng (y)
The direction of the gradient resembles that of our formulation in Theorem 4.1 However, the weight coefficients are different. The core difference of DPO and P3O is that P3O is an online RL algorithm while DPO is not. P3O learns to trials and errors by alternating between generate and update from human feedback while DPO is applied to fixed dataset. We empirically observe that DPO falls short on KL-control FFigure aand Figure 2) compared to P3O, we hypothesis that this because DPO aligns the policy towards the goal policy while doesn't directly consider the reward of the intermediate policies. Unlike P3O, which applies policy gradient based on the idea of strict policy improvement for every gradient update (Schulman et al. 2015a), DPO aligns the policy via an alternate "distance" where the intermediary steps are not guaranteed to maximize the KL-Reward trade-off. We note that P3O combines the benefits of PPO and DPO, offering guaranteed policy im improvement akin to policy gradient.
4.3 Reward Equivalence
We formally define the concept of reward equivalence (Definition 4.3) similar to the scale invariance defined in (Yan et al. 2022). We show that BTL is invariant under this equivalent

---

Published as a conference paper at COLM 2024
relationship in lemma 4.4
We then discuss why it leads to desirable property named invariance (Definition 4.5) that we want RL algorithms to satisfy. In the end, we present our main theorem (Theorem 4.6) which shows that PPO does not satisfy this property, contributing to its instability.
Definition 4.3 (Reward Equivalence) Two reward functions r(y(x) and (y(lx) are termed equivalent, denoted as ~ r', if and only if there exist a function 5(x) depend solely on the prompt x, such that for every prompt and response pair (x,y),
rlylx) 7 (ylx)=
The equivalent class associated with reward r is represented as [r]
Note that comparative losses such as Bradley-Teryy loss and Plackett-Luce loss, is unaffected
by shift in the prompt's reward in definition 4.3 This observation leads to the following Lemma:
Lemma 4.4 (Invariance of BTL) For two reward functions that satisfy Y ~ r', they both yield identical loss for any response pairs (or K responses) under the Bradley-Terry Loss (or Plackett-Luce Loss).
Lemma 4.4 underscores that the only information can learn from the preference data is the reward difference of two responses to the same prompt. This implies that direct comparison of responses stemming from different prompts should be avoided. This is because we can craft an arbitrary function denoted as á and replace P with the identical f f 8, while flipping the sign of F(ylxx (''''xx') As a result, an ideal algorithm should focus only on the relevant information within the reward function, filtering out the noise represented by This leads the following definition:
Definition 4.5 (Invariance) An algorithm is said to be invariant with respect to the equivalent relation "~",, for any two equivalent reward functions r r' and a fixed set of prompt and response pairs, the algorithm perform identical updates to the policy.
To illustrate definition 4.5 assume that we have two equivalent reward functions f and P' P 5. Notably, even when initialized with the same random seed, PPO can result in distinct updates for an identical batch. This behavior can be attributed to PPO's reliance on learning a V function estimate advantage. In the simplest scenario, where the advantage is estimated via one-step TD (Adv(y)x) = r(ylx)) Vxx), corresponding to ACAE 0) and single token, we should expect the advantage function stay unchanged. However, following the derivation
Advx(y|x) = Adv:(1/22)
f(ylx) - Vx(x) = +'(y|x) - Vp (x)
Vx(x) V,(x) =d(x)
We can see that even though and /' are equivalent, they yield different updates for function. This give rise to our main theorem (full proof in insubsecction C.2):
Theorem 4.6 (Non-invariance of PPO) P3O is invariant with respect to "~". ". In contrast, PPO not, given the same initialization of V.
Experiments
We conduct experiments on two widely-adopted RLHF tasks, summarization and question- answering, and we find that P3O achieves better performance in terms of both KL-Reward trade-off and quality of generation, against several strong baselines. Refer to Appendix D for detailed hyper parameter selection process.
Tasks. We explore two different open-ended text generation tasks, i.e. summarization and question-answering, For both tasks, algorithms are given reward model pre-trained from

---

Published as a conference paper at COLM 2024
HH with 18 Model
HH with 10 Model
Rewerd
Reward
P304G
P2042
Kl-divergence per sample
Ki-divergence per sample
HH with 68 Model
with 68 Model
Rewlard
Reward
PED
DPO
KL-divergence per sample
KL-diverpence per sample
Figure 2: KL-Reward frontier for HH: x-axis and y-axis represents ( (Ttl| and the reward respectively. Each point represent an average of results over 280 test prompts and calculated every 500 gradient updates. Left two figure compare P30-V1 and PPO with varying base model sizes; Right two figures compare P30-V2 and online-DPO. Results showing that P3O can not only achieve higher reward but also yield better KL control.
a dataset of preference (x(1), Yw yi h and the goal to obtain a policy π xyy(x) that can generate high-quality response y given prompt x. In summarization, we use the TL;DR (too long; didn't read) dataset (Völske et al. 2017), where x is a forum post from Reddit, and is corresponding summary. We use 6B SFT model CarperAI/openal summar tldr sft as the initial policy and EleutherAI/ gpt-j-tt- j-bb as the reward model. In question-answerrin is a human query, which may come from diverse topics, and the policy should learn to produce engaging and helpful response y. Following prior work, we use the Anthropic Helpful and Harmless (HH) dataset (Bai et al. 2022a). We fine-tune two policies of sizes and Dahoas Seahhii----sttti--ee
(2)
Both models have gone through supervised tine-tuning with labeled prompt-reyponse, pairs, similar to the protocol in Ouyang et al. (2022) and Ramamurthy et al. (2022). For the reward model, we use the 6B model Dahoas/gptj gptj--m-stan on EleutherAl/gpt-jioe
(1B,6B),
trained from the same dataset based a proxy of human preference.
Methods. We compare two versions of P3O, P30-V1 and P3O-V2, which represent clipping separately and jointly respectively, with several effective and representative approaches for LLM alignment. We start with the SFT policy trained by token-wise supervised fine- tuning. hasn't gone through further alignment; Every other method uses the SFT model as initialization. For RI. algorithms we consider the dominant approach PPO (Schulman et al. 2017 2017 Ouyang al. 2022) with reward specified in[uquation 2) We follow the implementation of trlx (Castricato et al. 2023). Besides, we also consider the newly proposed DPO [Rafailov et al. 2023) method that directly optimizes the policy towards the closed-fomm solution of the KL-constrained reward maximization. Although DPO is proposed as an offline
Among methods directly involving RL, we note that PPO and a modified version A2C Mnih et al.
2016 Lee et al. 2023) are the only two current online RI. methods for L.M alignment. However, there no strong evidence showing the supremacy of A2C over PPO, choose PPO baseline.
7

---

Published as a conference paper at COLM 2024
65
50.5
77.9
70
-6
Mooor Mi natt [%6]
DPD 45.4
50.0
54.0
67.3
55
DPO
49.5
50.0
62.1
76.3
40
40
SFT
30.7
32.7
38.4
50.0
SFT
22.1
23.7
30.4
50.0
Figure 3: flead-to-head comparisons between (P3O, DPO, PPO, SFT). Left figure displays the win rate as evaluated by GPT-4. Right figure presents the win rate based on comparison of the proxy reward. Despite the high correlation between the figures, we found that the reward win rate must be adjusted according to the KL to align with the GPT-4 win rate.
alignment method, we notice that we can make it online with the help of a proxy reward function. (More details can be found in mssubeection B.3
Evaluations. Deviating too much from the reference policy (egg. SFT model) would lead the online policy to cut corners of the reward model and produce incoherent continuations, as pointed out by previous works (Ziegler et al., 2019). Gao et al. (2023) studied the a synthetic setup, where labels supplied
scaling law of reward over-optimization by "gold standard" reward model. They empirically find out the golden reward can be approximated by a simple function form involving the square-root KL-divergence from the reference policy: Therefore, it is important to balance the trade-off between the KI.- divergence and asymptotic reward, and we measure the effectiveness of each algorithm by its frontier of achieved reward and KL-divergence from the reference policy (KL-Reward Frontier). To directly evaluate the quality of generated responses, we also perform Head- to-Head Comparisons between every pair of algorithms in the HH dataset. We two metrics for evaluation: (1) Reward, the optimized target during online RL, (2) GPT-4, as faithful proxy for human evaluation of response helpfulness. For the latter metric, shall point out that previous studies show that LLMs can be better automated evaluators than existing metrics (Chen et al., 2002),, and GPT-4 judgments correlate strongly with humans, with human agreement with GPT-4 typically similar or higher than inter-human annotator agreement (Rafailov al. 2023 Additional details can be found in Appendix D
5.1 KL-Reward Frontier
We conduct experiments on both TL;DR and HH datasets to evaluate the efficacy of the alignment algorithms in optimizing reward while restricting policy deviation from the reference. Figure and Figure demon- strate the Kl.-Reward frontier for IL,DR and HH respectively. Each point represents the average evaluation ery 500-step interval. The x-axis represents the average sequence-level KL-divergence ( I whereas the y-axis stands for the average reward given by the proxy re- ward model. For summarization task, we find that P30-V1 can reach a slightly higher reward than P3O-V2, while with a worse KL- Reward trade-off. Consequently, only P3O- V2 is included in Figure 4ffor comparison. We find that P30-V2 able to produce almost
Reward
730
test prompts at ev-
KL-divergence sample
Figure 4: KL-Reward Frontier for TL;DR:
The x-axis represent DKL ( ( y-axis represent the reward evaluated by the re- ward model, both averaged over 200 test prompts and evaluate every 500 gradient steps. We find that simple linear function fit the curve well. P30 have the best KL- Reward trade-off among the three.
8

---

Published as a conference paper at COLM 2024
the same highest reward whilst maintaining superior KL efficiency. DPO, despite its faster convergence, exhibits a 25% higher KL-divergence than P3O-V2 under the same reward. For the question-answering task, P3O-V1 and P3O-V2 have strictly dominant frontiers than PPO and DPO respectively in both model sizes, shown by Figure 2 Empirical findings establish P3O's superior trade-off between KL. and Reward over other baselines, delivering a substantial higher reward in the range of 0.1-0.3.
5.2 Head-to-Head Comparisons
We further conduct head-to- head comparisons between each algorithm pair among P30, DPO, PPO and SFT. Since the KL-Reward frontier indicates that joint-clipping (P3O-V2) produces more sta- ble results than separate- clipping (P3O-V1), we only consider P3O-V2 in this sec- tion and refer it as P3O. We sample completions from dif- ferent policies on the test of the HH dataset at default temperature 1.0, and we compute the average pairwise win rate using (1) reward and (2) GPT-4 as evaluators. Previous studies (Chen et al.) 2023 Rafailov et al., 2023) have shown that GPTT-4 is faithful proxy for human preference and is widely adopted for comparisons. The prompt used for evaluation is presented in sub- section D.2 Figure 3 presents the comprehensive pairwise comparison results, both via proxy reward and GPT-4. The average KL-divergence and reward ranking of these models DPO P30 PPO > SFT. Although DPO marginally surpasses P3O reward, has considerably higher KL-divergence (Table D, which may be detrimental to the quality of generation. As result, DPO has reward win rate 49.5% against P3O, but only 45.4% as evaluated by GPT-4. Compared with other methods, P30 exhibits GPT-4 win rate of 57.0% against PPÒ and 69.3% against SFT. This result consistent with our findings from the KL-RReaard frontier section, affirming that P3O could better align with human preference than previous baselines.
Table 1: Statistics for the checkpoints we used in GPT-4 eval- uation: PPO and SFT tends to generate long responses while P30 and DPO generate shorter responses. Moreover, P3O achieves nearly the same reward with DPO while incurring much less KI.
P3O
DPO
PPO
SFT
Reward f
-03302
-0.298
-0.613
-1.15
KL (sample) 4
9.83
12.01
7.02
0
KL (token) + 0.12
0.14
0.06
0
Token num
80.46
88.84
109.03
112.70
Conclusion & Future Works
This work presents new insights into aligning large language models with human prefer- ences via reinforcement learning. We introduced the Reinforcement Learning with Com- parative Feedback framework, which unifies the core principles of reward modeling and RL fine-tuning, Our empirical evidence compellingly supports the notion that comparative RL, by leveraging direct comparisons, presents a more effective approach for alignment than traditional non-comparative RL methods. Within this framework, we developed P3O, which utilizes pairwise comparisons to perform policy updates. Our empirical assessments have shown that P3O not only surpasses previous methodologies in achieving a better balance on the KL-Reward frontier but also demonstrates superior performance in GPT-4 win-rate comparisons. P3O encapsulates the benefits of policy gradient techniques while simplifying both the algorithmic construction and function approximation.
Looking ahead, several intriguing questions arise for future exploration. Firstly, aim to understand the impacts of reward over-optimization on trajectory- RL algorithms and token-based RL algorithms. Secondly, we are interested in whether can generalize the policy gradient algorithm to accommodate more than two ranked responses, potentially enabling better trade-off between human effort and AT alignment. Finally, wish to explore the benefits of applying P3O in contexts beyond training language models with human feedback. We eagerly anticipate investigating these questions in our future work.
We select checkpoints with the highest reward for generation.

---

Published as a conference paper at COLM 2024
References
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. Xiv preprint arXiv:2204.05862, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from feedback. arXiv preprint arXin:2212.08077, 2022b.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, (9(3/4)324-344 1952.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. ar preprint ar arXi::2303.127110
2023.
Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preference-based reinforcement learning: evolutionary direct policy search using preference-based racing algorithm. Machine learning, 97:327-355 351,, 2014.
Louis Castricato, Alex Havrilla, Shahbuland Matiana, Duy V. Phung, Aman Tiwari, Jonathan Tow, and Maksym Zhuravinsky. trlX: A scalable framework for RLHF, June 2023. URL
http::::/ggiiibbb..mm/Crrrrr///p com/Carrerrtttrll
Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. arviv preprint arXiv:2304.00723, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint ar arXiv:2204.02311 2022.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. aiv preprint arXio:2304.0677 0666770.77777. 2023a.
Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. Steerim: Attribute conditioned sft as an (user-steerable) alternative to rlhf. arXiv preprint axXiv:2310.05344, 2023b.
Miroslav Dudík, Katja Hofmann, Robert E. Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual dueling bandits. In Conference on Learning Theory, pp. 563-587. PMLR, 2015.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. arvio preprint arXiv:2005.12729, 2020.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. ar preprint arXit:2209.077 078888 2022.
10

---

Published as a conference paper at COLM 2024
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, PP. 10835.10866. PMLR, 2023.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjje Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXie:2308.0899 2023.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arv preprint ar Xrte:23306 11644, 2023.
Ashesh Jain, Brian Wojcik, Thorsten Joachims, and Ashutosh Saxena. Learning trajectory preferences for manipulators via iterative improvement. Advances in neural information processing systemis, 26, 2013.
Andras Kupcsik, David Hsu, and Wee Sun Lee. Learning dynamic robot-to-human object handover from human feedback. Robotics Research: Volume 1, pp. 161-176, 2018.
Nathan Lambert and Roberto Calandra. The alignment ceiling: Objective mismatch in reinforcement learning from human feedback. arXiv preprint arXiv:2311.00163 2023.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arxiv preprint ar XrXiv2339.0006:1 2023.
Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Salch, Simon Baumgartner, Jialu Liu, et al. Lipo: Listwise preference opti- mization through barminn--to-rank. ar arv preprint ar arXi::2402.018 2024.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928-1937. PMLR, 2016.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diego Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. Xrviv preprint ar XXiv:2202.03288 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. ar preprint ar Xix::2305..18900 2023.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv:2210.01241, 2022.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia, Active preference-based learning of reward functions. 2017.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897.
PMLR, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arv preprint arXit:1506.022333 2015b.
11

---

Published as a conference paper at COLM 2024
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithmss arXio preprint Xiiv:1007.06441 2017.
Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language learning. ar preprint urXiv:2206.11871, 2022.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. arXiv preprint (rXio:2306.17492, 2023.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour, Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. Xrv preprint ar Stte:22010022333 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar 1rX1912302.13971, 2023.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TI; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pp. 59 63, 2017.
Tianhao Wu, Yunchang Yang, Han Zhong, Liwei Wang, Simon Du, and Jiantao Jiao. Nearly optimal policy optimization with stable at any time guarantee. In International Conference on Machine Learning, pp. 14243-24266 PMLR, 2022.
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judae Xiv preprint drXivv2407.199 19594, 2024.
Jun Xu, Zeng Wei, Long Xia, Yanyan Lan, Dawei Yin, Xueqi Cheng, and Ji-Rong Wen. Reinforcement learning to rank with pairwise policy gradient. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.509 509-518, 2020.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for I1m alignment? a comprehensive study. arXiv preprint ar Xiv:2204. 10719, 2024.
Le Yan, Zhen Qin, Xuanhui Wang, Michael Bendersky, and Marc Najork. Scale calibration of deep ranking models. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4300-4309, 2022.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint XXXx:2441.1102000 2024.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. aiv preprint ar Xix::3044.0533 2023.
Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences, 78(5):1538-1555 2012.
Rui Zheng, Shihan Dou, Songyang Gao, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Limao Xiong, Lu Chen, et al. Secrets of rlhf in large language models part i: Ppo. ar Xiv preprint ar axX1::2307.04964 2023.
12

---

Published as a conference paper at COLM 2024
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving Ilm helpfulness & harmlessness with rlaif, November 2023a.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving Ilm helpfulness & harmlessness with rlaif, 2023b.
Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael Jordan, and Jiantao Jiao. Fine-tuning language models with advantage-induced policy alignment. arv preprint ar arXiv:2306.02233 2023c.
Daniel M Ziegler, Nisan Stienmon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. ar Xio preprint ar Xix::1909.08593, 2019.
A Appendix
13

---

Published as a conference paper at COLM 2024
B Algorithms
B.1 Pseudocodes
Algorithm 2 Vanilla Policy Gradient
Initialization: Initialize policy parameters &g and value function parameters do 2: for = 0,1,2... do
3: Collect trajectories D; = { by running policy πe, starting from a batch of prompts and generate single trajectory from each prompt.
4: Compute token-wise rewards contain both token-wise KL and preference reward as
in Equation 2 And then rewards-to-go R,.
5: Estimate advantage estimates Adv; via GAE or other methods.
6: Estimate policy gradient via:
1
8x
Σ
Σ
Adv; Ve log (a((s:))
[D R-D
7: Apply gradient updates 0g using gradient descent.
8: Fit value function by regression on mean-squared error via gradient descent:
1 T
T
4 = arg min
Σ
Σ
(Vo(st) - R₂)²
D
TCD, !=
9: end for
We present the pseudocode for both the Vanilla Policy Gradient (VPG) and our proposed algorithm P3O. While both algorithms follow the similar procedure of collecting trajectories and leveraging these trajectories to estimate the gradient, there are key differences: Our method collect pairwise trajectories and compute trajectory-wise rewards. This approach eliminates the need for estimating the value function V and bypasses the requirement of estimating the advantage function using Generalized Advantage Estimation (GAE). Consequently, P3O is not only simpler to implement but also introduces less bias into the estimation of the policy gradient.
B.2 P30: combine PPG with clipping
Clipping Separately (Version 1): For {i,}} {1,2},
(r(y:lx)- - r(())x))) 7&0 (y:]xx)
ne (ayyy)xz
C
P00 (x)
sg
TT
T8wu (gj(x)
TE (y:(x)
Te (x
clip Tip To(y;
(x) CP30 i,clip
sg
(r(ys/x) -r(y)xx))
To (yi(x) 1 E, +e)
V1 old
new (z/(x)
CP30
P30 min (x) (,1ipp (x)) + min (x), CP30
ip (2)11((x)) /2
X
Clipping Jointly (Version 2):
Tep (1/ (x)
Tep (12(2x
log TTp TT
(x) C
E yT
(r(y)lx) - -r(y21x))
Teaaaa (y112²)
Teala (y2|x)
TTp (x
Top (5/ (x)
To (ro(y21) xx)
CP30
(x)
E sg y1 MA old
(r(y)1x) r(y2lx))
TT (3112))
Th (y2) xx)
log [1(()1x)) .log
Toda (y1lx)
Toss (y/12)
clip
e, log
Te (y2 +
π(p T x))
nep (y2(x)
CP30
E min
P30 (x) CP30 (x))
jji x~D
14

---

Published as a conference paper at COLM 2024
B.3 Derivation of DPO
DPO start with preference dataset 'D and minimize the loss:
Aep (p((ylxx) Blog (yFl()
2i8 (yr (yr1(x) 7SFT (y)lx)
CDPO
logo (x.yorM))
Blog
However, this is offline since the algorithm only make use of fixed dataset. Instead, notice that if we have a reward function r, we can use the reward function label the preference result in an online fashion. Assume there are two new generated responses 1/, that have reward 72. Then we simply label the preference according Bradley & Terry (1952).
y1 12
w.p. o(1 - r2)
1/2 > y1 w.p.
0(r2 - r1)
We would like use the notation yv and yi to represent the preferred and less preferred response chosen by the reward. We collect all the newly generated responses into replay buffer D Preplay, therefore we can optimize the same DPÒ loss here:
Tep (yw STTT (1)
78 (y x))
DPO
E
logo
(llog
Blog
S (y)
(x.Me.yi)) Jo Ir replay
We can further reduce the variance of the loss by eliminating the randomness labelling the preference by incorporating the known labeling probability explicitly in the formula,
To ( [x) SFT (y xx
7o (y(22) (x)
CDPO_
E
logo EB EBllog
Eß
(x,y) 1-20201000 -D replay
SFT (y2 x
Here, Ber(o((r - r2)) is the two point Bernoulli distribution on { 111,
C Proofs
C.1 Proof Theorem 4.1
In the contextual bandit setting, VPG aims for estimating the gradient VLG V (x), where CPG (x) can be expressed as:
VPG (x)
E
r(y|x) V log 7 (ylx)
y
The expectation can be replaced with summation, leading to:
VC VPG
Σ
(5)
Σ [ E ((m
(6)
91.822
Σ ((gn)x))modppp)))V7T
Σ 31.42
(7)
Mi.222
= Σ (r(v1lx) - r(y2lx)) r(y2))) Tep (1222 V Te (x/lx)
(8)
91.42
(r(y)lx)~r(yal) r((2)x)) Ne (y2(xx V V
EB
(9)
91.22 700
Te xx New (y1
In Equation 6
we subtract the latter term which equals to because En Vx V = O. We further multiply the first term by 1 = Σy, To (1/2 |x) in
in Typ 1/1 TV x
Equation Finally, we rephrase the previous equation using importance sampling and
yield Equattin 9
15

---

Published as a conference paper at COLM 2024
Swap actions y2 and average togetherr get the desired form: VTp (x)
To(y2(x)
TV(V11
VT (122 (v2lx)
(x) =
IE
(r(y1|x) r(y21x))
/2
y1.92 old
ood (32) x) ooold (y1 (x)
Data (y1 xx)
Tidata (y22 x)
7ta (y (x)
70 (12 x)
7C0 (y1 |x)
IE y1-22-Rolld old
(r(y)(x) r(y2lx))
V
log
/2
Todal (y1 x) Tooal (
TTA (12 xx
C.2 Proof of Lemma
In this proof, we aim to show that two equivalent reward functions r and r' yield the same loss under the Tradley-Terry model. Assume that r ~ r', then by definition there exist ((x) such that for any prompt and response pair (x, r' /(ylx) = rlylx) + 0(x).
Consider any prompt and two responses yeu y/ labeled by human. According to Equation 1 the Bradley-Terry loss for this pair given reward is:
loss = log 0 (r(ywl>) r(ylx))
Similarly, the Bradley-Terry loss for this pair given reward r' is:
loss' log
o (r (F'w/) r
By substituting r' (y)xx) with r(y]x) + 8(x) in loss', we get:
r' r'yy x) - /' [ = (r(y)] + (x)) (r'(v)x)) + 8(x)) = r(ywlx)) - -((/(xx)
This shows that loss' loss, indicating that the two reward functions and r' are indeed equivalent with respect to the tradley-Terry loss. The same proof would go through for the Plackett-Luce loss, which we omit here for brevity.
C.3 Proof of Lemma 4.6
We first prove the invariance for P3O and DPO, then we prove that PPO is not invariant.
Assume that we have two equivalent reward functions by definition there exist 8(x) such that for any prompt and response pair (x, r' (y(xx) = r(ylx) + 5(x).
Invariance of P3O: This is trivial since the gradient directly involve ' - r2 We take P3O-V2
as an example and write the gradient formulation with respect to the prompt responses pair
(x, yı, y2):
If the reward is the update follows:
7i (111 (x)
Ti T
log 7ig TT (y)l1)) 7Ip (y2)x)
CP sg
(r(y))x) r(y2lx))
Tooa (y (x)
oo (12)x)
T [y1 (x)
TTp (y21x)
rP3
(r(v)lx) r(y2lx))
relip
sg
oid (1/1 (x)
Ti (12(x)
log Tg(yy12))
log Te (11 (x)
E, Log Ti (y: (x) ne (1/ xx) + 6
x clip
7Tp (1)22x
no (121x)
V C300 V min( P30 c,2211))
rji
Similarly, if the reward is ,', the gradient
Tpply) (x)
To (y(2))))
T (x) TT TT x)
c.30 sg (x((1xx) -x(yxxx)
log
Te (y)(x)
Te (y2(x))
P30
(f())lx)) - (y)2xx)
Ttp (v(l)xx
Ie (x(222))
Pllppp sg
Ti (y)1x)
Teas (y2/x)
[og T xx)
Tobas (11 |x)
Tedaa (y1(x) + + nea (y2(x)
x clip
log
Jo (12/x) - e, eolog
rTo (ro(yylx²
V P30
minn
PP3
jui
cllpp
16

---

Published as a conference paper at COLM 2024
We can see that the only difference between these two updates in the reward difference part. However, due to the fact that
(r'(11xx) - (y2 (r(y) (x) 5(x) (122))) (r(y)xx) r((2(xx)
We conclude that 2,30 cp and r,clip
C. ",lippp
Consequently, the two updates
v LP0 V P are the same.
Invariance of DPO: Assume the same setting as in the previous paragraph:
The gradient of DPO given reward r can be written as:
TEa (a2yy/2) Te eBlog log 7SFT (3/2 xx) logo (BBlog SFT (v:
V. C.P e-BBrrfe(r m2))
E
Similarly, the gradient of DPO given reward r' can be expressed as:
Tep Ty(l)) SFT (y)lx)
Ttp (122 xx)
VC
E e-Berrferr, x)))
logo
(BBlog
eplog SFT (y2)x))
The only difference between these two equations the sampling distribution of the Bernoulli
distribution. Easy to verify that they are the same since ( r2) or² -r²)..
PPO is not Invariant: The loss of PPO the combination of policy-loss and V-loss, with the trade-off of these two terms controlled by hyper-parameter y/:
CPPO
Pollky + nLv
Suppose the policy network and the V network have separate parameters, then taking the gradient of _PPO is simply taking gradient of Epolicy. We aim to prove that the gradient of pplicy is not identical for two equivalent rewards. We first recap the formula of Epolicy:
[a((((s)) Te (at
Ag(at(5))
Cpolicy =
E
min
Adv(a)ls;),
c, + Adv(a:lst)
)
Aa (#r(s))*
Where the Adv is estimated via GAE:
Adv(ai(s:) 5, + + (17) T-141 87
8; r(s;, a;) + (enl))+VV((s /V(s=11) - Vss))
For simplicity, we consider the one-sample case, where we are taking gradient with respect to the sample as), According to the formula of Adv, and combine with the fact that the reward is only appended to the last token T'. We have the following relation,
Adv (s)) (dv + (A7) 1+
Here, 8 8(x), represent the prompt corresponding to st, which is prefix of st. As
result, V min
mo(a(ls)
Adv(ai clip
Ta (4,),)) e, +e Adv(arls:) will not stay
mg(4)14)
Aa (4,55)
unchanged for different reward r, since Adv can arbitrary real number by choosing
D
Additional Experiment Results
D.1 Setup
For the hyper-parameter tuning, we first run PPO to search the learning rate among (0.5, 1, 4, x 10 that yields the best KL-Reward frontier (Figure 5). We then use the same learning rate for P3O and online-DPO without further hyper-parameter tuning. To ensure fair comparison, we double the batch size of PPO such that every algorithms can see the same number of responses, although P3O and online-DPO only see half the prompts.
17

---

Published as a conference paper at COLM 2024
Besides learning rate, we also tuned discount factor, GAE lambda, and clipping threshold. We found that a discount factor 1 performed best, and adjustments to GAE lambda and clipping within wide ranges did not significantly impact the training curve.
We identified learning rate and batch size as the most critical factors affecting PPO's perfor- mance. We observed that smaller batch size improved convergence speed, while larger batch sizes stabilized training but at a slower pace. We also considered GPU memory limitations and set the gpu-mini-batch size accordingly.
We repeated each experiment three times.
D.2 Prompt
For the following query to chatbot, which response is more helpful?
Query: <user_query>
Response A:
<response_a>
Response
<response_b>
FIRST provide one-sentence comparison of the two responses and explain which you feel is more helpful. SECOND, on a new line, state only "A" or "8" to indicate which response is more helpful. Your response should use the format:
Comparison: <one sentence comparison and explanation> More helpful: <"A" or "B">
To eliminate any potential bias from the order of the responses, the order of Response A and Response is randomized for each prompt. We note that although we ask GPT-4 directly respond A or B to indicate the preference, in approximately 3% of evaluations, GPT-4 found that neither response adequately addressed the user's query.
18

---

Published as a conference paper at COLM 2024
D.3
GPT-4 Evaluation Results
Prompt
Human:
help with understanding bitcoin
Assistant:
P30
Yes, that's fun question! Bitcoin can be seen as digital currency, but there are few differences.
What makes Bitcoin different is that it uses a digital wallet, instead of a cash-like system. This allows you to spend anywhere you want, without needing a card or bank account.
You can also use Bitcoin to spend other forms of currency.
This use is growing in popularity, and Bitcoin is also helping people move towards digital currency use.
DPO
Can you describe bit more about what Bitcoin and what it's used for?
PPO
What would you like to know?
SFT
What is Bitcoin?
GPT-4
Response A[P3O] provides brief explanation about Bitcoin and uses, while Response B[DPO] asks for more information from the user instead of providing any information.
Comparison:
Comparison:
Response A[PPO] is vague and doesn't provide any information, while Response B[P3OJ provides a detailed explanation about Bitcoin and its uses. Comparison: Response A[SFT] is question rather than an explanation, while Response B[P3O] provides detailed explanation of what Bitcoin is and how it works, making it more informative and helpful.
Table 2:
P3O outperforms the others by providing comprehensive explanation about Bitcoin and its uses, while the other algorithms respond with questions or requests for more information, rather than providing substantive information.
19

---

Published as a conference paper at COLM 2024
Prompt
Human: need help deciding what city to move
You could just use Google for that, or get specific about the criteria you're looking for, like weather and job markets. I'd like mid sized city with fun activities, good job market, and nice weather if possible
Assistant:
Human:
Assistant:
P30
Maybe try Austin, Dallas, or Phoenix. They might be possible choices.
DPO
That's a very reasonable description, and it comes with an affordable cost of living.
PPO
think that depends a lot on your situation. Is there job you're particularly interested in?
SFT
Is this for work, or pleasure?
GPT-4
Response AJDPO] is vague and doesn't provide any specific sugges- tions, while Response B[P3O] provides three specific city suggestions that could potentially meet the user's 's criteria.
Comparison:
Response A[PPO] seeks to gather more information to provide more tailored suggestion, while Response B[P3O] provides immediate suggestions based on the criteria given.
Comparison:
Comparison: Response A[SFT] seeks further clarification while Response B[P3O] provides direct suggestions based on the criteria provided by the user, making it more helpful.
Table 3: In each comparison, P3O outperformed the other algorithms,, effectively suggesting three potential cities (Austin, Dallas, or Phoenix) that align with the contrast, DPO's response was rather vague, offering no specific suggestions. PPO and SFT sought to gather more information, delaying immediate assistance.
user's criteria. In
20

---

Published as a conference paper at COLM 2024
Prompt
Human: How can I make my own soap at home? Assistant:
P3O
There are few different methods for making your soap at home. Oil or lye are usually the most common, but you can also use a few different types of chemicals to get similar effect create unique variation. Some effective ways include freezing the ingredients, using deodorants or oils, and adding essential oils vinegar give it nice drying effect.
DPO
I wouldn't recommend it, it's little fancier and not something you can do at home.
PPO
Making your own soap is a nice way to get involved in your own community. You help provide vital assistance to the local community, and learn about effective treatments and treatments for health problems.
SFT
Well historically, in Europe soap has typically not been watering shells and Ocean Aerosols, combination generally indicated as "EST date" secret date from the manufacturer. Soil type and utilization may be shedding light on the most efficient method for neutralizing and purifying chlorine, various chemicals involved in natural fragrances and essential oils. To use composting, cooking oils temperature has to be low enough to assist foam building. Additionally, using an electric pan, "overcup" for cures, and "a015" for the final soap. How do you plan to save your energy and waste water?
GPT-4
Response A[P30] provides detailed explanation of different meth- ods for making soap at home, while Response BJDPO] discourages the idea without providing any useful information.
Comparison:
Comparison:
Response A[PPO] is vague and doesn't provide any specific steps or ingredients for making soap, while Response B[P3O] gives general idea of the process and ingredients involved in soap making.
Comparisons Response A[P30] provides more straightforward and understand- able explanation of how to make soap at home, while Response B[SFT] is confusing and uses unclear terminology.
Table 4: P30 is assessed as the most helpful by GPT-4. It provides a detailed explanation of different methods for homemade soap creation, mentioning common ingredients and specific methods. Conversely, DPO discourages the idea without giving any constructive guidance. PPO fails to offer any specific steps or ingredients for soap creation. Finally, SFT delivers response that is complex and difficult to understand, featuring unclear terminology.
21


Reviews:
[{"review_id": "CrX36iQ0xi", "reviewer": "Reviewer_uGsp", "summary": "This paper proposes a new algorithm for RLHF, called Pairwise Proximal Policy Optimization (P3O). In an online training fashion similar to PPO, this algorithm uses a comparative signal between two sampled responses to train the policy model, which aligns better with how the reward model is trained. It also demonstrates some other properties, such as removing the need for a value model and being invariant to \"information-equivalent\" reward functions. The author provides detailed derivations of the P3O algorithm based off vanilla Policy Gradient, and also conducts experiments on summarization and question answering tasks to show the superiority of P3O over PPO and DPO, in terms of KL-Reward frontier and GPT4-judged win-rate.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate your recognition of the novelty of our paper. We will endeavor to address your concerns in our revised draft.\n>They mainly use some relatively old and weak pretrained models…\n\nThe decision to employ these specific models was driven by considerations of flexibility and consistency. The Pythia series offers a wide range of model sizes, from 70M to 12B parameters. This flexibility in sizes is particularly valuable in research settings, where we’re able to initiate our experiments with smaller, more manageable 1B models on V100s and then scale up to larger 6B models on A100s. Furthermore, employing the pythia models ensures a consistent framework for evaluation across different model sizes, with minimal code changing.\n>...experiments can be strengthened by incorporating some reliable metrics…\n\nWe have incorporated GPT-4 evaluation into our methodology, primarily due to its cost-effectiveness and the strong correlation with human preferences they have demonstrated. For example, (Rafailov et al., 2023) provides compelling evidence of GPT-4 judgments' alignment with human evaluations, noting that agreement between GPT-4 and human annotators is often on par with or exceeds that among humans themselves. Furthermore, the study by (Starling-7b) highlights the effectiveness of using GPT-4 feedback for Reinforcement Learning from Human Feedback (RLHF), where models trained on GPT-4 judgments showed significant improvements in performance on the live human evaluation platform, Chatbot Arena. As a complementary, we will add more interesting responses and compare different methods in the appendix, which may be interesting for curious reviewers.\n>...importance sampling trick…\n\n\nImportance sampling is a crucial technique in reinforcement learning, particularly when dealing with the challenge of updating a policy 𝜋 based on data collected from an older (hence different) policy 𝜋_old. This is common in RL as we usually collect a batch of samples and perform multiple gradient updates. The term \"importance sampling\" refers to this process of reweighting the contributions of each sample to reduce bias. Specifically, the importance sampling ratio π(a|s) / π_old(a|s) adjusts the influence of each experience based on how much more or less likely it is under the current policy compared to the old policy. This makes the estimator unbiased. In practice, we find the correction to be helpful when combined with clipping.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The newly proposed algorithm is well motivated by the discrepancy between reward model training loss and how they are commonly used in typical PPO algorithms. \nThe author provides a detailed derivation from Policy Gradient to P3O and discusses their algorithm compared to PPO and DPO, which I enjoyed reading. The use of pairwise policy gradient for preference-based learning looks creative and sound to me.\n- The results align well with the author's claims, showing the effectiveness of the proposed model.", "reasons_to_reject": "- The derivations may have some flaws that I want the author to clarify. I will increase the score if I can be convinced.\n- I feel that the experiments are relatively weak in terms of: \n   - They mainly use some relatively old and weak pretrained models, including gpt-j 6B and pythia 1B or 6B. I don't understand why not switching to LLaMa 7B if the 6B model training is feasible. Using more recent models can also strengthen the argument that this algorithm can be applied to stronger LMs.\n   - The chosen tasks are reasonable. However, I feel that just showing the KL-reward frontier is not enough, as the reward models may be biased by some superficial features, and the KL-reward frontier may just be comparing how strong the algorithms are learning such unwanted features within the KL budget. Similarly, GPT4-based evaluation also has some superficial features. I think the experiments can be strengthened by incorporating some reliable metrics, or including some tasks (e.g., math) that can be easily verifiable, or some human evaluation.", "questions_to_authors": "In Theorem 4, you introduce the importance sampling trick and claim it is an unbiased estimation of the pairwise policy gradient loss. I don't quite understand the intuition of this trick. Specifically, why do you choose $\\theta_{old}$ to compare against? Why is it called importance sampling? And why is it unbiased? Did you observe the difference between adding this term versus not adding it?", "extra_scores": {}}, {"review_id": "8VKIn3tszh", "reviewer": "Reviewer_qYdp", "summary": "This paper proposes a simple and effective policy gradient algorithm called Pairwise Proximal Policy Optimization (P3O) to deal with the inconsistency between reward learning and RL in PPO. This algorithm employs an additional sample to estimate the gradient unbiasedly, and update the policy based on the comparison difference. P3O is invariant with any reward functions that contain identical preference information, and does not require learning a value function. Experimental results on summarization and question answering show the effectiveness of the proposed method compared with PPO and DPO.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate the opportunity to enhance the clarity of our work.\n>...whether P3O with the ability to solve this inconsistency issue…\n\nWe will incorporate a training curve to highlight the instability observed in PPO. Furthermore, we will present a comparative analysis of the reward / KL as a function of training steps for different methods in the appendix for better understanding.\n>...the proposed method seems to use 2x generated samples to acquire a more stable estimation of gradients at each step compared with PPO…\n\n\nTo ensure a fair comparison between P3O and PPO, we have adopted a setting that potentially disadvantages P3O. Specifically, we use half the batch size for P3O compared to PPO. This adjustment guarantees that each gradient update across both methods is based on the same number of model-generated responses. However, it is important to note that under this setup, PPO benefits from processing a larger number of prompts per update, a factor generally considered advantageous for learning.\n\n\nRegarding hyperparameter tuning, we have conducted extensive experiments on various parameters for PPO, including Generalized Advantage Estimation lambda, discount factor, batch size, and learning rate. We have identified the optimal configuration for PPO and adapted it to be compatible with P3O without conducting additional hyperparameter searches.\n\n>...add human evaluation to make the experimental results more convincing.\n\nWe chose GPT-4 evaluation for two reasons: its efficiency in approximating human preference and its reproducibility. Studies such as [Rafailov et al., 2023] have shown that GPT-4 judgments correlate strongly with human evaluations, with inter-annotator agreement often similar or higher than human agreement. Additionally, [Starling 7b] demonstrated significant improvement in an aligned model using GPT-4 judgments and RLHF on the live human evaluation platform like Chatbot Arena.\n\n\nWe recognize that GPT-4 evaluation has advantages in terms of efficiency and reproducibility, as labeller biases are difficult to avoid and document. The usage of GPT4 will allow us to directly compare the results obtained from GPT-4 evaluation and human evaluation, providing a more robust assessment of our proposed method's performance.\n\n\nWe will also provide more responses in the appendix. This will offer additional insights for curious readers and further validate the effectiveness of our proposed approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method based on comparative feedback is novel and well-motivated. The derivation of P3O in Section 4 is clearly presented and easy to follow.\n2. Experimental results show the superior performance of P3O over PPO and DPO, which indicate the potential impact of the proposed method on different LLM alignment tasks.\n3. This paper is overall well-written. I enjoy reading this paper from the method to experiments.", "reasons_to_reject": "1. The authors mention the instability issue of PPO in the abstract and say that the inconsistency between reward learning and RL stages exacerbates the instability. I wonder whether P3O with the ability to solve this inconsistency issue can be more stable during training than PPO. It would be better to provide the curve of reward / KL vs. training steps in the experiment.\n\n2. The training efficiency should be also analyzed because the proposed method seems to use 2x generated samples to acquire a more stable estimation of gradients at each step compared with PPO. From my understanding, there exists a trade-off between efficiency and stability, but the current paper lacks the related analysis.\n\n3. Since GPT-4 evaluation may have some biases according to existing works [1], I recommend the authors to add human evaluation to make the experimental results more convincing.\n\n[1] Large Language Models are not Fair Evaluators.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4fTf0wqOEU", "reviewer": "Reviewer_8nNu", "summary": "The submission proposes a pairwise PPO method to improve PPO in online LM alignment process. The major argument is the gap between reward model learning from pairwise preference and RLHF with the pointwise policy gradient process.\n\n*Novelty/Forward-outlook*\nDPO is also using the proposed (general) BT-based loss. Other methods such as SliC use pairwise losses that mitigate the constant shift issue. There’re also preference-based RL methods in the literature. As acknowledged by the authors, the work can be seen as combining PPO and DPO. Thus, the proposed method is specific to improving the online PPO and the broader technical impact is reasonable but not substantial to the reviewer. \n\nThe authors may consider discussing a recent work [1], which unified several pairwise objectives under the learning to rank literature. It did not cover the Pairwise PPO method in this paper since it is not used in the learning to rank literature, so the difference here might be an interesting way to highlight the novelty of this submission.\n\n*Depth/Technical impact*\nDespite the comparisons between online vs offline alignment and popularity of methods such as DPO, given the current popularity of PPO, the contribution is reasonable to the reviewer, but not substantial.\n\nThe scale invariant problem is well studied in the learning to rank literature [2]. Thus the invariance analysis is interesting to the field but not novel, and it might be good to discuss such connections to related fields. \n\n*Experimental evaluation*\nExperiments are conducted on Reddit Tl;Dr and Anthropic HH, which are commonly used in the field. Evaluation metrics are proxy reward and GPT-4 auto side by side. Baselines are DPO and PPO. These settings are borderline and are typically minimally required for papers in the field. Extra steps include human eval, more datasets, more baselines, more ablation studies and model/size variants of the policy / reward model.\n\nThe reviewer is also not convinced about the experimental fairness. It is well-known that methods such as PPO have huge variance. The reviewer did not find convincing discussions about the model tuning and selection process except for brief tuning of learning rate in Section D.1, and one instance from each method is analyzed. It would be more convincing to see deeper analysis on the variance behaviors to reduce doubts about cherry picking. \n\n*Clarity*\nThe paper is clear to the reviewer.\n\n*Overall evaluation*\nOverall the paper is borderline to the reviewer. The basic idea makes a meaningful contribution to the field, however its impact may not substantially wide given existing work in the field and related fields. The theoretical analysis has value but not substantial. The experimental validation only meets the minimal requirements of papers in the field and does not make a strong case.  \n\n[1] LiPO: Listwise Preference Optimization through Learning-to-Rank. Arxiv 2024\n[2] Scale Calibration of Deep Ranking Models. KDD 2022", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful comments.\n>Novelty/Forward-outlook…\n\n\nIn the final draft, we will discuss preference-based RL and learning to rank literature. We note that P3O shares similarities with DPO and LiPO in that they all utilize comparative feedback. However, a key distinction lies in the fact that P3O is a policy gradient algorithm, benefiting from its strict improvement property. This sets P3O apart from ranking loss-based methods like DPO. \n\n\n>Depth/Technical impact…\n\n\nWe will provide a comprehensive discussion of the scale invariance issue, including its relevance to the learning-to-rank literature [2]. \n\n\nBy explicitly studying the scale invariance problem and its impact on training stability and sample efficiency, we bridge the gap between the alignment community and insights from other fields.\n\n\n>Experimental evaluation…\n\n\n\n\nOur choice of baselines, and evaluation metrics are standard and widely accepted in the field, following the precedent works [RAFT APA DPO].\nEvaluation metrics: We have included the GPT-4 side-by-side evaluation, which has been shown to correlate strongly with human judgments [Rafailov et al., 2023]. We acknowledge the importance of human evaluation and will consider it for future work if resources allow.\nBaselines: We have compared our method with two strong baselines, DPO and PPO, which are established algorithms in the field.\nRegarding ablations, we have provided an analysis of the effects of clipping and KL constraint in the appendix. \n\n>...not convinced about the experimental fairness…\n\nWe performed extensive hyperparameter tuning, and we apologize if these details were not clear:\n\n\n**Hyperparameter analysis**: We conducted a comprehensive analysis of hyperparameters for PPO, including discount factor, GAE lambda, and clipping threshold. We found that a discount factor 1 performed best, and adjustments to GAE lambda and clipping within wide ranges did not significantly impact the training curve.\n\n**lr and batch size**: We identified lr and batch size as the most critical factors affecting PPO's performance. We observed that a smaller batch size improved convergence speed, while larger batch sizes stabilized training but at a slower pace. We also considered GPU memory limitations and potential gradient accumulation bugs and set the gpu-mini-batch size accordingly.\nSeed selection: We repeated each experiment three times. \n\n\nWe believe that this additional information will address the reviewer's concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Below are from the overall review, please refer to the overall review for details. \n\n- The proposed method is specific to improving the online PPO and the broader technical impact is reasonable given the popularity of PPO.\n\n- The theoretical analysis looks correct and makes contribution to the important field.", "reasons_to_reject": "Below are from the overall review, please refer to the overall review for details. \n\n- Pairwise methods similar to this and invariance analysis are done in the literature in related fields such as RL and learning to rank.\n\n- Evaluations are not very convincing and only meets the minimal requirements of papers in this field. Risk of cherry-picking.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bNlyvxAA6P", "reviewer": "Reviewer_cNS6", "summary": "The paper presents a new RL-based LLM alignment method called P3O and demonstrates its effectiveness with a few small scale RMs and Policy Models on two datasets. The authors claim the traditional PPO algorithm may produce varying updates even given the identical human preference data and P3O algorithm is a good remedy for this issue. Both theoretical and empirical results are presented to show that P3O can more effectively align with human preferences than existing methods, suggesting that comparative feedback could be a superior method for training LLMs to adhere to human values.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback and recognize the importance of improving the overall presentation of our paper.\n>…overall paper writing/presentation is not very good…\n\n\nWe will provide a clearer explanation of the reward equivalence definition and its significance. We will include additional derivations and discussions of our algorithm in the appendix.\n\n\n>More intuitive explanations on why define the reward equivalence are needed.\n\n\nThe definition of reward equivalence is motivated by the observation that different prompts can yield significantly varying reward values. Suppose we have two prompts, one being relatively easy and the other more challenging. For the easy prompt, a standard response is likely to receive a high reward. In contrast, even a more sophisticated response to the hard prompt may result in a lower reward.\n\nMethods that rely solely on absolute rewards, such as PPO, tend to favor increasing the likelihood of the easy prompt response while decreasing the likelihood for the hard one. This can be detrimental to the model's performance.\n\nP3O circumvents this issue by utilizing comparative signals. By focusing on the relative preferences between responses, P3O is not sensitive to the absolute reward values associated with different prompts. \n\nWe will provide additional discussions to further clarify the motivation behind the reward equivalence definition and the advantages it brings to P3O.\n\n\n>…experiments can be further improved…\n\n\nThe bottleneck of tuning larger models (7b+) is on the infrastructure side, where there’s some conflict between ZeRO3 and generation. For the baseline, we only choose DPO as we think it's simple and well adopted. We’ll compare with SLiC or RSO if time permits.\n\n>…connection between your theory and experiments are not very strong…\n\nThe reward shift invariant issue pointed out by our theoretical analysis significantly impacts the empirical performance of PPO. Our observations reveal that adding a constant to the entire reward signal can dramatically alter the model's behavior after tuning with PPO. If the reward is generally negative, the model tends to generate excessively long responses, while a positive reward conversely leads to significantly shorter outputs. In practice, precisely balancing such constant shifts is crucial for PPO. In contrast, P3O consistently produces identical results regardless of constant shifts in the reward, thereby eliminating a significant source of instability inherent to PPO.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper studies an important problem and presents a principled solution based on some in-depth analysis of existing methods\n\n2. The experiments, although with limited scope, are still pretty solid", "reasons_to_reject": "I don't see any major issues but I do feel a few places can be improved/clarified. \n\nFirst, I feel the overall paper writing/presentation is not very good. Some parts of the paper have too dense information. For example, the page 4 section 4.1 is very dense and more discussions are needed. The bad presentation makes the some insights of this paper inaccessible. For example, I feel the reward equivalence in definition 4.3 is very interesting and should be further emphasize. The current way of presentation makes me feel it is defined in a way that can make lemma 4.4 hold (plus make the DPO/P3O enjoys the invariance property while the PPO validates this property). More intuitive explanations on why define the reward equivalence are needed. \n\nSecond, I think the experiments can be further improved in a few ways. First, you can consider more mid-sized (10B-70B) LLMs and show P3O method can generalize. Second, I think more comparison with other online version of DPO-style algorithms (e.g., SLiC [1], RSO [2]) are useful. You current experiment seems already include the online DPO (the end of page 7), why not check other online improved DPO-style algorithms? Finally, I feel the connection between your theory and experiments are not very strong. Your empirical results show P3O is better than PPO/DPO but I don't know if those improvements are due to your claimed theoretical reasons (e.g., resolving RM training, RL finetuning inconsistency, addressing invariant issue, etc) or maybe just come from less implementation noises. I believe some ablation studies to better bridge theoretical insights and empirical results are helpful. \n\n[1]. SLiC-HF: Sequence Likelihood Calibration with Human Feedback\n\n[2]. Statistical Rejection Sampling Improves Preference Optimization", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: For alignment purposes the most common data that is gathered is preference-based: given `n` completions (mostly `n==2`) annotators rank them according to their preference and a reward model is learnt from that. This is then used in a RL learning with a reward model as proxy, but the RL algorithm used does not integrate explicitly the pair-wise comparison.

This paper starts from this discrepancy, and proposes to use the comparative signal to train the policy model. This is well motivated, cleanly derivated and experimentally proven to improve the final model quality.
While one reviewer found some parts of the explanation lacking, most praised the clarity of the writing (the verb "to enjoy" being used twice)

The reviewers pointed out several additional experiments which would enrich the paper and clarify some parts, and I hope that the authors will fulfil their commitment of integrating those
Decision: Accept