Pairwise Proximal Policy Optimization: Language Model Alignment with Comparative RL

Authors: tianhao  wu
banghua  zhu
ruoyu  zhang
zhaojin  wen
kannan  ramchandran
jiantao  jiao
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 06:59:08

Abstract: LLMs may exhibit harmful behavior without aligning with human values. The dominant approach for steering LLMs towards beneficial behavior is Reinforcement Learning with Human Feedback (RLHF). This involves training a reward model with a human-labeled ranking dataset and fine-tuning the LLM with the reward signal using RL. Despite the fact that the reward is learned from comparing different responses, the RL stage doesn't involve direct comparisons. This inconsistency between reward learning and reinforcement learning stages exacerbates RL's instability. An example would be that the well adopted RL optimizer, Proximal Policy Optimization (PPO), could perform different gradient updates even for batches with identical human preference information. To address this, we propose a new framework, reinforcement learning with comparative feedback, and a simple policy gradient algorithm, Pairwise Proximal Policy Optimization (P3O), that learns to improve from direct comparison. Theoretically, P3O has the nice property of being invariant with any reward functions that contain identical preference information, while doesn't require learning a value function. Empirical evaluations demonstrate that P3O can align with human preferences better than existing methods. This suggest that comparative RL is strong candidate for aligning LLM with preference data.

Main paper: 
 introduction: 
 Large Language Models (LLMs) have made remarkable progress, profoundly influencing the AI community (Chowdhery et al., 2022; Brown et al., 2020; Touvron et al., 2023; Bubeck et al., 2023) . However, due to the reliance on massive corpora of internet data, which encompasses a high portion of low-quality data, LLMs are likely to express unintended behavior. These include fabricating facts, generating biased or toxic text, and even harmful content to humans (Perez et al., 2022; Ganguli et al., 2022) . Consequently, it is crucial to align LLMs with human values, e.g., helpful, honest, harmless (Bai et al., 2022a) .A leading method in AI Alignment for Large Language Models (LLMs), known as Reinforcement Learning from Human Feedback (RLHF), involves learning a reward function and fine-tuning the model with this reward feedback using reinforcement learning (RL) (Ziegler et al., 2019; Ouyang et al., 2022) . Specifically, a reward model is trained to rank candidate responses to align with the human-labeled ground-truth. As for RL, Proximal Policy Optimization (PPO) is widely adopted as the default optimizer (Schulman et al., 2017) . PPO alternate between generate new responses and adjust the likelihood toward responses with higher reward. Despite its acclaimed efficiency, we identify the inconsistency between these two stages:Inconsistency Between Reward Learning and RL. Although rewards are derived by ranking responses according to human judgments, the RL phase does not incorporate comparisons between generated samples. This leads to a scenario where the reward signal can be highly variable, often being lower for more challenging prompts and higher for simpler ones.Figure 1 : The figure on the left illustrates the prevalent method for fine-tuning LMs using RL, which relies on Absolute Feedback. In this paradigm, algorithms like PPO has to learn a V function, which capture not only the valuable relative preference information, but also less part, which is the scale of the reward for a given prompt. Contrastingly, the figure on the right presents RL paradigm that improve from direct comparison. Our algorithm generate a pair of responses per prompt, leveraging only the Comparative Feedback -derived from the difference in reward -for policy gradient updates. This method obviates the need for additional critic learning and intricate components like Generalized Advantage Estimation (Schulman et al., 2015b) .When employing conventional algorithms like Proximal Policy Optimization (PPO) to optimize against such a noisy reward, it becomes likely for PPO to reduce the likelihood of a quality response to a difficult prompt due to the absence of comparisons. This issue is further compounded by PPO's sensitivity to various factors including reward normalization, scaling, clipping, KL control, advantage normalization, and critic initialization (Zheng et al., 2023; Engstrom et al., 2020) , contributing to its fragility. As a consequence, we observe that PPO's reward experiences a decline during a warm-up period, and the training is sensitive to random seed. From a formal perspective, we argue this inconsistency by noting that the reward training objective, the Bradley-Terry Loss, is invariant to a constant shift (Yan et al., 2022) , whereas PPO is not. This implies that even if two reward models contain identical human preference information, their optimization via PPO could lead to disparate results. (subsection 4.3) In this paper, we provide new insights to address the inconsistency:• We define an equivalent relationship for reward functions trained from human preferences. We identify that the widely adopted reward training loss, Bradley-Terry Loss, is invariant under this equivalent relationship, while PPO is not. As a result, PPO may be less efficient at learning the reward.• We introduce Pairwise Proximal Policy Optimization (P3O), under the framework of Reinforcement Learning with Comparative Feedback (Figure 1 ). P3O learns by comparing pairs of responses, avoiding learning critic functions, advantage estimation and various normalization techniques (Zheng et al., 2023) . Empirical evaluations show that P3O consistently outperforms PPO and Direct Preference Optimization (DPO) in terms of GPT-4 Evaluation. 
related work: 
 Significant efforts have been made towards aligning LLMs with human values. These alignment strategies broadly fall into two categories: offline training and online training.Offline training typically involve a static dataset, and doesn't require additional evaluations or generations. For instance, Thoppilan et al. (2022) ; Gunasekar et al. (2023) use instruction fine-tuning to update the model on a high quality dataset tailored to a specific downstream task of interest. Snell et al. (2022) proposed to employ offline Q Learning to learn an add-on term for decoding. While Rafailov et al. (2023) introduced DPO, an offline approach that can directly align LM with human preference data, drawing from the closed-form solution of the Contextual Bandit with KL control problem. There are also methods like PRO (Song et al., 2023) and RRHF (Yuan et al., 2023) that fine-tune the model based on ranking of the rewards. Liu et al. (2024) unified several pairwise objectives under the LiPO framework and establish connection with Learning-to-Rank Literature.Our work is categorized under online training, which consist of a loop of generating new responses from the updated policy, evaluating them with the reward model and updating the policy. The current dominant approach RLHF relies on online RL methods such as PPO (Schulman et al., 2017) , A2C (Mnih et al., 2016) or their variants (Ramamurthy et al., 2022; Zhu et al., 2023c) . There are also few methods that deviate from this standard. For instance, ReST (Gulcehre et al., 2023) use offline RL in the policy improvement phase. RAFT (Dong et al., 2023a) iteratively fine-tune the policy on the responses generated by the Best-of-N policy. Another paradigm parallel to RLHF is Reinforcement Learning with AI Feedback (RLAIF) (Zhu et al., 2023a; b; Bai et al., 2022b; Lee et al., 2023; Yuan et al., 2024) , which aim for using AI to improve AI. For example, Wu et al. (2024) proposed Meta-Rewarding LMs, which significantly improve the instruction-following and judging ability without relying on human data.Outside of the context of language, Contextual dueling bandit (Dudík et al., 2015; Yue et al., 2012) use preferences or rankings of actions to adjust the policy, rather than rewards. Similarly, PbRL (Xu et al., 2020; Jain et al., 2013; Busa-Fekete et al., 2014; Christiano et al., 2017; Sadigh et al., 2017; Kupcsik et al., 2018) learn from binary preferences generated by some unknown scoring function.The necessity of reinforcement learning (RL) for aligning large language models (LLMs) has been a topic of much debate. There are alternative approaches, such as Direct Policy Optimization (DPO), which is a simpler method that utilizes a pre-collected offline dataset of preferences. On the other hand, RL involves a more intricate process of interacting with the language model in real-time. This process includes generating new responses, evaluating them using a reward model, and then updating the language model based on the rewards.  
preliminaries: 
 We briefly reviewing the RLHF pipeline in (Ziegler et al., 2019) .• SFT Phase (Supervised Fine-Tuning): This stage start with a pre-trained LM, and then fine-tuned with supervised learning (typically maximum likelihood loss) on a high quality dataset for the downstream task of interest. Outcome of this stage is denoted as π SFT . • Reward Learning Phase. In the second phase the SFT model is prompted with prompts x to produce pairs of answers y 1 , y 2 ∼ π SFT (y|x). The responses pairs are then presented to human labelers who express preferences for one answer, denoted as y w ≻ y l |x, where y w is the one favored by the labeler and y l is the one less favored. Under these preferences is the inaccessible latent reward model r * (y, x). According to the Bradley-Terry (Bradley & Terry, 1952) model, the human preference distribution p * can be expressed as:p * (y 1 ≻ y 2 |x) = 1 1 + exp (r * (y 2 |x) -r * (y 1 |x))Assuming the access to a dataset {(x i , y i w , y i l )} N i=1 sampled from p * . We parameterize the reward as r ϕ and estimate it via maximum log-likelihood:N ∑ i=1 1 N log σ r ϕ (y i w |x i ) -r ϕ (y i l |x i ) (1)where σ is the sigmoid function. r ϕ is initialized with π SFT augmented by additional linear layers on top. Constraints like E [r(y|x)] = 0 might be incorporated to lower the variance. • RL Fine-Tuning Phase. Prior work formulate the optimization problem as maximizing:EQUATIONThe βD KL (π θ (•|x)∥π SFT (•|x)) term is used to regulate the deviation from the SFT model, it is important to prevent the model from completely forget the world knowledge acquired in the pre-training stage. The standard approach is to directly employ PPO (Schulman et al., 2017; Ouyang et al., 2022) to optimize the modified reward r ϕ (y|x)β log π θ (y|x)log π SFT (y|x) . 
proximal pairwise policy optimization (p3o): 
 To derive P3O, we start from Vanilla Policy Gradient (VPG, Pseudo 2) (Sutton et al., 1999; Schulman et al., 2017; Wu et al., 2022) . For clarity, we'll focus on the bandit setting, though it can be extended to contextual bandits as in Theorem 4.1.In the bandit setting, assume we are updating a parameterized policy π θ with actions denoted as y. The VPG aims for estimating the following formula with samples:EQUATIONwhere b is a baseline used for variance reduction. A common choice for the baseline is the mean reward, which gives:EQUATIONEquation 4 highlights the reliance on the relative difference between rewards. Symmetrizing for y 1 , y 2 yield:∇L VPG = E y 1 ,y 2 ∼π θ (r(y 1 ) -r(y 2 ))∇ log π θ (y 1 ) π θ (y 2 ) /2Its immediate generalization to contextual bandit is the following:Theorem 4.1 (Pairwise Policy Gradient) For any prompt x, the policy gradient can be expressed as∇L VPG = E x∼D ∇L PPG (x), where ∇L PPG (x) can be expressed as:E y 1 ,y 2 ∼π θ (r(y 1 |x) -r(y 2 |x)) ∇ log π θ (y 1 |x) π θ (y 2 |x) /2To estimate the policy gradient with finite samples, considering that the replay buffer is collected using the previous policy π θ old , we further utilize importance sampling to correct the bias. The following theorem provides an unbiased estimation of the pairwise policy gradient:Theorem 4.2 (Estimate PPG with Importance Sampling) For replay-bufferD k = {τ i = (x i , y i 1 , y i 2 , r i 1 , r i 2 )} n i=1 collected using π θ old .The following is an unbiased estimation of pairwise policy gradient:1 2n n ∑ i=1 (r i 1 -r i 2 ) π θ (y i 1 |x i ) π θ old (y i 2 |x i ) π θ (y i 2 |x i ) π θ old (y i 2 |x i ) • ∇ log π θ (y i 1 |x i ) π θ (y i 2 |x i ) /2We further employ clipping to stabilize the policy updates, for more details refer to subsection B.2.1: Initialization: Initialize policy from the SFT model with parameters θ 0 2:for k = 0, 1, 2 • • • do 3:Sampling n prompts {x i } n i=1 from a prompt dataset. Collect pairwise responses for each prompt by sampling from the latest policy π θ k :y i 1 , y i 2 ∼ π θ k (•|x i ) 4:Score all the responses with a reward model r model , and aggregate the reward with KL divergence:r(y|x) = r model (y|x) -βD KL π θ k (•|x)∥π θ 0 (•|x) 5:Estimate policy gradient on the scored replay-bufferD k = {τ i = (x i , y i 1 , y i 2 , r i 1 , r i 2 )} n i=1 via (subsection B.2): ĝk = ∇ θ L P3O joi (D k ) 6:Update the θ k via gradient descent and yield θ k+1 . 7: end forComparison with PPO: Although PPO and P3O both fall into the online RL framework, they differ in the way they perform policy update: PPO update based on an estimated advantage, while P3O update based on direct comparison of two responses. Consider a simplified version of PPO applied to contextual bandit:L PPO no clip = - E y∼π θ old (•|x) (r(y|x) -V ϕ (x)) π θ (y|x) π θ old (y|x)Where V ϕ (x) is a proxy to the ground truth value function V π θ old = E y∼π θ old r(y|x), usually learnt via an additional regression loss. In contrast, P3O doesn't require learning the value function, this significantly reduce engineering efforts. P3O employ an additional sample y 2 to estimate the gradient unbiasedly, and update the policy based on the comparison r 1r 2 .The gradient of DPO's objective function ∇L DPO (x, y w , y l ) takes the following form:βσ β log π θ (y l |x) π SFT (y l |x) -β log π θ (y w |x) π SFT (y w |x) • ∇ log π θ (y w |x) π θ (y l |x) /2The direction of the gradient resembles that of our formulation in Theorem 4.1. However, the weight coefficients are different. The core difference of DPO and P3O is that P3O is an online RL algorithm while DPO is not. P3O learns to trials and errors by alternating between generate and update from human feedback while DPO is applied to a fixed dataset. We empirically observe that DPO falls short on KL-control (Figure 4 and Figure 2 ) compared to P3O, we hypothesis that this is because DPO aligns the policy towards the goal policy while doesn't directly consider the reward of the intermediate policies. Unlike P3O, which applies policy gradient based on the idea of strict policy improvement for every gradient update (Schulman et al., 2015a) , DPO aligns the policy via an alternate "distance", where the intermediary steps are not guaranteed to maximize the KL-Reward trade-off. We note that P3O combines the benefits of PPO and DPO, offering guaranteed policy improvement akin to policy gradient.We formally define the concept of reward equivalence (Definition 4.3), similar to the scale invariance defined in (Yan et al., 2022) . We show that BTL is invariant under this equivalent relationship in lemma 4.4. We then discuss why it leads to a desirable property named invariance (Definition 4.5) that we want RL algorithms to satisfy. In the end, we present our main theorem (Theorem 4.6) which shows that PPO does not satisfy this property, contributing to its instability.Definition 4.3 (Reward Equivalence) Two reward functions r(y|x) and r ′ (y|x) are termed equivalent, denoted as r ∼ r ′ , if and only if there exist a function δ(x) depend solely on the prompt x, such that for every prompt and response pair (x, y),r(y|x) -r ′ (y|x) = δ(x)The equivalent class associated with reward r is represented as [r] .Note that comparative losses such as Bradley-Terry loss and Plackett-Luce loss, is unaffected by a shift in the prompt's reward as in definition 4.3. This observation leads to the following Lemma:Lemma 4.4 (Invariance of BTL) For two reward functions that satisfy r ∼ r ′ , they both yield identical loss for any response pairs (or K responses) under the Bradley-Terry Loss (or Plackett-Luce Loss).Lemma 4.4 underscores that the only information we can learn from the preference data is the reward difference of two responses to the same prompt. This implies that direct comparison of responses stemming from different prompts should be avoided. This is because we can craft an arbitrary function denoted as δ and replace r with the identical r + δ, while flipping the sign of r(y|x) -r′ (y ′ |x ′ ). As a result, an ideal algorithm should focus only on the relevant information within the reward function, filtering out the noise represented by δ. This leads the following definition:Definition 4.5 (Invariance) An algorithm is said to be invariant with respect to the equivalent relation "∼", if for any two equivalent reward functions r ∼ r ′ and a fixed set of prompt and response pairs, the algorithm perform identical updates to the policy.To illustrate definition 4.5, assume that we have two equivalent reward functions r and r′ = r + δ. Notably, even when initialized with the same random seed, PPO can result in distinct updates for an identical batch. This behavior can be attributed to PPO's reliance on learning a V function to estimate advantage. In the simplest scenario, where the advantage is estimated via one-step TD (Adv(y|x) = r(y|x) -V(x), corresponding to λ GAE = 0) and y is a single token, we should expect the advantage function to stay unchanged. However, following the derivationAdv r (y|x) = Adv r′ (y|x) ⇐⇒ r(y|x) -V r (x) = r′ (y|x) -V r′ (x) ⇐⇒ V r′ (x) -V r (x) = δ(x)We can see that even though r and r′ are equivalent, they yield different updates for V function. This give rise to our main theorem (full proof in subsection C.2):Theorem 4.6 (Non-invariance of PPO) P3O is invariant with respect to "∼". In contrast, PPO is not, given the same initialization of V. 
experiments: 
 We conduct experiments on two widely-adopted RLHF tasks, summarization and questionanswering, and we find that P3O achieves better performance in terms of both KL-Reward trade-off and quality of generation, against several strong baselines. Refer to Appendix D for detailed hyper parameter selection process.Tasks. We explore two different open-ended text generation tasks, i.e. summarization and question-answering. For both tasks, algorithms are given a reward model pre-trained from l }, and the goal is to obtain a policy π(y|x) that can generate high-quality response y given prompt x. In summarization, we use the TL;DR (too long; didn't read) dataset (V ölske et al., 2017) , where x is a forum post from Reddit, and y is a corresponding summary. We use a 6B SFT model CarperAI/openai summarize tldr sft as the initial policy and EleutherAI/gpt-j-6b as the reward model. In question-answering, x is a human query, which may come from diverse topics, and the policy should learn to produce an engaging and helpful response y. Following prior work, we use the Anthropic Helpful and Harmless (HH) dataset (Bai et al., 2022a) . We fine-tune two policies of sizes {1B,6B}, Dahoas/pythia-1B-static-sft and Dahoas/pythia-6B-static-sft. Both models have gone through supervised fine-tuning with labeled prompt-response pairs, similar to the protocol in Ouyang et al. (2022) and Ramamurthy et al. (2022) . For the reward model, we use the 6B model Dahoas/gptj-rm-static trained from the same dataset based on EleutherAI/gpt-j-6b as a proxy of human preference.Methods. We compare two versions of P3O, P3O-V1 and P3O-V2, which represent clipping separately and jointly respectively, with several effective and representative approaches for LLM alignment. We start with the SFT policy trained by token-wise supervised finetuning. It hasn't gone through further alignment; Every other method uses the SFT model as initialization. For RL algorithms1 , we consider the dominant approach PPO (Schulman et al., 2017; Ouyang et al., 2022) with reward specified in Equation 2. We follow the implementation of trlx (Castricato et al., 2023) . Besides, we also consider the newly proposed DPO (Rafailov et al., 2023) , a method that directly optimizes the policy towards the closed-form solution of the KL-constrained reward maximization. Although DPO is proposed as an offline 2023) studied the scaling law of reward over-optimization in a synthetic setup, where labels are supplied by a "gold standard" reward model. They empirically find out the golden reward can be approximated by a simple function form involving the square-root KL-divergence from the reference policy. Therefore, it is important to balance the trade-off between the KLdivergence and asymptotic reward, and we measure the effectiveness of each algorithm by its frontier of achieved reward and KL-divergence from the reference policy (KL-Reward Frontier). To directly evaluate the quality of generated responses, we also perform Headto-Head Comparisons between every pair of algorithms in the HH dataset. We use two metrics for evaluation: (1) Reward, the optimized target during online RL, (2) GPT-4, as a faithful proxy for human evaluation of response helpfulness. For the latter metric, we shall point out that previous studies show that LLMs can be better automated evaluators than existing metrics (Chen et al., 2023) , and GPT-4 judgments correlate strongly with humans, with human agreement with GPT-4 typically similar or higher than inter-human annotator agreement (Rafailov et al., 2023) . Additional details can be found in Appendix D. We conduct experiments on both TL;DR and HH datasets to evaluate the efficacy of the alignment algorithms in optimizing reward while restricting policy deviation from the reference. Figure 4 and Figure 2 demonstrate the KL-Reward frontier for TL;DR and HH respectively. Each point represents the average evaluation over test prompts at every 500-step interval. The x-axis represents the average sequence-level KL-divergence D KL (π θ ∥π SFT ), whereas the y-axis stands for the average reward given by the proxy reward model. For summarization task, we find that P3O-V1 can reach a slightly higher reward than P3O-V2, while with a worse KL-Reward trade-off. Consequently, only P3O-V2 is included in Figure 4 for comparison. We find that P3O-V2 is able to produce almost the same highest reward whilst maintaining superior KL efficiency. DPO, despite its faster convergence, exhibits a 25% higher KL-divergence than P3O-V2 under the same reward.For the question-answering task, P3O-V1 and P3O-V2 have strictly dominant frontiers than PPO and DPO respectively in both model sizes, shown by Figure 2 . Empirical findings establish P3O's superior trade-off between KL and Reward over other baselines, delivering a substantial higher reward in the range of 0.1-0.3. We sample completions from different policies2 on the test set of the HH dataset at default temperature 1.0, and we compute the average pairwise win rate using (1) reward and (2) GPT-4 as evaluators. Previous studies (Chen et al., 2023; Rafailov et al., 2023) have shown that GPT-4 is a faithful proxy for human preference and is widely adopted for comparisons. The prompt used for evaluation is presented in subsection D.2. Figure 3 presents the comprehensive pairwise comparison results, both via proxy reward and GPT-4. The average KL-divergence and reward ranking of these models is DPO > P3O > PPO > SFT. Although DPO marginally surpasses P3O in reward, it has a considerably higher KL-divergence (Table 1 ), which may be detrimental to the quality of generation. As a result, DPO has a reward win rate 49.5% against P3O, but only 45.4% as evaluated by GPT-4. Compared with other methods, P3O exhibits a GPT-4 win rate of 57.0% against PPO and 69.3% against SFT. This result is consistent with our findings from the KL-Reward frontier section, affirming that P3O could better align with human preference than previous baselines. 
conclusion & future works: 
 This work presents new insights into aligning large language models with human preferences via reinforcement learning. We introduced the Reinforcement Learning with Comparative Feedback framework, which unifies the core principles of reward modeling and RL fine-tuning. Our empirical evidence compellingly supports the notion that comparative RL, by leveraging direct comparisons, presents a more effective approach for alignment than traditional non-comparative RL methods. Within this framework, we developed P3O, which utilizes pairwise comparisons to perform policy updates. Our empirical assessments have shown that P3O not only surpasses previous methodologies in achieving a better balance on the KL-Reward frontier but also demonstrates superior performance in GPT-4 win-rate comparisons. P3O encapsulates the benefits of policy gradient techniques while simplifying both the algorithmic construction and function approximation.Looking ahead, several intriguing questions arise for future exploration. Firstly, we aim to understand the impacts of reward over-optimization on trajectory-based RL algorithms and token-based RL algorithms. Secondly, we are interested in whether we can generalize the policy gradient algorithm to accommodate more than two ranked responses, potentially enabling a better trade-off between human effort and AI alignment. Finally, we wish to explore the benefits of applying P3O in contexts beyond training language models with human feedback. We eagerly anticipate investigating these questions in our future work. 
 
 Appendix: Algorithm 2 Vanilla Policy Gradient 1: Initialization: Initialize policy parameters θ 0 and value function parameters ϕ 0 2:for k = 0, 1, 2 • • • do 3:Collect trajectories D k = {τ i } by running policy π θ k starting from a batch of prompts and generate single trajectory from each prompt.Compute token-wise rewards contain both token-wise KL and preference reward as in Equation 2. And then rewards-to-go Rt .Estimate advantage estimates Adv t via GAE or other methods.Estimate policy gradient via:ĝk = 1 |D k | ∑ τ∈D k T ∑ k=0 Adv t ∇ θ log π θ (a t |s t ) 7:Apply gradient updates to θ k using gradient descent.Fit value function by regression on mean-squared error via gradient descent:ϕ k+1 = arg min ϕ 1 |D k |T ∑ τ∈D k T ∑ t=0 (V ϕ (s t ) -Rt ) 2 9: end forWe present the pseudocode for both the Vanilla Policy Gradient (VPG) and our proposed algorithm P3O. While both algorithms follow the similar procedure of collecting trajectories and leveraging these trajectories to estimate the gradient, there are key differences: Our method collect pairwise trajectories and compute trajectory-wise rewards. This approach eliminates the need for estimating the value function V and bypasses the requirement of estimating the advantage function using Generalized Advantage Estimation (GAE). Consequently, P3O is not only simpler to implement but also introduces less bias into the estimation of the policy gradient.Clipping Separately (Version 1): For {i, j} = {1, 2},L P3O i (x) = E y 1 ,y 2 ∼π θ old sg r(y i |x) -r(y j |x) π θ (y j |x) π θ old (y j |x) π θ (y i |x) π θ old (y i |x) L P3O i,clip (x) = E y 1 ,y 2 ∼π θ old sg r(y i |x) -r(y j |x) π θ (y j |x) π θ old (y j |x) clip( π θ (y i |x) π θ old (y i |x) , 1 -ϵ, 1 + ϵ) L P3O sep = E x∼D min(L P3O 1 (x), L P3O 1,clip (x)) + min(L P3O 2 (x), L P3O 2,clip (x)) /2Clipping Jointly (Version 2):L P3O (x) = -ϵ, log π θ old (y 1 |x) π θ old (y 2 |x) + ϵ L P3O joi = E x∼D min(L P3O (x), L P3O clip (x))DPO start with a preference dataset D and minimize the loss:L DPO = - E (x,y w ,y l )∼D log σ β log π θ (y w |x) π SFT (y w |x) -β log π θ (y l |x) π SFT (y l |x)However, this is offline since the algorithm only make use of a fixed dataset. Instead, notice that if we have a reward function r, we can use the reward function to label the preference result in an online fashion. Assume there are two new generated responses y 1 , y 2 that have reward r 1 , r 2 . Then we simply label the preference according to Bradley & Terry (1952) ,y 1 ≻ y 2 w.p. σ(r 1 -r 2 ) y 2 ≻ y 1 w.p. σ(r 2 -r 1 )We would like to use the notation y w and y l to represent the preferred and less preferred response chosen by the reward. We collect all the newly generated responses into a replay buffer D replay , therefore we can optimize the same DPO loss here:L DPO = - E (x,y w ,y l )∼D replay log σ β log π θ (y w |x) π SFT (y w |x) -β log π θ (y l |x) π SFT (y l |x)We can further reduce the variance of the loss by eliminating the randomness in labelling the preference by incorporating the known labeling probability explicitly in the formula,L DPO = - E (x,y 1 ,y 2 )∼D replay ϵ∼Ber(σ(r 1 -r 2 )) log σ ϵβ log π θ (y 1 |x) π SFT (y 1 |x) -ϵβ log π θ (y 2 |x) π SFT (y 2 |x)Here, Ber(σ(r 1r 2 )) is the two point Bernoulli distribution on {-1, 1}.C.1 Proof of Theorem 4.1In the contextual bandit setting, VPG aims for estimating the gradient ∇L VPG = E x∼D ∇L VPG (x), where ∇L VPG (x) can be expressed as:∇L VPG (x) = E y∼π θ (y|x) r(y|x)∇ log π θ (y|x)The expectation can be replaced with a summation, leading to:EQUATIONIn Equation 6we subtract the latter term which equals to 0 because ∑ y 1 ∇π θ (y1 |x) = ∇ ∑ y 1 π θ (y 1 |x) = ∇ • 1 = 0.We further multiply the first term by 1 = ∑ y 2 π θ (y 2 |x) in Equation 7. Finally, we rephrase the previous equation using importance sampling and yield Equation 9.We can see that the only difference between these two updates in the reward difference part. However, due to the fact thatr ′ (y 1 |x) -r ′ (y 2 |x) = (r(y 1 |x) + δ(x) -r(y 2 |x) -δ(x)) = (r(y 1 |x) -r(y 2 |x)) We conclude that L P3O r = L P3O r ′and L P3O r,clip = L P3O r ′ ,clip . Consequently, the two updates ∇L P3O r,joi , ∇L P3O r ′ ,joi are the same. Invariance of DPO: Assume the same setting as in the previous paragraph:The gradient of DPO given reward r can be written as:∇L DPO r = - E ϵ∼Ber(σ(r 1 -r 2 )) log σ ϵβ log π θ (y 1 |x) π SFT (y 1 |x) -ϵβ log π θ (y 2 |x) π SFT (y 2 |x)Similarly, the gradient of DPO given reward r ′ can be expressed as:∇L DPO r ′ = - E ϵ∼Ber(σ(r ′ 1 -r ′ 2 )) log σ ϵβ log π θ (y 1 |x) π SFT (y 1 |x) -ϵβ log π θ (y 2 |x) π SFT (y 2 |x)The only difference between these two equations is the sampling distribution of the Bernoulli distribution. Easy to verify that they are the same since σ(r 1r 2 ) = σ(r ′ 1r ′ 2 ). PPO is not Invariant: The loss of PPO is the combination of policy-loss and V-loss, with the trade-off of these two terms controlled by hyper-parameter η:L PPO = L policy + ηL VSuppose the policy network and the V network have separate parameters, then taking the gradient of L PPO is simply taking gradient of L policy . We aim to prove that the gradient of L policy is not identical for two equivalent rewards. We first recap the formula of L policy : Where the Adv is estimated via GAE:L policy = - E (s t ,Adv(a t |s t ) = δ t + (λγ)δ t+1 + • • • + (λγ) T-t+1 δ T-1 δ t = r(s t , a t ) + γV(s t+1 ) -V(s t )For simplicity, we consider the one-sample case, where we are taking gradient with respect to the sample (s t , a t ). According to the formula of Adv, and combine with the fact that the reward is only appended to the last token T. We have the following relation, For the hyper-parameter tuning, we first run PPO to search the learning rate among {0.5, 1, 2, 4, 8} × 10 -6 that yields the best KL-Reward frontier (Figure 5 ). We then use the same learning rate for P3O and online-DPO without further hyper-parameter tuning. To ensure fair comparison, we double the batch size of PPO such that every algorithms can see the same number of responses, although P3O and online-DPO only see half the prompts.Besides learning rate, we also tuned discount factor, GAE lambda, and clipping threshold. We found that a discount factor 1 performed best, and adjustments to GAE lambda and clipping within wide ranges did not significantly impact the training curve.We identified learning rate and batch size as the most critical factors affecting PPO's performance. We observed that a smaller batch size improved convergence speed, while larger batch sizes stabilized training but at a slower pace. We also considered GPU memory limitations and set the gpu-mini-batch size accordingly.We repeated each experiment three times.For the following query to a chatbot , which response is more helpful ? Comparison : <one -sentence comparison and explanation > More helpful : <"A" or "B"> To eliminate any potential bias from the order of the responses, the order of Response A and Response B is randomized for each prompt. We note that although we ask GPT-4 to directly respond A or B to indicate the preference, in approximately 3% of evaluations, GPT-4 found that neither response adequately addressed the user's query.Human: help with understanding bitcoin Assistant:Yes, that's a fun question! Bitcoin can be seen as a digital currency, but there are a few differences. Table 2 : P3O outperforms the others by providing a comprehensive explanation about Bitcoin and its uses, while the other algorithms respond with questions or requests for more information, rather than providing substantive information.Human: How can I make my own soap at home? Assistant:There are a few different methods for making your own soap at home. Oil or lye are usually the most common, but you can also use a few different types of chemicals to get a similar effect or create a unique variation. Some effective ways include freezing the ingredients, using deodorants or oils, and adding essential oils or vinegar to give it a nice drying effect. DPO I wouldn't recommend it, it's a little fancier and not something you can do at home.Making your own soap is a nice way to get involved in your own community. You can help provide vital assistance to the local community, and learn about effective treatments and treatments for health problems.Well historically, in Europe soap has typically not been watering shells and Ocean Aerosols, a combination generally indicated as "EST date" a secret date from the manufacturer. Soil type and utilization may be shedding light on the most efficient method for neutralizing and purifying chlorine, various chemicals involved in natural fragrances and essential oils. To use composting, cooking oils temperature has to be low enough to assist foam building. Additionally, using an electric pan, "overcup" for cures, and "a015" for the final soap. How do you plan to save your energy and waste water?Comparison: Response A[P3O] provides a detailed explanation of different methods for making soap at home, while Response B[DPO] discourages the idea without providing any useful information.Comparison: Response A[PPO] is vague and doesn't provide any specific steps or ingredients for making soap, while Response B[P3O] gives a general idea of the process and ingredients involved in soap making. Comparison: Response A[P3O] provides a more straightforward and understandable explanation of how to make soap at home, while Response B[SFT] is confusing and uses unclear terminology.Table 4 : P3O is assessed as the most helpful by GPT-4. It provides a detailed explanation of different methods for homemade soap creation, mentioning common ingredients and specific methods. Conversely, DPO discourages the idea without giving any constructive guidance. PPO fails to offer any specific steps or ingredients for soap creation. Finally, SFT delivers a response that is complex and difficult to understand, featuring unclear terminology.Among methods directly involving RL, we note that PPO and a modified version A2C(Mnih et al.,2016;Lee et al., 2023) are the only two current online RL methods for LLM alignment. However, there is no strong evidence showing the supremacy of A2C over PPO, so we choose PPO as our baseline.We select checkpoints with the highest reward for generation. 


Decision: For alignment purposes the most common data that is gathered is preference-based: given `n` completions (mostly `n==2`) annotators rank them according to their preference and a reward model is learnt from that. This is then used in a RL learning with a reward model as proxy, but the RL algorithm used does not integrate explicitly the pair-wise comparison.

This paper starts from this discrepancy, and proposes to use the comparative signal to train the policy model. This is well motivated, cleanly derivated and experimentally proven to improve the final model quality.
While one reviewer found some parts of the explanation lacking, most praised the clarity of the writing (the verb "to enjoy" being used twice)

The reviewers pointed out several additional experiments which would enrich the paper and clarify some parts, and I hope that the authors will fulfil their commitment of integrating those

Reviews: review: paper_summary: paper_summary: This paper proposes a new algorithm for RLHF, called Pairwise Proximal Policy Optimization (P3O). In an online training fashion similar to PPO, this algorithm uses a comparative signal between two sampled responses to train the policy model, which aligns better with how the reward model is trained. It also demonstrates some other properties, such as removing the need for a value model and being invariant to "information-equivalent" reward functions. The author provides detailed derivations of the P3O algorithm based off vanilla Policy Gradient, and also conducts experiments on summarization and question answering tasks to show the superiority of P3O over PPO and DPO, in terms of KL-Reward frontier and GPT4-judged win-rate.
strength_weakness: strength_weakness: - The derivations may have some flaws that I want the author to clarify. I will increase the score if I can be convinced.
- I feel that the experiments are relatively weak in terms of: 
   - They mainly use some relatively old and weak pretrained models, including gpt-j 6B and pythia 1B or 6B. I don't understand why not switching to LLaMa 7B if the 6B model training is feasible. Using more recent models can also strengthen the argument that this algorithm can be applied to stronger LMs.
   - The chosen tasks are reasonable. However, I feel that just showing the KL-reward frontier is not enough, as the reward models may be biased by some superficial features, and the KL-reward frontier may just be comparing how strong the algorithms are learning such unwanted features within the KL budget. Similarly, GPT4-based evaluation also has some superficial features. I think the experiments can be strengthened by incorporating some reliable metrics, or including some tasks (e.g., math) that can be easily verifiable, or some human evaluation.
questions: questions: In Theorem 4, you introduce the importance sampling trick and claim it is an unbiased estimation of the pairwise policy gradient loss. I don't quite understand the intuition of this trick. Specifically, why do you choose $\theta_{old}$ to compare against? Why is it called importance sampling? And why is it unbiased? Did you observe the difference between adding this term versus not adding it?
paper_summary: paper_summary: This paper proposes a simple and effective policy gradient algorithm called Pairwise Proximal Policy Optimization (P3O) to deal with the inconsistency between reward learning and RL in PPO. This algorithm employs an additional sample to estimate the gradient unbiasedly, and update the policy based on the comparison difference. P3O is invariant with any reward functions that contain identical preference information, and does not require learning a value function. Experimental results on summarization and question answering show the effectiveness of the proposed method compared with PPO and DPO.
strength_weakness: strength_weakness: 1. The authors mention the instability issue of PPO in the abstract and say that the inconsistency between reward learning and RL stages exacerbates the instability. I wonder whether P3O with the ability to solve this inconsistency issue can be more stable during training than PPO. It would be better to provide the curve of reward / KL vs. training steps in the experiment.

2. The training efficiency should be also analyzed because the proposed method seems to use 2x generated samples to acquire a more stable estimation of gradients at each step compared with PPO. From my understanding, there exists a trade-off between efficiency and stability, but the current paper lacks the related analysis.

3. Since GPT-4 evaluation may have some biases according to existing works [1], I recommend the authors to add human evaluation to make the experimental results more convincing.

[1] Large Language Models are not Fair Evaluators.
paper_summary: paper_summary: The submission proposes a pairwise PPO method to improve PPO in online LM alignment process. The major argument is the gap between reward model learning from pairwise preference and RLHF with the pointwise policy gradient process.

*Novelty/Forward-outlook*
DPO is also using the proposed (general) BT-based loss. Other methods such as SliC use pairwise losses that mitigate the constant shift issue. There’re also preference-based RL methods in the literature. As acknowledged by the authors, the work can be seen as combining PPO and DPO. Thus, the proposed method is specific to improving the online PPO and the broader technical impact is reasonable but not substantial to the reviewer. 

The authors may consider discussing a recent work [1], which unified several pairwise objectives under the learning to rank literature. It did not cover the Pairwise PPO method in this paper since it is not used in the learning to rank literature, so the difference here might be an interesting way to highlight the novelty of this submission.

*Depth/Technical impact*
Despite the comparisons between online vs offline alignment and popularity of methods such as DPO, given the current popularity of PPO, the contribution is reasonable to the reviewer, but not substantial.

The scale invariant problem is well studied in the learning to rank literature [2]. Thus the invariance analysis is interesting to the field but not novel, and it might be good to discuss such connections to related fields. 

*Experimental evaluation*
Experiments are conducted on Reddit Tl;Dr and Anthropic HH, which are commonly used in the field. Evaluation metrics are proxy reward and GPT-4 auto side by side. Baselines are DPO and PPO. These settings are borderline and are typically minimally required for papers in the field. Extra steps include human eval, more datasets, more baselines, more ablation studies and model/size variants of the policy / reward model.

The reviewer is also not convinced about the experimental fairness. It is well-known that methods such as PPO have huge variance. The reviewer did not find convincing discussions about the model tuning and selection process except for brief tuning of learning rate in Section D.1, and one instance from each method is analyzed. It would be more convincing to see deeper analysis on the variance behaviors to reduce doubts about cherry picking. 

*Clarity*
The paper is clear to the reviewer.

*Overall evaluation*
Overall the paper is borderline to the reviewer. The basic idea makes a meaningful contribution to the field, however its impact may not substantially wide given existing work in the field and related fields. The theoretical analysis has value but not substantial. The experimental validation only meets the minimal requirements of papers in the field and does not make a strong case.  

[1] LiPO: Listwise Preference Optimization through Learning-to-Rank. Arxiv 2024
[2] Scale Calibration of Deep Ranking Models. KDD 2022
strength_weakness: strength_weakness: Below are from the overall review, please refer to the overall review for details. 

- Pairwise methods similar to this and invariance analysis are done in the literature in related fields such as RL and learning to rank.

- Evaluations are not very convincing and only meets the minimal requirements of papers in this field. Risk of cherry-picking.
paper_summary: paper_summary: The paper presents a new RL-based LLM alignment method called P3O and demonstrates its effectiveness with a few small scale RMs and Policy Models on two datasets. The authors claim the traditional PPO algorithm may produce varying updates even given the identical human preference data and P3O algorithm is a good remedy for this issue. Both theoretical and empirical results are presented to show that P3O can more effectively align with human preferences than existing methods, suggesting that comparative feedback could be a superior method for training LLMs to adhere to human values.
strength_weakness: strength_weakness: I don't see any major issues but I do feel a few places can be improved/clarified. 

First, I feel the overall paper writing/presentation is not very good. Some parts of the paper have too dense information. For example, the page 4 section 4.1 is very dense and more discussions are needed. The bad presentation makes the some insights of this paper inaccessible. For example, I feel the reward equivalence in definition 4.3 is very interesting and should be further emphasize. The current way of presentation makes me feel it is defined in a way that can make lemma 4.4 hold (plus make the DPO/P3O enjoys the invariance property while the PPO validates this property). More intuitive explanations on why define the reward equivalence are needed. 

Second, I think the experiments can be further improved in a few ways. First, you can consider more mid-sized (10B-70B) LLMs and show P3O method can generalize. Second, I think more comparison with other online version of DPO-style algorithms (e.g., SLiC [1], RSO [2]) are useful. You current experiment seems already include the online DPO (the end of page 7), why not check other online improved DPO-style algorithms? Finally, I feel the connection between your theory and experiments are not very strong. Your empirical results show P3O is better than PPO/DPO but I don't know if those improvements are due to your claimed theoretical reasons (e.g., resolving RM training, RL finetuning inconsistency, addressing invariant issue, etc) or maybe just come from less implementation noises. I believe some ablation studies to better bridge theoretical insights and empirical results are helpful. 

[1]. SLiC-HF: Sequence Likelihood Calibration with Human Feedback

[2]. Statistical Rejection Sampling Improves Preference Optimization

Comments: comment: Thanks for your response. I have read the rebuttal which solves most of my concerns. I decide to maintain my score.
Thank you for your response. I've read it and my assessment remains the same.
Thanks for the response. I think they are reasonable and will keep the score.