Geometric-Averaged Preference Optimization for Soft Preference Labels

Authors: hiroki  furuta
kuang-huei  lee
shixiang shane gu
yutaka  matsuo
aleksandra  faust
heiga  zen
izzeddin  gur
google  deepmind
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:13:44

Abstract: Many algorithms for aligning LLMs with human preferences assume that human preferences are binary and deterministic.
However, human preferences can vary across individuals, and therefore should be represented distributionally.
In this work, we introduce the distributional soft preference labels and improve Direct Preference Optimization (DPO) with a weighted geometric average of the LLM output likelihood in the loss function.
This approach adjusts the scale of learning loss based on the soft labels such that the loss would approach zero when the responses are closer to equally preferred.
This simple modification can be easily applied to any DPO-based methods and mitigate over-optimization and objective mismatch, which prior works suffer from.
Our experiments simulate the soft preference labels with AI feedback from LLMs and demonstrate that geometric averaging consistently improves performance on standard benchmarks for alignment research. 
In particular, we observe more preferable responses than binary labels and significant improvements where modestly-confident labels are in the majority.

Main paper: 
 introduction: 
 Large Language Models (LLMs) [1, 7, 33] capture a wide range of behaviors and values from training data. However, we would usually prefer these models to focus on useful and safe expressions and abide by social norms. To solve these problems, preference optimization approaches have been popular, either way through reinforcement learning from feedback (RLHF) [12, 34, 4] or direct preference optimization (DPO) methods [41, 52] . These methods usually finetune supervised models on preference data and labels generated by human raters with a wide variety of priorities, backgrounds, knowledge, and skill sets. Nevertheless, existing RLHF and direct preference optimization methods usually assume binary preferences, which ignore the subtle relationship and amplify the bias in the preference labels.To address this issue, we introduce the concept of distributional soft preference labels and improve DPO and its algorithmic families by incorporating a weighted geometric average of LLM output likelihood into the loss function. This approach adjusts the scale of learning loss based on the soft labels and effectively minimizes the loss when presented with equally preferred responses.In the experiments, we simulate the soft preference labels with AI feedback from LLMs [4, 24] and show that soft preference labels and weighted geometric averaging achieve consistent improvement to the baselines on popular benchmarks for the alignment research literature, such as Reddit TL;DR [49] , and Anthropic Helpful and Harmless [3] , as well as original natural language planning dataset based on Plasma [6] . In particular, our results highlight that the proposed methods significantly improve the performance with the data dominated by modestly-confident labels, while conservative DPO (cDPO) [30] , a method leveraging soft labels via linear interpolation of objectives, is stuck to suboptimal performances there. When the models are trained with rich modestly-confident labels, the responses are preferable to those from the models trained with binary labels biased to high-confidence regions. The performance on preference label classification also reveals that cDPO struggles with objective mismatch between the text generation and preference modeling and the weighted geometric averaging could successfully balance both.Our primary contributions are:• We introduce soft preference labels, which can reflect the distributional preference and the finegrained relationship between the response pairs (Section 2.1). Soft preference labels contribute to mitigating over-optimization issues (Section 5.3) and aligning the models to more preferable responses than binary labels (Section 5.1). • We propose the weighted geometric averaging of the output likelihood in the loss function. This can be applied to a family of any algorithms derived from DPO (Section 3). • We point out the objective mismatch between text generation and preference modeling. The better preference accuracy from DPO-style objectives does not ensure better alignment, which conservative DPO suffers from and our geometric averaging can resolve (Section 5.2). 
preliminaries: 
 We denote x ∈ X as a text prompt from the set of prompts X , y ∈ Y as an answer corresponding to the prompts from the set of possible candidates Y, and π(y | x) as a LLM (i.e. policy). We use y 1 y 2 to indicate that y 1 is more preferable than y 2 , and denote a dataset of the paired preference as D = {(x (n) , y2 )} N n=1 . We assume that y 1 y 2 always holds (y 1 is always preferred or equal) in this paper unless specified otherwise.In the RLHF pipeline, we typically go through three phases, such as supervised finetuning (SFT), reward model training, and RL-finetuning [34, 69] . The SFT phase is maximum likelihood training of pre-trained LLMs on downstream tasks, which results in an initial model or reference model π ref for the later RL-finetuning. For the reward modeling, the Bradley-Terry model [5] is often assumed as underlying modeling for the oracle human preference such as p * (y 1 y 2 | x) = exp(r * (x, y 1 )) exp(r * (x, y 1 )) + exp(r * (x, y 2 ))= σ(r * (x, y 1 ) -r * (x, y 2 )),where r * (x, y) is a true reward function and σ(•) is a sigmoid function. Following this assumption, the parameterized reward function r ψ is initialized with a supervisedly-finetuned LLM π ref and trained with negative log-likelihood loss: min ψ -E [log σ(r ψ (x, y 1 ) -r ψ (x, y 2 ))]. RL-finetuning phase leverages the learned reward to update the LLM π θ by optimizing the following objective [34, 69] ,EQUATIONwhere β > 0 is a coefficient to control the KL-divergence regularization. Online RL approaches, such as PPO [45] , are often used to maximize Equation 2, but they are usually computational inefficiency and require a complex pipeline in practice. In contrast, offline preference optimization approaches, such as DPO [41] , are relatively simpler and lightweight in terms of implementation.While a reward model is often trained with binary preferences, we can usually assume distributional soft feedback via majority voting among the human raters or AI feedback with scoring [24] (e.g. y 1 is better than y 2 at a 70% chance). With soft preference labels, we can still easily recover the binary preference with a threshold.We assume that the binary preference labels, l(y 1 y 2 |x) = 1, are sampled from the Bradley-Terry model preference distribution with the parameter p * (y 1 y 2 |x). We define soft preference labels as estimates of the true preference probability: px,y1,y2 := p(y 1 y 2 |x) ≈ p * (y 1 y 2 |x).We denote px,y1,y2 as p ∈ [0.5, 1.0] for simplicity in the later sections. For instance, we can estimate this via Monte Carlo sampling such as p = 1 M M i=1 l i where l i ∈ {0, 1} is a sampled binary label, which is done via majority voting among M people in practice. Because soft preference labels reflect fine-grained relationships between the responses, they may contribute to aligning the models to more preferable responses than binary labels. Alternatively, we can also estimate the soft preference directly via Bradley-Terry models with some reward function. This direct estimation is often adopted in AI feedback with scoring (see Section 4.1 for further details) or the cases with multiple reward models.The sampled binary preference may sometimes flip with probability (i.e. label noise [11, 27, 30] ).If the degree of label noise is known, we may consider the expectation over the noise such as:p = (1 -1 M M i=1 i ) 1 M M i=1 l i + 1 M M i=1 i 1 M M i=1(1 -l i ), or we may ignore the noise when i is small and M is sufficiently large.Let's start with a brief review of DPO and the variants derived from it, such as conservative DPO, IPO, and ROPO. DPO maximizes the estimate of preference probability under the Bradley-Terry model, p θ (y 1 y 2 | x) = σ(r θ (x, y 1 ) -r θ (x, y 2 )), by parameterizing reward models with the policy model π θ itself, which comes from the following relationship in the constraint Lagrangian of RLHF objective (Equation 2),EQUATIONwhereZ(x) = y π ref (y | x) exp 1 β r(x, y) is the partition function. Substituting Equation 4 into log p θ (y 1 y 2 | x), the following objective is derived:EQUATIONNote that we define the reward difference function as h θ (x, y 1 , y 2 ) := r θ (x, y 1 ) -r θ (x, y 2 ).Conservative Direct Preference Optimization Conservative DPO (cDPO) [30] is the most representative work that incorporates soft labels. cDPO smooths the objective functions with soft preference labels via linear interpolation, such asEQUATIONwhere the later term is the DPO loss under flipped labels (i.e. y 2 y 1 ). Moreover, prior works incorporating an extra reward model r ψ to DPO objective have also adopted this formulation [9, 21] , by replacing p into σ(r ψ (x, y 1 ) -r ψ (x, y 2 )).Assuming the Bradley-Terry model as an underlying preference modeling causes over-optimization issues in DPO [2, 52] . To mitigate this problem, IPO [2] has been introduced by replacing reward maximization in Equation 2 with preference distribution maximization. The objective of IPO can be written as,EQUATIONwhere β > 0 is a regularization hyper-parameter. Similar to cDPO, we can also introduce conservative IPO (cIPO) [2, 27] , which results inEQUATION)Robust Preference Optimization ROPO [27] designs the objective to resolve the instability under noisy label problems, which is inspired by the unhinged loss [54] and reverse cross-entropy loss [59] in the noise-tolerant supervised learning literature. The objective is a combination of the regularization term and original DPO loss such as,EQUATIONwhere α > 0 and γ > 0 are extra hyper-parameters to balance the contribution of each term. Geometric averaging (GDPO) can adjust the scale of gradient based on the soft preference labels; if soft preference labels are close to 1 (p = 0.95), the scaling factor of GDPO is almost the same, and small soft labels (p = 0.55) make the scaling factor small while the norm reaches zero. (Right) A 1-D bandit problem with 100 actions, illustrating the histogram of train data and true reward function, preference distribution, and action distribution from the learned policies. Although cDPO accurately fits the data distribution and has the mode in a low-reward region, DPO and GDPO can assign a probability mass in a high-reward region. 
methods: 
 As DPO and the related methods assume binary preference, they cannot reflect the fine-grained relationship between the pair of responses during training. The conservative formulation of DPO can use the soft preference labels, but we found that it could not achieve good performance if modestlyconfident labels shape the distribution as a majority (see Section 5) . In this section, we propose a simple yet effective modification, weighted geometric averaging of LLM output likelihood in the learning loss, which can be applied to a family of algorithms derived from DPO.We assume that the pairs of winner and loser outputs (y w , y l ) are sampled from the weighted geometric average of LLM policies π(• | x) such as,EQUATIONwhere Z π,w (x) := yj ,y k , p π(y j | x) pπ(y k | x) 1-p and Z π,l (x) := yj ,y k , p π(y j | x) 1-pπ(y k | x) p (y j y k ). Because it is difficult to obtain precise estimation of these values with sampling, we set those normalization terms to constant and ignore them in practice, which is a common assumption in deep RL literature [18, 39, 46, 61] . If we have true binary labels (i.e. p = 1), Equation 10 reduces to the original formulation under the assumption of y 1 y 2 .Weighted geometric averaging can be considered as a regularization, which pushes the large likelihood down to small when the soft preference is far from 1. In the following, we present three modified DPO-based methods: Geometric DPO (GDPO), Geometric IPO (GIPO), and Geometric ROPO (GROPO), by replacing the winner output likelihood π(y 1 | x) → π(y 1 | x) pπ(y 2 | x)EQUATIONGeometric Identity Preference Optimization (GIPO)EQUATIONGeometric Robust Preference Optimization (GROPO)EQUATIONThese objectives are consistent with original ones (Equation 5, 7, 9) when we have binary preferences.To analyze the role of weighted geometric averaging, we consider the gradient of loss function with respect to model parameters θ in a general form, which can be written as:∇ θ L = -βE (x,y1,y2, p)∼D   wθ(x, y 1 , y 2 , p) scaling factor [∇ θ log π θ (y 1 | x) -∇ θ log π θ (y 2 | x)]positive and negative policy gradientsEQUATIONwhere w θ (x, y 1 , y 2 , p) is a scaling factor of positive and negative gradients. While defining an estimated preference probability by their own policy LLMs under the Bradly-Terry model as:EQUATIONwe summarize the scaling factor of each method in Table 1 . Comparing ∇ θ L DPO and ∇ θ L cDPO , DPO optimizes the model until the estimate preference ρ θ reaches 1 (w θ = 1 -ρ θ ), and cDPO does until ρ θ matches the soft preference p by assigning a high weight when the estimation is wrong (w θ = p -ρ θ ). DPO pushes the distribution to the oracle preferable outputs, and cDPO may work well as a regularization if the label has high confidence (e.g. p = 0.95). However, the gradient of cDPO may also cause unnecessary model updates around p = 0.5. Intuitively, p = 0.5 means either candidate answers (y 1 , y 2 ) are equally good, but ∇ θ L cDPO forces their likelihoods to be balanced.Scaling Factor wθDPO [41] 1 -ρθ cDPO [30] p -ρθ GDPO (ours) (2p -1)(1 -ρ θ ) IPO [41] 1 β 2 -2 β log ρ θ 1-ρ θ cIPO [27] 2 p-1 β 2 -2 β log ρ θ 1-ρ θ GIPO (ours) (2p -1) 2 1 β 2 -2 β log ρ θ 1-ρ θ ROPO [27] (γ -αρθ)(1 -ρθ) GROPO (ours) (2p -1)(γ -αρ θ )(1 -ρ θ )Table 1 : Scaling factor w θ (x, y1, y2, p) in the gradient of loss function (Equation 14). The estimated preference probabilities ρ θ and ρ θ are defined in Equation 15. Compared to others, geometric averaging has a product of (2p -1) in a scaling factor, which forces the norm of gradients from the equally preferable responses close to zero.In contrast, GDPO adjusts the gradient scale based on soft preference by multiplying (2p -1), which can also ignore the gradient from even candidate pairs. Figure 1 (left) visualizes that weighted geometric averaging can adjust the scale of gradient based on the soft preference labels. If soft preference labels are close to 1 (e.g. p = 0.95), the norm of the scaling factor is almost the same, and small soft preference makes the scaling factor small while the norm reaches zero (e.g. p = 0.55). This maintains the effect from clear relationship pairs, reduces the effect from equally good outputs, and reflects the detailed preference signals among the responses. In practice, we set a larger value for β in GDPO than in DPO to maintain and amplify the scale of the gradient for acceptable preference pairs, which works as an implicit filtering of soft preference labels. We will explain this in Section 5.1.To highlight the advantage of geometric averaging and the failure case of linear interpolation as done in cDPO (Equation 6), we consider a 1-D bandit problem with 100 discrete actions and a linear reward function. Figure 1 (right) illustrates the histogram of train data and true reward function, paired preference distribution, and action distributions from the learned policies. The 500,000 training instances are sampled from a bimodal mixture of Gaussian distribution (with the mode in the 20-th and 70-th indices), and we prepare the paired data from those while labeling preferences with the Bradley-Terry model. We train the parameterized reward r ψ by minimizing L DPO , L cDPO , and L GDPO , and then recover the learned policies analytically as π r ψ (y) ∝ π data (y) exp(r ψ (y)), where π data (y) is an underlying train data distribution. The results demonstrate that cDPO accurately fits the data distribution, which is because the linear interpolation of the loss function in Equation 6 can be interpreted as a minimization of KL divergence E[D KL (p || ρ θ )]. However, this could result in a sub-optimal solution when the train data has a peak in a low-reward region. Because greedy decoding considers the mode of learned distributions, this accurate modeling in cDPO is not aligned with the text generation objectives. On the other hand, DPO and GDPO can assign a probability mass in a high-reward region. GDPO has an advantage against cDPO by resolving such an objective mismatch. Similar trends can be observed in the LLM experiments (Section 5). 2-L, instruction-tuned on Flan dataset. We prepare Reddit TL;DR [69, 49] , Anthropic Helpful and Harmless [4] , and Plasma Plan [6] . We construct competitive paired samples with winner responses and PaLM 2-L to simulate diverse preference distributions that have a peak around the modest confidence (e.g. p ∈ [0.7, 0.9)).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: Our experiments are based on the open-source datasets [49, 4, 6] . We also constructed a novel dataset using the responses from LLMs, but we have not conducted an extensive toxicity check for those outputs. Currently, we are working on it for future release.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In the experiments, we use PaLM 2-XS [1] for the base LLM, as done in prior works [16, 20, 24, 43] (Appendix L uses Gemma-2B/7B as base LLMs). We use the popular RLHF datasets, such as Reddit TL;DR [49, 55] (summarization), and Anthropic Helpful and Harmless [3] (conversation) for the benchmark. To simulate the soft preference labels, we relabel the preference to the datasets by leveraging AI feedback [4, 24] from instruction-tuned PaLM 2-L (Section 4.1). However, because we found that the soft label distributions in popular RLHF datasets only have similar shapes concentrating on high-confidence regions such as p ∈ [0.95, 1.0] (Figure 10 in Appendix I), we prepared (1) new competitive paired responses from a winner in the original dataset and from LLMs and (2) the novel preference dataset based on Plasma Plan [6] , a dataset of daily-life natural language planning, which simulate more diverse preference label distributions we may face in a practical scenario. For instance, Plasma Plan has a pair of instruction x (e.g. see a movie) and the human-written gold plan y (e.g.Step 1: Choose a movie, Step 2: Buy a ticket, Step 3: Go to the theater). To construct a pair of plans, we generated the plans to all the instructions using PaLM 2-L with few-shot prompting, and then obtained the triplet (x, y data , y PaLM ). We gathered about 60K response pairs for train split and 861 examples for test split. Following this procedure, we prepared about 93K (Reddit TL;DR), 44K (Anthropic Helpful), and 42K (Harmless) response pairs as train split. To reduce the inference cost, we sample 1000 test prompt-response tuples in Reddit TL;DR while removing the duplicated ones.For other datasets, we have 1639 (Helpful) and 1614 (Harmless) examples in the test split.To prepare the SFT models, we finetune PaLM 2-XS using 50% of winner responses in train split for Reddit TL;DR, Anthropic Helpful, and Harmless, and using the responses from PaLM 2-L for Plasma Plan. We use those SFT models as an initial checkpoint of preference methods and the reference models π ref .See Appendix B for further details on training.Following prior works [4, 10, 15, 24] , as reliable alternatives to human raters, we simulate the soft preference labeling with AI feedback from LLMs. AI rating is well aligned with humans and is often used as a proxy of human evaluation in the RLHF literature [68] . Throughout the work, we use PaLM 2-L instruction-tuned on Flan dataset [13] as an AI rater. To obtain the soft preferences, we put the context x, first output y 1 , second output y 2 , and the statement such as "The more preferable output is: ", and then get the log probability (score) of token "(1)" and "(2)" from LLMs. Assuming the Bradley-Terry model, we compute the AI preference as follows:EQUATION)Lastly, to reduce the position bias [40, 58] in LLM rating, we take the average of pAI by flipping the ordering of (y 1 , y 2 ) in the prompt. See Appendix F for the prompts of AI rating. For a fair comparison, we prepare the binary labels based on pAI rather than the original labels in the dataset.Figure 2 shows the histogram of soft preference labels from the AI feedback in the preference datasets. We construct competitive paired samples with winner responses and the ones from PaLM 2-L to simulate diverse preference distributions that have uniformity or a peak around the modest confidence (e.g. p ∈ [0.7, 0.9)). We also prepare two other datasets based on Plasma Plan, with different distributions; Plasma Plan Skewed is the more skewed preference dataset by cutting off the high soft preference labels such as p ≥ 0. The methods applying geometric averaging (GDPO, GIPO, GROPO) achieve consistently better performances against binary preference methods (DPO, IPO, ROPO) or their conservative variants (cDPO, cIPO). In particular, geometric averaging has a larger performance gain on Plasma Plan, Skewed, and Stairs datasets, which have richer modestly-confident labels, than others. The improvement of GDPO, GIPO, and GROPO compared to baselines have statistical significance with p < 0.01 on the Wilcoxon signed-rank test, compared to SFT, binary preference methods, and soft preference methods with linear interpolation.(see Appendix E) and also help more clearly demonstrate the behavior of each algorithm when modestly-confident labels are in the majority. They could lead to better performance than preference labels concentrating on high confidence. The train split of Plasma Plan Skewed and Stairs consists of about 30K/27K response pairs, and the test splits are shared among the dataset from Plasma Plan.For the evaluation, we conduct a pairwise comparison between the response from the trained models (y llm ) and the reference response from PaLM 2-L, and GPT-4 [33] (y ref ). The reference responses from PaLM 2-L and GPT-4 are generated with few-shot prompting (see Appendix G). In addition, we directly compare our methods and corresponding baselines (e.g. GIPO v.s. IPO or cIPO).As evaluation metrics, we use the winning rate from binary and percentage judge. We first calculate the AI preference between the response from the trained models and evaluation data as explained in Section 4.1. We calculate the average binary and percent winning rate as follows:binary = 1 |D| (x,yref,yllm) 1 [p AI (y llm y ref | x) ≥ .5] , percent = 1 |D| (x,yref,yllm) pAI (y llm y ref | x).(17) Note that pAI is also averaged among the flipped order to alleviate the position bias. 
results: 
 We first compare the alignment performance among the algorithms with binary feedback (DPO, IPO, ROPO), their conservative variants (cDPO, cIPO), and weighted geometric averaging with soft feedback (GDPO, GIPO, GROPO) on six preference datasets (Section 5.1), and evaluate the preference label classification by the learned models (Section 5.2). We also analyze the log-likelihood ratio and reward gap during training (Section 5.4), and then demonstrate the online alignment performances (Section 5.3). and the corresponding baselines (e.g. DPO, cDPO). The results against SFT are averaged among GDPO, GIPO, and GROPO (Figure 9 ). Geometric averaging consistently outputs more preferable responses than competitive baselines with about 70% winning rate on average. See Appendix K for the example responses. 2 highlights that rich soft labels help align LLMs better than those binary ones. Focusing on DPO variants, cDPO does not work well while GDPO performs the best. We hypothesize that this comes from the objective mismatch between the text generation and preference modeling, which we verify in Section 5.2. Moreover, Figure 3 shows the binary winning rates in the direct comparison between corresponding methods, such as GDPO v.s DPO, and GIPO v.s. cIPO, etc, which also reveals that geometric averaging consistently outputs more preferable responses.Large β as Implicit Preference Filtering As discussed in Section 3.2, weighted geometric averaging makes the norm of the gradient smaller based on soft preference label p. However, an unnecessarily small gradient could stick to sub-optimal solutions. It would be necessary to maintain and even amplify the scale of the gradient from reliable preference pairs. For the rescaling of the gradient, we set larger β because, in geometric averaging, we can regard as using smaller β := βE[2p -1] < β. Such a larger β works as an implicit filtering of soft preference labels. Figure 4 (left) presents the binary winning rate of DPO and GDPO with different β ∈ [0.1, 0.5] on Plasma Plan dataset. GDPO has a peak at β = 0.3, which is larger than that of DPO (β = 0.1), and GDPO can achieve better performance. See Appendix B for further details of hyper-parameters.Since DPO objective (Equation 5) is derived from the assumption under the Bradley-Terry model, we can regard it as training reward models and implicitly estimating preference probability. We here compare DPO, cDPO, and GDPO, estimate the preference probability ρ θ from Equation 15, make a binary label classification (as done in Equation 17), and then compute the average accuracy between predicted labels and true labels given via AI rating. We use Plasma Plan and prepare three different pairs of outputs between PaLM 2-L and (1) humans, (2) GPT-4, and (3) GPT-3.5.Figure 4 (left) shows that all the methods can classify preference labels well when the test split is composed of the responses from PaLM 2-L and humans, which is the same data distribution as the train split. However, DPO sharply decreases the performance for classifying out-of-distribution pairs, such as from GPT-4 and GPT-3.5 (94.0% → 61.6%/66.3%). cDPO achieves the best classification accuracy on average, and GDPO mitigates the performance drop in DPO. Despite the best accuracy through the proper preference modeling, cDPO does not work well in text generation (Section 5.1).As pointed out in Section 3.3, this can be attributed to an objective mismatch between text generation and preference modeling. While preference modeling aims to fit the models into the given data distribution, the model in the text generation outputs the mode of distribution with greedy decoding, which might cause a significant mismatch when the mode of distribution is in the low-reward region. These empirical results highlight that GDPO successfully incorporates the strong performance in DPO and the nuanced relationship from the soft labels while avoiding a mismatch.The analysis of the log-likelihood ratio and the estimated reward gap can characterize the behavior of offline alignment algorithms [51] . In Figure 5 , we measure the log-likelihood ratio of winner/loser responses and estimated reward gap on Plasma Plan and Anthropic Harmless.DPO aggressively pushes down both log ratios and increases the reward gap, since DPO objective forces the model to achieve r θ (x, y w ) -r θ (x, y l ) → ∞, which causes an over-optimization issue. cDPO is more conservative in pushing down the log ratio while leading to worse alignment quality due to objective mismatch. GDPO mitigates the issues of such objective mismatch and over-optimization by maintaining the reward gap increase modestly. Note that, because our paper has focused on open-ended generation tasks, the decrease in the log-likelihood measured with preferable responses does not always matter in contrast to mathematical reasoning or code generation [36, 65, 42] . Our target tasks require pushing down the likelihood of both winner and loser responses to further improve the response quality through the exploration into out-of-distribution regions.Offline alignment methods can be extended to online updates [64, 20] by introducing online feedback processes such as extra reward models or self-rewarding [8] . Due to the cost constraints, online feedback is often asked to be fast and lightweight. However, the quality of preference labels significantly affects the alignment performances. In this section, we demonstrate that weighted geometric averaging can improve online alignment performance by mitigating the quality issues in online feedback. We employ the following two feedback processes: incorporating an extra reward model r ψ (x, y) and leveraging estimated self-preference ρ θ = σ(β log π θ (x,yw)πref(x,y l ) πref(x,yw)π θ (x,y l ) ). Note that we apply stop gradient operation for the self-preference. For the extra reward model, we use PaLM 2-XS, the same as a policy LLM.Figure 6 shows that GDPO performs the best in both settings. This is because GDPO can cancel the gradient from less-confident soft preferences as discussed in Section 3.2, which comes from the case when the on-policy responses are equally good or the estimated preferences in online feedback are not calibrated enough. GDPO demonstrates a significant gain in self-preference. In contrast, DPO degrades the performance worse because the binarization increases the gap from the true preference. We show that geometric averaging consistently improves the performance of DPO, IPO, and ROPO with soft preference labels. We also observe that uniformly distributed soft preference labels achieve better alignments than the original dataset (Appendix I). In fact, the modestly-confident labels do not always mean that the paired responses are noisy or low-quality but even also are more informative, because that could often happen if both responses are good enough. While, as seen in Figure 10 (Appendix I), most datasets for RLHF research only consist of highly-confident pairs, rethinking the effect of preference data distribution on the performances is an important future direction for the practitioners.The automatic AI rating has a good correlation to the human rating, and is a popular and scalable alternative recently [10, 15, 24, 68] .Our experiments have been conducted on datasets labeled by LLMs as a proximal simulation of soft preference due to the cost constraints.Leveraging actual human preferences labeled via majority voting is another possible future work. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We describe the details of experiments in Section 4 and other necessary information in Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
related works: 
 From the helpfulness and safety perspective, it is important to align the outputs from LLMs to the social agreements and our common sense. RLHF [12, 49, 63] is the most popular choice, where we train the reward models to score the predictions and maximize the learned reward with deep RL algorithms [45, 62] . However, this requires additional computational costs from the two independent LLMs and complex pipelines due to on-policy samples. As appealing alternatives, offline algorithms with a single model have been proposed; one of the most representative is DPO [41] , which has been actively extended with different constraints [52, 56] , loss function [2, 17, 67, 64] , iterative online training [9, 20, 66] , nash equilibrium [32, 44, 50] , and combination to rejection sampling [28] .In addition to algorithmic improvements, the alignment problem has been studied from the data perspective [14, 22, 23, 60] , which argues that the high-quality, fine-grained preference data without label noise is critical for the performance [19, 31, 35, 37] . Since the preference labels from the human raters must have disagreements and be diverse, Bayesian [57] or distributional reward modeling [26, 47] and noise-tolerant objectives [11, 27] have been investigated, to maintain the high-quality learning signals even from practical diverse preferences.Our work newly introduces the notion of soft preference labels -a more general and practical formalization of noisy labels -and then a simple yet effective technique to incorporate the distributional preference into algorithms that have only accepted the binary preference before.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: For the main results in Table 2 , we confirmed that the performance of GDPO, GIPO, and GROPO have statistical significance with p < 0.01 on the Wilcoxon signedrank test, compared to SFT, binary preference methods (DPO, IPO, ROPO), and prior soft preference methods with linear interpolation (cDPO, cIPO). We run experiments with one seed due to the computational costs.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
conclusion: 
 While the preference is inherently diverse among humans, most prior works only focus on binary labels. To reflect a more detailed preference relationship, we introduce soft preference labels and a simple yet effective modification via weighted geometric averaging that can be applicable to any DPO algorithmic variants. The results demonstrate that soft labels and geometric averaging consistently improve the alignment performance compared to binary labels and conservative methods with linear interpolation of objectives. Using soft labels improves model responses over binary labels by mitigating over-optimization. We also identify that conservative methods, that can fit the preference distribution much better, suffer from the objective mismatch between the text generation and preference modeling. In contrast, geometric averaging can balance both and empirically works better. We hope our work encourages more uses of soft preference labels for alignment in future.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: See Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] .Justification: We believe our research conforms, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Appendix A.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This work proposes novel algorithms for aligning large language models with human preferences using proportional soft preference labels. This can lead to LLMs that generate outputs that are more tailored to user needs and desires, improving the overall user experience and satisfaction. On the other hand, if the soft preference labels used for training are biased, the resulting LLM outputs could be biased as well, which might lead to insufficient alignment with the social agreement, common sense, and mitigating discriminative responses. It would be an important future study to work on detecting label bias or debiasing preference labels themselves.The use of LLMs for AI feedback and synthetic data generation has significantly reduced the costs associated with manual annotation and data curation, enabling scalable learning. While agreement between human and LLM preferences is generally high (around 80-85%), the remaining 20% of disagreements could contribute to the accumulation of errors through iterative feedback processes, amplifying the less preferred preferences. Continuous human monitoring is therefore crucial to ensure safety and mitigate potential risks. Furthermore, learning with synthetic data, particularly in pre-training, has shown potential for catastrophic performance degradation due to data distribution shifts. It is also important to be mindful of potential performance deterioration during post-training phases, including alignment, when using synthetic data.For SFT and preference methods, we trained PaLM 2-XS with batch size 32, input length 1024, and output length 256. We used cloud TPU-v3, which has a 32 GiB HBM memory space, with a proper number of cores. We run experiments with one seed per setting. Each run took about one day.We set β = 0.1 (Anthropic Helpful, Harmless, Plasma Plan) and β = 0.5 (Reddit TL;DR) for DPO, cDPO, ROPO following Rafailov et al. [41] . As discussed in Section 4, geometric averaging may require larger β to maintain the scale of gradient from the reliable training samples; GDPO and GROPO used β = 0.3 (Anthropic Helpful, Harmless, Plasma Plan) and β = 0.5 (Reddit TL;DR). For IPO and cIPO , we used β = 1.0 (Reddit TL;DR, Anthropic Helpful, Harmless) and β = 0.1 (Plasma Plan) as recommended in Guo et al. [20] . In contrast to DPO and GDPO, the scaling factor of IPO increases as β becomes small (Figure 7 ). For GIPO, we set β to 0.5 (Reddit TL;DR, Anthropic Helpful, Harmless) and 0.05 (Plasma Plan). For ROPO and GROPO, we employed α = 2.0 and γ = 0.1 as described in Liang et al. [27] . In online experiments, we train LLMs in a pure on-policy setting without any reuse of generated data and sample only 2 responses per prompt. It is an interesting future direction to optimize the number of gradient steps to reuse the generated samples (such as batched iteration methods [64, 29] ) and the number of responses sampled per prompt.We save the checkpoint every 200 iterations. To select the final checkpoint after RL-finetuning, we picked the last 4 checkpoints just before the length of outputs to the validation prompts started exceeding the max output tokens (or after pre-defined max gradient steps if such corruption does not happen). We then evaluated the responses from those by AI rating with the reference responses from PaLM 2-L, and selected the best-performed checkpoint. For the main results in Table 2 , we confirmed that the improvement of the performance in GDPO, GIPO, and GROPO against the baselines have statistical significance with p < 0.01 on the Wilcoxon signed-rank test2 for all the pairs as follows: (1) GDPO v.s. SFT, (2) GDPO v.s. DPO, (3) GDPO v.s. cDPO, (4) GIPO v.s. SFT, (5) GIPO v.s. IPO, (6) GIPO v.s. cIPO, (7) GROPO v.s. SFT, (8) GROPO v.s. ROPO.Based on Equation 14and Table 1 , the gradient of each loss function can be written as: First, discuss the helpfulness and honesty of each response. Then, explain which one is better overall and why. Finally, select which response is the most helpful and honest.∇ θ L DPO = -βE (x,y1,y2)∼D [(1 -ρ θ ) [∇ θ log π θ (y 1 | x) -∇ θ log π θ (y 2 | x)]] , ∇ θ L cDPO = -βE (x,y1,y2, p)∼D [(p -ρ θ ) [∇ θ log π θ (y 1 | x) -∇ θ log π θ (y 2 | x)]] , ∇ θ L GDPO = -βE (x,y1,y2, p)∼D [(2p -1) (1 -ρ θ ) [∇ θ log π θ (y 1 | x) -∇ θ log π θ (y 2 | x)]] , ∇ θ L IPO = -E (x,y1,y2)∼D 2 1 2β -β log π θ (y 1 | x)π ref (y 2 | x) π ref (y 1 | x)π θ (y 2 | x) [∇ θ log π θ (y 1 | x) -∇ θ log π θ (y 2 | x)] = -βE (x,y1,y2)∼D 1 β 2 - 2 β log ρ θ 1 -ρ θ [∇ θ log π θ (y 1 | x) -∇ θ log π θ (y 2 | x)] , ∇ θ L cIPO = -βE (x,y1,y2, p)∼D 2p -1 β 2 - 2 β log ρ θ 1 -ρ θ [∇ θ log π θ (y 1 | x) -∇ θ log π θ (y 2 | x)] , ∇ θ L GIPO = -βE (x,y1,y2, p)∼D (2p -1) 2 1 β 2 - 2 β log ρ θ 1 -ρ θ [∇ θ log π θ (y 1 | x) -∇ θ log π θ (y 2 | x)] , ∇ θ L ROPO = -βE (x,y1,y2)∼D [(γ -αρ θ ) (1 -ρ θ ) [∇ θ log π θ (y 1 | x) -∇ θ log π θ (y 2 | x)]] , ∇ θ L GROPO = -βE (x,y1,y2, p)∼D [(2p -1) (γ -αρ θ ) (1 -ρ θ ) [∇ θ log π θ (y 1 | x) -∇ θ log π θ (y 2 | x)]] .Rationale:Prompt for AI Feedback (Train) on Anthropic Harmless (from Lee et al. [24] )You are an expert rater of harmful assistant responses. Given the context and the two responses choose the most harmful response.» For the evaluation, we prepared high-quality reference responses from capable LLMs such as PaLM 2-L and GPT-4 (Section 4.2). In this section, we show the prompts used for reference generation with PaLM 2-L. We adopted few-shot prompts by taking exemplars from the train split. Anthropic Harmless adopted zero-shot prompts because capable LLMs are often well-aligned and refuse the response if the harmful contents are in the prompts. To avoid redundancy, some few-shot exemplars are omitted here.Prompt for Reference Generation on Reddit TL;DR You are an expert AI assistant to help the human. Please fill helpful and concise response to the following conversation.Human: What is the customary amount to tip a taxi driver? Assistant: The standard tip for taxi drivers in the United States is typically a 20% tip. This is a fairly standard tip that is given to taxi drivers for a job well done, and it's also a nice gesture to show your appreciation for their service. The tip can be more or less depending on the driver's service and the level of professionalism they showed. If you're considering tipping a taxi driver, I would recommend giving them a tip of at least 20%.{exemplar_1} {exemplar_2} {conversation}You are an expert AI assistant to help the human. Please fill harmless and concise response to the following conversation.{conversation}Given a goal write down a list of steps to achieve the goal. Please do not do a conversation with users. You are only allowed to respond the step-by-step plan.Goal: spend a year living a healthier lifestyle Plan:- In Section 4 and 5, we augment the standard RLHF benchmarks (Reddit TL;DR [49] , and Anthropic Helpful and Harmless [3] ) with responses from LLMs to simulate rich soft preference distributions. In this section, we relabel the original paired responses in the dataset with AI feedback and then compare the performance between the baseline algorithms (SFT, DPO, cDPO, IPO, cIPO, ROPO) and the ones applying geometric averaging (GDPO, GIPO, GROPO). Leveraging the original paired responses with soft labels, we compare the winning rate on Reddit TL;DR, and Anthropic Helpful and Harmless in Table 3 (against the responses from PaLM 2-L and GPT-4), and Table 4 (against winner response in the dataset). As demonstrated in Section 5, weighted geometric averaging consistently improves the performance against binary preference methods or soft preference methods, even with a soft-label dataset biased to high-confident pairs. However, the average of absolute difference (Ave.∆(+Geom.)) is lower, and the winning rates themselves on Anthropic Helpful and Harmless are also lower than Table 2 where the methods can fully benefit the rich soft label distributions. When the soft preference labels concentrate on a low-confident region (e.g. p ∈ [0.5, 0.55)) as in Reddit TL;DR (Figure 2 ), the winning rate with the original dataset can beat the one with rich soft labels, while the average of absolute difference (Ave.∆(+Geom.)) in this setting is still lower than the one from rich soft-label distribution.Moreover, Figure 11 provides the binary winning rates in the direct comparison between geometric averaging and corresponding baselines. The results show that geometric averaging can respond with more preferable outputs by about 70%. These results and comparison to Section 5 demonstrate that (1) weighted geometric averaging can improve the performance even when many soft labels concentrate around p ∈ [0.5, 0.55) or p ∈ [0.95, 1.0], and that (2) rich soft preference labels help improve the performance more than deterministic ones. It is a critical problem whether human feedback and AI feedback are correlated with each other. Table 5 provides the agreement evaluation between human and LLM judges (by PaLM 2-L) on Plasma Plan. We compare the responses from PaLM 2-L and GPT-3.5, and ask to judge which is better than another. The agreement accuracy reaches 81.3%, which is consistent with previous works [68, 24] . In this section, we provide the example responses from the models trained by DPO, cDPO, GDPO, IPO, cIPO, GIPO, ROPO, and GROPO, on Plasma Plan dataset. The context x is shared across similar algorithms for the comparison.Example Responses from DPO on Plasma Plan To demonstrate the scalability of our proposed method, we here provide the additional results with Gemma-2B/7B model [53] , which is an open LLM model with an architecture and pre-training different from PaLM 2-XS; an LLM we mainly used in this paper. In this section, we provide the theoretical analysis of the optimality gap in GDPO. We here derive a corollary stemming from Theorem 4.1 in Song et al. [48] , which shows the bound of optimality gap is improved by GDPO: from O(C √ dpo ) (DPO) to O(C √ dpop) (GDPO). We start with the review of the assumption and results in Song et al. [48] .O.1 Brief Review of Song et al. [48] Assumption O.1 (Global Coverage [48] ). For all π, we haveEQUATION)Definition O.2 (DPO Implicit Reward Class [48] ). DPO constructs the implicit reward class with the policy class Π:EQUATIONWe assume that the learned reward r dpo (x, y) = β log πdpo(y|x) πref(y|x)Z(x) ∈ R dpo satisfies the following assumption: Assumption O.3 (In Distribution Reward Learning [48] ). We assume the DPO policy π dpo satisfies that:EQUATIONTheorem O.4 (Optimality Gap in DPO; from Song et al. [48] ). Let π ref be any reference policy such that Assumption O.1 holds. For any policy π dpo such that the event in Assumption O.3 holds, we have thatEQUATIONFor the proof of Theorem O.4, we have the following Lemma: Lemma O.5 (Objective Decomposition [48] ). Let J(π) be the KL-regularized reward maximization objective, and for reward function r, we letEQUATIONthen we haveEQUATIONProof of Lemma O.5. [48] NeurIPS Paper Checklist • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test 


Decision: This paper studies the alignment of LLMs, and proposes to utilize a richer feedback than the usual binary preference labels. The authors assume they have access to distributional soft preference labels, and propose a weighted geometric averaging technique that can be applied to DPO and its variants. The main claim is that their training objective adjusts the scale of learning loss based on soft labels, which can improve performance on standard alignment benchmarks. 

The reviewing team all agree that the main idea of the paper is interesting, and that the paper introduces an incremental yet solid contribution to the literature. They have raised several issues—which I will not repeat here—that authors should incorporate in a revised version of the submission. In particular, the authors should provide a clear discussion of why the geometric averaging schemes allow incorporating soft labels.

Furthermore, I recommend the authors discuss where they expect soft preference labels to be derived from in practice. LLM-generated soft labels may not always be reliable (esp. outside of research datasets) and it is unclear what the cost of such finer-grained information would be.

Reviews: review: paper_summary: paper_summary: Many existing algorithms for aligning large language models (LLMs) with human preferences assume these preferences are binary and deterministic. However, preferences can vary among individuals and should be represented distributionally to capture subtle relationships between responses. This work introduces distributional soft preference labels and enhances Direct Preference Optimization (DPO) by incorporating a weighted geometric average of the LLM output likelihood in the loss function. This adjustment scales the learning loss based on soft labels, minimizing loss for equally preferred responses. This modification is applicable to any DPO family, addressing the objective mismatch seen in prior work. Experiments using simulated soft preference labels with AI feedback from LLMs show that geometric averaging consistently improves performance on standard alignment benchmarks, particularly with data containing a majority of modestly-confident labels.
strength_weakness: strength_weakness: Given the multiple aspect of human preference, the proposed method is only using a soft label scalar scoring which is established based on the basic assumption of transivity, derived from BT model and its formulation. Therefore, the trade-off between diverse aspects of preferences, especially when the preferences are contradictory, is not explicitly addressed.
questions: questions: It would be interesting to understand how could a model derived based on transitivity assumption or soft label based scalar ratings be able to reflect multiple aspects of real world preferences, e.g., the balance between helpfulness and harmlessness in Anthropic dataset.
limitations: limitations: N/A
paper_summary: paper_summary: This paper introduces the concept of soft preference labels and proposes leveraging this distributional information, alongside deterministic binary preferences, to enhance Direct Preference Optimization (DPO) losses. By incorporating a weighted geometric average of the LLM output likelihood in the loss function, the authors demonstrate that geometric averaging consistently improves performance on standard benchmarks.
strength_weakness: strength_weakness: The writing in general is very hard to follow and at times very unclear. Some examples are as follows:

1. The soft Preference Labels $\hat{p}$ is poorly defined. What exactly does the perturbation model refer to? Is it the personalized preference model relative to the population-averaged model? If so, over what is the expectation taken? Why is this $\hat{p}$ readily available data? Should it be $\widehat{p}_{x,y_1,y_2} = 1 - \frac{1}{M}\sum_{i=1}^M \epsilon_i(x,y_1,y_2) \approx 1 - \mathbb{E}[\epsilon(x,y_1,y_2)] = p^\star(y_1>y_2 | x)$ ? 

2. The introduction is not easy to parse. Providing a concrete example of the problem being studied would help. For instance, the statement "Nevertheless, many existing RLHF algorithms and Direct Preference Optimization (DPO) [38] variants assume binary deterministic preferences" is confusing, as the usual assumption for DPO is a Bradley-Terry preference model, which is not deterministic. This isn't clarified until the next section.

3. There is no justification for why weighted geometric averaging over the LLM output makes sense. If this approach makes mathematical sense after reparametrization, it would be very helpful to explain this clearly.

4. The explanation in chapter 3.3 is very hard to follow.

5. Are soft preference labels readily available with the current data curation process, or are you advocating for their utilization and thus encouraging the community to propose methods for obtaining them?
questions: questions: As previously mentioned in the weaknesses section, I am mainly confused about the soft preference labels: are you proposing to obtain these soft preference labels instead of the binary deterministic ones currently used, or are you suggesting that we have been under-exploiting our data, which already contains this information?

Additionally, it would be helpful to explain how this distributional information is philosophically different from the Bradley-Terry model, where preference labels are random draws from a fixed distribution representing population-level preferences.
limitations: limitations: The authors address some limitation in their writing.
paper_summary: paper_summary: This paper introduces a novel approach to aligning Large Language Models (LLMs) with human preferences by incorporating distributional soft preference labels. The authors argue that existing methods like Direct Preference Optimization (DPO) assume binary, deterministic preferences, which may not accurately reflect the nuanced nature of human judgments. To address this, they propose a modification to DPO that uses a weighted geometric average of LLM output likelihood in the loss function, allowing for a more nuanced representation of preferences. The method can be easily applied to any DPO-based algorithm and shows consistent improvements on standard alignment benchmarks, particularly for data with modestly-confident labels. The authors simulate soft preference labels using AI feedback from LLMs in their experiments.
strength_weakness: strength_weakness: The paper lacks sufficient theoretical analysis to justify the use of "Geometric Averaging" for soft labels in preference optimization. While the authors compare their method to others in terms of scaling factors, they fail to provide an in-depth analysis of its effectiveness. Key questions remain unanswered: What are the core weaknesses of previous scaling factors? Why are scaling factors crucial in preference optimization? How does the proposed method fundamentally differ from prior work? The paper would benefit from a more rigorous examination of the method's impact on training dynamics, such as reward gaps and convergence stability. Without this analysis, it's challenging to fully understand and evaluate the method's contributions to the field.
questions: questions: The soft label addresses preference variability, which relates to label noise. While the authors discuss robustness to noise, it's unclear how this approach actually performs when faced with inconsistent or noisy preference labels. Can the authors provide a more detailed analysis of the method's robustness to label inconsistencies? This would help clarify the practical advantages of soft labels in real-world scenarios where human preferences may be inconsistent or contradictory.
limitations: limitations: The authors adequately discussed the limitations in the seperated section.
paper_summary: paper_summary: The paper proposes a variation to Direct Preference Optimization which takes into account "soft labels," which aim to reflect that not every evaluator might agree on the relative ordering of a pair of model outputs, or that there may otherwise be a lack of confidence in the ordering of outputs. They model this as the likelihood that one output is better than another and use this likelihood in selecting the winner from two prompts using geometric averaging of the confidence in each output.

This approach is straightforwardly extended to several algorithms in the DPO family. This is then used in fine-tuning a model. Experiments compare their variations of DPO with other DPO algorithms and show that an LLM evaluation agent prefers their model outputs over those of PaLM-2L and GPT4 more often than the evaluator prefers other DPO outputs.
strength_weakness: strength_weakness: I am only loosely familiar with the current research in this area but I am aware of criticism of the Bradley-Terry model. While this might help to learn preferences under the BT model, it might aid the paper to include a comment on whether this model is worth continued analysis.

Some aspects of the paper could be written more clearly to explain what is happening or why. In particular, I find the paper likely assumes that readers are extremely familiar the most recent approaches to LLM evaluation and corresponding norms of how to present these results. Not being familiar with these things, I feel there are several aspects of the paper that could use more explanation; some are listed below.


The definition of p^ is vital to a thorough understanding of the paper. Right now, I find it's meaning somewhat difficult to parse when there are no subscripts. The paper would be improved by a more clear explanation of what it means in context. This would also make Figure 2 and some of Figure 1 more clear.

Similarly, in Eq. 10, y_w and y_l seem disconnected from y_1 and y_2. I generally understand what's going on but a more clear explanation would aid many readers.


The explanation of exactly what models you applied your techniques to may be clear to someone actively engaged in very similar research but was unclear to me.

Eq 17 needs clarification. From study it's meaning can be interpreted but seeing as both y's come from llm's (as far as I understand) and are involved in testing it is not immediately clear which meaning to assign to y_llm and y_test.


Some of the minor editing issues:

Sentence on line 113 is not correct.

line 129 - "Bradly-Terry"

line 158/159 - maybe "500,000 training instances are ..."?
questions: questions: Can you expand on the reasonableness of using an LLM to judge LLM performance? Can this lead to potential damage to model training in the future?
limitations: limitations: See question above.
paper_summary: paper_summary: Pretrained large language models know information about a wide range of topics, but since pre-training is often done on internet scale data, these models are not aligned with human values. Offline preference learning methods such as DPO are getting increasingly popular for this task.

A key assumption for DPO is that it sees a binary preference dataset  of $(x, y_w, y_l)$ tuples, where $x$ is the prompt, $y_w$ is the preferred and $y_l$ is the dispreferred response. However, this binary labeling is often too hard and does not count for the difference between two responses: while responses $y_1$ and $y_2$ may have a very clear preferred-dispreferred relationship, another pair $y_3$ and $y_4$ might have a much smaller difference. 

This paper proposes an algorithm to take into account the relative preference between two responses. The relative preference is not binary and can take any value between 0.5 and 1.0: closer to 1.0 means the preference is almost binary, whereas closer to 0.5 means it is a tossup between two responses. Prior work such as cDPO has used a linear interpolation between two different DPO losses with reversed preference relationships. In comparison, this paper assumes geometric averaging between two LLM policies, and this results in a simple modification to the existing DPO algorithm and its derivatives. This paper’s method, GDPO, outperforms DPO in the case of different soft preference labels.
strength_weakness: strength_weakness: I will list the weaknesses from most important to less important, in my opinion.

**(What about DPO pushing down probabilities of both positive and negative examples)**

The biggest problem with DPO in my opinion, is that it pushes down the log probabilities of both positives and negatives, though it pushes down that of negatives much more, increasing the reward margin. Recent work such as [1] discusses this issue. This makes DPO extrapolate to an OOD region, which might be either good or bad. Discussing the results of [1] in context of this paper’s algorithm would be necessary: we would not want an algorithm that would push down probabilities of winning responses. On the other hand, GDPO’s gradient weighting term, $w_{\theta}$, can actually improve this situation too, and it would be good to know. 

Prior work such as RPO [2] has attempted to fix this issue by adding a SFT loss to the DPO loss, whereas SimPO [3] shows that a lot of the mismatches happen because of the reference model’s wrong reward attribution, and removing the reference model from the loss calculation can help. While comparing all of these might be out-of-scope for this paper, at least documenting the basic results from [1], i.e., what happens to the log probabilities of winning vs losing responses under GDPO, DPO and cDPO, would strengthen the paper. Also how does this vary depending on the soft preference distribution?

**(Online DPO)**

Multiple works have shown since DPO that offline preference tuning methods are sub-optimal, and the DPO objective, coupled with samples generated from the model itself, is generally better [1, 4, 5, 6, 7]. These methods can either use the reward from the language model itself to train it [6], or a separate reward model [1]. Could the authors expand this paper’s method to the online variant of DPO? Two problems I can see from the start: in order to use a self-rewarding scheme, the probability/reward from the policy needs to somehow inform us of the soft preference label. Would model miscalibration hurt this? Also for a separate reward model, if not sufficiently strong enough, obtaining the soft preference label can be hard. Also obtaining it during training, for every training sample, can become computationally expensive.


**(More architectures/models tried out)**

Trying this algorithm with more recent models, such as LLama3, or other more commonly used models such as Pythia or Mistral would strengthen the paper. **Note that I do not consider this a major weakness, just something that would make the paper more comprehensive.**
questions: questions: **Possibility of extension to token level labels**

Could this paper’s method be extended to token level labels? Eg, assume a math problem that has 5 steps. Assume responses $y_1$ and $y_2$ that are both wrong, but wrong in different steps, i.e., $y_1$ gets it correct up till 3rd step, whereas $y_2$ gets it correct till 2nd step. Can the soft preference labels reflect that/can we still make the model learn something here?

# References

[1] Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data, https://arxiv.org/abs/2404.14367

[2] Iterative Reasoning Preference Optimization, https://arxiv.org/abs/2404.19733

[3] SimPO: Simple Preference Optimization with a Reference-Free Reward, https://arxiv.org/abs/2405.14734

[4] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study, https://arxiv.org/abs/2404.10719

[5] Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint, https://arxiv.org/abs/2312.11456

[6] Self-Rewarding Language Models, https://arxiv.org/abs/2401.10020

[7] Direct Language Model Alignment from Online AI Feedback, https://arxiv.org/abs/2402.04792
limitations: limitations: Yes

Comments: title: Response to the Authors
Followup from Reviewer 98AM
Response to Reviewer EtLq
Response to Reviewer 98AM (3/3)
Response to Reviewer 98AM (2/3)
Response to Reviewer 98AM (1/3)
comment: Dear Authors,

Thank you for the detailed response. I have read the rebuttals that the authors provided as well as other concurrent discussions, and I am willing to raise my score accordingly since I think the idea is interesting enough.
Dear Authors,

Thanks a lot for the updated results! This is satisfactory to me, I am increasing the score from 5 to 6, since **the authors have updated the rebuttals with results of 7B parameter range LLMs, as discussed before.**

Kudos on driving a successful rebuttal.
Dear Reviewer,

While this is a last minute before closing discussion period, we would appreciate it if you could check our updates and feel free to raise further questions if you have any. We are happy to address them further. Thank you so much for your time.

Sincerely,

Authors
We thank the reviewer for reading the responses and your thoughtful consideration.

As the reviewer pointed out, it is an important direction in practical scenarios to align LLMs to multiple preferences conflicting with each other. In addition to our experimental results, we will include this point in the revision. Thank you again for taking your time.
We thank the reviewer for reading the rebuttal and the thoughtful consideration.

For the benchmark, we believe that Reddit TL;DR and Anthropic Helpfulness and Harmlessness datasets are widely adopted in previous RLHF/alignment works (e.g. https://arxiv.org/abs/2305.10425, https://arxiv.org/abs/2309.06657, https://arxiv.org/abs/2309.16240, and more). As the reviewer mentioned, because the soft labels have not been used much ever and the community did not have suitable public dataset, we needed to build our experiments on top of popular configurations. We hope our work inspire the community to leverage the soft labels and release the open dataset with soft label annotations in the future.

We will include all the discussion with the reviewer (analysis on training dynamics & theoretical analysis) in the revision. Thank you again for taking your time.
**> Discussion about Log Ratio and Training Dynamics**

We thank the reviewer for discussing this thoroughly. We also agree with your final recap and will include all the experimental results, discussions, and references you provided during the rebuttal in the revised paper.

**> Winning Rate on Plasma Plan, Skewed and Stairs with Gemma-7B**

In the following table, we compare the winning rate of Gemma-7B among DPO, cDPO, and GDPO (against PaLM 2-L). The results show that, following the results on PaLM 2-XS and Gemma-2B, applying soft labels and weighted geometric averaging improves the alignment quality compared to the binary methods (DPO) and conservative methods with soft labels. Geometric averaging can be effective for various model sizes and architectures.

| Method | Plasma Plan (Binary) | (Percentage) | Skewed (Binary) | (Percentage) | Stairs (Binary) | (Percentage)
|--|--|--|--|--|--|--|
|SFT| 42.62% | 44.45% | 42.62% | 44.45% | 42.62% | 44.45% |
|DPO| 79.56% | 61.53% | 78.63% | 61.47% | 75.49% | 60.12% |
|cDPO| 74.33%| 59.91% | 73.52% | 56.79% | 71.89%| 59.91% |
|GDPO| 82.58% | 64.11% | 82.23% | 63.73% | 80.37% | 62.61% |
|$\Delta(+\text{Geom.})$| +5.64% | +3.39% | +6.16% | +4.60% | +6.68% | +2.60%|

Lastly, thank the reviewer again for your active engagement with the rebuttal and discussion despite the limited time. Your thoughtful feedback helped improve our paper.
I have read the rebuttal and thank the authors for their detailed and candid responses.

It is straightforward to understand that the proposed method outperforms all baseline methods, despite that the structural gap in handling contracting real-world preferences remains unresolved.

I maintain my positive opinion on this paper.
Thank you for your response.

After reviewing the authors' replies to the other reviewers and following the ongoing discussions, I find the concept of "soft preference labels with AI feedback" quite interesting, and I have raised my score accordingly.

I acknowledge that the proposed method effectively addresses some algorithmic challenges in DPO, and the analysis conducted on it appears promising. My main concern now is whether the TLDR and Anthropic HH datasets are sufficient to fully validate the method's efficacy. However, I understand that this limitation is unavoidable given the scarcity of available public data, so I did not consider this part when deciding my score.
I thank the authors for the prompt response!

I agree with the points the authors present. As long as the authors present the log-probability argument in this paper in the final version of the paper, along with discussions of the following papers:

[1] Iterative Reasoning Preference Optimization, https://arxiv.org/abs/2404.19733

[2]  Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data, https://arxiv.org/abs/2404.14367

[3] From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function, https://arxiv.org/abs/2404.12358

I am satisfied with the results. The reason I insist on this is: It is certain now that DPO over-extrapolates to an OOD region. Whether this OOD region is good or bad depends on the task: for math problems, as [1] shows, it is probably bad, because only a narrow distribution of responses is good, whereas for RLHF tasks that this paper considers, this might be okay. **It is important to note the effect of new variations of DPO in this regard, specially their learning dynamics.**

Section 5.3 of [3] seems to show a theoretical understanding of why this happens, and possibly including a discussion of this in the paper should improve the quality of this work.

I am willing to increase my scores once the authors produce results on models of around 7B parameter count range.

Kudos to the authors for leading a nice and thorough rebuttal!
We appreciate the quick response and clarification. The potential social impacts have been discussed in Appendix A,  "Broader Impacts" section (in the original submission), and following your clarification, we can further extend it considering the effect of AI feedback or training data synthesized by LLMs in the future.

The use of LLMs for AI feedback and synthetic data generation has significantly reduced the costs associated with manual annotation and data curation, enabling scalable learning. However, it remains unclear whether LLMs can accurately identify biases such as prejudice and discrimination when providing AI feedback; potentially LLMs wrongly provide preference labels leading to undesirable biases. Additionally, while agreement between human and LLM preferences is generally high (around 80-85%), the remaining 20% of disagreements could contribute to the accumulation of errors through iterative feedback processes, amplifying the less preferred preferences. Continuous human monitoring is therefore crucial to ensure safety and mitigate potential risks. Furthermore, learning with synthetic data, particularly in pre-training, has shown potential for catastrophic performance degradation due to data distribution shifts. It is also important to be mindful of potential performance deterioration during post-training phases, including alignment, when using synthetic data.

We will include these discussions in the revised version.
**> Reference Model Free Version (Geometric SimPO)**

We think we can apply geometric averaging to the reference model free SimPO. We start with the DPO objective without a reference model (we omit the input $x$ for readability). The derivation of the objective is as follows: 

$E[\beta \log\pi_{\theta}(y_w) - \beta \log\pi_{\theta}(y_l) ]$

$= E[\beta \log \frac{\pi_{\theta}(y_w)^{\hat{p}}\pi_{\theta}(y_l)^{1-\hat{p}}/Z_{w}}{\pi_{\theta}(y_l)^{\hat{p}}\pi_{\theta}(y_w)^{1-\hat{p}}/Z_{l}}]$

$= E[\beta (2\hat{p}-1) \log\pi_{\theta}(y_w) - \beta (2\hat{p}-1) \log\pi_{\theta}(y_l) - \beta\log\frac{Z_{w}}{Z_{l}}]$

where $Z_{w}$ and $Z_{l}$ is a partition function. Because they are hard to estimate accurately, we may treat the term $\beta\log\frac{Z_{w}}{Z_{l}}$ as a constant value $\gamma \geq 0$. By adding length normalization coefficients, we have an objective for Geometric SimPO.

$L_{\text{GSimPO}} := E[\frac{\beta(2\hat{p}-1)}{|y_w|} \log\pi_{\theta}(y_w) - \frac{\beta(2\hat{p}-1)}{|y_l|} \log\pi_{\theta}(y_l) - \gamma]$.
**(Continuing from the last thread)**

- (6) The target tasks or metrics can be related to the necessity of the increasing trends of the preferable log-likelihood. For instance, mathematical reasoning or code generation tasks can measure the performance with exact match/unit tests and need to maintain reasonable responses, because the best responses do not significantly differ from the preferable responses in the dataset. In these tasks, the degradation of preferable outputs in DPO may cause significant issues. DPO v.s. PPO paper (https://arxiv.org/abs/2404.10719) has stated `However, we will demonstrate in Sec. 6 that even with a nearly perfect annotator, the performance of DPO remains unsatisfactory in challenging tasks such as code generation.` (from Section 4.3: Practical Remark). In contrast, our paper has focused on "open-ended generation" tasks such as summarization or conversation, where we cannot measure the performance with an exact match, rather requiring a human or AI rater to evaluate the quality. In these tasks, the exploration into out-of-distribution regions makes more sense, and the policy needs to push down the likelihood of the (both winner and loser) responses to further improve the response quality.

In summary, we believe that our additional experiments (Figure 1 (f) in the rebuttal PDF) characterize the behavior of geometric averaging well;  cDPO suffers from objective mismatch due to the conservative update (both log probabilities do not push down much, and the winning rate is the worst). DPO faces an over-optimization issue induced by the maximization objective $r_{\theta}(x,y_w)-r_{\theta}(x,y_l) \rightarrow \infty$. This is aligned with our experimental observations (both log probability pushes down the most and the largest reward gap. The performance is the second.) GDPO resolves those two issues by adjusting the scaling factor of the gradient with soft labels (the decrease of log probability suppresses, and the reward gap stays in a modest range. The performance is the best). The trend of the log probability/reward gap is not proportional to the alignment quality. They need to be in a reasonable range. Because the policy has been initialized with SFT models finetuned with (50%) winner responses in the dataset, there is no margin for any algorithms to increase the preferable log-likelihood further. RPO increases log probability, but this comes from the experimental settings. The RPO paper has started the experiments from the checkpoints, which are not finetuned with the target tasks and have enough margin for improvement.

Lastly, we provide the table to compare the relationship among log-likelihood, reward gap, and the binary winning rate (the results are the same as Figure 1 (f) in the rebuttal PDF).

||400|800|1200|1600|2000|
|--|--|--|--|--|--|
|DPO (Pref ratio)|-3.97|-5.23|-7.77|-9.96|-11.9
|cDPO (Pref ratio)|-1.14|-1.46|-1.79|-2.00|-2.00
|GDPO (Pref ratio)|-3.24|-3.38|-4.49|-5.83|-9.18
|DPO (R gap)|33.7|38.4|43.3|49.0|54.5
|cDPO (R gap)|13.6|14.7|15.3|16.0|16.1
|GDPO (R gap)|23.1|26.6|29.4|33.5|40.8
|DPO (WR)|68.18|73.29|78.75|81.53|83.16
|cDPO (WR)|51.57|61.41|70.62|75.96|72.13
|GDPO (WR)|71.66|75.49|80.95|84.55|85.48

We appreciate the reviewer deepening the discussion. We are happy to address your concerns if you have any.


**> The online performances do not reach the offline performance.**

We'd like to clarify that our additional experiments (Figure 1 (h) in the rebuttal PDF) intend to verify the scalability of our geometric-averaging methods to online (on-policy) settings; i.e. online GDPO > online DPO or online cDPO, and do not intend to compare/discuss the performance between offline methods and online methods. The results reveal that, as the same as offline settings, online GDPO consistently outperforms online DPO and cDPO in both the extra reward model and self-rewarding settings.

For the discussion about online DPO and offline DPO, we would also like to clarify that their performances significantly depend on the scale of reward models (i.e. the preference label quality). In the Online AI Feedback paper (https://arxiv.org/abs/2402.04792), Figure 3 shows that, when the preference labels are annotated by PaLM 2-L, online DPO achieves a 95% winning rate against the SFT model on Reddit TL;DR dataset, and offline DPO achieves 90% winning rate. In contrast, Figure 5 also shows that even online DPO achieves a lower winning rate when the scale of reward models is small; online DPO with PaLM 2-S achieves an 86% winning rate against the SFT model (blue bars) and online DPO with PaLM 2-XS achieves 82%.

Because our online experiments use PaLM 2-XS for the extra reward model and offline experiments use PaLM 2-L, our performance gap between online and offline methods is consistent with the previous literature. Our experiments emphasize the scalability of geometric averaging under the same conditions.

We are glad to provide further clarification upon request.
We appreciate the quick and detailed response. Please let us know if our follow-up responses address your concerns.

**> What are the parameter counts of these two models? What is their agreement rate?**

The number of parameters in PaLM 2 models is not publicly available in the paper (https://arxiv.org/abs/2305.10403). However, the former version of PaLM models opens their parameter counts (https://arxiv.org/abs/2204.02311). The smallest PaLM has 8B parameters, and the largest PaLM has 540B. PaLM 2 might have similar configurations to PaLM.

For the agreement rate between trained reward models (PaLM 2-XS) and AI rater (PaLM 2-L), we can refer to the preference classification accuracy after the convergence: PaLM 2-XS achieves 90.1% train accuracy and 93.8% validation accuracy. For the agreement rate on **out-of-distribution** data, we can also refer to Figure 4 (right) in the main text. While a preference model analytically recovered from the DPO policy (based on PaLM 2-XS) achieves 94.0% accuracy between PaLM 2-L v.s. Human samples (the same distribution to the training data), it drops the accuracy to 61.6% (PaLM 2-L v.s. GPT-4 samples) and 66.3% (PaLM 2-L v.s. GPT-3.5 samples) when the response pairs are far from training data distribution. This implies that trained external reward models may not be so accurate when DPO/cDPO/GDPO outputs out-of-distribution responses.

**> 2B LLMs are not sufficient. Results with 7-8B LLMs?**

Because the similar size LLMs -- Pythia-1.4B/2.8B (https://arxiv.org/abs/2304.01373) -- are often used in RLHF/offline alignment papers (e.g. DPO: https://arxiv.org/abs/2305.18290, Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data: https://arxiv.org/abs/2404.14367) as a default, we still believe that our additional results from Gemma-2B are enough to claim the scalability of geometric-averaging methods to different LLMs.

However, following the reviewer's request, we can also provide the results from Gemma-7B (https://arxiv.org/abs/2403.08295). We are running the experiments now and please let us follow this up in a few days.

**> What about DPO pushing down probabilities of both positive and negative examples**

First, we -- the author and the reviewer -- can agree that **the log probabilities of preferable/less-preferable samples from the dataset (or their gap) can characterize the training dynamics and algorithm behaviors, but cannot be surrogate metrics proportional to alignment performance**.

Iterative RPO (https://arxiv.org/abs/2404.19733) paper and our experiments provide the evidence to support this:

- (1) Figure 2 (a) in the Iterative RPO paper has shown that SFT achieves a larger log probability of winner samples than RPO (DPO+SFT) with monotonic improvement (i.e. SFT > RPO). However, in terms of performance (test accuracy of GSM8K in Table 1), RPO surpasses SFT (i.e. RPO > SFT). This suggests that it is unclear whether a larger log probability of winner samples correlates to the improvement of target metrics or not.

- (2) Our results (Figure 1 (f) in the rebuttal PDF) also reveal that while the order of log probability of winner samples is cDPO > GDPO > DPO,  the order of the winning rate (Table 1 in the main text) is GDPO > DPO > cDPO.

- (3) The analysis paper (https://arxiv.org/abs/2404.14367) initiating this log probability discussion has only observed the fact that DPO pushes down both preferable log probability and less-preferable log probability, but has not claimed that the increasing trend of the preferable log probability improves the performance.

Moreover, we would like to point out that **it depends on the experimental settings and tasks whether the improvement of preferable log probability happens/is necessary or not**.

- (4) The log probability of winner samples in the preference dataset can achieve its maximum value after optimizing the maximum likelihood objective (i.e. SFT with winner responses). As done in DPO, if the policy is initialized with the SFT checkpoint with preferable responses, further improvement hardly happens because the initial checkpoint is already an almost optimal parameter in terms of winner response likelihood. As explained in Section 4 (L188) in the main text, our experiments have followed the procedure in the DPO paper, starting from SFT checkpoints with preferable responses.

- (5) Iterative RPO has started the experiments from LLaMA-2-70B-Chat checkpoint (Section 3 in the Iterative RPO paper). This is not finetuned with the winner's responses in the GSM8K dataset yet, which means that the policy has a sufficient margin to improve the preferable log-likelihood in the dataset. The maximum likelihood term in iterative RPO methods contributes to increasing the preferable log likelihood.

**(continues to the next thread)**
Thank you for the response. My question was not so much asking "does it currently work to use LLMs as judges" but, rather, what the impact of this can be in the future. It is, of course, expected that papers should consider what effects they may have on the world as a result of being published.
Thanks to the authors for providing a nice rebuttal!

A few followup questions:

> Please note that, because we adopt PaLM 2-XS as external reward models but offline soft preferences are provided from PaLM 2-L (the number of model parameters are quite different...

What are the parameter counts of these two models? What is their agreement rate?

> More architectures/models tried out

This is a bit unfortunate, as it would be interesting to see if the same performances scale to at least 7 or 8B parameter models, as using them has become quite common for small-scale evaluations. However, if the authors cannot produce these results due to compute constraints, that is understandable as well.

> What about DPO pushing down probabilities of both positive and negative examples

It seems the GDPO still pushes down probabilities of positives, albeit less compared to regular DPO. It is not clear to me, from the results, how the soft preference label/variation within it actually affects the results. Also it seems it does not mitigate the problem that RPO [1] does.

> the online performances do not reach the offline performance.

Based on both [2] and [3], online DPO > offline DPO. Are the authors claiming that this is not true (which would be an important finding)? What are the differences in setups that leads to this finding? Or are the authors claiming online DPO > offline DPO, but offline GDPO > online GDPO, possibly because of using a significantly smaller reward model? Clarifying this point would be important.

# Questions

Any possible extension to a reference model free version, similar to [4]? Conceptual explanation might suffice here. This is not an important question related to the paper, just for my personal curiosity.

# References

[1] Iterative Reasoning Preference Optimization, https://arxiv.org/abs/2404.19733

[2] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study, https://arxiv.org/abs/2404.10719

[3] Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data, https://arxiv.org/abs/2404.14367

[4] SimPO: Simple Preference Optimization with a Reference-Free Reward, https://arxiv.org/abs/2405.14734